From patchwork Tue Jun 18 06:50:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 13701833 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 811CC210F8 for ; Tue, 18 Jun 2024 06:51:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718693478; cv=none; b=MYKQI5+LQcszKK8X+KyMkuv5Y387wwBqjJcTULrp7fpvuZmrbTGF9ij8MniYZ+22OIiSL96ejwQyCFFp9wt4MYAd2a9xsTEvHfRXpUzGjtAUdJ8OF15jL75ERE4/K23Q6hMjqkRTdQX9bPp+fzmP2hyi4xP+ZYDP4L2x20Qhp6g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718693478; c=relaxed/simple; bh=k12AEBGc7nCNBQXaOB46zdsCZMtUjziUStIyAsNstsY=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Mgf8pzmR33Uul24npLCmyvIVCb0ZRmuzSc86SWlyNu6ZJJo5kNmiewVT8H33XS59a8bj3Eh9I0dwhlw7vM4vtV3Y0wnUSVhnHFvcq+TRulsAiYPkz4TOxo839sjP594u6TlmegD8Wzg8VAUWrv4snnwMLAxWdIL5LQeeU5zWf+0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b=DCZgiweO; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b=DCZgiweO; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="DCZgiweO"; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="DCZgiweO" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B227E22805 for ; Tue, 18 Jun 2024 06:51:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1718693473; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hEleke7vIjYg16jnSyikfqs5Lm7/9hLfRPI6DcIelGI=; b=DCZgiweOzr+p0DKCat/idIliSretgmsFfGw6dstNZQ4KH9YaC5DTNIL5dQhbraG9u0eSAD lWFwJU6g8roUGPeDwzgoenPWqOdqJwPV7b9cwZlWBKLdj75AxsOkxbHHnaEqzbvuHCv0oi 6jhCWLHpiybCiw9uhgp3/JexhJhB89U= Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1718693473; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hEleke7vIjYg16jnSyikfqs5Lm7/9hLfRPI6DcIelGI=; b=DCZgiweOzr+p0DKCat/idIliSretgmsFfGw6dstNZQ4KH9YaC5DTNIL5dQhbraG9u0eSAD lWFwJU6g8roUGPeDwzgoenPWqOdqJwPV7b9cwZlWBKLdj75AxsOkxbHHnaEqzbvuHCv0oi 6jhCWLHpiybCiw9uhgp3/JexhJhB89U= Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id C69161369F for ; Tue, 18 Jun 2024 06:51:12 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id aIH3HmAucWZ+ZQAAD6G6ig (envelope-from ) for ; Tue, 18 Jun 2024 06:51:12 +0000 From: Qu Wenruo To: linux-btrfs@vger.kernel.org Subject: [PATCH 1/2] btrfs-progs: change-csum: fix the wrong metadata space reservation Date: Tue, 18 Jun 2024 16:20:48 +0930 Message-ID: <0a232d728355ecc03465933b70b2b3d682858820.1718693318.git.wqu@suse.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-btrfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Score: -2.80 X-Spam-Level: X-Spam-Flag: NO X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; NEURAL_HAM_LONG(-1.00)[-1.000]; MID_CONTAINS_FROM(1.00)[]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.com:email]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[linux-btrfs@vger.kernel.org]; RCVD_TLS_ALL(0.00)[] [BUG] `btrfstune --csum` would always fail for a newly created btrfs: # truncates -s 1G test.img # ./mkfs.btrfs -f test.img # ./btrsftune --csum xxhash test.img WARNING: Experimental build with unstable or unfinished features WARNING: Switching checksums is experimental, do not use for valuable data! Proceed to switch checksums ERROR: failed to start transaction: Unknown error -28 ERROR: failed to start transaction: No space left on device ERROR: failed to generate new data csums: No space left on device ERROR: btrfstune failed [CAUSE] After commit e79f18a4a75f ("btrfs-progs: introduce a basic metadata free space reservation check"), btrfs_start_transaction() would check the metadata space. But at the time of introduction of csum conversion, the parameter for btrfs_start_transaction() is incorrect. The 2nd parameter is the *number* of items to be added (if we're deleting items, just pass 1). However commit 08a3bd7694b6 ("btrfs-progs: tune: add the ability to generate new data checksums") is using the item size, not the number of items to be added. This means we're passing a number 8 * nodesize times larger than the original size, no wonder we would error out with -ENOSPC. [FIX] Use proper calcuation to convert the new csum item size to number of leaves needed, and double it just in case. Signed-off-by: Qu Wenruo --- tune/change-csum.c | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/tune/change-csum.c b/tune/change-csum.c index f5fc3c7f32cb..371e0aa76fc1 100644 --- a/tune/change-csum.c +++ b/tune/change-csum.c @@ -224,14 +224,26 @@ out: * item. */ #define CSUM_CHANGE_BYTES_THRESHOLD (SZ_2M) + +static unsigned int calc_csum_change_nr_items(struct btrfs_fs_info *fs_info, + u16 new_csum_type) +{ + const u32 new_csum_size = btrfs_csum_type_size(new_csum_type); + const u32 csum_item_size = CSUM_CHANGE_BYTES_THRESHOLD / + fs_info->sectorsize * new_csum_size; + + return round_up(csum_item_size, fs_info->nodesize) / fs_info->nodesize * 2; +} + static int generate_new_data_csums_range(struct btrfs_fs_info *fs_info, u64 start, u16 new_csum_type) { + const unsigned int nr_items = calc_csum_change_nr_items(fs_info, + new_csum_type); struct btrfs_root *csum_root = btrfs_csum_root(fs_info, 0); struct btrfs_trans_handle *trans; struct btrfs_path path = { 0 }; struct btrfs_key key; - const u32 new_csum_size = btrfs_csum_type_size(new_csum_type); void *csum_buffer; u64 converted_bytes = 0; u64 last_csum; @@ -248,9 +260,7 @@ static int generate_new_data_csums_range(struct btrfs_fs_info *fs_info, u64 star if (!csum_buffer) return -ENOMEM; - trans = btrfs_start_transaction(csum_root, - CSUM_CHANGE_BYTES_THRESHOLD / fs_info->sectorsize * - new_csum_size); + trans = btrfs_start_transaction(csum_root, nr_items); if (IS_ERR(trans)) { ret = PTR_ERR(trans); errno = -ret; @@ -306,9 +316,7 @@ static int generate_new_data_csums_range(struct btrfs_fs_info *fs_info, u64 star return -EUCLEAN; if (ret < 0) goto out; - trans = btrfs_start_transaction(csum_root, - CSUM_CHANGE_BYTES_THRESHOLD / - fs_info->sectorsize * new_csum_size); + trans = btrfs_start_transaction(csum_root, nr_items); if (IS_ERR(trans)) { ret = PTR_ERR(trans); goto out; From patchwork Tue Jun 18 06:50:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 13701834 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 44BF21B813 for ; Tue, 18 Jun 2024 06:51:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718693479; cv=none; b=YoQABu6NZVwAHcdxZEsL3WZtMJz2gHGXoxxTulSPrN8IamTbAxFiyPqELBlWfHw7ABDr2vGA0+PE0C5ApCE9YJiR+cvHmlAylegbNuvuKpFhTbW/7ZhL3M33awtGjz7F8hybmA8TfOa5jEpBbeNwSau6ea75FxLclPM0Wq9/5C0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718693479; c=relaxed/simple; bh=CNxnfNeXjv1PqwhvCh4O1l9b0GTx7x951s1SW7aE4nc=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UkZ4wYDq8QZ8sFBZjw3XCJu4iNhepEh0zYNz5JZHtN21t9qDqiDLLa3vWcnkmA0JkqwcF2hoZYeNsIxmpLEaDykroh5NDWVyFYYRHiyZg11G4NlNuWyrsDh9u62IjC/3O4Kw6VDvVMnlc9MS5I8gcHZlLq+UUNrcdjW7B2GzNpo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b=EM4r9zwJ; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b=EM4r9zwJ; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="EM4r9zwJ"; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="EM4r9zwJ" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 319CD1FFD6 for ; Tue, 18 Jun 2024 06:51:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1718693475; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g3d6M1Vdj3qC5u8lGCo9YAS3/0rOdpREN/0QPqz02E0=; b=EM4r9zwJQv2hVwmqcaWyXTG7NEy42X3dWB+WrRr63PWGXdyBWYB0QoBwT3T4xPqvBj3HCw /UqIo732JTJaf7yGTXJGpYRaREWgccP6XXh9a+G2te+boSnGcUzqJsOpD+qCIiVxN5H7r7 MrzMNlLA42SQfmiAHimFGoQRBzcleNQ= Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1718693475; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g3d6M1Vdj3qC5u8lGCo9YAS3/0rOdpREN/0QPqz02E0=; b=EM4r9zwJQv2hVwmqcaWyXTG7NEy42X3dWB+WrRr63PWGXdyBWYB0QoBwT3T4xPqvBj3HCw /UqIo732JTJaf7yGTXJGpYRaREWgccP6XXh9a+G2te+boSnGcUzqJsOpD+qCIiVxN5H7r7 MrzMNlLA42SQfmiAHimFGoQRBzcleNQ= Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 488B71369F for ; Tue, 18 Jun 2024 06:51:14 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id wOk8AGIucWZ+ZQAAD6G6ig (envelope-from ) for ; Tue, 18 Jun 2024 06:51:14 +0000 From: Qu Wenruo To: linux-btrfs@vger.kernel.org Subject: [PATCH 2/2] btrfs-progs: misc-tests: add a test case for basic csum conversion Date: Tue, 18 Jun 2024 16:20:49 +0930 Message-ID: <5a92af80237ac06efc5223a9f42094d51404761b.1718693318.git.wqu@suse.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-btrfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Score: -2.32 X-Spam-Level: X-Spam-Flag: NO X-Spamd-Result: default: False [-2.32 / 50.00]; BAYES_HAM(-2.52)[97.84%]; NEURAL_HAM_LONG(-1.00)[-1.000]; MID_CONTAINS_FROM(1.00)[]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.com:email]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[linux-btrfs@vger.kernel.org]; RCVD_TLS_ALL(0.00)[] The new test case would: - Create a btrfs with crc32c csum - Populate the fs - Convert the fs to the following csums: * xxhash * blake2 * sha256 * crc32c Signed-off-by: Qu Wenruo --- .../064-csum-conversion-basic/test.sh | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 tests/misc-tests/064-csum-conversion-basic/test.sh diff --git a/tests/misc-tests/064-csum-conversion-basic/test.sh b/tests/misc-tests/064-csum-conversion-basic/test.sh new file mode 100755 index 000000000000..2f8be0e9b324 --- /dev/null +++ b/tests/misc-tests/064-csum-conversion-basic/test.sh @@ -0,0 +1,32 @@ +#!/bin/bash +# +# Verify the csum conversion works as expected. +# +source "$TEST_TOP/common" || exit +source "$TEST_TOP/common.convert" || exit + +check_experimental_build +setup_root_helper +prepare_test_dev + +convert_to_csum() +{ + local new_csum=$1 + + run_check "$TOP/btrfstune" --csum "$new_csum" "$TEST_DEV" + run_check "$TOP/btrfs" check --check-data-csum "$TEST_DEV" +} + +run_check_mkfs_test_dev --csum crc32c + +# We only mount the fs once to populate its contents, +# later one we would never mount the fs (to reduce the dependency on +# kernel features). +run_check_mount_test_dev +populate_fs +run_check_umount_test_dev + +convert_to_csum xxhash +convert_to_csum blake2 +convert_to_csum sha256 +convert_to_csum crc32c