From patchwork Wed Oct 28 07:24:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 11865111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 745EAC55179 for ; Thu, 29 Oct 2020 02:45:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0EEB02076A for ; Thu, 29 Oct 2020 02:45:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="Z1/KH/UW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726265AbgJ1Vdp (ORCPT ); Wed, 28 Oct 2020 17:33:45 -0400 Received: from mx2.suse.de ([195.135.220.15]:58260 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726229AbgJ1Vdn (ORCPT ); Wed, 28 Oct 2020 17:33:43 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1603869879; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sSxAAiojQg4WtgUjngHPfBfxoieX0dE2a75p6+YQCzQ=; b=Z1/KH/UW83xFFdpabGDkOODYgLN4NcB/iXGd+tfkfbB33zdN6X1+CGV7POrmEdS4+50wnA DIenvKWu8RdRYn/Q6DlTsRvwrVgvEc72woonk884+MUVJdBkssy/81n/VBHIuGAHF0PFqV PQy6oT8sEhibCfU7hdh8BCJ+FIuOrig= Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 3A771ADE4 for ; Wed, 28 Oct 2020 07:24:39 +0000 (UTC) From: Qu Wenruo To: linux-btrfs@vger.kernel.org Subject: [PATCH 1/3] btrfs: file-item: use nodesize to determine whether we need readhead for btrfs_lookup_bio_sums() Date: Wed, 28 Oct 2020 15:24:30 +0800 Message-Id: <20201028072432.86907-2-wqu@suse.com> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201028072432.86907-1-wqu@suse.com> References: <20201028072432.86907-1-wqu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org In btrfs_lookup_bio_sums() if the bio is pretty large, we want to readahead the csum tree. However the threshold is an immediate number, (PAGE_SIZE * 8), from the initial btrfs merge. The value itself is pretty hard to guess the meaning, especially when the immediate number is from the age where 4K sectorsize is the default and only CRC32 is supported. For the most common btrfs setup, CRC32 csum algorithme 4K sectorsize, it means just 32K read would kick readahead, while the csum itself is only 32 bytes in size. Now let's be more reasonable by taking both csum size and node size into consideration. If the csum size for the bio is larger than one node, then we kick the readahead. This means for current default btrfs, the threshold will be 16M. This change should not change performance observably, thus this is mostly a readability enhancement. Signed-off-by: Qu Wenruo Reviewed-by: Nikolay Borisov --- fs/btrfs/file-item.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/file-item.c b/fs/btrfs/file-item.c index 7d5ec71615b8..fbc60948b2c4 100644 --- a/fs/btrfs/file-item.c +++ b/fs/btrfs/file-item.c @@ -295,7 +295,11 @@ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, csum = dst; } - if (bio->bi_iter.bi_size > PAGE_SIZE * 8) + /* + * If needed csum size is larger than a node, kick the readahead for + * csum tree would be a good idea. + */ + if (nblocks * csum_size > fs_info->nodesize) path->reada = READA_FORWARD; /* From patchwork Wed Oct 28 07:24:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 11865113 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 980B1C4363A for ; Thu, 29 Oct 2020 02:45:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 45E742076B for ; Thu, 29 Oct 2020 02:45:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="XNZhPJfR" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726268AbgJ2CpY (ORCPT ); Wed, 28 Oct 2020 22:45:24 -0400 Received: from mx2.suse.de ([195.135.220.15]:58246 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726222AbgJ1Vdo (ORCPT ); Wed, 28 Oct 2020 17:33:44 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1603869881; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=73Q1XAdsFGmy0qR+eHlvhZ0IPSLSoj79+XTz1nqcVPY=; b=XNZhPJfR+uylo0I5bIVrgfonlivpwzciy3Iq21+YpDGbSHCTVDOb+dWllWjcIQgJ3/ShUD TpyecdyXpavCZswU9BMT6o2JnyLuBs6Za5SwXhu7kvM2STMkG4fuHIPpVgFdWVHdsxS6gX eTLDpldjPr3LPjmAuj8eURf0G2bzYDc= Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 0C41DADDF for ; Wed, 28 Oct 2020 07:24:41 +0000 (UTC) From: Qu Wenruo To: linux-btrfs@vger.kernel.org Subject: [PATCH 2/3] btrfs: ordered-data: rename parameter @len to @nr_sectors Date: Wed, 28 Oct 2020 15:24:31 +0800 Message-Id: <20201028072432.86907-3-wqu@suse.com> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201028072432.86907-1-wqu@suse.com> References: <20201028072432.86907-1-wqu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org The parameter is the number of sectors of the range to search. While most "len" we used in other locations are in byte size, this can lead to confusion. Rename @len to @nr_sectors to make it more clear and avoid confusion. Signed-off-by: Qu Wenruo Reviewed-by: Nikolay Borisov --- fs/btrfs/ordered-data.c | 9 ++++++--- fs/btrfs/ordered-data.h | 2 +- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/fs/btrfs/ordered-data.c b/fs/btrfs/ordered-data.c index ebac13389e7e..10c13f8a1603 100644 --- a/fs/btrfs/ordered-data.c +++ b/fs/btrfs/ordered-data.c @@ -802,9 +802,11 @@ btrfs_lookup_first_ordered_extent(struct inode *inode, u64 file_offset) * search the ordered extents for one corresponding to 'offset' and * try to find a checksum. This is used because we allow pages to * be reclaimed before their checksum is actually put into the btree + * + * @nr_sectors: The length of the search range, in sectors. */ int btrfs_find_ordered_sum(struct inode *inode, u64 offset, u64 disk_bytenr, - u8 *sum, int len) + u8 *sum, int nr_sectors) { struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); struct btrfs_ordered_sum *ordered_sum; @@ -828,12 +830,13 @@ int btrfs_find_ordered_sum(struct inode *inode, u64 offset, u64 disk_bytenr, inode->i_sb->s_blocksize_bits; num_sectors = ordered_sum->len >> inode->i_sb->s_blocksize_bits; - num_sectors = min_t(int, len - index, num_sectors - i); + num_sectors = min_t(int, nr_sectors - index, + num_sectors - i); memcpy(sum + index, ordered_sum->sums + i * csum_size, num_sectors * csum_size); index += (int)num_sectors * csum_size; - if (index == len) + if (index == nr_sectors) goto out; disk_bytenr += num_sectors * sectorsize; } diff --git a/fs/btrfs/ordered-data.h b/fs/btrfs/ordered-data.h index d61ea9c880a3..3ffc1c27ee30 100644 --- a/fs/btrfs/ordered-data.h +++ b/fs/btrfs/ordered-data.h @@ -175,7 +175,7 @@ struct btrfs_ordered_extent *btrfs_lookup_ordered_range( u64 file_offset, u64 len); int btrfs_find_ordered_sum(struct inode *inode, u64 offset, u64 disk_bytenr, - u8 *sum, int len); + u8 *sum, int nr_sectors); u64 btrfs_wait_ordered_extents(struct btrfs_root *root, u64 nr, const u64 range_start, const u64 range_len); void btrfs_wait_ordered_roots(struct btrfs_fs_info *fs_info, u64 nr, From patchwork Wed Oct 28 07:24:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 11865105 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 561CBC388F7 for ; Thu, 29 Oct 2020 02:45:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D7E022076A for ; Thu, 29 Oct 2020 02:45:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="P/gnJ80/" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729693AbgJ2CpT (ORCPT ); Wed, 28 Oct 2020 22:45:19 -0400 Received: from mx2.suse.de ([195.135.220.15]:58198 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726224AbgJ1Vdo (ORCPT ); Wed, 28 Oct 2020 17:33:44 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1603869882; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K4W88er9Sf1WrhRkZapVJY5WShpyacdDxDYHZuOCSm8=; b=P/gnJ80/LmSu8YxxaSYw0++9hsC323bWR43GDLr8T0MTrtNEFAYSgPbXLr1YPEH+6YA5g7 UsTovXEolwrKUE80R5mW/os8wTLhjWoo+5gEwZg5VQe8dZZY5uvCVXnpWz1yhz0O4ihzeV hYqj6IQGiRJC8WtyFadL82Ut76cr5qM= Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id B1D84AF22 for ; Wed, 28 Oct 2020 07:24:42 +0000 (UTC) From: Qu Wenruo To: linux-btrfs@vger.kernel.org Subject: [PATCH 3/3] btrfs: file-item: refactor btrfs_lookup_bio_sums() to handle out-of-order bvecs Date: Wed, 28 Oct 2020 15:24:32 +0800 Message-Id: <20201028072432.86907-4-wqu@suse.com> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201028072432.86907-1-wqu@suse.com> References: <20201028072432.86907-1-wqu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Refactor btrfs_lookup_bio_sums() by: - Extract the csum tree lookup into find_csum_tree_sums() The new function will handle the csum search in csum tree. The return value is the same as btrfs_find_ordered_sum(), returning the found number of sectors who has checksum. - Change how we do the main loop The parameter bio is in fact just a distraction. The only needed info from bio is: * the on-disk bytenr * the file_offset (if file_offset == (u64)-1) * the length After extracting above info, we can do the search without bio at all, which makes the main loop much simpler: for (cur_offset = orig_file_offset; cur_offset < orig_file_offset + orig_len; cur_offset += count * sectorsize) { /* Lookup ordered csum */ count = btrfs_find_ordered_sum(inode, cur_offset, cur_disk_bytenr, csum_dst, search_len / sectorsize); if (count) continue; /* Lookup csum tree */ count = find_csum_tree_sums(fs_info, path, cur_disk_bytenr, search_len, csum_dst); if (!count) { /* Csum hole handling */ } } - Use single variable as core to calculate all other offsets Instead of all differnt type of variables, we use only one core variable, cur_offset, which represents the current file offset. All involves values can be calculated from that core variable, and all those variable will only be visible in the inner loop. diff_sectors = (cur_offset - orig_file_offset) / sectorsize; cur_disk_bytenr = orig_disk_bytenr + diff_sectors * sectorsize; csum_dst = csum + diff_sectors * csum_size; - Fix bugs related to the bio iteration There are several hidden pitfalls related to the csum lookup. The biggest one is related to the bvec iteration. There are cases that the bvecs are not in linear bytenr order, like the following case with added debug info: btrfs_lookup_bio_sums: file_offset=-1 expected_bvec_offset=532480 page_offset=716800 bv_offset=0 btrfs_lookup_bio_sums: orig_file_offset=520192 bvec_index=3 root=259 ino=260 page_owner_ino=260 This is even more dangerous for subpage support, as for subpage case, we can have bvec with non-zero bv_offset, and if they get re-ordered, we can easily get incorrect csum skip and lead to false csum warning. Signed-off-by: Qu Wenruo --- fs/btrfs/file-item.c | 236 ++++++++++++++++++++++++++----------------- 1 file changed, 142 insertions(+), 94 deletions(-) diff --git a/fs/btrfs/file-item.c b/fs/btrfs/file-item.c index fbc60948b2c4..5f60ce6f227a 100644 --- a/fs/btrfs/file-item.c +++ b/fs/btrfs/file-item.c @@ -239,44 +239,117 @@ int btrfs_lookup_file_extent(struct btrfs_trans_handle *trans, return ret; } +/* + * Helper to find csums for logical bytenr range + * [disk_bytenr, disk_bytenr + len) and restore the result to @dst. + * + * Return >0 for the number of sectors we found. + * Return 0 for the range [disk_bytenr, disk_bytenr + sectorsize) has no csum + * for it. Caller may want to try next sector until one range is hit. + * Return <0 for fatal error. + */ +static int find_csum_tree_sums(struct btrfs_fs_info *fs_info, + struct btrfs_path *path, u64 disk_bytenr, + u64 len, u8 *dst) +{ + struct btrfs_csum_item *item = NULL; + struct btrfs_key key; + u32 csum_size = btrfs_super_csum_size(fs_info->super_copy); + u32 sectorsize = fs_info->sectorsize; + int ret; + u64 csum_start; + u64 csum_len; + + ASSERT(IS_ALIGNED(disk_bytenr, sectorsize) && + IS_ALIGNED(len, sectorsize)); + + /* Check if the current csum item covers disk_bytenr */ + if (path->nodes[0]) { + item = btrfs_item_ptr(path->nodes[0], path->slots[0], + struct btrfs_csum_item); + btrfs_item_key_to_cpu(path->nodes[0], &key, path->slots[0]); + csum_start = key.offset; + csum_len = btrfs_item_size_nr(path->nodes[0], path->slots[0]) / + csum_size * sectorsize; + + if (csum_start <= disk_bytenr && + csum_start + csum_len > disk_bytenr) + goto found; + } + + /* Current item doesn't contain the desired range, re-search */ + btrfs_release_path(path); + item = btrfs_lookup_csum(NULL, fs_info->csum_root, path, + disk_bytenr, 0); + if (IS_ERR(item)) { + ret = PTR_ERR(item); + goto out; + } +found: + btrfs_item_key_to_cpu(path->nodes[0], &key, path->slots[0]); + csum_start = key.offset; + csum_len = btrfs_item_size_nr(path->nodes[0], path->slots[0]) / + csum_size * sectorsize; + ASSERT(csum_start <= disk_bytenr && + csum_start + csum_len > disk_bytenr); + + ret = div_u64(min(csum_start + csum_len, disk_bytenr + len) - + disk_bytenr, sectorsize); + read_extent_buffer(path->nodes[0], dst, (unsigned long)item, + ret * csum_size); +out: + if (ret == -ENOENT) { + ret = 0; + memset(dst, 0, csum_size); + } + return ret; +} + /** * btrfs_lookup_bio_sums - Look up checksums for a bio. - * @inode: inode that the bio is for. - * @bio: bio to look up. - * @offset: Unless (u64)-1, look up checksums for this offset in the file. - * If (u64)-1, use the page offsets from the bio instead. - * @dst: Buffer of size nblocks * btrfs_super_csum_size() used to return - * checksum (nblocks = bio->bi_iter.bi_size / fs_info->sectorsize). If - * NULL, the checksum buffer is allocated and returned in - * btrfs_io_bio(bio)->csum instead. + * @inode: Inode that the bio is for. + * @bio: Bio to look up. + * NOTE: The bio is only used to determine the file_offset + * and length. + * @file_offset: File offset of the bio. + * If (u64)-1, will use the bio to determine the + * file offset. + * @dst: Csum destination. + * Should be at least (bio->bi_iter.bi_size / + * fs_info->sectorsize * csum_size) bytes in size. * * Return: BLK_STS_RESOURCE if allocating memory fails, BLK_STS_OK otherwise. */ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, - u64 offset, u8 *dst) + u64 file_offset, u8 *dst) { struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); - struct bio_vec bvec; - struct bvec_iter iter; - struct btrfs_csum_item *item = NULL; struct extent_io_tree *io_tree = &BTRFS_I(inode)->io_tree; struct btrfs_path *path; - const bool page_offsets = (offset == (u64)-1); + u32 csum_size = btrfs_super_csum_size(fs_info->super_copy); + u32 sectorsize = fs_info->sectorsize; + u64 orig_file_offset; + u64 orig_len; + u64 orig_disk_bytenr = bio->bi_iter.bi_sector << 9; + /* Current file offset, is used to calculate all other values */ + u64 cur_offset; u8 *csum; - u64 item_start_offset = 0; - u64 item_last_offset = 0; - u64 disk_bytenr; - u64 page_bytes_left; - u32 diff; int nblocks; int count = 0; - u16 csum_size = btrfs_super_csum_size(fs_info->super_copy); + + if (file_offset == (u64)-1) + orig_file_offset = page_offset(bio_first_page_all(bio)) + + bio_first_bvec_all(bio)->bv_offset; + else + orig_file_offset = file_offset; + + orig_len = bio->bi_iter.bi_size; + nblocks = orig_len >> inode->i_sb->s_blocksize_bits; path = btrfs_alloc_path(); if (!path) return BLK_STS_RESOURCE; - nblocks = bio->bi_iter.bi_size >> inode->i_sb->s_blocksize_bits; if (!dst) { struct btrfs_io_bio *btrfs_bio = btrfs_io_bio(bio); @@ -313,85 +386,60 @@ blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, path->skip_locking = 1; } - disk_bytenr = (u64)bio->bi_iter.bi_sector << 9; - - bio_for_each_segment(bvec, bio, iter) { - page_bytes_left = bvec.bv_len; - if (count) - goto next; - - if (page_offsets) - offset = page_offset(bvec.bv_page) + bvec.bv_offset; - count = btrfs_find_ordered_sum(inode, offset, disk_bytenr, - csum, nblocks); + /* + * In fact, for csum lookup we don't really need bio at all. + * + * We know the on-disk bytenr, the file_offset, and length. + * That's enough to search csum. The bio is in fact just a distraction + * and following bio bvec would make thing much hard to go. + * As we could have subpage bvec (with different bv_len) and non-linear + * bvec. + * + * So here we don't bother bio at all, just use @cur_offset to do the + * iteration. + */ + for (cur_offset = orig_file_offset; cur_offset < orig_file_offset + orig_len; + cur_offset += count * sectorsize) { + u64 cur_disk_bytenr; + int search_len = orig_file_offset + orig_len - cur_offset; + int diff_sectors; + u8 *csum_dst; + + diff_sectors = div_u64(cur_offset - orig_file_offset, + sectorsize); + cur_disk_bytenr = orig_disk_bytenr + + diff_sectors * sectorsize; + csum_dst = csum + diff_sectors * csum_size; + + count = btrfs_find_ordered_sum(inode, cur_offset, + cur_disk_bytenr, csum_dst, + search_len / sectorsize); if (count) - goto found; - - if (!item || disk_bytenr < item_start_offset || - disk_bytenr >= item_last_offset) { - struct btrfs_key found_key; - u32 item_size; - - if (item) - btrfs_release_path(path); - item = btrfs_lookup_csum(NULL, fs_info->csum_root, - path, disk_bytenr, 0); - if (IS_ERR(item)) { - count = 1; - memset(csum, 0, csum_size); - if (BTRFS_I(inode)->root->root_key.objectid == - BTRFS_DATA_RELOC_TREE_OBJECTID) { - set_extent_bits(io_tree, offset, - offset + fs_info->sectorsize - 1, - EXTENT_NODATASUM); - } else { - btrfs_info_rl(fs_info, - "no csum found for inode %llu start %llu", - btrfs_ino(BTRFS_I(inode)), offset); - } - item = NULL; - btrfs_release_path(path); - goto found; + continue; + count = find_csum_tree_sums(fs_info, path, cur_disk_bytenr, + search_len, csum_dst); + if (!count) { + /* + * For not found case, the csum has been zeroed + * in find_csum_tree_sums() already, just skip + * to next sector. + */ + count = 1; + if (BTRFS_I(inode)->root->root_key.objectid == + BTRFS_DATA_RELOC_TREE_OBJECTID) { + set_extent_bits(io_tree, cur_offset, + cur_offset + sectorsize - 1, + EXTENT_NODATASUM); + } else { + btrfs_warn_rl(fs_info, + "csum hole found for root %lld inode %llu range [%llu, %llu)", + BTRFS_I(inode)->root->root_key.objectid, + btrfs_ino(BTRFS_I(inode)), + cur_offset, cur_offset + sectorsize); } - btrfs_item_key_to_cpu(path->nodes[0], &found_key, - path->slots[0]); - - item_start_offset = found_key.offset; - item_size = btrfs_item_size_nr(path->nodes[0], - path->slots[0]); - item_last_offset = item_start_offset + - (item_size / csum_size) * - fs_info->sectorsize; - item = btrfs_item_ptr(path->nodes[0], path->slots[0], - struct btrfs_csum_item); - } - /* - * this byte range must be able to fit inside - * a single leaf so it will also fit inside a u32 - */ - diff = disk_bytenr - item_start_offset; - diff = diff / fs_info->sectorsize; - diff = diff * csum_size; - count = min_t(int, nblocks, (item_last_offset - disk_bytenr) >> - inode->i_sb->s_blocksize_bits); - read_extent_buffer(path->nodes[0], csum, - ((unsigned long)item) + diff, - csum_size * count); -found: - csum += count * csum_size; - nblocks -= count; -next: - while (count > 0) { - count--; - disk_bytenr += fs_info->sectorsize; - offset += fs_info->sectorsize; - page_bytes_left -= fs_info->sectorsize; - if (!page_bytes_left) - break; /* move to next bio */ } } - WARN_ON_ONCE(count); btrfs_free_path(path); return BLK_STS_OK; }