From patchwork Thu Jul 25 09:33:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058311 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3913B13A0 for ; Thu, 25 Jul 2019 09:34:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2889B28998 for ; Thu, 25 Jul 2019 09:34:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1C9E02899D; Thu, 25 Jul 2019 09:34:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AD5A228998 for ; Thu, 25 Jul 2019 09:34:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403772AbfGYJeK (ORCPT ); Thu, 25 Jul 2019 05:34:10 -0400 Received: from mx2.suse.de ([195.135.220.15]:52304 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2390575AbfGYJeK (ORCPT ); Thu, 25 Jul 2019 05:34:10 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 00A80AD8A for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 1/4] btrfs: turn checksum type define into a union Date: Thu, 25 Jul 2019 11:33:48 +0200 Message-Id: <6601e14425550304e1ba8b5317e74a5f806d6a34.1564046812.git.jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Turn the checksum type definition into a union. This eases later addition of new checksums. Signed-off-by: Johannes Thumshirn --- include/uapi/linux/btrfs_tree.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/include/uapi/linux/btrfs_tree.h b/include/uapi/linux/btrfs_tree.h index 34d5b34286fa..babef98181a2 100644 --- a/include/uapi/linux/btrfs_tree.h +++ b/include/uapi/linux/btrfs_tree.h @@ -300,7 +300,9 @@ #define BTRFS_CSUM_SIZE 32 /* csum types */ -#define BTRFS_CSUM_TYPE_CRC32 0 +enum btrfs_csum_type { + BTRFS_CSUM_TYPE_CRC32 = 0, +}; /* * flags definitions for directory entry item type From patchwork Thu Jul 25 09:33:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058315 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 070D017EF for ; Thu, 25 Jul 2019 09:34:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EB9BB28998 for ; Thu, 25 Jul 2019 09:34:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E02B12899B; Thu, 25 Jul 2019 09:34:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 76DBB28998 for ; Thu, 25 Jul 2019 09:34:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403795AbfGYJeL (ORCPT ); Thu, 25 Jul 2019 05:34:11 -0400 Received: from mx2.suse.de ([195.135.220.15]:52306 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2390576AbfGYJeK (ORCPT ); Thu, 25 Jul 2019 05:34:10 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 048A6AD9C for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 2/4] btrfs: create structure to encode checksum type and length Date: Thu, 25 Jul 2019 11:33:49 +0200 Message-Id: <944b685765a68c3389888159d3fe228c2e78eb22.1564046812.git.jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Create a structure to encode the type and length for the known on-disk checksums. Also add a table and a convenience macro for adding the checksum types to the table. This makes it easier to add new checksums later. Signed-off-by: Johannes Thumshirn --- fs/btrfs/ctree.h | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index da97ff10f421..099401f5dd47 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -82,9 +82,15 @@ struct btrfs_ref; */ #define BTRFS_LINK_MAX 65535U -/* four bytes for CRC32 */ -static const int btrfs_csum_sizes[] = { 4 }; -static const char *btrfs_csum_names[] = { "crc32c" }; +#define BTRFS_CHECKSUM_TYPE(_type, _size, _name) \ + [_type] = { .size = _size, .name = _name } + +static const struct btrfs_csums { + u16 size; + const char *name; +} btrfs_csums[] = { + BTRFS_CHECKSUM_TYPE(BTRFS_CSUM_TYPE_CRC32, 4, "crc32c"), +}; #define BTRFS_EMPTY_DIR_SIZE 0 @@ -2373,13 +2379,13 @@ static inline int btrfs_super_csum_size(const struct btrfs_super_block *s) /* * csum type is validated at mount time */ - return btrfs_csum_sizes[t]; + return btrfs_csums[t].size; } static inline const char *btrfs_super_csum_name(u16 csum_type) { /* csum type is validated at mount time */ - return btrfs_csum_names[csum_type]; + return btrfs_csums[csum_type].name; } /* From patchwork Thu Jul 25 09:33:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058313 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9CA2F112C for ; Thu, 25 Jul 2019 09:34:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8BE0028999 for ; Thu, 25 Jul 2019 09:34:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 800192899D; Thu, 25 Jul 2019 09:34:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0E1B128999 for ; Thu, 25 Jul 2019 09:34:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403792AbfGYJeL (ORCPT ); Thu, 25 Jul 2019 05:34:11 -0400 Received: from mx2.suse.de ([195.135.220.15]:52314 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2390579AbfGYJeK (ORCPT ); Thu, 25 Jul 2019 05:34:10 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 1557DAE1C for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 3/4] btrfs: use xxhash64 for checksumming Date: Thu, 25 Jul 2019 11:33:50 +0200 Message-Id: <4ac3332af1fa5ba7ecb92181329151382b800f3d.1564046812.git.jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Johannes Thumshirn --- fs/btrfs/Kconfig | 1 + fs/btrfs/ctree.h | 1 + fs/btrfs/disk-io.c | 1 + fs/btrfs/super.c | 1 + include/uapi/linux/btrfs_tree.h | 1 + 5 files changed, 5 insertions(+) diff --git a/fs/btrfs/Kconfig b/fs/btrfs/Kconfig index 38651fae7f21..6d5a01c57da3 100644 --- a/fs/btrfs/Kconfig +++ b/fs/btrfs/Kconfig @@ -5,6 +5,7 @@ config BTRFS_FS select CRYPTO select CRYPTO_CRC32C select LIBCRC32C + select CRYPTO_XXHASH select ZLIB_INFLATE select ZLIB_DEFLATE select LZO_COMPRESS diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 099401f5dd47..b34f22e55304 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -90,6 +90,7 @@ static const struct btrfs_csums { const char *name; } btrfs_csums[] = { BTRFS_CHECKSUM_TYPE(BTRFS_CSUM_TYPE_CRC32, 4, "crc32c"), + BTRFS_CHECKSUM_TYPE(BTRFS_CSUM_TYPE_XXHASH, 8, "xxhash64"), }; #define BTRFS_EMPTY_DIR_SIZE 0 diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 5f7ee70b3d1a..54a8ef489850 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -351,6 +351,7 @@ static bool btrfs_supported_super_csum(u16 csum_type) { switch (csum_type) { case BTRFS_CSUM_TYPE_CRC32: + case BTRFS_CSUM_TYPE_XXHASH: return true; default: return false; diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c index 78de9d5d80c6..7312f675d702 100644 --- a/fs/btrfs/super.c +++ b/fs/btrfs/super.c @@ -2464,3 +2464,4 @@ module_exit(exit_btrfs_fs) MODULE_LICENSE("GPL"); MODULE_SOFTDEP("pre: crc32c"); +MODULE_SOFTDEP("pre: xxhash64"); diff --git a/include/uapi/linux/btrfs_tree.h b/include/uapi/linux/btrfs_tree.h index babef98181a2..af4f5dec10b7 100644 --- a/include/uapi/linux/btrfs_tree.h +++ b/include/uapi/linux/btrfs_tree.h @@ -302,6 +302,7 @@ /* csum types */ enum btrfs_csum_type { BTRFS_CSUM_TYPE_CRC32 = 0, + BTRFS_CSUM_TYPE_XXHASH = 1, }; /* From patchwork Thu Jul 25 09:33:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058345 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 60AB313A0 for ; Thu, 25 Jul 2019 09:34:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4F23A28998 for ; Thu, 25 Jul 2019 09:34:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 43B592899B; Thu, 25 Jul 2019 09:34:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D04F028998 for ; Thu, 25 Jul 2019 09:34:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403765AbfGYJeK (ORCPT ); Thu, 25 Jul 2019 05:34:10 -0400 Received: from mx2.suse.de ([195.135.220.15]:52318 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2390581AbfGYJeK (ORCPT ); Thu, 25 Jul 2019 05:34:10 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 1C627AE65 for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 4/4] btrfs: sysfs: export supported checksums Date: Thu, 25 Jul 2019 11:33:51 +0200 Message-Id: X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: David Sterba Export supported checksum algorithms via sysfs. Signed-off-by: David Sterba Signed-off-by: Johannes Thumshirn --- fs/btrfs/sysfs.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c index 9539f8143b7a..920282a3452b 100644 --- a/fs/btrfs/sysfs.c +++ b/fs/btrfs/sysfs.c @@ -182,6 +182,30 @@ static umode_t btrfs_feature_visible(struct kobject *kobj, return mode; } +static ssize_t btrfs_checksums_show(struct kobject *kobj, + struct kobj_attribute *a, char *buf) +{ + ssize_t ret = 0; + int i; + + for (i = 0; i < ARRAY_SIZE(btrfs_csums); i++) { + ret += snprintf(buf + ret, PAGE_SIZE, "%s%s", + (i == 0 ? "" : ", "), + btrfs_csums[i].name); + + } + + ret += snprintf(buf + ret, PAGE_SIZE, "\n"); + return ret; +} + +static ssize_t btrfs_checksums_store(struct kobject *kobj, + struct kobj_attribute *a, + const char *buf, size_t count) +{ + return -EPERM; +} + BTRFS_FEAT_ATTR_INCOMPAT(mixed_backref, MIXED_BACKREF); BTRFS_FEAT_ATTR_INCOMPAT(default_subvol, DEFAULT_SUBVOL); BTRFS_FEAT_ATTR_INCOMPAT(mixed_groups, MIXED_GROUPS); @@ -195,6 +219,14 @@ BTRFS_FEAT_ATTR_INCOMPAT(no_holes, NO_HOLES); BTRFS_FEAT_ATTR_INCOMPAT(metadata_uuid, METADATA_UUID); BTRFS_FEAT_ATTR_COMPAT_RO(free_space_tree, FREE_SPACE_TREE); +static struct btrfs_feature_attr btrfs_attr_features_checksums_name = { + .kobj_attr = __INIT_KOBJ_ATTR(checksums, S_IRUGO, + btrfs_checksums_show, + btrfs_checksums_store), + .feature_set = FEAT_INCOMPAT, + .feature_bit = 0, +}; + static struct attribute *btrfs_supported_feature_attrs[] = { BTRFS_FEAT_ATTR_PTR(mixed_backref), BTRFS_FEAT_ATTR_PTR(default_subvol), @@ -208,6 +240,9 @@ static struct attribute *btrfs_supported_feature_attrs[] = { BTRFS_FEAT_ATTR_PTR(no_holes), BTRFS_FEAT_ATTR_PTR(metadata_uuid), BTRFS_FEAT_ATTR_PTR(free_space_tree), + + &btrfs_attr_features_checksums_name.kobj_attr.attr, + NULL }; From patchwork Thu Jul 25 09:33:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058331 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BB90413A0 for ; Thu, 25 Jul 2019 09:34:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ACBCA28999 for ; Thu, 25 Jul 2019 09:34:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A19E12899F; Thu, 25 Jul 2019 09:34:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 22C0328999 for ; Thu, 25 Jul 2019 09:34:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403780AbfGYJeT (ORCPT ); Thu, 25 Jul 2019 05:34:19 -0400 Received: from mx2.suse.de ([195.135.220.15]:52346 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2403758AbfGYJeL (ORCPT ); Thu, 25 Jul 2019 05:34:11 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 2CC56AE6D for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 05/17] btrfs-progs: add option for checksum type to mkfs Date: Thu, 25 Jul 2019 11:33:52 +0200 Message-Id: X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add an option to mkfs to specify which checksum algorithm will be used for the filesystem. XXX this patch should go last in the series. Signed-off-by: Johannes Thumshirn --- mkfs/common.c | 2 +- mkfs/common.h | 2 ++ mkfs/main.c | 21 ++++++++++++++++++++- 3 files changed, 23 insertions(+), 2 deletions(-) diff --git a/mkfs/common.c b/mkfs/common.c index caca5e707233..8249492704ad 100644 --- a/mkfs/common.c +++ b/mkfs/common.c @@ -201,7 +201,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) super.__unused_leafsize = cpu_to_le32(cfg->nodesize); btrfs_set_super_nodesize(&super, cfg->nodesize); btrfs_set_super_stripesize(&super, cfg->stripesize); - btrfs_set_super_csum_type(&super, BTRFS_CSUM_TYPE_CRC32); + btrfs_set_super_csum_type(&super, cfg->csum_type); btrfs_set_super_chunk_root_generation(&super, 1); btrfs_set_super_cache_generation(&super, -1); btrfs_set_super_incompat_flags(&super, cfg->features); diff --git a/mkfs/common.h b/mkfs/common.h index 28912906d0a9..384c1ef6a753 100644 --- a/mkfs/common.h +++ b/mkfs/common.h @@ -53,6 +53,8 @@ struct btrfs_mkfs_config { u64 features; /* Size of the filesystem in bytes */ u64 num_bytes; + /* checksum algorithm to use */ + u16 csum_type; /* Output fields, set during creation */ diff --git a/mkfs/main.c b/mkfs/main.c index 8dbec0717b89..3deff76045ba 100644 --- a/mkfs/main.c +++ b/mkfs/main.c @@ -346,6 +346,7 @@ static void print_usage(int ret) printf("\t--shrink (with --rootdir) shrink the filled filesystem to minimal size\n"); printf("\t-K|--nodiscard do not perform whole device TRIM\n"); printf("\t-f|--force force overwrite of existing filesystem\n"); + printf("\t-C|--checksum checksum algorithm to use (default: crc32c)\n"); printf(" general:\n"); printf("\t-q|--quiet no messages except errors\n"); printf("\t-V|--version print the mkfs.btrfs version and exit\n"); @@ -380,6 +381,18 @@ static u64 parse_profile(const char *s) return 0; } +static u16 parse_csum_type(const char *s) +{ + if (strcasecmp(s, "crc32c") == 0) { + return BTRFS_CSUM_TYPE_CRC32; + } else { + error("unknown csum type %s", s); + exit(1); + } + /* not reached */ + return 0; +} + static char *parse_label(const char *input) { int len = strlen(input); @@ -826,6 +839,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv) u64 features = BTRFS_MKFS_DEFAULT_FEATURES; struct mkfs_allocation allocation = { 0 }; struct btrfs_mkfs_config mkfs_cfg; + int csum_type = BTRFS_CSUM_TYPE_CRC32; crc32c_optimization_init(); @@ -835,6 +849,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv) static const struct option long_options[] = { { "alloc-start", required_argument, NULL, 'A'}, { "byte-count", required_argument, NULL, 'b' }, + { "checksum", required_argument, NULL, 'C' }, { "force", no_argument, NULL, 'f' }, { "leafsize", required_argument, NULL, 'l' }, { "label", required_argument, NULL, 'L'}, @@ -854,7 +869,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv) { NULL, 0, NULL, 0} }; - c = getopt_long(argc, argv, "A:b:fl:n:s:m:d:L:O:r:U:VMKq", + c = getopt_long(argc, argv, "A:b:C:fl:n:s:m:d:L:O:r:U:VMKq", long_options, NULL); if (c < 0) break; @@ -932,6 +947,9 @@ int BOX_MAIN(mkfs)(int argc, char **argv) case GETOPT_VAL_SHRINK: shrink_rootdir = true; break; + case 'C': + csum_type = parse_csum_type(optarg); + break; case GETOPT_VAL_HELP: default: print_usage(c != GETOPT_VAL_HELP); @@ -1170,6 +1188,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv) mkfs_cfg.sectorsize = sectorsize; mkfs_cfg.stripesize = stripesize; mkfs_cfg.features = features; + mkfs_cfg.csum_type = csum_type; ret = make_btrfs(fd, &mkfs_cfg); if (ret) { From patchwork Thu Jul 25 09:33:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058321 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D632B112C for ; Thu, 25 Jul 2019 09:34:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C795A28999 for ; Thu, 25 Jul 2019 09:34:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BC7492899D; Thu, 25 Jul 2019 09:34:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3712E28999 for ; Thu, 25 Jul 2019 09:34:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390575AbfGYJeN (ORCPT ); Thu, 25 Jul 2019 05:34:13 -0400 Received: from mx2.suse.de ([195.135.220.15]:52348 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2403759AbfGYJeL (ORCPT ); Thu, 25 Jul 2019 05:34:11 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 38D41AE82 for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 06/17] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size() Date: Thu, 25 Jul 2019 11:33:53 +0200 Message-Id: X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The callers of csum_tree_block_size() blindly assume we're only having crc32c as a possible checksum and thus pass in btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of csum_tree_block_size(). Signed-off-by: Johannes Thumshirn Reviewed-by: Nikolay Borisov --- mkfs/common.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mkfs/common.c b/mkfs/common.c index 8249492704ad..d63a9267bca3 100644 --- a/mkfs/common.c +++ b/mkfs/common.c @@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg, } /* generate checksum */ - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); /* write back root tree */ ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_ROOT_TREE]); @@ -292,7 +292,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]); btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -380,7 +380,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]); btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -420,7 +420,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]); btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -433,7 +433,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]); btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID); btrfs_set_header_nritems(buf, 0); - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -445,7 +445,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]); btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID); btrfs_set_header_nritems(buf, 0); - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -456,7 +456,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE); memcpy(buf->data, &super, sizeof(super)); buf->len = BTRFS_SUPER_INFO_SIZE; - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE, cfg->blocks[MKFS_SUPER_BLOCK]); if (ret != BTRFS_SUPER_INFO_SIZE) { From patchwork Thu Jul 25 09:33:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058335 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 375BC13A0 for ; Thu, 25 Jul 2019 09:34:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 287E028999 for ; Thu, 25 Jul 2019 09:34:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1D1472899F; Thu, 25 Jul 2019 09:34:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AF93828999 for ; Thu, 25 Jul 2019 09:34:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403814AbfGYJeW (ORCPT ); Thu, 25 Jul 2019 05:34:22 -0400 Received: from mx2.suse.de ([195.135.220.15]:52350 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2403776AbfGYJeL (ORCPT ); Thu, 25 Jul 2019 05:34:11 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 40429AEAC for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 07/17] btrfs-progs: use btrfs_csum_data() in __csum_tree_block_size() Date: Thu, 25 Jul 2019 11:33:54 +0200 Message-Id: X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Use the btrfs_csum_data() wrapper in __csum_tree_block_size() instead of directly calling crc32c(). This helps us when plumbing new checksum algorithms into the FS. Signed-off-by: Johannes Thumshirn --- disk-io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/disk-io.c b/disk-io.c index be44eead5cef..01314504a50a 100644 --- a/disk-io.c +++ b/disk-io.c @@ -156,7 +156,7 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, u32 crc = ~(u32)0; len = buf->len - BTRFS_CSUM_SIZE; - crc = crc32c(crc, buf->data + BTRFS_CSUM_SIZE, len); + crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, crc, len); btrfs_csum_final(crc, result); if (verify) { From patchwork Thu Jul 25 09:33:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058341 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9D60E112C for ; Thu, 25 Jul 2019 09:34:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8DDD928999 for ; Thu, 25 Jul 2019 09:34:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 827552899D; Thu, 25 Jul 2019 09:34:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 290F928999 for ; Thu, 25 Jul 2019 09:34:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403824AbfGYJeY (ORCPT ); Thu, 25 Jul 2019 05:34:24 -0400 Received: from mx2.suse.de ([195.135.220.15]:52352 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2403773AbfGYJeL (ORCPT ); Thu, 25 Jul 2019 05:34:11 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 4D6B5AEFF for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 08/17] btrfs-progs: cache csum_type in recover_control Date: Thu, 25 Jul 2019 11:33:55 +0200 Message-Id: <9f3f9dd60d4c7dbfc854ad98e6133914bb47f7dc.1564046972.git.jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Cache the super-block's checksum type field in 'struct recover_control'. This will be needed for further refactoring the checksum handling. Signed-off-by: Johannes Thumshirn Reviewed-by: Nikolay Borisov --- cmds/rescue-chunk-recover.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c index 608af9d49407..308731ea5ea6 100644 --- a/cmds/rescue-chunk-recover.c +++ b/cmds/rescue-chunk-recover.c @@ -47,6 +47,7 @@ struct recover_control { int yes; u16 csum_size; + u16 csum_type; u32 sectorsize; u32 nodesize; u64 generation; @@ -1530,6 +1531,7 @@ static int recover_prepare(struct recover_control *rc, const char *path) rc->generation = btrfs_super_generation(sb); rc->chunk_root_generation = btrfs_super_chunk_root_generation(sb); rc->csum_size = btrfs_super_csum_size(sb); + rc->csum_type = btrfs_super_csum_type(sb); /* if seed, the result of scanning below will be partial */ if (btrfs_super_flags(sb) & BTRFS_SUPER_FLAG_SEEDING) { From patchwork Thu Jul 25 09:33:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058337 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 80FC914DB for ; Thu, 25 Jul 2019 09:34:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 720EE28999 for ; Thu, 25 Jul 2019 09:34:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 66CD82899D; Thu, 25 Jul 2019 09:34:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EBDAC2899B for ; Thu, 25 Jul 2019 09:34:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403816AbfGYJeX (ORCPT ); Thu, 25 Jul 2019 05:34:23 -0400 Received: from mx2.suse.de ([195.135.220.15]:52354 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2403778AbfGYJeL (ORCPT ); Thu, 25 Jul 2019 05:34:11 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 59097AF1F for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 09/17] progs: pass in a btrfs_mkfs_config to write_temp_extent_buffer Date: Thu, 25 Jul 2019 11:33:56 +0200 Message-Id: <042f116c565320bc134a6bca5b5d91b0754a19a8.1564046972.git.jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Pass in a btrfs_mkfs_config to write_temp_extent_buffer(), this is needed so we can grab the checksum type for checksum buffer verification in later patches. Signed-off-by: Johannes Thumshirn Reviewed-by: Nikolay Borisov --- convert/common.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/convert/common.c b/convert/common.c index af4f8d372299..dea5f5b20d50 100644 --- a/convert/common.c +++ b/convert/common.c @@ -218,7 +218,8 @@ static void insert_temp_root_item(struct extent_buffer *buf, * Setup an extent buffer for tree block. */ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf, - u64 bytenr) + u64 bytenr, + struct btrfs_mkfs_config *cfg) { int ret; @@ -281,7 +282,7 @@ static int setup_temp_root_tree(int fd, struct btrfs_mkfs_config *cfg, insert_temp_root_item(buf, cfg, &slot, &itemoff, BTRFS_CSUM_TREE_OBJECTID, csum_bytenr); - ret = write_temp_extent_buffer(fd, buf, root_bytenr); + ret = write_temp_extent_buffer(fd, buf, root_bytenr, cfg); out: free(buf); return ret; @@ -456,7 +457,7 @@ static int setup_temp_chunk_tree(int fd, struct btrfs_mkfs_config *cfg, BTRFS_BLOCK_GROUP_METADATA); if (ret < 0) goto out; - ret = write_temp_extent_buffer(fd, buf, chunk_bytenr); + ret = write_temp_extent_buffer(fd, buf, chunk_bytenr, cfg); out: free(buf); @@ -515,7 +516,7 @@ static int setup_temp_dev_tree(int fd, struct btrfs_mkfs_config *cfg, BTRFS_MKFS_SYSTEM_GROUP_SIZE); insert_temp_dev_extent(buf, &slot, &itemoff, meta_chunk_start, BTRFS_CONVERT_META_GROUP_SIZE); - ret = write_temp_extent_buffer(fd, buf, dev_bytenr); + ret = write_temp_extent_buffer(fd, buf, dev_bytenr, cfg); out: free(buf); return ret; @@ -537,7 +538,7 @@ static int setup_temp_fs_tree(int fd, struct btrfs_mkfs_config *cfg, /* * Temporary fs tree is completely empty. */ - ret = write_temp_extent_buffer(fd, buf, fs_bytenr); + ret = write_temp_extent_buffer(fd, buf, fs_bytenr, cfg); out: free(buf); return ret; @@ -559,7 +560,7 @@ static int setup_temp_csum_tree(int fd, struct btrfs_mkfs_config *cfg, /* * Temporary csum tree is completely empty. */ - ret = write_temp_extent_buffer(fd, buf, csum_bytenr); + ret = write_temp_extent_buffer(fd, buf, csum_bytenr, cfg); out: free(buf); return ret; @@ -765,7 +766,7 @@ static int setup_temp_extent_tree(int fd, struct btrfs_mkfs_config *cfg, if (ret < 0) goto out; - ret = write_temp_extent_buffer(fd, buf, extent_bytenr); + ret = write_temp_extent_buffer(fd, buf, extent_bytenr, cfg); out: free(buf); return ret; From patchwork Thu Jul 25 09:33:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058319 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D8777112C for ; Thu, 25 Jul 2019 09:34:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C7BF728998 for ; Thu, 25 Jul 2019 09:34:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BC2972899B; Thu, 25 Jul 2019 09:34:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 034B128998 for ; Thu, 25 Jul 2019 09:34:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403806AbfGYJeN (ORCPT ); Thu, 25 Jul 2019 05:34:13 -0400 Received: from mx2.suse.de ([195.135.220.15]:52356 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2403783AbfGYJeM (ORCPT ); Thu, 25 Jul 2019 05:34:12 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 659E5AF37 for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 10/17] btrfs-progs: add checksum type to checksumming functions Date: Thu, 25 Jul 2019 11:33:57 +0200 Message-Id: <71245b857a7214b284a57989fe677101c2ea13e5.1564046972.git.jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add the checksum type to csum_tree_block_size(), __csum_tree_block_size() and verify_tree_block_csum_silent(). Signed-off-by: Johannes Thumshirn --- btrfs-corrupt-block.c | 3 ++- cmds/rescue-chunk-recover.c | 3 ++- convert/common.c | 3 ++- convert/main.c | 3 ++- disk-io.c | 21 ++++++++++++--------- disk-io.h | 5 +++-- mkfs/common.c | 21 ++++++++++++++------- 7 files changed, 37 insertions(+), 22 deletions(-) diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c index bbef0c02e5d1..1dde9594bdcc 100644 --- a/btrfs-corrupt-block.c +++ b/btrfs-corrupt-block.c @@ -158,7 +158,8 @@ static void corrupt_keys(struct btrfs_trans_handle *trans, if (!trans) { u16 csum_size = btrfs_super_csum_size(fs_info->super_copy); - csum_tree_block_size(eb, csum_size, 0); + u16 csum_type = btrfs_super_csum_type(fs_info->super_copy); + csum_tree_block_size(eb, csum_size, 0, csum_type); write_extent_to_disk(eb); } } diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c index 308731ea5ea6..1a368310d895 100644 --- a/cmds/rescue-chunk-recover.c +++ b/cmds/rescue-chunk-recover.c @@ -768,7 +768,8 @@ static int scan_one_device(void *dev_scan_struct) continue; } - if (verify_tree_block_csum_silent(buf, rc->csum_size)) { + if (verify_tree_block_csum_silent(buf, rc->csum_size, + rc->csum_type)) { bytenr += rc->sectorsize; continue; } diff --git a/convert/common.c b/convert/common.c index dea5f5b20d50..f8bbb23cba89 100644 --- a/convert/common.c +++ b/convert/common.c @@ -223,7 +223,8 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf, { int ret; - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); /* Temporary extent buffer is always mapped 1:1 on disk */ ret = pwrite(fd, buf->data, buf->len, bytenr); diff --git a/convert/main.c b/convert/main.c index 9711874bd137..5e6b12431f59 100644 --- a/convert/main.c +++ b/convert/main.c @@ -1058,7 +1058,8 @@ static int migrate_super_block(int fd, u64 old_bytenr) BUG_ON(btrfs_super_bytenr(super) != old_bytenr); btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET); - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0, + btrfs_super_csum_type(super)); ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE, BTRFS_SUPER_INFO_OFFSET); if (ret != BTRFS_SUPER_INFO_SIZE) diff --git a/disk-io.c b/disk-io.c index 01314504a50a..a4995a628210 100644 --- a/disk-io.c +++ b/disk-io.c @@ -149,7 +149,7 @@ void btrfs_csum_final(u32 crc, u8 *result) } static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, - int verify, int silent) + int verify, int silent, u16 csum_type) { u8 result[BTRFS_CSUM_SIZE]; u32 len; @@ -174,24 +174,27 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, return 0; } -int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify) +int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify, + u16 csum_type) { - return __csum_tree_block_size(buf, csum_size, verify, 0); + return __csum_tree_block_size(buf, csum_size, verify, 0, csum_type); } -int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size) +int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size, + u16 csum_type) { - return __csum_tree_block_size(buf, csum_size, 1, 1); + return __csum_tree_block_size(buf, csum_size, 1, 1, csum_type); } int csum_tree_block(struct btrfs_fs_info *fs_info, struct extent_buffer *buf, int verify) { - u16 csum_size = - btrfs_super_csum_size(fs_info->super_copy); + u16 csum_size = btrfs_super_csum_size(fs_info->super_copy); + u16 csum_type = btrfs_super_csum_type(fs_info->super_copy); + if (verify && fs_info->suppress_check_block_errors) - return verify_tree_block_csum_silent(buf, csum_size); - return csum_tree_block_size(buf, csum_size, verify); + return verify_tree_block_csum_silent(buf, csum_size, csum_type); + return csum_tree_block_size(buf, csum_size, verify, csum_type); } struct extent_buffer *btrfs_find_tree_block(struct btrfs_fs_info *fs_info, diff --git a/disk-io.h b/disk-io.h index 7b5c3806ba98..394997ad72cb 100644 --- a/disk-io.h +++ b/disk-io.h @@ -191,8 +191,9 @@ void btrfs_csum_final(u32 crc, u8 *result); int btrfs_open_device(struct btrfs_device *dev); int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize, - int verify); -int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size); + int verify, u16 csum_type); +int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size, + u16 csum_type); int btrfs_read_buffer(struct extent_buffer *buf, u64 parent_transid); int write_tree_block(struct btrfs_trans_handle *trans, struct btrfs_fs_info *fs_info, diff --git a/mkfs/common.c b/mkfs/common.c index d63a9267bca3..4a417bd7a306 100644 --- a/mkfs/common.c +++ b/mkfs/common.c @@ -101,7 +101,8 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg, } /* generate checksum */ - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); /* write back root tree */ ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_ROOT_TREE]); @@ -292,7 +293,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]); btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -380,7 +382,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]); btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -420,7 +423,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]); btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -433,7 +437,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]); btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID); btrfs_set_header_nritems(buf, 0); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -445,7 +450,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]); btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID); btrfs_set_header_nritems(buf, 0); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -456,7 +462,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE); memcpy(buf->data, &super, sizeof(super)); buf->len = BTRFS_SUPER_INFO_SIZE; - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE, cfg->blocks[MKFS_SUPER_BLOCK]); if (ret != BTRFS_SUPER_INFO_SIZE) { From patchwork Thu Jul 25 09:33:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058333 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 19F7E112C for ; Thu, 25 Jul 2019 09:34:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0985A28999 for ; Thu, 25 Jul 2019 09:34:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F1D432899D; Thu, 25 Jul 2019 09:34:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5ECF02899B for ; Thu, 25 Jul 2019 09:34:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389563AbfGYJeT (ORCPT ); Thu, 25 Jul 2019 05:34:19 -0400 Received: from mx2.suse.de ([195.135.220.15]:52358 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2403780AbfGYJeL (ORCPT ); Thu, 25 Jul 2019 05:34:11 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 72931AF40 for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 11/17] btrfs-progs: don't assume checksums are always 4 bytes Date: Thu, 25 Jul 2019 11:33:58 +0200 Message-Id: X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Johannes Thumshirn --- btrfs-sb-mod.c | 9 ++++++--- check/main.c | 2 +- cmds/inspect-dump-super.c | 3 ++- cmds/rescue-chunk-recover.c | 2 +- convert/common.c | 3 ++- disk-io.c | 12 ++++++------ disk-io.h | 2 +- file-item.c | 2 +- 8 files changed, 20 insertions(+), 15 deletions(-) diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c index 16a26f772494..932c2a0432ef 100644 --- a/btrfs-sb-mod.c +++ b/btrfs-sb-mod.c @@ -37,7 +37,8 @@ static int check_csum_superblock(void *sb) u32 crc = ~(u32)0; crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, - crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); + (u8 *)&crc, + BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); btrfs_csum_final(crc, result); return !memcmp(sb, &result, csum_size); @@ -50,10 +51,12 @@ static void update_block_csum(void *block, int is_sb) u32 crc = ~(u32)0; if (is_sb) { - crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, crc, + crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, + (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); } else { - crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, crc, + crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, + (u8 *)&crc, BLOCKSIZE - BTRFS_CSUM_SIZE); } btrfs_csum_final(crc, result); diff --git a/check/main.c b/check/main.c index bb57933b83fc..92a7e40c1ebf 100644 --- a/check/main.c +++ b/check/main.c @@ -5622,7 +5622,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr, tmp = offset + data_checked; csum = btrfs_csum_data((char *)data + tmp, - csum, fs_info->sectorsize); + (u8 *)&csum, fs_info->sectorsize); btrfs_csum_final(csum, (u8 *)&csum); csum_offset = leaf_offset + diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c index 65fb3506eac6..96ad3deca3d8 100644 --- a/cmds/inspect-dump-super.c +++ b/cmds/inspect-dump-super.c @@ -41,7 +41,8 @@ static int check_csum_sblock(void *sb, int csum_size) u32 crc = ~(u32)0; crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, - crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); + (u8 *)&crc, + BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); btrfs_csum_final(crc, result); return !memcmp(sb, &result, csum_size); diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c index 1a368310d895..0fddb5dd8ead 100644 --- a/cmds/rescue-chunk-recover.c +++ b/cmds/rescue-chunk-recover.c @@ -1902,7 +1902,7 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum) goto out; } ret = 0; - csum_result = btrfs_csum_data(data, csum_result, len); + csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len); btrfs_csum_final(csum_result, (u8 *)&csum_result); if (csum_result != tree_csum) ret = 1; diff --git a/convert/common.c b/convert/common.c index f8bbb23cba89..ab8e6b9f4749 100644 --- a/convert/common.c +++ b/convert/common.c @@ -65,7 +65,8 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb, u32 crc = ~(u32)0; int ret; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc, + crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, + (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); btrfs_csum_final(crc, &sb->csum[0]); ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr); diff --git a/disk-io.c b/disk-io.c index a4995a628210..a0c37c569d58 100644 --- a/disk-io.c +++ b/disk-io.c @@ -138,9 +138,9 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info, } } -u32 btrfs_csum_data(char *data, u32 seed, size_t len) +u32 btrfs_csum_data(char *data, u8 *seed, size_t len) { - return crc32c(seed, data, len); + return crc32c(*(u32*)seed, data, len); } void btrfs_csum_final(u32 crc, u8 *result) @@ -156,7 +156,7 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, u32 crc = ~(u32)0; len = buf->len - BTRFS_CSUM_SIZE; - crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, crc, len); + crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len); btrfs_csum_final(crc, result); if (verify) { @@ -1376,7 +1376,7 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags) csum_size = btrfs_csum_sizes[csum_type]; crc = ~(u32)0; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc, + crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); btrfs_csum_final(crc, result); @@ -1631,7 +1631,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) { btrfs_set_super_bytenr(sb, fs_info->super_bytenr); crc = ~(u32)0; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc, + crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); btrfs_csum_final(crc, &sb->csum[0]); @@ -1667,7 +1667,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, btrfs_set_super_bytenr(sb, bytenr); crc = ~(u32)0; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc, + crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); btrfs_csum_final(crc, &sb->csum[0]); diff --git a/disk-io.h b/disk-io.h index 394997ad72cb..92c87f28f8b2 100644 --- a/disk-io.h +++ b/disk-io.h @@ -186,7 +186,7 @@ int btrfs_free_fs_root(struct btrfs_root *root); void btrfs_mark_buffer_dirty(struct extent_buffer *buf); int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid); int btrfs_set_buffer_uptodate(struct extent_buffer *buf); -u32 btrfs_csum_data(char *data, u32 seed, size_t len); +u32 btrfs_csum_data(char *data, u8 *seed, size_t len); void btrfs_csum_final(u32 crc, u8 *result); int btrfs_open_device(struct btrfs_device *dev); diff --git a/file-item.c b/file-item.c index 3bf48c68913d..5f6548e9a74f 100644 --- a/file-item.c +++ b/file-item.c @@ -312,7 +312,7 @@ csum: item = (struct btrfs_csum_item *)((unsigned char *)item + csum_offset * csum_size); found: - csum_result = btrfs_csum_data(data, csum_result, len); + csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len); btrfs_csum_final(csum_result, (u8 *)&csum_result); if (csum_result == 0) { printk("csum result is 0 for block %llu\n", From patchwork Thu Jul 25 09:33:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058323 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7671513A0 for ; Thu, 25 Jul 2019 09:34:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5FDAB28999 for ; Thu, 25 Jul 2019 09:34:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 547032899E; Thu, 25 Jul 2019 09:34:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7328028999 for ; Thu, 25 Jul 2019 09:34:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403810AbfGYJeQ (ORCPT ); Thu, 25 Jul 2019 05:34:16 -0400 Received: from mx2.suse.de ([195.135.220.15]:52360 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2403784AbfGYJeM (ORCPT ); Thu, 25 Jul 2019 05:34:12 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 83B9BAF5B for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 12/17] btrfs-progs: pass checksum type to btrfs_csum_data() Date: Thu, 25 Jul 2019 11:33:59 +0200 Message-Id: X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Johannes Thumshirn --- check/main.c | 6 ++++-- cmds/inspect-dump-super.c | 8 ++++---- cmds/rescue-chunk-recover.c | 10 ++++++---- convert/common.c | 5 +++-- disk-io.c | 39 +++++++++++++++++++++++++++------------ disk-io.h | 4 ++-- file-item.c | 7 +++++-- free-space-cache.c | 2 +- image/main.c | 2 +- 9 files changed, 53 insertions(+), 30 deletions(-) diff --git a/check/main.c b/check/main.c index 92a7e40c1ebf..eabe328b6800 100644 --- a/check/main.c +++ b/check/main.c @@ -5581,6 +5581,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr, struct btrfs_fs_info *fs_info = root->fs_info; u64 offset = 0; u16 csum_size = btrfs_super_csum_size(fs_info->super_copy); + u16 csum_type = btrfs_super_csum_type(fs_info->super_copy); char *data; unsigned long csum_offset; u32 csum; @@ -5621,9 +5622,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr, csum = ~(u32)0; tmp = offset + data_checked; - csum = btrfs_csum_data((char *)data + tmp, + csum = btrfs_csum_data(csum_type, + (char *)data + tmp, (u8 *)&csum, fs_info->sectorsize); - btrfs_csum_final(csum, (u8 *)&csum); + btrfs_csum_final(csum_type, csum, (u8 *)&csum); csum_offset = leaf_offset + tmp / fs_info->sectorsize * csum_size; diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c index 96ad3deca3d8..40019a6670ef 100644 --- a/cmds/inspect-dump-super.c +++ b/cmds/inspect-dump-super.c @@ -35,15 +35,15 @@ #include "kernel-lib/crc32c.h" #include "common/help.h" -static int check_csum_sblock(void *sb, int csum_size) +static int check_csum_sblock(void *sb, int csum_size, u16 csum_type) { u8 result[BTRFS_CSUM_SIZE]; u32 crc = ~(u32)0; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, + crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(crc, result); + btrfs_csum_final(csum_type, crc, result); return !memcmp(sb, &result, csum_size); } @@ -348,7 +348,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full) if (csum_type != BTRFS_CSUM_TYPE_CRC32 || csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]) printf(" [UNKNOWN CSUM TYPE OR SIZE]"); - else if (check_csum_sblock(sb, csum_size)) + else if (check_csum_sblock(sb, csum_size, csum_type)) printf(" [match]"); else printf(" [DON'T MATCH]"); diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c index 0fddb5dd8ead..cddae471f50c 100644 --- a/cmds/rescue-chunk-recover.c +++ b/cmds/rescue-chunk-recover.c @@ -1887,7 +1887,8 @@ static u64 calc_data_offset(struct btrfs_key *key, return dev_offset + data_offset; } -static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum) +static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum, + u16 csum_type) { char *data; int ret = 0; @@ -1902,8 +1903,8 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum) goto out; } ret = 0; - csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len); - btrfs_csum_final(csum_result, (u8 *)&csum_result); + csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len); + btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result); if (csum_result != tree_csum) ret = 1; out: @@ -2102,7 +2103,8 @@ next_csum: devext->objectid, 1)); ret = check_one_csum(dev->fd, data_offset, blocksize, - tree_csum); + tree_csum, + btrfs_super_csum_type(root->fs_info->super_copy)); if (ret < 0) goto fail_out; else if (ret > 0) diff --git a/convert/common.c b/convert/common.c index ab8e6b9f4749..894a6ee0ba90 100644 --- a/convert/common.c +++ b/convert/common.c @@ -63,12 +63,13 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb, u64 sb_bytenr) { u32 crc = ~(u32)0; + u16 csum_type = btrfs_super_csum_type(sb); int ret; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, + crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(crc, &sb->csum[0]); + btrfs_csum_final(csum_type, crc, &sb->csum[0]); ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr); if (ret < BTRFS_SUPER_INFO_SIZE) ret = (ret < 0 ? -errno : -EIO); diff --git a/disk-io.c b/disk-io.c index a0c37c569d58..7e538969c57a 100644 --- a/disk-io.c +++ b/disk-io.c @@ -138,14 +138,26 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info, } } -u32 btrfs_csum_data(char *data, u8 *seed, size_t len) +u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len) { - return crc32c(*(u32*)seed, data, len); + switch (csum_type) { + case BTRFS_CSUM_TYPE_CRC32: + return crc32c(*(u32*)seed, data, len); + default: /* Not reached */ + return ~(u32)0; + } + } -void btrfs_csum_final(u32 crc, u8 *result) +void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result) { - put_unaligned_le32(~crc, result); + switch (csum_type) { + case BTRFS_CSUM_TYPE_CRC32: + put_unaligned_le32(~crc, result); + break; + default: /* Not reached */ + break; + } } static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, @@ -156,8 +168,9 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, u32 crc = ~(u32)0; len = buf->len - BTRFS_CSUM_SIZE; - crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len); - btrfs_csum_final(crc, result); + crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE, + (u8 *)&crc, len); + btrfs_csum_final(csum_type, crc, result); if (verify) { if (memcmp_extent_buffer(buf, result, 0, csum_size)) { @@ -1376,9 +1389,10 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags) csum_size = btrfs_csum_sizes[csum_type]; crc = ~(u32)0; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, + crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE, + (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(crc, result); + btrfs_csum_final(csum_type, crc, result); if (memcmp(result, sb->csum, csum_size)) { error("superblock checksum mismatch"); @@ -1616,6 +1630,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, u64 bytenr; u32 crc; int i, ret; + u16 csum_type = btrfs_super_csum_type(sb); /* * We need to write super block after all metadata written. @@ -1631,9 +1646,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) { btrfs_set_super_bytenr(sb, fs_info->super_bytenr); crc = ~(u32)0; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, + crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(crc, &sb->csum[0]); + btrfs_csum_final(csum_type, crc, &sb->csum[0]); /* * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is @@ -1667,9 +1682,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, btrfs_set_super_bytenr(sb, bytenr); crc = ~(u32)0; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, + crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(crc, &sb->csum[0]); + btrfs_csum_final(csum_type, crc, &sb->csum[0]); /* * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is diff --git a/disk-io.h b/disk-io.h index 92c87f28f8b2..4b5e9ea86385 100644 --- a/disk-io.h +++ b/disk-io.h @@ -186,8 +186,8 @@ int btrfs_free_fs_root(struct btrfs_root *root); void btrfs_mark_buffer_dirty(struct extent_buffer *buf); int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid); int btrfs_set_buffer_uptodate(struct extent_buffer *buf); -u32 btrfs_csum_data(char *data, u8 *seed, size_t len); -void btrfs_csum_final(u32 crc, u8 *result); +u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len); +void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result); int btrfs_open_device(struct btrfs_device *dev); int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize, diff --git a/file-item.c b/file-item.c index 5f6548e9a74f..78fdcecd0bab 100644 --- a/file-item.c +++ b/file-item.c @@ -202,6 +202,9 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans, u16 csum_size = btrfs_super_csum_size(root->fs_info->super_copy); + u16 csum_type = + btrfs_super_csum_type(root->fs_info->super_copy); + path = btrfs_alloc_path(); if (!path) return -ENOMEM; @@ -312,8 +315,8 @@ csum: item = (struct btrfs_csum_item *)((unsigned char *)item + csum_offset * csum_size); found: - csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len); - btrfs_csum_final(csum_result, (u8 *)&csum_result); + csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len); + btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result); if (csum_result == 0) { printk("csum result is 0 for block %llu\n", (unsigned long long)bytenr); diff --git a/free-space-cache.c b/free-space-cache.c index e872eb6a00db..8a57f86dc650 100644 --- a/free-space-cache.c +++ b/free-space-cache.c @@ -213,7 +213,7 @@ static int io_ctl_check_crc(struct io_ctl *io_ctl, int index) io_ctl_map_page(io_ctl, 0); crc = crc32c(crc, io_ctl->orig + offset, io_ctl->root->fs_info->sectorsize - offset); - btrfs_csum_final(crc, (u8 *)&crc); + put_unaligned_le32(~crc, (u8 *)&crc); if (val != crc) { printk("btrfs: csum mismatch on free space cache\n"); io_ctl_unmap_page(io_ctl); diff --git a/image/main.c b/image/main.c index 28ef1609b5ff..0c8ffede56f5 100644 --- a/image/main.c +++ b/image/main.c @@ -124,7 +124,7 @@ static void csum_block(u8 *buf, size_t len) u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]]; u32 crc = ~(u32)0; crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE); - btrfs_csum_final(crc, result); + put_unaligned_le32(~crc, result); memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]); } From patchwork Thu Jul 25 09:34:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058343 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 52DBD13A0 for ; Thu, 25 Jul 2019 09:34:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 41FD928999 for ; Thu, 25 Jul 2019 09:34:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 367912899D; Thu, 25 Jul 2019 09:34:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C7F9628999 for ; Thu, 25 Jul 2019 09:34:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403827AbfGYJeZ (ORCPT ); Thu, 25 Jul 2019 05:34:25 -0400 Received: from mx2.suse.de ([195.135.220.15]:52362 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2403782AbfGYJeL (ORCPT ); Thu, 25 Jul 2019 05:34:11 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 89B2DAFF9 for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 13/17] btrfs-progs: pass checksum type to btrfs_csum_data() Date: Thu, 25 Jul 2019 11:34:00 +0200 Message-Id: X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Johannes Thumshirn --- btrfs-sb-mod.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c index 932c2a0432ef..d9630f187d0f 100644 --- a/btrfs-sb-mod.c +++ b/btrfs-sb-mod.c @@ -35,11 +35,13 @@ static int check_csum_superblock(void *sb) { u8 result[csum_size]; u32 crc = ~(u32)0; + u16 csum_type = btrfs_super_csum_type(sb); - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, + crc = btrfs_csum_data(csum_type, + (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(crc, result); + btrfs_csum_final(csum_type, crc, result); return !memcmp(sb, &result, csum_size); } @@ -49,17 +51,20 @@ static void update_block_csum(void *block, int is_sb) u8 result[csum_size]; struct btrfs_header *hdr; u32 crc = ~(u32)0; + u16 csum_type = btrfs_super_csum_type(block); if (is_sb) { - crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, + crc = btrfs_csum_data(csum_type, + (char *)block + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); } else { - crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, + crc = btrfs_csum_data(csum_type, + (char *)block + BTRFS_CSUM_SIZE, (u8 *)&crc, BLOCKSIZE - BTRFS_CSUM_SIZE); } - btrfs_csum_final(crc, result); + btrfs_csum_final(csum_type, crc, result); memset(block, 0, BTRFS_CSUM_SIZE); hdr = (struct btrfs_header *)block; memcpy(&hdr->csum, result, csum_size); From patchwork Thu Jul 25 09:34:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058339 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B4886112C for ; Thu, 25 Jul 2019 09:34:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A2DBA28999 for ; Thu, 25 Jul 2019 09:34:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 976B92899B; Thu, 25 Jul 2019 09:34:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 33A66289A0 for ; Thu, 25 Jul 2019 09:34:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403820AbfGYJeX (ORCPT ); Thu, 25 Jul 2019 05:34:23 -0400 Received: from mx2.suse.de ([195.135.220.15]:52364 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2403781AbfGYJeL (ORCPT ); Thu, 25 Jul 2019 05:34:11 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 9704AAFFE for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 14/17] btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c Date: Thu, 25 Jul 2019 11:34:01 +0200 Message-Id: X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP update_block_csum() in btrfs-sb-mod.c is always called with the 'is_sb' argument set to 1. Get rid of the special case for is_sb == 0. Signed-off-by: Johannes Thumshirn Reviewed-by: Nikolay Borisov --- btrfs-sb-mod.c | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c index d9630f187d0f..9e64b34d2b8f 100644 --- a/btrfs-sb-mod.c +++ b/btrfs-sb-mod.c @@ -46,24 +46,17 @@ static int check_csum_superblock(void *sb) return !memcmp(sb, &result, csum_size); } -static void update_block_csum(void *block, int is_sb) +static void update_block_csum(void *block) { u8 result[csum_size]; struct btrfs_header *hdr; u32 crc = ~(u32)0; u16 csum_type = btrfs_super_csum_type(block); - if (is_sb) { - crc = btrfs_csum_data(csum_type, - (char *)block + BTRFS_CSUM_SIZE, - (u8 *)&crc, - BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - } else { - crc = btrfs_csum_data(csum_type, - (char *)block + BTRFS_CSUM_SIZE, - (u8 *)&crc, - BLOCKSIZE - BTRFS_CSUM_SIZE); - } + crc = btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE, + (u8 *)&crc, + BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); + btrfs_csum_final(csum_type, crc, result); memset(block, 0, BTRFS_CSUM_SIZE); hdr = (struct btrfs_header *)block; @@ -354,7 +347,7 @@ int main(int argc, char **argv) { if (changed) { printf("Update csum\n"); - update_block_csum(buf, 1); + update_block_csum(buf); ret = pwrite(fd, buf, BLOCKSIZE, off); if (ret <= 0) { printf("pwrite error %d at offset %llu\n", From patchwork Thu Jul 25 09:34:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058327 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2DD04112C for ; Thu, 25 Jul 2019 09:34:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1CB7128998 for ; Thu, 25 Jul 2019 09:34:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1146D2899D; Thu, 25 Jul 2019 09:34:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C26A82899B for ; Thu, 25 Jul 2019 09:34:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403807AbfGYJeP (ORCPT ); Thu, 25 Jul 2019 05:34:15 -0400 Received: from mx2.suse.de ([195.135.220.15]:52352 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2403797AbfGYJeN (ORCPT ); Thu, 25 Jul 2019 05:34:13 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id A44B3B00E for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 15/17] btrfs-progs: update checksumming api Date: Thu, 25 Jul 2019 11:34:02 +0200 Message-Id: X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: David Sterba [XXX Not done!!! ] This is based on a patch by David Sterba. Signed-off-by: David Sterba Signed-off-by: Johannes Thumshirn --- btrfs-sb-mod.c | 21 ++++++---------- check/main.c | 20 +++++++-------- cmds/inspect-dump-super.c | 9 +++---- cmds/rescue-chunk-recover.c | 12 ++++++--- convert/common.c | 9 +++---- disk-io.c | 59 ++++++++++++++++++--------------------------- disk-io.h | 3 +-- file-item.c | 8 +++--- 8 files changed, 60 insertions(+), 81 deletions(-) diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c index 9e64b34d2b8f..105b556b0cf1 100644 --- a/btrfs-sb-mod.c +++ b/btrfs-sb-mod.c @@ -33,31 +33,24 @@ static int csum_size; static int check_csum_superblock(void *sb) { - u8 result[csum_size]; - u32 crc = ~(u32)0; + u8 result[BTRFS_CSUM_SIZE]; u16 csum_type = btrfs_super_csum_type(sb); - crc = btrfs_csum_data(csum_type, - (char *)sb + BTRFS_CSUM_SIZE, - (u8 *)&crc, - BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(csum_type, crc, result); + btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, + result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - return !memcmp(sb, &result, csum_size); + return !memcmp(sb, result, csum_size); } static void update_block_csum(void *block) { - u8 result[csum_size]; + u8 result[BTRFS_CSUM_SIZE]; struct btrfs_header *hdr; - u32 crc = ~(u32)0; u16 csum_type = btrfs_super_csum_type(block); - crc = btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE, - (u8 *)&crc, - BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); + btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE, + result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(csum_type, crc, result); memset(block, 0, BTRFS_CSUM_SIZE); hdr = (struct btrfs_header *)block; memcpy(&hdr->csum, result, csum_size); diff --git a/check/main.c b/check/main.c index eabe328b6800..429edc4aa735 100644 --- a/check/main.c +++ b/check/main.c @@ -5582,10 +5582,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr, u64 offset = 0; u16 csum_size = btrfs_super_csum_size(fs_info->super_copy); u16 csum_type = btrfs_super_csum_type(fs_info->super_copy); - char *data; + u8 *data; unsigned long csum_offset; - u32 csum; - u32 csum_expected; + u8 result[BTRFS_CSUM_SIZE]; + u8 csum_expected[BTRFS_CSUM_SIZE]; u64 read_len; u64 data_checked = 0; u64 tmp; @@ -5611,7 +5611,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr, for (mirror = 1; mirror <= num_copies; mirror++) { read_len = num_bytes - offset; /* read as much space once a time */ - ret = read_extent_data(fs_info, data + offset, + ret = read_extent_data(fs_info, (char *)data + offset, bytenr + offset, &read_len, mirror); if (ret) goto out; @@ -5619,24 +5619,22 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr, data_checked = 0; /* verify every 4k data's checksum */ while (data_checked < read_len) { - csum = ~(u32)0; tmp = offset + data_checked; - csum = btrfs_csum_data(csum_type, - (char *)data + tmp, - (u8 *)&csum, fs_info->sectorsize); - btrfs_csum_final(csum_type, csum, (u8 *)&csum); + btrfs_csum_data(csum_type, data + tmp, + result, fs_info->sectorsize); csum_offset = leaf_offset + tmp / fs_info->sectorsize * csum_size; read_extent_buffer(eb, (char *)&csum_expected, csum_offset, csum_size); - if (csum != csum_expected) { + if (memcmp(result, csum_expected, csum_size) != 0) { csum_mismatch = true; + /* FIXME: format */ fprintf(stderr, "mirror %d bytenr %llu csum %u expected csum %u\n", mirror, bytenr + tmp, - csum, csum_expected); + result[0], csum_expected[0]); } data_checked += fs_info->sectorsize; } diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c index 40019a6670ef..58bf82b0bbd3 100644 --- a/cmds/inspect-dump-super.c +++ b/cmds/inspect-dump-super.c @@ -38,14 +38,11 @@ static int check_csum_sblock(void *sb, int csum_size, u16 csum_type) { u8 result[BTRFS_CSUM_SIZE]; - u32 crc = ~(u32)0; - crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, - (u8 *)&crc, - BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(csum_type, crc, result); + btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, + result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - return !memcmp(sb, &result, csum_size); + return !memcmp(sb, result, csum_size); } static void print_sys_chunk_array(struct btrfs_super_block *sb) diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c index cddae471f50c..4b6426ae0346 100644 --- a/cmds/rescue-chunk-recover.c +++ b/cmds/rescue-chunk-recover.c @@ -1892,7 +1892,11 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum, { char *data; int ret = 0; - u32 csum_result = ~(u32)0; + u8 result[BTRFS_CSUM_SIZE]; + int csum_size = 0; + u8 expected_csum[BTRFS_CSUM_SIZE]; + + ASSERT(0); data = malloc(len); if (!data) @@ -1903,9 +1907,9 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum, goto out; } ret = 0; - csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len); - btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result); - if (csum_result != tree_csum) + put_unaligned_le32(tree_csum, expected_csum); + btrfs_csum_data(csum_type, (u8 *)data, result, len); + if (memcmp(result, expected_csum, csum_size) != 0) ret = 1; out: free(data); diff --git a/convert/common.c b/convert/common.c index 894a6ee0ba90..8cae507ec0f7 100644 --- a/convert/common.c +++ b/convert/common.c @@ -62,14 +62,13 @@ static int reserve_free_space(struct cache_tree *free_tree, u64 len, static inline int write_temp_super(int fd, struct btrfs_super_block *sb, u64 sb_bytenr) { - u32 crc = ~(u32)0; + u8 result[BTRFS_CSUM_SIZE]; u16 csum_type = btrfs_super_csum_type(sb); int ret; - crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, - (u8 *)&crc, - BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(csum_type, crc, &sb->csum[0]); + btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, + result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); + memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE); ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr); if (ret < BTRFS_SUPER_INFO_SIZE) ret = (ret < 0 ? -errno : -EIO); diff --git a/disk-io.c b/disk-io.c index 7e538969c57a..810c2e14294a 100644 --- a/disk-io.c +++ b/disk-io.c @@ -138,26 +138,22 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info, } } -u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len) +int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len) { - switch (csum_type) { - case BTRFS_CSUM_TYPE_CRC32: - return crc32c(*(u32*)seed, data, len); - default: /* Not reached */ - return ~(u32)0; - } - -} + u32 crc = ~(u32)0; + memset(out, 0, BTRFS_CSUM_SIZE); -void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result) -{ switch (csum_type) { case BTRFS_CSUM_TYPE_CRC32: - put_unaligned_le32(~crc, result); - break; - default: /* Not reached */ - break; + crc = crc32c(crc, data, len); + put_unaligned_le32(~crc, out); + return 0; + default: + fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type); + ASSERT(0); } + + return -1; } static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, @@ -165,20 +161,19 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, { u8 result[BTRFS_CSUM_SIZE]; u32 len; - u32 crc = ~(u32)0; len = buf->len - BTRFS_CSUM_SIZE; - crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE, - (u8 *)&crc, len); - btrfs_csum_final(csum_type, crc, result); + btrfs_csum_data(csum_type, (u8 *)buf->data + BTRFS_CSUM_SIZE, + result, len); if (verify) { if (memcmp_extent_buffer(buf, result, 0, csum_size)) { + /* FIXME: format */ if (!silent) printk("checksum verify failed on %llu found %08X wanted %08X\n", (unsigned long long)buf->start, - *((u32 *)result), - *((u32*)(char *)buf->data)); + result[0], + buf->data[0]); return 1; } } else { @@ -1367,7 +1362,6 @@ struct btrfs_root *open_ctree_fd(int fp, const char *path, u64 sb_bytenr, int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags) { u8 result[BTRFS_CSUM_SIZE]; - u32 crc; u16 csum_type; int csum_size; u8 *metadata_uuid; @@ -1388,11 +1382,8 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags) } csum_size = btrfs_csum_sizes[csum_type]; - crc = ~(u32)0; - crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE, - (u8 *)&crc, - BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(csum_type, crc, result); + btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, + result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); if (memcmp(result, sb->csum, csum_size)) { error("superblock checksum mismatch"); @@ -1628,7 +1619,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, struct btrfs_device *device) { u64 bytenr; - u32 crc; + u8 result[BTRFS_CSUM_SIZE]; int i, ret; u16 csum_type = btrfs_super_csum_type(sb); @@ -1645,10 +1636,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, } if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) { btrfs_set_super_bytenr(sb, fs_info->super_bytenr); - crc = ~(u32)0; - crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, - BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(csum_type, crc, &sb->csum[0]); + btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, result, + BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); + memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE); /* * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is @@ -1681,10 +1671,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, btrfs_set_super_bytenr(sb, bytenr); - crc = ~(u32)0; - crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, + btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(csum_type, crc, &sb->csum[0]); + memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE); /* * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is diff --git a/disk-io.h b/disk-io.h index 4b5e9ea86385..545cacda9a79 100644 --- a/disk-io.h +++ b/disk-io.h @@ -186,8 +186,7 @@ int btrfs_free_fs_root(struct btrfs_root *root); void btrfs_mark_buffer_dirty(struct extent_buffer *buf); int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid); int btrfs_set_buffer_uptodate(struct extent_buffer *buf); -u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len); -void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result); +int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len); int btrfs_open_device(struct btrfs_device *dev); int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize, diff --git a/file-item.c b/file-item.c index 78fdcecd0bab..c6e9d212bcab 100644 --- a/file-item.c +++ b/file-item.c @@ -195,7 +195,7 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans, struct btrfs_csum_item *item; struct extent_buffer *leaf = NULL; u64 csum_offset; - u32 csum_result = ~(u32)0; + u8 csum_result[BTRFS_CSUM_SIZE]; u32 sectorsize = root->fs_info->sectorsize; u32 nritems; u32 ins_size; @@ -315,14 +315,14 @@ csum: item = (struct btrfs_csum_item *)((unsigned char *)item + csum_offset * csum_size); found: - csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len); - btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result); + btrfs_csum_data(csum_type, (u8 *)data, csum_result, len); + /* FIXME: does not make sense for non-crc32c */ if (csum_result == 0) { printk("csum result is 0 for block %llu\n", (unsigned long long)bytenr); } - write_extent_buffer(leaf, &csum_result, (unsigned long)item, + write_extent_buffer(leaf, csum_result, (unsigned long)item, csum_size); btrfs_mark_buffer_dirty(path->nodes[0]); fail: From patchwork Thu Jul 25 09:34:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058325 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0F85914DB for ; Thu, 25 Jul 2019 09:34:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E9DFA28999 for ; Thu, 25 Jul 2019 09:34:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DE7EF2899D; Thu, 25 Jul 2019 09:34:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8D1D928998 for ; Thu, 25 Jul 2019 09:34:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389991AbfGYJeO (ORCPT ); Thu, 25 Jul 2019 05:34:14 -0400 Received: from mx2.suse.de ([195.135.220.15]:52350 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2403796AbfGYJeN (ORCPT ); Thu, 25 Jul 2019 05:34:13 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id B6AEEB01C for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 16/17] btrfs-progs: add xxhash sources Date: Thu, 25 Jul 2019 11:34:03 +0200 Message-Id: <4175a2c8521b5eb86b10476a15b5c2d147373463.1564046972.git.jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: David Sterba git://github.com/Cyan4973/xxHash Signed-off-by: David Sterba Signed-off-by: Johannes Thumshirn --- crypto/xxhash.c | 1024 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ crypto/xxhash.h | 445 ++++++++++++++++++++++++ 2 files changed, 1469 insertions(+) create mode 100644 crypto/xxhash.c create mode 100644 crypto/xxhash.h diff --git a/crypto/xxhash.c b/crypto/xxhash.c new file mode 100644 index 000000000000..af9d02795ac6 --- /dev/null +++ b/crypto/xxhash.c @@ -0,0 +1,1024 @@ +/* +* xxHash - Fast Hash algorithm +* Copyright (C) 2012-2016, Yann Collet +* +* BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) +* +* Redistribution and use in source and binary forms, with or without +* modification, are permitted provided that the following conditions are +* met: +* +* * Redistributions of source code must retain the above copyright +* notice, this list of conditions and the following disclaimer. +* * Redistributions in binary form must reproduce the above +* copyright notice, this list of conditions and the following disclaimer +* in the documentation and/or other materials provided with the +* distribution. +* +* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +* +* You can contact the author at : +* - xxHash homepage: http://www.xxhash.com +* - xxHash source repository : https://github.com/Cyan4973/xxHash +*/ + + +/* ************************************* +* Tuning parameters +***************************************/ +/*!XXH_FORCE_MEMORY_ACCESS : + * By default, access to unaligned memory is controlled by `memcpy()`, which is safe and portable. + * Unfortunately, on some target/compiler combinations, the generated assembly is sub-optimal. + * The below switch allow to select different access method for improved performance. + * Method 0 (default) : use `memcpy()`. Safe and portable. + * Method 1 : `__packed` statement. It depends on compiler extension (ie, not portable). + * This method is safe if your compiler supports it, and *generally* as fast or faster than `memcpy`. + * Method 2 : direct access. This method doesn't depend on compiler but violate C standard. + * It can generate buggy code on targets which do not support unaligned memory accesses. + * But in some circumstances, it's the only known way to get the most performance (ie GCC + ARMv6) + * See http://stackoverflow.com/a/32095106/646947 for details. + * Prefer these methods in priority order (0 > 1 > 2) + */ +#ifndef XXH_FORCE_MEMORY_ACCESS /* can be defined externally, on command line for example */ +# if defined(__GNUC__) && ( defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) \ + || defined(__ARM_ARCH_6K__) || defined(__ARM_ARCH_6Z__) \ + || defined(__ARM_ARCH_6ZK__) || defined(__ARM_ARCH_6T2__) ) +# define XXH_FORCE_MEMORY_ACCESS 2 +# elif (defined(__INTEL_COMPILER) && !defined(_WIN32)) || \ + (defined(__GNUC__) && ( defined(__ARM_ARCH_7__) || defined(__ARM_ARCH_7A__) \ + || defined(__ARM_ARCH_7R__) || defined(__ARM_ARCH_7M__) \ + || defined(__ARM_ARCH_7S__) )) +# define XXH_FORCE_MEMORY_ACCESS 1 +# endif +#endif + +/*!XXH_ACCEPT_NULL_INPUT_POINTER : + * If input pointer is NULL, xxHash default behavior is to dereference it, triggering a segfault. + * When this macro is enabled, xxHash actively checks input for null pointer. + * It it is, result for null input pointers is the same as a null-length input. + */ +#ifndef XXH_ACCEPT_NULL_INPUT_POINTER /* can be defined externally */ +# define XXH_ACCEPT_NULL_INPUT_POINTER 0 +#endif + +/*!XXH_FORCE_ALIGN_CHECK : + * This is a minor performance trick, only useful with lots of very small keys. + * It means : check for aligned/unaligned input. + * The check costs one initial branch per hash; + * set it to 0 when the input is guaranteed to be aligned, + * or when alignment doesn't matter for performance. + */ +#ifndef XXH_FORCE_ALIGN_CHECK /* can be defined externally */ +# if defined(__i386) || defined(_M_IX86) || defined(__x86_64__) || defined(_M_X64) +# define XXH_FORCE_ALIGN_CHECK 0 +# else +# define XXH_FORCE_ALIGN_CHECK 1 +# endif +#endif + + +/* ************************************* +* Includes & Memory related functions +***************************************/ +/*! Modify the local functions below should you wish to use some other memory routines +* for malloc(), free() */ +#include +static void* XXH_malloc(size_t s) { return malloc(s); } +static void XXH_free (void* p) { free(p); } +/*! and for memcpy() */ +#include +static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); } + +#include /* assert */ + +#define XXH_STATIC_LINKING_ONLY +#include "xxhash.h" + + +/* ************************************* +* Compiler Specific Options +***************************************/ +#ifdef _MSC_VER /* Visual Studio */ +# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ +# define XXH_FORCE_INLINE static __forceinline +# define XXH_NO_INLINE static __declspec(noinline) +#else +# if defined (__cplusplus) || defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */ +# ifdef __GNUC__ +# define XXH_FORCE_INLINE static inline __attribute__((always_inline)) +# define XXH_NO_INLINE static __attribute__((noinline)) +# else +# define XXH_FORCE_INLINE static inline +# define XXH_NO_INLINE static +# endif +# else +# define XXH_FORCE_INLINE static +# define XXH_NO_INLINE static +# endif /* __STDC_VERSION__ */ +#endif + + +/* ************************************* +* Basic Types +***************************************/ +#ifndef MEM_MODULE +# if !defined (__VMS) \ + && (defined (__cplusplus) \ + || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) +# include + typedef uint8_t BYTE; + typedef uint16_t U16; + typedef uint32_t U32; +# else + typedef unsigned char BYTE; + typedef unsigned short U16; + typedef unsigned int U32; +# endif +#endif + + +/* === Memory access === */ + +#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2)) + +/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */ +static U32 XXH_read32(const void* memPtr) { return *(const U32*) memPtr; } + +#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1)) + +/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */ +/* currently only defined for gcc and icc */ +typedef union { U32 u32; } __attribute__((packed)) unalign; +static U32 XXH_read32(const void* ptr) { return ((const unalign*)ptr)->u32; } + +#else + +/* portable and safe solution. Generally efficient. + * see : http://stackoverflow.com/a/32095106/646947 + */ +static U32 XXH_read32(const void* memPtr) +{ + U32 val; + memcpy(&val, memPtr, sizeof(val)); + return val; +} + +#endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */ + + +/* === Endianess === */ +typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess; + +/* XXH_CPU_LITTLE_ENDIAN can be defined externally, for example on the compiler command line */ +#ifndef XXH_CPU_LITTLE_ENDIAN +static int XXH_isLittleEndian(void) +{ + const union { U32 u; BYTE c[4]; } one = { 1 }; /* don't use static : performance detrimental */ + return one.c[0]; +} +# define XXH_CPU_LITTLE_ENDIAN XXH_isLittleEndian() +#endif + + + + +/* **************************************** +* Compiler-specific Functions and Macros +******************************************/ +#define XXH_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) + +/* Note : although _rotl exists for minGW (GCC under windows), performance seems poor */ +#if defined(_MSC_VER) +# define XXH_rotl32(x,r) _rotl(x,r) +# define XXH_rotl64(x,r) _rotl64(x,r) +#else +# define XXH_rotl32(x,r) (((x) << (r)) | ((x) >> (32 - (r)))) +# define XXH_rotl64(x,r) (((x) << (r)) | ((x) >> (64 - (r)))) +#endif + +#if defined(_MSC_VER) /* Visual Studio */ +# define XXH_swap32 _byteswap_ulong +#elif XXH_GCC_VERSION >= 403 +# define XXH_swap32 __builtin_bswap32 +#else +static U32 XXH_swap32 (U32 x) +{ + return ((x << 24) & 0xff000000 ) | + ((x << 8) & 0x00ff0000 ) | + ((x >> 8) & 0x0000ff00 ) | + ((x >> 24) & 0x000000ff ); +} +#endif + + +/* *************************** +* Memory reads +*****************************/ +typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment; + +XXH_FORCE_INLINE U32 XXH_readLE32(const void* ptr) +{ + return XXH_CPU_LITTLE_ENDIAN ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr)); +} + +static U32 XXH_readBE32(const void* ptr) +{ + return XXH_CPU_LITTLE_ENDIAN ? XXH_swap32(XXH_read32(ptr)) : XXH_read32(ptr); +} + +XXH_FORCE_INLINE U32 +XXH_readLE32_align(const void* ptr, XXH_alignment align) +{ + if (align==XXH_unaligned) { + return XXH_readLE32(ptr); + } else { + return XXH_CPU_LITTLE_ENDIAN ? *(const U32*)ptr : XXH_swap32(*(const U32*)ptr); + } +} + + +/* ************************************* +* Macros +***************************************/ +#define XXH_STATIC_ASSERT(c) { enum { XXH_sa = 1/(int)(!!(c)) }; } /* use after variable declarations */ +XXH_PUBLIC_API unsigned XXH_versionNumber (void) { return XXH_VERSION_NUMBER; } + + +/* ******************************************************************* +* 32-bit hash functions +*********************************************************************/ +static const U32 PRIME32_1 = 2654435761U; /* 0b10011110001101110111100110110001 */ +static const U32 PRIME32_2 = 2246822519U; /* 0b10000101111010111100101001110111 */ +static const U32 PRIME32_3 = 3266489917U; /* 0b11000010101100101010111000111101 */ +static const U32 PRIME32_4 = 668265263U; /* 0b00100111110101001110101100101111 */ +static const U32 PRIME32_5 = 374761393U; /* 0b00010110010101100110011110110001 */ + +static U32 XXH32_round(U32 acc, U32 input) +{ + acc += input * PRIME32_2; + acc = XXH_rotl32(acc, 13); + acc *= PRIME32_1; +#if defined(__GNUC__) && defined(__SSE4_1__) && !defined(XXH_ENABLE_AUTOVECTORIZE) + /* UGLY HACK: + * This inline assembly hack forces acc into a normal register. This is the + * only thing that prevents GCC and Clang from autovectorizing the XXH32 loop + * (pragmas and attributes don't work for some resason) without globally + * disabling SSE4.1. + * + * The reason we want to avoid vectorization is because despite working on + * 4 integers at a time, there are multiple factors slowing XXH32 down on + * SSE4: + * - There's a ridiculous amount of lag from pmulld (10 cycles of latency on newer chips!) + * making it slightly slower to multiply four integers at once compared to four + * integers independently. Even when pmulld was fastest, Sandy/Ivy Bridge, it is + * still not worth it to go into SSE just to multiply unless doing a long operation. + * + * - Four instructions are required to rotate, + * movqda tmp, v // not required with VEX encoding + * pslld tmp, 13 // tmp <<= 13 + * psrld v, 19 // x >>= 19 + * por v, tmp // x |= tmp + * compared to one for scalar: + * roll v, 13 // reliably fast across the board + * shldl v, v, 13 // Sandy Bridge and later prefer this for some reason + * + * - Instruction level parallelism is actually more beneficial here because the + * SIMD actually serializes this operation: While v1 is rotating, v2 can load data, + * while v3 can multiply. SSE forces them to operate together. + * + * How this hack works: + * __asm__("" // Declare an assembly block but don't declare any instructions + * : // However, as an Input/Output Operand, + * "+r" // constrain a read/write operand (+) as a general purpose register (r). + * (acc) // and set acc as the operand + * ); + * + * Because of the 'r', the compiler has promised that seed will be in a + * general purpose register and the '+' says that it will be 'read/write', + * so it has to assume it has changed. It is like volatile without all the + * loads and stores. + * + * Since the argument has to be in a normal register (not an SSE register), + * each time XXH32_round is called, it is impossible to vectorize. */ + __asm__("" : "+r" (acc)); +#endif + return acc; +} + +/* mix all bits */ +static U32 XXH32_avalanche(U32 h32) +{ + h32 ^= h32 >> 15; + h32 *= PRIME32_2; + h32 ^= h32 >> 13; + h32 *= PRIME32_3; + h32 ^= h32 >> 16; + return(h32); +} + +#define XXH_get32bits(p) XXH_readLE32_align(p, align) + +static U32 +XXH32_finalize(U32 h32, const void* ptr, size_t len, XXH_alignment align) + +{ + const BYTE* p = (const BYTE*)ptr; + +#define PROCESS1 \ + h32 += (*p++) * PRIME32_5; \ + h32 = XXH_rotl32(h32, 11) * PRIME32_1 ; + +#define PROCESS4 \ + h32 += XXH_get32bits(p) * PRIME32_3; \ + p+=4; \ + h32 = XXH_rotl32(h32, 17) * PRIME32_4 ; + + switch(len&15) /* or switch(bEnd - p) */ + { + case 12: PROCESS4; + /* fallthrough */ + case 8: PROCESS4; + /* fallthrough */ + case 4: PROCESS4; + return XXH32_avalanche(h32); + + case 13: PROCESS4; + /* fallthrough */ + case 9: PROCESS4; + /* fallthrough */ + case 5: PROCESS4; + PROCESS1; + return XXH32_avalanche(h32); + + case 14: PROCESS4; + /* fallthrough */ + case 10: PROCESS4; + /* fallthrough */ + case 6: PROCESS4; + PROCESS1; + PROCESS1; + return XXH32_avalanche(h32); + + case 15: PROCESS4; + /* fallthrough */ + case 11: PROCESS4; + /* fallthrough */ + case 7: PROCESS4; + /* fallthrough */ + case 3: PROCESS1; + /* fallthrough */ + case 2: PROCESS1; + /* fallthrough */ + case 1: PROCESS1; + /* fallthrough */ + case 0: return XXH32_avalanche(h32); + } + assert(0); + return h32; /* reaching this point is deemed impossible */ +} + +XXH_FORCE_INLINE U32 +XXH32_endian_align(const void* input, size_t len, U32 seed, XXH_alignment align) +{ + const BYTE* p = (const BYTE*)input; + const BYTE* bEnd = p + len; + U32 h32; + +#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1) + if (p==NULL) { + len=0; + bEnd=p=(const BYTE*)(size_t)16; + } +#endif + + if (len>=16) { + const BYTE* const limit = bEnd - 15; + U32 v1 = seed + PRIME32_1 + PRIME32_2; + U32 v2 = seed + PRIME32_2; + U32 v3 = seed + 0; + U32 v4 = seed - PRIME32_1; + + do { + v1 = XXH32_round(v1, XXH_get32bits(p)); p+=4; + v2 = XXH32_round(v2, XXH_get32bits(p)); p+=4; + v3 = XXH32_round(v3, XXH_get32bits(p)); p+=4; + v4 = XXH32_round(v4, XXH_get32bits(p)); p+=4; + } while (p < limit); + + h32 = XXH_rotl32(v1, 1) + XXH_rotl32(v2, 7) + + XXH_rotl32(v3, 12) + XXH_rotl32(v4, 18); + } else { + h32 = seed + PRIME32_5; + } + + h32 += (U32)len; + + return XXH32_finalize(h32, p, len&15, align); +} + + +XXH_PUBLIC_API unsigned int XXH32 (const void* input, size_t len, unsigned int seed) +{ +#if 0 + /* Simple version, good for code maintenance, but unfortunately slow for small inputs */ + XXH32_state_t state; + XXH32_reset(&state, seed); + XXH32_update(&state, input, len); + return XXH32_digest(&state); + +#else + + if (XXH_FORCE_ALIGN_CHECK) { + if ((((size_t)input) & 3) == 0) { /* Input is 4-bytes aligned, leverage the speed benefit */ + return XXH32_endian_align(input, len, seed, XXH_aligned); + } } + + return XXH32_endian_align(input, len, seed, XXH_unaligned); +#endif +} + + + +/*====== Hash streaming ======*/ + +XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void) +{ + return (XXH32_state_t*)XXH_malloc(sizeof(XXH32_state_t)); +} +XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr) +{ + XXH_free(statePtr); + return XXH_OK; +} + +XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dstState, const XXH32_state_t* srcState) +{ + memcpy(dstState, srcState, sizeof(*dstState)); +} + +XXH_PUBLIC_API XXH_errorcode XXH32_reset(XXH32_state_t* statePtr, unsigned int seed) +{ + XXH32_state_t state; /* using a local state to memcpy() in order to avoid strict-aliasing warnings */ + memset(&state, 0, sizeof(state)); + state.v1 = seed + PRIME32_1 + PRIME32_2; + state.v2 = seed + PRIME32_2; + state.v3 = seed + 0; + state.v4 = seed - PRIME32_1; + /* do not write into reserved, planned to be removed in a future version */ + memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved)); + return XXH_OK; +} + + +XXH_PUBLIC_API XXH_errorcode +XXH32_update(XXH32_state_t* state, const void* input, size_t len) +{ + if (input==NULL) +#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1) + return XXH_OK; +#else + return XXH_ERROR; +#endif + + { const BYTE* p = (const BYTE*)input; + const BYTE* const bEnd = p + len; + + state->total_len_32 += (XXH32_hash_t)len; + state->large_len |= (XXH32_hash_t)((len>=16) | (state->total_len_32>=16)); + + if (state->memsize + len < 16) { /* fill in tmp buffer */ + XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, len); + state->memsize += (XXH32_hash_t)len; + return XXH_OK; + } + + if (state->memsize) { /* some data left from previous update */ + XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, 16-state->memsize); + { const U32* p32 = state->mem32; + state->v1 = XXH32_round(state->v1, XXH_readLE32(p32)); p32++; + state->v2 = XXH32_round(state->v2, XXH_readLE32(p32)); p32++; + state->v3 = XXH32_round(state->v3, XXH_readLE32(p32)); p32++; + state->v4 = XXH32_round(state->v4, XXH_readLE32(p32)); + } + p += 16-state->memsize; + state->memsize = 0; + } + + if (p <= bEnd-16) { + const BYTE* const limit = bEnd - 16; + U32 v1 = state->v1; + U32 v2 = state->v2; + U32 v3 = state->v3; + U32 v4 = state->v4; + + do { + v1 = XXH32_round(v1, XXH_readLE32(p)); p+=4; + v2 = XXH32_round(v2, XXH_readLE32(p)); p+=4; + v3 = XXH32_round(v3, XXH_readLE32(p)); p+=4; + v4 = XXH32_round(v4, XXH_readLE32(p)); p+=4; + } while (p<=limit); + + state->v1 = v1; + state->v2 = v2; + state->v3 = v3; + state->v4 = v4; + } + + if (p < bEnd) { + XXH_memcpy(state->mem32, p, (size_t)(bEnd-p)); + state->memsize = (unsigned)(bEnd-p); + } + } + + return XXH_OK; +} + + +XXH_PUBLIC_API unsigned int XXH32_digest (const XXH32_state_t* state) +{ + U32 h32; + + if (state->large_len) { + h32 = XXH_rotl32(state->v1, 1) + + XXH_rotl32(state->v2, 7) + + XXH_rotl32(state->v3, 12) + + XXH_rotl32(state->v4, 18); + } else { + h32 = state->v3 /* == seed */ + PRIME32_5; + } + + h32 += state->total_len_32; + + return XXH32_finalize(h32, state->mem32, state->memsize, XXH_aligned); +} + + +/*====== Canonical representation ======*/ + +/*! Default XXH result types are basic unsigned 32 and 64 bits. +* The canonical representation follows human-readable write convention, aka big-endian (large digits first). +* These functions allow transformation of hash result into and from its canonical format. +* This way, hash values can be written into a file or buffer, remaining comparable across different systems. +*/ + +XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash) +{ + XXH_STATIC_ASSERT(sizeof(XXH32_canonical_t) == sizeof(XXH32_hash_t)); + if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap32(hash); + memcpy(dst, &hash, sizeof(*dst)); +} + +XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src) +{ + return XXH_readBE32(src); +} + + +#ifndef XXH_NO_LONG_LONG + +/* ******************************************************************* +* 64-bit hash functions +*********************************************************************/ + +/*====== Memory access ======*/ + +#ifndef MEM_MODULE +# define MEM_MODULE +# if !defined (__VMS) \ + && (defined (__cplusplus) \ + || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) +# include + typedef uint64_t U64; +# else + /* if compiler doesn't support unsigned long long, replace by another 64-bit type */ + typedef unsigned long long U64; +# endif +#endif + + +#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2)) + +/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */ +static U64 XXH_read64(const void* memPtr) { return *(const U64*) memPtr; } + +#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1)) + +/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */ +/* currently only defined for gcc and icc */ +typedef union { U32 u32; U64 u64; } __attribute__((packed)) unalign64; +static U64 XXH_read64(const void* ptr) { return ((const unalign64*)ptr)->u64; } + +#else + +/* portable and safe solution. Generally efficient. + * see : http://stackoverflow.com/a/32095106/646947 + */ + +static U64 XXH_read64(const void* memPtr) +{ + U64 val; + memcpy(&val, memPtr, sizeof(val)); + return val; +} + +#endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */ + +#if defined(_MSC_VER) /* Visual Studio */ +# define XXH_swap64 _byteswap_uint64 +#elif XXH_GCC_VERSION >= 403 +# define XXH_swap64 __builtin_bswap64 +#else +static U64 XXH_swap64 (U64 x) +{ + return ((x << 56) & 0xff00000000000000ULL) | + ((x << 40) & 0x00ff000000000000ULL) | + ((x << 24) & 0x0000ff0000000000ULL) | + ((x << 8) & 0x000000ff00000000ULL) | + ((x >> 8) & 0x00000000ff000000ULL) | + ((x >> 24) & 0x0000000000ff0000ULL) | + ((x >> 40) & 0x000000000000ff00ULL) | + ((x >> 56) & 0x00000000000000ffULL); +} +#endif + +XXH_FORCE_INLINE U64 XXH_readLE64(const void* ptr) +{ + return XXH_CPU_LITTLE_ENDIAN ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr)); +} + +static U64 XXH_readBE64(const void* ptr) +{ + return XXH_CPU_LITTLE_ENDIAN ? XXH_swap64(XXH_read64(ptr)) : XXH_read64(ptr); +} + +XXH_FORCE_INLINE U64 +XXH_readLE64_align(const void* ptr, XXH_alignment align) +{ + if (align==XXH_unaligned) + return XXH_readLE64(ptr); + else + return XXH_CPU_LITTLE_ENDIAN ? *(const U64*)ptr : XXH_swap64(*(const U64*)ptr); +} + + +/*====== xxh64 ======*/ + +static const U64 PRIME64_1 = 11400714785074694791ULL; /* 0b1001111000110111011110011011000110000101111010111100101010000111 */ +static const U64 PRIME64_2 = 14029467366897019727ULL; /* 0b1100001010110010101011100011110100100111110101001110101101001111 */ +static const U64 PRIME64_3 = 1609587929392839161ULL; /* 0b0001011001010110011001111011000110011110001101110111100111111001 */ +static const U64 PRIME64_4 = 9650029242287828579ULL; /* 0b1000010111101011110010100111011111000010101100101010111001100011 */ +static const U64 PRIME64_5 = 2870177450012600261ULL; /* 0b0010011111010100111010110010111100010110010101100110011111000101 */ + +static U64 XXH64_round(U64 acc, U64 input) +{ + acc += input * PRIME64_2; + acc = XXH_rotl64(acc, 31); + acc *= PRIME64_1; + return acc; +} + +static U64 XXH64_mergeRound(U64 acc, U64 val) +{ + val = XXH64_round(0, val); + acc ^= val; + acc = acc * PRIME64_1 + PRIME64_4; + return acc; +} + +static U64 XXH64_avalanche(U64 h64) +{ + h64 ^= h64 >> 33; + h64 *= PRIME64_2; + h64 ^= h64 >> 29; + h64 *= PRIME64_3; + h64 ^= h64 >> 32; + return h64; +} + + +#define XXH_get64bits(p) XXH_readLE64_align(p, align) + +static U64 +XXH64_finalize(U64 h64, const void* ptr, size_t len, XXH_alignment align) +{ + const BYTE* p = (const BYTE*)ptr; + +#define PROCESS1_64 \ + h64 ^= (*p++) * PRIME64_5; \ + h64 = XXH_rotl64(h64, 11) * PRIME64_1; + +#define PROCESS4_64 \ + h64 ^= (U64)(XXH_get32bits(p)) * PRIME64_1; \ + p+=4; \ + h64 = XXH_rotl64(h64, 23) * PRIME64_2 + PRIME64_3; + +#define PROCESS8_64 { \ + U64 const k1 = XXH64_round(0, XXH_get64bits(p)); \ + p+=8; \ + h64 ^= k1; \ + h64 = XXH_rotl64(h64,27) * PRIME64_1 + PRIME64_4; \ +} + + switch(len&31) { + case 24: PROCESS8_64; + /* fallthrough */ + case 16: PROCESS8_64; + /* fallthrough */ + case 8: PROCESS8_64; + return XXH64_avalanche(h64); + + case 28: PROCESS8_64; + /* fallthrough */ + case 20: PROCESS8_64; + /* fallthrough */ + case 12: PROCESS8_64; + /* fallthrough */ + case 4: PROCESS4_64; + return XXH64_avalanche(h64); + + case 25: PROCESS8_64; + /* fallthrough */ + case 17: PROCESS8_64; + /* fallthrough */ + case 9: PROCESS8_64; + PROCESS1_64; + return XXH64_avalanche(h64); + + case 29: PROCESS8_64; + /* fallthrough */ + case 21: PROCESS8_64; + /* fallthrough */ + case 13: PROCESS8_64; + /* fallthrough */ + case 5: PROCESS4_64; + PROCESS1_64; + return XXH64_avalanche(h64); + + case 26: PROCESS8_64; + /* fallthrough */ + case 18: PROCESS8_64; + /* fallthrough */ + case 10: PROCESS8_64; + PROCESS1_64; + PROCESS1_64; + return XXH64_avalanche(h64); + + case 30: PROCESS8_64; + /* fallthrough */ + case 22: PROCESS8_64; + /* fallthrough */ + case 14: PROCESS8_64; + /* fallthrough */ + case 6: PROCESS4_64; + PROCESS1_64; + PROCESS1_64; + return XXH64_avalanche(h64); + + case 27: PROCESS8_64; + /* fallthrough */ + case 19: PROCESS8_64; + /* fallthrough */ + case 11: PROCESS8_64; + PROCESS1_64; + PROCESS1_64; + PROCESS1_64; + return XXH64_avalanche(h64); + + case 31: PROCESS8_64; + /* fallthrough */ + case 23: PROCESS8_64; + /* fallthrough */ + case 15: PROCESS8_64; + /* fallthrough */ + case 7: PROCESS4_64; + /* fallthrough */ + case 3: PROCESS1_64; + /* fallthrough */ + case 2: PROCESS1_64; + /* fallthrough */ + case 1: PROCESS1_64; + /* fallthrough */ + case 0: return XXH64_avalanche(h64); + } + + /* impossible to reach */ + assert(0); + return 0; /* unreachable, but some compilers complain without it */ +} + +XXH_FORCE_INLINE U64 +XXH64_endian_align(const void* input, size_t len, U64 seed, XXH_alignment align) +{ + const BYTE* p = (const BYTE*)input; + const BYTE* bEnd = p + len; + U64 h64; + +#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1) + if (p==NULL) { + len=0; + bEnd=p=(const BYTE*)(size_t)32; + } +#endif + + if (len>=32) { + const BYTE* const limit = bEnd - 32; + U64 v1 = seed + PRIME64_1 + PRIME64_2; + U64 v2 = seed + PRIME64_2; + U64 v3 = seed + 0; + U64 v4 = seed - PRIME64_1; + + do { + v1 = XXH64_round(v1, XXH_get64bits(p)); p+=8; + v2 = XXH64_round(v2, XXH_get64bits(p)); p+=8; + v3 = XXH64_round(v3, XXH_get64bits(p)); p+=8; + v4 = XXH64_round(v4, XXH_get64bits(p)); p+=8; + } while (p<=limit); + + h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18); + h64 = XXH64_mergeRound(h64, v1); + h64 = XXH64_mergeRound(h64, v2); + h64 = XXH64_mergeRound(h64, v3); + h64 = XXH64_mergeRound(h64, v4); + + } else { + h64 = seed + PRIME64_5; + } + + h64 += (U64) len; + + return XXH64_finalize(h64, p, len, align); +} + + +XXH_PUBLIC_API unsigned long long XXH64 (const void* input, size_t len, unsigned long long seed) +{ +#if 0 + /* Simple version, good for code maintenance, but unfortunately slow for small inputs */ + XXH64_state_t state; + XXH64_reset(&state, seed); + XXH64_update(&state, input, len); + return XXH64_digest(&state); + +#else + + if (XXH_FORCE_ALIGN_CHECK) { + if ((((size_t)input) & 7)==0) { /* Input is aligned, let's leverage the speed advantage */ + return XXH64_endian_align(input, len, seed, XXH_aligned); + } } + + return XXH64_endian_align(input, len, seed, XXH_unaligned); + +#endif +} + +/*====== Hash Streaming ======*/ + +XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void) +{ + return (XXH64_state_t*)XXH_malloc(sizeof(XXH64_state_t)); +} +XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr) +{ + XXH_free(statePtr); + return XXH_OK; +} + +XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dstState, const XXH64_state_t* srcState) +{ + memcpy(dstState, srcState, sizeof(*dstState)); +} + +XXH_PUBLIC_API XXH_errorcode XXH64_reset(XXH64_state_t* statePtr, unsigned long long seed) +{ + XXH64_state_t state; /* using a local state to memcpy() in order to avoid strict-aliasing warnings */ + memset(&state, 0, sizeof(state)); + state.v1 = seed + PRIME64_1 + PRIME64_2; + state.v2 = seed + PRIME64_2; + state.v3 = seed + 0; + state.v4 = seed - PRIME64_1; + /* do not write into reserved, planned to be removed in a future version */ + memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved)); + return XXH_OK; +} + +XXH_PUBLIC_API XXH_errorcode +XXH64_update (XXH64_state_t* state, const void* input, size_t len) +{ + if (input==NULL) +#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1) + return XXH_OK; +#else + return XXH_ERROR; +#endif + + { const BYTE* p = (const BYTE*)input; + const BYTE* const bEnd = p + len; + + state->total_len += len; + + if (state->memsize + len < 32) { /* fill in tmp buffer */ + XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, len); + state->memsize += (U32)len; + return XXH_OK; + } + + if (state->memsize) { /* tmp buffer is full */ + XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, 32-state->memsize); + state->v1 = XXH64_round(state->v1, XXH_readLE64(state->mem64+0)); + state->v2 = XXH64_round(state->v2, XXH_readLE64(state->mem64+1)); + state->v3 = XXH64_round(state->v3, XXH_readLE64(state->mem64+2)); + state->v4 = XXH64_round(state->v4, XXH_readLE64(state->mem64+3)); + p += 32-state->memsize; + state->memsize = 0; + } + + if (p+32 <= bEnd) { + const BYTE* const limit = bEnd - 32; + U64 v1 = state->v1; + U64 v2 = state->v2; + U64 v3 = state->v3; + U64 v4 = state->v4; + + do { + v1 = XXH64_round(v1, XXH_readLE64(p)); p+=8; + v2 = XXH64_round(v2, XXH_readLE64(p)); p+=8; + v3 = XXH64_round(v3, XXH_readLE64(p)); p+=8; + v4 = XXH64_round(v4, XXH_readLE64(p)); p+=8; + } while (p<=limit); + + state->v1 = v1; + state->v2 = v2; + state->v3 = v3; + state->v4 = v4; + } + + if (p < bEnd) { + XXH_memcpy(state->mem64, p, (size_t)(bEnd-p)); + state->memsize = (unsigned)(bEnd-p); + } + } + + return XXH_OK; +} + + +XXH_PUBLIC_API unsigned long long XXH64_digest (const XXH64_state_t* state) +{ + U64 h64; + + if (state->total_len >= 32) { + U64 const v1 = state->v1; + U64 const v2 = state->v2; + U64 const v3 = state->v3; + U64 const v4 = state->v4; + + h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18); + h64 = XXH64_mergeRound(h64, v1); + h64 = XXH64_mergeRound(h64, v2); + h64 = XXH64_mergeRound(h64, v3); + h64 = XXH64_mergeRound(h64, v4); + } else { + h64 = state->v3 /*seed*/ + PRIME64_5; + } + + h64 += (U64) state->total_len; + + return XXH64_finalize(h64, state->mem64, (size_t)state->total_len, XXH_aligned); +} + + +/*====== Canonical representation ======*/ + +XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash) +{ + XXH_STATIC_ASSERT(sizeof(XXH64_canonical_t) == sizeof(XXH64_hash_t)); + if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap64(hash); + memcpy(dst, &hash, sizeof(*dst)); +} + +XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src) +{ + return XXH_readBE64(src); +} + + + +/* ********************************************************************* +* XXH3 +* New generation hash designed for speed on small keys and vectorization +************************************************************************ */ + +#include "xxh3.h" + + +#endif /* XXH_NO_LONG_LONG */ diff --git a/crypto/xxhash.h b/crypto/xxhash.h new file mode 100644 index 000000000000..9ee05e5dc146 --- /dev/null +++ b/crypto/xxhash.h @@ -0,0 +1,445 @@ +/* + xxHash - Extremely Fast Hash algorithm + Header File + Copyright (C) 2012-2016, Yann Collet. + + BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following disclaimer + in the documentation and/or other materials provided with the + distribution. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + You can contact the author at : + - xxHash source repository : https://github.com/Cyan4973/xxHash +*/ + +/* Notice extracted from xxHash homepage : + +xxHash is an extremely fast Hash algorithm, running at RAM speed limits. +It also successfully passes all tests from the SMHasher suite. + +Comparison (single thread, Windows Seven 32 bits, using SMHasher on a Core 2 Duo @3GHz) + +Name Speed Q.Score Author +xxHash 5.4 GB/s 10 +CrapWow 3.2 GB/s 2 Andrew +MumurHash 3a 2.7 GB/s 10 Austin Appleby +SpookyHash 2.0 GB/s 10 Bob Jenkins +SBox 1.4 GB/s 9 Bret Mulvey +Lookup3 1.2 GB/s 9 Bob Jenkins +SuperFastHash 1.2 GB/s 1 Paul Hsieh +CityHash64 1.05 GB/s 10 Pike & Alakuijala +FNV 0.55 GB/s 5 Fowler, Noll, Vo +CRC32 0.43 GB/s 9 +MD5-32 0.33 GB/s 10 Ronald L. Rivest +SHA1-32 0.28 GB/s 10 + +Q.Score is a measure of quality of the hash function. +It depends on successfully passing SMHasher test set. +10 is a perfect score. + +A 64-bit version, named XXH64, is available since r35. +It offers much better speed, but for 64-bit applications only. +Name Speed on 64 bits Speed on 32 bits +XXH64 13.8 GB/s 1.9 GB/s +XXH32 6.8 GB/s 6.0 GB/s +*/ + +#ifndef XXHASH_H_5627135585666179 +#define XXHASH_H_5627135585666179 1 + +#if defined (__cplusplus) +extern "C" { +#endif + + +/* **************************** +* Definitions +******************************/ +#include /* size_t */ +typedef enum { XXH_OK=0, XXH_ERROR } XXH_errorcode; + + +/* **************************** + * API modifier + ******************************/ +/** XXH_INLINE_ALL (and XXH_PRIVATE_API) + * This is useful to include xxhash functions in `static` mode + * in order to inline them, and remove their symbol from the public list. + * Inlining can offer dramatic performance improvement on small keys. + * Methodology : + * #define XXH_INLINE_ALL + * #include "xxhash.h" + * `xxhash.c` is automatically included. + * It's not useful to compile and link it as a separate module. + */ +#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API) +# ifndef XXH_STATIC_LINKING_ONLY +# define XXH_STATIC_LINKING_ONLY +# endif +# if defined(__GNUC__) +# define XXH_PUBLIC_API static __inline __attribute__((unused)) +# elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) +# define XXH_PUBLIC_API static inline +# elif defined(_MSC_VER) +# define XXH_PUBLIC_API static __inline +# else + /* this version may generate warnings for unused static functions */ +# define XXH_PUBLIC_API static +# endif +#else +# if defined(WIN32) && defined(_MSC_VER) && (defined(XXH_IMPORT) || defined(XXH_EXPORT)) +# ifdef XXH_EXPORT +# define XXH_PUBLIC_API __declspec(dllexport) +# elif XXH_IMPORT +# define XXH_PUBLIC_API __declspec(dllimport) +# endif +# else +# define XXH_PUBLIC_API /* do nothing */ +# endif +#endif /* XXH_INLINE_ALL || XXH_PRIVATE_API */ + +/*! XXH_NAMESPACE, aka Namespace Emulation : + * + * If you want to include _and expose_ xxHash functions from within your own library, + * but also want to avoid symbol collisions with other libraries which may also include xxHash, + * + * you can use XXH_NAMESPACE, to automatically prefix any public symbol from xxhash library + * with the value of XXH_NAMESPACE (therefore, avoid NULL and numeric values). + * + * Note that no change is required within the calling program as long as it includes `xxhash.h` : + * regular symbol name will be automatically translated by this header. + */ +#ifdef XXH_NAMESPACE +# define XXH_CAT(A,B) A##B +# define XXH_NAME2(A,B) XXH_CAT(A,B) +# define XXH_versionNumber XXH_NAME2(XXH_NAMESPACE, XXH_versionNumber) +# define XXH32 XXH_NAME2(XXH_NAMESPACE, XXH32) +# define XXH32_createState XXH_NAME2(XXH_NAMESPACE, XXH32_createState) +# define XXH32_freeState XXH_NAME2(XXH_NAMESPACE, XXH32_freeState) +# define XXH32_reset XXH_NAME2(XXH_NAMESPACE, XXH32_reset) +# define XXH32_update XXH_NAME2(XXH_NAMESPACE, XXH32_update) +# define XXH32_digest XXH_NAME2(XXH_NAMESPACE, XXH32_digest) +# define XXH32_copyState XXH_NAME2(XXH_NAMESPACE, XXH32_copyState) +# define XXH32_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH32_canonicalFromHash) +# define XXH32_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH32_hashFromCanonical) +# define XXH64 XXH_NAME2(XXH_NAMESPACE, XXH64) +# define XXH64_createState XXH_NAME2(XXH_NAMESPACE, XXH64_createState) +# define XXH64_freeState XXH_NAME2(XXH_NAMESPACE, XXH64_freeState) +# define XXH64_reset XXH_NAME2(XXH_NAMESPACE, XXH64_reset) +# define XXH64_update XXH_NAME2(XXH_NAMESPACE, XXH64_update) +# define XXH64_digest XXH_NAME2(XXH_NAMESPACE, XXH64_digest) +# define XXH64_copyState XXH_NAME2(XXH_NAMESPACE, XXH64_copyState) +# define XXH64_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH64_canonicalFromHash) +# define XXH64_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH64_hashFromCanonical) +#endif + + +/* ************************************* +* Version +***************************************/ +#define XXH_VERSION_MAJOR 0 +#define XXH_VERSION_MINOR 7 +#define XXH_VERSION_RELEASE 0 +#define XXH_VERSION_NUMBER (XXH_VERSION_MAJOR *100*100 + XXH_VERSION_MINOR *100 + XXH_VERSION_RELEASE) +XXH_PUBLIC_API unsigned XXH_versionNumber (void); + + +/*-********************************************************************** +* 32-bit hash +************************************************************************/ +typedef unsigned int XXH32_hash_t; + +/*! XXH32() : + Calculate the 32-bit hash of sequence "length" bytes stored at memory address "input". + The memory between input & input+length must be valid (allocated and read-accessible). + "seed" can be used to alter the result predictably. + Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark) : 5.4 GB/s */ +XXH_PUBLIC_API XXH32_hash_t XXH32 (const void* input, size_t length, unsigned int seed); + +/*====== Streaming ======*/ +typedef struct XXH32_state_s XXH32_state_t; /* incomplete type */ +XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void); +XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr); +XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dst_state, const XXH32_state_t* src_state); + +XXH_PUBLIC_API XXH_errorcode XXH32_reset (XXH32_state_t* statePtr, unsigned int seed); +XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length); +XXH_PUBLIC_API XXH32_hash_t XXH32_digest (const XXH32_state_t* statePtr); + +/* + * Streaming functions generate the xxHash of an input provided in multiple segments. + * Note that, for small input, they are slower than single-call functions, due to state management. + * For small inputs, prefer `XXH32()` and `XXH64()`, which are better optimized. + * + * XXH state must first be allocated, using XXH*_createState() . + * + * Start a new hash by initializing state with a seed, using XXH*_reset(). + * + * Then, feed the hash state by calling XXH*_update() as many times as necessary. + * The function returns an error code, with 0 meaning OK, and any other value meaning there is an error. + * + * Finally, a hash value can be produced anytime, by using XXH*_digest(). + * This function returns the nn-bits hash as an int or long long. + * + * It's still possible to continue inserting input into the hash state after a digest, + * and generate some new hashes later on, by calling again XXH*_digest(). + * + * When done, free XXH state space if it was allocated dynamically. + */ + +/*====== Canonical representation ======*/ + +typedef struct { unsigned char digest[4]; } XXH32_canonical_t; +XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash); +XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src); + +/* Default result type for XXH functions are primitive unsigned 32 and 64 bits. + * The canonical representation uses human-readable write convention, aka big-endian (large digits first). + * These functions allow transformation of hash result into and from its canonical format. + * This way, hash values can be written into a file / memory, and remain comparable on different systems and programs. + */ + + +#ifndef XXH_NO_LONG_LONG +/*-********************************************************************** +* 64-bit hash +************************************************************************/ +typedef unsigned long long XXH64_hash_t; + +/*! XXH64() : + Calculate the 64-bit hash of sequence of length "len" stored at memory address "input". + "seed" can be used to alter the result predictably. + This function runs faster on 64-bit systems, but slower on 32-bit systems (see benchmark). +*/ +XXH_PUBLIC_API XXH64_hash_t XXH64 (const void* input, size_t length, unsigned long long seed); + +/*====== Streaming ======*/ +typedef struct XXH64_state_s XXH64_state_t; /* incomplete type */ +XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void); +XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr); +XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dst_state, const XXH64_state_t* src_state); + +XXH_PUBLIC_API XXH_errorcode XXH64_reset (XXH64_state_t* statePtr, unsigned long long seed); +XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* statePtr, const void* input, size_t length); +XXH_PUBLIC_API XXH64_hash_t XXH64_digest (const XXH64_state_t* statePtr); + +/*====== Canonical representation ======*/ +typedef struct { unsigned char digest[8]; } XXH64_canonical_t; +XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash); +XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src); + + +#endif /* XXH_NO_LONG_LONG */ + + + +#ifdef XXH_STATIC_LINKING_ONLY + +/* ================================================================================================ + This section contains declarations which are not guaranteed to remain stable. + They may change in future versions, becoming incompatible with a different version of the library. + These declarations should only be used with static linking. + Never use them in association with dynamic linking ! +=================================================================================================== */ + +/* These definitions are only present to allow + * static allocation of XXH state, on stack or in a struct for example. + * Never **ever** use members directly. */ + +#if !defined (__VMS) \ + && (defined (__cplusplus) \ + || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) +# include + +struct XXH32_state_s { + uint32_t total_len_32; + uint32_t large_len; + uint32_t v1; + uint32_t v2; + uint32_t v3; + uint32_t v4; + uint32_t mem32[4]; + uint32_t memsize; + uint32_t reserved; /* never read nor write, might be removed in a future version */ +}; /* typedef'd to XXH32_state_t */ + +struct XXH64_state_s { + uint64_t total_len; + uint64_t v1; + uint64_t v2; + uint64_t v3; + uint64_t v4; + uint64_t mem64[4]; + uint32_t memsize; + uint32_t reserved[2]; /* never read nor write, might be removed in a future version */ +}; /* typedef'd to XXH64_state_t */ + +# else + +struct XXH32_state_s { + XXH32_hash_t total_len_32; + XXH32_hash_t large_len; + XXH32_hash_t v1; + XXH32_hash_t v2; + XXH32_hash_t v3; + XXH32_hash_t v4; + XXH32_hash_t mem32[4]; + XXH32_hash_t memsize; + XXH32_hash_t reserved; /* never read nor write, might be removed in a future version */ +}; /* typedef'd to XXH32_state_t */ + +# ifndef XXH_NO_LONG_LONG /* remove 64-bit support */ +struct XXH64_state_s { + XXH64_hash_t total_len; + XXH64_hash_t v1; + XXH64_hash_t v2; + XXH64_hash_t v3; + XXH64_hash_t v4; + XXH64_hash_t mem64[4]; + XXH32_hash_t memsize; + XXH32_hash_t reserved[2]; /* never read nor write, might be removed in a future version */ +}; /* typedef'd to XXH64_state_t */ +# endif + +# endif + + +/*-********************************************************************** +* XXH3 +* New experimental hash +************************************************************************/ +#ifndef XXH_NO_LONG_LONG + + +/* ============================================ + * XXH3 is a new hash algorithm, + * featuring vastly improved speed performance + * for both small and large inputs. + * A full speed analysis will be published, + * it requires a lot more space than this comment can handle. + * In general, expect XXH3 to run about ~2x faster on large inputs, + * and >3x faster on small ones, though exact difference depend on platform. + * + * The algorithm is portable, will generate the same hash on all platforms. + * It benefits greatly from vectorization units, but does not require it. + * + * XXH3 offers 2 variants, _64bits and _128bits. + * When only 64 bits are needed, prefer calling the _64bits variant : + * it reduces the amount of mixing, resulting in faster speed on small inputs. + * It's also generally simpler to manipulate a scalar type than a struct. + * Note : the low 64-bit field of the _128bits variant is the same as _64bits result. + * + * The XXH3 algorithm is still considered experimental. + * It's possible to use it for ephemeral data, but avoid storing long-term values for later re-use. + * While labelled experimental, the produced result can still change between versions. + * + * The API currently supports one-shot hashing only. + * The full version will include streaming capability, and canonical representation + * Long term optional feature may include custom secret keys, and secret key generation. + * + * There are still a number of opened questions that community can influence during the experimental period. + * I'm trying to list a few of them below, though don't consider this list as complete. + * + * - 128-bits output type : currently defined as a structure of 2 64-bits fields. + * That's because 128-bit values do not exist in C standard. + * Note that it means that, at byte level, result is not identical depending on endianess. + * However, at field level, they are identical on all platforms. + * The canonical representation will solve the issue of identical byte-level representation across platforms, + * which is necessary for serialization. + * Would there be a better representation for a 128-bit hash result ? + * Are the names of the inner 64-bit fields important ? Should they be changed ? + * + * - Canonical representation : for the 64-bit variant, canonical representation is the same as XXH64() (aka big-endian). + * What should it be for the 128-bit variant ? + * Since it's no longer a scalar value, big-endian representation is no longer an obvious choice. + * One possibility : represent it as the concatenation of two 64-bits canonical representation (aka 2x big-endian) + * Another one : represent it in the same order as natural order in the struct for little-endian platforms. + * Less consistent with existing convention for XXH32/XXH64, but may be more natural for little-endian platforms. + * + * - Associated functions for 128-bit hash : simple things, such as checking if 2 hashes are equal, become more difficult with struct. + * Granted, it's not terribly difficult to create a comparator, but it's still a workload. + * Would it be beneficial to declare and define a comparator function for XXH128_hash_t ? + * Are there other operations on XXH128_hash_t which would be desirable ? + * + * - Variant compatibility : The low 64-bit field of the _128bits variant is the same as the result of _64bits. + * This is not a compulsory behavior. It just felt that it "wouldn't hurt", and might even help in some (unidentified) cases. + * But it might influence the design of XXH128_hash_t, in ways which may block other possibilities. + * Good idea, bad idea ? + * + * - Seed type for 128-bits variant : currently, it's a single 64-bit value, like the 64-bit variant. + * It could be argued that it's more logical to offer a 128-bit seed input parameter for a 128-bit hash. + * Although it's also more difficult to use, since it requires to declare and pass a structure instead of a value. + * It would either replace current choice, or add a new one. + * Farmhash, for example, offers both variants (the 128-bits seed variant is called `doubleSeed`). + * If both 64-bit and 128-bit seeds are possible, which variant should be called XXH128 ? + * + * - Result for len==0 : Currently, the result of hashing a zero-length input is the seed. + * This mimics the behavior of a crc : in which case, a seed is effectively an accumulator, so it's not updated if input is empty. + * Consequently, by default, when no seed specified, it returns zero. That part seems okay (it used to be a request for XXH32/XXH64). + * But is it still fine to return the seed when the seed is non-zero ? + * Are there use case which would depend on this behavior, or would prefer a mixing of the seed ? + */ + +#ifdef XXH_NAMESPACE +# define XXH128 XXH_NAME2(XXH_NAMESPACE, XXH128) +# define XXH3_64bits XXH_NAME2(XXH_NAMESPACE, XXH3_64bits) +# define XXH3_64bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSeed) +# define XXH3_128bits XXH_NAME2(XXH_NAMESPACE, XXH3_128bits) +# define XXH3_128bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSeed) +#endif + + +typedef struct { + XXH64_hash_t low64; + XXH64_hash_t high64; +} XXH128_hash_t; + +XXH_PUBLIC_API XXH128_hash_t XXH128(const void* data, size_t len, unsigned long long seed); + +/* note : variants without seed produce same result as variant with seed == 0 */ +XXH_PUBLIC_API XXH64_hash_t XXH3_64bits(const void* data, size_t len); +XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSeed(const void* data, size_t len, unsigned long long seed); +XXH_PUBLIC_API XXH128_hash_t XXH3_128bits(const void* data, size_t len); +XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSeed(const void* data, size_t len, unsigned long long seed); /* == XXH128() */ + + +#endif /* XXH_NO_LONG_LONG */ + + +/*-********************************************************************** +* XXH_INLINE_ALL +************************************************************************/ +#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API) +# include "xxhash.c" /* include xxhash function bodies as `static`, for inlining */ +#endif + + + +#endif /* XXH_STATIC_LINKING_ONLY */ + + +#if defined (__cplusplus) +} +#endif + +#endif /* XXHASH_H_5627135585666179 */ From patchwork Thu Jul 25 09:34:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11058329 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 476AF13A0 for ; Thu, 25 Jul 2019 09:34:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3448B28999 for ; Thu, 25 Jul 2019 09:34:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 276952899D; Thu, 25 Jul 2019 09:34:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 215F72899B for ; Thu, 25 Jul 2019 09:34:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403812AbfGYJeR (ORCPT ); Thu, 25 Jul 2019 05:34:17 -0400 Received: from mx2.suse.de ([195.135.220.15]:52362 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2403798AbfGYJeM (ORCPT ); Thu, 25 Jul 2019 05:34:12 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id C4B67B021 for ; Thu, 25 Jul 2019 09:34:09 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [RFC PATCH 17/17] btrfs-progs: add xxhash64 as checksum algorithm Date: Thu, 25 Jul 2019 11:34:04 +0200 Message-Id: X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: David Sterba Add xxhash64 as another checksumming algorithm. Signed-off-by: David Sterba Signed-off-by: Johannes Thumshirn --- Makefile | 3 ++- cmds/inspect-dump-super.c | 25 ++++++++++++++++--------- convert/common.c | 2 +- convert/main.c | 2 +- crypto/hash.c | 16 ++++++++++++++++ crypto/hash.h | 10 ++++++++++ crypto/xxhash.c | 2 +- ctree.h | 18 +++++++++++++----- disk-io.c | 7 +++++-- image/main.c | 5 +++-- mkfs/common.c | 14 +++++++------- mkfs/main.c | 6 +++++- 12 files changed, 80 insertions(+), 30 deletions(-) create mode 100644 crypto/hash.c create mode 100644 crypto/hash.h diff --git a/Makefile b/Makefile index 82417d19a9f8..1982e6f5d70e 100644 --- a/Makefile +++ b/Makefile @@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \ mkfs/common.o check/mode-common.o check/mode-lowmem.o libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \ kernel-lib/crc32c.o common/messages.o \ - uuid-tree.o utils-lib.o common/rbtree-utils.o + uuid-tree.o utils-lib.o common/rbtree-utils.o \ + crypto/hash.o crypto/xxhash.o libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \ kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \ kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \ diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c index 58bf82b0bbd3..1001c8aa5c85 100644 --- a/cmds/inspect-dump-super.c +++ b/cmds/inspect-dump-super.c @@ -311,6 +311,17 @@ static void print_readable_super_flag(u64 flag) super_flags_num, BTRFS_SUPER_FLAG_SUPP); } +static bool is_valid_csum_type(u16 csum_type) +{ + switch (csum_type) { + case BTRFS_CSUM_TYPE_CRC32: + case BTRFS_CSUM_TYPE_XXHASH: + return true; + default: + return false; + } +} + static void dump_superblock(struct btrfs_super_block *sb, int full) { int i; @@ -326,15 +337,11 @@ static void dump_superblock(struct btrfs_super_block *sb, int full) csum_type = btrfs_super_csum_type(sb); csum_size = BTRFS_CSUM_SIZE; printf("csum_type\t\t%hu (", csum_type); - if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) { + if (csum_type >= ARRAY_SIZE(btrfs_csums)) { printf("INVALID"); } else { - if (csum_type == BTRFS_CSUM_TYPE_CRC32) { - printf("crc32c"); - csum_size = btrfs_csum_sizes[csum_type]; - } else { - printf("unknown"); - } + printf("%s", btrfs_csums[csum_type].name); + csum_size = btrfs_csums[csum_type].size; } printf(")\n"); printf("csum_size\t\t%llu\n", (unsigned long long)csum_size); @@ -342,8 +349,8 @@ static void dump_superblock(struct btrfs_super_block *sb, int full) printf("csum\t\t\t0x"); for (i = 0, p = sb->csum; i < csum_size; i++) printf("%02x", p[i]); - if (csum_type != BTRFS_CSUM_TYPE_CRC32 || - csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]) + if (!is_valid_csum_type(csum_type) || + csum_size != btrfs_csums[csum_type].size) printf(" [UNKNOWN CSUM TYPE OR SIZE]"); else if (check_csum_sblock(sb, csum_size, csum_type)) printf(" [match]"); diff --git a/convert/common.c b/convert/common.c index 8cae507ec0f7..2b9613f01787 100644 --- a/convert/common.c +++ b/convert/common.c @@ -224,7 +224,7 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf, { int ret; - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); /* Temporary extent buffer is always mapped 1:1 on disk */ diff --git a/convert/main.c b/convert/main.c index 5e6b12431f59..5eb2a59fb68a 100644 --- a/convert/main.c +++ b/convert/main.c @@ -1058,7 +1058,7 @@ static int migrate_super_block(int fd, u64 old_bytenr) BUG_ON(btrfs_super_bytenr(super) != old_bytenr); btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET); - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0, + csum_tree_block_size(buf, btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size, 0, btrfs_super_csum_type(super)); ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE, BTRFS_SUPER_INFO_OFFSET); diff --git a/crypto/hash.c b/crypto/hash.c new file mode 100644 index 000000000000..fda7fc4e9f23 --- /dev/null +++ b/crypto/hash.c @@ -0,0 +1,16 @@ +#include "crypto/hash.h" +#include "crypto/xxhash.h" + +int hash_xxhash(const u8 *buf, size_t length, u8 *out) +{ + XXH64_hash_t hash; + + hash = XXH64(buf, length, 0); + /* NOTE: we're not taking the canonical form here but the plain hash to + * be compatible with the kernel implementation! + */ + memcpy(out, &hash, 8); + + return 0; +} + diff --git a/crypto/hash.h b/crypto/hash.h new file mode 100644 index 000000000000..45c1ef17bc57 --- /dev/null +++ b/crypto/hash.h @@ -0,0 +1,10 @@ +#ifndef CRYPTO_HASH_H +#define CRYPTO_HASH_H + +#include "../kerncompat.h" + +#define CRYPTO_HASH_SIZE_MAX 32 + +int hash_xxhash(const u8 *buf, size_t length, u8 *out); + +#endif diff --git a/crypto/xxhash.c b/crypto/xxhash.c index af9d02795ac6..7f381c8b56a0 100644 --- a/crypto/xxhash.c +++ b/crypto/xxhash.c @@ -1018,7 +1018,7 @@ XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src * New generation hash designed for speed on small keys and vectorization ************************************************************************ */ -#include "xxh3.h" +/* #include "xxh3.h" */ #endif /* XXH_NO_LONG_LONG */ diff --git a/ctree.h b/ctree.h index c63fad8310db..bb10080c9238 100644 --- a/ctree.h +++ b/ctree.h @@ -165,10 +165,18 @@ struct btrfs_free_space_ctl; #define BTRFS_CSUM_SIZE 32 /* csum types */ -#define BTRFS_CSUM_TYPE_CRC32 0 +enum btrfs_csum_type { + BTRFS_CSUM_TYPE_CRC32 = 0, + BTRFS_CSUM_TYPE_XXHASH = 1, +}; -/* four bytes for CRC32 */ -static int btrfs_csum_sizes[] = { 4 }; +static struct btrfs_csum { + u16 size; + const char *name; +} btrfs_csums[] = { + [BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" }, + [BTRFS_CSUM_TYPE_XXHASH] = { 8, "xxhash64" }, +}; #define BTRFS_EMPTY_DIR_SIZE 0 @@ -2264,8 +2272,8 @@ BTRFS_SETGET_STACK_FUNCS(super_magic, struct btrfs_super_block, magic, 64); static inline int btrfs_super_csum_size(struct btrfs_super_block *s) { int t = btrfs_super_csum_type(s); - BUG_ON(t >= ARRAY_SIZE(btrfs_csum_sizes)); - return btrfs_csum_sizes[t]; + BUG_ON(t >= ARRAY_SIZE(btrfs_csums)); + return btrfs_csums[t].size; } static inline unsigned long btrfs_leaf_data(struct extent_buffer *l) diff --git a/disk-io.c b/disk-io.c index 810c2e14294a..ce0b746f4db9 100644 --- a/disk-io.c +++ b/disk-io.c @@ -34,6 +34,7 @@ #include "print-tree.h" #include "common/rbtree-utils.h" #include "common/device-scan.h" +#include "crypto/hash.h" /* specified errno for check_tree_block */ #define BTRFS_BAD_BYTENR (-1) @@ -148,6 +149,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len) crc = crc32c(crc, data, len); put_unaligned_le32(~crc, out); return 0; + case BTRFS_CSUM_TYPE_XXHASH: + return hash_xxhash(data, len, out); default: fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type); ASSERT(0); @@ -1376,11 +1379,11 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags) } csum_type = btrfs_super_csum_type(sb); - if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) { + if (csum_type >= ARRAY_SIZE(btrfs_csums)) { error("unsupported checksum algorithm %u", csum_type); return -EIO; } - csum_size = btrfs_csum_sizes[csum_type]; + csum_size = btrfs_csums[csum_type].size; btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); diff --git a/image/main.c b/image/main.c index 0c8ffede56f5..1265152cf524 100644 --- a/image/main.c +++ b/image/main.c @@ -121,11 +121,12 @@ static struct extent_buffer *alloc_dummy_eb(u64 bytenr, u32 size); static void csum_block(u8 *buf, size_t len) { - u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]]; + u16 csum_size = btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size; + u8 result[csum_size]; u32 crc = ~(u32)0; crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE); put_unaligned_le32(~crc, result); - memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]); + memcpy(buf, result, csum_size); } static int has_name(struct btrfs_key *key) diff --git a/mkfs/common.c b/mkfs/common.c index 4a417bd7a306..939be5eb2dc2 100644 --- a/mkfs/common.c +++ b/mkfs/common.c @@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg, } /* generate checksum */ - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); /* write back root tree */ @@ -293,7 +293,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]); btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]); if (ret != cfg->nodesize) { @@ -382,7 +382,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]); btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]); if (ret != cfg->nodesize) { @@ -423,7 +423,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]); btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]); if (ret != cfg->nodesize) { @@ -437,7 +437,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]); btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID); btrfs_set_header_nritems(buf, 0); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]); if (ret != cfg->nodesize) { @@ -450,7 +450,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]); btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID); btrfs_set_header_nritems(buf, 0); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]); if (ret != cfg->nodesize) { @@ -462,7 +462,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE); memcpy(buf->data, &super, sizeof(super)); buf->len = BTRFS_SUPER_INFO_SIZE; - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE, cfg->blocks[MKFS_SUPER_BLOCK]); diff --git a/mkfs/main.c b/mkfs/main.c index 3deff76045ba..4d244d56bbac 100644 --- a/mkfs/main.c +++ b/mkfs/main.c @@ -385,6 +385,9 @@ static u16 parse_csum_type(const char *s) { if (strcasecmp(s, "crc32c") == 0) { return BTRFS_CSUM_TYPE_CRC32; + } else if (strcasecmp(s, "xxhash64") == 0 || + strcasecmp(s, "xxhash") == 0) { + return BTRFS_CSUM_TYPE_XXHASH; } else { error("unknown csum type %s", s); exit(1); @@ -1370,7 +1373,8 @@ raid_groups: pretty_size(allocation.system)); printf("SSD detected: %s\n", ssd ? "yes" : "no"); btrfs_parse_features_to_string(features_buf, features); - printf("Incompat features: %s", features_buf); + printf("Incompat features: %s\n", features_buf); + printf("Checksum: %s", btrfs_csums[csum_type].name); printf("\n"); list_all_devices(root);