From patchwork Tue Sep 3 15:00:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11128111 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CBB0D1813 for ; Tue, 3 Sep 2019 15:00:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B5A5723789 for ; Tue, 3 Sep 2019 15:00:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729714AbfICPAv (ORCPT ); Tue, 3 Sep 2019 11:00:51 -0400 Received: from mx2.suse.de ([195.135.220.15]:57440 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727667AbfICPAt (ORCPT ); Tue, 3 Sep 2019 11:00:49 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 00D33AD4E; Tue, 3 Sep 2019 15:00:48 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size() Date: Tue, 3 Sep 2019 17:00:35 +0200 Message-Id: <20190903150046.14926-2-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190903150046.14926-1-jthumshirn@suse.de> References: <20190903150046.14926-1-jthumshirn@suse.de> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org The callers of csum_tree_block_size() blindly assume we're only having crc32c as a possible checksum and thus pass in btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of csum_tree_block_size(). Signed-off-by: Johannes Thumshirn Reviewed-by: Nikolay Borisov --- mkfs/common.c | 14 +++++++------- mkfs/common.h | 2 ++ 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/mkfs/common.c b/mkfs/common.c index caca5e707233..b6e549b19272 100644 --- a/mkfs/common.c +++ b/mkfs/common.c @@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg, } /* generate checksum */ - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); /* write back root tree */ ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_ROOT_TREE]); @@ -292,7 +292,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]); btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -380,7 +380,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]); btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -420,7 +420,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]); btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -433,7 +433,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]); btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID); btrfs_set_header_nritems(buf, 0); - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -445,7 +445,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]); btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID); btrfs_set_header_nritems(buf, 0); - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -456,7 +456,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE); memcpy(buf->data, &super, sizeof(super)); buf->len = BTRFS_SUPER_INFO_SIZE; - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE, cfg->blocks[MKFS_SUPER_BLOCK]); if (ret != BTRFS_SUPER_INFO_SIZE) { diff --git a/mkfs/common.h b/mkfs/common.h index 28912906d0a9..1ca71a4fcce5 100644 --- a/mkfs/common.h +++ b/mkfs/common.h @@ -53,6 +53,8 @@ struct btrfs_mkfs_config { u64 features; /* Size of the filesystem in bytes */ u64 num_bytes; + /* checksum algorithm to use */ + enum btrfs_csum_type csum_type; /* Output fields, set during creation */ From patchwork Tue Sep 3 15:00:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11128107 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A13F7112C for ; Tue, 3 Sep 2019 15:00:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8B921238C4 for ; Tue, 3 Sep 2019 15:00:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729686AbfICPAt (ORCPT ); Tue, 3 Sep 2019 11:00:49 -0400 Received: from mx2.suse.de ([195.135.220.15]:57448 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728854AbfICPAt (ORCPT ); Tue, 3 Sep 2019 11:00:49 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 0DBE2AD63; Tue, 3 Sep 2019 15:00:48 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [PATCH v4 02/12] btrfs-progs: cache csum_type in recover_control Date: Tue, 3 Sep 2019 17:00:36 +0200 Message-Id: <20190903150046.14926-3-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190903150046.14926-1-jthumshirn@suse.de> References: <20190903150046.14926-1-jthumshirn@suse.de> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Cache the super-block's checksum type field in 'struct recover_control'. This will be needed for further refactoring the checksum handling. Signed-off-by: Johannes Thumshirn Reviewed-by: Nikolay Borisov --- cmds/rescue-chunk-recover.c | 2 ++ ctree.h | 4 +++- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c index c9b268781159..1959a2047c17 100644 --- a/cmds/rescue-chunk-recover.c +++ b/cmds/rescue-chunk-recover.c @@ -47,6 +47,7 @@ struct recover_control { int yes; u16 csum_size; + u16 csum_type; u32 sectorsize; u32 nodesize; u64 generation; @@ -1530,6 +1531,7 @@ static int recover_prepare(struct recover_control *rc, const char *path) rc->generation = btrfs_super_generation(sb); rc->chunk_root_generation = btrfs_super_chunk_root_generation(sb); rc->csum_size = btrfs_super_csum_size(sb); + rc->csum_type = btrfs_super_csum_type(sb); /* if seed, the result of scanning below will be partial */ if (btrfs_super_flags(sb) & BTRFS_SUPER_FLAG_SEEDING) { diff --git a/ctree.h b/ctree.h index 0d12563b7261..870d9f4948de 100644 --- a/ctree.h +++ b/ctree.h @@ -165,7 +165,9 @@ struct btrfs_free_space_ctl; #define BTRFS_CSUM_SIZE 32 /* csum types */ -#define BTRFS_CSUM_TYPE_CRC32 0 +enum btrfs_csum_type { + BTRFS_CSUM_TYPE_CRC32 = 0, +}; /* four bytes for CRC32 */ static int btrfs_csum_sizes[] = { 4 }; From patchwork Tue Sep 3 15:00:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11128117 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8C2101813 for ; Tue, 3 Sep 2019 15:00:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6E1352339D for ; Tue, 3 Sep 2019 15:00:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729726AbfICPAx (ORCPT ); Tue, 3 Sep 2019 11:00:53 -0400 Received: from mx2.suse.de ([195.135.220.15]:57454 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729109AbfICPAv (ORCPT ); Tue, 3 Sep 2019 11:00:51 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 23980AE14; Tue, 3 Sep 2019 15:00:48 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [PATCH v4 03/12] btrfs-progs: add checksum type to checksumming functions Date: Tue, 3 Sep 2019 17:00:37 +0200 Message-Id: <20190903150046.14926-4-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190903150046.14926-1-jthumshirn@suse.de> References: <20190903150046.14926-1-jthumshirn@suse.de> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Add the checksum type to csum_tree_block_size(), __csum_tree_block_size() and verify_tree_block_csum_silent(). Signed-off-by: Johannes Thumshirn --- btrfs-corrupt-block.c | 3 ++- cmds/rescue-chunk-recover.c | 3 ++- convert/common.c | 3 ++- convert/main.c | 3 ++- disk-io.c | 21 ++++++++++++--------- disk-io.h | 5 +++-- mkfs/common.c | 21 ++++++++++++++------- 7 files changed, 37 insertions(+), 22 deletions(-) diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c index bbef0c02e5d1..1dde9594bdcc 100644 --- a/btrfs-corrupt-block.c +++ b/btrfs-corrupt-block.c @@ -158,7 +158,8 @@ static void corrupt_keys(struct btrfs_trans_handle *trans, if (!trans) { u16 csum_size = btrfs_super_csum_size(fs_info->super_copy); - csum_tree_block_size(eb, csum_size, 0); + u16 csum_type = btrfs_super_csum_type(fs_info->super_copy); + csum_tree_block_size(eb, csum_size, 0, csum_type); write_extent_to_disk(eb); } } diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c index 1959a2047c17..c2a26ad040ee 100644 --- a/cmds/rescue-chunk-recover.c +++ b/cmds/rescue-chunk-recover.c @@ -768,7 +768,8 @@ static int scan_one_device(void *dev_scan_struct) continue; } - if (verify_tree_block_csum_silent(buf, rc->csum_size)) { + if (verify_tree_block_csum_silent(buf, rc->csum_size, + rc->csum_type)) { bytenr += rc->sectorsize; continue; } diff --git a/convert/common.c b/convert/common.c index dea5f5b20d50..f8bbb23cba89 100644 --- a/convert/common.c +++ b/convert/common.c @@ -223,7 +223,8 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf, { int ret; - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); /* Temporary extent buffer is always mapped 1:1 on disk */ ret = pwrite(fd, buf->data, buf->len, bytenr); diff --git a/convert/main.c b/convert/main.c index 9711874bd137..5e6b12431f59 100644 --- a/convert/main.c +++ b/convert/main.c @@ -1058,7 +1058,8 @@ static int migrate_super_block(int fd, u64 old_bytenr) BUG_ON(btrfs_super_bytenr(super) != old_bytenr); btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET); - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0, + btrfs_super_csum_type(super)); ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE, BTRFS_SUPER_INFO_OFFSET); if (ret != BTRFS_SUPER_INFO_SIZE) diff --git a/disk-io.c b/disk-io.c index 01314504a50a..a4995a628210 100644 --- a/disk-io.c +++ b/disk-io.c @@ -149,7 +149,7 @@ void btrfs_csum_final(u32 crc, u8 *result) } static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, - int verify, int silent) + int verify, int silent, u16 csum_type) { u8 result[BTRFS_CSUM_SIZE]; u32 len; @@ -174,24 +174,27 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, return 0; } -int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify) +int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify, + u16 csum_type) { - return __csum_tree_block_size(buf, csum_size, verify, 0); + return __csum_tree_block_size(buf, csum_size, verify, 0, csum_type); } -int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size) +int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size, + u16 csum_type) { - return __csum_tree_block_size(buf, csum_size, 1, 1); + return __csum_tree_block_size(buf, csum_size, 1, 1, csum_type); } int csum_tree_block(struct btrfs_fs_info *fs_info, struct extent_buffer *buf, int verify) { - u16 csum_size = - btrfs_super_csum_size(fs_info->super_copy); + u16 csum_size = btrfs_super_csum_size(fs_info->super_copy); + u16 csum_type = btrfs_super_csum_type(fs_info->super_copy); + if (verify && fs_info->suppress_check_block_errors) - return verify_tree_block_csum_silent(buf, csum_size); - return csum_tree_block_size(buf, csum_size, verify); + return verify_tree_block_csum_silent(buf, csum_size, csum_type); + return csum_tree_block_size(buf, csum_size, verify, csum_type); } struct extent_buffer *btrfs_find_tree_block(struct btrfs_fs_info *fs_info, diff --git a/disk-io.h b/disk-io.h index 7b5c3806ba98..394997ad72cb 100644 --- a/disk-io.h +++ b/disk-io.h @@ -191,8 +191,9 @@ void btrfs_csum_final(u32 crc, u8 *result); int btrfs_open_device(struct btrfs_device *dev); int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize, - int verify); -int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size); + int verify, u16 csum_type); +int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size, + u16 csum_type); int btrfs_read_buffer(struct extent_buffer *buf, u64 parent_transid); int write_tree_block(struct btrfs_trans_handle *trans, struct btrfs_fs_info *fs_info, diff --git a/mkfs/common.c b/mkfs/common.c index b6e549b19272..9762391a8d2b 100644 --- a/mkfs/common.c +++ b/mkfs/common.c @@ -101,7 +101,8 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg, } /* generate checksum */ - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); /* write back root tree */ ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_ROOT_TREE]); @@ -292,7 +293,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]); btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -380,7 +382,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]); btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -420,7 +423,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]); btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -433,7 +437,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]); btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID); btrfs_set_header_nritems(buf, 0); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -445,7 +450,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]); btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID); btrfs_set_header_nritems(buf, 0); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]); if (ret != cfg->nodesize) { ret = (ret < 0 ? -errno : -EIO); @@ -456,7 +462,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE); memcpy(buf->data, &super, sizeof(super)); buf->len = BTRFS_SUPER_INFO_SIZE; - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0); + csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + cfg->csum_type); ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE, cfg->blocks[MKFS_SUPER_BLOCK]); if (ret != BTRFS_SUPER_INFO_SIZE) { From patchwork Tue Sep 3 15:00:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11128109 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7BDDA112C for ; Tue, 3 Sep 2019 15:00:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 674CA23789 for ; Tue, 3 Sep 2019 15:00:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729137AbfICPAu (ORCPT ); Tue, 3 Sep 2019 11:00:50 -0400 Received: from mx2.suse.de ([195.135.220.15]:57462 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729678AbfICPAt (ORCPT ); Tue, 3 Sep 2019 11:00:49 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 3024AAEA1; Tue, 3 Sep 2019 15:00:48 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [PATCH v4 04/12] btrfs-progs: don't assume checksums are always 4 bytes Date: Tue, 3 Sep 2019 17:00:38 +0200 Message-Id: <20190903150046.14926-5-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190903150046.14926-1-jthumshirn@suse.de> References: <20190903150046.14926-1-jthumshirn@suse.de> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Signed-off-by: Johannes Thumshirn --- btrfs-sb-mod.c | 9 ++++++--- check/main.c | 2 +- cmds/inspect-dump-super.c | 3 ++- cmds/rescue-chunk-recover.c | 2 +- convert/common.c | 3 ++- disk-io.c | 12 ++++++------ disk-io.h | 2 +- file-item.c | 2 +- 8 files changed, 20 insertions(+), 15 deletions(-) diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c index 16a26f772494..932c2a0432ef 100644 --- a/btrfs-sb-mod.c +++ b/btrfs-sb-mod.c @@ -37,7 +37,8 @@ static int check_csum_superblock(void *sb) u32 crc = ~(u32)0; crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, - crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); + (u8 *)&crc, + BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); btrfs_csum_final(crc, result); return !memcmp(sb, &result, csum_size); @@ -50,10 +51,12 @@ static void update_block_csum(void *block, int is_sb) u32 crc = ~(u32)0; if (is_sb) { - crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, crc, + crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, + (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); } else { - crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, crc, + crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, + (u8 *)&crc, BLOCKSIZE - BTRFS_CSUM_SIZE); } btrfs_csum_final(crc, result); diff --git a/check/main.c b/check/main.c index 2e16b4e6f05b..0f02c85368ec 100644 --- a/check/main.c +++ b/check/main.c @@ -5660,7 +5660,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr, tmp = offset + data_checked; csum = btrfs_csum_data((char *)data + tmp, - csum, fs_info->sectorsize); + (u8 *)&csum, fs_info->sectorsize); btrfs_csum_final(csum, (u8 *)&csum); csum_offset = leaf_offset + diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c index 65fb3506eac6..96ad3deca3d8 100644 --- a/cmds/inspect-dump-super.c +++ b/cmds/inspect-dump-super.c @@ -41,7 +41,8 @@ static int check_csum_sblock(void *sb, int csum_size) u32 crc = ~(u32)0; crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, - crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); + (u8 *)&crc, + BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); btrfs_csum_final(crc, result); return !memcmp(sb, &result, csum_size); diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c index c2a26ad040ee..37c024ed8965 100644 --- a/cmds/rescue-chunk-recover.c +++ b/cmds/rescue-chunk-recover.c @@ -1902,7 +1902,7 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum) goto out; } ret = 0; - csum_result = btrfs_csum_data(data, csum_result, len); + csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len); btrfs_csum_final(csum_result, (u8 *)&csum_result); if (csum_result != tree_csum) ret = 1; diff --git a/convert/common.c b/convert/common.c index f8bbb23cba89..ab8e6b9f4749 100644 --- a/convert/common.c +++ b/convert/common.c @@ -65,7 +65,8 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb, u32 crc = ~(u32)0; int ret; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc, + crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, + (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); btrfs_csum_final(crc, &sb->csum[0]); ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr); diff --git a/disk-io.c b/disk-io.c index a4995a628210..a0c37c569d58 100644 --- a/disk-io.c +++ b/disk-io.c @@ -138,9 +138,9 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info, } } -u32 btrfs_csum_data(char *data, u32 seed, size_t len) +u32 btrfs_csum_data(char *data, u8 *seed, size_t len) { - return crc32c(seed, data, len); + return crc32c(*(u32*)seed, data, len); } void btrfs_csum_final(u32 crc, u8 *result) @@ -156,7 +156,7 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, u32 crc = ~(u32)0; len = buf->len - BTRFS_CSUM_SIZE; - crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, crc, len); + crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len); btrfs_csum_final(crc, result); if (verify) { @@ -1376,7 +1376,7 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags) csum_size = btrfs_csum_sizes[csum_type]; crc = ~(u32)0; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc, + crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); btrfs_csum_final(crc, result); @@ -1631,7 +1631,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) { btrfs_set_super_bytenr(sb, fs_info->super_bytenr); crc = ~(u32)0; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc, + crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); btrfs_csum_final(crc, &sb->csum[0]); @@ -1667,7 +1667,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, btrfs_set_super_bytenr(sb, bytenr); crc = ~(u32)0; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc, + crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); btrfs_csum_final(crc, &sb->csum[0]); diff --git a/disk-io.h b/disk-io.h index 394997ad72cb..92c87f28f8b2 100644 --- a/disk-io.h +++ b/disk-io.h @@ -186,7 +186,7 @@ int btrfs_free_fs_root(struct btrfs_root *root); void btrfs_mark_buffer_dirty(struct extent_buffer *buf); int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid); int btrfs_set_buffer_uptodate(struct extent_buffer *buf); -u32 btrfs_csum_data(char *data, u32 seed, size_t len); +u32 btrfs_csum_data(char *data, u8 *seed, size_t len); void btrfs_csum_final(u32 crc, u8 *result); int btrfs_open_device(struct btrfs_device *dev); diff --git a/file-item.c b/file-item.c index 3bf48c68913d..5f6548e9a74f 100644 --- a/file-item.c +++ b/file-item.c @@ -312,7 +312,7 @@ csum: item = (struct btrfs_csum_item *)((unsigned char *)item + csum_offset * csum_size); found: - csum_result = btrfs_csum_data(data, csum_result, len); + csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len); btrfs_csum_final(csum_result, (u8 *)&csum_result); if (csum_result == 0) { printk("csum result is 0 for block %llu\n", From patchwork Tue Sep 3 15:00:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11128115 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18C96112C for ; Tue, 3 Sep 2019 15:00:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EE65922D6D for ; Tue, 3 Sep 2019 15:00:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729730AbfICPAx (ORCPT ); Tue, 3 Sep 2019 11:00:53 -0400 Received: from mx2.suse.de ([195.135.220.15]:57490 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728854AbfICPAw (ORCPT ); Tue, 3 Sep 2019 11:00:52 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 3C948AECA; Tue, 3 Sep 2019 15:00:48 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [PATCH v4 05/12] btrfs-progs: pass checksum type to btrfs_csum_data()/btrfs_csum_final() Date: Tue, 3 Sep 2019 17:00:39 +0200 Message-Id: <20190903150046.14926-6-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190903150046.14926-1-jthumshirn@suse.de> References: <20190903150046.14926-1-jthumshirn@suse.de> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org In preparation to supporting new checksum algorithm pass the checksum type to btrfs_csum_data/btrfs_csum_final, this allows us to encapsulate any differences in processing into the respective functions Signed-off-by: Johannes Thumshirn --- Changes to v1: - Update changelog and squashed the other identical patch --- btrfs-sb-mod.c | 15 ++++++++++----- check/main.c | 6 ++++-- cmds/inspect-dump-super.c | 8 ++++---- cmds/rescue-chunk-recover.c | 10 ++++++---- convert/common.c | 5 +++-- disk-io.c | 39 +++++++++++++++++++++++++++------------ disk-io.h | 4 ++-- file-item.c | 7 +++++-- free-space-cache.c | 2 +- image/main.c | 2 +- 10 files changed, 63 insertions(+), 35 deletions(-) diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c index 932c2a0432ef..d9630f187d0f 100644 --- a/btrfs-sb-mod.c +++ b/btrfs-sb-mod.c @@ -35,11 +35,13 @@ static int check_csum_superblock(void *sb) { u8 result[csum_size]; u32 crc = ~(u32)0; + u16 csum_type = btrfs_super_csum_type(sb); - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, + crc = btrfs_csum_data(csum_type, + (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(crc, result); + btrfs_csum_final(csum_type, crc, result); return !memcmp(sb, &result, csum_size); } @@ -49,17 +51,20 @@ static void update_block_csum(void *block, int is_sb) u8 result[csum_size]; struct btrfs_header *hdr; u32 crc = ~(u32)0; + u16 csum_type = btrfs_super_csum_type(block); if (is_sb) { - crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, + crc = btrfs_csum_data(csum_type, + (char *)block + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); } else { - crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, + crc = btrfs_csum_data(csum_type, + (char *)block + BTRFS_CSUM_SIZE, (u8 *)&crc, BLOCKSIZE - BTRFS_CSUM_SIZE); } - btrfs_csum_final(crc, result); + btrfs_csum_final(csum_type, crc, result); memset(block, 0, BTRFS_CSUM_SIZE); hdr = (struct btrfs_header *)block; memcpy(&hdr->csum, result, csum_size); diff --git a/check/main.c b/check/main.c index 0f02c85368ec..1896a28f157d 100644 --- a/check/main.c +++ b/check/main.c @@ -5619,6 +5619,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr, struct btrfs_fs_info *fs_info = root->fs_info; u64 offset = 0; u16 csum_size = btrfs_super_csum_size(fs_info->super_copy); + u16 csum_type = btrfs_super_csum_type(fs_info->super_copy); char *data; unsigned long csum_offset; u32 csum; @@ -5659,9 +5660,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr, csum = ~(u32)0; tmp = offset + data_checked; - csum = btrfs_csum_data((char *)data + tmp, + csum = btrfs_csum_data(csum_type, + (char *)data + tmp, (u8 *)&csum, fs_info->sectorsize); - btrfs_csum_final(csum, (u8 *)&csum); + btrfs_csum_final(csum_type, csum, (u8 *)&csum); csum_offset = leaf_offset + tmp / fs_info->sectorsize * csum_size; diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c index 96ad3deca3d8..40019a6670ef 100644 --- a/cmds/inspect-dump-super.c +++ b/cmds/inspect-dump-super.c @@ -35,15 +35,15 @@ #include "kernel-lib/crc32c.h" #include "common/help.h" -static int check_csum_sblock(void *sb, int csum_size) +static int check_csum_sblock(void *sb, int csum_size, u16 csum_type) { u8 result[BTRFS_CSUM_SIZE]; u32 crc = ~(u32)0; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, + crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(crc, result); + btrfs_csum_final(csum_type, crc, result); return !memcmp(sb, &result, csum_size); } @@ -348,7 +348,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full) if (csum_type != BTRFS_CSUM_TYPE_CRC32 || csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]) printf(" [UNKNOWN CSUM TYPE OR SIZE]"); - else if (check_csum_sblock(sb, csum_size)) + else if (check_csum_sblock(sb, csum_size, csum_type)) printf(" [match]"); else printf(" [DON'T MATCH]"); diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c index 37c024ed8965..d556323ec34a 100644 --- a/cmds/rescue-chunk-recover.c +++ b/cmds/rescue-chunk-recover.c @@ -1887,7 +1887,8 @@ static u64 calc_data_offset(struct btrfs_key *key, return dev_offset + data_offset; } -static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum) +static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum, + u16 csum_type) { char *data; int ret = 0; @@ -1902,8 +1903,8 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum) goto out; } ret = 0; - csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len); - btrfs_csum_final(csum_result, (u8 *)&csum_result); + csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len); + btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result); if (csum_result != tree_csum) ret = 1; out: @@ -2102,7 +2103,8 @@ next_csum: devext->objectid, 1)); ret = check_one_csum(dev->fd, data_offset, blocksize, - tree_csum); + tree_csum, + btrfs_super_csum_type(root->fs_info->super_copy)); if (ret < 0) goto fail_out; else if (ret > 0) diff --git a/convert/common.c b/convert/common.c index ab8e6b9f4749..894a6ee0ba90 100644 --- a/convert/common.c +++ b/convert/common.c @@ -63,12 +63,13 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb, u64 sb_bytenr) { u32 crc = ~(u32)0; + u16 csum_type = btrfs_super_csum_type(sb); int ret; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, + crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(crc, &sb->csum[0]); + btrfs_csum_final(csum_type, crc, &sb->csum[0]); ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr); if (ret < BTRFS_SUPER_INFO_SIZE) ret = (ret < 0 ? -errno : -EIO); diff --git a/disk-io.c b/disk-io.c index a0c37c569d58..7e538969c57a 100644 --- a/disk-io.c +++ b/disk-io.c @@ -138,14 +138,26 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info, } } -u32 btrfs_csum_data(char *data, u8 *seed, size_t len) +u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len) { - return crc32c(*(u32*)seed, data, len); + switch (csum_type) { + case BTRFS_CSUM_TYPE_CRC32: + return crc32c(*(u32*)seed, data, len); + default: /* Not reached */ + return ~(u32)0; + } + } -void btrfs_csum_final(u32 crc, u8 *result) +void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result) { - put_unaligned_le32(~crc, result); + switch (csum_type) { + case BTRFS_CSUM_TYPE_CRC32: + put_unaligned_le32(~crc, result); + break; + default: /* Not reached */ + break; + } } static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, @@ -156,8 +168,9 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, u32 crc = ~(u32)0; len = buf->len - BTRFS_CSUM_SIZE; - crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len); - btrfs_csum_final(crc, result); + crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE, + (u8 *)&crc, len); + btrfs_csum_final(csum_type, crc, result); if (verify) { if (memcmp_extent_buffer(buf, result, 0, csum_size)) { @@ -1376,9 +1389,10 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags) csum_size = btrfs_csum_sizes[csum_type]; crc = ~(u32)0; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, + crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE, + (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(crc, result); + btrfs_csum_final(csum_type, crc, result); if (memcmp(result, sb->csum, csum_size)) { error("superblock checksum mismatch"); @@ -1616,6 +1630,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, u64 bytenr; u32 crc; int i, ret; + u16 csum_type = btrfs_super_csum_type(sb); /* * We need to write super block after all metadata written. @@ -1631,9 +1646,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) { btrfs_set_super_bytenr(sb, fs_info->super_bytenr); crc = ~(u32)0; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, + crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(crc, &sb->csum[0]); + btrfs_csum_final(csum_type, crc, &sb->csum[0]); /* * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is @@ -1667,9 +1682,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, btrfs_set_super_bytenr(sb, bytenr); crc = ~(u32)0; - crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, + crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(crc, &sb->csum[0]); + btrfs_csum_final(csum_type, crc, &sb->csum[0]); /* * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is diff --git a/disk-io.h b/disk-io.h index 92c87f28f8b2..4b5e9ea86385 100644 --- a/disk-io.h +++ b/disk-io.h @@ -186,8 +186,8 @@ int btrfs_free_fs_root(struct btrfs_root *root); void btrfs_mark_buffer_dirty(struct extent_buffer *buf); int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid); int btrfs_set_buffer_uptodate(struct extent_buffer *buf); -u32 btrfs_csum_data(char *data, u8 *seed, size_t len); -void btrfs_csum_final(u32 crc, u8 *result); +u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len); +void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result); int btrfs_open_device(struct btrfs_device *dev); int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize, diff --git a/file-item.c b/file-item.c index 5f6548e9a74f..78fdcecd0bab 100644 --- a/file-item.c +++ b/file-item.c @@ -202,6 +202,9 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans, u16 csum_size = btrfs_super_csum_size(root->fs_info->super_copy); + u16 csum_type = + btrfs_super_csum_type(root->fs_info->super_copy); + path = btrfs_alloc_path(); if (!path) return -ENOMEM; @@ -312,8 +315,8 @@ csum: item = (struct btrfs_csum_item *)((unsigned char *)item + csum_offset * csum_size); found: - csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len); - btrfs_csum_final(csum_result, (u8 *)&csum_result); + csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len); + btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result); if (csum_result == 0) { printk("csum result is 0 for block %llu\n", (unsigned long long)bytenr); diff --git a/free-space-cache.c b/free-space-cache.c index e872eb6a00db..8a57f86dc650 100644 --- a/free-space-cache.c +++ b/free-space-cache.c @@ -213,7 +213,7 @@ static int io_ctl_check_crc(struct io_ctl *io_ctl, int index) io_ctl_map_page(io_ctl, 0); crc = crc32c(crc, io_ctl->orig + offset, io_ctl->root->fs_info->sectorsize - offset); - btrfs_csum_final(crc, (u8 *)&crc); + put_unaligned_le32(~crc, (u8 *)&crc); if (val != crc) { printk("btrfs: csum mismatch on free space cache\n"); io_ctl_unmap_page(io_ctl); diff --git a/image/main.c b/image/main.c index 28ef1609b5ff..0c8ffede56f5 100644 --- a/image/main.c +++ b/image/main.c @@ -124,7 +124,7 @@ static void csum_block(u8 *buf, size_t len) u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]]; u32 crc = ~(u32)0; crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE); - btrfs_csum_final(crc, result); + put_unaligned_le32(~crc, result); memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]); } From patchwork Tue Sep 3 15:00:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11128131 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6F73A16B1 for ; Tue, 3 Sep 2019 15:01:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5B0E92339D for ; Tue, 3 Sep 2019 15:01:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729777AbfICPBE (ORCPT ); Tue, 3 Sep 2019 11:01:04 -0400 Received: from mx2.suse.de ([195.135.220.15]:57492 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729357AbfICPAu (ORCPT ); Tue, 3 Sep 2019 11:00:50 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 49B4CAEFF; Tue, 3 Sep 2019 15:00:48 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [PATCH v4 06/12] btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c Date: Tue, 3 Sep 2019 17:00:40 +0200 Message-Id: <20190903150046.14926-7-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190903150046.14926-1-jthumshirn@suse.de> References: <20190903150046.14926-1-jthumshirn@suse.de> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org update_block_csum() in btrfs-sb-mod.c is always called with the 'is_sb' argument set to 1. Get rid of the special case for is_sb == 0. Signed-off-by: Johannes Thumshirn Reviewed-by: Nikolay Borisov --- btrfs-sb-mod.c | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c index d9630f187d0f..9e64b34d2b8f 100644 --- a/btrfs-sb-mod.c +++ b/btrfs-sb-mod.c @@ -46,24 +46,17 @@ static int check_csum_superblock(void *sb) return !memcmp(sb, &result, csum_size); } -static void update_block_csum(void *block, int is_sb) +static void update_block_csum(void *block) { u8 result[csum_size]; struct btrfs_header *hdr; u32 crc = ~(u32)0; u16 csum_type = btrfs_super_csum_type(block); - if (is_sb) { - crc = btrfs_csum_data(csum_type, - (char *)block + BTRFS_CSUM_SIZE, - (u8 *)&crc, - BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - } else { - crc = btrfs_csum_data(csum_type, - (char *)block + BTRFS_CSUM_SIZE, - (u8 *)&crc, - BLOCKSIZE - BTRFS_CSUM_SIZE); - } + crc = btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE, + (u8 *)&crc, + BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); + btrfs_csum_final(csum_type, crc, result); memset(block, 0, BTRFS_CSUM_SIZE); hdr = (struct btrfs_header *)block; @@ -354,7 +347,7 @@ int main(int argc, char **argv) { if (changed) { printf("Update csum\n"); - update_block_csum(buf, 1); + update_block_csum(buf); ret = pwrite(fd, buf, BLOCKSIZE, off); if (ret <= 0) { printf("pwrite error %d at offset %llu\n", From patchwork Tue Sep 3 15:00:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11128113 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9EFF7112C for ; Tue, 3 Sep 2019 15:00:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6A0942339D for ; Tue, 3 Sep 2019 15:00:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729721AbfICPAw (ORCPT ); Tue, 3 Sep 2019 11:00:52 -0400 Received: from mx2.suse.de ([195.135.220.15]:57500 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729696AbfICPAv (ORCPT ); Tue, 3 Sep 2019 11:00:51 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 4CAB9AF06; Tue, 3 Sep 2019 15:00:48 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [PATCH v4 07/12] btrfs-progs: update checksumming api Date: Tue, 3 Sep 2019 17:00:41 +0200 Message-Id: <20190903150046.14926-8-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190903150046.14926-1-jthumshirn@suse.de> References: <20190903150046.14926-1-jthumshirn@suse.de> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: David Sterba Update the checksumming API to be able to cope with more checksum types than just CRC32C. Co-developed-by: David Sterba Signed-off-by: David Sterba Signed-off-by: Johannes Thumshirn --- btrfs-sb-mod.c | 21 ++++++---------- check/main.c | 20 +++++++-------- cmds/inspect-dump-super.c | 9 +++---- cmds/rescue-chunk-recover.c | 12 ++++++--- convert/common.c | 9 +++---- disk-io.c | 59 ++++++++++++++++++--------------------------- disk-io.h | 3 +-- file-item.c | 8 +++--- 8 files changed, 60 insertions(+), 81 deletions(-) diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c index 9e64b34d2b8f..105b556b0cf1 100644 --- a/btrfs-sb-mod.c +++ b/btrfs-sb-mod.c @@ -33,31 +33,24 @@ static int csum_size; static int check_csum_superblock(void *sb) { - u8 result[csum_size]; - u32 crc = ~(u32)0; + u8 result[BTRFS_CSUM_SIZE]; u16 csum_type = btrfs_super_csum_type(sb); - crc = btrfs_csum_data(csum_type, - (char *)sb + BTRFS_CSUM_SIZE, - (u8 *)&crc, - BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(csum_type, crc, result); + btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, + result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - return !memcmp(sb, &result, csum_size); + return !memcmp(sb, result, csum_size); } static void update_block_csum(void *block) { - u8 result[csum_size]; + u8 result[BTRFS_CSUM_SIZE]; struct btrfs_header *hdr; - u32 crc = ~(u32)0; u16 csum_type = btrfs_super_csum_type(block); - crc = btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE, - (u8 *)&crc, - BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); + btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE, + result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(csum_type, crc, result); memset(block, 0, BTRFS_CSUM_SIZE); hdr = (struct btrfs_header *)block; memcpy(&hdr->csum, result, csum_size); diff --git a/check/main.c b/check/main.c index 1896a28f157d..5028599a773a 100644 --- a/check/main.c +++ b/check/main.c @@ -5620,10 +5620,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr, u64 offset = 0; u16 csum_size = btrfs_super_csum_size(fs_info->super_copy); u16 csum_type = btrfs_super_csum_type(fs_info->super_copy); - char *data; + u8 *data; unsigned long csum_offset; - u32 csum; - u32 csum_expected; + u8 result[BTRFS_CSUM_SIZE]; + u8 csum_expected[BTRFS_CSUM_SIZE]; u64 read_len; u64 data_checked = 0; u64 tmp; @@ -5649,7 +5649,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr, for (mirror = 1; mirror <= num_copies; mirror++) { read_len = num_bytes - offset; /* read as much space once a time */ - ret = read_extent_data(fs_info, data + offset, + ret = read_extent_data(fs_info, (char *)data + offset, bytenr + offset, &read_len, mirror); if (ret) goto out; @@ -5657,24 +5657,22 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr, data_checked = 0; /* verify every 4k data's checksum */ while (data_checked < read_len) { - csum = ~(u32)0; tmp = offset + data_checked; - csum = btrfs_csum_data(csum_type, - (char *)data + tmp, - (u8 *)&csum, fs_info->sectorsize); - btrfs_csum_final(csum_type, csum, (u8 *)&csum); + btrfs_csum_data(csum_type, data + tmp, + result, fs_info->sectorsize); csum_offset = leaf_offset + tmp / fs_info->sectorsize * csum_size; read_extent_buffer(eb, (char *)&csum_expected, csum_offset, csum_size); - if (csum != csum_expected) { + if (memcmp(result, csum_expected, csum_size) != 0) { csum_mismatch = true; + /* FIXME: format */ fprintf(stderr, "mirror %d bytenr %llu csum %u expected csum %u\n", mirror, bytenr + tmp, - csum, csum_expected); + result[0], csum_expected[0]); } data_checked += fs_info->sectorsize; } diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c index 40019a6670ef..58bf82b0bbd3 100644 --- a/cmds/inspect-dump-super.c +++ b/cmds/inspect-dump-super.c @@ -38,14 +38,11 @@ static int check_csum_sblock(void *sb, int csum_size, u16 csum_type) { u8 result[BTRFS_CSUM_SIZE]; - u32 crc = ~(u32)0; - crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, - (u8 *)&crc, - BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(csum_type, crc, result); + btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, + result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - return !memcmp(sb, &result, csum_size); + return !memcmp(sb, result, csum_size); } static void print_sys_chunk_array(struct btrfs_super_block *sb) diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c index d556323ec34a..329a608dfc6b 100644 --- a/cmds/rescue-chunk-recover.c +++ b/cmds/rescue-chunk-recover.c @@ -1892,7 +1892,11 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum, { char *data; int ret = 0; - u32 csum_result = ~(u32)0; + u8 result[BTRFS_CSUM_SIZE]; + int csum_size = 0; + u8 expected_csum[BTRFS_CSUM_SIZE]; + + ASSERT(0); data = malloc(len); if (!data) @@ -1903,9 +1907,9 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum, goto out; } ret = 0; - csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len); - btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result); - if (csum_result != tree_csum) + put_unaligned_le32(tree_csum, expected_csum); + btrfs_csum_data(csum_type, (u8 *)data, result, len); + if (memcmp(result, expected_csum, csum_size) != 0) ret = 1; out: free(data); diff --git a/convert/common.c b/convert/common.c index 894a6ee0ba90..8cae507ec0f7 100644 --- a/convert/common.c +++ b/convert/common.c @@ -62,14 +62,13 @@ static int reserve_free_space(struct cache_tree *free_tree, u64 len, static inline int write_temp_super(int fd, struct btrfs_super_block *sb, u64 sb_bytenr) { - u32 crc = ~(u32)0; + u8 result[BTRFS_CSUM_SIZE]; u16 csum_type = btrfs_super_csum_type(sb); int ret; - crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, - (u8 *)&crc, - BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(csum_type, crc, &sb->csum[0]); + btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, + result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); + memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE); ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr); if (ret < BTRFS_SUPER_INFO_SIZE) ret = (ret < 0 ? -errno : -EIO); diff --git a/disk-io.c b/disk-io.c index 7e538969c57a..810c2e14294a 100644 --- a/disk-io.c +++ b/disk-io.c @@ -138,26 +138,22 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info, } } -u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len) +int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len) { - switch (csum_type) { - case BTRFS_CSUM_TYPE_CRC32: - return crc32c(*(u32*)seed, data, len); - default: /* Not reached */ - return ~(u32)0; - } - -} + u32 crc = ~(u32)0; + memset(out, 0, BTRFS_CSUM_SIZE); -void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result) -{ switch (csum_type) { case BTRFS_CSUM_TYPE_CRC32: - put_unaligned_le32(~crc, result); - break; - default: /* Not reached */ - break; + crc = crc32c(crc, data, len); + put_unaligned_le32(~crc, out); + return 0; + default: + fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type); + ASSERT(0); } + + return -1; } static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, @@ -165,20 +161,19 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, { u8 result[BTRFS_CSUM_SIZE]; u32 len; - u32 crc = ~(u32)0; len = buf->len - BTRFS_CSUM_SIZE; - crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE, - (u8 *)&crc, len); - btrfs_csum_final(csum_type, crc, result); + btrfs_csum_data(csum_type, (u8 *)buf->data + BTRFS_CSUM_SIZE, + result, len); if (verify) { if (memcmp_extent_buffer(buf, result, 0, csum_size)) { + /* FIXME: format */ if (!silent) printk("checksum verify failed on %llu found %08X wanted %08X\n", (unsigned long long)buf->start, - *((u32 *)result), - *((u32*)(char *)buf->data)); + result[0], + buf->data[0]); return 1; } } else { @@ -1367,7 +1362,6 @@ struct btrfs_root *open_ctree_fd(int fp, const char *path, u64 sb_bytenr, int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags) { u8 result[BTRFS_CSUM_SIZE]; - u32 crc; u16 csum_type; int csum_size; u8 *metadata_uuid; @@ -1388,11 +1382,8 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags) } csum_size = btrfs_csum_sizes[csum_type]; - crc = ~(u32)0; - crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE, - (u8 *)&crc, - BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(csum_type, crc, result); + btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, + result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); if (memcmp(result, sb->csum, csum_size)) { error("superblock checksum mismatch"); @@ -1628,7 +1619,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, struct btrfs_device *device) { u64 bytenr; - u32 crc; + u8 result[BTRFS_CSUM_SIZE]; int i, ret; u16 csum_type = btrfs_super_csum_type(sb); @@ -1645,10 +1636,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, } if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) { btrfs_set_super_bytenr(sb, fs_info->super_bytenr); - crc = ~(u32)0; - crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, - BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(csum_type, crc, &sb->csum[0]); + btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, result, + BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); + memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE); /* * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is @@ -1681,10 +1671,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info, btrfs_set_super_bytenr(sb, bytenr); - crc = ~(u32)0; - crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc, + btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); - btrfs_csum_final(csum_type, crc, &sb->csum[0]); + memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE); /* * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is diff --git a/disk-io.h b/disk-io.h index 4b5e9ea86385..545cacda9a79 100644 --- a/disk-io.h +++ b/disk-io.h @@ -186,8 +186,7 @@ int btrfs_free_fs_root(struct btrfs_root *root); void btrfs_mark_buffer_dirty(struct extent_buffer *buf); int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid); int btrfs_set_buffer_uptodate(struct extent_buffer *buf); -u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len); -void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result); +int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len); int btrfs_open_device(struct btrfs_device *dev); int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize, diff --git a/file-item.c b/file-item.c index 78fdcecd0bab..c6e9d212bcab 100644 --- a/file-item.c +++ b/file-item.c @@ -195,7 +195,7 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans, struct btrfs_csum_item *item; struct extent_buffer *leaf = NULL; u64 csum_offset; - u32 csum_result = ~(u32)0; + u8 csum_result[BTRFS_CSUM_SIZE]; u32 sectorsize = root->fs_info->sectorsize; u32 nritems; u32 ins_size; @@ -315,14 +315,14 @@ csum: item = (struct btrfs_csum_item *)((unsigned char *)item + csum_offset * csum_size); found: - csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len); - btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result); + btrfs_csum_data(csum_type, (u8 *)data, csum_result, len); + /* FIXME: does not make sense for non-crc32c */ if (csum_result == 0) { printk("csum result is 0 for block %llu\n", (unsigned long long)bytenr); } - write_extent_buffer(leaf, &csum_result, (unsigned long)item, + write_extent_buffer(leaf, csum_result, (unsigned long)item, csum_size); btrfs_mark_buffer_dirty(path->nodes[0]); fail: From patchwork Tue Sep 3 15:00:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11128127 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6952D1813 for ; Tue, 3 Sep 2019 15:01:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 540CA2339D for ; Tue, 3 Sep 2019 15:01:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729734AbfICPBD (ORCPT ); Tue, 3 Sep 2019 11:01:03 -0400 Received: from mx2.suse.de ([195.135.220.15]:57498 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729700AbfICPAv (ORCPT ); Tue, 3 Sep 2019 11:00:51 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 5DBD9AF1E; Tue, 3 Sep 2019 15:00:48 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs Date: Tue, 3 Sep 2019 17:00:42 +0200 Message-Id: <20190903150046.14926-9-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190903150046.14926-1-jthumshirn@suse.de> References: <20190903150046.14926-1-jthumshirn@suse.de> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Add an option to mkfs to specify which checksum algorithm will be used for the filesystem. Signed-off-by: Johannes Thumshirn --- convert/common.c | 2 +- mkfs/common.c | 2 +- mkfs/main.c | 21 ++++++++++++++++++++- 3 files changed, 22 insertions(+), 3 deletions(-) diff --git a/convert/common.c b/convert/common.c index 8cae507ec0f7..2e2318a5863e 100644 --- a/convert/common.c +++ b/convert/common.c @@ -135,7 +135,7 @@ static int setup_temp_super(int fd, struct btrfs_mkfs_config *cfg, super->__unused_leafsize = cpu_to_le32(cfg->nodesize); btrfs_set_super_nodesize(super, cfg->nodesize); btrfs_set_super_stripesize(super, cfg->stripesize); - btrfs_set_super_csum_type(super, BTRFS_CSUM_TYPE_CRC32); + btrfs_set_super_csum_type(super, cfg->csum_type); btrfs_set_super_chunk_root(super, chunk_bytenr); btrfs_set_super_cache_generation(super, -1); btrfs_set_super_incompat_flags(super, cfg->features); diff --git a/mkfs/common.c b/mkfs/common.c index 9762391a8d2b..4a417bd7a306 100644 --- a/mkfs/common.c +++ b/mkfs/common.c @@ -202,7 +202,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) super.__unused_leafsize = cpu_to_le32(cfg->nodesize); btrfs_set_super_nodesize(&super, cfg->nodesize); btrfs_set_super_stripesize(&super, cfg->stripesize); - btrfs_set_super_csum_type(&super, BTRFS_CSUM_TYPE_CRC32); + btrfs_set_super_csum_type(&super, cfg->csum_type); btrfs_set_super_chunk_root_generation(&super, 1); btrfs_set_super_cache_generation(&super, -1); btrfs_set_super_incompat_flags(&super, cfg->features); diff --git a/mkfs/main.c b/mkfs/main.c index b752da13aba9..e96cbc5399a2 100644 --- a/mkfs/main.c +++ b/mkfs/main.c @@ -352,6 +352,7 @@ static void print_usage(int ret) printf("\t--shrink (with --rootdir) shrink the filled filesystem to minimal size\n"); printf("\t-K|--nodiscard do not perform whole device TRIM\n"); printf("\t-f|--force force overwrite of existing filesystem\n"); + printf("\t-C|--checksum checksum algorithm to use (default: crc32c)\n"); printf(" general:\n"); printf("\t-q|--quiet no messages except errors\n"); printf("\t-V|--version print the mkfs.btrfs version and exit\n"); @@ -386,6 +387,18 @@ static u64 parse_profile(const char *s) return 0; } +static enum btrfs_csum_type parse_csum_type(const char *s) +{ + if (strcasecmp(s, "crc32c") == 0) { + return BTRFS_CSUM_TYPE_CRC32; + } else { + error("unknown csum type %s", s); + exit(1); + } + /* not reached */ + return 0; +} + static char *parse_label(const char *input) { int len = strlen(input); @@ -832,6 +845,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv) u64 features = BTRFS_MKFS_DEFAULT_FEATURES; struct mkfs_allocation allocation = { 0 }; struct btrfs_mkfs_config mkfs_cfg; + enum btrfs_csum_type csum_type = BTRFS_CSUM_TYPE_CRC32; crc32c_optimization_init(); @@ -841,6 +855,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv) static const struct option long_options[] = { { "alloc-start", required_argument, NULL, 'A'}, { "byte-count", required_argument, NULL, 'b' }, + { "checksum", required_argument, NULL, 'C' }, { "force", no_argument, NULL, 'f' }, { "leafsize", required_argument, NULL, 'l' }, { "label", required_argument, NULL, 'L'}, @@ -860,7 +875,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv) { NULL, 0, NULL, 0} }; - c = getopt_long(argc, argv, "A:b:fl:n:s:m:d:L:O:r:U:VMKq", + c = getopt_long(argc, argv, "A:b:C:fl:n:s:m:d:L:O:r:U:VMKq", long_options, NULL); if (c < 0) break; @@ -938,6 +953,9 @@ int BOX_MAIN(mkfs)(int argc, char **argv) case GETOPT_VAL_SHRINK: shrink_rootdir = true; break; + case 'C': + csum_type = parse_csum_type(optarg); + break; case GETOPT_VAL_HELP: default: print_usage(c != GETOPT_VAL_HELP); @@ -1176,6 +1194,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv) mkfs_cfg.sectorsize = sectorsize; mkfs_cfg.stripesize = stripesize; mkfs_cfg.features = features; + mkfs_cfg.csum_type = csum_type; ret = make_btrfs(fd, &mkfs_cfg); if (ret) { From patchwork Tue Sep 3 15:00:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11128121 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B350316B1 for ; Tue, 3 Sep 2019 15:00:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 71A3E2339D for ; Tue, 3 Sep 2019 15:00:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729739AbfICPAz (ORCPT ); Tue, 3 Sep 2019 11:00:55 -0400 Received: from mx2.suse.de ([195.135.220.15]:57508 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729701AbfICPAw (ORCPT ); Tue, 3 Sep 2019 11:00:52 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 6C11AAF35; Tue, 3 Sep 2019 15:00:48 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [PATCH v4 09/12] btrfs-progs: add xxhash sources Date: Tue, 3 Sep 2019 17:00:43 +0200 Message-Id: <20190903150046.14926-10-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190903150046.14926-1-jthumshirn@suse.de> References: <20190903150046.14926-1-jthumshirn@suse.de> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: David Sterba git://github.com/Cyan4973/xxHash Signed-off-by: David Sterba Signed-off-by: Johannes Thumshirn --- crypto/xxhash.c | 1025 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ crypto/xxhash.h | 445 ++++++++++++++++++++++++ 2 files changed, 1470 insertions(+) create mode 100644 crypto/xxhash.c create mode 100644 crypto/xxhash.h diff --git a/crypto/xxhash.c b/crypto/xxhash.c new file mode 100644 index 000000000000..58bb749d3078 --- /dev/null +++ b/crypto/xxhash.c @@ -0,0 +1,1025 @@ +/* +* xxHash - Fast Hash algorithm +* Copyright (C) 2012-2016, Yann Collet +* +* BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) +* +* Redistribution and use in source and binary forms, with or without +* modification, are permitted provided that the following conditions are +* met: +* +* * Redistributions of source code must retain the above copyright +* notice, this list of conditions and the following disclaimer. +* * Redistributions in binary form must reproduce the above +* copyright notice, this list of conditions and the following disclaimer +* in the documentation and/or other materials provided with the +* distribution. +* +* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +* +* You can contact the author at : +* - xxHash homepage: http://www.xxhash.com +* - xxHash source repository : https://github.com/Cyan4973/xxHash +*/ + + +/* ************************************* +* Tuning parameters +***************************************/ +/*!XXH_FORCE_MEMORY_ACCESS : + * By default, access to unaligned memory is controlled by `memcpy()`, which is safe and portable. + * Unfortunately, on some target/compiler combinations, the generated assembly is sub-optimal. + * The below switch allow to select different access method for improved performance. + * Method 0 (default) : use `memcpy()`. Safe and portable. + * Method 1 : `__packed` statement. It depends on compiler extension (ie, not portable). + * This method is safe if your compiler supports it, and *generally* as fast or faster than `memcpy`. + * Method 2 : direct access. This method doesn't depend on compiler but violate C standard. + * It can generate buggy code on targets which do not support unaligned memory accesses. + * But in some circumstances, it's the only known way to get the most performance (ie GCC + ARMv6) + * See http://stackoverflow.com/a/32095106/646947 for details. + * Prefer these methods in priority order (0 > 1 > 2) + */ +#ifndef XXH_FORCE_MEMORY_ACCESS /* can be defined externally, on command line for example */ +# if defined(__GNUC__) && ( defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) \ + || defined(__ARM_ARCH_6K__) || defined(__ARM_ARCH_6Z__) \ + || defined(__ARM_ARCH_6ZK__) || defined(__ARM_ARCH_6T2__) ) +# define XXH_FORCE_MEMORY_ACCESS 2 +# elif (defined(__INTEL_COMPILER) && !defined(_WIN32)) || \ + (defined(__GNUC__) && ( defined(__ARM_ARCH_7__) || defined(__ARM_ARCH_7A__) \ + || defined(__ARM_ARCH_7R__) || defined(__ARM_ARCH_7M__) \ + || defined(__ARM_ARCH_7S__) )) +# define XXH_FORCE_MEMORY_ACCESS 1 +# endif +#endif + +/*!XXH_ACCEPT_NULL_INPUT_POINTER : + * If input pointer is NULL, xxHash default behavior is to dereference it, triggering a segfault. + * When this macro is enabled, xxHash actively checks input for null pointer. + * It it is, result for null input pointers is the same as a null-length input. + */ +#ifndef XXH_ACCEPT_NULL_INPUT_POINTER /* can be defined externally */ +# define XXH_ACCEPT_NULL_INPUT_POINTER 0 +#endif + +/*!XXH_FORCE_ALIGN_CHECK : + * This is a minor performance trick, only useful with lots of very small keys. + * It means : check for aligned/unaligned input. + * The check costs one initial branch per hash; + * set it to 0 when the input is guaranteed to be aligned, + * or when alignment doesn't matter for performance. + */ +#ifndef XXH_FORCE_ALIGN_CHECK /* can be defined externally */ +# if defined(__i386) || defined(_M_IX86) || defined(__x86_64__) || defined(_M_X64) +# define XXH_FORCE_ALIGN_CHECK 0 +# else +# define XXH_FORCE_ALIGN_CHECK 1 +# endif +#endif + + +/* ************************************* +* Includes & Memory related functions +***************************************/ +/*! Modify the local functions below should you wish to use some other memory routines +* for malloc(), free() */ +#include +static void* XXH_malloc(size_t s) { return malloc(s); } +static void XXH_free (void* p) { free(p); } +/*! and for memcpy() */ +#include +static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); } + +#include /* assert */ + +#define XXH_STATIC_LINKING_ONLY +#include "xxhash.h" + + +/* ************************************* +* Compiler Specific Options +***************************************/ +#ifdef _MSC_VER /* Visual Studio */ +# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ +# define XXH_FORCE_INLINE static __forceinline +# define XXH_NO_INLINE static __declspec(noinline) +#else +# if defined (__cplusplus) || defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */ +# ifdef __GNUC__ +# define XXH_FORCE_INLINE static inline __attribute__((always_inline)) +# define XXH_NO_INLINE static __attribute__((noinline)) +# else +# define XXH_FORCE_INLINE static inline +# define XXH_NO_INLINE static +# endif +# else +# define XXH_FORCE_INLINE static +# define XXH_NO_INLINE static +# endif /* __STDC_VERSION__ */ +#endif + + +/* ************************************* +* Basic Types +***************************************/ +#ifndef MEM_MODULE +# if !defined (__VMS) \ + && (defined (__cplusplus) \ + || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) +# include + typedef uint8_t BYTE; + typedef uint16_t U16; + typedef uint32_t U32; +# else + typedef unsigned char BYTE; + typedef unsigned short U16; + typedef unsigned int U32; +# endif +#endif + + +/* === Memory access === */ + +#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2)) + +/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */ +static U32 XXH_read32(const void* memPtr) { return *(const U32*) memPtr; } + +#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1)) + +/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */ +/* currently only defined for gcc and icc */ +typedef union { U32 u32; } __attribute__((packed)) unalign; +static U32 XXH_read32(const void* ptr) { return ((const unalign*)ptr)->u32; } + +#else + +/* portable and safe solution. Generally efficient. + * see : http://stackoverflow.com/a/32095106/646947 + */ +static U32 XXH_read32(const void* memPtr) +{ + U32 val; + memcpy(&val, memPtr, sizeof(val)); + return val; +} + +#endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */ + + +/* === Endianess === */ +typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess; + +/* XXH_CPU_LITTLE_ENDIAN can be defined externally, for example on the compiler command line */ +#ifndef XXH_CPU_LITTLE_ENDIAN +static int XXH_isLittleEndian(void) +{ + const union { U32 u; BYTE c[4]; } one = { 1 }; /* don't use static : performance detrimental */ + return one.c[0]; +} +# define XXH_CPU_LITTLE_ENDIAN XXH_isLittleEndian() +#endif + + + + +/* **************************************** +* Compiler-specific Functions and Macros +******************************************/ +#define XXH_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) + +/* Note : although _rotl exists for minGW (GCC under windows), performance seems poor */ +#if defined(_MSC_VER) +# define XXH_rotl32(x,r) _rotl(x,r) +# define XXH_rotl64(x,r) _rotl64(x,r) +#else +# define XXH_rotl32(x,r) (((x) << (r)) | ((x) >> (32 - (r)))) +# define XXH_rotl64(x,r) (((x) << (r)) | ((x) >> (64 - (r)))) +#endif + +#if defined(_MSC_VER) /* Visual Studio */ +# define XXH_swap32 _byteswap_ulong +#elif XXH_GCC_VERSION >= 403 +# define XXH_swap32 __builtin_bswap32 +#else +static U32 XXH_swap32 (U32 x) +{ + return ((x << 24) & 0xff000000 ) | + ((x << 8) & 0x00ff0000 ) | + ((x >> 8) & 0x0000ff00 ) | + ((x >> 24) & 0x000000ff ); +} +#endif + + +/* *************************** +* Memory reads +*****************************/ +typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment; + +XXH_FORCE_INLINE U32 XXH_readLE32(const void* ptr) +{ + return XXH_CPU_LITTLE_ENDIAN ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr)); +} + +static U32 XXH_readBE32(const void* ptr) +{ + return XXH_CPU_LITTLE_ENDIAN ? XXH_swap32(XXH_read32(ptr)) : XXH_read32(ptr); +} + +XXH_FORCE_INLINE U32 +XXH_readLE32_align(const void* ptr, XXH_alignment align) +{ + if (align==XXH_unaligned) { + return XXH_readLE32(ptr); + } else { + return XXH_CPU_LITTLE_ENDIAN ? *(const U32*)ptr : XXH_swap32(*(const U32*)ptr); + } +} + + +/* ************************************* +* Macros +***************************************/ +#define XXH_STATIC_ASSERT(c) { enum { XXH_sa = 1/(int)(!!(c)) }; } /* use after variable declarations */ +XXH_PUBLIC_API unsigned XXH_versionNumber (void) { return XXH_VERSION_NUMBER; } + + +/* ******************************************************************* +* 32-bit hash functions +*********************************************************************/ +static const U32 PRIME32_1 = 2654435761U; /* 0b10011110001101110111100110110001 */ +static const U32 PRIME32_2 = 2246822519U; /* 0b10000101111010111100101001110111 */ +static const U32 PRIME32_3 = 3266489917U; /* 0b11000010101100101010111000111101 */ +static const U32 PRIME32_4 = 668265263U; /* 0b00100111110101001110101100101111 */ +static const U32 PRIME32_5 = 374761393U; /* 0b00010110010101100110011110110001 */ + +static U32 XXH32_round(U32 acc, U32 input) +{ + acc += input * PRIME32_2; + acc = XXH_rotl32(acc, 13); + acc *= PRIME32_1; +#if defined(__GNUC__) && defined(__SSE4_1__) && !defined(XXH_ENABLE_AUTOVECTORIZE) + /* UGLY HACK: + * This inline assembly hack forces acc into a normal register. This is the + * only thing that prevents GCC and Clang from autovectorizing the XXH32 loop + * (pragmas and attributes don't work for some resason) without globally + * disabling SSE4.1. + * + * The reason we want to avoid vectorization is because despite working on + * 4 integers at a time, there are multiple factors slowing XXH32 down on + * SSE4: + * - There's a ridiculous amount of lag from pmulld (10 cycles of latency on newer chips!) + * making it slightly slower to multiply four integers at once compared to four + * integers independently. Even when pmulld was fastest, Sandy/Ivy Bridge, it is + * still not worth it to go into SSE just to multiply unless doing a long operation. + * + * - Four instructions are required to rotate, + * movqda tmp, v // not required with VEX encoding + * pslld tmp, 13 // tmp <<= 13 + * psrld v, 19 // x >>= 19 + * por v, tmp // x |= tmp + * compared to one for scalar: + * roll v, 13 // reliably fast across the board + * shldl v, v, 13 // Sandy Bridge and later prefer this for some reason + * + * - Instruction level parallelism is actually more beneficial here because the + * SIMD actually serializes this operation: While v1 is rotating, v2 can load data, + * while v3 can multiply. SSE forces them to operate together. + * + * How this hack works: + * __asm__("" // Declare an assembly block but don't declare any instructions + * : // However, as an Input/Output Operand, + * "+r" // constrain a read/write operand (+) as a general purpose register (r). + * (acc) // and set acc as the operand + * ); + * + * Because of the 'r', the compiler has promised that seed will be in a + * general purpose register and the '+' says that it will be 'read/write', + * so it has to assume it has changed. It is like volatile without all the + * loads and stores. + * + * Since the argument has to be in a normal register (not an SSE register), + * each time XXH32_round is called, it is impossible to vectorize. */ + __asm__("" : "+r" (acc)); +#endif + return acc; +} + +/* mix all bits */ +static U32 XXH32_avalanche(U32 h32) +{ + h32 ^= h32 >> 15; + h32 *= PRIME32_2; + h32 ^= h32 >> 13; + h32 *= PRIME32_3; + h32 ^= h32 >> 16; + return(h32); +} + +#define XXH_get32bits(p) XXH_readLE32_align(p, align) + +static U32 +XXH32_finalize(U32 h32, const void* ptr, size_t len, XXH_alignment align) + +{ + const BYTE* p = (const BYTE*)ptr; + +#define PROCESS1 \ + h32 += (*p++) * PRIME32_5; \ + h32 = XXH_rotl32(h32, 11) * PRIME32_1 ; + +#define PROCESS4 \ + h32 += XXH_get32bits(p) * PRIME32_3; \ + p+=4; \ + h32 = XXH_rotl32(h32, 17) * PRIME32_4 ; + + switch(len&15) /* or switch(bEnd - p) */ + { + case 12: PROCESS4; + /* fallthrough */ + case 8: PROCESS4; + /* fallthrough */ + case 4: PROCESS4; + return XXH32_avalanche(h32); + + case 13: PROCESS4; + /* fallthrough */ + case 9: PROCESS4; + /* fallthrough */ + case 5: PROCESS4; + PROCESS1; + return XXH32_avalanche(h32); + + case 14: PROCESS4; + /* fallthrough */ + case 10: PROCESS4; + /* fallthrough */ + case 6: PROCESS4; + PROCESS1; + PROCESS1; + return XXH32_avalanche(h32); + + case 15: PROCESS4; + /* fallthrough */ + case 11: PROCESS4; + /* fallthrough */ + case 7: PROCESS4; + /* fallthrough */ + case 3: PROCESS1; + /* fallthrough */ + case 2: PROCESS1; + /* fallthrough */ + case 1: PROCESS1; + /* fallthrough */ + case 0: return XXH32_avalanche(h32); + } + assert(0); + return h32; /* reaching this point is deemed impossible */ +} + +XXH_FORCE_INLINE U32 +XXH32_endian_align(const void* input, size_t len, U32 seed, XXH_alignment align) +{ + const BYTE* p = (const BYTE*)input; + const BYTE* bEnd = p + len; + U32 h32; + +#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1) + if (p==NULL) { + len=0; + bEnd=p=(const BYTE*)(size_t)16; + } +#endif + + if (len>=16) { + const BYTE* const limit = bEnd - 15; + U32 v1 = seed + PRIME32_1 + PRIME32_2; + U32 v2 = seed + PRIME32_2; + U32 v3 = seed + 0; + U32 v4 = seed - PRIME32_1; + + do { + v1 = XXH32_round(v1, XXH_get32bits(p)); p+=4; + v2 = XXH32_round(v2, XXH_get32bits(p)); p+=4; + v3 = XXH32_round(v3, XXH_get32bits(p)); p+=4; + v4 = XXH32_round(v4, XXH_get32bits(p)); p+=4; + } while (p < limit); + + h32 = XXH_rotl32(v1, 1) + XXH_rotl32(v2, 7) + + XXH_rotl32(v3, 12) + XXH_rotl32(v4, 18); + } else { + h32 = seed + PRIME32_5; + } + + h32 += (U32)len; + + return XXH32_finalize(h32, p, len&15, align); +} + + +XXH_PUBLIC_API unsigned int XXH32 (const void* input, size_t len, unsigned int seed) +{ +#if 0 + /* Simple version, good for code maintenance, but unfortunately slow for small inputs */ + XXH32_state_t state; + XXH32_reset(&state, seed); + XXH32_update(&state, input, len); + return XXH32_digest(&state); + +#else + + if (XXH_FORCE_ALIGN_CHECK) { + if ((((size_t)input) & 3) == 0) { /* Input is 4-bytes aligned, leverage the speed benefit */ + return XXH32_endian_align(input, len, seed, XXH_aligned); + } } + + return XXH32_endian_align(input, len, seed, XXH_unaligned); +#endif +} + + + +/*====== Hash streaming ======*/ + +XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void) +{ + return (XXH32_state_t*)XXH_malloc(sizeof(XXH32_state_t)); +} +XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr) +{ + XXH_free(statePtr); + return XXH_OK; +} + +XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dstState, const XXH32_state_t* srcState) +{ + memcpy(dstState, srcState, sizeof(*dstState)); +} + +XXH_PUBLIC_API XXH_errorcode XXH32_reset(XXH32_state_t* statePtr, unsigned int seed) +{ + XXH32_state_t state; /* using a local state to memcpy() in order to avoid strict-aliasing warnings */ + memset(&state, 0, sizeof(state)); + state.v1 = seed + PRIME32_1 + PRIME32_2; + state.v2 = seed + PRIME32_2; + state.v3 = seed + 0; + state.v4 = seed - PRIME32_1; + /* do not write into reserved, planned to be removed in a future version */ + memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved)); + return XXH_OK; +} + + +XXH_PUBLIC_API XXH_errorcode +XXH32_update(XXH32_state_t* state, const void* input, size_t len) +{ + if (input==NULL) +#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1) + return XXH_OK; +#else + return XXH_ERROR; +#endif + + { const BYTE* p = (const BYTE*)input; + const BYTE* const bEnd = p + len; + + state->total_len_32 += (XXH32_hash_t)len; + state->large_len |= (XXH32_hash_t)((len>=16) | (state->total_len_32>=16)); + + if (state->memsize + len < 16) { /* fill in tmp buffer */ + XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, len); + state->memsize += (XXH32_hash_t)len; + return XXH_OK; + } + + if (state->memsize) { /* some data left from previous update */ + XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, 16-state->memsize); + { const U32* p32 = state->mem32; + state->v1 = XXH32_round(state->v1, XXH_readLE32(p32)); p32++; + state->v2 = XXH32_round(state->v2, XXH_readLE32(p32)); p32++; + state->v3 = XXH32_round(state->v3, XXH_readLE32(p32)); p32++; + state->v4 = XXH32_round(state->v4, XXH_readLE32(p32)); + } + p += 16-state->memsize; + state->memsize = 0; + } + + if (p <= bEnd-16) { + const BYTE* const limit = bEnd - 16; + U32 v1 = state->v1; + U32 v2 = state->v2; + U32 v3 = state->v3; + U32 v4 = state->v4; + + do { + v1 = XXH32_round(v1, XXH_readLE32(p)); p+=4; + v2 = XXH32_round(v2, XXH_readLE32(p)); p+=4; + v3 = XXH32_round(v3, XXH_readLE32(p)); p+=4; + v4 = XXH32_round(v4, XXH_readLE32(p)); p+=4; + } while (p<=limit); + + state->v1 = v1; + state->v2 = v2; + state->v3 = v3; + state->v4 = v4; + } + + if (p < bEnd) { + XXH_memcpy(state->mem32, p, (size_t)(bEnd-p)); + state->memsize = (unsigned)(bEnd-p); + } + } + + return XXH_OK; +} + + +XXH_PUBLIC_API unsigned int XXH32_digest (const XXH32_state_t* state) +{ + U32 h32; + + if (state->large_len) { + h32 = XXH_rotl32(state->v1, 1) + + XXH_rotl32(state->v2, 7) + + XXH_rotl32(state->v3, 12) + + XXH_rotl32(state->v4, 18); + } else { + h32 = state->v3 /* == seed */ + PRIME32_5; + } + + h32 += state->total_len_32; + + return XXH32_finalize(h32, state->mem32, state->memsize, XXH_aligned); +} + + +/*====== Canonical representation ======*/ + +/*! Default XXH result types are basic unsigned 32 and 64 bits. +* The canonical representation follows human-readable write convention, aka big-endian (large digits first). +* These functions allow transformation of hash result into and from its canonical format. +* This way, hash values can be written into a file or buffer, remaining comparable across different systems. +*/ + +XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash) +{ + XXH_STATIC_ASSERT(sizeof(XXH32_canonical_t) == sizeof(XXH32_hash_t)); + if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap32(hash); + memcpy(dst, &hash, sizeof(*dst)); +} + +XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src) +{ + return XXH_readBE32(src); +} + + +#ifndef XXH_NO_LONG_LONG + +/* ******************************************************************* +* 64-bit hash functions +*********************************************************************/ + +/*====== Memory access ======*/ + +#ifndef MEM_MODULE +# define MEM_MODULE +# if !defined (__VMS) \ + && (defined (__cplusplus) \ + || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) +# include + typedef uint64_t U64; +# else + /* if compiler doesn't support unsigned long long, replace by another 64-bit type */ + typedef unsigned long long U64; +# endif +#endif + + +#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2)) + +/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */ +static U64 XXH_read64(const void* memPtr) { return *(const U64*) memPtr; } + +#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1)) + +/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */ +/* currently only defined for gcc and icc */ +typedef union { U32 u32; U64 u64; } __attribute__((packed)) unalign64; +static U64 XXH_read64(const void* ptr) { return ((const unalign64*)ptr)->u64; } + +#else + +/* portable and safe solution. Generally efficient. + * see : http://stackoverflow.com/a/32095106/646947 + */ + +static U64 XXH_read64(const void* memPtr) +{ + U64 val; + memcpy(&val, memPtr, sizeof(val)); + return val; +} + +#endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */ + +#if defined(_MSC_VER) /* Visual Studio */ +# define XXH_swap64 _byteswap_uint64 +#elif XXH_GCC_VERSION >= 403 +# define XXH_swap64 __builtin_bswap64 +#else +static U64 XXH_swap64 (U64 x) +{ + return ((x << 56) & 0xff00000000000000ULL) | + ((x << 40) & 0x00ff000000000000ULL) | + ((x << 24) & 0x0000ff0000000000ULL) | + ((x << 8) & 0x000000ff00000000ULL) | + ((x >> 8) & 0x00000000ff000000ULL) | + ((x >> 24) & 0x0000000000ff0000ULL) | + ((x >> 40) & 0x000000000000ff00ULL) | + ((x >> 56) & 0x00000000000000ffULL); +} +#endif + +XXH_FORCE_INLINE U64 XXH_readLE64(const void* ptr) +{ + return XXH_CPU_LITTLE_ENDIAN ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr)); +} + +static U64 XXH_readBE64(const void* ptr) +{ + return XXH_CPU_LITTLE_ENDIAN ? XXH_swap64(XXH_read64(ptr)) : XXH_read64(ptr); +} + +XXH_FORCE_INLINE U64 +XXH_readLE64_align(const void* ptr, XXH_alignment align) +{ + if (align==XXH_unaligned) + return XXH_readLE64(ptr); + else + return XXH_CPU_LITTLE_ENDIAN ? *(const U64*)ptr : XXH_swap64(*(const U64*)ptr); +} + + +/*====== xxh64 ======*/ + +static const U64 PRIME64_1 = 11400714785074694791ULL; /* 0b1001111000110111011110011011000110000101111010111100101010000111 */ +static const U64 PRIME64_2 = 14029467366897019727ULL; /* 0b1100001010110010101011100011110100100111110101001110101101001111 */ +static const U64 PRIME64_3 = 1609587929392839161ULL; /* 0b0001011001010110011001111011000110011110001101110111100111111001 */ +static const U64 PRIME64_4 = 9650029242287828579ULL; /* 0b1000010111101011110010100111011111000010101100101010111001100011 */ +static const U64 PRIME64_5 = 2870177450012600261ULL; /* 0b0010011111010100111010110010111100010110010101100110011111000101 */ + +static U64 XXH64_round(U64 acc, U64 input) +{ + acc += input * PRIME64_2; + acc = XXH_rotl64(acc, 31); + acc *= PRIME64_1; + return acc; +} + +static U64 XXH64_mergeRound(U64 acc, U64 val) +{ + val = XXH64_round(0, val); + acc ^= val; + acc = acc * PRIME64_1 + PRIME64_4; + return acc; +} + +static U64 XXH64_avalanche(U64 h64) +{ + h64 ^= h64 >> 33; + h64 *= PRIME64_2; + h64 ^= h64 >> 29; + h64 *= PRIME64_3; + h64 ^= h64 >> 32; + return h64; +} + + +#define XXH_get64bits(p) XXH_readLE64_align(p, align) + +static U64 +XXH64_finalize(U64 h64, const void* ptr, size_t len, XXH_alignment align) +{ + const BYTE* p = (const BYTE*)ptr; + +#define PROCESS1_64 \ + h64 ^= (*p++) * PRIME64_5; \ + h64 = XXH_rotl64(h64, 11) * PRIME64_1; + +#define PROCESS4_64 \ + h64 ^= (U64)(XXH_get32bits(p)) * PRIME64_1; \ + p+=4; \ + h64 = XXH_rotl64(h64, 23) * PRIME64_2 + PRIME64_3; + +#define PROCESS8_64 { \ + U64 const k1 = XXH64_round(0, XXH_get64bits(p)); \ + p+=8; \ + h64 ^= k1; \ + h64 = XXH_rotl64(h64,27) * PRIME64_1 + PRIME64_4; \ +} + + switch(len&31) { + case 24: PROCESS8_64; + /* fallthrough */ + case 16: PROCESS8_64; + /* fallthrough */ + case 8: PROCESS8_64; + return XXH64_avalanche(h64); + + case 28: PROCESS8_64; + /* fallthrough */ + case 20: PROCESS8_64; + /* fallthrough */ + case 12: PROCESS8_64; + /* fallthrough */ + case 4: PROCESS4_64; + return XXH64_avalanche(h64); + + case 25: PROCESS8_64; + /* fallthrough */ + case 17: PROCESS8_64; + /* fallthrough */ + case 9: PROCESS8_64; + PROCESS1_64; + return XXH64_avalanche(h64); + + case 29: PROCESS8_64; + /* fallthrough */ + case 21: PROCESS8_64; + /* fallthrough */ + case 13: PROCESS8_64; + /* fallthrough */ + case 5: PROCESS4_64; + PROCESS1_64; + return XXH64_avalanche(h64); + + case 26: PROCESS8_64; + /* fallthrough */ + case 18: PROCESS8_64; + /* fallthrough */ + case 10: PROCESS8_64; + PROCESS1_64; + PROCESS1_64; + return XXH64_avalanche(h64); + + case 30: PROCESS8_64; + /* fallthrough */ + case 22: PROCESS8_64; + /* fallthrough */ + case 14: PROCESS8_64; + /* fallthrough */ + case 6: PROCESS4_64; + PROCESS1_64; + PROCESS1_64; + return XXH64_avalanche(h64); + + case 27: PROCESS8_64; + /* fallthrough */ + case 19: PROCESS8_64; + /* fallthrough */ + case 11: PROCESS8_64; + PROCESS1_64; + PROCESS1_64; + PROCESS1_64; + return XXH64_avalanche(h64); + + case 31: PROCESS8_64; + /* fallthrough */ + case 23: PROCESS8_64; + /* fallthrough */ + case 15: PROCESS8_64; + /* fallthrough */ + case 7: PROCESS4_64; + /* fallthrough */ + case 3: PROCESS1_64; + /* fallthrough */ + case 2: PROCESS1_64; + /* fallthrough */ + case 1: PROCESS1_64; + /* fallthrough */ + case 0: return XXH64_avalanche(h64); + } + + /* impossible to reach */ + assert(0); + return 0; /* unreachable, but some compilers complain without it */ +} + +XXH_FORCE_INLINE U64 +XXH64_endian_align(const void* input, size_t len, U64 seed, XXH_alignment align) +{ + const BYTE* p = (const BYTE*)input; + const BYTE* bEnd = p + len; + U64 h64; + +#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1) + if (p==NULL) { + len=0; + bEnd=p=(const BYTE*)(size_t)32; + } +#endif + + if (len>=32) { + const BYTE* const limit = bEnd - 32; + U64 v1 = seed + PRIME64_1 + PRIME64_2; + U64 v2 = seed + PRIME64_2; + U64 v3 = seed + 0; + U64 v4 = seed - PRIME64_1; + + do { + v1 = XXH64_round(v1, XXH_get64bits(p)); p+=8; + v2 = XXH64_round(v2, XXH_get64bits(p)); p+=8; + v3 = XXH64_round(v3, XXH_get64bits(p)); p+=8; + v4 = XXH64_round(v4, XXH_get64bits(p)); p+=8; + } while (p<=limit); + + h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18); + h64 = XXH64_mergeRound(h64, v1); + h64 = XXH64_mergeRound(h64, v2); + h64 = XXH64_mergeRound(h64, v3); + h64 = XXH64_mergeRound(h64, v4); + + } else { + h64 = seed + PRIME64_5; + } + + h64 += (U64) len; + + return XXH64_finalize(h64, p, len, align); +} + + +XXH_PUBLIC_API unsigned long long XXH64 (const void* input, size_t len, unsigned long long seed) +{ +#if 0 + /* Simple version, good for code maintenance, but unfortunately slow for small inputs */ + XXH64_state_t state; + XXH64_reset(&state, seed); + XXH64_update(&state, input, len); + return XXH64_digest(&state); + +#else + + if (XXH_FORCE_ALIGN_CHECK) { + if ((((size_t)input) & 7)==0) { /* Input is aligned, let's leverage the speed advantage */ + return XXH64_endian_align(input, len, seed, XXH_aligned); + } } + + return XXH64_endian_align(input, len, seed, XXH_unaligned); + +#endif +} + +/*====== Hash Streaming ======*/ + +XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void) +{ + return (XXH64_state_t*)XXH_malloc(sizeof(XXH64_state_t)); +} +XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr) +{ + XXH_free(statePtr); + return XXH_OK; +} + +XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dstState, const XXH64_state_t* srcState) +{ + memcpy(dstState, srcState, sizeof(*dstState)); +} + +XXH_PUBLIC_API XXH_errorcode XXH64_reset(XXH64_state_t* statePtr, unsigned long long seed) +{ + XXH64_state_t state; /* using a local state to memcpy() in order to avoid strict-aliasing warnings */ + memset(&state, 0, sizeof(state)); + state.v1 = seed + PRIME64_1 + PRIME64_2; + state.v2 = seed + PRIME64_2; + state.v3 = seed + 0; + state.v4 = seed - PRIME64_1; + /* do not write into reserved, planned to be removed in a future version */ + memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved)); + return XXH_OK; +} + +XXH_PUBLIC_API XXH_errorcode +XXH64_update (XXH64_state_t* state, const void* input, size_t len) +{ + if (input==NULL) +#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1) + return XXH_OK; +#else + return XXH_ERROR; +#endif + + { const BYTE* p = (const BYTE*)input; + const BYTE* const bEnd = p + len; + + state->total_len += len; + + if (state->memsize + len < 32) { /* fill in tmp buffer */ + XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, len); + state->memsize += (U32)len; + return XXH_OK; + } + + if (state->memsize) { /* tmp buffer is full */ + XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, 32-state->memsize); + state->v1 = XXH64_round(state->v1, XXH_readLE64(state->mem64+0)); + state->v2 = XXH64_round(state->v2, XXH_readLE64(state->mem64+1)); + state->v3 = XXH64_round(state->v3, XXH_readLE64(state->mem64+2)); + state->v4 = XXH64_round(state->v4, XXH_readLE64(state->mem64+3)); + p += 32-state->memsize; + state->memsize = 0; + } + + if (p+32 <= bEnd) { + const BYTE* const limit = bEnd - 32; + U64 v1 = state->v1; + U64 v2 = state->v2; + U64 v3 = state->v3; + U64 v4 = state->v4; + + do { + v1 = XXH64_round(v1, XXH_readLE64(p)); p+=8; + v2 = XXH64_round(v2, XXH_readLE64(p)); p+=8; + v3 = XXH64_round(v3, XXH_readLE64(p)); p+=8; + v4 = XXH64_round(v4, XXH_readLE64(p)); p+=8; + } while (p<=limit); + + state->v1 = v1; + state->v2 = v2; + state->v3 = v3; + state->v4 = v4; + } + + if (p < bEnd) { + XXH_memcpy(state->mem64, p, (size_t)(bEnd-p)); + state->memsize = (unsigned)(bEnd-p); + } + } + + return XXH_OK; +} + + +XXH_PUBLIC_API unsigned long long XXH64_digest (const XXH64_state_t* state) +{ + U64 h64; + + if (state->total_len >= 32) { + U64 const v1 = state->v1; + U64 const v2 = state->v2; + U64 const v3 = state->v3; + U64 const v4 = state->v4; + + h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18); + h64 = XXH64_mergeRound(h64, v1); + h64 = XXH64_mergeRound(h64, v2); + h64 = XXH64_mergeRound(h64, v3); + h64 = XXH64_mergeRound(h64, v4); + } else { + h64 = state->v3 /*seed*/ + PRIME64_5; + } + + h64 += (U64) state->total_len; + + return XXH64_finalize(h64, state->mem64, (size_t)state->total_len, XXH_aligned); +} + + +/*====== Canonical representation ======*/ + +XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash) +{ + XXH_STATIC_ASSERT(sizeof(XXH64_canonical_t) == sizeof(XXH64_hash_t)); + if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap64(hash); + memcpy(dst, &hash, sizeof(*dst)); +} + +XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src) +{ + return XXH_readBE64(src); +} + + + +/* ********************************************************************* +* XXH3 +* New generation hash designed for speed on small keys and vectorization +************************************************************************ */ + +/* #include "xxh3.h" */ + + + +#endif /* XXH_NO_LONG_LONG */ diff --git a/crypto/xxhash.h b/crypto/xxhash.h new file mode 100644 index 000000000000..9ee05e5dc146 --- /dev/null +++ b/crypto/xxhash.h @@ -0,0 +1,445 @@ +/* + xxHash - Extremely Fast Hash algorithm + Header File + Copyright (C) 2012-2016, Yann Collet. + + BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following disclaimer + in the documentation and/or other materials provided with the + distribution. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + You can contact the author at : + - xxHash source repository : https://github.com/Cyan4973/xxHash +*/ + +/* Notice extracted from xxHash homepage : + +xxHash is an extremely fast Hash algorithm, running at RAM speed limits. +It also successfully passes all tests from the SMHasher suite. + +Comparison (single thread, Windows Seven 32 bits, using SMHasher on a Core 2 Duo @3GHz) + +Name Speed Q.Score Author +xxHash 5.4 GB/s 10 +CrapWow 3.2 GB/s 2 Andrew +MumurHash 3a 2.7 GB/s 10 Austin Appleby +SpookyHash 2.0 GB/s 10 Bob Jenkins +SBox 1.4 GB/s 9 Bret Mulvey +Lookup3 1.2 GB/s 9 Bob Jenkins +SuperFastHash 1.2 GB/s 1 Paul Hsieh +CityHash64 1.05 GB/s 10 Pike & Alakuijala +FNV 0.55 GB/s 5 Fowler, Noll, Vo +CRC32 0.43 GB/s 9 +MD5-32 0.33 GB/s 10 Ronald L. Rivest +SHA1-32 0.28 GB/s 10 + +Q.Score is a measure of quality of the hash function. +It depends on successfully passing SMHasher test set. +10 is a perfect score. + +A 64-bit version, named XXH64, is available since r35. +It offers much better speed, but for 64-bit applications only. +Name Speed on 64 bits Speed on 32 bits +XXH64 13.8 GB/s 1.9 GB/s +XXH32 6.8 GB/s 6.0 GB/s +*/ + +#ifndef XXHASH_H_5627135585666179 +#define XXHASH_H_5627135585666179 1 + +#if defined (__cplusplus) +extern "C" { +#endif + + +/* **************************** +* Definitions +******************************/ +#include /* size_t */ +typedef enum { XXH_OK=0, XXH_ERROR } XXH_errorcode; + + +/* **************************** + * API modifier + ******************************/ +/** XXH_INLINE_ALL (and XXH_PRIVATE_API) + * This is useful to include xxhash functions in `static` mode + * in order to inline them, and remove their symbol from the public list. + * Inlining can offer dramatic performance improvement on small keys. + * Methodology : + * #define XXH_INLINE_ALL + * #include "xxhash.h" + * `xxhash.c` is automatically included. + * It's not useful to compile and link it as a separate module. + */ +#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API) +# ifndef XXH_STATIC_LINKING_ONLY +# define XXH_STATIC_LINKING_ONLY +# endif +# if defined(__GNUC__) +# define XXH_PUBLIC_API static __inline __attribute__((unused)) +# elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) +# define XXH_PUBLIC_API static inline +# elif defined(_MSC_VER) +# define XXH_PUBLIC_API static __inline +# else + /* this version may generate warnings for unused static functions */ +# define XXH_PUBLIC_API static +# endif +#else +# if defined(WIN32) && defined(_MSC_VER) && (defined(XXH_IMPORT) || defined(XXH_EXPORT)) +# ifdef XXH_EXPORT +# define XXH_PUBLIC_API __declspec(dllexport) +# elif XXH_IMPORT +# define XXH_PUBLIC_API __declspec(dllimport) +# endif +# else +# define XXH_PUBLIC_API /* do nothing */ +# endif +#endif /* XXH_INLINE_ALL || XXH_PRIVATE_API */ + +/*! XXH_NAMESPACE, aka Namespace Emulation : + * + * If you want to include _and expose_ xxHash functions from within your own library, + * but also want to avoid symbol collisions with other libraries which may also include xxHash, + * + * you can use XXH_NAMESPACE, to automatically prefix any public symbol from xxhash library + * with the value of XXH_NAMESPACE (therefore, avoid NULL and numeric values). + * + * Note that no change is required within the calling program as long as it includes `xxhash.h` : + * regular symbol name will be automatically translated by this header. + */ +#ifdef XXH_NAMESPACE +# define XXH_CAT(A,B) A##B +# define XXH_NAME2(A,B) XXH_CAT(A,B) +# define XXH_versionNumber XXH_NAME2(XXH_NAMESPACE, XXH_versionNumber) +# define XXH32 XXH_NAME2(XXH_NAMESPACE, XXH32) +# define XXH32_createState XXH_NAME2(XXH_NAMESPACE, XXH32_createState) +# define XXH32_freeState XXH_NAME2(XXH_NAMESPACE, XXH32_freeState) +# define XXH32_reset XXH_NAME2(XXH_NAMESPACE, XXH32_reset) +# define XXH32_update XXH_NAME2(XXH_NAMESPACE, XXH32_update) +# define XXH32_digest XXH_NAME2(XXH_NAMESPACE, XXH32_digest) +# define XXH32_copyState XXH_NAME2(XXH_NAMESPACE, XXH32_copyState) +# define XXH32_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH32_canonicalFromHash) +# define XXH32_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH32_hashFromCanonical) +# define XXH64 XXH_NAME2(XXH_NAMESPACE, XXH64) +# define XXH64_createState XXH_NAME2(XXH_NAMESPACE, XXH64_createState) +# define XXH64_freeState XXH_NAME2(XXH_NAMESPACE, XXH64_freeState) +# define XXH64_reset XXH_NAME2(XXH_NAMESPACE, XXH64_reset) +# define XXH64_update XXH_NAME2(XXH_NAMESPACE, XXH64_update) +# define XXH64_digest XXH_NAME2(XXH_NAMESPACE, XXH64_digest) +# define XXH64_copyState XXH_NAME2(XXH_NAMESPACE, XXH64_copyState) +# define XXH64_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH64_canonicalFromHash) +# define XXH64_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH64_hashFromCanonical) +#endif + + +/* ************************************* +* Version +***************************************/ +#define XXH_VERSION_MAJOR 0 +#define XXH_VERSION_MINOR 7 +#define XXH_VERSION_RELEASE 0 +#define XXH_VERSION_NUMBER (XXH_VERSION_MAJOR *100*100 + XXH_VERSION_MINOR *100 + XXH_VERSION_RELEASE) +XXH_PUBLIC_API unsigned XXH_versionNumber (void); + + +/*-********************************************************************** +* 32-bit hash +************************************************************************/ +typedef unsigned int XXH32_hash_t; + +/*! XXH32() : + Calculate the 32-bit hash of sequence "length" bytes stored at memory address "input". + The memory between input & input+length must be valid (allocated and read-accessible). + "seed" can be used to alter the result predictably. + Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark) : 5.4 GB/s */ +XXH_PUBLIC_API XXH32_hash_t XXH32 (const void* input, size_t length, unsigned int seed); + +/*====== Streaming ======*/ +typedef struct XXH32_state_s XXH32_state_t; /* incomplete type */ +XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void); +XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr); +XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dst_state, const XXH32_state_t* src_state); + +XXH_PUBLIC_API XXH_errorcode XXH32_reset (XXH32_state_t* statePtr, unsigned int seed); +XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length); +XXH_PUBLIC_API XXH32_hash_t XXH32_digest (const XXH32_state_t* statePtr); + +/* + * Streaming functions generate the xxHash of an input provided in multiple segments. + * Note that, for small input, they are slower than single-call functions, due to state management. + * For small inputs, prefer `XXH32()` and `XXH64()`, which are better optimized. + * + * XXH state must first be allocated, using XXH*_createState() . + * + * Start a new hash by initializing state with a seed, using XXH*_reset(). + * + * Then, feed the hash state by calling XXH*_update() as many times as necessary. + * The function returns an error code, with 0 meaning OK, and any other value meaning there is an error. + * + * Finally, a hash value can be produced anytime, by using XXH*_digest(). + * This function returns the nn-bits hash as an int or long long. + * + * It's still possible to continue inserting input into the hash state after a digest, + * and generate some new hashes later on, by calling again XXH*_digest(). + * + * When done, free XXH state space if it was allocated dynamically. + */ + +/*====== Canonical representation ======*/ + +typedef struct { unsigned char digest[4]; } XXH32_canonical_t; +XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash); +XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src); + +/* Default result type for XXH functions are primitive unsigned 32 and 64 bits. + * The canonical representation uses human-readable write convention, aka big-endian (large digits first). + * These functions allow transformation of hash result into and from its canonical format. + * This way, hash values can be written into a file / memory, and remain comparable on different systems and programs. + */ + + +#ifndef XXH_NO_LONG_LONG +/*-********************************************************************** +* 64-bit hash +************************************************************************/ +typedef unsigned long long XXH64_hash_t; + +/*! XXH64() : + Calculate the 64-bit hash of sequence of length "len" stored at memory address "input". + "seed" can be used to alter the result predictably. + This function runs faster on 64-bit systems, but slower on 32-bit systems (see benchmark). +*/ +XXH_PUBLIC_API XXH64_hash_t XXH64 (const void* input, size_t length, unsigned long long seed); + +/*====== Streaming ======*/ +typedef struct XXH64_state_s XXH64_state_t; /* incomplete type */ +XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void); +XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr); +XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dst_state, const XXH64_state_t* src_state); + +XXH_PUBLIC_API XXH_errorcode XXH64_reset (XXH64_state_t* statePtr, unsigned long long seed); +XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* statePtr, const void* input, size_t length); +XXH_PUBLIC_API XXH64_hash_t XXH64_digest (const XXH64_state_t* statePtr); + +/*====== Canonical representation ======*/ +typedef struct { unsigned char digest[8]; } XXH64_canonical_t; +XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash); +XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src); + + +#endif /* XXH_NO_LONG_LONG */ + + + +#ifdef XXH_STATIC_LINKING_ONLY + +/* ================================================================================================ + This section contains declarations which are not guaranteed to remain stable. + They may change in future versions, becoming incompatible with a different version of the library. + These declarations should only be used with static linking. + Never use them in association with dynamic linking ! +=================================================================================================== */ + +/* These definitions are only present to allow + * static allocation of XXH state, on stack or in a struct for example. + * Never **ever** use members directly. */ + +#if !defined (__VMS) \ + && (defined (__cplusplus) \ + || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) +# include + +struct XXH32_state_s { + uint32_t total_len_32; + uint32_t large_len; + uint32_t v1; + uint32_t v2; + uint32_t v3; + uint32_t v4; + uint32_t mem32[4]; + uint32_t memsize; + uint32_t reserved; /* never read nor write, might be removed in a future version */ +}; /* typedef'd to XXH32_state_t */ + +struct XXH64_state_s { + uint64_t total_len; + uint64_t v1; + uint64_t v2; + uint64_t v3; + uint64_t v4; + uint64_t mem64[4]; + uint32_t memsize; + uint32_t reserved[2]; /* never read nor write, might be removed in a future version */ +}; /* typedef'd to XXH64_state_t */ + +# else + +struct XXH32_state_s { + XXH32_hash_t total_len_32; + XXH32_hash_t large_len; + XXH32_hash_t v1; + XXH32_hash_t v2; + XXH32_hash_t v3; + XXH32_hash_t v4; + XXH32_hash_t mem32[4]; + XXH32_hash_t memsize; + XXH32_hash_t reserved; /* never read nor write, might be removed in a future version */ +}; /* typedef'd to XXH32_state_t */ + +# ifndef XXH_NO_LONG_LONG /* remove 64-bit support */ +struct XXH64_state_s { + XXH64_hash_t total_len; + XXH64_hash_t v1; + XXH64_hash_t v2; + XXH64_hash_t v3; + XXH64_hash_t v4; + XXH64_hash_t mem64[4]; + XXH32_hash_t memsize; + XXH32_hash_t reserved[2]; /* never read nor write, might be removed in a future version */ +}; /* typedef'd to XXH64_state_t */ +# endif + +# endif + + +/*-********************************************************************** +* XXH3 +* New experimental hash +************************************************************************/ +#ifndef XXH_NO_LONG_LONG + + +/* ============================================ + * XXH3 is a new hash algorithm, + * featuring vastly improved speed performance + * for both small and large inputs. + * A full speed analysis will be published, + * it requires a lot more space than this comment can handle. + * In general, expect XXH3 to run about ~2x faster on large inputs, + * and >3x faster on small ones, though exact difference depend on platform. + * + * The algorithm is portable, will generate the same hash on all platforms. + * It benefits greatly from vectorization units, but does not require it. + * + * XXH3 offers 2 variants, _64bits and _128bits. + * When only 64 bits are needed, prefer calling the _64bits variant : + * it reduces the amount of mixing, resulting in faster speed on small inputs. + * It's also generally simpler to manipulate a scalar type than a struct. + * Note : the low 64-bit field of the _128bits variant is the same as _64bits result. + * + * The XXH3 algorithm is still considered experimental. + * It's possible to use it for ephemeral data, but avoid storing long-term values for later re-use. + * While labelled experimental, the produced result can still change between versions. + * + * The API currently supports one-shot hashing only. + * The full version will include streaming capability, and canonical representation + * Long term optional feature may include custom secret keys, and secret key generation. + * + * There are still a number of opened questions that community can influence during the experimental period. + * I'm trying to list a few of them below, though don't consider this list as complete. + * + * - 128-bits output type : currently defined as a structure of 2 64-bits fields. + * That's because 128-bit values do not exist in C standard. + * Note that it means that, at byte level, result is not identical depending on endianess. + * However, at field level, they are identical on all platforms. + * The canonical representation will solve the issue of identical byte-level representation across platforms, + * which is necessary for serialization. + * Would there be a better representation for a 128-bit hash result ? + * Are the names of the inner 64-bit fields important ? Should they be changed ? + * + * - Canonical representation : for the 64-bit variant, canonical representation is the same as XXH64() (aka big-endian). + * What should it be for the 128-bit variant ? + * Since it's no longer a scalar value, big-endian representation is no longer an obvious choice. + * One possibility : represent it as the concatenation of two 64-bits canonical representation (aka 2x big-endian) + * Another one : represent it in the same order as natural order in the struct for little-endian platforms. + * Less consistent with existing convention for XXH32/XXH64, but may be more natural for little-endian platforms. + * + * - Associated functions for 128-bit hash : simple things, such as checking if 2 hashes are equal, become more difficult with struct. + * Granted, it's not terribly difficult to create a comparator, but it's still a workload. + * Would it be beneficial to declare and define a comparator function for XXH128_hash_t ? + * Are there other operations on XXH128_hash_t which would be desirable ? + * + * - Variant compatibility : The low 64-bit field of the _128bits variant is the same as the result of _64bits. + * This is not a compulsory behavior. It just felt that it "wouldn't hurt", and might even help in some (unidentified) cases. + * But it might influence the design of XXH128_hash_t, in ways which may block other possibilities. + * Good idea, bad idea ? + * + * - Seed type for 128-bits variant : currently, it's a single 64-bit value, like the 64-bit variant. + * It could be argued that it's more logical to offer a 128-bit seed input parameter for a 128-bit hash. + * Although it's also more difficult to use, since it requires to declare and pass a structure instead of a value. + * It would either replace current choice, or add a new one. + * Farmhash, for example, offers both variants (the 128-bits seed variant is called `doubleSeed`). + * If both 64-bit and 128-bit seeds are possible, which variant should be called XXH128 ? + * + * - Result for len==0 : Currently, the result of hashing a zero-length input is the seed. + * This mimics the behavior of a crc : in which case, a seed is effectively an accumulator, so it's not updated if input is empty. + * Consequently, by default, when no seed specified, it returns zero. That part seems okay (it used to be a request for XXH32/XXH64). + * But is it still fine to return the seed when the seed is non-zero ? + * Are there use case which would depend on this behavior, or would prefer a mixing of the seed ? + */ + +#ifdef XXH_NAMESPACE +# define XXH128 XXH_NAME2(XXH_NAMESPACE, XXH128) +# define XXH3_64bits XXH_NAME2(XXH_NAMESPACE, XXH3_64bits) +# define XXH3_64bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSeed) +# define XXH3_128bits XXH_NAME2(XXH_NAMESPACE, XXH3_128bits) +# define XXH3_128bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSeed) +#endif + + +typedef struct { + XXH64_hash_t low64; + XXH64_hash_t high64; +} XXH128_hash_t; + +XXH_PUBLIC_API XXH128_hash_t XXH128(const void* data, size_t len, unsigned long long seed); + +/* note : variants without seed produce same result as variant with seed == 0 */ +XXH_PUBLIC_API XXH64_hash_t XXH3_64bits(const void* data, size_t len); +XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSeed(const void* data, size_t len, unsigned long long seed); +XXH_PUBLIC_API XXH128_hash_t XXH3_128bits(const void* data, size_t len); +XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSeed(const void* data, size_t len, unsigned long long seed); /* == XXH128() */ + + +#endif /* XXH_NO_LONG_LONG */ + + +/*-********************************************************************** +* XXH_INLINE_ALL +************************************************************************/ +#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API) +# include "xxhash.c" /* include xxhash function bodies as `static`, for inlining */ +#endif + + + +#endif /* XXH_STATIC_LINKING_ONLY */ + + +#if defined (__cplusplus) +} +#endif + +#endif /* XXHASH_H_5627135585666179 */ From patchwork Tue Sep 3 15:00:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11128123 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1DB7E112C for ; Tue, 3 Sep 2019 15:00:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F37E22339D for ; Tue, 3 Sep 2019 15:00:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729756AbfICPA5 (ORCPT ); Tue, 3 Sep 2019 11:00:57 -0400 Received: from mx2.suse.de ([195.135.220.15]:57510 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729706AbfICPAv (ORCPT ); Tue, 3 Sep 2019 11:00:51 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 7BBF6AF38; Tue, 3 Sep 2019 15:00:48 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm Date: Tue, 3 Sep 2019 17:00:44 +0200 Message-Id: <20190903150046.14926-11-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190903150046.14926-1-jthumshirn@suse.de> References: <20190903150046.14926-1-jthumshirn@suse.de> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: David Sterba Add xxhash64 as another checksumming algorithm. Signed-off-by: David Sterba Signed-off-by: Johannes Thumshirn Reviewed-by: Nikolay Borisov --- Changes to v3: - Fix usage of is_valid_csum_type() (Nikolay) - Remove unrelated whitespace change (Nikolay) Changes to v2: - Integrated comments from Nikolay --- Makefile | 3 ++- cmds/inspect-dump-super.c | 24 +++++++++++++++--------- convert/common.c | 2 +- convert/main.c | 2 +- crypto/hash.c | 16 ++++++++++++++++ crypto/hash.h | 10 ++++++++++ ctree.h | 14 ++++++++++---- disk-io.c | 7 +++++-- image/main.c | 5 +++-- mkfs/common.c | 14 +++++++------- mkfs/main.c | 6 +++++- 11 files changed, 75 insertions(+), 28 deletions(-) create mode 100644 crypto/hash.c create mode 100644 crypto/hash.h diff --git a/Makefile b/Makefile index 370e0c37ff65..45530749e2b9 100644 --- a/Makefile +++ b/Makefile @@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \ mkfs/common.o check/mode-common.o check/mode-lowmem.o libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \ kernel-lib/crc32c.o common/messages.o \ - uuid-tree.o utils-lib.o common/rbtree-utils.o + uuid-tree.o utils-lib.o common/rbtree-utils.o \ + crypto/hash.o crypto/xxhash.o libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \ kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \ kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \ diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c index 58bf82b0bbd3..f9f38751f429 100644 --- a/cmds/inspect-dump-super.c +++ b/cmds/inspect-dump-super.c @@ -311,6 +311,17 @@ static void print_readable_super_flag(u64 flag) super_flags_num, BTRFS_SUPER_FLAG_SUPP); } +static bool is_valid_csum_type(u16 csum_type) +{ + switch (csum_type) { + case BTRFS_CSUM_TYPE_CRC32: + case BTRFS_CSUM_TYPE_XXHASH: + return true; + default: + return false; + } +} + static void dump_superblock(struct btrfs_super_block *sb, int full) { int i; @@ -326,15 +337,11 @@ static void dump_superblock(struct btrfs_super_block *sb, int full) csum_type = btrfs_super_csum_type(sb); csum_size = BTRFS_CSUM_SIZE; printf("csum_type\t\t%hu (", csum_type); - if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) { + if (!is_valid_csum_type(csum_type)) { printf("INVALID"); } else { - if (csum_type == BTRFS_CSUM_TYPE_CRC32) { - printf("crc32c"); - csum_size = btrfs_csum_sizes[csum_type]; - } else { - printf("unknown"); - } + printf("%s", btrfs_csums[csum_type].name); + csum_size = btrfs_csums[csum_type].size; } printf(")\n"); printf("csum_size\t\t%llu\n", (unsigned long long)csum_size); @@ -342,8 +349,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full) printf("csum\t\t\t0x"); for (i = 0, p = sb->csum; i < csum_size; i++) printf("%02x", p[i]); - if (csum_type != BTRFS_CSUM_TYPE_CRC32 || - csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]) + if (!is_valid_csum_type(csum_type)) printf(" [UNKNOWN CSUM TYPE OR SIZE]"); else if (check_csum_sblock(sb, csum_size, csum_type)) printf(" [match]"); diff --git a/convert/common.c b/convert/common.c index 2e2318a5863e..5dd1a2644bf6 100644 --- a/convert/common.c +++ b/convert/common.c @@ -224,7 +224,7 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf, { int ret; - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); /* Temporary extent buffer is always mapped 1:1 on disk */ diff --git a/convert/main.c b/convert/main.c index 5e6b12431f59..5eb2a59fb68a 100644 --- a/convert/main.c +++ b/convert/main.c @@ -1058,7 +1058,7 @@ static int migrate_super_block(int fd, u64 old_bytenr) BUG_ON(btrfs_super_bytenr(super) != old_bytenr); btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET); - csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0, + csum_tree_block_size(buf, btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size, 0, btrfs_super_csum_type(super)); ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE, BTRFS_SUPER_INFO_OFFSET); diff --git a/crypto/hash.c b/crypto/hash.c new file mode 100644 index 000000000000..fda7fc4e9f23 --- /dev/null +++ b/crypto/hash.c @@ -0,0 +1,16 @@ +#include "crypto/hash.h" +#include "crypto/xxhash.h" + +int hash_xxhash(const u8 *buf, size_t length, u8 *out) +{ + XXH64_hash_t hash; + + hash = XXH64(buf, length, 0); + /* NOTE: we're not taking the canonical form here but the plain hash to + * be compatible with the kernel implementation! + */ + memcpy(out, &hash, 8); + + return 0; +} + diff --git a/crypto/hash.h b/crypto/hash.h new file mode 100644 index 000000000000..45c1ef17bc57 --- /dev/null +++ b/crypto/hash.h @@ -0,0 +1,10 @@ +#ifndef CRYPTO_HASH_H +#define CRYPTO_HASH_H + +#include "../kerncompat.h" + +#define CRYPTO_HASH_SIZE_MAX 32 + +int hash_xxhash(const u8 *buf, size_t length, u8 *out); + +#endif diff --git a/ctree.h b/ctree.h index 870d9f4948de..4ded8161d149 100644 --- a/ctree.h +++ b/ctree.h @@ -167,10 +167,16 @@ struct btrfs_free_space_ctl; /* csum types */ enum btrfs_csum_type { BTRFS_CSUM_TYPE_CRC32 = 0, + BTRFS_CSUM_TYPE_XXHASH = 1, }; -/* four bytes for CRC32 */ -static int btrfs_csum_sizes[] = { 4 }; +static struct btrfs_csum { + u16 size; + const char *name; +} btrfs_csums[] = { + [BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" }, + [BTRFS_CSUM_TYPE_XXHASH] = { 8, "xxhash64" }, +}; #define BTRFS_EMPTY_DIR_SIZE 0 @@ -2266,8 +2272,8 @@ BTRFS_SETGET_STACK_FUNCS(super_magic, struct btrfs_super_block, magic, 64); static inline int btrfs_super_csum_size(struct btrfs_super_block *s) { int t = btrfs_super_csum_type(s); - BUG_ON(t >= ARRAY_SIZE(btrfs_csum_sizes)); - return btrfs_csum_sizes[t]; + BUG_ON(t >= ARRAY_SIZE(btrfs_csums)); + return btrfs_csums[t].size; } static inline unsigned long btrfs_leaf_data(struct extent_buffer *l) diff --git a/disk-io.c b/disk-io.c index 810c2e14294a..ce0b746f4db9 100644 --- a/disk-io.c +++ b/disk-io.c @@ -34,6 +34,7 @@ #include "print-tree.h" #include "common/rbtree-utils.h" #include "common/device-scan.h" +#include "crypto/hash.h" /* specified errno for check_tree_block */ #define BTRFS_BAD_BYTENR (-1) @@ -148,6 +149,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len) crc = crc32c(crc, data, len); put_unaligned_le32(~crc, out); return 0; + case BTRFS_CSUM_TYPE_XXHASH: + return hash_xxhash(data, len, out); default: fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type); ASSERT(0); @@ -1376,11 +1379,11 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags) } csum_type = btrfs_super_csum_type(sb); - if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) { + if (csum_type >= ARRAY_SIZE(btrfs_csums)) { error("unsupported checksum algorithm %u", csum_type); return -EIO; } - csum_size = btrfs_csum_sizes[csum_type]; + csum_size = btrfs_csums[csum_type].size; btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE); diff --git a/image/main.c b/image/main.c index 0c8ffede56f5..1265152cf524 100644 --- a/image/main.c +++ b/image/main.c @@ -121,11 +121,12 @@ static struct extent_buffer *alloc_dummy_eb(u64 bytenr, u32 size); static void csum_block(u8 *buf, size_t len) { - u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]]; + u16 csum_size = btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size; + u8 result[csum_size]; u32 crc = ~(u32)0; crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE); put_unaligned_le32(~crc, result); - memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]); + memcpy(buf, result, csum_size); } static int has_name(struct btrfs_key *key) diff --git a/mkfs/common.c b/mkfs/common.c index 4a417bd7a306..939be5eb2dc2 100644 --- a/mkfs/common.c +++ b/mkfs/common.c @@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg, } /* generate checksum */ - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); /* write back root tree */ @@ -293,7 +293,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]); btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]); if (ret != cfg->nodesize) { @@ -382,7 +382,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]); btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]); if (ret != cfg->nodesize) { @@ -423,7 +423,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]); btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID); btrfs_set_header_nritems(buf, nritems); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]); if (ret != cfg->nodesize) { @@ -437,7 +437,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]); btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID); btrfs_set_header_nritems(buf, 0); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]); if (ret != cfg->nodesize) { @@ -450,7 +450,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]); btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID); btrfs_set_header_nritems(buf, 0); - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]); if (ret != cfg->nodesize) { @@ -462,7 +462,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg) memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE); memcpy(buf->data, &super, sizeof(super)); buf->len = BTRFS_SUPER_INFO_SIZE; - csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0, + csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0, cfg->csum_type); ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE, cfg->blocks[MKFS_SUPER_BLOCK]); diff --git a/mkfs/main.c b/mkfs/main.c index e96cbc5399a2..64806dac7706 100644 --- a/mkfs/main.c +++ b/mkfs/main.c @@ -391,6 +391,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s) { if (strcasecmp(s, "crc32c") == 0) { return BTRFS_CSUM_TYPE_CRC32; + } else if (strcasecmp(s, "xxhash64") == 0 || + strcasecmp(s, "xxhash") == 0) { + return BTRFS_CSUM_TYPE_XXHASH; } else { error("unknown csum type %s", s); exit(1); @@ -1376,7 +1379,8 @@ raid_groups: pretty_size(allocation.system)); printf("SSD detected: %s\n", ssd ? "yes" : "no"); btrfs_parse_features_to_string(features_buf, features); - printf("Incompat features: %s", features_buf); + printf("Incompat features: %s\n", features_buf); + printf("Checksum: %s", btrfs_csums[csum_type].name); printf("\n"); list_all_devices(root); From patchwork Tue Sep 3 15:00:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11128125 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 081E116B1 for ; Tue, 3 Sep 2019 15:01:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D5A9222D6D for ; Tue, 3 Sep 2019 15:01:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729748AbfICPA4 (ORCPT ); Tue, 3 Sep 2019 11:00:56 -0400 Received: from mx2.suse.de ([195.135.220.15]:57512 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729705AbfICPAw (ORCPT ); Tue, 3 Sep 2019 11:00:52 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 8FAB6AF54; Tue, 3 Sep 2019 15:00:48 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [PATCH v4 11/12] btrfs-progs: move crc32c implementation to crypto/ Date: Tue, 3 Sep 2019 17:00:45 +0200 Message-Id: <20190903150046.14926-12-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190903150046.14926-1-jthumshirn@suse.de> References: <20190903150046.14926-1-jthumshirn@suse.de> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org With the introduction of xxhash64 to btrfs-progs we created a crypto/ directory for all the hashes used in btrfs (although no cryptographically secure hash is there yet). Move the crc32c implementation from kernel-lib/ to crypto/ as well so we have all hashes consolidated. Signed-off-by: Johannes Thumshirn Reviewed-by: Nikolay Borisov --- Android.mk | 4 ++-- Makefile | 4 ++-- btrfs-crc.c | 2 +- btrfs-find-root.c | 2 +- btrfs-sb-mod.c | 2 +- btrfs.c | 2 +- cmds/inspect-dump-super.c | 2 +- cmds/rescue-chunk-recover.c | 2 +- cmds/rescue-super-recover.c | 2 +- common/utils.c | 2 +- convert/main.c | 2 +- {kernel-lib => crypto}/crc32c.c | 2 +- {kernel-lib => crypto}/crc32c.h | 0 disk-io.c | 2 +- extent-tree.c | 2 +- file-item.c | 2 +- free-space-cache.c | 2 +- hash.h | 2 +- image/main.c | 2 +- image/sanitize.c | 2 +- library-test.c | 2 +- mkfs/main.c | 2 +- send-stream.c | 2 +- 23 files changed, 24 insertions(+), 24 deletions(-) rename {kernel-lib => crypto}/crc32c.c (99%) rename {kernel-lib => crypto}/crc32c.h (100%) diff --git a/Android.mk b/Android.mk index e8de47eb4617..8288ba7356f4 100644 --- a/Android.mk +++ b/Android.mk @@ -32,10 +32,10 @@ cmds_objects := cmds-subvolume.c cmds-filesystem.c cmds-device.c cmds-scrub.c \ cmds-inspect-dump-super.c cmds-inspect-tree-stats.c cmds-fi-du.c \ mkfs/common.c libbtrfs_objects := send-stream.c send-utils.c kernel-lib/rbtree.c btrfs-list.c \ - kernel-lib/crc32c.c messages.c \ + crypto/crc32c.c messages.c \ uuid-tree.c utils-lib.c rbtree-utils.c libbtrfs_headers := send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \ - kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \ + crypto/crc32c.h kernel-lib/list.h kerncompat.h \ kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \ extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h blkid_objects := partition/ superblocks/ topology/ diff --git a/Makefile b/Makefile index 45530749e2b9..c241c22d1018 100644 --- a/Makefile +++ b/Makefile @@ -150,11 +150,11 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \ cmds/inspect-dump-super.o cmds/inspect-tree-stats.o cmds/filesystem-du.o \ mkfs/common.o check/mode-common.o check/mode-lowmem.o libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \ - kernel-lib/crc32c.o common/messages.o \ + crypto/crc32c.o common/messages.o \ uuid-tree.o utils-lib.o common/rbtree-utils.o \ crypto/hash.o crypto/xxhash.o libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \ - kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \ + crypto/crc32c.h kernel-lib/list.h kerncompat.h \ kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \ extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h libbtrfsutil_major := $(shell sed -rn 's/^\#define BTRFS_UTIL_VERSION_MAJOR ([0-9])+$$/\1/p' libbtrfsutil/btrfsutil.h) diff --git a/btrfs-crc.c b/btrfs-crc.c index bcf25df8b46a..c4f81fc65f67 100644 --- a/btrfs-crc.c +++ b/btrfs-crc.c @@ -19,7 +19,7 @@ #include #include #include -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "common/utils.h" void print_usage(int status) diff --git a/btrfs-find-root.c b/btrfs-find-root.c index e46fa8723b33..741eb9a95ac5 100644 --- a/btrfs-find-root.c +++ b/btrfs-find-root.c @@ -32,7 +32,7 @@ #include "kernel-lib/list.h" #include "volumes.h" #include "common/utils.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "extent-cache.h" #include "find-root.h" #include "common/help.h" diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c index 105b556b0cf1..348991b39451 100644 --- a/btrfs-sb-mod.c +++ b/btrfs-sb-mod.c @@ -24,7 +24,7 @@ #include #include #include -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "disk-io.h" #define BLOCKSIZE (4096) diff --git a/btrfs.c b/btrfs.c index 6c8aabe24dc8..72dad6fb3983 100644 --- a/btrfs.c +++ b/btrfs.c @@ -20,7 +20,7 @@ #include #include "volumes.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "cmds/commands.h" #include "common/utils.h" #include "common/help.h" diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c index f9f38751f429..ef3d9094e661 100644 --- a/cmds/inspect-dump-super.c +++ b/cmds/inspect-dump-super.c @@ -32,7 +32,7 @@ #include "kernel-lib/list.h" #include "common/utils.h" #include "cmds/commands.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "common/help.h" static int check_csum_sblock(void *sb, int csum_size, u16 csum_type) diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c index 329a608dfc6b..bf35693ddbfa 100644 --- a/cmds/rescue-chunk-recover.c +++ b/cmds/rescue-chunk-recover.c @@ -36,7 +36,7 @@ #include "disk-io.h" #include "volumes.h" #include "transaction.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "common/utils.h" #include "check/common.h" #include "cmds/commands.h" diff --git a/cmds/rescue-super-recover.c b/cmds/rescue-super-recover.c index ea3a00caf56c..5d6bea836c8b 100644 --- a/cmds/rescue-super-recover.c +++ b/cmds/rescue-super-recover.c @@ -31,7 +31,7 @@ #include "disk-io.h" #include "kernel-lib/list.h" #include "common/utils.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "volumes.h" #include "cmds/commands.h" #include "cmds/rescue.h" diff --git a/common/utils.c b/common/utils.c index f2a10cccca86..fa49c01ad102 100644 --- a/common/utils.c +++ b/common/utils.c @@ -47,7 +47,7 @@ #include "ctree.h" #include "disk-io.h" #include "transaction.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "common/utils.h" #include "common/path-utils.h" #include "common/device-scan.h" diff --git a/convert/main.c b/convert/main.c index 5eb2a59fb68a..fdbddc846e39 100644 --- a/convert/main.c +++ b/convert/main.c @@ -102,7 +102,7 @@ #include "mkfs/common.h" #include "convert/common.h" #include "convert/source-fs.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "common/fsfeatures.h" #include "common/box.h" diff --git a/kernel-lib/crc32c.c b/crypto/crc32c.c similarity index 99% rename from kernel-lib/crc32c.c rename to crypto/crc32c.c index 36bb6f189971..bd6283d5baeb 100644 --- a/kernel-lib/crc32c.c +++ b/crypto/crc32c.c @@ -8,7 +8,7 @@ * */ #include "kerncompat.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include #include #include diff --git a/kernel-lib/crc32c.h b/crypto/crc32c.h similarity index 100% rename from kernel-lib/crc32c.h rename to crypto/crc32c.h diff --git a/disk-io.c b/disk-io.c index ce0b746f4db9..4093982cf3dc 100644 --- a/disk-io.c +++ b/disk-io.c @@ -29,7 +29,7 @@ #include "disk-io.h" #include "volumes.h" #include "transaction.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "common/utils.h" #include "print-tree.h" #include "common/rbtree-utils.h" diff --git a/extent-tree.c b/extent-tree.c index 932af2c644bd..a8f57776bd73 100644 --- a/extent-tree.c +++ b/extent-tree.c @@ -26,7 +26,7 @@ #include "disk-io.h" #include "print-tree.h" #include "transaction.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "volumes.h" #include "free-space-cache.h" #include "free-space-tree.h" diff --git a/file-item.c b/file-item.c index c6e9d212bcab..64af57693baf 100644 --- a/file-item.c +++ b/file-item.c @@ -24,7 +24,7 @@ #include "disk-io.h" #include "transaction.h" #include "print-tree.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "common/internal.h" #define MAX_CSUM_ITEMS(r, size) ((((BTRFS_LEAF_DATA_SIZE(r->fs_info) - \ diff --git a/free-space-cache.c b/free-space-cache.c index 8a57f86dc650..6e7d7e1ef561 100644 --- a/free-space-cache.c +++ b/free-space-cache.c @@ -22,7 +22,7 @@ #include "transaction.h" #include "disk-io.h" #include "extent_io.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "kernel-lib/bitops.h" #include "common/internal.h" #include "common/utils.h" diff --git a/hash.h b/hash.h index 5398e8869316..51e842047093 100644 --- a/hash.h +++ b/hash.h @@ -19,7 +19,7 @@ #ifndef __BTRFS_HASH_H__ #define __BTRFS_HASH_H__ -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" static inline u64 btrfs_name_hash(const char *name, int len) { diff --git a/image/main.c b/image/main.c index 1265152cf524..92ee28d7fe1a 100644 --- a/image/main.c +++ b/image/main.c @@ -28,7 +28,7 @@ #include #include "kerncompat.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "ctree.h" #include "disk-io.h" #include "transaction.h" diff --git a/image/sanitize.c b/image/sanitize.c index 9caa5deaf2dd..cd2bd6fe2379 100644 --- a/image/sanitize.c +++ b/image/sanitize.c @@ -18,7 +18,7 @@ #include "common/internal.h" #include "common/messages.h" #include "common/utils.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "image/sanitize.h" #include "extent_io.h" diff --git a/library-test.c b/library-test.c index c44bad228e50..e47917c25830 100644 --- a/library-test.c +++ b/library-test.c @@ -21,7 +21,7 @@ #include "version.h" #include "kernel-lib/rbtree.h" #include "kernel-lib/radix-tree.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "kernel-lib/list.h" #include "kernel-lib/sizes.h" #include "ctree.h" diff --git a/mkfs/main.c b/mkfs/main.c index 64806dac7706..a46205e7237f 100644 --- a/mkfs/main.c +++ b/mkfs/main.c @@ -45,7 +45,7 @@ #include "common/rbtree-utils.h" #include "mkfs/common.h" #include "mkfs/rootdir.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "common/fsfeatures.h" #include "common/box.h" diff --git a/send-stream.c b/send-stream.c index 08687e5e6904..3b8ada2110aa 100644 --- a/send-stream.c +++ b/send-stream.c @@ -21,7 +21,7 @@ #include "send.h" #include "send-stream.h" -#include "kernel-lib/crc32c.h" +#include "crypto/crc32c.h" #include "common/utils.h" struct btrfs_send_stream { From patchwork Tue Sep 3 15:00:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 11128129 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 340ED112C for ; Tue, 3 Sep 2019 15:01:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2115B2339D for ; Tue, 3 Sep 2019 15:01:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729771AbfICPBE (ORCPT ); Tue, 3 Sep 2019 11:01:04 -0400 Received: from mx2.suse.de ([195.135.220.15]:57514 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729702AbfICPAv (ORCPT ); Tue, 3 Sep 2019 11:00:51 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 90D89AF79; Tue, 3 Sep 2019 15:00:48 +0000 (UTC) From: Johannes Thumshirn To: David Sterba Cc: Linux BTRFS Mailinglist , Johannes Thumshirn Subject: [PATCH v4 12/12] btrfs-progs: add test-case for mkfs with xxhash64 Date: Tue, 3 Sep 2019 17:00:46 +0200 Message-Id: <20190903150046.14926-13-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190903150046.14926-1-jthumshirn@suse.de> References: <20190903150046.14926-1-jthumshirn@suse.de> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Add test-cases for creating a file-system xxhash64 as checksumming algorithm. Signed-off-by: Johannes Thumshirn Reviewed-by: Nikolay Borisov --- tests/mkfs-tests/001-basic-profiles/test.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/mkfs-tests/001-basic-profiles/test.sh b/tests/mkfs-tests/001-basic-profiles/test.sh index 6e295274119d..3fa3c8ad42d1 100755 --- a/tests/mkfs-tests/001-basic-profiles/test.sh +++ b/tests/mkfs-tests/001-basic-profiles/test.sh @@ -46,6 +46,8 @@ test_mkfs_single -d single -m dup test_mkfs_single -d dup -m single test_mkfs_single -d dup -m dup test_mkfs_single -d dup -m dup --mixed +test_mkfs_single -C xxhash64 +test_mkfs_single -C xxhash test_mkfs_multi test_mkfs_multi -d single -m single