From patchwork Fri Apr 1 06:34:52 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 8720411 Return-Path: X-Original-To: patchwork-linux-btrfs@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 6911FC0554 for ; Fri, 1 Apr 2016 06:35:37 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 4DC93203AA for ; Fri, 1 Apr 2016 06:35:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2E3D0203AD for ; Fri, 1 Apr 2016 06:35:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758241AbcDAGfd (ORCPT ); Fri, 1 Apr 2016 02:35:33 -0400 Received: from cn.fujitsu.com ([222.73.24.84]:62526 "EHLO song.cn.fujitsu.com" rhost-flags-OK-FAIL-OK-OK) by vger.kernel.org with ESMTP id S1758006AbcDAGfc (ORCPT ); Fri, 1 Apr 2016 02:35:32 -0400 X-IronPort-AV: E=Sophos;i="5.20,367,1444665600"; d="scan'208";a="419975" Received: from unknown (HELO cn.fujitsu.com) ([10.167.250.3]) by song.cn.fujitsu.com with ESMTP; 01 Apr 2016 14:35:16 +0800 Received: from localhost.localdomain (unknown [10.167.226.34]) by cn.fujitsu.com (Postfix) with ESMTP id 0FE2E4056403; Fri, 1 Apr 2016 14:35:13 +0800 (CST) From: Qu Wenruo To: linux-btrfs@vger.kernel.org Cc: Wang Xiaoguang Subject: [PATCH v10 01/21] btrfs: dedupe: Introduce dedupe framework and its header Date: Fri, 1 Apr 2016 14:34:52 +0800 Message-Id: <1459492512-31435-2-git-send-email-quwenruo@cn.fujitsu.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1459492512-31435-1-git-send-email-quwenruo@cn.fujitsu.com> References: <1459492512-31435-1-git-send-email-quwenruo@cn.fujitsu.com> MIME-Version: 1.0 X-yoursite-MailScanner-ID: 0FE2E4056403.AD0F2 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: quwenruo@cn.fujitsu.com X-Spam-Status: No, score=-7.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Wang Xiaoguang Introduce the header for btrfs online(write time) de-duplication framework and needed header. The new de-duplication framework is going to support 2 different dedupe methods and 1 dedupe hash. Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang --- fs/btrfs/ctree.h | 5 ++ fs/btrfs/dedupe.h | 134 +++++++++++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/disk-io.c | 1 + 3 files changed, 140 insertions(+) create mode 100644 fs/btrfs/dedupe.h diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 84a6a5b..022ab61 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -1860,6 +1860,11 @@ struct btrfs_fs_info { struct list_head pinned_chunks; int creating_free_space_tree; + + /* Inband de-duplication related structures*/ + unsigned int dedupe_enabled:1; + struct btrfs_dedupe_info *dedupe_info; + struct mutex dedupe_ioctl_lock; }; struct btrfs_subvolume_writers { diff --git a/fs/btrfs/dedupe.h b/fs/btrfs/dedupe.h new file mode 100644 index 0000000..40f4808 --- /dev/null +++ b/fs/btrfs/dedupe.h @@ -0,0 +1,134 @@ +/* + * Copyright (C) 2015 Fujitsu. All rights reserved. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public + * License v2 as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + * + * You should have received a copy of the GNU General Public + * License along with this program; if not, write to the + * Free Software Foundation, Inc., 59 Temple Place - Suite 330, + * Boston, MA 021110-1307, USA. + */ + +#ifndef __BTRFS_DEDUPE__ +#define __BTRFS_DEDUPE__ + +#include +#include +#include + +/* + * Dedup storage backend + * On disk is persist storage but overhead is large + * In memory is fast but will lose all its hash on umount + */ +#define BTRFS_DEDUPE_BACKEND_INMEMORY 0 +#define BTRFS_DEDUPE_BACKEND_ONDISK 1 + +/* Only support inmemory yet, so count is still only 1 */ +#define BTRFS_DEDUPE_BACKEND_COUNT 1 + +/* Dedup block size limit and default value */ +#define BTRFS_DEDUPE_BLOCKSIZE_MAX (8 * 1024 * 1024) +#define BTRFS_DEDUPE_BLOCKSIZE_MIN (16 * 1024) +#define BTRFS_DEDUPE_BLOCKSIZE_DEFAULT (128 * 1024) + +/* Hash algorithm, only support SHA256 yet */ +#define BTRFS_DEDUPE_HASH_SHA256 0 + +static int btrfs_dedupe_sizes[] = { 32 }; + +/* + * For caller outside of dedup.c + * + * Different dedupe backends should have their own hash structure + */ +struct btrfs_dedupe_hash { + u64 bytenr; + u32 num_bytes; + + /* last field is a variable length array of dedupe hash */ + u8 hash[]; +}; + +struct btrfs_dedupe_info { + /* dedupe blocksize */ + u64 blocksize; + u16 backend; + u16 hash_type; + + struct crypto_shash *dedupe_driver; + struct mutex lock; + + /* following members are only used in in-memory dedupe mode */ + struct rb_root hash_root; + struct rb_root bytenr_root; + struct list_head lru_list; + u64 limit_nr; + u64 current_nr; +}; + +struct btrfs_trans_handle; + +static inline int btrfs_dedupe_hash_hit(struct btrfs_dedupe_hash *hash) +{ + return (hash && hash->bytenr); +} + +int btrfs_dedupe_hash_size(u16 type); +struct btrfs_dedupe_hash *btrfs_dedupe_alloc_hash(u16 type); + +/* + * Initial inband dedupe info + * Called at dedupe enable time. + */ +int btrfs_dedupe_enable(struct btrfs_fs_info *fs_info, u16 type, u16 backend, + u64 blocksize, u64 limit_nr, u64 limit_mem); + +/* + * Disable dedupe and invalidate all its dedupe data. + * Called at dedupe disable time. + */ +int btrfs_dedupe_disable(struct btrfs_fs_info *fs_info); + +/* + * Calculate hash for dedup. + * Caller must ensure [start, start + dedupe_bs) has valid data. + */ +int btrfs_dedupe_calc_hash(struct btrfs_fs_info *fs_info, + struct inode *inode, u64 start, + struct btrfs_dedupe_hash *hash); + +/* + * Search for duplicated extents by calculated hash + * Caller must call btrfs_dedupe_calc_hash() first to get the hash. + * + * @inode: the inode for we are writing + * @file_pos: offset inside the inode + * As we will increase extent ref immediately after a hash match, + * we need @file_pos and @inode in this case. + * + * Return > 0 for a hash match, and the extent ref will be + * *INCREASED*, and hash->bytenr/num_bytes will record the existing + * extent data. + * Return 0 for a hash miss. Nothing is done + */ +int btrfs_dedupe_search(struct btrfs_fs_info *fs_info, + struct inode *inode, u64 file_pos, + struct btrfs_dedupe_hash *hash); + +/* Add a dedupe hash into dedupe info */ +int btrfs_dedupe_add(struct btrfs_trans_handle *trans, + struct btrfs_fs_info *fs_info, + struct btrfs_dedupe_hash *hash); + +/* Remove a dedupe hash from dedupe info */ +int btrfs_dedupe_del(struct btrfs_trans_handle *trans, + struct btrfs_fs_info *fs_info, u64 bytenr); +#endif diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index c95e3ce..3cf4c11 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -2584,6 +2584,7 @@ int open_ctree(struct super_block *sb, mutex_init(&fs_info->reloc_mutex); mutex_init(&fs_info->delalloc_root_mutex); mutex_init(&fs_info->cleaner_delayed_iput_mutex); + mutex_init(&fs_info->dedupe_ioctl_lock); seqlock_init(&fs_info->profiles_lock); INIT_LIST_HEAD(&fs_info->dirty_cowonly_roots);