From patchwork Fri Nov 4 01:32:52 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 9411721 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 348A7601C2 for ; Fri, 4 Nov 2016 01:33:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 25E172B027 for ; Fri, 4 Nov 2016 01:33:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1716E2B030; Fri, 4 Nov 2016 01:33:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 32D7B2B035 for ; Fri, 4 Nov 2016 01:33:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933425AbcKDBdp (ORCPT ); Thu, 3 Nov 2016 21:33:45 -0400 Received: from cn.fujitsu.com ([222.73.24.84]:17959 "EHLO song.cn.fujitsu.com" rhost-flags-OK-FAIL-OK-OK) by vger.kernel.org with ESMTP id S933204AbcKDBdo (ORCPT ); Thu, 3 Nov 2016 21:33:44 -0400 X-IronPort-AV: E=Sophos;i="5.20,367,1444665600"; d="scan'208";a="943827" Received: from unknown (HELO cn.fujitsu.com) ([10.167.250.3]) by song.cn.fujitsu.com with ESMTP; 04 Nov 2016 09:33:15 +0800 Received: from localhost.localdomain (unknown [10.167.226.34]) by cn.fujitsu.com (Postfix) with ESMTP id 06AAB41B388A; Fri, 4 Nov 2016 09:33:15 +0800 (CST) From: Qu Wenruo To: linux-btrfs@vger.kernel.org Cc: Wang Xiaoguang Subject: [PATCH v14 03/15] btrfs: dedupe: Introduce dedupe framework and its header Date: Fri, 4 Nov 2016 09:32:52 +0800 Message-Id: <20161104013304.11365-4-quwenruo@cn.fujitsu.com> X-Mailer: git-send-email 2.10.1 In-Reply-To: <20161104013304.11365-1-quwenruo@cn.fujitsu.com> References: <20161104013304.11365-1-quwenruo@cn.fujitsu.com> MIME-Version: 1.0 X-yoursite-MailScanner-ID: 06AAB41B388A.AED86 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: quwenruo@cn.fujitsu.com Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Wang Xiaoguang Introduce the header for btrfs in-band(write time) de-duplication framework and needed header. The new de-duplication framework is going to support 2 different dedupe methods and 1 dedupe hash. Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang --- fs/btrfs/ctree.h | 7 +++ fs/btrfs/dedupe.h | 137 ++++++++++++++++++++++++++++++++++++++++++++- fs/btrfs/disk-io.c | 1 + include/uapi/linux/btrfs.h | 34 +++++++++++ 4 files changed, 177 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 3ec0d16..0564de6 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -1097,6 +1097,13 @@ struct btrfs_fs_info { /* Used to record internally whether fs has been frozen */ int fs_frozen; + + /* + * Inband de-duplication related structures + */ + unsigned long dedupe_enabled:1; + struct btrfs_dedupe_info *dedupe_info; + struct mutex dedupe_ioctl_lock; }; struct btrfs_subvolume_writers { diff --git a/fs/btrfs/dedupe.h b/fs/btrfs/dedupe.h index 83ebfe2..5ecc321 100644 --- a/fs/btrfs/dedupe.h +++ b/fs/btrfs/dedupe.h @@ -19,6 +19,139 @@ #ifndef __BTRFS_DEDUPE__ #define __BTRFS_DEDUPE__ -/* later in-band dedupe will expand this struct */ -struct btrfs_dedupe_hash; +#include +#include +#include + +static const int btrfs_hash_sizes[] = { 32 }; + +/* + * For caller outside of dedupe.c + * + * Different dedupe backends should have their own hash structure + */ +struct btrfs_dedupe_hash { + u64 bytenr; + u32 num_bytes; + + /* last field is a variable length array of dedupe hash */ + u8 hash[]; +}; + +struct btrfs_dedupe_info { + /* dedupe blocksize */ + u64 blocksize; + u16 backend; + u16 hash_algo; + + struct crypto_shash *dedupe_driver; + + /* + * Use mutex to portect both backends + * Even for in-memory backends, the rb-tree can be quite large, + * so mutex is better for such use case. + */ + struct mutex lock; + + /* following members are only used in in-memory backend */ + struct rb_root hash_root; + struct rb_root bytenr_root; + struct list_head lru_list; + u64 limit_nr; + u64 current_nr; +}; + +struct btrfs_trans_handle; + +static inline int btrfs_dedupe_hash_hit(struct btrfs_dedupe_hash *hash) +{ + return (hash && hash->bytenr); +} + +int btrfs_dedupe_hash_size(u16 algo); +struct btrfs_dedupe_hash *btrfs_dedupe_alloc_hash(u16 algo); + +/* + * Initial inband dedupe info + * Called at dedupe enable time. + * + * Return 0 for success + * Return <0 for any error + * (from unsupported param to tree creation error for some backends) + */ +int btrfs_dedupe_enable(struct btrfs_fs_info *fs_info, + struct btrfs_ioctl_dedupe_args *dargs); + +/* + * Disable dedupe and invalidate all its dedupe data. + * Called at dedupe disable time. + * + * Return 0 for success + * Return <0 for any error + * (tree operation error for some backends) + */ +int btrfs_dedupe_disable(struct btrfs_fs_info *fs_info); + +/* + * Get current dedupe status. + * Return 0 for success + * No possible error yet + */ +void btrfs_dedupe_status(struct btrfs_fs_info *fs_info, + struct btrfs_ioctl_dedupe_args *dargs); + +/* + * Calculate hash for dedupe. + * Caller must ensure [start, start + dedupe_bs) has valid data. + * + * Return 0 for success + * Return <0 for any error + * (error from hash codes) + */ +int btrfs_dedupe_calc_hash(struct btrfs_fs_info *fs_info, + struct inode *inode, u64 start, + struct btrfs_dedupe_hash *hash); + +/* + * Search for duplicated extents by calculated hash + * Caller must call btrfs_dedupe_calc_hash() first to get the hash. + * + * @inode: the inode for we are writing + * @file_pos: offset inside the inode + * As we will increase extent ref immediately after a hash match, + * we need @file_pos and @inode in this case. + * + * Return > 0 for a hash match, and the extent ref will be + * *INCREASED*, and hash->bytenr/num_bytes will record the existing + * extent data. + * Return 0 for a hash miss. Nothing is done + * Return <0 for any error + * (tree operation error for some backends) + */ +int btrfs_dedupe_search(struct btrfs_fs_info *fs_info, + struct inode *inode, u64 file_pos, + struct btrfs_dedupe_hash *hash); + +/* + * Add a dedupe hash into dedupe info + * Return 0 for success + * Return <0 for any error + * (tree operation error for some backends) + */ +int btrfs_dedupe_add(struct btrfs_trans_handle *trans, + struct btrfs_fs_info *fs_info, + struct btrfs_dedupe_hash *hash); + +/* + * Remove a dedupe hash from dedupe info + * Return 0 for success + * Return <0 for any error + * (tree operation error for some backends) + * + * NOTE: if hash deletion error is not handled well, it will lead + * to corrupted fs, as later dedupe write can points to non-exist or even + * wrong extent. + */ +int btrfs_dedupe_del(struct btrfs_trans_handle *trans, + struct btrfs_fs_info *fs_info, u64 bytenr); #endif diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 3a57f99..48511a4 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -2659,6 +2659,7 @@ int open_ctree(struct super_block *sb, mutex_init(&fs_info->reloc_mutex); mutex_init(&fs_info->delalloc_root_mutex); mutex_init(&fs_info->cleaner_delayed_iput_mutex); + mutex_init(&fs_info->dedupe_ioctl_lock); seqlock_init(&fs_info->profiles_lock); INIT_LIST_HEAD(&fs_info->dirty_cowonly_roots); diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h index db4c253..db1596c 100644 --- a/include/uapi/linux/btrfs.h +++ b/include/uapi/linux/btrfs.h @@ -629,6 +629,40 @@ struct btrfs_ioctl_get_dev_stats { __u64 unused[128 - 2 - BTRFS_DEV_STAT_VALUES_MAX]; /* pad to 1k */ }; +/* In-band dedupe related */ +#define BTRFS_DEDUPE_BACKEND_INMEMORY 0 +#define BTRFS_DEDUPE_BACKEND_ONDISK 1 + +/* Only support inmemory yet, so count is still only 1 */ +#define BTRFS_DEDUPE_BACKEND_COUNT 1 + +/* Dedup block size limit and default value */ +#define BTRFS_DEDUPE_BLOCKSIZE_MAX (8 * 1024 * 1024) +#define BTRFS_DEDUPE_BLOCKSIZE_MIN (16 * 1024) +#define BTRFS_DEDUPE_BLOCKSIZE_DEFAULT (128 * 1024) + +/* Hash algorithm, only support SHA256 yet */ +#define BTRFS_DEDUPE_HASH_SHA256 0 + +/* + * This structure is used for dedupe enable/disable/configure + * and status ioctl. + * Reserved range should be set to 0xff. + */ +struct btrfs_ioctl_dedupe_args { + __u16 cmd; /* In: command */ + __u64 blocksize; /* In/Out: blocksize */ + __u64 limit_nr; /* In/Out: limit nr for inmem backend */ + __u64 limit_mem; /* In/Out: limit mem for inmem backend */ + __u64 current_nr; /* Out: current hash nr */ + __u16 backend; /* In/Out: current backend */ + __u16 hash_algo; /* In/Out: hash algorithm */ + u8 status; /* Out: enabled or disabled */ + u8 flags; /* In: special flags for ioctl */ + u8 __unused[472]; /* Pad to 512 bytes */ +}; + + #define BTRFS_QUOTA_CTL_ENABLE 1 #define BTRFS_QUOTA_CTL_DISABLE 2 #define BTRFS_QUOTA_CTL_RESCAN__NOTUSED 3