From patchwork Fri Aug 21 07:39:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728461 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CFA561392 for ; Fri, 21 Aug 2020 07:40:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B338D207DF for ; Fri, 21 Aug 2020 07:40:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="vGjTBteg" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728270AbgHUHk3 (ORCPT ); Fri, 21 Aug 2020 03:40:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37186 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727935AbgHUHkR (ORCPT ); Fri, 21 Aug 2020 03:40:17 -0400 Received: from mail-pl1-x641.google.com (mail-pl1-x641.google.com [IPv6:2607:f8b0:4864:20::641]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43180C061385 for ; Fri, 21 Aug 2020 00:40:17 -0700 (PDT) Received: by mail-pl1-x641.google.com with SMTP id s14so519164plp.4 for ; Fri, 21 Aug 2020 00:40:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5SqT5HAFYRljt1AMWeBXhRdOtC1aJb3/tKpBWuypvUM=; b=vGjTBtegLouDINSX89/DUbl0rXrvGMosTXgf3yh/KvRO+XaYrs13qKTStO39lW2D1v e71FCzZqXXyTo5b5NdWFb+1SNsyZZ8ethiDK9sz1r9bj1nmuPBGtQ7q4JBohtBJNrQOQ sOwHxaI+rpxGbY/xTgxsHcDplJcvrf7KjcyapKfQRCu53rB60SgmSX03EzpYXweEtL5S KuruJfdEq/nKjJKKGIBvnwjYCcVvjlsp7rKcVwVCdbcFRDviA2Xk5/vZpD3V/ERoqMOx MbLlkRF9GoKP+ODevyrSHp0wqnZvkwbyK1QntGRoRyel+JVMu0bic9qDMBpTd61a9mYE 3JhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5SqT5HAFYRljt1AMWeBXhRdOtC1aJb3/tKpBWuypvUM=; b=qipSiKbiDlElj1piNNwfvzE38KTa4kuUSXqnPC2nMVwlT/XLh1ONLr97LPZPUT6kgK 2zo4nEDj7Wpla+qU2R0/b1gCf6fewXvQLyd3Ly/iQkaBnX/uBiOaI5DOBebg2M8V7FQD LXSgJMD8Uj1/HxQDq320Ublgy0TPnZqXLpz4xRzBKkvj6Biu1yTN5Ek7WvmQfaCX8aHJ tvaYrz2drEP3q8oVGPgU81fn1RrHvn2wKxeT9BTlu60NJniPwHOWKrNkaSVbDxlcosFh pUHmOyR1JRQ/VfdbNyrOTtbjYdRE+t6XrxawqI7cA6IMdHXlzar51Plcn6wJfdHBQ7ee 4Q5A== X-Gm-Message-State: AOAM530l5sLWwrv4i2LOOeDs3RdsKl3xCEU1PSs3M3CCk/BrpdvJoCQH MwVHNwZi49T4bOBJdf4sfuAu9w== X-Google-Smtp-Source: ABdhPJxN9ZmKpfnmcT+flno4G92mVF8eJPuMYv1H6pNBYN2fp+LMtlORB8iGhREqfVBYgy4HaRl81Q== X-Received: by 2002:a17:902:b60d:: with SMTP id b13mr1467416pls.48.1597995616710; Fri, 21 Aug 2020 00:40:16 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:14 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 1/9] btrfs: send: get rid of i_size logic in send_write() Date: Fri, 21 Aug 2020 00:39:51 -0700 Message-Id: <16f28691ff66e8aeb280532cc146c8ee49d6cda4.1597994106.git.osandov@osandov.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Omar Sandoval send_write()/fill_read_buf() have some logic for avoiding reading past i_size. However, everywhere that we call send_write()/send_extent_data(), we've already clamped the length down to i_size. Get rid of the i_size handling, which simplifies the next change. Signed-off-by: Omar Sandoval Reviewed-by: Filipe Manana Reviewed-by: Josef Bacik --- fs/btrfs/send.c | 37 ++++++++++--------------------------- 1 file changed, 10 insertions(+), 27 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 7c7c09fc65e8..8af5e867e4ca 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -4794,7 +4794,7 @@ static int process_all_new_xattrs(struct send_ctx *sctx) return ret; } -static ssize_t fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len) +static int fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len) { struct btrfs_root *root = sctx->send_root; struct btrfs_fs_info *fs_info = root->fs_info; @@ -4804,21 +4804,13 @@ static ssize_t fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len) pgoff_t index = offset >> PAGE_SHIFT; pgoff_t last_index; unsigned pg_offset = offset_in_page(offset); - ssize_t ret = 0; + int ret = 0; + size_t read = 0; inode = btrfs_iget(fs_info->sb, sctx->cur_ino, root); if (IS_ERR(inode)) return PTR_ERR(inode); - if (offset + len > i_size_read(inode)) { - if (offset > i_size_read(inode)) - len = 0; - else - len = offset - i_size_read(inode); - } - if (len == 0) - goto out; - last_index = (offset + len - 1) >> PAGE_SHIFT; /* initial readahead */ @@ -4859,16 +4851,15 @@ static ssize_t fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len) } addr = kmap(page); - memcpy(sctx->read_buf + ret, addr + pg_offset, cur_len); + memcpy(sctx->read_buf + read, addr + pg_offset, cur_len); kunmap(page); unlock_page(page); put_page(page); index++; pg_offset = 0; len -= cur_len; - ret += cur_len; + read += cur_len; } -out: iput(inode); return ret; } @@ -4882,7 +4873,6 @@ static int send_write(struct send_ctx *sctx, u64 offset, u32 len) struct btrfs_fs_info *fs_info = sctx->send_root->fs_info; int ret = 0; struct fs_path *p; - ssize_t num_read = 0; p = fs_path_alloc(); if (!p) @@ -4890,12 +4880,9 @@ static int send_write(struct send_ctx *sctx, u64 offset, u32 len) btrfs_debug(fs_info, "send_write offset=%llu, len=%d", offset, len); - num_read = fill_read_buf(sctx, offset, len); - if (num_read <= 0) { - if (num_read < 0) - ret = num_read; + ret = fill_read_buf(sctx, offset, len); + if (ret < 0) goto out; - } ret = begin_cmd(sctx, BTRFS_SEND_C_WRITE); if (ret < 0) @@ -4907,16 +4894,14 @@ static int send_write(struct send_ctx *sctx, u64 offset, u32 len) TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p); TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset); - TLV_PUT(sctx, BTRFS_SEND_A_DATA, sctx->read_buf, num_read); + TLV_PUT(sctx, BTRFS_SEND_A_DATA, sctx->read_buf, len); ret = send_cmd(sctx); tlv_put_failure: out: fs_path_free(p); - if (ret < 0) - return ret; - return num_read; + return ret; } /* @@ -5095,9 +5080,7 @@ static int send_extent_data(struct send_ctx *sctx, ret = send_write(sctx, offset + sent, size); if (ret < 0) return ret; - if (!ret) - break; - sent += ret; + sent += size; } return 0; } From patchwork Fri Aug 21 07:39:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728471 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CA43B1575 for ; Fri, 21 Aug 2020 07:40:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A9F232087D for ; Fri, 21 Aug 2020 07:40:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="yF3ZUILB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728341AbgHUHkj (ORCPT ); Fri, 21 Aug 2020 03:40:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727902AbgHUHkT (ORCPT ); Fri, 21 Aug 2020 03:40:19 -0400 Received: from mail-pg1-x543.google.com (mail-pg1-x543.google.com [IPv6:2607:f8b0:4864:20::543]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98220C061387 for ; Fri, 21 Aug 2020 00:40:19 -0700 (PDT) Received: by mail-pg1-x543.google.com with SMTP id 189so599926pgg.13 for ; Fri, 21 Aug 2020 00:40:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rV0Iw2CCSZHG6p1G4uFOZPtC+8d3NmwhtMPWnU59b4o=; b=yF3ZUILBJ6pRuJOTEVflPp/vEYfpVErHnkY9F4KHaPMN2rchhmZ4blW6ivIKX/t5kk Bc4jBnsEvq0Al4xyHfDG4KcvMwGdOqguFCG/BabKcZLu4fgFmeNlh3/vqHK4mqim4N5n wPoJpKwrMsQiiES8kp/0YZWkWxUeXm2Ar0tWDe/Zh3t62KJ9z/HdxmIdwHaUyck3BnKe UanAV49SOd2H+bW+EVEGqhpOvnCElcpIIzhcoLeKJBy4juaDdohUU1H1H4WUZ85079tf IGbrthiM0giWDVo/bWkwdWKRRtp1blu0V3440gZZ0AugisuBgt+CCGN94WASxhEDo/Nr 0Jvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rV0Iw2CCSZHG6p1G4uFOZPtC+8d3NmwhtMPWnU59b4o=; b=W5j5DtTMSH4Oq4/3XqiEd8e4Q5uxQGKW8jRdpwBiuMKrnMsURHThut8n7/TlUMj1CL tsV2uaH1l0PNcQnH8gqK0VICyyIOyDJy4e5B8Rm+Irzh1LUKsYkCvCGM88G+dyucxARe LP7A/Z2hgj9mdBgr6/kTV21JrIR/9pbPBlJaV8whZFV8msG9vcwK2DLZn9hwwS2S8owh tNu7f3bzt/Ewh76Z77F9JrLCC+pZyTbrxE0K12ennJxyHjhJ11baJte5el8yX12K8x8W yryB/Bkjvdzf89+pfOr7MXPsiyWrU49iE2peMd4ohCMW7ySAG5W++J3T5Z1KCXBRfqBE 4AUA== X-Gm-Message-State: AOAM530kH0RBGLGCARxOL/k/SYiPSJLNItg75XMFtq8WgJZbBHcAGRIM k9NWBFheWjyh1YWNZ16xIpUMUl1KSeMbVw== X-Google-Smtp-Source: ABdhPJx5BYSVv2feYz5UTuKAlmjkIsTLVPVDIM/X9wiNoLdnQYy0W7Vg3XqVJIGd7kHmLA05UdiIAQ== X-Received: by 2002:a62:9246:: with SMTP id o67mr1416722pfd.249.1597995619020; Fri, 21 Aug 2020 00:40:19 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:17 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 2/9] btrfs: send: avoid copying file data Date: Fri, 21 Aug 2020 00:39:52 -0700 Message-Id: X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Omar Sandoval send_write() currently copies from the page cache to sctx->read_buf, and then from sctx->read_buf to sctx->send_buf. Similarly, send_hole() zeroes sctx->read_buf and then copies from sctx->read_buf to sctx->send_buf. However, if we write the TLV header manually, we can copy to sctx->send_buf directly and get rid of sctx->read_buf. Signed-off-by: Omar Sandoval Reviewed-by: Filipe Manana Reviewed-by: Josef Bacik --- fs/btrfs/send.c | 65 +++++++++++++++++++++++++++++-------------------- fs/btrfs/send.h | 1 - 2 files changed, 39 insertions(+), 27 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 8af5e867e4ca..e70f5ceb3261 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -122,8 +122,6 @@ struct send_ctx { struct file_ra_state ra; - char *read_buf; - /* * We process inodes by their increasing order, so if before an * incremental send we reverse the parent/child relationship of @@ -4794,7 +4792,25 @@ static int process_all_new_xattrs(struct send_ctx *sctx) return ret; } -static int fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len) +static u64 max_send_read_size(struct send_ctx *sctx) +{ + return sctx->send_max_size - SZ_16K; +} + +static int put_data_header(struct send_ctx *sctx, u32 len) +{ + struct btrfs_tlv_header *hdr; + + if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len) + return -EOVERFLOW; + hdr = (struct btrfs_tlv_header *)(sctx->send_buf + sctx->send_size); + hdr->tlv_type = cpu_to_le16(BTRFS_SEND_A_DATA); + hdr->tlv_len = cpu_to_le16(len); + sctx->send_size += sizeof(*hdr); + return 0; +} + +static int put_file_data(struct send_ctx *sctx, u64 offset, u32 len) { struct btrfs_root *root = sctx->send_root; struct btrfs_fs_info *fs_info = root->fs_info; @@ -4804,8 +4820,11 @@ static int fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len) pgoff_t index = offset >> PAGE_SHIFT; pgoff_t last_index; unsigned pg_offset = offset_in_page(offset); - int ret = 0; - size_t read = 0; + int ret; + + ret = put_data_header(sctx, len); + if (ret) + return ret; inode = btrfs_iget(fs_info->sb, sctx->cur_ino, root); if (IS_ERR(inode)) @@ -4851,14 +4870,15 @@ static int fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len) } addr = kmap(page); - memcpy(sctx->read_buf + read, addr + pg_offset, cur_len); + memcpy(sctx->send_buf + sctx->send_size, addr + pg_offset, + cur_len); kunmap(page); unlock_page(page); put_page(page); index++; pg_offset = 0; len -= cur_len; - read += cur_len; + sctx->send_size += cur_len; } iput(inode); return ret; @@ -4880,10 +4900,6 @@ static int send_write(struct send_ctx *sctx, u64 offset, u32 len) btrfs_debug(fs_info, "send_write offset=%llu, len=%d", offset, len); - ret = fill_read_buf(sctx, offset, len); - if (ret < 0) - goto out; - ret = begin_cmd(sctx, BTRFS_SEND_C_WRITE); if (ret < 0) goto out; @@ -4894,7 +4910,9 @@ static int send_write(struct send_ctx *sctx, u64 offset, u32 len) TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p); TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset); - TLV_PUT(sctx, BTRFS_SEND_A_DATA, sctx->read_buf, len); + ret = put_file_data(sctx, offset, len); + if (ret < 0) + goto out; ret = send_cmd(sctx); @@ -5013,8 +5031,8 @@ static int send_update_extent(struct send_ctx *sctx, static int send_hole(struct send_ctx *sctx, u64 end) { struct fs_path *p = NULL; + u64 read_size = max_send_read_size(sctx); u64 offset = sctx->cur_inode_last_extent; - u64 len; int ret = 0; /* @@ -5041,16 +5059,19 @@ static int send_hole(struct send_ctx *sctx, u64 end) ret = get_cur_path(sctx, sctx->cur_ino, sctx->cur_inode_gen, p); if (ret < 0) goto tlv_put_failure; - memset(sctx->read_buf, 0, BTRFS_SEND_READ_SIZE); while (offset < end) { - len = min_t(u64, end - offset, BTRFS_SEND_READ_SIZE); + u64 len = min(end - offset, read_size); ret = begin_cmd(sctx, BTRFS_SEND_C_WRITE); if (ret < 0) break; TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p); TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset); - TLV_PUT(sctx, BTRFS_SEND_A_DATA, sctx->read_buf, len); + ret = put_data_header(sctx, len); + if (ret < 0) + break; + memset(sctx->send_buf + sctx->send_size, 0, len); + sctx->send_size += len; ret = send_cmd(sctx); if (ret < 0) break; @@ -5066,17 +5087,16 @@ static int send_extent_data(struct send_ctx *sctx, const u64 offset, const u64 len) { + u64 read_size = max_send_read_size(sctx); u64 sent = 0; if (sctx->flags & BTRFS_SEND_FLAG_NO_FILE_DATA) return send_update_extent(sctx, offset, len); while (sent < len) { - u64 size = len - sent; + u64 size = min(len - sent, read_size); int ret; - if (size > BTRFS_SEND_READ_SIZE) - size = BTRFS_SEND_READ_SIZE; ret = send_write(sctx, offset + sent, size); if (ret < 0) return ret; @@ -7145,12 +7165,6 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) goto out; } - sctx->read_buf = kvmalloc(BTRFS_SEND_READ_SIZE, GFP_KERNEL); - if (!sctx->read_buf) { - ret = -ENOMEM; - goto out; - } - sctx->pending_dir_moves = RB_ROOT; sctx->waiting_dir_moves = RB_ROOT; sctx->orphan_dirs = RB_ROOT; @@ -7354,7 +7368,6 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) kvfree(sctx->clone_roots); kvfree(sctx->send_buf); - kvfree(sctx->read_buf); name_cache_free(sctx); diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index ead397f7034f..de91488b7cd0 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -13,7 +13,6 @@ #define BTRFS_SEND_STREAM_VERSION 1 #define BTRFS_SEND_BUF_SIZE SZ_64K -#define BTRFS_SEND_READ_SIZE (48 * SZ_1K) enum btrfs_tlv_type { BTRFS_TLV_U8, From patchwork Fri Aug 21 07:39:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728473 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9532A739 for ; Fri, 21 Aug 2020 07:40:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7917A2087D for ; Fri, 21 Aug 2020 07:40:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="fQi6hg2l" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728250AbgHUHkl (ORCPT ); Fri, 21 Aug 2020 03:40:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728251AbgHUHkV (ORCPT ); Fri, 21 Aug 2020 03:40:21 -0400 Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 82AC0C061344 for ; Fri, 21 Aug 2020 00:40:21 -0700 (PDT) Received: by mail-pf1-x444.google.com with SMTP id u20so680065pfn.0 for ; Fri, 21 Aug 2020 00:40:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WxMUvNLFoM8CqgnAU/FexkWAf/EILGzRJKuKOkapjZs=; b=fQi6hg2lyIXY/tp2l5XOHtW4GiLOPYUvJ9Jr6FnWnG3kPcU99T8Gtfvit2LRq0kXAc zvaztwb5qb3/YiI5V17A91xL06hsJQj6je+Avg/r4+jVP7DjEMq0U7XwbGSEICgTwRjI BYzTB3Kv9npv0UhNa0l+VtyleNqkc9XT0x6Gv2mnTeG4TkOFpZvl+VEfzZuTxb6os814 UbTcE9PUWSlIhy59UIeNTzn2kLwQWeIYDmUptVjRclK5l51jXYp2U786nFY2cLU0fG7I QspN9ijLN3riItYOBzLi2XV/YXDkpW0xpQI4E0PIKvuffJWFrgCgPmcZRaCN0JTxNBR2 Yj0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WxMUvNLFoM8CqgnAU/FexkWAf/EILGzRJKuKOkapjZs=; b=LfUTAmeF1/ZCE3jD5kmsAVDvjvkVjxKgYx00D9nb+q3mUYNGnmEFOxFoGaytnTZ0op QpQWrY5uz/zMJr2hUnODJvbP8HOsr4UosPC6cllJaebkJ1+jnBMo+jYCyGDbBMyLk9kZ 2lO/NRQOgKria5yiGG3TIcOOcTWuqZiHdiZrGBZfFGL7Np+Uq4pFFV4yUR9pH04IHZe3 FcwDvIr4oh19lRsuZfJM2lWcbEFIOgFqagxrN1PgS6kgb4HR6c9gHUBZFA+mH7mjXxE6 OVXUeJKSPHSILMDS6kMAVyU5TIxDlBCEZNVHWNNrJw9/+KdJvYqUBcVV+IsG+940v4yb p/VA== X-Gm-Message-State: AOAM532niCNeNhyKVOHnvkSAzUlAUMpYQeoFi4LwtRrEjcs6GIYZ3uc/ X3uOEJgenFCDVdreoKn/3UnCBQ== X-Google-Smtp-Source: ABdhPJzKGa7BAFakQj/fH6OwBwbA7yTK86To7UfDRqnod0zu9Ve1yVQN5nfGzPIxB8zbNFFaAKmJIw== X-Received: by 2002:a05:6a00:15d0:: with SMTP id o16mr1486151pfu.230.1597995620998; Fri, 21 Aug 2020 00:40:20 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:19 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 3/9] btrfs: send: use btrfs_file_extent_end() in send_write_or_clone() Date: Fri, 21 Aug 2020 00:39:53 -0700 Message-Id: X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Omar Sandoval send_write_or_clone() basically has an open-coded copy of btrfs_file_extent_end() except that it (incorrectly) aligns to PAGE_SIZE instead of sectorsize. Fix and simplify the code by using btrfs_file_extent_end(). Signed-off-by: Omar Sandoval Reviewed-by: Filipe Manana --- fs/btrfs/send.c | 44 +++++++++++--------------------------------- 1 file changed, 11 insertions(+), 33 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index e70f5ceb3261..37ce21361782 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -5400,51 +5400,29 @@ static int send_write_or_clone(struct send_ctx *sctx, struct clone_root *clone_root) { int ret = 0; - struct btrfs_file_extent_item *ei; u64 offset = key->offset; - u64 len; - u8 type; + u64 end; u64 bs = sctx->send_root->fs_info->sb->s_blocksize; - ei = btrfs_item_ptr(path->nodes[0], path->slots[0], - struct btrfs_file_extent_item); - type = btrfs_file_extent_type(path->nodes[0], ei); - if (type == BTRFS_FILE_EXTENT_INLINE) { - len = btrfs_file_extent_ram_bytes(path->nodes[0], ei); - /* - * it is possible the inline item won't cover the whole page, - * but there may be items after this page. Make - * sure to send the whole thing - */ - len = PAGE_ALIGN(len); - } else { - len = btrfs_file_extent_num_bytes(path->nodes[0], ei); - } - - if (offset >= sctx->cur_inode_size) { - ret = 0; - goto out; - } - if (offset + len > sctx->cur_inode_size) - len = sctx->cur_inode_size - offset; - if (len == 0) { - ret = 0; - goto out; - } + end = min(btrfs_file_extent_end(path), sctx->cur_inode_size); + if (offset >= end) + return 0; - if (clone_root && IS_ALIGNED(offset + len, bs)) { + if (clone_root && IS_ALIGNED(end, bs)) { + struct btrfs_file_extent_item *ei; u64 disk_byte; u64 data_offset; + ei = btrfs_item_ptr(path->nodes[0], path->slots[0], + struct btrfs_file_extent_item); disk_byte = btrfs_file_extent_disk_bytenr(path->nodes[0], ei); data_offset = btrfs_file_extent_offset(path->nodes[0], ei); ret = clone_range(sctx, clone_root, disk_byte, data_offset, - offset, len); + offset, end - offset); } else { - ret = send_extent_data(sctx, offset, len); + ret = send_extent_data(sctx, offset, end - offset); } - sctx->cur_inode_next_write_offset = offset + len; -out: + sctx->cur_inode_next_write_offset = end; return ret; } From patchwork Fri Aug 21 07:40:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728505 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 60938739 for ; Fri, 21 Aug 2020 07:41:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 47E89208DB for ; Fri, 21 Aug 2020 07:41:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="IP6qcyz3" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728437AbgHUHlN (ORCPT ); Fri, 21 Aug 2020 03:41:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728358AbgHUHkn (ORCPT ); Fri, 21 Aug 2020 03:40:43 -0400 Received: from mail-pl1-x641.google.com (mail-pl1-x641.google.com [IPv6:2607:f8b0:4864:20::641]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77D36C061385 for ; Fri, 21 Aug 2020 00:40:43 -0700 (PDT) Received: by mail-pl1-x641.google.com with SMTP id z23so513833plo.8 for ; Fri, 21 Aug 2020 00:40:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HPllMP1dggYArlaBG8NbdTKVXA2/TOK/hKcMMPnu8LA=; b=IP6qcyz3YWCExIoUyhje7azbZk90z3J/+FZDf98t6ckQrKngJGknpqKrixQJHnL6n/ nJo6BM45hA6TVWMpLVxhEBKr7SLPZsVQvGGtl8pwv/9tZUaozMggjPKKrSOhf39LmKEe kREF5Uijlym1NEz/PLMdJvy5Y/QnT9z1YczOrRmuEwhQKtxdKseemhOOhuotGdKouSNB 7xDy2d2dLB2NLD3hUjaWuR898WiAss4ammDiyezszjDsLinXOoWPZHjw7KRR0EwD+84g zQwLNiYR2WMPjlG+cr212Sk1vTbNdDGyScUDD7S54qCiYBRttx5MatrFui/tNtqplOtl TDDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HPllMP1dggYArlaBG8NbdTKVXA2/TOK/hKcMMPnu8LA=; b=TO7yaXJYNsEgSf0oajj2xw2CuTfUCNaoJ+gALFcmC6noSLgeNaCLWqAoZHzlMND24m 1uz/n3d/id3VY+rHPIjEzwDOIyN4AHg7d6NNZKJG0uOD7JH3Ma2D5Fz94Kfpa1PVT7Zv jHsX7bu5WRRTqDBOjB0fqtsb5DyCgaPhQb25zP7/hoXO/lnIOzBgH8qSHzC9Us3UO27x 4nRj2RJXHuVSHgzkRreWFFwISA8guTdmxWaebhwPJouCxEQrktYAD4fdFmZUh2wIUU8O kSps6Y3UyTdn0ybKude/Lf2EeBZ05EwDyDwDmtI5BZjDlKNmPZW1CjuRgG6r8KSo1WTw uJgQ== X-Gm-Message-State: AOAM531PLI3AKStkpPVSwWkyPCIhKP+uH4AuX9Uqm6WfY6RHIem6979u o0A4/DMAoWuskU7wNaXkC+Dw3/pT1vaVtw== X-Google-Smtp-Source: ABdhPJzjSz6B/wIUhrw81Khq6id6K/AFXO+kMSQ2LmrsdpT9MO9QNi0H3um10Hmj7H+rctPHlxvQAQ== X-Received: by 2002:a17:90a:c787:: with SMTP id gn7mr1497597pjb.90.1597995642917; Fri, 21 Aug 2020 00:40:42 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:42 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 04/11] btrfs-progs: receive: add send stream v2 cmds and attrs to send.h Date: Fri, 21 Aug 2020 00:40:03 -0700 Message-Id: <3477da4106d103099b41705e2a84fb58c18cbd29.1597994354.git.osandov@osandov.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Boris Burkov Send stream v2 adds three commands and several attributes associated to those commands. Before we implement processing them, add all the commands and attributes. This avoids leaving the enums in an intermediate state that doesn't correspond to any version of send stream. Signed-off-by: Boris Burkov --- send.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/send.h b/send.h index 228928a0..3c47e0c7 100644 --- a/send.h +++ b/send.h @@ -98,6 +98,11 @@ enum btrfs_send_cmd { BTRFS_SEND_C_END, BTRFS_SEND_C_UPDATE_EXTENT, + + BTRFS_SEND_C_FALLOCATE, + BTRFS_SEND_C_SETFLAGS, + BTRFS_SEND_C_ENCODED_WRITE, + __BTRFS_SEND_C_MAX, }; #define BTRFS_SEND_C_MAX (__BTRFS_SEND_C_MAX - 1) @@ -136,6 +141,16 @@ enum { BTRFS_SEND_A_CLONE_OFFSET, BTRFS_SEND_A_CLONE_LEN, + BTRFS_SEND_A_FALLOCATE_MODE, + + BTRFS_SEND_A_SETFLAGS_FLAGS, + + BTRFS_SEND_A_UNENCODED_FILE_LEN, + BTRFS_SEND_A_UNENCODED_LEN, + BTRFS_SEND_A_UNENCODED_OFFSET, + BTRFS_SEND_A_COMPRESSION, + BTRFS_SEND_A_ENCRYPTION, + __BTRFS_SEND_A_MAX, }; #define BTRFS_SEND_A_MAX (__BTRFS_SEND_A_MAX - 1) From patchwork Fri Aug 21 07:39:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728483 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ECFA81575 for ; Fri, 21 Aug 2020 07:40:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D43922078D for ; Fri, 21 Aug 2020 07:40:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="dSg2e+H2" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728364AbgHUHko (ORCPT ); Fri, 21 Aug 2020 03:40:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728266AbgHUHk0 (ORCPT ); Fri, 21 Aug 2020 03:40:26 -0400 Received: from mail-pg1-x541.google.com (mail-pg1-x541.google.com [IPv6:2607:f8b0:4864:20::541]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4235EC061388 for ; Fri, 21 Aug 2020 00:40:26 -0700 (PDT) Received: by mail-pg1-x541.google.com with SMTP id o13so627629pgf.0 for ; Fri, 21 Aug 2020 00:40:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1WFgjJ3rZW2t7bP/7ZGExcYanOQgvJvNl08xr/OUhp8=; b=dSg2e+H2NFSciBdWMN+2zt/ZVTY+R/pe0F+3RGzZK41krLC+OjGma5ZjnKJyfc3nZE Ztr9zGKRsIn8jIb4exSJiIiZkHITbW36ekN401fQengv4nO2CrTBv8NrN411ivDKZihR om7XiqZXRJU4UFpspAWhOFyjWYtvoPl+RShLOe9ucOOJ7ZVFZrgN9FKYUWCCzX3otcu3 irpkJIWAh0ZPgkR087E9sjjlLvKX5HQfYRMNmUOY/R+PWjxM4o2cvkdOIIY7tn76llsL 18Pm6SlITDGkWEF1oMdpXTL3YR5kLMJxh0/++yrWGvcJGmWNxSyFNqWQVnJPUSVsynFO yHvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1WFgjJ3rZW2t7bP/7ZGExcYanOQgvJvNl08xr/OUhp8=; b=k+qpRFLyglEFu53LfkG1Ev5G6kppPeHt2Jp9Vlgs+RaERzxVDU6sOgE356BcJW81HI 9m/If0TJbWnbLPFpPKdItLJXPL4xo7FuK7vG2PzlAeoYt4EBkbTgqZ0gJWlJoqb/GYoa 6gDJr2EGhYyMCSC/K4YbGga+jAqhy8mvIaFCdQQSGVGVYggkHyAT6LizcLFUfiAs2oyG n3jNjiNM4C9/uQNnCKCHbvkRRLlm1jnUzICHuBfgUdLf9LnSLRf8SpMIlheaFU81IX7k yzTLeiMepKoLye5JvhITuGhFgm0gEj779miqFgS0w/2EGDjdHudKBMhak2CbJk0316+B 8LIQ== X-Gm-Message-State: AOAM532V0SppyixcNVRe3jnly/Vi4qZzWhoPqqk8vn+lUX1PCwdgu6PJ NBRMAKg9wEjCzIACCvAhKFTTGQ== X-Google-Smtp-Source: ABdhPJxKlmIXI+kh+VsqvVebfzvPnWqxe1vtPxCacXLgvBfG6sEyFX0MiDh0Ot2V48zgw7hzbioy1w== X-Received: by 2002:a63:5559:: with SMTP id f25mr1407868pgm.369.1597995625667; Fri, 21 Aug 2020 00:40:25 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:23 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 5/9] btrfs: add send stream v2 definitions Date: Fri, 21 Aug 2020 00:39:55 -0700 Message-Id: X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Omar Sandoval This adds the definitions of the new commands for send stream version 2 and their respective attributes: fallocate, FS_IOC_SETFLAGS (a.k.a. chattr), and encoded writes. It also documents two changes to the send stream format in v2: the receiver shouldn't assume a maximum command size, and the DATA attribute is encoded differently to allow for writes larger than 64k. These will be implemented in subsequent changes, and then the ioctl will accept the new flags. Signed-off-by: Omar Sandoval Reviewed-by: Josef Bacik --- fs/btrfs/send.c | 2 +- fs/btrfs/send.h | 30 +++++++++++++++++++++++++++++- include/uapi/linux/btrfs.h | 13 +++++++++++++ 3 files changed, 43 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 37ce21361782..e25c3391fc02 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -7136,7 +7136,7 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) sctx->clone_roots_cnt = arg->clone_sources_count; - sctx->send_max_size = BTRFS_SEND_BUF_SIZE; + sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); if (!sctx->send_buf) { ret = -ENOMEM; diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index de91488b7cd0..9f4f7b96b1eb 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -12,7 +12,11 @@ #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" #define BTRFS_SEND_STREAM_VERSION 1 -#define BTRFS_SEND_BUF_SIZE SZ_64K +/* + * In send stream v1, no command is larger than 64k. In send stream v2, no limit + * should be assumed. + */ +#define BTRFS_SEND_BUF_SIZE_V1 SZ_64K enum btrfs_tlv_type { BTRFS_TLV_U8, @@ -76,6 +80,13 @@ enum btrfs_send_cmd { BTRFS_SEND_C_END, BTRFS_SEND_C_UPDATE_EXTENT, + + /* The following commands were added in send stream v2. */ + + BTRFS_SEND_C_FALLOCATE, + BTRFS_SEND_C_SETFLAGS, + BTRFS_SEND_C_ENCODED_WRITE, + __BTRFS_SEND_C_MAX, }; #define BTRFS_SEND_C_MAX (__BTRFS_SEND_C_MAX - 1) @@ -106,6 +117,11 @@ enum { BTRFS_SEND_A_PATH_LINK, BTRFS_SEND_A_FILE_OFFSET, + /* + * In send stream v2, this attribute is special: it must be the last + * attribute in a command, its header contains only the type, and its + * length is implicitly the remaining length of the command. + */ BTRFS_SEND_A_DATA, BTRFS_SEND_A_CLONE_UUID, @@ -114,6 +130,18 @@ enum { BTRFS_SEND_A_CLONE_OFFSET, BTRFS_SEND_A_CLONE_LEN, + /* The following attributes were added in send stream v2. */ + + BTRFS_SEND_A_FALLOCATE_MODE, + + BTRFS_SEND_A_SETFLAGS_FLAGS, + + BTRFS_SEND_A_UNENCODED_FILE_LEN, + BTRFS_SEND_A_UNENCODED_LEN, + BTRFS_SEND_A_UNENCODED_OFFSET, + BTRFS_SEND_A_COMPRESSION, + BTRFS_SEND_A_ENCRYPTION, + __BTRFS_SEND_A_MAX, }; #define BTRFS_SEND_A_MAX (__BTRFS_SEND_A_MAX - 1) diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h index 2c39d15a2beb..51e69f28d22d 100644 --- a/include/uapi/linux/btrfs.h +++ b/include/uapi/linux/btrfs.h @@ -769,6 +769,19 @@ struct btrfs_ioctl_received_subvol_args { */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ From patchwork Fri Aug 21 07:39:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728475 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3F8A2739 for ; Fri, 21 Aug 2020 07:40:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 236CA207DF for ; Fri, 21 Aug 2020 07:40:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="iau3GpV8" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728369AbgHUHkp (ORCPT ); Fri, 21 Aug 2020 03:40:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37186 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728267AbgHUHk2 (ORCPT ); Fri, 21 Aug 2020 03:40:28 -0400 Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 55F20C061385 for ; Fri, 21 Aug 2020 00:40:28 -0700 (PDT) Received: by mail-pf1-x443.google.com with SMTP id m71so678164pfd.1 for ; Fri, 21 Aug 2020 00:40:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MpMHOROWH1K8dL9ToH1ce+0j8obccQVesOcDCSINSNg=; b=iau3GpV8ekB9m+DmlUkMpAcGc34fTiQC1Xkrmkdflfo80BK9SRQVIbkJcXexjwHMhf yLwurQrHHV9kx2jJUr9eCK2J26VhrOmBW2gMzW3eLN/3lVQ3en5/3/zx5QFRLq2X0qUa XSjhYR/xJkNn+x5tLTM4sRNfIugFG8JEDe7zqC7EGKLmrBJPm7AWBKjdIkZHD0ZDk/Hx vsCyLAeCDewv3odZ7axGSkfGsm+ponE2mDza8rBgO61tKP7QUSeqg3JDO31OmC8kOrWc wE6x4QH8fMoR+qUSc2Vvkrbk9rMoBM881+8o94PDHOKImxZZ9xEYL66k1TrVKYEMa5Qw Esfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MpMHOROWH1K8dL9ToH1ce+0j8obccQVesOcDCSINSNg=; b=QI48NU3R0ptHF2siZos3OnNTyJiBFQJzxhF/l8z5d3O5GqW7SSIQ4mCcO+7jsanZ16 TW2fRXYihWyVCsc+RmRkFlsAq9H1CejlHbVsyNPVD1EOBWfnxuzCgkq+6DT58zQqv1Cv Rh0TiU6V00uC1DG6zOEWFz5A8TzUpZeFHsWYK4xzb9Z2V/P7QqDW0JQE9szvigTwHB0E 0tkRK7RbwaECE7X9pdiZflU1haX+EP9oJjgmnhQ2B6vI1wFsI2pIMkAxbhaDeXnBw92D qRkKZrbfAdAaF0mN9atotlDzHuzregidksETc9YERZfaKBmgtfWrAN/3P8UPeEwgvVtc Rx4g== X-Gm-Message-State: AOAM531riVVMCjFHRwMHZdhVt/4n5Khlp26jPAgjfCPHinLGGkb89W0n 104DQ6GA6FhD0IAW/6vaZI+wF9TXeuIKBQ== X-Google-Smtp-Source: ABdhPJydIf99Q5Hfrly97cyxhvALDoesvJp9B1+WUNcmfgiAj0fKuLIqZ6rzXogAAWcrSVCLW+xVig== X-Received: by 2002:a62:8141:: with SMTP id t62mr1426534pfd.282.1597995627765; Fri, 21 Aug 2020 00:40:27 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:26 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 6/9] btrfs: send: write larger chunks when using stream v2 Date: Fri, 21 Aug 2020 00:39:56 -0700 Message-Id: X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Omar Sandoval The length field of the send stream TLV header is 16 bits. This means that the maximum amount of data that can be sent for one write is 64k minus one. However, encoded writes must be able to send the maximum compressed extent (128k) in one command. To support this, send stream version 2 encodes the DATA attribute differently: it has no length field, and the length is implicitly up to the end of containing command (which has a 32-bit length field). Although this is necessary for encoded writes, normal writes can benefit from it, too. For v2, let's bump up the send buffer to the maximum compressed extent size plus 16k for the other metadata (144k total). Since this will most likely be vmalloc'd (and always will be after the next commit), we round it up to the next page since we might as well use the rest of the page on systems with >16k pages. Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index e25c3391fc02..c0f81d302f49 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -4799,14 +4799,27 @@ static u64 max_send_read_size(struct send_ctx *sctx) static int put_data_header(struct send_ctx *sctx, u32 len) { - struct btrfs_tlv_header *hdr; + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) { + __le16 tlv_type; + + if (sctx->send_max_size - sctx->send_size < + sizeof(tlv_type) + len) + return -EOVERFLOW; + tlv_type = cpu_to_le16(BTRFS_SEND_A_DATA); + memcpy(sctx->send_buf + sctx->send_size, &tlv_type, + sizeof(tlv_type)); + sctx->send_size += sizeof(tlv_type); + } else { + struct btrfs_tlv_header *hdr; - if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len) - return -EOVERFLOW; - hdr = (struct btrfs_tlv_header *)(sctx->send_buf + sctx->send_size); - hdr->tlv_type = cpu_to_le16(BTRFS_SEND_A_DATA); - hdr->tlv_len = cpu_to_le16(len); - sctx->send_size += sizeof(*hdr); + if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len) + return -EOVERFLOW; + hdr = (struct btrfs_tlv_header *)(sctx->send_buf + + sctx->send_size); + hdr->tlv_type = cpu_to_le16(BTRFS_SEND_A_DATA); + hdr->tlv_len = cpu_to_le16(len); + sctx->send_size += sizeof(*hdr); + } return 0; } @@ -7136,7 +7149,12 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) sctx->clone_roots_cnt = arg->clone_sources_count; - sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) { + sctx->send_max_size = ALIGN(SZ_16K + BTRFS_MAX_COMPRESSED, + PAGE_SIZE); + } else { + sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; + } sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); if (!sctx->send_buf) { ret = -ENOMEM; From patchwork Fri Aug 21 07:40:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728511 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A4FD2739 for ; Fri, 21 Aug 2020 07:41:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 88D902087D for ; Fri, 21 Aug 2020 07:41:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="WGb9lD/E" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728460AbgHUHlY (ORCPT ); Fri, 21 Aug 2020 03:41:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728379AbgHUHkv (ORCPT ); Fri, 21 Aug 2020 03:40:51 -0400 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0D94C061348 for ; Fri, 21 Aug 2020 00:40:49 -0700 (PDT) Received: by mail-pl1-x644.google.com with SMTP id v16so525635plo.1 for ; Fri, 21 Aug 2020 00:40:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JJzpMUwPLLywPgZkTrDSxUbJuyFfFwLySMrCBI6an1Q=; b=WGb9lD/EEk978nTwkBnFUsrmz3dmqQKyQbLPwgQoKrWdtYB6kznfddOnyC0TdeBF0y vRqm5dQyCRM/c26lXGuBu09O8VS7qa24Q4ynJEPOGjk4trJu8hpNWtnNVfFpU4nyXrP8 5O8lbSNrFT2hzzbVAdXBHUMDyZzjVka/HpqCf+BKVu/DYHMxxizDGCprS/Bajbni5IuY xvXdpZ2lQoSHVMECaT7QhLOd5EE4Yn94SM+QNMwchDZYXVnMAotJBP2JSHUzxGyNRpYJ 8ZwGdqt8ZnbSel36mhQzN1e58pODamtIHPUft1QUHp3ZIqauxrV5EoV7DmcpUrsN83JD grgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JJzpMUwPLLywPgZkTrDSxUbJuyFfFwLySMrCBI6an1Q=; b=iparRGNzLB3WTVSiN7KsROPZ+ZLx5IQxxrgDirB3ATPvRtC2To446GUtiZ7cAPSgMY BckxItP6ofna0F0XwWTsZ3UmnZOrcg7+KfZsy8jxHYHcTDnfQP/q1n0ymNFTJ/JY3oBY mZ6IktXkFnhtAu/ZJ6WiZO6xqQnryR/v1lwRZlAa87ZcSUoJPFqSQoO1HKZg0frQ79B1 2o+YFd0S8/TSBXanqo5zjcmLiOUUY39ZvNJMK6BmZw6bbD53AK34BYvTuTNgtgMN59sQ keYwtMuSZ6rOkXOuw/m0qZzAXF0IRY85edZQSA4wzh8tjfMF952+Z9bilzrWIyOP/f5N bcog== X-Gm-Message-State: AOAM53200Mtp42d3Y4buIQ6VTjqczzlqEdu1YjPTMneFNAcyWu6PFKQy Ya/33ZmMmmqg9l0Ak8MDC3iFaw== X-Google-Smtp-Source: ABdhPJxsk3zqP0HOjMeW9POjNCWYqpmClhHKnIxSUapiYItczePylr2VHUZuKznq9sCJyt45kJAimw== X-Received: by 2002:a17:90a:bf86:: with SMTP id d6mr1394211pjs.83.1597995649334; Fri, 21 Aug 2020 00:40:49 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:47 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 07/11] btrfs-progs: receive: encoded_write fallback to explicit decode and write Date: Fri, 21 Aug 2020 00:40:06 -0700 Message-Id: <8e4bf8b613ad4504376e9f00ff87ee2e51222a23.1597994354.git.osandov@osandov.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Boris Burkov An encoded_write can fail if the file system it is being applied to does not support encoded writes or if it can't find enough contiguous space to accommodate the encoded extent. In those cases, we can likely still process an encoded_write by explicitly decoding the data and doing a normal write. Add the necessary fallback path for decoding data compressed with zlib, lzo, or zstd. zlib and zstd have reusable decoding context data structures which we cache in the receive context so that we don't have to recreate them on every encoded_write. Finally, add a command line flag for force-decompress which causes receive to always use the fallback path rather than first attempting the encoded write. Signed-off-by: Boris Burkov --- cmds/receive.c | 273 +++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 262 insertions(+), 11 deletions(-) diff --git a/cmds/receive.c b/cmds/receive.c index cd0f47ec..c67d4653 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -41,6 +41,10 @@ #include #include +#include +#include +#include + #include "kernel-shared/ctree.h" #include "ioctl.h" #include "cmds/commands.h" @@ -82,6 +86,8 @@ struct btrfs_receive int honor_end_cmd; + int force_decompress; + /* * Buffer to store capabilities from security.capabilities xattr, * usually 20 bytes, but make same room for potentially larger @@ -89,6 +95,10 @@ struct btrfs_receive */ char cached_capabilities[64]; int cached_capabilities_len; + + /* Reuse stream objects for encoded_write decompression fallback */ + ZSTD_DStream *zstd_dstream; + z_stream *zlib_stream; }; static int finish_subvol(struct btrfs_receive *rctx) @@ -1073,9 +1083,210 @@ static int process_update_extent(const char *path, u64 offset, u64 len, return 0; } +static int decompress_zlib(struct btrfs_receive *rctx, const void *encoded_data, + u64 encoded_len, char *unencoded_data, + u64 unencoded_len) +{ + int status = 0; + bool init = false; + int ret; + + if (!rctx->zlib_stream) { + init = true; + rctx->zlib_stream = malloc(sizeof(z_stream)); + if (!rctx->zlib_stream) { + error("failed to allocate zlib stream %m"); + status = -ENOMEM; + goto out; + } + } + rctx->zlib_stream->next_in = (void *)encoded_data; + rctx->zlib_stream->avail_in = encoded_len; + rctx->zlib_stream->next_out = (void *)unencoded_data; + rctx->zlib_stream->avail_out = unencoded_len; + + if (!init) + ret = inflateReset(rctx->zlib_stream); + else { + rctx->zlib_stream->zalloc = Z_NULL; + rctx->zlib_stream->zfree = Z_NULL; + rctx->zlib_stream->opaque = Z_NULL; + ret = inflateInit(rctx->zlib_stream); + } + if (ret != Z_OK) { + error("zlib inflate init failed %d", ret); + status = -EIO; + goto out; + } + + while (rctx->zlib_stream->avail_in > 0 && + rctx->zlib_stream->avail_out > 0) { + ret = inflate(rctx->zlib_stream, Z_FINISH); + if (ret == Z_STREAM_END) { + break; + } else if (ret != Z_OK) { + error("zlib inflate failed %d", ret); + status = -EIO; + break; + } + } +out: + return status; +} + +static int decompress_lzo(const void *encoded_data, u64 encoded_len, + char *unencoded_data, u64 unencoded_len) +{ + uint32_t total_len; + size_t in_pos, out_pos; + + if (encoded_len < 4) { + error("lzo header is truncated"); + return -EIO; + } + memcpy(&total_len, encoded_data, 4); + total_len = le32toh(total_len); + if (total_len > encoded_len) { + error("lzo header is invalid"); + return -EIO; + } + + in_pos = 4; + out_pos = 0; + while (in_pos < total_len && out_pos < unencoded_len) { + uint32_t src_len; + lzo_uint dst_len = unencoded_len - out_pos; + int ret; + + if (total_len - in_pos < 4) { + error("lzo segment header is truncated"); + return -EIO; + } + memcpy(&src_len, encoded_data + in_pos, 4); + src_len = le32toh(src_len); + in_pos += 4; + if (src_len > total_len - in_pos) { + error("lzo segment header is invalid\n"); + return -EIO; + } + + ret = lzo1x_decompress_safe((void *)(encoded_data + in_pos), + src_len, (void *)(unencoded_data + out_pos), &dst_len, + NULL); + if (ret != LZO_E_OK) { + error("lzo1x_decompress_safe failed: %d", ret); + return -EIO; + } + + in_pos += src_len; + out_pos += dst_len; + } + return 0; +} + +static int decompress_zstd(struct btrfs_receive *rctx, const void *encoded_buf, + u64 encoded_len, char *unencoded_buf, + u64 unencoded_len) +{ + ZSTD_inBuffer in_buf = { + .src = encoded_buf, + .size = encoded_len + }; + ZSTD_outBuffer out_buf = { + .dst = unencoded_buf, + .size = unencoded_len + }; + int status = 0; + size_t ret; + + if (!rctx->zstd_dstream) { + rctx->zstd_dstream = ZSTD_createDStream(); + if (!rctx->zstd_dstream) { + error("failed to create zstd dstream"); + status = -ENOMEM; + goto out; + } + } + ret = ZSTD_initDStream(rctx->zstd_dstream); + if (ZSTD_isError(ret)) { + error("failed to init zstd stream %s", ZSTD_getErrorName(ret)); + status = -EIO; + goto out; + } + while (in_buf.pos < in_buf.size && out_buf.pos < out_buf.size) { + ret = ZSTD_decompressStream(rctx->zstd_dstream, &out_buf, &in_buf); + if (ret == 0) { + break; + } else if (ZSTD_isError(ret)) { + error("failed to decompress zstd stream: %s", + ZSTD_getErrorName(ret)); + status = -EIO; + goto out; + } + } + +out: + return status; +} + +static int decompress_and_write(const void *encoded_data, u64 encoded_len, + u64 unencoded_file_len, u64 unencoded_len, + u64 unencoded_offset, u32 compression, + void *user) +{ + int ret = 0; + size_t pos; + ssize_t w; + struct btrfs_receive *rctx = user; + char *unencoded_data; + + unencoded_data = calloc(unencoded_len, sizeof(*unencoded_data)); + if (!unencoded_data) { + error("allocating space for unencoded data failed: %m"); + return -errno; + } + + switch (compression) { + case ENCODED_IOV_COMPRESSION_ZLIB: + ret = decompress_zlib(rctx, encoded_data, encoded_len, + unencoded_data, unencoded_len); + if (ret) + goto out; + break; + case ENCODED_IOV_COMPRESSION_LZO: + ret = decompress_lzo(encoded_data, encoded_len, + unencoded_data, unencoded_len); + if (ret) + goto out; + break; + case ENCODED_IOV_COMPRESSION_ZSTD: + ret = decompress_zstd(rctx, encoded_data, encoded_len, + unencoded_data, unencoded_len); + if (ret) + goto out; + break; + } + + pos = unencoded_offset; + while (pos < unencoded_file_len) { + w = pwrite(rctx->write_fd, unencoded_data + pos, + unencoded_file_len - pos, unencoded_offset + pos); + if (w < 0) { + ret = -errno; + error("writing unencoded data failed: %m"); + goto out; + } + pos += w; + } +out: + free(unencoded_data); + return ret; +} + static int process_encoded_write(const char *path, const void *data, u64 offset, - u64 len, u64 unencoded_file_len, u64 unencoded_len, - u64 unencoded_offset, u32 compression, u32 encryption, void *user) + u64 len, u64 unencoded_file_len, + u64 unencoded_len, u64 unencoded_offset, + u32 compression, u32 encryption, void *user) { int ret; struct btrfs_receive *rctx = user; @@ -1091,6 +1302,14 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, { &encoded, sizeof(encoded) }, { (char *)data, len } }; + bool encoded_write = !rctx->force_decompress; + bool decompress = rctx->force_decompress; + + if (encryption) { + error("encoded_write: encryption not supported\n"); + ret = -EOPNOTSUPP; + goto out; + } ret = path_cat_out(full_path, rctx->full_subvol_path, path); if (ret < 0) { @@ -1102,15 +1321,37 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, if (ret < 0) goto out; - /* - * NOTE: encoded writes guarantee no partial writes, - * so we don't need to handle that possibility. - */ - ret = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); - if (ret < 0) { - ret = -errno; - error("encoded_write: writing to %s failed: %m", path); + if (encoded_write) { + /* + * NOTE: encoded writes guarantee no partial writes, + * so we don't need to handle that possibility. + */ + ret = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); + if (ret < 0) { + /* + * error conditions where fallback to manual decompress + * and write make sense. + */ + if (errno == ENOSPC || + errno == EOPNOTSUPP || + errno == EINVAL) + decompress = true; + else { + ret = -errno; + error("encoded_write: writing to %s failed: %m", path); + goto out; + } + } } + + if (decompress) { + ret = decompress_and_write(data, len, unencoded_file_len, + unencoded_len, unencoded_offset, + compression, user); + if (ret < 0) + goto out; + } + ret = 0; out: return ret; } @@ -1300,6 +1541,12 @@ out: close(rctx->dest_dir_fd); rctx->dest_dir_fd = -1; } + if (rctx->zstd_dstream) + ZSTD_freeDStream(rctx->zstd_dstream); + if (rctx->zlib_stream) { + inflateEnd(rctx->zlib_stream); + free(rctx->zlib_stream); + } return ret; } @@ -1373,12 +1620,13 @@ static int cmd_receive(const struct cmd_struct *cmd, int argc, char **argv) optind = 0; while (1) { int c; - enum { GETOPT_VAL_DUMP = 257 }; + enum { GETOPT_VAL_DUMP = 257, GETOPT_VAL_FORCE_DECOMPRESS }; static const struct option long_opts[] = { { "max-errors", required_argument, NULL, 'E' }, { "chroot", no_argument, NULL, 'C' }, { "dump", no_argument, NULL, GETOPT_VAL_DUMP }, { "quiet", no_argument, NULL, 'q' }, + { "force-decompress", no_argument, NULL, GETOPT_VAL_FORCE_DECOMPRESS }, { NULL, 0, NULL, 0 } }; @@ -1421,6 +1669,9 @@ static int cmd_receive(const struct cmd_struct *cmd, int argc, char **argv) case GETOPT_VAL_DUMP: dump = 1; break; + case GETOPT_VAL_FORCE_DECOMPRESS: + rctx.force_decompress = 1; + break; default: usage_unknown_option(cmd, argv); } From patchwork Fri Aug 21 07:40:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728519 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5277E1392 for ; Fri, 21 Aug 2020 07:41:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3BE0E207DF for ; Fri, 21 Aug 2020 07:41:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="qEZIapEt" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728451AbgHUHlW (ORCPT ); Fri, 21 Aug 2020 03:41:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728391AbgHUHkw (ORCPT ); Fri, 21 Aug 2020 03:40:52 -0400 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E7A59C06134A for ; Fri, 21 Aug 2020 00:40:51 -0700 (PDT) Received: by mail-pl1-x644.google.com with SMTP id f5so510022plr.9 for ; Fri, 21 Aug 2020 00:40:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=imB0DQxAT8UunMGY6iRGaYrAjtodauS9bVt6p7lUxh8=; b=qEZIapEtUSXCOYAjJh9pfNtL2V5UP/ULp/F1qG8YiJdY62JXcBc9vIBxI4tSFJEh5I 4nKw60kl556qYa3kE+L3GNho9GVg3XmPF8PcVBfFoiymS5pjI8GFPfyQ7nzQdoUsPxah KdC2WKm957IO1O+KzD0PGnuf2jSvuk2TLRQlRFgo/167LLjp4DQZQlImVZK38o4xpd9z zDWsQiNnxdpgO0qijN51OJ/a++/HzQrZkZAvzoAgWlRuZFxoVRPgu/xzWQZ9bdTljC7S zTE/g6dAEGg8sxc2uKSye0vN+i9vyrw2yb9CjB2Sv8mdNfrHX3TY1OrxdgthFlF5F861 dVIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=imB0DQxAT8UunMGY6iRGaYrAjtodauS9bVt6p7lUxh8=; b=nK/oyqshGKbaiO9yGX3SkaShCWiKBhST9Hp+08/zBBgPF/VQKrMncjfzlVvfE27vZ/ qLHR5tOVtq+RSpt9RDkDfXDMiYlZ0Rb9xjHBtOF/vLVlFp5i4JUzu/ecizjE8g0+YViR CCJlEPfBFypfAUAkr5HH3IL7KPUO21Yq0EPv295BI9foNSJJGjaFCDdhNjsPzoW43eok SPvJDQtp1/wdUxIQoIAvUssvidjePW1VGo+b6/AA52Wpjk5BRlV4M3JJF5MZRYxWbpOI yaPvQR8mzLQRCQq5MObjNgxncwgH9Tq8acj4LidyUT3Nsm3EJA+lcbT/n7/rAJIV2EHe v0KA== X-Gm-Message-State: AOAM531mMZpHDKqWqC2J++4xasq7Zg8BfxUBDHL6axbkdNwPkLkFeifY exj20R0TtpGxEvsEph9AjS26lg== X-Google-Smtp-Source: ABdhPJyc/afzXaSJriTeoXao7RoNflEWnzbYOs3q8ATN+UmcRjuyXEnIaiZrqdNCqgL+nyNV36CRCA== X-Received: by 2002:a17:90b:18b:: with SMTP id t11mr1422021pjs.105.1597995651357; Fri, 21 Aug 2020 00:40:51 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:49 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 08/11] btrfs-progs: receive: process fallocate commands Date: Fri, 21 Aug 2020 00:40:07 -0700 Message-Id: <1957f12abea48a115fb27e62f7c7fc93976e80d3.1597994354.git.osandov@osandov.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Boris Burkov Send stream v2 can emit fallocate commands, so receive must support them as well. The implementation simply passes along the arguments to the syscall. Note that mode is encoded as a u32 in send stream but fallocate takes an int, so there is a unsigned->signed conversion there. Signed-off-by: Boris Burkov --- cmds/receive-dump.c | 9 +++++++++ cmds/receive.c | 26 ++++++++++++++++++++++++++ common/send-stream.c | 9 +++++++++ common/send-stream.h | 2 ++ 4 files changed, 46 insertions(+) diff --git a/cmds/receive-dump.c b/cmds/receive-dump.c index 20ec2b70..acc0ba32 100644 --- a/cmds/receive-dump.c +++ b/cmds/receive-dump.c @@ -329,6 +329,14 @@ static int print_encoded_write(const char *path, const void *data, u64 offset, unencoded_offset, compression, encryption); } +static int print_fallocate(const char *path, int mode, u64 offset, u64 len, + void *user) +{ + return PRINT_DUMP(user, path, "fallocate", + "mode=%d offset=%llu len=%llu", + mode, offset, len); +} + struct btrfs_send_ops btrfs_print_send_ops = { .subvol = print_subvol, .snapshot = print_snapshot, @@ -352,4 +360,5 @@ struct btrfs_send_ops btrfs_print_send_ops = { .utimes = print_utimes, .update_extent = print_update_extent, .encoded_write = print_encoded_write, + .fallocate = print_fallocate, }; diff --git a/cmds/receive.c b/cmds/receive.c index c67d4653..5c0930cb 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -1356,6 +1356,31 @@ out: return ret; } +static int process_fallocate(const char *path, int mode, u64 offset, u64 len, + void *user) +{ + int ret; + struct btrfs_receive *rctx = user; + char full_path[PATH_MAX]; + + ret = path_cat_out(full_path, rctx->full_subvol_path, path); + if (ret < 0) { + error("fallocate: path invalid: %s", path); + goto out; + } + ret = open_inode_for_write(rctx, full_path, false); + if (ret < 0) + goto out; + ret = fallocate(rctx->write_fd, mode, offset, len); + if (ret < 0) { + ret = -errno; + error("fallocate: fallocate on %s failed: %m", path); + goto out; + } +out: + return ret; +} + static struct btrfs_send_ops send_ops = { .subvol = process_subvol, .snapshot = process_snapshot, @@ -1379,6 +1404,7 @@ static struct btrfs_send_ops send_ops = { .utimes = process_utimes, .update_extent = process_update_extent, .encoded_write = process_encoded_write, + .fallocate = process_fallocate, }; static int do_receive(struct btrfs_receive *rctx, const char *tomnt, diff --git a/common/send-stream.c b/common/send-stream.c index 1376e00b..d455cdfb 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -369,6 +369,7 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) u64 unencoded_offset; int len; int xattr_len; + int fallocate_mode; ret = read_cmd(sctx); if (ret) @@ -514,6 +515,14 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) case BTRFS_SEND_C_END: ret = 1; break; + case BTRFS_SEND_C_FALLOCATE: + TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); + TLV_GET_U32(sctx, BTRFS_SEND_A_FALLOCATE_MODE, &fallocate_mode); + TLV_GET_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, &offset); + TLV_GET_U64(sctx, BTRFS_SEND_A_SIZE, &tmp); + ret = sctx->ops->fallocate(path, fallocate_mode, offset, tmp, + sctx->user); + break; } tlv_get_failed: diff --git a/common/send-stream.h b/common/send-stream.h index 607bc007..a58739bb 100644 --- a/common/send-stream.h +++ b/common/send-stream.h @@ -70,6 +70,8 @@ struct btrfs_send_ops { u64 len, u64 unencoded_file_len, u64 unencoded_len, u64 unencoded_offset, u32 compression, u32 encryption, void *user); + int (*fallocate)(const char *path, int mode, u64 offset, u64 len, + void *user); }; int btrfs_read_and_process_send_stream(int fd, From patchwork Fri Aug 21 07:39:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728489 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3E7931392 for ; Fri, 21 Aug 2020 07:41:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 26C7C208DB for ; Fri, 21 Aug 2020 07:41:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="FfIoIiJ2" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728324AbgHUHk7 (ORCPT ); Fri, 21 Aug 2020 03:40:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728107AbgHUHkf (ORCPT ); Fri, 21 Aug 2020 03:40:35 -0400 Received: from mail-pj1-x1043.google.com (mail-pj1-x1043.google.com [IPv6:2607:f8b0:4864:20::1043]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B0D2C061387 for ; Fri, 21 Aug 2020 00:40:35 -0700 (PDT) Received: by mail-pj1-x1043.google.com with SMTP id kr4so466966pjb.2 for ; Fri, 21 Aug 2020 00:40:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eHIY7+yv9iDvme5i3eTWSs94I5h2qd9+JUn/jl34owY=; b=FfIoIiJ2nK41QK2HVGnMsZpckArAX6hK2ETKBNhWm5hoyst/Q7RMTNJHh2WfQ63bcd S+6+0MxY+QR62LAO4H71HSNbmdNU8xJQeTBkhfMtO0sjB4859KDP4ydCpXytu9tEX+uR KlHutbcts1KwOzZ9Akiw/tCnxx0dN6DRNNrHXA+VHMKMGPph8T4mq54XeTGqM3Au8Hyj T44tD0X4TUpDiYJq1XlKWPFwqKNIqtxrhAc7mUCGcF6hQa1RTpRYxeZoJ5aJcRHN2dod heL2bKJq/0QcYONzF46wke951RPk9Y4NqjxvXVeHWoAkScrVXRHJjtgRTxNahr1x0pf0 0GGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eHIY7+yv9iDvme5i3eTWSs94I5h2qd9+JUn/jl34owY=; b=aVw+gQxEzpTlMTQQMc2HmAnKl/sRr45HAmes4ZBmwEBQQL13fAMt2gb26Kf/4XWNM6 FqQ4t91iRunoql1waLoso516O0wLFYO0oFyRAeXmH2+4/cyw0puM3KOecZkG1WS38r4x zo0y73Jro+1B5v3gC6midwMvvlgFQnkDQdSsl7g4lpIXtVJtkndwuVvfDm4t7wuUV9fg CGTBlsLmpTaCGIMeisMwZsuJO7qxUleS6YoA/0a/uSv48oIF3DtcKlizveYyAC+1KInu 3t0cdwxDpvSZUqt4KluuFxWaChz9EyvL+m0VQHzFzGCL2NWAhm5/zTX/ZExyN/Tr1fWS Qe4g== X-Gm-Message-State: AOAM531p7IMUBd9vPVwdrOfTXY+hlhOKpU6hkp7JKlnv15jC8foFrwKL lHVJLS23Amb7CBuOGAqu7Dt84w== X-Google-Smtp-Source: ABdhPJztre1VZSD47fIOEgc25Ha4Tb12UKVQQ6BZY0AZui9Q3RIoZ2N+y9OZyrxJj4d/4T5SBB64PQ== X-Received: by 2002:a17:902:c286:: with SMTP id i6mr1373034pld.63.1597995635098; Fri, 21 Aug 2020 00:40:35 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:32 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 9/9] btrfs: send: enable support for stream v2 and compressed writes Date: Fri, 21 Aug 2020 00:39:59 -0700 Message-Id: X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Omar Sandoval Now that the new support is implemented, allow the ioctl to accept the flags and update the version in sysfs. Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 10 +++++++++- fs/btrfs/send.h | 2 +- include/uapi/linux/btrfs.h | 4 +++- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index df6882b3ab2b..e87dea7bd915 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -670,7 +670,10 @@ static int send_header(struct send_ctx *sctx) struct btrfs_stream_header hdr; strcpy(hdr.magic, BTRFS_SEND_STREAM_MAGIC); - hdr.version = cpu_to_le32(BTRFS_SEND_STREAM_VERSION); + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) + hdr.version = cpu_to_le32(2); + else + hdr.version = cpu_to_le32(1); return write_buf(sctx->send_filp, &hdr, sizeof(hdr), &sctx->send_off); @@ -7315,6 +7318,11 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) ret = -EINVAL; goto out; } + if ((arg->flags & BTRFS_SEND_FLAG_COMPRESSED) && + !(arg->flags & BTRFS_SEND_FLAG_STREAM_V2)) { + ret = -EINVAL; + goto out; + } sctx = kzalloc(sizeof(struct send_ctx), GFP_KERNEL); if (!sctx) { diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index 9f4f7b96b1eb..9c83e14a43b2 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -10,7 +10,7 @@ #include "ctree.h" #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" -#define BTRFS_SEND_STREAM_VERSION 1 +#define BTRFS_SEND_STREAM_VERSION 2 /* * In send stream v1, no command is larger than 64k. In send stream v2, no limit diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h index 51e69f28d22d..6f29c456e4d7 100644 --- a/include/uapi/linux/btrfs.h +++ b/include/uapi/linux/btrfs.h @@ -785,7 +785,9 @@ struct btrfs_ioctl_received_subvol_args { #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ From patchwork Fri Aug 21 07:40:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728531 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9A004739 for ; Fri, 21 Aug 2020 07:41:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7290D2078D for ; Fri, 21 Aug 2020 07:41:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="eUl2gyDC" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728400AbgHUHlm (ORCPT ); Fri, 21 Aug 2020 03:41:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728398AbgHUHk4 (ORCPT ); Fri, 21 Aug 2020 03:40:56 -0400 Received: from mail-pl1-x642.google.com (mail-pl1-x642.google.com [IPv6:2607:f8b0:4864:20::642]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D154EC061350 for ; Fri, 21 Aug 2020 00:40:55 -0700 (PDT) Received: by mail-pl1-x642.google.com with SMTP id g15so515338plj.6 for ; Fri, 21 Aug 2020 00:40:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yvoSycF4wGFmZEGYefbZtpG0hIG/VlSxdUHeeNSYgs0=; b=eUl2gyDCmKzsToDNfWl5aI9WA5NN6pL3sCIY+3KLlYrhjm1f9ukiy1AlZYTxLKl4NL asfxVR0yHk5HohAHb65P1vNM4IJWtebZAUzQQu8nHBFrhl3ZSROs8nL52Hv1Me48GDEP PrpN445otWPkYgg3gKGnUjPuoMuX+PZHN6Ystz2pgWubOuIdCbFsZIbLxj+YUT1krvtg 81wzD9rg3xfsFUOeioDf/LmXJfwoeuHIhSChKDDcwjw3o//gCigedIK4P1WUIs0uVeU1 xSy3w71XHvH305imyHqkS6xaenQ2N9olR1jdQp1KWUSOn55StedbfPgVquxqUR8ny7La WeGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yvoSycF4wGFmZEGYefbZtpG0hIG/VlSxdUHeeNSYgs0=; b=VUcqx0Gjb3nl+m/OjTSyuhfAXBzuJu8ira/Wb6ZMRgmh0K5QMItfQEk+lMOKRRoDIM 9KcMhNrqKAKomMPyUd9lDcfKVJ5Z3BYiur18AWXIj3eRfAGqUHovdKMqhvDFZGgAK05z StT0JLBTc12LSnaWq+HmzD9ShNLvvJoI1mVTPRn9DsKB5rM/Hd+hQlLnMFrPiqKX9e/E WBIyy2wmfzBWEQoE3CDlvFSaigZ9hBZw4OvY9ASMrqVPGWTntHOP11WZKOP8Fm3qug+g VqihE7PhKmjL1JxlgRinHd/Oo83+afaiBRiaEru+9+Ru8GMpbEBo7Ow+IFPTMGz3uK8F UyOw== X-Gm-Message-State: AOAM533aU/neuTcBo0a7yPcI4eQVQGeLXuvn9JWv68i70LyruZ+OBBIP Zn8wOqkDL03t/DCoc8MPdO1Di1wU1d3d3g== X-Google-Smtp-Source: ABdhPJwHgF0DGJDsQ1TwGS7d7/rnUydxJiwSq4qInbfmmIxhUcFpFQy3UgHN33CeMdXWkQgr6cGylg== X-Received: by 2002:a17:90a:17ab:: with SMTP id q40mr1530279pja.28.1597995655218; Fri, 21 Aug 2020 00:40:55 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:54 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 10/11] btrfs-progs: send: stream v2 ioctl flags Date: Fri, 21 Aug 2020 00:40:09 -0700 Message-Id: X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Boris Burkov To make the btrfs send ioctl use the stream v2 format requires passing BTRFS_SEND_FLAG_STREAM_V2 in flags. Further, to cause the ioctl to emit encoded_write commands for encoded extents, we must set that flag as well as BTRFS_SEND_FLAG_COMPRESSED. Finally, we bump up the version in send.h as well, since we are now fully compatible with v2. Add two command line arguments to btrfs send: --stream-version and --compressed. --stream-version requires an argument which it parses as an integer and sets STREAM_V2 if the argument is 2. --compressed does not require an argument and automatically implies STREAM_V2 as well (COMPRESSED alone causes the ioctl to error out). Some examples to illustrate edge cases: // v1, old format and no encoded_writes btrfs send subvol btrfs send --stream-version 1 subvol // v2 and compressed, we will see encoded_writes btrfs send --compressed subvol btrfs send --compressed --stream-version 2 subvol // v2 only, new format but no encoded_writes btrfs send --stream-version 2 subvol // error: compressed needs version >= 2 btrfs send --compressed --stream-version 1 subvol // error: invalid version (not 1 or 2) btrfs send --stream-version 3 subvol btrfs send --compressed --stream-version 0 subvol btrfs send --compressed --stream-version 10 subvol Signed-off-by: Boris Burkov --- cmds/send.c | 39 +++++++++++++++++++++++++++++++++++++-- ioctl.h | 17 ++++++++++++++++- libbtrfsutil/btrfs.h | 17 ++++++++++++++++- send.h | 2 +- 4 files changed, 70 insertions(+), 5 deletions(-) diff --git a/cmds/send.c b/cmds/send.c index b8e3ba12..4c4eaa84 100644 --- a/cmds/send.c +++ b/cmds/send.c @@ -474,6 +474,7 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) int full_send = 1; int new_end_cmd_semantic = 0; u64 send_flags = 0; + long stream_version = 0; memset(&send, 0, sizeof(send)); send.dump_fd = fileno(stdout); @@ -492,11 +493,17 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) optind = 0; while (1) { - enum { GETOPT_VAL_SEND_NO_DATA = 256 }; + enum { + GETOPT_VAL_SEND_NO_DATA = 256, + GETOPT_VAL_SEND_STREAM_V2, + GETOPT_VAL_SEND_COMPRESSED + }; static const struct option long_options[] = { { "verbose", no_argument, NULL, 'v' }, { "quiet", no_argument, NULL, 'q' }, - { "no-data", no_argument, NULL, GETOPT_VAL_SEND_NO_DATA } + { "no-data", no_argument, NULL, GETOPT_VAL_SEND_NO_DATA }, + { "stream-version", required_argument, NULL, GETOPT_VAL_SEND_STREAM_V2 }, + { "compressed", no_argument, NULL, GETOPT_VAL_SEND_COMPRESSED } }; int c = getopt_long(argc, argv, "vqec:f:i:p:", long_options, NULL); @@ -584,10 +591,38 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) case GETOPT_VAL_SEND_NO_DATA: send_flags |= BTRFS_SEND_FLAG_NO_FILE_DATA; break; + case GETOPT_VAL_SEND_STREAM_V2: + stream_version = strtol(optarg, NULL, 10); + if (stream_version < 1 || stream_version > 2) { + ret = 1; + error("invalid --stream-version. valid values: {1, 2}"); + goto out; + } + if (stream_version == 2) + send_flags |= BTRFS_SEND_FLAG_STREAM_V2; + break; + case GETOPT_VAL_SEND_COMPRESSED: + send_flags |= BTRFS_SEND_FLAG_COMPRESSED; + /* + * We want to default to stream v2 if only compressed is + * set. If stream_version is explicitly set to 0, that + * will trigger its own error condition for being an + * invalid version. + */ + if (stream_version == 0) { + stream_version = 2; + send_flags |= BTRFS_SEND_FLAG_STREAM_V2; + } + break; default: usage_unknown_option(cmd, argv); } } + if (stream_version < 2 && (send_flags & BTRFS_SEND_FLAG_COMPRESSED)) { + ret = 1; + error("--compressed requires --stream-version >= 2"); + goto out; + } if (check_argc_min(argc - optind, 1)) return 1; diff --git a/ioctl.h b/ioctl.h index ade6dcb9..46de8ac8 100644 --- a/ioctl.h +++ b/ioctl.h @@ -653,10 +653,25 @@ BUILD_ASSERT(sizeof(struct btrfs_ioctl_received_subvol_args_32) == 192); */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ diff --git a/libbtrfsutil/btrfs.h b/libbtrfsutil/btrfs.h index 60d51ff6..8430a40d 100644 --- a/libbtrfsutil/btrfs.h +++ b/libbtrfsutil/btrfs.h @@ -731,10 +731,25 @@ struct btrfs_ioctl_received_subvol_args { */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ diff --git a/send.h b/send.h index 3c47e0c7..fac90588 100644 --- a/send.h +++ b/send.h @@ -31,7 +31,7 @@ extern "C" { #endif #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" -#define BTRFS_SEND_STREAM_VERSION 1 +#define BTRFS_SEND_STREAM_VERSION 2 #define BTRFS_SEND_BUF_SIZE_V1 SZ_64K #define BTRFS_SEND_READ_SIZE (1024 * 48) From patchwork Fri Aug 21 07:40:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728525 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CA7771392 for ; Fri, 21 Aug 2020 07:41:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B2B12207DF for ; Fri, 21 Aug 2020 07:41:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="lEgZZeZz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728464AbgHUHlg (ORCPT ); Fri, 21 Aug 2020 03:41:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37336 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728407AbgHUHk6 (ORCPT ); Fri, 21 Aug 2020 03:40:58 -0400 Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D80BAC061344 for ; Fri, 21 Aug 2020 00:40:57 -0700 (PDT) Received: by mail-pf1-x432.google.com with SMTP id y206so657088pfb.10 for ; Fri, 21 Aug 2020 00:40:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=83qY1yiGV6Av8ygbOO41/2N//qJALrswYkrshLQjEaI=; b=lEgZZeZz3dHdOoJtt5d8VNuVhl8oh1GerNwL6riHZgnlCETtA26dy9skXtG8r4+LZd DoASVZ0kDL/fuv4jLYePgszynxrI+WQs5E0FTiWaU4D6mQvcGDszmyowaArlV+h11YVN 3hjp7/cObCFoR+6rs+/fCl8S6fL85bTSWEVF+P32lm2sd11oyQo7nAqx0CNNuZx0noOa YR4VEfJ1aic9wQJxsaWGUF8EZbBdgpDhGgqbImoHam6O0uu0lxCi62UG9tHVwFofrnnf H8D/gorWe6W2zjrEcgLuxKrOSXbS01dIZCJT7QXLCu4jZKnE5bl2pFVJ/6GXpEdOGXCa xSZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=83qY1yiGV6Av8ygbOO41/2N//qJALrswYkrshLQjEaI=; b=LkAE6vEV9Ayhh0BxRfw4S47O6l1xwXhG2tLBCeAOiYwOXWhvKlVxC1xTKPi1StS5X/ Aezeb0HLyzzPoTJmjXYAW0nsq2h81O+QVZQjKSqm1pZc/O8sywg7TQ6MRgfATS+rd7Jc BlOM58OsDWLwJvTTnWUGPoPmJJhq0oRXYfYgOtkPsfbvh7Ibklr6h+m9zf2MnvNo+VwW Nj+Ac8+dWW1tsUXvuhGICV69ectp14zMVsnYndixb5PxutqGbT7U4GxHkzXERHPvQtu1 fEfy4Ix87/pSMllaB+H28WS8FuOI97+Gn48HTGC8ZS3rfBpqeaxjyspKCLfuL7Zi5IwK pZqA== X-Gm-Message-State: AOAM533YoUSg21Nc1oDX5hklQYu907qPK+zxOjVWEL+7j7iPS4xmw3qg 6dn1DRcgUHUhHB2sUBvotGr2MQ== X-Google-Smtp-Source: ABdhPJwqhfIDFP068+V2W1ChVQO5SsJKLf5jPoSErAlzXLd7vD0uTnfmOQfhNU/zO9ptzsOqTDGIyA== X-Received: by 2002:a63:fe0a:: with SMTP id p10mr1391421pgh.255.1597995657124; Fri, 21 Aug 2020 00:40:57 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:55 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 11/11] btrfs-progs: receive: add tests for basic encoded_write send/receive Date: Fri, 21 Aug 2020 00:40:10 -0700 Message-Id: X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Boris Burkov Adapt the existing send/receive tests by passing '-o --force-compress' to the mount commands in a new test. After writing a few files in the various compression formats, send/receive them with and without --force-decompress to test both the encoded_write path and the fallback to decode+write. Signed-off-by: Boris Burkov --- .../040-receive-write-encoded/test.sh | 114 ++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100755 tests/misc-tests/040-receive-write-encoded/test.sh diff --git a/tests/misc-tests/040-receive-write-encoded/test.sh b/tests/misc-tests/040-receive-write-encoded/test.sh new file mode 100755 index 00000000..4df6ccd6 --- /dev/null +++ b/tests/misc-tests/040-receive-write-encoded/test.sh @@ -0,0 +1,114 @@ +#!/bin/bash +# +# test that we can send and receive encoded writes for three modes of +# transparent compression: zlib, lzo, and zstd. + +source "$TEST_TOP/common" + +check_prereq mkfs.btrfs +check_prereq btrfs + +setup_root_helper +prepare_test_dev + +here=`pwd` + +# assumes the filesystem exists, and does mount, write, snapshot, send, unmount +# for the specified encoding option +send_one() { + local str + local subv + local snap + + algorithm="$1" + shift + str="$1" + shift + + subv="subv-$algorithm" + snap="snap-$algorithm" + + run_check_mount_test_dev "-o" "compress-force=$algorithm" + cd "$TEST_MNT" || _fail "cannot chdir to TEST_MNT" + + run_check $SUDO_HELPER "$TOP/btrfs" subvolume create "$subv" + run_check $SUDO_HELPER dd if=/dev/zero of="$subv/file1" bs=1M count=1 + run_check $SUDO_HELPER dd if=/dev/zero of="$subv/file2" bs=500K count=1 + run_check $SUDO_HELPER "$TOP/btrfs" subvolume snapshot -r "$subv" "$snap" + run_check $SUDO_HELPER "$TOP/btrfs" send -f "$str" "$snap" "$@" + + cd "$here" || _fail "cannot chdir back to test directory" + run_check_umount_test_dev +} + +receive_one() { + local str + str="$1" + shift + + run_check_mkfs_test_dev + run_check_mount_test_dev + run_check $SUDO_HELPER "$TOP/btrfs" receive "$@" -v -f "$str" "$TEST_MNT" + run_check_umount_test_dev + run_check rm -f -- "$str" +} + +test_one_write_encoded() { + local str + local algorithm + algorithm="$1" + shift + str="$here/stream-$algorithm.stream" + + run_check_mkfs_test_dev + send_one "$algorithm" "$str" --compressed + receive_one "$str" "$@" +} + +test_one_stream_v1() { + local str + local algorithm + algorithm="$1" + shift + str="$here/stream-$algorithm.stream" + + run_check_mkfs_test_dev + send_one "$algorithm" "$str" --stream-version 1 + receive_one "$str" "$@" +} + +test_mix_write_encoded() { + local strzlib + local strlzo + local strzstd + strzlib="$here/stream-zlib.stream" + strlzo="$here/stream-lzo.stream" + strzstd="$here/stream-zstd.stream" + + run_check_mkfs_test_dev + + send_one "zlib" "$strzlib" --compressed + send_one "lzo" "$strlzo" --compressed + send_one "zstd" "$strzstd" --compressed + + receive_one "$strzlib" + receive_one "$strlzo" + receive_one "$strzstd" +} + +test_one_write_encoded "zlib" +test_one_write_encoded "lzo" +test_one_write_encoded "zstd" + +# with decompression forced +test_one_write_encoded "zlib" "--force-decompress" +test_one_write_encoded "lzo" "--force-decompress" +test_one_write_encoded "zstd" "--force-decompress" + +# send stream v1 +test_one_stream_v1 "zlib" +test_one_stream_v1 "lzo" +test_one_stream_v1 "zstd" + +# files use a mix of compression algorithms +test_mix_write_encoded