From patchwork Wed Nov 18 19:18:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39FE6C56201 for ; Wed, 18 Nov 2020 19:19:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C538822227 for ; Wed, 18 Nov 2020 19:19:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="TvyNk7Qz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727253AbgKRTTK (ORCPT ); Wed, 18 Nov 2020 14:19:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727208AbgKRTTJ (ORCPT ); Wed, 18 Nov 2020 14:19:09 -0500 Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 26C38C0613D4 for ; Wed, 18 Nov 2020 11:19:09 -0800 (PST) Received: by mail-pf1-x442.google.com with SMTP id i8so2029828pfk.10 for ; Wed, 18 Nov 2020 11:19:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xPW+Dna2wRVVrHRfE5h6s7X411Swfso/95bBBe/E3W4=; b=TvyNk7QzgpB5yJSoGrRZyBM+BmM6MmaPBeXY299u5a8Mh7WaxRNh2OGWJ3Tv7G9UHR fNUkvb3o5FbhTBrcRzcu2vAolfe8OTgh+wxhYHppJyE8DE97mM5bUxTUca0W+ehVYkV4 yRSJP+qzJnkaYY/x7EKcuYeEu4HatNT1EyNY1PaF5gz/T0iy22vkiRUYkaNtmILkge5t AixXUekNT00Kn24cj1AhQ4wKN96Ey4l9f1Hs1UZpS7li3JRNououXp4ZWGfPHDU453oB h+pzRJ2w7lGCYpV7E8CRuU8BCpyDQc3N2A295lWiqHro8Tcfr6ccIxgznNWDJnS1Ofgz cmnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xPW+Dna2wRVVrHRfE5h6s7X411Swfso/95bBBe/E3W4=; b=cWcF8EwnlOycJ13EiYqxq1cKzD2ZR5V8w3pibe5bZ8xZtQFG49paJlIMGSn6fdP3yX dVyC7AbbivKVNdKdIwdGtzRK0AvDvVw/V0/Vfe5UeHYlLYuuyNpyNAnN31zK7NdmfpA6 URpyd911D9JQGD+LcQ4ntf0T7r9CHy1H6nCpy/YirX2077n6ydmmCeB3r6w7kKRt6njN Gz6LSZDD9ROykhllCHmUGS27O3SRU0cfyS5+4t3CQvpWVnSsGErip5ewtrtodWUnYsf7 7QvrIgDl4pz8gjnJEtFL7xlu1Il+LgEpycTBtiv4ZrXS7sMU1g7bnMbs/Pk9zcqqjYlQ g3Qg== X-Gm-Message-State: AOAM533f5WCFZAz1oMzlhgbgRsCl7fpnGxqGd4+PhtSuzAQEbymDw6x1 oqqFfCJG39GqGR5SVy0wDRsNPx2sToSkSw== X-Google-Smtp-Source: ABdhPJwmZL45UiAH7bJUkojbwqQ/1KAcPRf9x0k4WefT4RTlv7Z6ssxk4zGch9SrbClsJ1FTsUgW2g== X-Received: by 2002:a63:3c10:: with SMTP id j16mr9524663pga.140.1605727148644; Wed, 18 Nov 2020 11:19:08 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:07 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 1/5] btrfs: add send stream v2 definitions Date: Wed, 18 Nov 2020 11:18:43 -0800 Message-Id: X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Omar Sandoval This adds the definitions of the new commands for send stream version 2 and their respective attributes: fallocate, FS_IOC_SETFLAGS (a.k.a. chattr), and encoded writes. It also documents two changes to the send stream format in v2: the receiver shouldn't assume a maximum command size, and the DATA attribute is encoded differently to allow for writes larger than 64k. These will be implemented in subsequent changes, and then the ioctl will accept the new flags. Reviewed-by: Josef Bacik Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 2 +- fs/btrfs/send.h | 30 +++++++++++++++++++++++++++++- include/uapi/linux/btrfs.h | 13 +++++++++++++ 3 files changed, 43 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index d719a2755a40..3cd090c3ffda 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -7255,7 +7255,7 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) sctx->clone_roots_cnt = arg->clone_sources_count; - sctx->send_max_size = BTRFS_SEND_BUF_SIZE; + sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); if (!sctx->send_buf) { ret = -ENOMEM; diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index de91488b7cd0..9f4f7b96b1eb 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -12,7 +12,11 @@ #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" #define BTRFS_SEND_STREAM_VERSION 1 -#define BTRFS_SEND_BUF_SIZE SZ_64K +/* + * In send stream v1, no command is larger than 64k. In send stream v2, no limit + * should be assumed. + */ +#define BTRFS_SEND_BUF_SIZE_V1 SZ_64K enum btrfs_tlv_type { BTRFS_TLV_U8, @@ -76,6 +80,13 @@ enum btrfs_send_cmd { BTRFS_SEND_C_END, BTRFS_SEND_C_UPDATE_EXTENT, + + /* The following commands were added in send stream v2. */ + + BTRFS_SEND_C_FALLOCATE, + BTRFS_SEND_C_SETFLAGS, + BTRFS_SEND_C_ENCODED_WRITE, + __BTRFS_SEND_C_MAX, }; #define BTRFS_SEND_C_MAX (__BTRFS_SEND_C_MAX - 1) @@ -106,6 +117,11 @@ enum { BTRFS_SEND_A_PATH_LINK, BTRFS_SEND_A_FILE_OFFSET, + /* + * In send stream v2, this attribute is special: it must be the last + * attribute in a command, its header contains only the type, and its + * length is implicitly the remaining length of the command. + */ BTRFS_SEND_A_DATA, BTRFS_SEND_A_CLONE_UUID, @@ -114,6 +130,18 @@ enum { BTRFS_SEND_A_CLONE_OFFSET, BTRFS_SEND_A_CLONE_LEN, + /* The following attributes were added in send stream v2. */ + + BTRFS_SEND_A_FALLOCATE_MODE, + + BTRFS_SEND_A_SETFLAGS_FLAGS, + + BTRFS_SEND_A_UNENCODED_FILE_LEN, + BTRFS_SEND_A_UNENCODED_LEN, + BTRFS_SEND_A_UNENCODED_OFFSET, + BTRFS_SEND_A_COMPRESSION, + BTRFS_SEND_A_ENCRYPTION, + __BTRFS_SEND_A_MAX, }; #define BTRFS_SEND_A_MAX (__BTRFS_SEND_A_MAX - 1) diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h index 2c39d15a2beb..51e69f28d22d 100644 --- a/include/uapi/linux/btrfs.h +++ b/include/uapi/linux/btrfs.h @@ -769,6 +769,19 @@ struct btrfs_ioctl_received_subvol_args { */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ From patchwork Wed Nov 18 19:18:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8A05C63798 for ; Wed, 18 Nov 2020 19:19:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5BB042222C for ; Wed, 18 Nov 2020 19:19:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="tSKDlvfh" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727278AbgKRTTR (ORCPT ); Wed, 18 Nov 2020 14:19:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727257AbgKRTTQ (ORCPT ); Wed, 18 Nov 2020 14:19:16 -0500 Received: from mail-pl1-x642.google.com (mail-pl1-x642.google.com [IPv6:2607:f8b0:4864:20::642]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 38CDDC0613D4 for ; Wed, 18 Nov 2020 11:19:16 -0800 (PST) Received: by mail-pl1-x642.google.com with SMTP id 5so1527695plj.8 for ; Wed, 18 Nov 2020 11:19:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=26N9VK0p5cJDz5jy+y4kzcNpExO8G7FO7M93wxXu0UU=; b=tSKDlvfhTgTzL5lokM5AI2dcEugmR/E9HDpNCOTivrWdxD/qz4lK0H37LL+0rK7ZIC FctY+zpUH3tLvRQcOuG3k8JbbMWl0D6Pgc+Epr4rKcHfE5nJ1KHxjpjelPt9Xm+F2o4v qa7fVlKEzWV9QVQexhqatvev+UC2CuLhYpG2NZ4VY6f00TPG7aJpGcX1GlQBc64X5abY uwaYFhnR2/A4Gn53cg2jt97U45Ckl4Cg4shzqx0h0cwNu0zDEEALT7NCGlTnHMWo0eJy 45ZgBDiE229Zg0PoWrjuVS/iopvJoSYUTy7zL936nnvi8v1TTeCkIi2jnJViEB5cCEj8 OdNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=26N9VK0p5cJDz5jy+y4kzcNpExO8G7FO7M93wxXu0UU=; b=Azu7moTF30boGWDMG64+kBtVqT4s4pN1c/l+ns/zZsfMJn0Q4l06Rr7ZZ2stPoYo3t irkhcKXN9atmSBt9IbbyEHu6iP95+2gVsdxy/f8es9vl5CjVNy/JircqE01dJI1m+v5U oSb5EH6hpNwE0oAe824laVvQWUUfXxUkl4iccJgFkvdUdBSgBPe6n8y2QNRqQKWrcMA+ BAt/B1ZAi4fWrtl+TO6I0jEmrVInsfQQoCHhvhHkeCs7UZkk4rrkhOOA4daZsBtJphLs S2dv7TgNyGR/ZhqSbiwbjVZ+HSlgU81mY2FU3GLBopnNiXcjqrlZ1bkZepVmfoozSLMi 9M0A== X-Gm-Message-State: AOAM531bm5UsLjjfWtcpbtaMEfKfRu/1MdPK2iqZdv5krU1i1YcS9KtC hX1psKPtxhXmYg/NetOL3eDrmA== X-Google-Smtp-Source: ABdhPJwOzZpCBLIh+2XGUvA4XyTM1PJb6t6LcUd1nFPlredd6XVqU325+opbjlvlVe5hyiElPyY1Iw== X-Received: by 2002:a17:902:eb42:b029:d6:ba60:ba41 with SMTP id i2-20020a170902eb42b02900d6ba60ba41mr5653407pli.0.1605727155762; Wed, 18 Nov 2020 11:19:15 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:14 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 2/5] btrfs: send: write larger chunks when using stream v2 Date: Wed, 18 Nov 2020 11:18:46 -0800 Message-Id: X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Omar Sandoval The length field of the send stream TLV header is 16 bits. This means that the maximum amount of data that can be sent for one write is 64k minus one. However, encoded writes must be able to send the maximum compressed extent (128k) in one command. To support this, send stream version 2 encodes the DATA attribute differently: it has no length field, and the length is implicitly up to the end of containing command (which has a 32-bit length field). Although this is necessary for encoded writes, normal writes can benefit from it, too. For v2, let's bump up the send buffer to the maximum compressed extent size plus 16k for the other metadata (144k total). Since this will most likely be vmalloc'd (and always will be after the next commit), we round it up to the next page since we might as well use the rest of the page on systems with >16k pages. Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 3cd090c3ffda..822458e6ffae 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -4918,14 +4918,27 @@ static inline u64 max_send_read_size(const struct send_ctx *sctx) static int put_data_header(struct send_ctx *sctx, u32 len) { - struct btrfs_tlv_header *hdr; + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) { + /* + * In v2, the data attribute header doesn't include a length; it + * is implicitly to the end of the command. + */ + if (sctx->send_max_size - sctx->send_size < 2 + len) + return -EOVERFLOW; + put_unaligned_le16(BTRFS_SEND_A_DATA, + sctx->send_buf + sctx->send_size); + sctx->send_size += 2; + } else { + struct btrfs_tlv_header *hdr; - if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len) - return -EOVERFLOW; - hdr = (struct btrfs_tlv_header *)(sctx->send_buf + sctx->send_size); - put_unaligned_le16(BTRFS_SEND_A_DATA, &hdr->tlv_type); - put_unaligned_le16(len, &hdr->tlv_len); - sctx->send_size += sizeof(*hdr); + if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len) + return -EOVERFLOW; + hdr = (struct btrfs_tlv_header *)(sctx->send_buf + + sctx->send_size); + put_unaligned_le16(BTRFS_SEND_A_DATA, &hdr->tlv_type); + put_unaligned_le16(len, &hdr->tlv_len); + sctx->send_size += sizeof(*hdr); + } return 0; } @@ -7255,7 +7268,12 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) sctx->clone_roots_cnt = arg->clone_sources_count; - sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) { + sctx->send_max_size = ALIGN(SZ_16K + BTRFS_MAX_COMPRESSED, + PAGE_SIZE); + } else { + sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; + } sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); if (!sctx->send_buf) { ret = -ENOMEM; From patchwork Wed Nov 18 19:18:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44FCBC64E69 for ; Wed, 18 Nov 2020 19:19:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DB9482222C for ; Wed, 18 Nov 2020 19:19:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="fMZQHOtx" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727288AbgKRTTT (ORCPT ); Wed, 18 Nov 2020 14:19:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727273AbgKRTTS (ORCPT ); Wed, 18 Nov 2020 14:19:18 -0500 Received: from mail-pg1-x543.google.com (mail-pg1-x543.google.com [IPv6:2607:f8b0:4864:20::543]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7D4B0C0613D4 for ; Wed, 18 Nov 2020 11:19:18 -0800 (PST) Received: by mail-pg1-x543.google.com with SMTP id t21so1870943pgl.3 for ; Wed, 18 Nov 2020 11:19:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/zsxWF6lVDz+7AA3ZWvsvy5Wo7+pjmYoohRaGzOoct0=; b=fMZQHOtxQhH+Wm6oQayF/Jis6x8CMcwPmGIwI8unHPGZdxEVNgU4wmCUbheZf7k+Qs tVHAWpCOqeqNRIUY+MwW4e2wg1vf+nLmG2elJL6T6gogP69u/KqwDN7HJYoUV3dKcXPk ZLeWJFBHkup16/cVKeUktQS+XbU0FguStreRV7/+pBwNMuoQW2RivtqCmakePUfS3mP9 omigH4NPnNBh9Ndb7T8MQXbTkTBTvNfsJxvAWl0+D343EQBGRsZymWMB5i8s2mvmvM12 nAJPJhNm9wTZoRkVAepPHbjTUklp+Byk4ossFZRVftpgNzqOyp2xzHNXh8SkxEHb5OvM BZng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/zsxWF6lVDz+7AA3ZWvsvy5Wo7+pjmYoohRaGzOoct0=; b=ngNubyGDN7UDfzDKwT3YYGqLDPhMhSFXMmmHDyp2eytYGtR5tzjsJRYhY/9TNOOrYt p1XdjxGbE6PxU2o8lIq5yIt+QoAeEAvPxNfBP7mqL0ZPXVYZ/cyy0cqia8/Jj2ylYt7Z J01JkxWqWKRusv70+Wx+2Q8GX+mkwjZIjC3/SnwM1/dxW33Xw/iDD3ktEEbOpIc+p+U6 RGo9AYndsYzNAORP/meo9kmHUI5Op5hsp3KS3plioO2cQQuEKRVQ+DAE7LvpheBccNNt CglZuELA0jZ0M7dXO22iMSMapbaud35xekOtWUKrzugAb5hH4dwHbppiXuwSRwLedJht 4ELg== X-Gm-Message-State: AOAM533OwDWQ27FeE2fPGYKqOTuUddHGwIPyVTdVUDhD3af/7wHtTGIY 6pKx5vTs4ZHlnL8mOeI7wudi3w== X-Google-Smtp-Source: ABdhPJzQACKgpEvQnGNJzbC9MgeDK/K2QmXPVpbgyKh4g6Zq+LKBixy7Ekh0P/zAGAtJVkLPSRDxFw== X-Received: by 2002:a17:90a:154a:: with SMTP id y10mr535452pja.6.1605727158044; Wed, 18 Nov 2020 11:19:18 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:17 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 03/13] btrfs-progs: receive: dynamically allocate sctx->read_buf Date: Wed, 18 Nov 2020 11:18:47 -0800 Message-Id: <0621eb3c44c2057b35ceb1ce5c1270a4c9c8edc5.1605723745.git.osandov@osandov.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Boris Burkov In send stream v2, write commands can now be an arbitrary size. For that reason, we can no longer allocate a fixed array in sctx for read_cmd. Instead, read_cmd dynamically allocates sctx->read_buf. To avoid needless reallocations, we reuse read_buf between read_cmd calls by also keeping track of the size of the allocated buffer in sctx->read_buf_sz. We do the first allocation of the old default size at the start of processing the stream, and we only reallocate if we encounter a command that needs a larger buffer. Signed-off-by: Boris Burkov --- common/send-stream.c | 55 ++++++++++++++++++++++++++++---------------- send.h | 2 +- 2 files changed, 36 insertions(+), 21 deletions(-) diff --git a/common/send-stream.c b/common/send-stream.c index 3bd21d3f..51a6a94a 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -36,10 +36,10 @@ struct btrfs_send_attribute { struct btrfs_send_stream { int fd; - char read_buf[BTRFS_SEND_BUF_SIZE]; + char *read_buf; + size_t read_buf_sz; int cmd; - struct btrfs_cmd_header *cmd_hdr; struct btrfs_send_attribute cmd_attrs[BTRFS_SEND_A_MAX + 1]; u32 version; @@ -111,11 +111,12 @@ static int read_cmd(struct btrfs_send_stream *sctx) u32 pos; u32 crc; u32 crc2; + struct btrfs_cmd_header *cmd_hdr; + size_t buf_len; memset(sctx->cmd_attrs, 0, sizeof(sctx->cmd_attrs)); - ASSERT(sizeof(*sctx->cmd_hdr) <= sizeof(sctx->read_buf)); - ret = read_buf(sctx, sctx->read_buf, sizeof(*sctx->cmd_hdr)); + ret = read_buf(sctx, sctx->read_buf, sizeof(*cmd_hdr)); if (ret < 0) goto out; if (ret) { @@ -124,18 +125,22 @@ static int read_cmd(struct btrfs_send_stream *sctx) goto out; } - sctx->cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf; - cmd = le16_to_cpu(sctx->cmd_hdr->cmd); - cmd_len = le32_to_cpu(sctx->cmd_hdr->len); - - if (cmd_len + sizeof(*sctx->cmd_hdr) >= sizeof(sctx->read_buf)) { - ret = -EINVAL; - error("command length %u too big for buffer %zu", - cmd_len, sizeof(sctx->read_buf)); - goto out; + cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf; + cmd_len = le32_to_cpu(cmd_hdr->len); + cmd = le16_to_cpu(cmd_hdr->cmd); + buf_len = sizeof(*cmd_hdr) + cmd_len; + if (sctx->read_buf_sz < buf_len) { + sctx->read_buf = realloc(sctx->read_buf, buf_len); + if (!sctx->read_buf) { + ret = -ENOMEM; + error("failed to reallocate read buffer for cmd"); + goto out; + } + sctx->read_buf_sz = buf_len; + /* We need to reset cmd_hdr after realloc of sctx->read_buf */ + cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf; } - - data = sctx->read_buf + sizeof(*sctx->cmd_hdr); + data = sctx->read_buf + sizeof(*cmd_hdr); ret = read_buf(sctx, data, cmd_len); if (ret < 0) goto out; @@ -145,11 +150,12 @@ static int read_cmd(struct btrfs_send_stream *sctx) goto out; } - crc = le32_to_cpu(sctx->cmd_hdr->crc); - sctx->cmd_hdr->crc = 0; + crc = le32_to_cpu(cmd_hdr->crc); + /* in send, crc is computed with header crc = 0, replicate that */ + cmd_hdr->crc = 0; crc2 = crc32c(0, (unsigned char*)sctx->read_buf, - sizeof(*sctx->cmd_hdr) + cmd_len); + sizeof(*cmd_hdr) + cmd_len); if (crc != crc2) { ret = -EINVAL; @@ -524,19 +530,28 @@ int btrfs_read_and_process_send_stream(int fd, goto out; } + sctx.read_buf = malloc(BTRFS_SEND_BUF_SIZE_V1); + if (!sctx.read_buf) { + ret = -ENOMEM; + error("unable to allocate send stream read buffer"); + goto out; + } + sctx.read_buf_sz = BTRFS_SEND_BUF_SIZE_V1; + while (1) { ret = read_and_process_cmd(&sctx); if (ret < 0) { last_err = ret; errors++; if (max_errors > 0 && errors >= max_errors) - goto out; + break; } else if (ret > 0) { if (!honor_end_cmd) ret = 0; - goto out; + break; } } + free(sctx.read_buf); out: if (last_err && !ret) diff --git a/send.h b/send.h index 8dd865ec..228928a0 100644 --- a/send.h +++ b/send.h @@ -33,7 +33,7 @@ extern "C" { #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" #define BTRFS_SEND_STREAM_VERSION 1 -#define BTRFS_SEND_BUF_SIZE SZ_64K +#define BTRFS_SEND_BUF_SIZE_V1 SZ_64K #define BTRFS_SEND_READ_SIZE (1024 * 48) enum btrfs_tlv_type { From patchwork Wed Nov 18 19:18:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3385DC64E7D for ; Wed, 18 Nov 2020 19:19:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E8E8322228 for ; Wed, 18 Nov 2020 19:19:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="HnwigP05" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727291AbgKRTTY (ORCPT ); Wed, 18 Nov 2020 14:19:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726416AbgKRTTY (ORCPT ); Wed, 18 Nov 2020 14:19:24 -0500 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C38FEC0613D4 for ; Wed, 18 Nov 2020 11:19:22 -0800 (PST) Received: by mail-pl1-x644.google.com with SMTP id l11so1547679plt.1 for ; Wed, 18 Nov 2020 11:19:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=P9Kdu1iOQxHZHPzLsKpiyc2lmM69laJRMX8/k+zeI9U=; b=HnwigP052OdRCU9TTvxKjsmCRiFnZCItgfljMLRQV+kUfZPqlxT76tlV5nK13Gn9ZV 8/Ab8Hh+MwMHIMPqCSjIYF+GsIRD5icuJTkNG9qX2f52JlPZo4VjO5oA7TrbT3EhX3Zr ioGBhNQsCOSJ9XDIf2jbz7arirspi7eDFG+j3aZtk+ybdQQxSANNvmlCnz2LByRnpBNV 1Z0cJPip3ujM2/hONn7eZKPW8wS8wpWkGdhDhhl8k5O0u/nMHSkVL3fKPqM+/spTJS2N Vek9l/Gd/s779eTHzJd01ifO7vS5nPgZOljuUb59vYd64tXsXBmsLENuFMzedrwkeIGp wSXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=P9Kdu1iOQxHZHPzLsKpiyc2lmM69laJRMX8/k+zeI9U=; b=FCno3Qcu3hemlc82qKjyhso+MUiDVHMP0EywC+twjkcfvp4phmNnc6SF8mG+R3WvIQ 4CzteUv2Z3fZUDkeeTasFLerQChheJqP3jtECVPhqD1X4Lt1p9gAChuUrUbQsgan1VDF DAtH+YxLuziRWsd47i7OwEvI2On4rA5KjuokPbuK/zdtC4jeyxLjsGp6ffC98FM5LYNL fNbQAtHscSAZqBBNl+sf9F7Rd2dkC/wwfJph7+I8JJl6hPEZGtV+aYFboIkLdIRq/V8l 1sQm15pHhzqx4e7IXUnW8klzGQsHdVEXzQXy+YEEbGhe2U90PJGkL4OVfdB7P6xUfyw8 5e6A== X-Gm-Message-State: AOAM533idxX2uXOGIUlev7ELcfrdpKXhJDywdsBFtqAQDAIstCNVzinJ SDupbZeJjtlk75EiubByNdWOaw== X-Google-Smtp-Source: ABdhPJynB/3z0rxiPUbMCU413edbSddUD8pbNkI6RYQQryN1QHsqK3UrdxKuCwPlg4Lqw15YCYw9GQ== X-Received: by 2002:a17:902:bb94:b029:d6:edb2:4f41 with SMTP id m20-20020a170902bb94b02900d6edb24f41mr5762832pls.3.1605727162314; Wed, 18 Nov 2020 11:19:22 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:21 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 04/13] btrfs-progs: receive: support v2 send stream DATA tlv format Date: Wed, 18 Nov 2020 11:18:49 -0800 Message-Id: <514c65bbb59958b3e7504cadc3c708a48771d39b.1605723745.git.osandov@osandov.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Boris Burkov The new format privileges the BTRFS_SEND_A_DATA attribute by guaranteeing it will always be the last attribute in any command that needs it, and by implicitly encoding the data length as the difference between the total command length in the command header and the sizes of the rest of the attributes (and of course the tlv_type identifying the DATA attribute). To parse the new stream, we must read the tlv_type and if it is not DATA, we proceed normally, but if it is DATA, we don't parse a tlv_len but simply compute the length. In addition, we add some bounds checking when parsing each chunk of data, as well as for the tlv_len itself. Signed-off-by: Boris Burkov --- common/send-stream.c | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-) diff --git a/common/send-stream.c b/common/send-stream.c index 51a6a94a..77d5cd04 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -165,28 +165,44 @@ static int read_cmd(struct btrfs_send_stream *sctx) pos = 0; while (pos < cmd_len) { - struct btrfs_tlv_header *tlv_hdr; u16 tlv_type; - u16 tlv_len; struct btrfs_send_attribute *send_attr; - tlv_hdr = (struct btrfs_tlv_header *)data; - tlv_type = le16_to_cpu(tlv_hdr->tlv_type); - tlv_len = le16_to_cpu(tlv_hdr->tlv_len); + if (cmd_len - pos < sizeof(__le16)) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } + tlv_type = le16_to_cpu(*(__le16 *)data); if (tlv_type == 0 || tlv_type > BTRFS_SEND_A_MAX) { - error("invalid tlv in cmd tlv_type = %hu, tlv_len = %hu", - tlv_type, tlv_len); + error("invalid tlv in cmd tlv_type = %hu", tlv_type); ret = -EINVAL; goto out; } send_attr = &sctx->cmd_attrs[tlv_type]; send_attr->tlv_type = tlv_type; - send_attr->tlv_len = tlv_len; - pos += sizeof(*tlv_hdr); - data += sizeof(*tlv_hdr); + pos += sizeof(tlv_type); + data += sizeof(tlv_type); + if (sctx->version == 2 && tlv_type == BTRFS_SEND_A_DATA) { + send_attr->tlv_len = cmd_len - pos; + } else { + if (cmd_len - pos < sizeof(__le16)) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } + send_attr->tlv_len = le16_to_cpu(*(__le16 *)data); + pos += sizeof(__le16); + data += sizeof(__le16); + } + if (cmd_len - pos < send_attr->tlv_len) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } send_attr->data = data; pos += send_attr->tlv_len; data += send_attr->tlv_len; From patchwork Wed Nov 18 19:18:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915707 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D49FC8300B for ; Wed, 18 Nov 2020 19:19:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0A4A32220B for ; Wed, 18 Nov 2020 19:19:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="XS7pdPT+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727321AbgKRTTc (ORCPT ); Wed, 18 Nov 2020 14:19:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727312AbgKRTTa (ORCPT ); Wed, 18 Nov 2020 14:19:30 -0500 Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E97BDC061A4D for ; Wed, 18 Nov 2020 11:19:28 -0800 (PST) Received: by mail-pf1-x442.google.com with SMTP id g7so2058979pfc.2 for ; Wed, 18 Nov 2020 11:19:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BlZlv11RxklRSJqWlFnwra0sdX85prYJ/jAmSAt0T7w=; b=XS7pdPT+/PSzyYEKRQkL6sC1Ujnc2RsKA/Bc8a5wng1Ofh5bFe3vfMzquf9OsTYxh4 R2BsjUX698ORk+wXLNQwCdl5MSvz7YK2xGF4+jrfyy07kErVMaHyhhwZBXCmnzlCpTKg U1J/jY0BL7S+6RRDe3XWgREr8brptvhZo0cVnFFRDx1nCejXVzzHdyWMbPQT33YHF4eI yyA1Vu3K4ydiZxwNR2WaVSi1ECCuFV6F23LHH6hS+6ET9z8LUaDaUoqAmyoam6RhvCCg rbvfQR5j7w+acWofcEz7E2Xvyje5mSV4dBpXr63WVtJNgvIKT0dvXEFEXb5e8HNh+kgz ELTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BlZlv11RxklRSJqWlFnwra0sdX85prYJ/jAmSAt0T7w=; b=O5pctkB9y8LKAQ9gQYxo3ttwsFm3IsU/+BXY7bM1UZV/PX24prWB5tZDQXkeIpkd54 S+CuqAgjJO7cwIfu0ir4H1znJB9G1NuUqXLZNvZkRA9gKvf5NiWm/lfU+dnav3rB4IlE CtRHi2zf7sa1vf0nWvSBRYj7x/Gs5nercJHvZIndeeHeGaMNFBQRhR+Rc7uYEnLj3Aev /DZhLSxaZP1DDnaZQxio3KcGGH/RgXjV1ZwIPUVifPaDHyWxDhEhjal1SMALmZj2dsKA uN9c4xjk+e8Y/Tc5bjA1lFrrj5GcJbF/LhrG4kJQ0ln592HmXZ/6IFgB6alcRMatI1R4 mEeQ== X-Gm-Message-State: AOAM531Di4XoHUSkaBcgv6UH1VseqGuUt9NfDG4XP/+QZdY7zhrC7ODF bOCwF6BfG14uisKYlQc3nUryGw== X-Google-Smtp-Source: ABdhPJzvRG4XF5j1xk4Nqrsni3oqzZ3ncomfE9kYtmbBgT+DGsI/BSV1m/HtrVoALYlDRBR/XGbI0Q== X-Received: by 2002:a63:7a51:: with SMTP id j17mr9271233pgn.186.1605727168499; Wed, 18 Nov 2020 11:19:28 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:27 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 5/5] btrfs: send: enable support for stream v2 and compressed writes Date: Wed, 18 Nov 2020 11:18:52 -0800 Message-Id: <836f1fbe6112ab89c6f592faaaea0dddb24fb8fe.1605723600.git.osandov@fb.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Omar Sandoval Now that the new support is implemented, allow the ioctl to accept the flags and update the version in sysfs. Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 10 +++++++++- fs/btrfs/send.h | 2 +- include/uapi/linux/btrfs.h | 4 +++- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 4d5cd90f4ef3..4b62bdbfbee0 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -670,7 +670,10 @@ static int send_header(struct send_ctx *sctx) struct btrfs_stream_header hdr; strcpy(hdr.magic, BTRFS_SEND_STREAM_MAGIC); - hdr.version = cpu_to_le32(BTRFS_SEND_STREAM_VERSION); + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) + hdr.version = cpu_to_le32(2); + else + hdr.version = cpu_to_le32(1); return write_buf(sctx->send_filp, &hdr, sizeof(hdr), &sctx->send_off); @@ -7434,6 +7437,11 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) ret = -EINVAL; goto out; } + if ((arg->flags & BTRFS_SEND_FLAG_COMPRESSED) && + !(arg->flags & BTRFS_SEND_FLAG_STREAM_V2)) { + ret = -EINVAL; + goto out; + } sctx = kzalloc(sizeof(struct send_ctx), GFP_KERNEL); if (!sctx) { diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index 9f4f7b96b1eb..9c83e14a43b2 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -10,7 +10,7 @@ #include "ctree.h" #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" -#define BTRFS_SEND_STREAM_VERSION 1 +#define BTRFS_SEND_STREAM_VERSION 2 /* * In send stream v1, no command is larger than 64k. In send stream v2, no limit diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h index 51e69f28d22d..6f29c456e4d7 100644 --- a/include/uapi/linux/btrfs.h +++ b/include/uapi/linux/btrfs.h @@ -785,7 +785,9 @@ struct btrfs_ioctl_received_subvol_args { #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ From patchwork Wed Nov 18 19:18:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8EBAC8300C for ; Wed, 18 Nov 2020 19:19:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4C9F922227 for ; Wed, 18 Nov 2020 19:19:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="roqYDntK" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727330AbgKRTTd (ORCPT ); Wed, 18 Nov 2020 14:19:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55872 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727312AbgKRTTd (ORCPT ); Wed, 18 Nov 2020 14:19:33 -0500 Received: from mail-pl1-x642.google.com (mail-pl1-x642.google.com [IPv6:2607:f8b0:4864:20::642]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B995CC0613D4 for ; Wed, 18 Nov 2020 11:19:31 -0800 (PST) Received: by mail-pl1-x642.google.com with SMTP id y22so1534250plr.6 for ; Wed, 18 Nov 2020 11:19:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BdTEku7J/1wmWyPZ7NOZDpBLdc5B/RHj8uVOrA+In/8=; b=roqYDntKPONpvGO4yrNgAnMiIvzffHjw1ItJjihDO0zAXyTqw7LaWVag9fQpotYUKd YfVB6eFRNdZ5s6MEsHcrcqaJFcYKJyMEfNku/pCinewZLtzbI4FIGXBqhiZkvNOg3A3B AAsgrrl7tDKgSZpiOIhXcoZ7ZDrPSrVaiXe02Q6CEiyYITVaT7I+IgwHFFaIsDnePx1e WhG4EGML51kiebsnX5Z3iIDjhfpdcS/8oh1i0RFjCOfvE90xXoiHIWBp+6lBvrBiS0fQ 16Y7ZDhSWkcCR2EJ/g4EwNQ7a/o4RLSksiEaCh4pS2UuBOyhD8vOYbv7lmLaJGGSgA1B RgFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BdTEku7J/1wmWyPZ7NOZDpBLdc5B/RHj8uVOrA+In/8=; b=Hqg56KByA7u7d4pqBy1JRUc8vYeIj2QSOTlPzKB3GvTGzsJE4yA7qA0gB6Z8g2eE0X OABrlPYPrbbzVsCR4/RP1zYe3aYWAYBnt2zYnZrAWOZnipvj+MoTs36HpfMXReKadvQW t4mQoM3u8x0tPN8jAlspNpundUG1w3Yc9DlSPg5z+x3/djbrgZT8DwO/8IRY6JmXvhkA Tda+/XewgUhkca+WqAXp/BlUPGbjBOeaxP4mREpc7nm9CMktyHaXfbzqIqdZvVQ/S+M8 Solip/yBx8/kdcVCYRBYHAwXlJeYVFNVk1ISlpnE1M0WgTeUf4DKNNBQxBc4jW8r822/ vlJg== X-Gm-Message-State: AOAM533QDV3suf5blamhSWaRdDOogkmJjyi+QqkwCdfAHk+iPvmqPHkM gZLjq/HBGMVaOtPNqxdRcxwlHw== X-Google-Smtp-Source: ABdhPJzLbh0+BDId9iRc2IZ73xfiwWhzpQU1qk3S1DKX/X95Jnkcr7rywmRMtBQwOSeUicRjgXvtWg== X-Received: by 2002:a17:902:6a83:b029:d5:e98f:2437 with SMTP id n3-20020a1709026a83b02900d5e98f2437mr5625373plk.38.1605727171319; Wed, 18 Nov 2020 11:19:31 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:29 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 06/13] btrfs-progs: receive: add stub implementation for pwritev2 Date: Wed, 18 Nov 2020 11:18:53 -0800 Message-Id: X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Boris Burkov Encoded writes in receive will use pwritev2. It is possible that the system libc does not export this function, so we stub it out and detect whether to build the stub code with autoconf. This syscall has special semantics in x32 (no hi lo, just takes loff_t) so we have to detect that case and use the appropriate arguments. Signed-off-by: Boris Burkov --- Makefile | 4 ++-- configure.ac | 1 + stubs.c | 24 ++++++++++++++++++++++++ stubs.h | 11 +++++++++++ 4 files changed, 38 insertions(+), 2 deletions(-) create mode 100644 stubs.c create mode 100644 stubs.h diff --git a/Makefile b/Makefile index 381b630d..505e39d7 100644 --- a/Makefile +++ b/Makefile @@ -173,12 +173,12 @@ libbtrfs_objects = common/send-stream.o common/send-utils.o kernel-lib/rbtree.o kernel-lib/raid56.o kernel-lib/tables.o \ common/device-scan.o common/path-utils.o \ common/utils.o libbtrfsutil/subvolume.o libbtrfsutil/stubs.o \ - crypto/hash.o crypto/xxhash.o $(CRYPTO_OBJECTS) + crypto/hash.o crypto/xxhash.o $(CRYPTO_OBJECTS) stubs.o libbtrfs_headers = common/send-stream.h common/send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \ crypto/crc32c.h kernel-lib/list.h kerncompat.h \ kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \ common/extent-cache.h kernel-shared/extent_io.h ioctl.h \ - kernel-shared/ctree.h btrfsck.h version.h + kernel-shared/ctree.h btrfsck.h version.h stubs.h libbtrfsutil_major := $(shell sed -rn 's/^\#define BTRFS_UTIL_VERSION_MAJOR ([0-9])+$$/\1/p' libbtrfsutil/btrfsutil.h) libbtrfsutil_minor := $(shell sed -rn 's/^\#define BTRFS_UTIL_VERSION_MINOR ([0-9])+$$/\1/p' libbtrfsutil/btrfsutil.h) libbtrfsutil_patch := $(shell sed -rn 's/^\#define BTRFS_UTIL_VERSION_PATCH ([0-9])+$$/\1/p' libbtrfsutil/btrfsutil.h) diff --git a/configure.ac b/configure.ac index dd4adedf..eaf353cc 100644 --- a/configure.ac +++ b/configure.ac @@ -57,6 +57,7 @@ AC_CHECK_FUNCS([openat], [], [AC_MSG_ERROR([cannot find openat() function])]) AC_CHECK_FUNCS([reallocarray]) +AC_CHECK_FUNCS([pwritev2]) m4_ifndef([PKG_PROG_PKG_CONFIG], [m4_fatal([Could not locate the pkg-config autoconf diff --git a/stubs.c b/stubs.c new file mode 100644 index 00000000..ab68a411 --- /dev/null +++ b/stubs.c @@ -0,0 +1,24 @@ +#if HAVE_PWRITEV2 != 1 + +#include "stubs.h" + +#include "kerncompat.h" + +#include +#include +#include + +ssize_t pwritev2(int fd, const struct iovec *iov, int iovcnt, off_t offset, + int flags) +{ +/* these conditions indicate an x32 system, which has a different pwritev2 */ +#if defined(__x86_64__) && defined(__ILP32__) + return syscall(SYS_pwritev2, fd, iov, iovcnt, offset, flags); +#else + unsigned long hi = offset >> (BITS_PER_LONG / 2) >> (BITS_PER_LONG / 2); + unsigned long lo = offset; + + return syscall(SYS_pwritev2, fd, iov, iovcnt, lo, hi, flags); +#endif // X32 +} +#endif /* HAVE_PWRIVEV2 */ diff --git a/stubs.h b/stubs.h new file mode 100644 index 00000000..b39f8a69 --- /dev/null +++ b/stubs.h @@ -0,0 +1,11 @@ +#ifndef _BTRFS_STUBS_H +#define _BTRFS_STUBS_H + +#include + +struct iovec; + +ssize_t pwritev2(int fd, const struct iovec *iov, int iovcnt, off_t offset, + int flags); + +#endif From patchwork Wed Nov 18 19:18:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC991C6379D for ; Wed, 18 Nov 2020 19:19:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8A5762220B for ; Wed, 18 Nov 2020 19:19:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="SXRyxESn" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727342AbgKRTTe (ORCPT ); Wed, 18 Nov 2020 14:19:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727324AbgKRTTd (ORCPT ); Wed, 18 Nov 2020 14:19:33 -0500 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9948FC0613D4 for ; Wed, 18 Nov 2020 11:19:33 -0800 (PST) Received: by mail-pl1-x644.google.com with SMTP id t18so1550541plo.0 for ; Wed, 18 Nov 2020 11:19:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8Juha+13op+j4M860hObsmkFcXreLNLajnjj5oyj6Fc=; b=SXRyxESnxqWtvijXr4ME6DRo29Bb+r4IbukyFtWQhUiOeWuZMiGoqk8zT99vU9CfIs sFsL4VFXw2Tdeu6ztQjIN8SWzVcjYb5tSsuwX2M7LgIRx5Vsjp0xhYMqn57o+cyV2Wcc Gd5uTWmqj4v7p5sN5d7gLtojV66ED7tqqRR5lXMvzAaTJksLxc6da5hefoE8IqFedSCV ZBQmOGce5/4zoRzCiE8Jo82GHMms5/lfwKHOoA2VmDX8VgTzqdPXpFhoCMXQqoxjNZFq XWGHYeHvM8PzVrbopREJ9MeummLpqJn5ttjLSbS+YtpopW8KYFNKXrNVcy0oyTa95rGD zRSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8Juha+13op+j4M860hObsmkFcXreLNLajnjj5oyj6Fc=; b=VzJzrivg9+Cf4a0OJXdJiJG5mwKE2GzgcHEGZs7RlNIzrMIgG06rWiaozGLVthoxYn nUQ1vqzM8x60oiYAwNPuNLLK2qy7YX+0j0c9Avzjn3jUt//HnPbq6A1hIk2521i/CR1f 1Zwf6FmSj6oHQgCwR2+V6w2yTHrOs/duePU32nDZCA3Q/AWMHMfepCvO9oQ3k3JexK4r yCIhtwuhinb7dcziu2uH4hFUJ/IigpJJWZCzwv7dENzpEmfRxGudhs9Ubu6NHuaC2CcM jDI7V9/dlRhcaAz71Nu9UiV1O6diNPxhlV7HbvZ3o3Yv4pcKbIYu1vkRCNaIPCLsWee0 Gu3A== X-Gm-Message-State: AOAM5309iJOnEiRZ2R+8zsCU9a+KmcyfyzQa19XKfmIBnIk12GjV9OY3 Rb3cOK7x7sBsCdK9M1BJnMtIig== X-Google-Smtp-Source: ABdhPJwtVAIOFL011TKnuC2lFW9Qq26zYzRJ4/EAzUsgjlotJzQpQnLq3yrQQp7SThqbk1uJ/r1pkw== X-Received: by 2002:a17:902:8ec7:b029:d8:e603:304c with SMTP id x7-20020a1709028ec7b02900d8e603304cmr5393411plo.85.1605727173156; Wed, 18 Nov 2020 11:19:33 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:32 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 07/13] btrfs-progs: receive: open files with O_CLOEXEC Date: Wed, 18 Nov 2020 11:18:54 -0800 Message-Id: <41f3e3f92372f48037c7cee18af808d4cc5352c5.1605723745.git.osandov@osandov.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Omar Sandoval Opening with O_ALLOW_ENCODED also requires O_CLOEXEC. Let's add O_CLOEXEC now in preparation. btrfs receive doesn't exec anything, so this should be a no-op. Signed-off-by: Omar Sandoval --- cmds/receive.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/cmds/receive.c b/cmds/receive.c index 2aaba3ff..2c56cea6 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -654,7 +654,11 @@ static int open_inode_for_write(struct btrfs_receive *rctx, const char *path) rctx->write_fd = -1; } - rctx->write_fd = open(path, O_RDWR); + /* + * When opening with O_ALLOW_ENCODED, O_CLOEXEC must also be specified. + * We might as well always use it even though we don't exec anything. + */ + rctx->write_fd = open(path, O_RDWR | O_CLOEXEC); if (rctx->write_fd < 0) { ret = -errno; error("cannot open %s: %m", path); From patchwork Wed Nov 18 19:18:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50A6CC6379F for ; Wed, 18 Nov 2020 19:19:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DC2AA22227 for ; Wed, 18 Nov 2020 19:19:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="iZIK8j5Z" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727355AbgKRTTi (ORCPT ); Wed, 18 Nov 2020 14:19:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727344AbgKRTTh (ORCPT ); Wed, 18 Nov 2020 14:19:37 -0500 Received: from mail-pl1-x641.google.com (mail-pl1-x641.google.com [IPv6:2607:f8b0:4864:20::641]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C48EAC0613D4 for ; Wed, 18 Nov 2020 11:19:35 -0800 (PST) Received: by mail-pl1-x641.google.com with SMTP id bj5so717894plb.4 for ; Wed, 18 Nov 2020 11:19:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GSKSDRR8K/+TKUuUq5SwiGfUO6KTVDHzgwHOrlirzyk=; b=iZIK8j5ZVlJW3ibHpPfRgjY/R2IWpjUmAQWNTUIUac8hV6LSOlpLK9uclduYXRW2EE 3tSywZp392AmcfprV0GK5UfY7rymX3b3iAWKAo52bXWzcK1XhSJYRFQcgsNDIHqmPlA2 kf5xvz0eSlVYL69Lw9eA4dkGfirWFyUU94WCUFYmG4NBNU4JsdjqUV5rS8vQgrj0pnmg 93P2NOSsMJHeNAyXqPP99nBqRuOMoZ1hpF05mpra2hGX08g17KHJ1+xMQTTqjXnohZJG 3yBggj3a0QCBMAFX0nuMkEZhdAdB0tDwVkEIWxG7YSdDkl0XXzgvfyYV+jjH2AjX+Rr5 NF1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GSKSDRR8K/+TKUuUq5SwiGfUO6KTVDHzgwHOrlirzyk=; b=pCrWRh5GOom7wqJWDWX2Pc1SaNtgPR2Yw6Stou+Aqc6krMTCE3audvZNZBkVPLgE8A xxt/qaINlZcH4yxD/DZBo+AdScgMtfbyEW2glDQu38Z/Va+cbFwZql0HKUbIkkNh11nQ Zys9SktI3mq595UIll4nmq8/1CBxce8lGgCfahFdEufEimUcW4tK8yj0GbawWdS/kv9Q HcKGsrTl9Pf4WKXN5qK7O2PxrI4HqEHzJFYdXNs0dkhnz/qjk8h/82cgebgPiH8JcbyU n0u/eJJ6nUq+RAJk8Qf/8iKW3lUId8kEJDY4V7nOHbLaz8/XGF+e9c304bSitYiY4z9J kB3w== X-Gm-Message-State: AOAM530eZJIHvetTObFXLzh1pMp8WYhS9hK6MC6Wci5ec1lXIDRCMRYp iTa2gnoSCp7doFBTFO7aKP5K2hQx6D/hAQ== X-Google-Smtp-Source: ABdhPJx5TBEnjr5vEOFv+PxN5aWsXsCncd4fanIQGwc7uRj2/+E+lXhHxJtEXiPIwMXWcJjYvl7R7A== X-Received: by 2002:a17:902:8c84:b029:d9:471:f0da with SMTP id t4-20020a1709028c84b02900d90471f0damr5225230plo.84.1605727175286; Wed, 18 Nov 2020 11:19:35 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:34 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 08/13] btrfs-progs: receive: process encoded_write commands Date: Wed, 18 Nov 2020 11:18:55 -0800 Message-Id: X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Boris Burkov Add a new btrfs_send_op and support for both dumping and proper receive processing which does actual encoded writes. Encoded writes are only allowed on a file descriptor opened with an extra flag that allows encoded writes, so we also add support for this flag when opening or reusing a file for writing. Signed-off-by: Boris Burkov --- cmds/receive-dump.c | 16 +++++- cmds/receive.c | 118 ++++++++++++++++++++++++++++++++++++++----- common/send-stream.c | 22 ++++++++ common/send-stream.h | 4 ++ stubs.h | 56 ++++++++++++++++++++ 5 files changed, 203 insertions(+), 13 deletions(-) diff --git a/cmds/receive-dump.c b/cmds/receive-dump.c index 648d9314..20ec2b70 100644 --- a/cmds/receive-dump.c +++ b/cmds/receive-dump.c @@ -316,6 +316,19 @@ static int print_update_extent(const char *path, u64 offset, u64 len, offset, len); } +static int print_encoded_write(const char *path, const void *data, u64 offset, + u64 len, u64 unencoded_file_len, + u64 unencoded_len, u64 unencoded_offset, + u32 compression, u32 encryption, void *user) +{ + return PRINT_DUMP(user, path, "encoded_write", + "offset=%llu len=%llu, unencoded_file_len=%llu, " + "unencoded_len=%llu, unencoded_offset=%llu, " + "compression=%u, encryption=%u", + offset, len, unencoded_file_len, unencoded_len, + unencoded_offset, compression, encryption); +} + struct btrfs_send_ops btrfs_print_send_ops = { .subvol = print_subvol, .snapshot = print_snapshot, @@ -337,5 +350,6 @@ struct btrfs_send_ops btrfs_print_send_ops = { .chmod = print_chmod, .chown = print_chown, .utimes = print_utimes, - .update_extent = print_update_extent + .update_extent = print_update_extent, + .encoded_write = print_encoded_write, }; diff --git a/cmds/receive.c b/cmds/receive.c index 2c56cea6..87f47a9a 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -30,12 +30,14 @@ #include #include #include +#include #include #include #include #include #include +#include #include #include @@ -52,6 +54,7 @@ #include "cmds/receive-dump.h" #include "common/help.h" #include "common/path-utils.h" +#include "stubs.h" struct btrfs_receive { @@ -60,6 +63,7 @@ struct btrfs_receive int write_fd; char write_path[PATH_MAX]; + int write_fd_allow_encoded; char *root_path; char *dest_dir_path; /* relative to root_path */ @@ -643,28 +647,69 @@ out: return ret; } -static int open_inode_for_write(struct btrfs_receive *rctx, const char *path) +static int set_write_fd_allow_encoded(struct btrfs_receive *rctx) +{ + int ret; + int flags; + + flags = fcntl(rctx->write_fd, F_GETFL); + if (flags < 0) { + ret = -errno; + error("failed to fetch old fd flags"); + goto close_fd; + } + ret = fcntl(rctx->write_fd, F_SETFL, flags | O_ALLOW_ENCODED); + if (ret < 0) { + ret = -errno; + error("failed to enable encoded writes"); + goto close_fd; + } + rctx->write_fd_allow_encoded = true; + ret = 0; + goto out; +close_fd: + close(rctx->write_fd); + rctx->write_fd = -1; + rctx->write_fd_allow_encoded = false; +out: + return ret; +} + +static int open_inode_for_write(struct btrfs_receive *rctx, const char *path, + bool allow_encoded) { int ret = 0; - - if (rctx->write_fd != -1) { - if (strcmp(rctx->write_path, path) == 0) - goto out; - close(rctx->write_fd); - rctx->write_fd = -1; - } - /* * When opening with O_ALLOW_ENCODED, O_CLOEXEC must also be specified. * We might as well always use it even though we don't exec anything. */ - rctx->write_fd = open(path, O_RDWR | O_CLOEXEC); + int flags = O_RDWR | O_CLOEXEC; + + if (rctx->write_fd != -1) { + /* + * if the existing fd is for this path and the needed flags are + * satisfied, no need to open a new one + */ + if (strcmp(rctx->write_path, path) == 0) { + /* fixup the allow encoded flag, if necessary */ + if (allow_encoded && !rctx->write_fd_allow_encoded) + ret = set_write_fd_allow_encoded(rctx); + goto out; + } + close(rctx->write_fd); + rctx->write_fd = -1; + } + + if (allow_encoded) + flags |= O_ALLOW_ENCODED; + rctx->write_fd = open(path, flags); if (rctx->write_fd < 0) { ret = -errno; error("cannot open %s: %m", path); goto out; } strncpy_null(rctx->write_path, path); + rctx->write_fd_allow_encoded = allow_encoded; out: return ret; @@ -695,7 +740,7 @@ static int process_write(const char *path, const void *data, u64 offset, goto out; } - ret = open_inode_for_write(rctx, full_path); + ret = open_inode_for_write(rctx, full_path, false); if (ret < 0) goto out; @@ -738,7 +783,7 @@ static int process_clone(const char *path, u64 offset, u64 len, goto out; } - ret = open_inode_for_write(rctx, full_path); + ret = open_inode_for_write(rctx, full_path, false); if (ret < 0) goto out; @@ -1032,6 +1077,54 @@ static int process_update_extent(const char *path, u64 offset, u64 len, return 0; } +static int process_encoded_write(const char *path, const void *data, u64 offset, + u64 len, u64 unencoded_file_len, u64 unencoded_len, + u64 unencoded_offset, u32 compression, u32 encryption, void *user) +{ + int ret; + ssize_t w; + struct btrfs_receive *rctx = user; + char full_path[PATH_MAX]; + struct encoded_iov encoded = { + .len = unencoded_file_len, + .unencoded_len = unencoded_len, + .unencoded_offset = unencoded_offset, + .compression = compression, + .encryption = encryption, + }; + struct iovec iov[2] = { + { &encoded, sizeof(encoded) }, + { (char *)data, len } + }; + + if (encryption) { + error("encoded_write: encryption not supported"); + return -EOPNOTSUPP; + } + + ret = path_cat_out(full_path, rctx->full_subvol_path, path); + if (ret < 0) { + error("encoded_write: path invalid: %s", path); + return ret; + } + + ret = open_inode_for_write(rctx, full_path, true); + if (ret < 0) + return ret; + + /* + * NOTE: encoded writes guarantee no partial writes, so we don't need to + * handle that possibility. + */ + w = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); + if (w < 0) { + ret = -errno; + error("encoded_write: writing to %s failed: %m", path); + return ret; + } + return 0; +} + static struct btrfs_send_ops send_ops = { .subvol = process_subvol, .snapshot = process_snapshot, @@ -1054,6 +1147,7 @@ static struct btrfs_send_ops send_ops = { .chown = process_chown, .utimes = process_utimes, .update_extent = process_update_extent, + .encoded_write = process_encoded_write, }; static int do_receive(struct btrfs_receive *rctx, const char *tomnt, diff --git a/common/send-stream.c b/common/send-stream.c index 77d5cd04..1376e00b 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -354,6 +354,8 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) struct timespec mt; u8 uuid[BTRFS_UUID_SIZE]; u8 clone_uuid[BTRFS_UUID_SIZE]; + u32 compression; + u32 encryption; u64 tmp; u64 tmp2; u64 ctransid; @@ -362,6 +364,9 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) u64 dev; u64 clone_offset; u64 offset; + u64 unencoded_file_len; + u64 unencoded_len; + u64 unencoded_offset; int len; int xattr_len; @@ -436,6 +441,23 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) TLV_GET(sctx, BTRFS_SEND_A_DATA, &data, &len); ret = sctx->ops->write(path, data, offset, len, sctx->user); break; + case BTRFS_SEND_C_ENCODED_WRITE: + TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); + TLV_GET_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, &offset); + TLV_GET_U64(sctx, BTRFS_SEND_A_UNENCODED_FILE_LEN, + &unencoded_file_len); + TLV_GET_U64(sctx, BTRFS_SEND_A_UNENCODED_LEN, &unencoded_len); + TLV_GET_U64(sctx, BTRFS_SEND_A_UNENCODED_OFFSET, + &unencoded_offset); + TLV_GET_U32(sctx, BTRFS_SEND_A_COMPRESSION, &compression); + TLV_GET_U32(sctx, BTRFS_SEND_A_ENCRYPTION, &encryption); + TLV_GET(sctx, BTRFS_SEND_A_DATA, &data, &len); + ret = sctx->ops->encoded_write(path, data, offset, len, + unencoded_file_len, + unencoded_len, unencoded_offset, + compression, encryption, + sctx->user); + break; case BTRFS_SEND_C_CLONE: TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); TLV_GET_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, &offset); diff --git a/common/send-stream.h b/common/send-stream.h index 39901f86..607bc007 100644 --- a/common/send-stream.h +++ b/common/send-stream.h @@ -66,6 +66,10 @@ struct btrfs_send_ops { struct timespec *mt, struct timespec *ct, void *user); int (*update_extent)(const char *path, u64 offset, u64 len, void *user); + int (*encoded_write)(const char *path, const void *data, u64 offset, + u64 len, u64 unencoded_file_len, u64 unencoded_len, + u64 unencoded_offset, u32 compression, + u32 encryption, void *user); }; int btrfs_read_and_process_send_stream(int fd, diff --git a/stubs.h b/stubs.h index b39f8a69..7574ae8f 100644 --- a/stubs.h +++ b/stubs.h @@ -1,6 +1,8 @@ #ifndef _BTRFS_STUBS_H #define _BTRFS_STUBS_H +#include +#include #include struct iovec; @@ -8,4 +10,58 @@ struct iovec; ssize_t pwritev2(int fd, const struct iovec *iov, int iovcnt, off_t offset, int flags); +#ifndef O_ALLOW_ENCODED +#if defined(__alpha__) +#define O_ALLOW_ENCODED 0200000000 +#elif defined(__hppa__) +#define O_ALLOW_ENCODED 100000000 +#elif defined(__sparc__) +#define O_ALLOW_ENCODED 0x8000000 +#else +#define O_ALLOW_ENCODED 040000000 #endif +#endif + +#ifndef RWF_ENCODED + +enum { + ENCODED_IOV_COMPRESSION_NONE, +#define ENCODED_IOV_COMPRESSION_NONE ENCODED_IOV_COMPRESSION_NONE + ENCODED_IOV_COMPRESSION_BTRFS_ZLIB, +#define ENCODED_IOV_COMPRESSION_BTRFS_ZLIB ENCODED_IOV_COMPRESSION_BTRFS_ZLIB + ENCODED_IOV_COMPRESSION_BTRFS_ZSTD, +#define ENCODED_IOV_COMPRESSION_BTRFS_ZSTD ENCODED_IOV_COMPRESSION_BTRFS_ZSTD + ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K, +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K + ENCODED_IOV_COMPRESSION_BTRFS_LZO_8K, +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_8K ENCODED_IOV_COMPRESSION_BTRFS_LZO_8K + ENCODED_IOV_COMPRESSION_BTRFS_LZO_16K, +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_16K ENCODED_IOV_COMPRESSION_BTRFS_LZO_16K + ENCODED_IOV_COMPRESSION_BTRFS_LZO_32K, +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_32K ENCODED_IOV_COMPRESSION_BTRFS_LZO_32K + ENCODED_IOV_COMPRESSION_BTRFS_LZO_64K, +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_64K ENCODED_IOV_COMPRESSION_BTRFS_LZO_64K + ENCODED_IOV_COMPRESSION_TYPES = ENCODED_IOV_COMPRESSION_BTRFS_LZO_64K, +}; + +enum { + ENCODED_IOV_ENCRYPTION_NONE, +#define ENCODED_IOV_ENCRYPTION_NONE ENCODED_IOV_ENCRYPTION_NONE + ENCODED_IOV_ENCRYPTION_TYPES = ENCODED_IOV_ENCRYPTION_NONE, +}; + +struct encoded_iov { + __aligned_u64 len; + __aligned_u64 unencoded_len; + __aligned_u64 unencoded_offset; + __u32 compression; + __u32 encryption; +}; + +#define ENCODED_IOV_SIZE_VER0 32 + +/* encoded (e.g., compressed and/or encrypted) IO */ +#define RWF_ENCODED ((__kernel_rwf_t)0x00000020) +#endif /* RWF_ENCODED */ + +#endif /* _BTRFS_STUBS_H */ From patchwork Wed Nov 18 19:18:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1B01C56202 for ; Wed, 18 Nov 2020 19:19:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6006222228 for ; Wed, 18 Nov 2020 19:19:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="nzxpNUj6" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727344AbgKRTTi (ORCPT ); Wed, 18 Nov 2020 14:19:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727350AbgKRTTi (ORCPT ); Wed, 18 Nov 2020 14:19:38 -0500 Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1206AC0613D4 for ; Wed, 18 Nov 2020 11:19:38 -0800 (PST) Received: by mail-pf1-x442.google.com with SMTP id 131so2035262pfb.9 for ; Wed, 18 Nov 2020 11:19:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Uou+avacjLUgrd6sYNlplF7cpTImH06UMIPWOWB8yac=; b=nzxpNUj6PoGBVmyVq2S5Fp1qk7EKOPe6nNjaOmvNKESVqAaIkL9LtQBkbYPlh17oxN QWNpL4lDd9zQgBUi64ePtOxUDaqQczmWuQOZFHlNYGzDO17xu8DPVkyFiiL/D4wBr5jB vap+fxZY6N98wIMdyKpebAWkqFmWk46TOyHe8UD/y0BefzXbQxXE7pUN1yqSc7FjqfdS ccC3zfmoQ0hL1Yjijm3osiQC5SRVXv9u+itrHDoODV+cyRSGzd4M8iBsIrJzKKC3q+5m 5uDJK0O2S8QT66/15D6WRfaeMBaygMxT9Fi28ILgSIR2xNCPJlRvBQDZRtdUKwbUqdiO o5qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Uou+avacjLUgrd6sYNlplF7cpTImH06UMIPWOWB8yac=; b=L5FIDOuf7KgOpUgEDO2iJ6psEz40jRfHyxpsZZ3awGoO4RFeioAfbxpeENCimmKNFD 1LK6Ccys6Fy1R/I0rS9p/lN3BQO2WcKoKfyY9vDiel07BlYBmS1nBsVcue29JDdKaBJs wy/vmbYrIwefsrl0uUir6RLSq2FU3GM/UwzPFPY9gfTVZ2bmbydoGSyJ0suqaZzNCp29 9YTS8R+o17VeWnkoVr4Mu9MdlMYhu9p1rZ5+CD0nP1WNAPiOn4CbWxJTnASHE0k9Poxi k18oSZZp8s8j6zn6394C0xhsnGJxfJEepEkfEAwTYRugMvbqvfn+XX9OzP7c6JcG6TYG CiJA== X-Gm-Message-State: AOAM530yR8GHQxYtfmta86Fbw3c7r+5/+OgTWEPF5kt3xK+DsxLh1UvG OpAuTuzDZ3XMUtDCU/SPhOPFNQ== X-Google-Smtp-Source: ABdhPJxeeAcMunQfSTcIXD946l5RX2PtUCFBhHPmRN80hYEI6+DTzpdaJ52wUPqWfy5Qhc4NJuKu6Q== X-Received: by 2002:a05:6a00:16c4:b029:162:bf9f:6458 with SMTP id l4-20020a056a0016c4b0290162bf9f6458mr5955906pfc.55.1605727177373; Wed, 18 Nov 2020 11:19:37 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:36 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 09/13] btrfs-progs: receive: encoded_write fallback to explicit decode and write Date: Wed, 18 Nov 2020 11:18:56 -0800 Message-Id: <003ced29dccaac8e5bf41aa4f571f0154d011ae0.1605723745.git.osandov@osandov.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Boris Burkov An encoded_write can fail if the file system it is being applied to does not support encoded writes or if it can't find enough contiguous space to accommodate the encoded extent. In those cases, we can likely still process an encoded_write by explicitly decoding the data and doing a normal write. Add the necessary fallback path for decoding data compressed with zlib, lzo, or zstd. zlib and zstd have reusable decoding context data structures which we cache in the receive context so that we don't have to recreate them on every encoded_write. Finally, add a command line flag for force-decompress which causes receive to always use the fallback path rather than first attempting the encoded write. Signed-off-by: Boris Burkov --- Documentation/btrfs-receive.asciidoc | 4 + cmds/receive.c | 274 +++++++++++++++++++++++++-- 2 files changed, 265 insertions(+), 13 deletions(-) diff --git a/Documentation/btrfs-receive.asciidoc b/Documentation/btrfs-receive.asciidoc index e4c4d2c0..354a71dc 100644 --- a/Documentation/btrfs-receive.asciidoc +++ b/Documentation/btrfs-receive.asciidoc @@ -60,6 +60,10 @@ By default the mountpoint is searched in '/proc/self/mounts'. If '/proc' is not accessible, eg. in a chroot environment, use this option to tell us where this filesystem is mounted. +--force-decompress:: +if the stream contains compressed data (see '--compressed-data' in +`btrfs-send`(8)), always decompress it instead of writing it with encoded I/O. + --dump:: dump the stream metadata, one line per operation + diff --git a/cmds/receive.c b/cmds/receive.c index 87f47a9a..361c7c8a 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -41,6 +41,10 @@ #include #include +#include +#include +#include + #include "kernel-shared/ctree.h" #include "ioctl.h" #include "cmds/commands.h" @@ -82,6 +86,8 @@ struct btrfs_receive int honor_end_cmd; + int force_decompress; + /* * Buffer to store capabilities from security.capabilities xattr, * usually 20 bytes, but make same room for potentially larger @@ -89,6 +95,10 @@ struct btrfs_receive */ char cached_capabilities[64]; int cached_capabilities_len; + + /* Reuse stream objects for encoded_write decompression fallback */ + ZSTD_DStream *zstd_dstream; + z_stream *zlib_stream; }; static int finish_subvol(struct btrfs_receive *rctx) @@ -1077,9 +1087,222 @@ static int process_update_extent(const char *path, u64 offset, u64 len, return 0; } +static int decompress_zlib(struct btrfs_receive *rctx, const char *encoded_data, + u64 encoded_len, char *unencoded_data, + u64 unencoded_len) +{ + bool init = false; + int ret; + + if (!rctx->zlib_stream) { + init = true; + rctx->zlib_stream = malloc(sizeof(z_stream)); + if (!rctx->zlib_stream) { + error("failed to allocate zlib stream %m"); + return -ENOMEM; + } + } + rctx->zlib_stream->next_in = (void *)encoded_data; + rctx->zlib_stream->avail_in = encoded_len; + rctx->zlib_stream->next_out = (void *)unencoded_data; + rctx->zlib_stream->avail_out = unencoded_len; + + if (init) { + rctx->zlib_stream->zalloc = Z_NULL; + rctx->zlib_stream->zfree = Z_NULL; + rctx->zlib_stream->opaque = Z_NULL; + ret = inflateInit(rctx->zlib_stream); + } else { + ret = inflateReset(rctx->zlib_stream); + } + if (ret != Z_OK) { + error("zlib inflate init failed: %d", ret); + return -EIO; + } + + while (rctx->zlib_stream->avail_in > 0 && + rctx->zlib_stream->avail_out > 0) { + ret = inflate(rctx->zlib_stream, Z_FINISH); + if (ret == Z_STREAM_END) { + break; + } else if (ret != Z_OK) { + error("zlib inflate failed: %d", ret); + return -EIO; + } + } + return 0; +} + +static int decompress_zstd(struct btrfs_receive *rctx, const char *encoded_buf, + u64 encoded_len, char *unencoded_buf, + u64 unencoded_len) +{ + ZSTD_inBuffer in_buf = { + .src = encoded_buf, + .size = encoded_len + }; + ZSTD_outBuffer out_buf = { + .dst = unencoded_buf, + .size = unencoded_len + }; + size_t ret; + + if (!rctx->zstd_dstream) { + rctx->zstd_dstream = ZSTD_createDStream(); + if (!rctx->zstd_dstream) { + error("failed to create zstd dstream"); + return -ENOMEM; + } + } + ret = ZSTD_initDStream(rctx->zstd_dstream); + if (ZSTD_isError(ret)) { + error("failed to init zstd stream: %s", ZSTD_getErrorName(ret)); + return -EIO; + } + while (in_buf.pos < in_buf.size && out_buf.pos < out_buf.size) { + ret = ZSTD_decompressStream(rctx->zstd_dstream, &out_buf, &in_buf); + if (ret == 0) { + break; + } else if (ZSTD_isError(ret)) { + error("failed to decompress zstd stream: %s", + ZSTD_getErrorName(ret)); + return -EIO; + } + } + return 0; +} + +static int decompress_lzo(const char *encoded_data, u64 encoded_len, + char *unencoded_data, u64 unencoded_len, + unsigned int page_size) +{ + uint32_t total_len; + size_t in_pos, out_pos; + + if (encoded_len < 4) { + error("lzo header is truncated"); + return -EIO; + } + memcpy(&total_len, encoded_data, 4); + total_len = le32toh(total_len); + if (total_len > encoded_len) { + error("lzo header is invalid"); + return -EIO; + } + + in_pos = 4; + out_pos = 0; + while (in_pos < total_len && out_pos < unencoded_len) { + size_t page_remaining; + uint32_t src_len; + lzo_uint dst_len; + int ret; + + page_remaining = -in_pos % page_size; + if (page_remaining < 4) { + if (total_len - in_pos <= page_remaining) + break; + in_pos += page_remaining; + } + + if (total_len - in_pos < 4) { + error("lzo segment header is truncated"); + return -EIO; + } + + memcpy(&src_len, encoded_data + in_pos, 4); + src_len = le32toh(src_len); + in_pos += 4; + if (src_len > total_len - in_pos) { + error("lzo segment header is invalid"); + return -EIO; + } + + dst_len = page_size; + ret = lzo1x_decompress_safe((void *)(encoded_data + in_pos), + src_len, + (void *)(unencoded_data + out_pos), + &dst_len, NULL); + if (ret != LZO_E_OK) { + error("lzo1x_decompress_safe failed: %d", ret); + return -EIO; + } + + in_pos += src_len; + out_pos += dst_len; + } + return 0; +} + +static int decompress_and_write(struct btrfs_receive *rctx, + const char *encoded_data, u64 offset, + u64 encoded_len, u64 unencoded_file_len, + u64 unencoded_len, u64 unencoded_offset, + u32 compression) +{ + int ret = 0; + size_t pos; + ssize_t w; + char *unencoded_data; + int page_shift; + + unencoded_data = calloc(unencoded_len, 1); + if (!unencoded_data) { + error("allocating space for unencoded data failed: %m"); + return -errno; + } + + switch (compression) { + case ENCODED_IOV_COMPRESSION_BTRFS_ZLIB: + ret = decompress_zlib(rctx, encoded_data, encoded_len, + unencoded_data, unencoded_len); + if (ret) + goto out; + break; + case ENCODED_IOV_COMPRESSION_BTRFS_ZSTD: + ret = decompress_zstd(rctx, encoded_data, encoded_len, + unencoded_data, unencoded_len); + if (ret) + goto out; + break; + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_8K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_16K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_32K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_64K: + page_shift = compression - ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K + 12; + ret = decompress_lzo(encoded_data, encoded_len, unencoded_data, + unencoded_len, 1U << page_shift); + if (ret) + goto out; + break; + default: + error("unknown compression: %d", compression); + ret = -EOPNOTSUPP; + goto out; + } + + pos = unencoded_offset; + while (pos < unencoded_file_len) { + w = pwrite(rctx->write_fd, unencoded_data + pos, + unencoded_file_len - pos, offset); + if (w < 0) { + ret = -errno; + error("writing unencoded data failed: %m"); + goto out; + } + pos += w; + offset += w; + } +out: + free(unencoded_data); + return ret; +} + static int process_encoded_write(const char *path, const void *data, u64 offset, - u64 len, u64 unencoded_file_len, u64 unencoded_len, - u64 unencoded_offset, u32 compression, u32 encryption, void *user) + u64 len, u64 unencoded_file_len, + u64 unencoded_len, u64 unencoded_offset, + u32 compression, u32 encryption, void *user) { int ret; ssize_t w; @@ -1096,6 +1319,7 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, { &encoded, sizeof(encoded) }, { (char *)data, len } }; + bool encoded_write = !rctx->force_decompress; if (encryption) { error("encoded_write: encryption not supported"); @@ -1112,17 +1336,25 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, if (ret < 0) return ret; - /* - * NOTE: encoded writes guarantee no partial writes, so we don't need to - * handle that possibility. - */ - w = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); - if (w < 0) { - ret = -errno; - error("encoded_write: writing to %s failed: %m", path); - return ret; + if (encoded_write) { + /* + * NOTE: encoded writes guarantee no partial writes, so we don't + * need to handle that possibility. + */ + w = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); + if (w >= 0) + return 0; + /* Fall back for these errors, fail hard for anything else. */ + if (errno != ENOSPC && errno != EOPNOTSUPP && errno != EINVAL) { + ret = -errno; + error("encoded_write: writing to %s failed: %m", path); + return ret; + } } - return 0; + + return decompress_and_write(rctx, data, offset, len, unencoded_file_len, + unencoded_len, unencoded_offset, + compression); } static struct btrfs_send_ops send_ops = { @@ -1310,6 +1542,12 @@ out: close(rctx->dest_dir_fd); rctx->dest_dir_fd = -1; } + if (rctx->zstd_dstream) + ZSTD_freeDStream(rctx->zstd_dstream); + if (rctx->zlib_stream) { + inflateEnd(rctx->zlib_stream); + free(rctx->zlib_stream); + } return ret; } @@ -1340,6 +1578,9 @@ static const char * const cmd_receive_usage[] = { "-m ROOTMOUNT the root mount point of the destination filesystem.", " If /proc is not accessible, use this to tell us where", " this file system is mounted.", + "--force-decompress", + " if the stream contains compressed data, always", + " decompress it instead of writing it with encoded I/O", "--dump dump stream metadata, one line per operation,", " does not require the MOUNT parameter", "-v deprecated, alias for global -v option", @@ -1383,12 +1624,16 @@ static int cmd_receive(const struct cmd_struct *cmd, int argc, char **argv) optind = 0; while (1) { int c; - enum { GETOPT_VAL_DUMP = 257 }; + enum { + GETOPT_VAL_DUMP = 257, + GETOPT_VAL_FORCE_DECOMPRESS, + }; static const struct option long_opts[] = { { "max-errors", required_argument, NULL, 'E' }, { "chroot", no_argument, NULL, 'C' }, { "dump", no_argument, NULL, GETOPT_VAL_DUMP }, { "quiet", no_argument, NULL, 'q' }, + { "force-decompress", no_argument, NULL, GETOPT_VAL_FORCE_DECOMPRESS }, { NULL, 0, NULL, 0 } }; @@ -1431,6 +1676,9 @@ static int cmd_receive(const struct cmd_struct *cmd, int argc, char **argv) case GETOPT_VAL_DUMP: dump = 1; break; + case GETOPT_VAL_FORCE_DECOMPRESS: + rctx.force_decompress = 1; + break; default: usage_unknown_option(cmd, argv); } From patchwork Wed Nov 18 19:18:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52A38C64E69 for ; Wed, 18 Nov 2020 19:19:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EAD5622227 for ; Wed, 18 Nov 2020 19:19:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="yrvZIKPZ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727365AbgKRTTm (ORCPT ); Wed, 18 Nov 2020 14:19:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55912 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727350AbgKRTTl (ORCPT ); Wed, 18 Nov 2020 14:19:41 -0500 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98A9FC0613D4 for ; Wed, 18 Nov 2020 11:19:41 -0800 (PST) Received: by mail-pf1-x441.google.com with SMTP id w6so2068843pfu.1 for ; Wed, 18 Nov 2020 11:19:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=V3j6zZr0VDMJvc7zZVcwrRVz+yGAYinHXHs1QBoNG4M=; b=yrvZIKPZURujIFP5Vio+JkFgfLuiFrd6IC2/KC0DAKghdLiaqSAXvFG9Dw96eg0IpQ U5rjt9W9lz2AkcBEFS8Ab6Euy/SWTtRYRPJb8OsfF3zOg0QD+pBao9NOBcTwI8ZZdljO uY2g0IaXfVk62dwcTpK7uZX2m+lBtjqRxI71QTPhMWrigsBzEJe0EKQILrVB6grrFOnm 7x458/pZCZ/bHxA5h6pjBNrzsTo5pOMJaDmmEw2U8qAXVhFEo8P6fhOLMFNOsuISFTTv eWHrrCH6zKfHYV6B+o//rJVn0Ch7v9Ce3rD9yovGPDuqCTueC0BP7o2MrVGAB5fMSY45 ctWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=V3j6zZr0VDMJvc7zZVcwrRVz+yGAYinHXHs1QBoNG4M=; b=kGHDdC8gL+elWnmKZ/73mmy2i1asu42oG7h+l5ilt0pS4SMEwQFjSsGReq3m5sWvoA gyN4GImg4DQ8PRDj1RSHM5vwvKxp3nhpHUIzeVuQ8aQj1wGvcp44EG1szYWHzLfRz5Eo VbLPjHGbrTljDYEp51AcDervuvPKZshxG4QYEIf8TWaJLdH5zsKhkaGIKUTLbDyIRXAd vfQHqcN+3ZxrjvUA6R7oiGhPGOz8foN+r4zkMUBFhUdgKiJGjRVMW3xvVSmHugQWWAg7 gKL6TMZdBbLYP/yJG33cbbGcfthVBpFtHEMKbglTpXoJXRNjEqzY3SRDO/Zqn+2DEtN8 iVUg== X-Gm-Message-State: AOAM533LiLucrvYTJIV4LvOQUGLZu5zqVW+OoTwzhtRZLJ1HgzFV1v1v 4ZgK2ARJQjYq/7iAAKPiq0A9Sw== X-Google-Smtp-Source: ABdhPJwTZa7tfZEaS8Ag0NQtDtafbzxzHuL/EumP+kMYVnhhH9fW02xHaqmCMvVWjSBDKP9VAIDR1g== X-Received: by 2002:a17:90b:705:: with SMTP id s5mr547489pjz.22.1605727181145; Wed, 18 Nov 2020 11:19:41 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:39 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 10/13] btrfs-progs: receive: process fallocate commands Date: Wed, 18 Nov 2020 11:18:57 -0800 Message-Id: X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Boris Burkov Send stream v2 can emit fallocate commands, so receive must support them as well. The implementation simply passes along the arguments to the syscall. Note that mode is encoded as a u32 in send stream but fallocate takes an int, so there is a unsigned->signed conversion there. Signed-off-by: Boris Burkov --- cmds/receive-dump.c | 9 +++++++++ cmds/receive.c | 25 +++++++++++++++++++++++++ common/send-stream.c | 9 +++++++++ common/send-stream.h | 2 ++ 4 files changed, 45 insertions(+) diff --git a/cmds/receive-dump.c b/cmds/receive-dump.c index 20ec2b70..acc0ba32 100644 --- a/cmds/receive-dump.c +++ b/cmds/receive-dump.c @@ -329,6 +329,14 @@ static int print_encoded_write(const char *path, const void *data, u64 offset, unencoded_offset, compression, encryption); } +static int print_fallocate(const char *path, int mode, u64 offset, u64 len, + void *user) +{ + return PRINT_DUMP(user, path, "fallocate", + "mode=%d offset=%llu len=%llu", + mode, offset, len); +} + struct btrfs_send_ops btrfs_print_send_ops = { .subvol = print_subvol, .snapshot = print_snapshot, @@ -352,4 +360,5 @@ struct btrfs_send_ops btrfs_print_send_ops = { .utimes = print_utimes, .update_extent = print_update_extent, .encoded_write = print_encoded_write, + .fallocate = print_fallocate, }; diff --git a/cmds/receive.c b/cmds/receive.c index 361c7c8a..ca9c3f5a 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -1357,6 +1357,30 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, compression); } +static int process_fallocate(const char *path, int mode, u64 offset, u64 len, + void *user) +{ + int ret; + struct btrfs_receive *rctx = user; + char full_path[PATH_MAX]; + + ret = path_cat_out(full_path, rctx->full_subvol_path, path); + if (ret < 0) { + error("fallocate: path invalid: %s", path); + return ret; + } + ret = open_inode_for_write(rctx, full_path, false); + if (ret < 0) + return ret; + ret = fallocate(rctx->write_fd, mode, offset, len); + if (ret < 0) { + ret = -errno; + error("fallocate: fallocate on %s failed: %m", path); + return ret; + } + return 0; +} + static struct btrfs_send_ops send_ops = { .subvol = process_subvol, .snapshot = process_snapshot, @@ -1380,6 +1404,7 @@ static struct btrfs_send_ops send_ops = { .utimes = process_utimes, .update_extent = process_update_extent, .encoded_write = process_encoded_write, + .fallocate = process_fallocate, }; static int do_receive(struct btrfs_receive *rctx, const char *tomnt, diff --git a/common/send-stream.c b/common/send-stream.c index 1376e00b..d455cdfb 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -369,6 +369,7 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) u64 unencoded_offset; int len; int xattr_len; + int fallocate_mode; ret = read_cmd(sctx); if (ret) @@ -514,6 +515,14 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) case BTRFS_SEND_C_END: ret = 1; break; + case BTRFS_SEND_C_FALLOCATE: + TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); + TLV_GET_U32(sctx, BTRFS_SEND_A_FALLOCATE_MODE, &fallocate_mode); + TLV_GET_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, &offset); + TLV_GET_U64(sctx, BTRFS_SEND_A_SIZE, &tmp); + ret = sctx->ops->fallocate(path, fallocate_mode, offset, tmp, + sctx->user); + break; } tlv_get_failed: diff --git a/common/send-stream.h b/common/send-stream.h index 607bc007..a58739bb 100644 --- a/common/send-stream.h +++ b/common/send-stream.h @@ -70,6 +70,8 @@ struct btrfs_send_ops { u64 len, u64 unencoded_file_len, u64 unencoded_len, u64 unencoded_offset, u32 compression, u32 encryption, void *user); + int (*fallocate)(const char *path, int mode, u64 offset, u64 len, + void *user); }; int btrfs_read_and_process_send_stream(int fd, From patchwork Wed Nov 18 19:18:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915725 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2368C63798 for ; Wed, 18 Nov 2020 19:19:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 67A1822227 for ; Wed, 18 Nov 2020 19:19:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="ctJn0jyR" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727375AbgKRTTo (ORCPT ); Wed, 18 Nov 2020 14:19:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727363AbgKRTTn (ORCPT ); Wed, 18 Nov 2020 14:19:43 -0500 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D896AC0613D4 for ; Wed, 18 Nov 2020 11:19:43 -0800 (PST) Received: by mail-pl1-x644.google.com with SMTP id j5so1531615plk.7 for ; Wed, 18 Nov 2020 11:19:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YMykUAF1jBHiR3hoO57Gb2atrtMf3A6zHbbf8nZiQAw=; b=ctJn0jyRQWnOpDMJwzJ+5wGIlsnQ3VJ3TXetiT77zEVZfeZInXDICwBHskcwPaagfw hVFzGIgVWpn/RmBRX3h+aVFw4RBIaLTXiMBWXUP0Pbb9VJttVxZuBwkUofvTQBbzVOgz Kct6Hqn5DOOCDvLZw1BEHq3M9yE8NW0PIpiunBFr4tWy31Zz/S5wczAJcdR2eHrNtyaB V3rR9mK5JV/NaVSjDhsX4Qki6XSsWcY3lY1WCPTrVuSxwqaEMbIxnBqRUAZCzUkWlIGs sDCcta50NGCyweNXAWog7CT4ioEj8vmkYsz98Y5K4Xn8bOj5F2rgUWIjojE5CAtLtt0o Mxog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YMykUAF1jBHiR3hoO57Gb2atrtMf3A6zHbbf8nZiQAw=; b=mmX4sRmzNfJB8RouAiv7jaonIXi2LiTqqmto/aaBX5KXi84oKQvcoPTsXk1mgptXdf JHAXHM4VKHKNNNW3JnvXwcPcCG0A70YM7PC2eg3RAa9pX3ZkKUnzmDao6y6IYSPOIl6G 0oGrVycSp53MaSTfuj/gtfMJnkuPpgpc96xAGwdIXQHv/YdkCUM5YfjuXolJDrRIioym rYJTCh5k/1Jgf1K9/ZIY4LjMXnPN/9r6neUVXUSyQ3sbG7sJnEMao0ABGyjN0yrbLT18 6Irl840kYbOPyup1cXnQf9Sl50sG8lI5x7dxJzrMirCS+oDeyagJA9Ckd3YndwRBnpF6 Mmxw== X-Gm-Message-State: AOAM533CypetC8mYyq8KyaiwwqX6jKTbXOslK+h2gVZDNn3NgkqS3API uqWj3DMyY3XtAtcmGbL15FMXZ9oRy3HrTw== X-Google-Smtp-Source: ABdhPJyjyV39XZbRXV15OJ4+YJbkxyshR9NPZsrXzqNxm89Fj2aeHfQnMcTtwS6DMunK8J7vfAGITQ== X-Received: by 2002:a17:902:9b93:b029:d6:d0e3:10c6 with SMTP id y19-20020a1709029b93b02900d6d0e310c6mr5855545plp.41.1605727183352; Wed, 18 Nov 2020 11:19:43 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:42 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 11/13] btrfs-progs: receive: process setflags ioctl commands Date: Wed, 18 Nov 2020 11:18:58 -0800 Message-Id: <40b7cf2fb2602cc6029eb36c15eab361ac4e5340.1605723745.git.osandov@osandov.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Boris Burkov In send stream v2, send can emit a command for setting inode flags via the setflags ioctl. Pass the flags attribute through to the ioctl call in receive. Signed-off-by: Boris Burkov --- cmds/receive-dump.c | 6 ++++++ cmds/receive.c | 24 ++++++++++++++++++++++++ common/send-stream.c | 7 +++++++ common/send-stream.h | 1 + 4 files changed, 38 insertions(+) diff --git a/cmds/receive-dump.c b/cmds/receive-dump.c index acc0ba32..40f07ad4 100644 --- a/cmds/receive-dump.c +++ b/cmds/receive-dump.c @@ -337,6 +337,11 @@ static int print_fallocate(const char *path, int mode, u64 offset, u64 len, mode, offset, len); } +static int print_setflags(const char *path, int flags, void *user) +{ + return PRINT_DUMP(user, path, "setflags", "flags=%d", flags); +} + struct btrfs_send_ops btrfs_print_send_ops = { .subvol = print_subvol, .snapshot = print_snapshot, @@ -361,4 +366,5 @@ struct btrfs_send_ops btrfs_print_send_ops = { .update_extent = print_update_extent, .encoded_write = print_encoded_write, .fallocate = print_fallocate, + .setflags = print_setflags, }; diff --git a/cmds/receive.c b/cmds/receive.c index ca9c3f5a..1863f881 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -1381,6 +1381,29 @@ static int process_fallocate(const char *path, int mode, u64 offset, u64 len, return 0; } +static int process_setflags(const char *path, int flags, void *user) +{ + int ret; + struct btrfs_receive *rctx = user; + char full_path[PATH_MAX]; + + ret = path_cat_out(full_path, rctx->full_subvol_path, path); + if (ret < 0) { + error("setflags: path invalid: %s", path); + return ret; + } + ret = open_inode_for_write(rctx, full_path, false); + if (ret < 0) + return ret; + ret = ioctl(rctx->write_fd, FS_IOC_SETFLAGS, &flags); + if (ret < 0) { + ret = -errno; + error("setflags: setflags ioctl on %s failed: %m", path); + return ret; + } + return 0; +} + static struct btrfs_send_ops send_ops = { .subvol = process_subvol, .snapshot = process_snapshot, @@ -1405,6 +1428,7 @@ static struct btrfs_send_ops send_ops = { .update_extent = process_update_extent, .encoded_write = process_encoded_write, .fallocate = process_fallocate, + .setflags = process_setflags, }; static int do_receive(struct btrfs_receive *rctx, const char *tomnt, diff --git a/common/send-stream.c b/common/send-stream.c index d455cdfb..da0c0e5d 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -370,6 +370,7 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) int len; int xattr_len; int fallocate_mode; + int setflags_flags; ret = read_cmd(sctx); if (ret) @@ -523,8 +524,14 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) ret = sctx->ops->fallocate(path, fallocate_mode, offset, tmp, sctx->user); break; + case BTRFS_SEND_C_SETFLAGS: + TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); + TLV_GET_U32(sctx, BTRFS_SEND_A_SETFLAGS_FLAGS, &setflags_flags); + ret = sctx->ops->setflags(path, setflags_flags, sctx->user); + break; } + tlv_get_failed: out: free(path); diff --git a/common/send-stream.h b/common/send-stream.h index a58739bb..5373bf69 100644 --- a/common/send-stream.h +++ b/common/send-stream.h @@ -72,6 +72,7 @@ struct btrfs_send_ops { u32 encryption, void *user); int (*fallocate)(const char *path, int mode, u64 offset, u64 len, void *user); + int (*setflags)(const char *path, int flags, void *user); }; int btrfs_read_and_process_send_stream(int fd, From patchwork Wed Nov 18 19:18:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915723 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62507C5519F for ; Wed, 18 Nov 2020 19:19:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 04EAE22228 for ; Wed, 18 Nov 2020 19:19:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="Uz9ORVnV" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726511AbgKRTTr (ORCPT ); Wed, 18 Nov 2020 14:19:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727376AbgKRTTq (ORCPT ); Wed, 18 Nov 2020 14:19:46 -0500 Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73A1FC0613D4 for ; Wed, 18 Nov 2020 11:19:46 -0800 (PST) Received: by mail-pl1-x643.google.com with SMTP id bj5so718190plb.4 for ; Wed, 18 Nov 2020 11:19:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=andgHX9wkZy2Trx0HRVKpuTygds8q/VC0NacIobtvWI=; b=Uz9ORVnVyGjkjVWIwMCuxG++lNNP+7VuE0dYpsFjPP38DN+VEoWSvf5IUHHinpw9ag 4wvpqvlQ9/uDVPXXepKmA2bcoE4scjaqma4iEiMGFSxIGcVOMruqWmpwrgskRUVHPBPM A6xclirhrFkYNnCzfEDX1UG6i/BZAZAb6MXCkPucbfGnwdtCPtDVjKHnEKkGxFkevleP r9JE13vLdDzQkYW5zrKOkKSe3Ke1LEj8UQ3mDh82ACZIkbMXr+j67R+zkvBr6otR9mVE Tl7UaRuucQNJbOk4/RL8F0RYyigcOAG0hbKfuYpal1gRntLXfXXvfyH20TVBvWRcPEZ/ T6zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=andgHX9wkZy2Trx0HRVKpuTygds8q/VC0NacIobtvWI=; b=U6n9sSNAC0vB6HiAluhmx2KETW2NZmXW8/SglKGNx3zwIuI9GyI6NXWcGyp/NcSK8d cyj5b0siSt7sOYjBP2hhwuyKqK2HMcQ3FIbdQT5VkceoC6RbEm8Y8q5mGGQbVZGNVagv gQl1PBXctLNHP9hYeP8FKG8HyAfiu7L5RL4mclfWiDGEu/M4joiFydzidbAGWVW6C59G D48sFcGSP85sg7vnSPGv8zWnPQ6VrEFXfzgOKwxopzAnCe7qp+S1TsGZHfaD6cgPDmZG OOlVo2hFaisg9sIUV7Ae7CNpqNxZ0DCwKATvQea5kLSDFM9ZXW5TCMYl5aTk70Y1aDht wKkQ== X-Gm-Message-State: AOAM533jVnvQUldKsnaDwVOPoIm0rMn2Z04iPo6SbKHG8NV/DSbM2W9o SsvPaEbLHmOtUur1BARhaGrVpA== X-Google-Smtp-Source: ABdhPJz8l0hqY52ZeX8nSamIt+o44kFCYTONvLjz7am2GGLBPhpxeTEMBov/ijXykagRnWrPM5GUCQ== X-Received: by 2002:a17:902:7606:b029:d4:c797:a186 with SMTP id k6-20020a1709027606b02900d4c797a186mr5340154pll.38.1605727185903; Wed, 18 Nov 2020 11:19:45 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:44 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 12/13] btrfs-progs: send: stream v2 ioctl flags Date: Wed, 18 Nov 2020 11:18:59 -0800 Message-Id: <340dcc141473301e5811b6c1eb6963a056118d0c.1605723745.git.osandov@osandov.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Boris Burkov To make the btrfs send ioctl use the stream v2 format requires passing BTRFS_SEND_FLAG_STREAM_V2 in flags. Further, to cause the ioctl to emit encoded_write commands for encoded extents, we must set that flag as well as BTRFS_SEND_FLAG_COMPRESSED. Finally, we bump up the version in send.h as well, since we are now fully compatible with v2. Add two command line arguments to btrfs send: --stream-version and --compressed-data. --stream-version requires an argument which it parses as an integer and sets STREAM_V2 if the argument is 2. --compressed-data does not require an argument and automatically implies STREAM_V2 as well (COMPRESSED alone causes the ioctl to error out). Some examples to illustrate edge cases: // v1, old format and no encoded_writes btrfs send subvol btrfs send --stream-version 1 subvol // v2 and compressed, we will see encoded_writes btrfs send --compressed-data subvol btrfs send --compressed-data --stream-version 2 subvol // v2 only, new format but no encoded_writes btrfs send --stream-version 2 subvol // error: compressed needs version >= 2 btrfs send --compressed-data --stream-version 1 subvol // error: invalid version (not 1 or 2) btrfs send --stream-version 3 subvol btrfs send --compressed-data --stream-version 0 subvol btrfs send --compressed-data --stream-version 10 subvol Signed-off-by: Boris Burkov --- Documentation/btrfs-send.asciidoc | 16 ++++++++- cmds/send.c | 54 ++++++++++++++++++++++++++++++- ioctl.h | 17 +++++++++- libbtrfsutil/btrfs.h | 17 +++++++++- send.h | 2 +- 5 files changed, 101 insertions(+), 5 deletions(-) diff --git a/Documentation/btrfs-send.asciidoc b/Documentation/btrfs-send.asciidoc index c4a05672..202bcd97 100644 --- a/Documentation/btrfs-send.asciidoc +++ b/Documentation/btrfs-send.asciidoc @@ -55,7 +55,21 @@ send in 'NO_FILE_DATA' mode The output stream does not contain any file data and thus cannot be used to transfer changes. This mode is faster and is useful to show the differences in metadata. --q|--quiet:::: + +--stream-version <1|2>:: +Use the given send stream version. The default is 1. Version 2 encodes file +data slightly more efficiently; it is also required for sending compressed data +directly (see '--compressed-data'). Version 2 requires at least btrfs-progs +5.12 on both the sender and receiver and at least Linux 5.12 on the sender. + +--compressed-data:: +Send data that is compressed on the filesystem directly without decompressing +it. If the receiver supports encoded I/O (see `encoded_io`(7)), it can also +write it directly without decompressing it. Otherwise, the receiver will fall +back to decompressing it and writing it normally. This implies +'--stream-version 2'. + +-q|--quiet:: (deprecated) alias for global '-q' option -v|--verbose:: (deprecated) alias for global '-v' option diff --git a/cmds/send.c b/cmds/send.c index 3bfc69f5..80eb2510 100644 --- a/cmds/send.c +++ b/cmds/send.c @@ -452,6 +452,21 @@ static const char * const cmd_send_usage[] = { " does not contain any file data and thus cannot be used", " to transfer changes. This mode is faster and useful to", " show the differences in metadata.", + "--stream-version <1|2>", + " Use the given send stream version. The default is", + " 1. Version 2 encodes file data slightly more", + " efficiently; it is also required for sending", + " compressed data directly (see --compressed-data).", + " Version 2 requires at least btrfs-progs 5.12 on both", + " the sender and receiver and at least Linux 5.12 on the", + " sender.", + "--compressed-data", + " Send data that is compressed on the filesystem", + " directly without decompressing it. If the receiver", + " supports encoded I/O, it can also write it directly", + " without decompressing it. Otherwise, the receiver will", + " fall back to decompressing it and writing it normally.", + " This implies --stream-version 2.", "-v|--verbose deprecated, alias for global -v option", "-q|--quiet deprecated, alias for global -q option", HELPINFO_INSERT_GLOBALS, @@ -463,6 +478,7 @@ static const char * const cmd_send_usage[] = { static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) { char *subvol = NULL; + char *end; int ret; char outname[PATH_MAX]; struct btrfs_send send; @@ -474,6 +490,7 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) int full_send = 1; int new_end_cmd_semantic = 0; u64 send_flags = 0; + long stream_version = 0; memset(&send, 0, sizeof(send)); send.dump_fd = fileno(stdout); @@ -492,11 +509,17 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) optind = 0; while (1) { - enum { GETOPT_VAL_SEND_NO_DATA = 256 }; + enum { + GETOPT_VAL_SEND_NO_DATA = 256, + GETOPT_VAL_SEND_STREAM_V2, + GETOPT_VAL_SEND_COMPRESSED_DATA, + }; static const struct option long_options[] = { { "verbose", no_argument, NULL, 'v' }, { "quiet", no_argument, NULL, 'q' }, { "no-data", no_argument, NULL, GETOPT_VAL_SEND_NO_DATA }, + { "stream-version", required_argument, NULL, GETOPT_VAL_SEND_STREAM_V2 }, + { "compressed-data", no_argument, NULL, GETOPT_VAL_SEND_COMPRESSED_DATA }, { NULL, 0, NULL, 0 } }; int c = getopt_long(argc, argv, "vqec:f:i:p:", long_options, NULL); @@ -585,10 +608,39 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) case GETOPT_VAL_SEND_NO_DATA: send_flags |= BTRFS_SEND_FLAG_NO_FILE_DATA; break; + case GETOPT_VAL_SEND_STREAM_V2: + stream_version = strtol(optarg, &end, 10); + if (*end != '\0' || + stream_version < 1 || stream_version > 2) { + ret = 1; + error("invalid --stream-version. valid values: {1, 2}"); + goto out; + } + if (stream_version == 2) + send_flags |= BTRFS_SEND_FLAG_STREAM_V2; + break; + case GETOPT_VAL_SEND_COMPRESSED_DATA: + send_flags |= BTRFS_SEND_FLAG_COMPRESSED; + /* + * We want to default to stream v2 if only compressed is + * set. If stream_version is explicitly set to 0, that + * will trigger its own error condition for being an + * invalid version. + */ + if (stream_version == 0) { + stream_version = 2; + send_flags |= BTRFS_SEND_FLAG_STREAM_V2; + } + break; default: usage_unknown_option(cmd, argv); } } + if (stream_version < 2 && (send_flags & BTRFS_SEND_FLAG_COMPRESSED)) { + ret = 1; + error("--compressed requires --stream-version >= 2"); + goto out; + } if (check_argc_min(argc - optind, 1)) return 1; diff --git a/ioctl.h b/ioctl.h index ade6dcb9..46de8ac8 100644 --- a/ioctl.h +++ b/ioctl.h @@ -653,10 +653,25 @@ BUILD_ASSERT(sizeof(struct btrfs_ioctl_received_subvol_args_32) == 192); */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ diff --git a/libbtrfsutil/btrfs.h b/libbtrfsutil/btrfs.h index 60d51ff6..8430a40d 100644 --- a/libbtrfsutil/btrfs.h +++ b/libbtrfsutil/btrfs.h @@ -731,10 +731,25 @@ struct btrfs_ioctl_received_subvol_args { */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ diff --git a/send.h b/send.h index 3c47e0c7..fac90588 100644 --- a/send.h +++ b/send.h @@ -31,7 +31,7 @@ extern "C" { #endif #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" -#define BTRFS_SEND_STREAM_VERSION 1 +#define BTRFS_SEND_STREAM_VERSION 2 #define BTRFS_SEND_BUF_SIZE_V1 SZ_64K #define BTRFS_SEND_READ_SIZE (1024 * 48) From patchwork Wed Nov 18 19:19:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82703C63798 for ; Wed, 18 Nov 2020 19:20:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 38F0D22227 for ; Wed, 18 Nov 2020 19:20:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="g9Pgdk80" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727408AbgKRTTt (ORCPT ); Wed, 18 Nov 2020 14:19:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727395AbgKRTTs (ORCPT ); Wed, 18 Nov 2020 14:19:48 -0500 Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44BE2C0613D4 for ; Wed, 18 Nov 2020 11:19:48 -0800 (PST) Received: by mail-pf1-x42d.google.com with SMTP id 131so2035611pfb.9 for ; Wed, 18 Nov 2020 11:19:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qASfdMugLX6dkJNB+g4DB9CS4PoJpLhsrJsEHm5JPHo=; b=g9Pgdk80zIKJmvTMpsBJ8RyqvR6ihfHqH60Twyb9UJwXyNRCAMOC2GbyKrKUy527mR iSg5zoJNfey4DZoUsFxzXjslAhiH1JIghrUVj435BWFNsjAKxc6XW/IqlR990B1EqHoU g3o0xkNgIPCxn4ijuiWumP+wXwF/5q2HC7J/nTk9RL9Wkrrs4Wp6MHtuFKvmWHiu53CN cCFnq2kk1fKjtC38KqsTNqbC5NoU/Ek0EriZ/8aL+tg0/olB9vuvJ3HvpfFkqEwWZ1Zv ROLAoVhI9gr82YZsnYlCEjSK6YcYDDOU7HOJj11qlBYbGJMOXr1GyTWufFY42VvH+okJ efeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qASfdMugLX6dkJNB+g4DB9CS4PoJpLhsrJsEHm5JPHo=; b=cvzQ7oU9N6ch0br9wC/cab7JbhLgZLC8RbUjMizdCr+M8virLLS+OY5G10XEkU2tVB +4rJT6JdhrKfhs+OFfd0UaIDEgUkw02zd9kML5kDI2l6j5w8JQZhpZvTaoR1enypvY19 HTKe+JW7BjQuDdKvQFut3T4tNsHYMX4rfwYZtOuQcgCug09o0/rsKfK34LnEowggYWV6 Gh2TJaOKwa+D8iRHDr8vWtTQHacabpkwn59m34V02op8cN3xZDw2NjWGf2Ct9n9+fSgg DFy/WDWQpLFF6d/xueuqPFqJneQg3OWQjCZjMin6FR8yMBNEhF9pN6KcT5iJ/TnQHpVU AaVg== X-Gm-Message-State: AOAM5301JziiYuvpIGwwNaL7dkdKKWq/1viK8xHRq63WW5jJDSfoVpI5 +heZ15nxBEVGZw+aoH7dk4S/Rw== X-Google-Smtp-Source: ABdhPJxZ7ymPheEqg1BwDPCXVWYQJR2jOlrQETu1Ux49c10JuOuG4RR1rxx/X3E8SOPzKEs4gwW2lA== X-Received: by 2002:a63:fc1c:: with SMTP id j28mr9106892pgi.95.1605727187768; Wed, 18 Nov 2020 11:19:47 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:46 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 13/13] btrfs-progs: receive: add tests for basic encoded_write send/receive Date: Wed, 18 Nov 2020 11:19:00 -0800 Message-Id: X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Boris Burkov Adapt the existing send/receive tests by passing '-o --force-compress' to the mount commands in a new test. After writing a few files in the various compression formats, send/receive them with and without --force-decompress to test both the encoded_write path and the fallback to decode+write. Signed-off-by: Boris Burkov --- .../042-receive-write-encoded/test.sh | 114 ++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100755 tests/misc-tests/042-receive-write-encoded/test.sh diff --git a/tests/misc-tests/042-receive-write-encoded/test.sh b/tests/misc-tests/042-receive-write-encoded/test.sh new file mode 100755 index 00000000..b9390e88 --- /dev/null +++ b/tests/misc-tests/042-receive-write-encoded/test.sh @@ -0,0 +1,114 @@ +#!/bin/bash +# +# test that we can send and receive encoded writes for three modes of +# transparent compression: zlib, lzo, and zstd. + +source "$TEST_TOP/common" + +check_prereq mkfs.btrfs +check_prereq btrfs + +setup_root_helper +prepare_test_dev + +here=`pwd` + +# assumes the filesystem exists, and does mount, write, snapshot, send, unmount +# for the specified encoding option +send_one() { + local str + local subv + local snap + + algorithm="$1" + shift + str="$1" + shift + + subv="subv-$algorithm" + snap="snap-$algorithm" + + run_check_mount_test_dev "-o" "compress-force=$algorithm" + cd "$TEST_MNT" || _fail "cannot chdir to TEST_MNT" + + run_check $SUDO_HELPER "$TOP/btrfs" subvolume create "$subv" + run_check $SUDO_HELPER dd if=/dev/zero of="$subv/file1" bs=1M count=1 + run_check $SUDO_HELPER dd if=/dev/zero of="$subv/file2" bs=500K count=1 + run_check $SUDO_HELPER "$TOP/btrfs" subvolume snapshot -r "$subv" "$snap" + run_check $SUDO_HELPER "$TOP/btrfs" send -f "$str" "$snap" "$@" + + cd "$here" || _fail "cannot chdir back to test directory" + run_check_umount_test_dev +} + +receive_one() { + local str + str="$1" + shift + + run_check_mkfs_test_dev + run_check_mount_test_dev + run_check $SUDO_HELPER "$TOP/btrfs" receive "$@" -v -f "$str" "$TEST_MNT" + run_check_umount_test_dev + run_check rm -f -- "$str" +} + +test_one_write_encoded() { + local str + local algorithm + algorithm="$1" + shift + str="$here/stream-$algorithm.stream" + + run_check_mkfs_test_dev + send_one "$algorithm" "$str" --compressed-data + receive_one "$str" "$@" +} + +test_one_stream_v1() { + local str + local algorithm + algorithm="$1" + shift + str="$here/stream-$algorithm.stream" + + run_check_mkfs_test_dev + send_one "$algorithm" "$str" --stream-version 1 + receive_one "$str" "$@" +} + +test_mix_write_encoded() { + local strzlib + local strlzo + local strzstd + strzlib="$here/stream-zlib.stream" + strlzo="$here/stream-lzo.stream" + strzstd="$here/stream-zstd.stream" + + run_check_mkfs_test_dev + + send_one "zlib" "$strzlib" --compressed-data + send_one "lzo" "$strlzo" --compressed-data + send_one "zstd" "$strzstd" --compressed-data + + receive_one "$strzlib" + receive_one "$strlzo" + receive_one "$strzstd" +} + +test_one_write_encoded "zlib" +test_one_write_encoded "lzo" +test_one_write_encoded "zstd" + +# with decompression forced +test_one_write_encoded "zlib" "--force-decompress" +test_one_write_encoded "lzo" "--force-decompress" +test_one_write_encoded "zstd" "--force-decompress" + +# send stream v1 +test_one_stream_v1 "zlib" +test_one_stream_v1 "lzo" +test_one_stream_v1 "zstd" + +# files use a mix of compression algorithms +test_mix_write_encoded