From patchwork Thu Mar 17 17:25:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12784359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34485C4332F for ; Thu, 17 Mar 2022 17:26:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236826AbiCQR1Z (ORCPT ); Thu, 17 Mar 2022 13:27:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54488 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230168AbiCQR1Y (ORCPT ); Thu, 17 Mar 2022 13:27:24 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E0495114FC9 for ; Thu, 17 Mar 2022 10:26:07 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id mz9-20020a17090b378900b001c657559290so5860867pjb.2 for ; Thu, 17 Mar 2022 10:26:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jf1xcgurY+im0KaPf7PvoqQR+BSeNSxrsqeoOhcfrkk=; b=Yi+9GpIuJvJs5KlfTtLg5XhEneji0geyrYVS0PErYGrC9hW1vlWUhxnhONsGqsgXgy 7xJ9jcy/pK/c2EZaval+ZtyJRu106QQFEEFF+iQX4RDU6ho8an6Dpqr+6GUujQpKwb7R UIG43O+vFTUdxhHBxnrE6WoU+x/bLfraCTIA+JngQml/yJRldLffdWNBZ6JlCvpia8mu zLhRuxcAmPqHqCeRAhoeCocQ4/O+UvEYrY3gx7u9N1cxrBhNR1OodfGC9RKzYLRDTIhI aQevED9INssIRJrDiGIsxKdkIhT1J3DcG2T7zaumEUQGXvDf+eihN3/vYYXadp8f/7Gc CJYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jf1xcgurY+im0KaPf7PvoqQR+BSeNSxrsqeoOhcfrkk=; b=aQI8fMUk3f8krfUYpYdtYAFTCfgr2vUmHT2AifQxzVF/EW04jwMg6YfmugeT4lf+7e Y563+jxc7MV3IC1+3757z9N+FYLwLNEJ7qysrzG2Q+WGI7bZ0Xn9P5onI5yhNiU1MOdW X7qt86zcFvFpa377K7TACr/yOqdnyV+6E3/0m10OiO4BFHA5EKnc9J8y/aVGKkoD2lx7 lgm8qeO/v+QFZ+xW6Wdhaq2FyPQSxrvhoITBWEqVe/wQ1ymRa7GZhRoQwDSyiD55BqMN 6hHCOcW0+vjyggcn0RnYB6scP0gcxvrIQ8K9Ikx0Oe/K/3g5PL2KnekqqbY8zG47ma0i yqPw== X-Gm-Message-State: AOAM530cTwhMSza8mfSGTlsNKi1flxhUNXOea+i8ZPf5tfep50S4Ecgf tDKmkf+e5gFHG+G4Uus00ZpjPaOo09BunQ== X-Google-Smtp-Source: ABdhPJy/ntVhxtNZewL9c3b912HL6JknQ2JX0mdHVq0qC9Zo+4r5rPfbZ9xCjEnGKR3sUFqXHoVW1w== X-Received: by 2002:a17:90b:3145:b0:1bc:5855:f94d with SMTP id ip5-20020a17090b314500b001bc5855f94dmr6617224pjb.55.1647537967081; Thu, 17 Mar 2022 10:26:07 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:624e]) by smtp.gmail.com with ESMTPSA id q10-20020a056a00088a00b004f7ceff389esm7815424pfj.152.2022.03.17.10.26.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Mar 2022 10:26:06 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: kernel-team@fb.com Subject: [PATCH v14 1/7] btrfs: send: remove unused send_ctx::{total,cmd}_send_size Date: Thu, 17 Mar 2022 10:25:37 -0700 Message-Id: X-Mailer: git-send-email 2.35.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval We collect these statistics but have never exposed them in any way. I also didn't find any patches that ever attempted to make use of them. Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index cf86f1eafcb7..6d36dee1505f 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -82,8 +82,6 @@ struct send_ctx { char *send_buf; u32 send_size; u32 send_max_size; - u64 total_send_size; - u64 cmd_send_size[BTRFS_SEND_C_MAX + 1]; u64 flags; /* 'flags' member of btrfs_ioctl_send_args is u64 */ /* Protocol version compatibility requested */ u32 proto; @@ -727,8 +725,6 @@ static int send_cmd(struct send_ctx *sctx) ret = write_buf(sctx->send_filp, sctx->send_buf, sctx->send_size, &sctx->send_off); - sctx->total_send_size += sctx->send_size; - sctx->cmd_send_size[get_unaligned_le16(&hdr->cmd)] += sctx->send_size; sctx->send_size = 0; return ret; From patchwork Thu Mar 17 17:25:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12784360 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8277AC433F5 for ; Thu, 17 Mar 2022 17:26:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236848AbiCQR13 (ORCPT ); Thu, 17 Mar 2022 13:27:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54518 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230168AbiCQR10 (ORCPT ); Thu, 17 Mar 2022 13:27:26 -0400 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07A5A114FC1 for ; Thu, 17 Mar 2022 10:26:09 -0700 (PDT) Received: by mail-pj1-x1029.google.com with SMTP id mm4-20020a17090b358400b001c68e836fa6so1712439pjb.3 for ; Thu, 17 Mar 2022 10:26:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PQT49Q1rZdF1CDHVKX7tZ94Ry6ex1JxOz+Fe5ZrpYs8=; b=X01umut7LF+VoQmza0yK3OrOJGvxpPEimK7JssIGH7CeQSS/beX8KJ/iZktvyrJIbI uisPxbPX4DRJVfYhJDw+X5aAh896Oefj7cN7N/ir1zvUQMsb+xYiOnKEyBAxVrknbKUl mN4XVXHKueL0ACnTfNH80UO3WHrhkbDlVh2SSCveZn2DE+/OOQYSHFSDXdotkpF95ioN jwRVYYyzjGnsmMpFalXyeXdtu5OYkw3djx8InQMIAXXV4A76/Jt/Jo25iMzjLklt44uN 43zN/4bI+75464xcnE+XLo8Q94xR5ueyx+/iA1v2EiR398VZye2zADhHRlcowtwslb8s wD/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PQT49Q1rZdF1CDHVKX7tZ94Ry6ex1JxOz+Fe5ZrpYs8=; b=blaK+vNU9Y9K4Hp39++Wf1kjjBkT5ZJwEtVSOnYsq7r23U0mD2kRgXk7Cqxse+kn5+ 3MxpClEDH18ulZKxNm1ybndVJMS/qeCgEVoDbHKW4Adx/QcRMfj2GduetM09Kv7fvoTE +A1qF5n7+KNm5mDBvN73wCTaSVAPBMx90q+YJ0Q68tgwCPrwaVZslCssZXiS6FN9+rfI uO4rkpJ035s4Bx3RY/uFWUO+tn4ISHDOKmQDk/RzdLMblyMcofeENKxuBzjCa5wq7tHL 1bjc50hmlG5TGFQUpr5TF12G0G6rM3cOEBFtLMs22shU+0X49VopBrjXMkyvkDTQkGNo nwCA== X-Gm-Message-State: AOAM530pnNcqzgsb7OyoyPXMmrnEVpYIK8qjLNvAxoAUU9wm6+Rqikw3 4Uc2AEcNCqUfnMtuXcgKTry7bfxRi9VXwA== X-Google-Smtp-Source: ABdhPJyjIgS7wnAbjQHE3qnJz459fglZRvgTHgfu50DE5I5KmY9ZNq38DmRCVfhGwSfJKO9Yq16nEg== X-Received: by 2002:a17:902:c407:b0:151:f794:ac5e with SMTP id k7-20020a170902c40700b00151f794ac5emr6147359plk.67.1647537968072; Thu, 17 Mar 2022 10:26:08 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:624e]) by smtp.gmail.com with ESMTPSA id q10-20020a056a00088a00b004f7ceff389esm7815424pfj.152.2022.03.17.10.26.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Mar 2022 10:26:07 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: kernel-team@fb.com Subject: [PATCH v14 2/7] btrfs: send: explicitly number commands and attributes Date: Thu, 17 Mar 2022 10:25:38 -0700 Message-Id: X-Mailer: git-send-email 2.35.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval Commit e77fbf990316 ("btrfs: send: prepare for v2 protocol") added _BTRFS_SEND_C_MAX_V* macros equal to the maximum command number for the version plus 1, but as written this creates gaps in the number space. The maximum command number is currently 22, and __BTRFS_SEND_C_MAX_V1 is accordingly 23. But then __BTRFS_SEND_C_MAX_V2 is 24, suggesting that v2 has a command numbered 23, and __BTRFS_SEND_C_MAX is 25, suggesting that 23 and 24 are valid commands. Instead, let's explicitly number all of the commands, attributes, and sentinel MAX constants. Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 4 +- fs/btrfs/send.h | 106 ++++++++++++++++++++++++------------------------ 2 files changed, 54 insertions(+), 56 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 6d36dee1505f..9363f625fa17 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -326,8 +326,8 @@ __maybe_unused static bool proto_cmd_ok(const struct send_ctx *sctx, int cmd) { switch (sctx->proto) { - case 1: return cmd < __BTRFS_SEND_C_MAX_V1; - case 2: return cmd < __BTRFS_SEND_C_MAX_V2; + case 1: return cmd <= BTRFS_SEND_C_MAX_V1; + case 2: return cmd <= BTRFS_SEND_C_MAX_V2; default: return false; } } diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index 08602fdd600a..67721e0281ba 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -46,84 +46,82 @@ struct btrfs_tlv_header { /* commands */ enum btrfs_send_cmd { - BTRFS_SEND_C_UNSPEC, + BTRFS_SEND_C_UNSPEC = 0, /* Version 1 */ - BTRFS_SEND_C_SUBVOL, - BTRFS_SEND_C_SNAPSHOT, + BTRFS_SEND_C_SUBVOL = 1, + BTRFS_SEND_C_SNAPSHOT = 2, - BTRFS_SEND_C_MKFILE, - BTRFS_SEND_C_MKDIR, - BTRFS_SEND_C_MKNOD, - BTRFS_SEND_C_MKFIFO, - BTRFS_SEND_C_MKSOCK, - BTRFS_SEND_C_SYMLINK, + BTRFS_SEND_C_MKFILE = 3, + BTRFS_SEND_C_MKDIR = 4, + BTRFS_SEND_C_MKNOD = 5, + BTRFS_SEND_C_MKFIFO = 6, + BTRFS_SEND_C_MKSOCK = 7, + BTRFS_SEND_C_SYMLINK = 8, - BTRFS_SEND_C_RENAME, - BTRFS_SEND_C_LINK, - BTRFS_SEND_C_UNLINK, - BTRFS_SEND_C_RMDIR, + BTRFS_SEND_C_RENAME = 9, + BTRFS_SEND_C_LINK = 10, + BTRFS_SEND_C_UNLINK = 11, + BTRFS_SEND_C_RMDIR = 12, - BTRFS_SEND_C_SET_XATTR, - BTRFS_SEND_C_REMOVE_XATTR, + BTRFS_SEND_C_SET_XATTR = 13, + BTRFS_SEND_C_REMOVE_XATTR = 14, - BTRFS_SEND_C_WRITE, - BTRFS_SEND_C_CLONE, + BTRFS_SEND_C_WRITE = 15, + BTRFS_SEND_C_CLONE = 16, - BTRFS_SEND_C_TRUNCATE, - BTRFS_SEND_C_CHMOD, - BTRFS_SEND_C_CHOWN, - BTRFS_SEND_C_UTIMES, + BTRFS_SEND_C_TRUNCATE = 17, + BTRFS_SEND_C_CHMOD = 18, + BTRFS_SEND_C_CHOWN = 19, + BTRFS_SEND_C_UTIMES = 20, - BTRFS_SEND_C_END, - BTRFS_SEND_C_UPDATE_EXTENT, - __BTRFS_SEND_C_MAX_V1, + BTRFS_SEND_C_END = 21, + BTRFS_SEND_C_UPDATE_EXTENT = 22, + BTRFS_SEND_C_MAX_V1 = 22, /* Version 2 */ - __BTRFS_SEND_C_MAX_V2, + BTRFS_SEND_C_MAX_V2 = 22, /* End */ - __BTRFS_SEND_C_MAX, + BTRFS_SEND_C_MAX = 22, }; -#define BTRFS_SEND_C_MAX (__BTRFS_SEND_C_MAX - 1) /* attributes in send stream */ enum { - BTRFS_SEND_A_UNSPEC, + BTRFS_SEND_A_UNSPEC = 0, - BTRFS_SEND_A_UUID, - BTRFS_SEND_A_CTRANSID, + BTRFS_SEND_A_UUID = 1, + BTRFS_SEND_A_CTRANSID = 2, - BTRFS_SEND_A_INO, - BTRFS_SEND_A_SIZE, - BTRFS_SEND_A_MODE, - BTRFS_SEND_A_UID, - BTRFS_SEND_A_GID, - BTRFS_SEND_A_RDEV, - BTRFS_SEND_A_CTIME, - BTRFS_SEND_A_MTIME, - BTRFS_SEND_A_ATIME, - BTRFS_SEND_A_OTIME, + BTRFS_SEND_A_INO = 3, + BTRFS_SEND_A_SIZE = 4, + BTRFS_SEND_A_MODE = 5, + BTRFS_SEND_A_UID = 6, + BTRFS_SEND_A_GID = 7, + BTRFS_SEND_A_RDEV = 8, + BTRFS_SEND_A_CTIME = 9, + BTRFS_SEND_A_MTIME = 10, + BTRFS_SEND_A_ATIME = 11, + BTRFS_SEND_A_OTIME = 12, - BTRFS_SEND_A_XATTR_NAME, - BTRFS_SEND_A_XATTR_DATA, + BTRFS_SEND_A_XATTR_NAME = 13, + BTRFS_SEND_A_XATTR_DATA = 14, - BTRFS_SEND_A_PATH, - BTRFS_SEND_A_PATH_TO, - BTRFS_SEND_A_PATH_LINK, + BTRFS_SEND_A_PATH = 15, + BTRFS_SEND_A_PATH_TO = 16, + BTRFS_SEND_A_PATH_LINK = 17, - BTRFS_SEND_A_FILE_OFFSET, - BTRFS_SEND_A_DATA, + BTRFS_SEND_A_FILE_OFFSET = 18, + BTRFS_SEND_A_DATA = 19, - BTRFS_SEND_A_CLONE_UUID, - BTRFS_SEND_A_CLONE_CTRANSID, - BTRFS_SEND_A_CLONE_PATH, - BTRFS_SEND_A_CLONE_OFFSET, - BTRFS_SEND_A_CLONE_LEN, + BTRFS_SEND_A_CLONE_UUID = 20, + BTRFS_SEND_A_CLONE_CTRANSID = 21, + BTRFS_SEND_A_CLONE_PATH = 22, + BTRFS_SEND_A_CLONE_OFFSET = 23, + BTRFS_SEND_A_CLONE_LEN = 24, - __BTRFS_SEND_A_MAX, + BTRFS_SEND_A_MAX = 24, }; -#define BTRFS_SEND_A_MAX (__BTRFS_SEND_A_MAX - 1) #ifdef __KERNEL__ long btrfs_ioctl_send(struct inode *inode, struct btrfs_ioctl_send_args *arg); From patchwork Thu Mar 17 17:25:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12784361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AA88C433EF for ; Thu, 17 Mar 2022 17:26:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236850AbiCQR13 (ORCPT ); Thu, 17 Mar 2022 13:27:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236840AbiCQR10 (ORCPT ); Thu, 17 Mar 2022 13:27:26 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 008B5114FC9 for ; Thu, 17 Mar 2022 10:26:09 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id z12-20020a17090ad78c00b001bf022b69d6so6097272pju.2 for ; Thu, 17 Mar 2022 10:26:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=q6mWpklEfCJElpbyli4O0vFR3E2bQpaD80/20rwpjwI=; b=gX6aIwz3vzUPGsvIOA7yh2C0YCkov1J+b/0PnVSr9Dhh0YsuiRmKrTU9KSEzNe2pWG uUWM4BQIOgCj6GN8j2ZSmPh/V/0uCQwJuTl6nqcpLFz0JuMy1j90pqGqjmy5yA9M8f4i wjRWp/B5DdvQAvsy4w5qvnDGjlmANjHbAcNMmqp4NgAFN8+Wqvu0zdtBoH3K9YBZGakH yP6y6TPoh8rpijpord+1OlTNPiHzlmEiTYbhkZ/ttKhR/zJET6NihBNV//nZoVbTDpCQ TFCByGjJrEY+LRfZZG4IjCfZUehdEcf27tbtPjggMAfIQKIvUn3RfmWqZZesyRRXxu1J 9NPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=q6mWpklEfCJElpbyli4O0vFR3E2bQpaD80/20rwpjwI=; b=Vm1Q5J+Kc1Ob0S+MJhR9L68akiWYScwqi8rISpLXBcMdwGmgrf+tpLiV38DBY29/gZ nKeCSiWtQcf88lO5EP3PwXusQ7kJXZTNa7Q6Kn+4FaBRsprPe8xgYE6WKjmNydpYzKgK PeqqTxmL9mj49XQCTvAR4do/tbY+1GNm7kSRdxM7UT8P+7GjdQgwOB9axBHGbl/SIrT/ SzvAmumMooyTLuH9wHWXzvrJZ9Pv1ADpmGcOcRKOaXViZWrImQM1x42z4PBGbwEgulQa ZCNTca3uYmTv74baaVa/OdBosQD5JG4juY4qIydqWBfYtEA4uWZ+yrr+UcdQqAsrwKBF 4PPw== X-Gm-Message-State: AOAM532htAqzdejyTr5cXsv6g/QSs6vk8SeX3l+/waJ2SYQEdPcRnea7 WNcWKdzYMXKb9g9VqYxyYnMizaLAxaPZAw== X-Google-Smtp-Source: ABdhPJwU1GOwtGgCbtKDoHIXX8HUI3920aAKSmUqj2gj3YT1TbNPVkj5zDFWDDx6oHjdu58+ILtuvQ== X-Received: by 2002:a17:902:7045:b0:153:a8b1:d666 with SMTP id h5-20020a170902704500b00153a8b1d666mr6224874plt.75.1647537969038; Thu, 17 Mar 2022 10:26:09 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:624e]) by smtp.gmail.com with ESMTPSA id q10-20020a056a00088a00b004f7ceff389esm7815424pfj.152.2022.03.17.10.26.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Mar 2022 10:26:08 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: kernel-team@fb.com Subject: [PATCH v14 3/7] btrfs: add send stream v2 definitions Date: Thu, 17 Mar 2022 10:25:39 -0700 Message-Id: <40148edfb135002d33242471a87d4e7dc860b48c.1647537027.git.osandov@fb.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval This adds the definitions of the new commands for send stream version 2 and their respective attributes: fallocate, FS_IOC_SETFLAGS (a.k.a. chattr), and encoded writes. It also documents two changes to the send stream format in v2: the receiver shouldn't assume a maximum command size, and the DATA attribute is encoded differently to allow for writes larger than 64k. These will be implemented in subsequent changes, and then the ioctl will accept the new version and flag. Reviewed-by: Josef Bacik Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 2 +- fs/btrfs/send.h | 40 ++++++++++++++++++++++++++++++++++---- include/uapi/linux/btrfs.h | 7 +++++++ 3 files changed, 44 insertions(+), 5 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 9363f625fa17..1f141de3a7d6 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -7459,7 +7459,7 @@ long btrfs_ioctl_send(struct inode *inode, struct btrfs_ioctl_send_args *arg) sctx->clone_roots_cnt = arg->clone_sources_count; - sctx->send_max_size = BTRFS_SEND_BUF_SIZE; + sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); if (!sctx->send_buf) { ret = -ENOMEM; diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index 67721e0281ba..805d8095209a 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -12,7 +12,11 @@ #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" #define BTRFS_SEND_STREAM_VERSION 1 -#define BTRFS_SEND_BUF_SIZE SZ_64K +/* + * In send stream v1, no command is larger than 64k. In send stream v2, no limit + * should be assumed. + */ +#define BTRFS_SEND_BUF_SIZE_V1 SZ_64K enum btrfs_tlv_type { BTRFS_TLV_U8, @@ -80,16 +84,20 @@ enum btrfs_send_cmd { BTRFS_SEND_C_MAX_V1 = 22, /* Version 2 */ - BTRFS_SEND_C_MAX_V2 = 22, + BTRFS_SEND_C_FALLOCATE = 23, + BTRFS_SEND_C_SETFLAGS = 24, + BTRFS_SEND_C_ENCODED_WRITE = 25, + BTRFS_SEND_C_MAX_V2 = 25, /* End */ - BTRFS_SEND_C_MAX = 22, + BTRFS_SEND_C_MAX = 25, }; /* attributes in send stream */ enum { BTRFS_SEND_A_UNSPEC = 0, + /* Version 1 */ BTRFS_SEND_A_UUID = 1, BTRFS_SEND_A_CTRANSID = 2, @@ -112,6 +120,11 @@ enum { BTRFS_SEND_A_PATH_LINK = 17, BTRFS_SEND_A_FILE_OFFSET = 18, + /* + * As of send stream v2, this attribute is special: it must be the last + * attribute in a command, its header contains only the type, and its + * length is implicitly the remaining length of the command. + */ BTRFS_SEND_A_DATA = 19, BTRFS_SEND_A_CLONE_UUID = 20, @@ -120,7 +133,26 @@ enum { BTRFS_SEND_A_CLONE_OFFSET = 23, BTRFS_SEND_A_CLONE_LEN = 24, - BTRFS_SEND_A_MAX = 24, + BTRFS_SEND_A_MAX_V1 = 24, + + /* Version 2 */ + BTRFS_SEND_A_FALLOCATE_MODE = 25, + + BTRFS_SEND_A_SETFLAGS_FLAGS = 26, + + BTRFS_SEND_A_UNENCODED_FILE_LEN = 27, + BTRFS_SEND_A_UNENCODED_LEN = 28, + BTRFS_SEND_A_UNENCODED_OFFSET = 29, + /* + * COMPRESSION and ENCRYPTION default to NONE (0) if omitted from + * BTRFS_SEND_C_ENCODED_WRITE. + */ + BTRFS_SEND_A_COMPRESSION = 30, + BTRFS_SEND_A_ENCRYPTION = 31, + BTRFS_SEND_A_MAX_V2 = 31, + + /* End */ + BTRFS_SEND_A_MAX = 31, }; #ifdef __KERNEL__ diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h index d956b2993970..b6f26a434b10 100644 --- a/include/uapi/linux/btrfs.h +++ b/include/uapi/linux/btrfs.h @@ -777,6 +777,13 @@ struct btrfs_ioctl_received_subvol_args { */ #define BTRFS_SEND_FLAG_VERSION 0x8 +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires protocol + * version >= 2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ From patchwork Thu Mar 17 17:25:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12784363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2C28C433FE for ; Thu, 17 Mar 2022 17:26:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236851AbiCQR1a (ORCPT ); Thu, 17 Mar 2022 13:27:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236846AbiCQR11 (ORCPT ); Thu, 17 Mar 2022 13:27:27 -0400 Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2A0E114FC8 for ; Thu, 17 Mar 2022 10:26:10 -0700 (PDT) Received: by mail-pj1-x1035.google.com with SMTP id l4-20020a17090a49c400b001c6840df4a3so2283321pjm.0 for ; Thu, 17 Mar 2022 10:26:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=khOJrdrDEWIV7HJtp0T3it85P1f2rHhG80yKCamxkTk=; b=ROBCBpTXXlO0lM4lyfnymfBEhTVCuXuKihXSb3ry5oV22pMzFGT26ikvhmRlwmvMhy trNAcsKkNF4wGyrVsQT0Vonn/MTSWIkrEpRDmaYrWcemT43ucLJ+PA9w7y/MI9iRzU5f iK+6LLWE/OWzdB25xfl+PaPWvug1SZKCKvn/7xw1GDH18UtKOCxX3qX8oVwbKrJpZxtm 30ao6mCgdLKaLPh43KHbJIfZDaxyu/bn9z2PdNY4vubPTH+/4TY4iMJVKPFItMpWI8tK Jm8mhiP3Z72Gjsik7ESS++gVlkLoiGOu8xhbHlNYlWasv1CyjLyRlDCJtr4OBxUh+WdJ 58Yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=khOJrdrDEWIV7HJtp0T3it85P1f2rHhG80yKCamxkTk=; b=A1f7gVswDWNhf/1tUfdB+G1sIrn6IcJJGg4DjknFMlx0Hhw4ChFLj/b0lJ+xn0/W/j khAY7pzMGZWm3DAJ5LAX/VKkgZghCH3eXDNst4HfNSVCiaU7yoaoyE+8RekJq3xIEm61 tue37ImNsKqxY4XuSZJ6ifErftT5MyavoAfJ2YLLdjYWiRn/cKQM3ep7y1XrloXJubGs krs5+RTyZdB0J2dHd+T8nZ3Czb2+Pe386KMS4h587JdigZqy8y8hE+3UnUmfCwf0ux0j askhszZbMECAGfb2VEOU72bwpRh+8j96E7igPuYYF7lvD54nfbiLPE9lKllJYK5I2/N/ VUgw== X-Gm-Message-State: AOAM530zp71cwgOsEkYlzX2hcVfFno3Zp0ckAN/9QK+xk6sf+HQefL4m udRDgcBVSwn92KRJ8LW27q9vHzFbl3/oUA== X-Google-Smtp-Source: ABdhPJyknfhUTrVMbQkNAz0MNRtZ3y/Vpfz1KZMRb+xVNr3Q5sVco3R9GPeyhfFeXkSOS52gML3Wzg== X-Received: by 2002:a17:902:d501:b0:153:4a6a:52de with SMTP id b1-20020a170902d50100b001534a6a52demr5939456plg.100.1647537970051; Thu, 17 Mar 2022 10:26:10 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:624e]) by smtp.gmail.com with ESMTPSA id q10-20020a056a00088a00b004f7ceff389esm7815424pfj.152.2022.03.17.10.26.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Mar 2022 10:26:09 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: kernel-team@fb.com Subject: [PATCH v14 4/7] btrfs: send: write larger chunks when using stream v2 Date: Thu, 17 Mar 2022 10:25:40 -0700 Message-Id: <2ac307b0d20f84a09302d9d4f7c4fa1207edccc7.1647537027.git.osandov@fb.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval The length field of the send stream TLV header is 16 bits. This means that the maximum amount of data that can be sent for one write is 64k minus one. However, encoded writes must be able to send the maximum compressed extent (128k) in one command. To support this, send stream version 2 encodes the DATA attribute differently: it has no length field, and the length is implicitly up to the end of containing command (which has a 32-bit length field). Although this is necessary for encoded writes, normal writes can benefit from it, too. Also add a check to enforce that the DATA attribute is last. It is only strictly necessary for v2, but we might as well make v1 consistent with it. For v2, let's bump up the send buffer to the maximum compressed extent size plus 16k for the other metadata (144k total). Since this will most likely be vmalloc'd (and always will be after the next commit), we round it up to the next page since we might as well use the rest of the page on systems with >16k pages. Reviewed-by: Nikolay Borisov Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 42 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 34 insertions(+), 8 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 1f141de3a7d6..02053fff80ca 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -82,6 +82,7 @@ struct send_ctx { char *send_buf; u32 send_size; u32 send_max_size; + bool put_data; u64 flags; /* 'flags' member of btrfs_ioctl_send_args is u64 */ /* Protocol version compatibility requested */ u32 proto; @@ -589,6 +590,9 @@ static int tlv_put(struct send_ctx *sctx, u16 attr, const void *data, int len) int total_len = sizeof(*hdr) + len; int left = sctx->send_max_size - sctx->send_size; + if (WARN_ON_ONCE(sctx->put_data)) + return -EINVAL; + if (unlikely(left < total_len)) return -EOVERFLOW; @@ -726,6 +730,7 @@ static int send_cmd(struct send_ctx *sctx) &sctx->send_off); sctx->send_size = 0; + sctx->put_data = false; return ret; } @@ -4853,14 +4858,30 @@ static inline u64 max_send_read_size(const struct send_ctx *sctx) static int put_data_header(struct send_ctx *sctx, u32 len) { - struct btrfs_tlv_header *hdr; + if (WARN_ON_ONCE(sctx->put_data)) + return -EINVAL; + sctx->put_data = true; + if (sctx->proto >= 2) { + /* + * In v2, the data attribute header doesn't include a length; it + * is implicitly to the end of the command. + */ + if (sctx->send_max_size - sctx->send_size < 2 + len) + return -EOVERFLOW; + put_unaligned_le16(BTRFS_SEND_A_DATA, + sctx->send_buf + sctx->send_size); + sctx->send_size += 2; + } else { + struct btrfs_tlv_header *hdr; - if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len) - return -EOVERFLOW; - hdr = (struct btrfs_tlv_header *)(sctx->send_buf + sctx->send_size); - put_unaligned_le16(BTRFS_SEND_A_DATA, &hdr->tlv_type); - put_unaligned_le16(len, &hdr->tlv_len); - sctx->send_size += sizeof(*hdr); + if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len) + return -EOVERFLOW; + hdr = (struct btrfs_tlv_header *)(sctx->send_buf + + sctx->send_size); + put_unaligned_le16(BTRFS_SEND_A_DATA, &hdr->tlv_type); + put_unaligned_le16(len, &hdr->tlv_len); + sctx->send_size += sizeof(*hdr); + } return 0; } @@ -7459,7 +7480,12 @@ long btrfs_ioctl_send(struct inode *inode, struct btrfs_ioctl_send_args *arg) sctx->clone_roots_cnt = arg->clone_sources_count; - sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; + if (sctx->proto >= 2) { + sctx->send_max_size = ALIGN(SZ_16K + BTRFS_MAX_COMPRESSED, + PAGE_SIZE); + } else { + sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; + } sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); if (!sctx->send_buf) { ret = -ENOMEM; From patchwork Thu Mar 17 17:25:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12784362 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61077C4332F for ; Thu, 17 Mar 2022 17:26:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236846AbiCQR1b (ORCPT ); Thu, 17 Mar 2022 13:27:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236834AbiCQR12 (ORCPT ); Thu, 17 Mar 2022 13:27:28 -0400 Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CA470114FCA for ; Thu, 17 Mar 2022 10:26:11 -0700 (PDT) Received: by mail-pl1-x62b.google.com with SMTP id e13so5026352plh.3 for ; Thu, 17 Mar 2022 10:26:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5Ema1oqCVoL3SOLFCC5gLkP1k9e610J2xmQvzR4oaQA=; b=0UoTKvukDlHDjNVtAdLAwMKKAyahmOxmrFPmBiZvnbHb0p26PALHtjf5xlZQ4obSkS onaddbon6yz0bqQsm1e9iV+kheWeZ3H4Hb+Kx4IHXW+Jih8rE2COWd1oeX9dOOBx+8lQ jgDNyvj56VmxQ1AORVTDLdaPfhlgPE+fn32uH/J4sFBT27T9z07IbLHkvNYX78n1kCqm jgg5ibTG53iTU5o9TX4bcRShvtTk0+dLndju7yk8S2jA6lv+jXcnO4vJbm7FFRg+KpE8 jondBMZKdKXwdd1UHQtUp6mFiFoU6hedEzpFx5ifVwRJPjzuAqugr4U0Gom+xuU+hnOb uFEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5Ema1oqCVoL3SOLFCC5gLkP1k9e610J2xmQvzR4oaQA=; b=oOyzhCO0neHj+nb8QQPyXVqrEWYRxUmzwUciMijLlUmn7Ew5cVW6JRI8tutUHWnDVQ fPxfAIGeE8FjoBdZEtWAFH7sJ+HUu02XbT0FhN/abvQ+rlX4zDaMqVGVXHUfxd6sF5ty zKbQADnAUKiPF/SB0hw9XgbiIOYQeTmrQO00eUQ3nCfCwuFYT/dOrSF4Qk1AgfCz7N4E lGmZWPFXjNFbK7IwQJS6iwvAXFR/nE+e0yh9qvzomghfXpfptv8mA2muoUpUA4ISXU51 C6hMwERhg7JfHPzJMaouEDplqiw0QKQRVZLFGPYgDMrk9A3Keq4xfzY90OfzBzbt5rkH 5pCA== X-Gm-Message-State: AOAM532Q6IV1j2NOOUm1Kt6v3+KbUogjYT07ajM1e0ubzOgN/CzfZmm3 x7sEiuLiYvZEsyWA9YeTNKYq7Pj2Kj2vgw== X-Google-Smtp-Source: ABdhPJxqey2tPftKrTdw4XtKJFIDnQGvOFdGlQO94D+N7W9Qb7Ann3aR+54W6eWnUVxz2ndAL7DGkQ== X-Received: by 2002:a17:90a:3906:b0:1bf:a0a6:d208 with SMTP id y6-20020a17090a390600b001bfa0a6d208mr16967877pjb.21.1647537970952; Thu, 17 Mar 2022 10:26:10 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:624e]) by smtp.gmail.com with ESMTPSA id q10-20020a056a00088a00b004f7ceff389esm7815424pfj.152.2022.03.17.10.26.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Mar 2022 10:26:10 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: kernel-team@fb.com Subject: [PATCH v14 5/7] btrfs: send: allocate send buffer with alloc_page() and vmap() for v2 Date: Thu, 17 Mar 2022 10:25:41 -0700 Message-Id: <4353fe7122eb0aae24e3c9ff2399f2b58b74f79e.1647537027.git.osandov@fb.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval For encoded writes, we need the raw pages for reading compressed data directly via a bio. So, replace kvmalloc() with vmap() so we have access to the raw pages. 144k is large enough that it usually gets allocated with vmalloc(), anyways. Reviewed-by: Nikolay Borisov Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 02053fff80ca..ac2a1297027a 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -83,6 +83,7 @@ struct send_ctx { u32 send_size; u32 send_max_size; bool put_data; + struct page **send_buf_pages; u64 flags; /* 'flags' member of btrfs_ioctl_send_args is u64 */ /* Protocol version compatibility requested */ u32 proto; @@ -7392,6 +7393,7 @@ long btrfs_ioctl_send(struct inode *inode, struct btrfs_ioctl_send_args *arg) struct btrfs_root *clone_root; struct send_ctx *sctx = NULL; u32 i; + u32 send_buf_num_pages = 0; u64 *clone_sources_tmp = NULL; int clone_sources_to_rollback = 0; size_t alloc_size; @@ -7483,10 +7485,28 @@ long btrfs_ioctl_send(struct inode *inode, struct btrfs_ioctl_send_args *arg) if (sctx->proto >= 2) { sctx->send_max_size = ALIGN(SZ_16K + BTRFS_MAX_COMPRESSED, PAGE_SIZE); + send_buf_num_pages = sctx->send_max_size >> PAGE_SHIFT; + sctx->send_buf_pages = kcalloc(send_buf_num_pages, + sizeof(*sctx->send_buf_pages), + GFP_KERNEL); + if (!sctx->send_buf_pages) { + send_buf_num_pages = 0; + ret = -ENOMEM; + goto out; + } + for (i = 0; i < send_buf_num_pages; i++) { + sctx->send_buf_pages[i] = alloc_page(GFP_KERNEL); + if (!sctx->send_buf_pages[i]) { + ret = -ENOMEM; + goto out; + } + } + sctx->send_buf = vmap(sctx->send_buf_pages, send_buf_num_pages, + VM_MAP, PAGE_KERNEL); } else { sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; + sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); } - sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); if (!sctx->send_buf) { ret = -ENOMEM; goto out; @@ -7679,7 +7699,16 @@ long btrfs_ioctl_send(struct inode *inode, struct btrfs_ioctl_send_args *arg) fput(sctx->send_filp); kvfree(sctx->clone_roots); - kvfree(sctx->send_buf); + if (sctx->proto >= 2) { + vunmap(sctx->send_buf); + for (i = 0; i < send_buf_num_pages; i++) { + if (sctx->send_buf_pages[i]) + __free_page(sctx->send_buf_pages[i]); + } + kfree(sctx->send_buf_pages); + } else { + kvfree(sctx->send_buf); + } name_cache_free(sctx); From patchwork Thu Mar 17 17:25:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12784365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63D57C433F5 for ; Thu, 17 Mar 2022 17:26:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236856AbiCQR1b (ORCPT ); Thu, 17 Mar 2022 13:27:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54574 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230168AbiCQR13 (ORCPT ); Thu, 17 Mar 2022 13:27:29 -0400 Received: from mail-pg1-x532.google.com (mail-pg1-x532.google.com [IPv6:2607:f8b0:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EAB5D114FC1 for ; Thu, 17 Mar 2022 10:26:12 -0700 (PDT) Received: by mail-pg1-x532.google.com with SMTP id 6so3158053pgg.0 for ; Thu, 17 Mar 2022 10:26:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5jxmnUqzbF2hZh7PkiPw555y1MidL2Q1O9KnbDhHysc=; b=VCYR5uC7RDWnXMn0SZS82K9g4jiaxcBW7f8HzSzFYB251wYQ9dfgJzeAd3gsqb2mU1 MX0ffTa3YOwF3k7x1AXa1r5WCdkIprFAZJJkb9buFCA+85Qq2MJzixnDPf1IF95x+FJ7 nr3eFafwwXxpOKIhaP6hBol1FPH0qQSuQ09f24zumvXB6fcidespU8igo+zWOhj8XnTw wZBFoQBe5WmxrnoPGT8a5R9fOgDFDQXKRlLU5ueqBpI5qeaiEIxrXpIb9Ie6vsEE+8vL gro3QjX/gpCespRHNAQ2PFHQJIXP1ZsD//ef6QXziVFWPaYm5VcAUe8qsEW/ic4diPsI TsPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5jxmnUqzbF2hZh7PkiPw555y1MidL2Q1O9KnbDhHysc=; b=gosza+8I+Fe0cykm0F0z7ViqdpZRyRsOsqWYUkIVs3lp6U3iGIun75XmMbPCKnZFFD 3nLWQuq+fNn6QIyyA/F0hDncA95cVhh2q3Cy+wdT47VD46GZTkxHzGIHYeq3ILdDkYQl eODtJBTQIT3L5uBr1gc5D5sUSiq1/1nqtXyxIi50cgG0KedCMqZrWJWFpcznVF6Wpw8i sOViLKfCg5+P9iwVHUcKlG0OS0dk0IdX0NJtGBn1dJADVloDlUepA1breqoVARZejDGi 52JuUJzydecL/7+ENZNjZeRNJF+tOoZYZlzIDDbIcIe3qjOHJ81JGj7WN4b5JG82zZ3j T/gg== X-Gm-Message-State: AOAM5337rCBPUIAk6w2ribmiV0yfUnxNyENMW20IFmUh0ClIZPztQ0jO SjShzvG3sOnEyZOm/BfcQSVYtOzXUhEzgA== X-Google-Smtp-Source: ABdhPJzd2M9BBjGyAHcPw3uC8/77NhDq0R6DPxVO4dQfjIbqhqjlVXyYZ7pp4hV/1GNIA76drMTdqw== X-Received: by 2002:aa7:968b:0:b0:4f6:fa51:c239 with SMTP id f11-20020aa7968b000000b004f6fa51c239mr5537258pfk.48.1647537971969; Thu, 17 Mar 2022 10:26:11 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:624e]) by smtp.gmail.com with ESMTPSA id q10-20020a056a00088a00b004f7ceff389esm7815424pfj.152.2022.03.17.10.26.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Mar 2022 10:26:11 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: kernel-team@fb.com Subject: [PATCH v14 6/7] btrfs: send: send compressed extents with encoded writes Date: Thu, 17 Mar 2022 10:25:42 -0700 Message-Id: X-Mailer: git-send-email 2.35.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval Now that all of the pieces are in place, we can use the ENCODED_WRITE command to send compressed extents when appropriate. Signed-off-by: Omar Sandoval --- fs/btrfs/ctree.h | 6 ++ fs/btrfs/inode.c | 13 +-- fs/btrfs/send.c | 234 +++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 228 insertions(+), 25 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 09b7b0b2d016..92bfee5b3ded 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -3355,6 +3355,12 @@ int btrfs_writepage_cow_fixup(struct page *page); void btrfs_writepage_endio_finish_ordered(struct btrfs_inode *inode, struct page *page, u64 start, u64 end, bool uptodate); +int btrfs_encoded_io_compression_from_extent(struct btrfs_fs_info *fs_info, + int compress_type); +int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode, + u64 file_offset, u64 disk_bytenr, + u64 disk_io_size, + struct page **pages); ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter, struct btrfs_ioctl_encoded_io_args *encoded); ssize_t btrfs_do_encoded_write(struct kiocb *iocb, struct iov_iter *from, diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 78a5145353e1..6b8c0f026545 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -10114,9 +10114,8 @@ void btrfs_set_range_writeback(struct btrfs_inode *inode, u64 start, u64 end) } } -static int btrfs_encoded_io_compression_from_extent( - struct btrfs_fs_info *fs_info, - int compress_type) +int btrfs_encoded_io_compression_from_extent(struct btrfs_fs_info *fs_info, + int compress_type) { switch (compress_type) { case BTRFS_COMPRESS_NONE: @@ -10321,11 +10320,9 @@ static void btrfs_encoded_read_endio(struct bio *bio) bio_put(bio); } -static int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode, - u64 file_offset, - u64 disk_bytenr, - u64 disk_io_size, - struct page **pages) +int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode, + u64 file_offset, u64 disk_bytenr, + u64 disk_io_size, struct page **pages) { struct btrfs_fs_info *fs_info = inode->root->fs_info; struct btrfs_encoded_read_private priv = { diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index ac2a1297027a..b0560be3053b 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -614,6 +614,7 @@ static int tlv_put(struct send_ctx *sctx, u16 attr, const void *data, int len) return tlv_put(sctx, attr, &__tmp, sizeof(__tmp)); \ } +TLV_PUT_DEFINE_INT(32) TLV_PUT_DEFINE_INT(64) static int tlv_put_string(struct send_ctx *sctx, u16 attr, @@ -5160,16 +5161,215 @@ static int send_hole(struct send_ctx *sctx, u64 end) return ret; } -static int send_extent_data(struct send_ctx *sctx, - const u64 offset, - const u64 len) +static int send_encoded_inline_extent(struct send_ctx *sctx, + struct btrfs_path *path, u64 offset, + u64 len) { + struct btrfs_root *root = sctx->send_root; + struct btrfs_fs_info *fs_info = root->fs_info; + struct inode *inode; + struct fs_path *p; + struct extent_buffer *leaf = path->nodes[0]; + struct btrfs_key key; + struct btrfs_file_extent_item *ei; + u64 ram_bytes; + size_t inline_size; + int ret; + + inode = btrfs_iget(fs_info->sb, sctx->cur_ino, root); + if (IS_ERR(inode)) + return PTR_ERR(inode); + + p = fs_path_alloc(); + if (!p) { + ret = -ENOMEM; + goto out; + } + + ret = begin_cmd(sctx, BTRFS_SEND_C_ENCODED_WRITE); + if (ret < 0) + goto out; + + ret = get_cur_path(sctx, sctx->cur_ino, sctx->cur_inode_gen, p); + if (ret < 0) + goto out; + + btrfs_item_key_to_cpu(leaf, &key, path->slots[0]); + ei = btrfs_item_ptr(leaf, path->slots[0], + struct btrfs_file_extent_item); + ram_bytes = btrfs_file_extent_ram_bytes(leaf, ei); + inline_size = btrfs_file_extent_inline_item_len(leaf, path->slots[0]); + + TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p); + TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_FILE_LEN, + min(key.offset + ram_bytes - offset, len)); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_LEN, ram_bytes); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_OFFSET, offset - key.offset); + ret = btrfs_encoded_io_compression_from_extent(fs_info, + btrfs_file_extent_compression(leaf, ei)); + if (ret < 0) + goto out; + TLV_PUT_U32(sctx, BTRFS_SEND_A_COMPRESSION, ret); + + ret = put_data_header(sctx, inline_size); + if (ret < 0) + goto out; + read_extent_buffer(leaf, sctx->send_buf + sctx->send_size, + btrfs_file_extent_inline_start(ei), inline_size); + sctx->send_size += inline_size; + + ret = send_cmd(sctx); + +tlv_put_failure: +out: + fs_path_free(p); + iput(inode); + return ret; +} + +static int send_encoded_extent(struct send_ctx *sctx, struct btrfs_path *path, + u64 offset, u64 len) +{ + struct btrfs_root *root = sctx->send_root; + struct btrfs_fs_info *fs_info = root->fs_info; + struct inode *inode; + struct fs_path *p; + struct extent_buffer *leaf = path->nodes[0]; + struct btrfs_key key; + struct btrfs_file_extent_item *ei; + u64 disk_bytenr, disk_num_bytes; + u32 data_offset; + struct btrfs_cmd_header *hdr; + u32 crc; + int ret; + + inode = btrfs_iget(fs_info->sb, sctx->cur_ino, root); + if (IS_ERR(inode)) + return PTR_ERR(inode); + + p = fs_path_alloc(); + if (!p) { + ret = -ENOMEM; + goto out; + } + + ret = begin_cmd(sctx, BTRFS_SEND_C_ENCODED_WRITE); + if (ret < 0) + goto out; + + ret = get_cur_path(sctx, sctx->cur_ino, sctx->cur_inode_gen, p); + if (ret < 0) + goto out; + + btrfs_item_key_to_cpu(leaf, &key, path->slots[0]); + ei = btrfs_item_ptr(leaf, path->slots[0], + struct btrfs_file_extent_item); + disk_bytenr = btrfs_file_extent_disk_bytenr(leaf, ei); + disk_num_bytes = btrfs_file_extent_disk_num_bytes(leaf, ei); + + TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p); + TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_FILE_LEN, + min(key.offset + btrfs_file_extent_num_bytes(leaf, ei) - offset, + len)); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_LEN, + btrfs_file_extent_ram_bytes(leaf, ei)); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_OFFSET, + offset - key.offset + btrfs_file_extent_offset(leaf, ei)); + ret = btrfs_encoded_io_compression_from_extent(fs_info, + btrfs_file_extent_compression(leaf, ei)); + if (ret < 0) + goto out; + TLV_PUT_U32(sctx, BTRFS_SEND_A_COMPRESSION, ret); + TLV_PUT_U32(sctx, BTRFS_SEND_A_ENCRYPTION, 0); + + ret = put_data_header(sctx, disk_num_bytes); + if (ret < 0) + goto out; + + /* + * We want to do I/O directly into the send buffer, so get the next page + * boundary in the send buffer. This means that there may be a gap + * between the beginning of the command and the file data. + */ + data_offset = ALIGN(sctx->send_size, PAGE_SIZE); + if (data_offset > sctx->send_max_size || + sctx->send_max_size - data_offset < disk_num_bytes) { + ret = -EOVERFLOW; + goto out; + } + + /* + * Note that send_buf is a mapping of send_buf_pages, so this is really + * reading into send_buf. + */ + ret = btrfs_encoded_read_regular_fill_pages(BTRFS_I(inode), offset, + disk_bytenr, disk_num_bytes, + sctx->send_buf_pages + + (data_offset >> PAGE_SHIFT)); + if (ret) + goto out; + + hdr = (struct btrfs_cmd_header *)sctx->send_buf; + hdr->len = cpu_to_le32(sctx->send_size + disk_num_bytes - sizeof(*hdr)); + hdr->crc = 0; + crc = btrfs_crc32c(0, sctx->send_buf, sctx->send_size); + crc = btrfs_crc32c(crc, sctx->send_buf + data_offset, disk_num_bytes); + hdr->crc = cpu_to_le32(crc); + + ret = write_buf(sctx->send_filp, sctx->send_buf, sctx->send_size, + &sctx->send_off); + if (!ret) { + ret = write_buf(sctx->send_filp, sctx->send_buf + data_offset, + disk_num_bytes, &sctx->send_off); + } + sctx->send_size = 0; + sctx->put_data = false; + +tlv_put_failure: +out: + fs_path_free(p); + iput(inode); + return ret; +} + +static int send_extent_data(struct send_ctx *sctx, struct btrfs_path *path, + const u64 offset, const u64 len) +{ + struct extent_buffer *leaf = path->nodes[0]; + struct btrfs_file_extent_item *ei; u64 read_size = max_send_read_size(sctx); u64 sent = 0; if (sctx->flags & BTRFS_SEND_FLAG_NO_FILE_DATA) return send_update_extent(sctx, offset, len); + ei = btrfs_item_ptr(leaf, path->slots[0], + struct btrfs_file_extent_item); + if ((sctx->flags & BTRFS_SEND_FLAG_COMPRESSED) && + btrfs_file_extent_compression(leaf, ei) != BTRFS_COMPRESS_NONE) { + bool is_inline = (btrfs_file_extent_type(leaf, ei) == + BTRFS_FILE_EXTENT_INLINE); + + /* + * Send the compressed extent unless the compressed data is + * larger than the decompressed data. This can happen if we're + * not sending the entire extent, either because it has been + * partially overwritten/truncated or because this is a part of + * the extent that we couldn't clone in clone_range(). + */ + if (is_inline && + btrfs_file_extent_inline_item_len(leaf, + path->slots[0]) <= len) { + return send_encoded_inline_extent(sctx, path, offset, + len); + } else if (!is_inline && + btrfs_file_extent_disk_num_bytes(leaf, ei) <= len) { + return send_encoded_extent(sctx, path, offset, len); + } + } + while (sent < len) { u64 size = min(len - sent, read_size); int ret; @@ -5240,12 +5440,9 @@ static int send_capabilities(struct send_ctx *sctx) return ret; } -static int clone_range(struct send_ctx *sctx, - struct clone_root *clone_root, - const u64 disk_byte, - u64 data_offset, - u64 offset, - u64 len) +static int clone_range(struct send_ctx *sctx, struct btrfs_path *dst_path, + struct clone_root *clone_root, const u64 disk_byte, + u64 data_offset, u64 offset, u64 len) { struct btrfs_path *path; struct btrfs_key key; @@ -5269,7 +5466,7 @@ static int clone_range(struct send_ctx *sctx, */ if (clone_root->offset == 0 && len == sctx->send_root->fs_info->sectorsize) - return send_extent_data(sctx, offset, len); + return send_extent_data(sctx, dst_path, offset, len); path = alloc_path_for_send(); if (!path) @@ -5366,7 +5563,8 @@ static int clone_range(struct send_ctx *sctx, if (hole_len > len) hole_len = len; - ret = send_extent_data(sctx, offset, hole_len); + ret = send_extent_data(sctx, dst_path, offset, + hole_len); if (ret < 0) goto out; @@ -5439,14 +5637,16 @@ static int clone_range(struct send_ctx *sctx, if (ret < 0) goto out; } - ret = send_extent_data(sctx, offset + slen, + ret = send_extent_data(sctx, dst_path, + offset + slen, clone_len - slen); } else { ret = send_clone(sctx, offset, clone_len, clone_root); } } else { - ret = send_extent_data(sctx, offset, clone_len); + ret = send_extent_data(sctx, dst_path, offset, + clone_len); } if (ret < 0) @@ -5478,7 +5678,7 @@ static int clone_range(struct send_ctx *sctx, } if (len > 0) - ret = send_extent_data(sctx, offset, len); + ret = send_extent_data(sctx, dst_path, offset, len); else ret = 0; out: @@ -5509,10 +5709,10 @@ static int send_write_or_clone(struct send_ctx *sctx, struct btrfs_file_extent_item); disk_byte = btrfs_file_extent_disk_bytenr(path->nodes[0], ei); data_offset = btrfs_file_extent_offset(path->nodes[0], ei); - ret = clone_range(sctx, clone_root, disk_byte, data_offset, - offset, end - offset); + ret = clone_range(sctx, path, clone_root, disk_byte, + data_offset, offset, end - offset); } else { - ret = send_extent_data(sctx, offset, end - offset); + ret = send_extent_data(sctx, path, offset, end - offset); } sctx->cur_inode_next_write_offset = end; return ret; From patchwork Thu Mar 17 17:25:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12784364 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DA48C433EF for ; Thu, 17 Mar 2022 17:26:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236811AbiCQR1c (ORCPT ); Thu, 17 Mar 2022 13:27:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236853AbiCQR1a (ORCPT ); Thu, 17 Mar 2022 13:27:30 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC87D114FCA for ; Thu, 17 Mar 2022 10:26:13 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id mp6-20020a17090b190600b001c6841b8a52so2275084pjb.5 for ; Thu, 17 Mar 2022 10:26:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Z2OFoefPGJQ+WHOMM+btFk0/kfeAvOV06RDV8glzYuA=; b=1gJ6IDIcG1NZ+YC5e2JtsLNGdmPKgpSyO9BZ7UIrVHH8HI4gUwoZUdoGCgx8k2a4lV liLVcGR/EH/B91xxwShAByPSK/RdjMX2/Qqu6IcPeQ9mDnDO0UjqT7vvNyIMLTx4YVB+ BEPmSO1RQ9mI++mY3iW7RrOq5NQ+R+RjWsRoGyvbC4lW584EpHBnkxe7lF3tFQ7Zkg95 LheHgOFvnE2bOgJf6INoEAlLZMVmzZDk8T/Ku+v58GqskhuKNPhbYdmt9n0/rimk/MVc PQryPsF8r5nv4lBxtdVmTQpI1cwifRbUG4K9fmsETu0iZH/FDBQjQNEiR0LfeogTWw4V TJlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Z2OFoefPGJQ+WHOMM+btFk0/kfeAvOV06RDV8glzYuA=; b=gko8ttpdFOPMzwIyU3gb3fwYkzs6UdTfg8IBsade3Tl4dDL0VViLSnUpbOTWOkpwi/ cLRuiOmaXMPImIn6TZh10awRnCCJN+ihcGmoV0PLw9QBYiOh4ZaEd74DlPiA4mv9Qplx /fcvY/VgJJF00hliDlC+VV0RWB0BXqv8JSM7wyc9Nxoqd2L4b3F2JV0NMViaa/NZXd1l 9gd3mKZL2upIu8/uZ4h2Rmy5dGxq4u553HwoPhwrs/il08ye3Tk5jQnEM3vvXLlC//QX M9Eqt3T+q/i8rnSPdoypxnlLTbQxfgzh+71nMCAhCtxzfD9rEqrUJB4EM6mMVh36afIU L/hw== X-Gm-Message-State: AOAM531/l3RQBDSFIx0LPoysedQZRRAQ1pdc2CBIDguHWsPQcNzV2/zB P+MnHVaSL/nO5NJwgpszXNKuIlQRE6nWuQ== X-Google-Smtp-Source: ABdhPJwZrlS0/YmC/faPOEbDTW23MNGqcv59XtblFmAmyQOimLLnMK9JaZ/sTYeIVboa9knARMsDdQ== X-Received: by 2002:a17:902:ecc2:b0:151:dd64:c77a with SMTP id a2-20020a170902ecc200b00151dd64c77amr5963564plh.154.1647537972913; Thu, 17 Mar 2022 10:26:12 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:624e]) by smtp.gmail.com with ESMTPSA id q10-20020a056a00088a00b004f7ceff389esm7815424pfj.152.2022.03.17.10.26.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Mar 2022 10:26:12 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: kernel-team@fb.com Subject: [PATCH v14 7/7] btrfs: send: enable support for stream v2 and compressed writes Date: Thu, 17 Mar 2022 10:25:43 -0700 Message-Id: <9473f6be9faadaea779f76bd9ef29dc638a58f46.1647537027.git.osandov@fb.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval Now that the new support is implemented, allow the ioctl to accept v2 and the compressed flag, and update the version in sysfs. Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 7 +++++-- fs/btrfs/send.h | 2 +- include/uapi/linux/btrfs.h | 3 ++- 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index b0560be3053b..4567271ce642 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -690,8 +690,7 @@ static int send_header(struct send_ctx *sctx) struct btrfs_stream_header hdr; strcpy(hdr.magic, BTRFS_SEND_STREAM_MAGIC); - hdr.version = cpu_to_le32(BTRFS_SEND_STREAM_VERSION); - + hdr.version = cpu_to_le32(sctx->proto); return write_buf(sctx->send_filp, &hdr, sizeof(hdr), &sctx->send_off); } @@ -7663,6 +7662,10 @@ long btrfs_ioctl_send(struct inode *inode, struct btrfs_ioctl_send_args *arg) } else { sctx->proto = 1; } + if ((arg->flags & BTRFS_SEND_FLAG_COMPRESSED) && sctx->proto < 2) { + ret = -EINVAL; + goto out; + } sctx->send_filp = fget(arg->send_fd); if (!sctx->send_filp) { diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index 805d8095209a..50a2aceae929 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -10,7 +10,7 @@ #include "ctree.h" #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" -#define BTRFS_SEND_STREAM_VERSION 1 +#define BTRFS_SEND_STREAM_VERSION 2 /* * In send stream v1, no command is larger than 64k. In send stream v2, no limit diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h index b6f26a434b10..f54dc91e4025 100644 --- a/include/uapi/linux/btrfs.h +++ b/include/uapi/linux/btrfs.h @@ -788,7 +788,8 @@ struct btrfs_ioctl_received_subvol_args { (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ BTRFS_SEND_FLAG_OMIT_END_CMD | \ - BTRFS_SEND_FLAG_VERSION) + BTRFS_SEND_FLAG_VERSION | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ From patchwork Thu Mar 17 17:25:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12784373 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 893E4C433FE for ; Thu, 17 Mar 2022 17:26:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236869AbiCQR1i (ORCPT ); Thu, 17 Mar 2022 13:27:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236870AbiCQR1h (ORCPT ); Thu, 17 Mar 2022 13:27:37 -0400 Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 612C314DFE5 for ; Thu, 17 Mar 2022 10:26:21 -0700 (PDT) Received: by mail-pj1-x1034.google.com with SMTP id b8so5435698pjb.4 for ; Thu, 17 Mar 2022 10:26:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=N4Zc2VDwcN7WNUNPczgMrS6RYIapeQEbHJjjRCTXvSI=; b=3lO9WAU0II71mjlNVRHDpbb9L8LWK9QbcEUeh/sbYmZsFAtmkdVP3K/JFhtcSEX4Sn RsVRbCf1iRx325ddjWX5F8bFrCuptdeurb7FVXIv7a+iA/IIuxqdNAglEAAFM3BZd4cv wPqH806FTBYcsgX/6z3zzTiicAM1BBeCpM+11JA9+andTbjPZ58l7lFW1DzxmIemUa8R EAgGla0wWc2LwEACyYsvt15svyXSVPR+oM3LlJbNhucOrA1LESbLToSfWylI/G0GYhjT pkxvdVVLmxxy6l+j9Kx4qPTkTjmj9W7o7LoRaWXu3sunIVG7nTGr01DAa0KW4LqVNwVQ BJIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=N4Zc2VDwcN7WNUNPczgMrS6RYIapeQEbHJjjRCTXvSI=; b=WFuSOBv1zZCMZgeHTKCCbYJjAx6u+F3zK4MMpTTUIpD4Lth/ZAWwye7VtbtY0m4QMX /Esq1DRyfNBe6v2sBymcOUimmN3lJpr0NOKxUgAIMg7DsOuDhbLl+7h+IwW1JbiIqMSV cMXBIPWBbYRH3IbhpY+iUVvDIfyRK6DKLKsIHyXHBMQgLmsVrw9seJULwGm7MBxH4G7s 5bms/ioiyMJ93URxnPjux+M/X7ZBJUY4Iht2aaeU4E5QHE3ffXoiHan+5Siu9q8jcVWn H4vIIsUxc2Rnf5ncllp/fEhAlSOUujAhUo/MWpBT5gUEf1PlYM3RcIgaWKNkhYw06yli MhrA== X-Gm-Message-State: AOAM532uQZZO8PMrRKTWmyNM6NFWB8/bt/E7+jFIzC3zQ8V4NoBXBQyD lW6i0wO0TPE5EYR4NEIR7omvf+t0ej6PaA== X-Google-Smtp-Source: ABdhPJx3WA+g6YIxvZY89r+/2SvsJL9xQJ9DMRA4oqu0Nrgwv0rICn8VdiQ9VxKOmaaN+nayp99cVQ== X-Received: by 2002:a17:902:8bc2:b0:149:1ce6:c28c with SMTP id r2-20020a1709028bc200b001491ce6c28cmr5939137plo.164.1647537980554; Thu, 17 Mar 2022 10:26:20 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:624e]) by smtp.gmail.com with ESMTPSA id q10-20020a056a00088a00b004f7ceff389esm7815424pfj.152.2022.03.17.10.26.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Mar 2022 10:26:20 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: kernel-team@fb.com Subject: [PATCH v14 08/10] btrfs-progs: receive: process setflags ioctl commands Date: Thu, 17 Mar 2022 10:25:51 -0700 Message-Id: <7e92fa68eafefadb4f384b3b8cb2879b6edbcd89.1647537098.git.osandov@fb.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov In send stream v2, send can emit a command for setting inode flags via the setflags ioctl. Pass the flags attribute through to the ioctl call in receive. Reviewed-by: Nikolay Borisov Signed-off-by: Boris Burkov --- cmds/receive-dump.c | 6 ++++++ cmds/receive.c | 25 +++++++++++++++++++++++++ common/send-stream.c | 7 +++++++ common/send-stream.h | 1 + 4 files changed, 39 insertions(+) diff --git a/cmds/receive-dump.c b/cmds/receive-dump.c index fa397bcf..df5991e1 100644 --- a/cmds/receive-dump.c +++ b/cmds/receive-dump.c @@ -339,6 +339,11 @@ static int print_fallocate(const char *path, int mode, u64 offset, u64 len, mode, offset, len); } +static int print_setflags(const char *path, int flags, void *user) +{ + return PRINT_DUMP(user, path, "setflags", "flags=%d", flags); +} + struct btrfs_send_ops btrfs_print_send_ops = { .subvol = print_subvol, .snapshot = print_snapshot, @@ -363,4 +368,5 @@ struct btrfs_send_ops btrfs_print_send_ops = { .update_extent = print_update_extent, .encoded_write = print_encoded_write, .fallocate = print_fallocate, + .setflags = print_setflags, }; diff --git a/cmds/receive.c b/cmds/receive.c index 4893d693..7f76a04f 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include @@ -1284,6 +1285,29 @@ static int process_fallocate(const char *path, int mode, u64 offset, u64 len, return 0; } +static int process_setflags(const char *path, int flags, void *user) +{ + int ret; + struct btrfs_receive *rctx = user; + char full_path[PATH_MAX]; + + ret = path_cat_out(full_path, rctx->full_subvol_path, path); + if (ret < 0) { + error("setflags: path invalid: %s", path); + return ret; + } + ret = open_inode_for_write(rctx, full_path); + if (ret < 0) + return ret; + ret = ioctl(rctx->write_fd, FS_IOC_SETFLAGS, &flags); + if (ret < 0) { + ret = -errno; + error("setflags: setflags ioctl on %s failed: %m", path); + return ret; + } + return 0; +} + static struct btrfs_send_ops send_ops = { .subvol = process_subvol, .snapshot = process_snapshot, @@ -1308,6 +1332,7 @@ static struct btrfs_send_ops send_ops = { .update_extent = process_update_extent, .encoded_write = process_encoded_write, .fallocate = process_fallocate, + .setflags = process_setflags, }; static int do_receive(struct btrfs_receive *rctx, const char *tomnt, diff --git a/common/send-stream.c b/common/send-stream.c index 2d0aa624..21295cbb 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -374,6 +374,7 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) int len; int xattr_len; int fallocate_mode; + int setflags_flags; ret = read_cmd(sctx); if (ret) @@ -546,8 +547,14 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) ret = sctx->ops->fallocate(path, fallocate_mode, offset, tmp, sctx->user); break; + case BTRFS_SEND_C_SETFLAGS: + TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); + TLV_GET_U32(sctx, BTRFS_SEND_A_SETFLAGS_FLAGS, &setflags_flags); + ret = sctx->ops->setflags(path, setflags_flags, sctx->user); + break; } + tlv_get_failed: out: free(path); diff --git a/common/send-stream.h b/common/send-stream.h index 61a88d3d..3189f889 100644 --- a/common/send-stream.h +++ b/common/send-stream.h @@ -59,6 +59,7 @@ struct btrfs_send_ops { u32 encryption, void *user); int (*fallocate)(const char *path, int mode, u64 offset, u64 len, void *user); + int (*setflags)(const char *path, int flags, void *user); }; int btrfs_read_and_process_send_stream(int fd, From patchwork Thu Mar 17 17:25:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12784374 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59A56C433F5 for ; Thu, 17 Mar 2022 17:26:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236873AbiCQR1k (ORCPT ); Thu, 17 Mar 2022 13:27:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236870AbiCQR1j (ORCPT ); Thu, 17 Mar 2022 13:27:39 -0400 Received: from mail-pg1-x529.google.com (mail-pg1-x529.google.com [IPv6:2607:f8b0:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA40F14DFE5 for ; Thu, 17 Mar 2022 10:26:22 -0700 (PDT) Received: by mail-pg1-x529.google.com with SMTP id t14so3164142pgr.3 for ; Thu, 17 Mar 2022 10:26:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Fe/P4ONb0TA5b3hvpbn+ZDGquFnInysKiMbJwA33xE0=; b=nbgnp9RxpXHRRh7dm8q0jph39LiBEc5FbfBDodwOKDBZKB13M0kmVh4GRAkPh/VOIR +5wNOw2eGIzQdGIHoBCS+5fVPyGBzSiNe3Njok3N3qLf3tQsJ4u6pFht3ncoOg4XBogY 3IrRbL6KPu2cr8tXd3yBl25/zSI6plxsiujHvPBy1Ss0E6tPxX7MN7W9zWE0wHn2I2Pz i1GS8qQZOQxJWjFCNV6pUP4hksqVZp/5Iyru2YMWuUa7Wa94FWFPI5fq9UkXy3dC7mtL r9i0KSXPLB7r267F/8yXWInLwNWXtNlwNNZFEonISITX05MxI8mI/Qs5EHXtOS7CbuqL lgkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Fe/P4ONb0TA5b3hvpbn+ZDGquFnInysKiMbJwA33xE0=; b=RkE1GWrwbBKQdfhVSdhG//fSzxh5xMb7SDkgk4Nlqmi4hImqnv0w3LYuNgW+1VJqTn W8JzxvT1XgjhopXZ36O3ku8Plqsu6UtExCiAlanhWl5a0jykIkKIH/UExn3k8FvP/hkG dOdB52cHVZUVbmXgscajzvl0GdpwBi+lRpf0CBrwhaJxsdBXPtqiSXTLN8YiX0z3QgpN 5wFgJtthqBKZ4g84f3M7os1dOstFXjO9HaMYfOXOalnsnMjIZAsu0Vqld2q4T3keI2b4 ukyo+bfjw6dMZVC9ubSAMCTVwCenYQrZAUdZQFtUk63lZT9+fS40g0AUP+RoF8eVyMfa aZPQ== X-Gm-Message-State: AOAM530bATfMcOdULsX0Vn1MIE6fffhCIXaAaaRZmwvBFonyDoNxWV6G Lfk5+9nMOTvwOMXWxVDgwYmIoJIob6fPCg== X-Google-Smtp-Source: ABdhPJxQjJJZvLBbsKy3nn7mPsPCPRsT4cRpGkbrpfM7Clnu/4XGEkec+gGOska8lnMoO31R0+0mog== X-Received: by 2002:a63:7e43:0:b0:374:75ce:4d80 with SMTP id o3-20020a637e43000000b0037475ce4d80mr4595993pgn.589.1647537981955; Thu, 17 Mar 2022 10:26:21 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:624e]) by smtp.gmail.com with ESMTPSA id q10-20020a056a00088a00b004f7ceff389esm7815424pfj.152.2022.03.17.10.26.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Mar 2022 10:26:21 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: kernel-team@fb.com Subject: [PATCH v14 09/10] btrfs-progs: send: stream v2 ioctl flags Date: Thu, 17 Mar 2022 10:25:52 -0700 Message-Id: X-Mailer: git-send-email 2.35.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval First, add a --proto option to allow specifying the desired send protocol version. It defaults to one, the original version. In a couple of releases once people are aware that protocol revisions are happening, we can change it to default to zero, which means the latest version supported by the kernel. This is based on Dave Sterba's patch. Also add a --compressed-data flag to instruct the kernel to use encoded_write commands for compressed extents. This requires an explicit opt in separate from the protocol version because: 1. The user may not want compression on the receiving side, or may want a different compression algorithm/level on the receiving side. 2. It has a soft requirement for kernel support on the receiving side (btrfs-progs can fall back to decompressing and writing if the kernel doesn't support BTRFS_IOC_ENCODED_WRITE, but the user may not be prepared to pay that CPU cost). Going forward, since it's easier to update progs than the kernel, I think we'll want to make new send features that require kernel support opt-in, whereas anything that only requires a progs update can happen automatically. Signed-off-by: Boris Burkov Signed-off-by: Omar Sandoval --- Documentation/btrfs-send.rst | 22 ++++++++ cmds/send.c | 100 ++++++++++++++++++++++++++++++++++- ioctl.h | 19 ++++++- kernel-shared/send.h | 2 +- 4 files changed, 138 insertions(+), 5 deletions(-) diff --git a/Documentation/btrfs-send.rst b/Documentation/btrfs-send.rst index 4526532e..291c537e 100644 --- a/Documentation/btrfs-send.rst +++ b/Documentation/btrfs-send.rst @@ -60,6 +60,28 @@ please see section *SUBVOLUME FLAGS* in ``btrfs-subvolume(8)``. used to transfer changes. This mode is faster and is useful to show the differences in metadata. +--proto + use send protocol version N + + The default is 1, which was the original protocol version. Version 2 + encodes file data slightly more efficiently; it is also required for + sending compressed data directly (see *--compressed-data*). Version 2 + requires at least btrfs-progs 5.18 on both the sender and receiver and + at least Linux 5.18 on the sender. Passing 0 means to use the highest + version supported by the running kernel. + +--compressed-data + send data that is compressed on the filesystem directly without + decompressing it + + If the receiver supports the *BTRFS_IOC_ENCODED_WRITE* ioctl (added in + Linux 5.18), it can also write it directly without decompressing it. + Otherwise, the receiver will fall back to decompressing it and writing + it normally. + + This requires protocol version 2 or higher. If *--proto* was not used, + then *--compressed-data* implies *--proto 2*. + -q|--quiet (deprecated) alias for global *-q* option diff --git a/cmds/send.c b/cmds/send.c index 087af05c..b1adfeca 100644 --- a/cmds/send.c +++ b/cmds/send.c @@ -57,6 +57,8 @@ struct btrfs_send { u64 clone_sources_count; char *root_path; + u32 proto; + u32 proto_supported; }; static int get_root_id(struct btrfs_send *sctx, const char *path, u64 *root_id) @@ -259,6 +261,16 @@ static int do_send(struct btrfs_send *send, u64 parent_root_id, memset(&io_send, 0, sizeof(io_send)); io_send.send_fd = pipefd[1]; send->send_fd = pipefd[0]; + io_send.flags = flags; + + if (send->proto_supported > 1) { + /* + * Versioned stream supported, requesting default or specific + * number. + */ + io_send.version = send->proto; + io_send.flags |= BTRFS_SEND_FLAG_VERSION; + } if (!ret) ret = pthread_create(&t_read, NULL, read_sent_data, send); @@ -269,7 +281,6 @@ static int do_send(struct btrfs_send *send, u64 parent_root_id, goto out; } - io_send.flags = flags; io_send.clone_sources = (__u64*)send->clone_sources; io_send.clone_sources_count = send->clone_sources_count; io_send.parent_root = parent_root_id; @@ -421,6 +432,36 @@ static void free_send_info(struct btrfs_send *sctx) sctx->root_path = NULL; } +static u32 get_sysfs_proto_supported(void) +{ + int fd; + int ret; + char buf[32] = {}; + char *end = NULL; + u64 version; + + fd = sysfs_open_file("features/send_stream_version"); + if (fd < 0) { + /* + * No file is either no version support or old kernel with just + * v1. + */ + return 1; + } + ret = sysfs_read_file(fd, buf, sizeof(buf)); + close(fd); + if (ret <= 0) + return 1; + version = strtoull(buf, &end, 10); + if (version == ULLONG_MAX && errno == ERANGE) + return 1; + if (version > U32_MAX) { + warning("sysfs/send_stream_version too big: %llu", version); + version = 1; + } + return version; +} + static const char * const cmd_send_usage[] = { "btrfs send [-ve] [-p ] [-c ] [-f ] [...]", "Send the subvolume(s) to stdout.", @@ -449,6 +490,11 @@ static const char * const cmd_send_usage[] = { " does not contain any file data and thus cannot be used", " to transfer changes. This mode is faster and useful to", " show the differences in metadata.", + "--proto N use protocol version N, or 0 to use the highest version", + " supported by the sending kernel (default: 1)", + "--compressed-data", + " send data that is compressed on the filesystem directly", + " without decompressing it", "-v|--verbose deprecated, alias for global -v option", "-q|--quiet deprecated, alias for global -q option", HELPINFO_INSERT_GLOBALS, @@ -471,9 +517,11 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) int full_send = 1; int new_end_cmd_semantic = 0; u64 send_flags = 0; + u64 proto = 0; memset(&send, 0, sizeof(send)); send.dump_fd = fileno(stdout); + send.proto = 1; outname[0] = 0; /* @@ -489,11 +537,17 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) optind = 0; while (1) { - enum { GETOPT_VAL_SEND_NO_DATA = 256 }; + enum { + GETOPT_VAL_SEND_NO_DATA = 256, + GETOPT_VAL_PROTO, + GETOPT_VAL_COMPRESSED_DATA, + }; static const struct option long_options[] = { { "verbose", no_argument, NULL, 'v' }, { "quiet", no_argument, NULL, 'q' }, { "no-data", no_argument, NULL, GETOPT_VAL_SEND_NO_DATA }, + { "proto", required_argument, NULL, GETOPT_VAL_PROTO }, + { "compressed-data", no_argument, NULL, GETOPT_VAL_COMPRESSED_DATA }, { NULL, 0, NULL, 0 } }; int c = getopt_long(argc, argv, "vqec:f:i:p:", long_options, NULL); @@ -582,6 +636,18 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) case GETOPT_VAL_SEND_NO_DATA: send_flags |= BTRFS_SEND_FLAG_NO_FILE_DATA; break; + case GETOPT_VAL_PROTO: + proto = arg_strtou64(optarg); + if (proto > U32_MAX) { + error("protocol version number too big %llu", proto); + ret = 1; + goto out; + } + send.proto = proto; + break; + case GETOPT_VAL_COMPRESSED_DATA: + send_flags |= BTRFS_SEND_FLAG_COMPRESSED; + break; default: usage_unknown_option(cmd, argv); } @@ -689,6 +755,36 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) if ((send_flags & BTRFS_SEND_FLAG_NO_FILE_DATA) && bconf.verbose > 1) if (bconf.verbose > 1) fprintf(stderr, "Mode NO_FILE_DATA enabled\n"); + send.proto_supported = get_sysfs_proto_supported(); + if (send.proto_supported == 1) { + if (send.proto > send.proto_supported) { + error("requested version %u but kernel supports only %u", + send.proto, send.proto_supported); + ret = -EPROTO; + goto out; + } + } + if (send_flags & BTRFS_SEND_FLAG_COMPRESSED) { + /* + * If no protocol version was explicitly requested, then + * --compressed-data implies --proto 2. + */ + if (send.proto == 1 && !proto) + send.proto = 2; + + if (send.proto == 1) { + error("--compressed-data requires protocol version >= 2 (requested 1)"); + ret = -EINVAL; + goto out; + } else if (send.proto == 0 && send.proto_supported < 2) { + error("kernel does not support --compressed-data"); + ret = -EINVAL; + goto out; + } + } + if (bconf.verbose > 1) + fprintf(stderr, "Protocol version requested: %u (supported %u)\n", + send.proto, send.proto_supported); for (i = optind; i < argc; i++) { int is_first_subvol; diff --git a/ioctl.h b/ioctl.h index 8adf63c2..f19695e3 100644 --- a/ioctl.h +++ b/ioctl.h @@ -655,10 +655,24 @@ BUILD_ASSERT(sizeof(struct btrfs_ioctl_received_subvol_args_32) == 192); */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Read the protocol version in the structure + */ +#define BTRFS_SEND_FLAG_VERSION 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires protocol + * version >= 2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_VERSION | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ @@ -666,7 +680,8 @@ struct btrfs_ioctl_send_args { __u64 __user *clone_sources; /* in */ __u64 parent_root; /* in */ __u64 flags; /* in */ - __u64 reserved[4]; /* in */ + __u32 version; /* in */ + __u8 reserved[28]; /* in */ }; /* * Size of structure depends on pointer width, was not caught in the early diff --git a/kernel-shared/send.h b/kernel-shared/send.h index b902d054..1f20d01a 100644 --- a/kernel-shared/send.h +++ b/kernel-shared/send.h @@ -31,7 +31,7 @@ extern "C" { #endif #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" -#define BTRFS_SEND_STREAM_VERSION 1 +#define BTRFS_SEND_STREAM_VERSION 2 /* * In send stream v1, no command is larger than 64k. In send stream v2, no limit From patchwork Thu Mar 17 17:25:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12784375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11491C433EF for ; Thu, 17 Mar 2022 17:26:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236871AbiCQR1l (ORCPT ); Thu, 17 Mar 2022 13:27:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236875AbiCQR1k (ORCPT ); Thu, 17 Mar 2022 13:27:40 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D486E21047B for ; Thu, 17 Mar 2022 10:26:23 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id v4so5454163pjh.2 for ; Thu, 17 Mar 2022 10:26:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RIflKMx6orNUBBzvTaVkzMgSRIIbO7dH/hPzc4Os/l8=; b=f8hjOBq53EbXo4sN7ojf8rUjF+GjY1rGIJSoTvvAPOJTKk0WrSx5nc5WkTcPi3bMUI 00US3FfFlKua27qIk4M085NqCsaLkCEhabYXuv4y/dfoQpx2CK5ml1L97AeDI64qHoz+ Z4hcq4UIZv3gCEda5TF6wYsAzikwFcMBU8AaXuczccWS7Uj+JmHFQ0XC4bIG+fnbd7T6 g9XexcytwVwcaVbnzEzL2VDn3SaVCojXkM+sTCIPG2W/T6H5cOm4M+VYKTBloxJLM76B h8B1DDgrOIepPxdi10jmG5LpVZ7aTSyEcxO8btegZvRrUJC9yi4Qh6E+180km1EDqgdX A9RQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RIflKMx6orNUBBzvTaVkzMgSRIIbO7dH/hPzc4Os/l8=; b=VXCDO9AD354RyoNZEeGLxYdV/Z/FBAr6X6ctdG4oFsMVbPG5fO4TtVsQ0RJDl4YM0L I5BqR+XDL8M6jAmbvzeLOsqvHcFrOAcnWjZrrd77WUV0x3+wumDzNV7z5biVt8xWd2EZ wMngxAtnpeczASXvksZZdfM9gGX5OY9jBhc4VSoPVJjNlly8NR6CDcIh6b8d4t4H1DOS YZPEazuVOlZX+baOnweI/siswiSMkHwFDOkkeTJoZVW6hbS0chtMpuGB6uHFSQdjI/cw osdfA4koEToCoHBsREz82El+WIuRSZ33k4Mc6pRz3kxoHi0U5pviH+IRy+4LVpDV4D0Q +jWQ== X-Gm-Message-State: AOAM532sSd2s4N9+T2oe2HgohAmWpLw5viDN4dTLleo18DJcJFboIPgj cg9cFunlvx7lsEBxhXOMjPpk3nS+4UhjpA== X-Google-Smtp-Source: ABdhPJwqFHPf9c9HxJ5Kihdzb4XSe0Ie19ArJ4kUP+HQ9335nTHOBKoezvl/5HqVPC8y3r8QHeySPQ== X-Received: by 2002:a17:902:ce08:b0:153:8d90:a109 with SMTP id k8-20020a170902ce0800b001538d90a109mr6190283plg.157.1647537982984; Thu, 17 Mar 2022 10:26:22 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:624e]) by smtp.gmail.com with ESMTPSA id q10-20020a056a00088a00b004f7ceff389esm7815424pfj.152.2022.03.17.10.26.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Mar 2022 10:26:22 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: kernel-team@fb.com Subject: [PATCH v14 10/10] btrfs-progs: receive: add tests for basic encoded_write send/receive Date: Thu, 17 Mar 2022 10:25:53 -0700 Message-Id: <7fe7eb90c6c1cbcb898a429d6235c551b3d9c3ba.1647537098.git.osandov@fb.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Adapt the existing send/receive tests by passing '-o compress-force' to the mount commands in a new test. After writing a few files in the various compression formats, send/receive them with and without --force-decompress to test both the encoded_write path and the fallback to decode+write. Signed-off-by: Boris Burkov --- .../052-receive-write-encoded/test.sh | 114 ++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100755 tests/misc-tests/052-receive-write-encoded/test.sh diff --git a/tests/misc-tests/052-receive-write-encoded/test.sh b/tests/misc-tests/052-receive-write-encoded/test.sh new file mode 100755 index 00000000..47330281 --- /dev/null +++ b/tests/misc-tests/052-receive-write-encoded/test.sh @@ -0,0 +1,114 @@ +#!/bin/bash +# +# test that we can send and receive encoded writes for three modes of +# transparent compression: zlib, lzo, and zstd. + +source "$TEST_TOP/common" + +check_prereq mkfs.btrfs +check_prereq btrfs + +setup_root_helper +prepare_test_dev + +here=`pwd` + +# assumes the filesystem exists, and does mount, write, snapshot, send, unmount +# for the specified encoding option +send_one() { + local str + local subv + local snap + + algorithm="$1" + shift + str="$1" + shift + + subv="subv-$algorithm" + snap="snap-$algorithm" + + run_check_mount_test_dev "-o" "compress-force=$algorithm" + cd "$TEST_MNT" || _fail "cannot chdir to TEST_MNT" + + run_check $SUDO_HELPER "$TOP/btrfs" subvolume create "$subv" + run_check $SUDO_HELPER dd if=/dev/zero of="$subv/file1" bs=1M count=1 + run_check $SUDO_HELPER dd if=/dev/zero of="$subv/file2" bs=500K count=1 + run_check $SUDO_HELPER "$TOP/btrfs" subvolume snapshot -r "$subv" "$snap" + run_check $SUDO_HELPER "$TOP/btrfs" send -f "$str" "$snap" "$@" + + cd "$here" || _fail "cannot chdir back to test directory" + run_check_umount_test_dev +} + +receive_one() { + local str + str="$1" + shift + + run_check_mkfs_test_dev + run_check_mount_test_dev + run_check $SUDO_HELPER "$TOP/btrfs" receive "$@" -v -f "$str" "$TEST_MNT" + run_check_umount_test_dev + run_check rm -f -- "$str" +} + +test_one_write_encoded() { + local str + local algorithm + algorithm="$1" + shift + str="$here/stream-$algorithm.stream" + + run_check_mkfs_test_dev + send_one "$algorithm" "$str" --compressed-data + receive_one "$str" "$@" +} + +test_one_stream_v1() { + local str + local algorithm + algorithm="$1" + shift + str="$here/stream-$algorithm.stream" + + run_check_mkfs_test_dev + send_one "$algorithm" "$str" --proto 1 + receive_one "$str" "$@" +} + +test_mix_write_encoded() { + local strzlib + local strlzo + local strzstd + strzlib="$here/stream-zlib.stream" + strlzo="$here/stream-lzo.stream" + strzstd="$here/stream-zstd.stream" + + run_check_mkfs_test_dev + + send_one "zlib" "$strzlib" --compressed-data + send_one "lzo" "$strlzo" --compressed-data + send_one "zstd" "$strzstd" --compressed-data + + receive_one "$strzlib" + receive_one "$strlzo" + receive_one "$strzstd" +} + +test_one_write_encoded "zlib" +test_one_write_encoded "lzo" +test_one_write_encoded "zstd" + +# with decompression forced +test_one_write_encoded "zlib" "--force-decompress" +test_one_write_encoded "lzo" "--force-decompress" +test_one_write_encoded "zstd" "--force-decompress" + +# send stream v1 +test_one_stream_v1 "zlib" +test_one_stream_v1 "lzo" +test_one_stream_v1 "zstd" + +# files use a mix of compression algorithms +test_mix_write_encoded