From patchwork Thu Feb 2 09:32:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13125602 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 695D9C63797 for ; Thu, 2 Feb 2023 09:32:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232103AbjBBJcq (ORCPT ); Thu, 2 Feb 2023 04:32:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40468 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230021AbjBBJco (ORCPT ); Thu, 2 Feb 2023 04:32:44 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 278A159E6F for ; Thu, 2 Feb 2023 01:32:42 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id k8-20020a05600c1c8800b003dc57ea0dfeso3263537wms.0 for ; Thu, 02 Feb 2023 01:32:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6Q710yGKSb1Do3Dgf89StzbHJ8k5WkyplrSoApE6W3E=; b=ZrwWLvOyy/LMFOQDKEKqnUDKSYP0/eTX5M+qYTeWWXsTR3zltCCNLxZRPxfSnJmeAR sOhkFitHFhQO3akLK39004TLKQLEEe0btStHjoqP/Ke4gGgqC5xuuQM7NgQhTuNiKLTy SCRejks6ktlBfdsEKXyt36vr5vikIXLWxc3yCxLJQXYOPtPkfJB+EQ9sU5Ikx7i/1ttL kofbkIZicA70Px97b95ifIk+DG4GKZFiBQlEwoF9xelTJLo3aI0OicVf2NSlhGt4TpaZ a3hKHU+aDdZXglYqIA4Dr1Xfse78scl9Zqin9z9t1FVM4XO3wfX+dycmA56o8JuS/wOt d1ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6Q710yGKSb1Do3Dgf89StzbHJ8k5WkyplrSoApE6W3E=; b=hd3Zy4Ba+RQbnwI8wJU+4zLX6xaE69rc6VPp5rllnUl+Uz7HGANERTyZM+iwX2sNft U/US+5ck/JiQmCOumWzf3LPe5RYJG194eltCJeqO03U0fjaeTjV1znBGuru65+LafG4s m9QdNP1/sUL5gGBb6on90JwqIEEOnw1C1FxldR9ZMZWwqIBKptVy3LaRacTKzTGwQvoS UypJjPqD9OT/UtxmGhdmVyo3z47A3GKxzt56o2TL0LQ/khbhoe/EUSQ65loEV3++JxjZ ffGZ4HWWCPFlTvyQ0uxTR2+Wjl92a6oRAkTLZrvVwhCGxQbpiMGJXpmSWU6HpgrE50XV n+gg== X-Gm-Message-State: AO0yUKWo5yTOITWoHoHXoaeaAasAEFA87QH84koPJl4iQ+IiiJwHS3Is hdrLHiqCQ6WhRQJsOxGEzNuxtEqjaXGnZ9+N X-Google-Smtp-Source: AK7set8U3euTJHXiioIpJcDoEa7lj80XALcwI2tXC87NzJzK6f3kPVUGS8F7pGXHOkjaFoSQ6VFu6A== X-Received: by 2002:a05:600c:4a9b:b0:3de:1d31:1043 with SMTP id b27-20020a05600c4a9b00b003de1d311043mr5248972wmp.21.1675330360182; Thu, 02 Feb 2023 01:32:40 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id l19-20020a05600c2cd300b003cf71b1f66csm4238753wmc.0.2023.02.02.01.32.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 01:32:39 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Eli Schwartz , =?utf-8?q?Ren=C3=A9_Scharfe?= , "brian m . carlson" , Konstantin Ryabitsev , =?utf-8?q?Michal_Suc?= =?utf-8?q?h=C3=A1nek?= , "Raymond E . Pasco" , demerphq , Theodore Ts'o , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFz?= =?utf-8?b?b24=?= Subject: [PATCH 1/9] archive & tar config docs: de-duplicate configuration section Date: Thu, 2 Feb 2023 10:32:21 +0100 Message-Id: X-Mailer: git-send-email 2.39.1.1392.g63e6d408230 In-Reply-To: References: <230131.86357rrtsg.gmgdl@evledraar.gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The "tar.umask" documentation was initially added in [1], and was duplicated from the start. Then with [2] the two started drifting apart. Let's consolidate them with a change like the ones made in the commits merged in [3]. 1. ce1a79b6a74 (tar-tree: add the "tar.umask" config option, 2006-07-20) 2. 687157c736d (Documentation: update tar.umask default, 2007-08-21) 3. 7a54d740451 (Merge branch 'ab/dedup-config-and-command-docs', 2022-09-14) Signed-off-by: Ævar Arnfjörð Bjarmason --- Documentation/config/tar.txt | 4 +++- Documentation/git-archive.txt | 8 +------- 2 files changed, 4 insertions(+), 8 deletions(-) diff --git a/Documentation/config/tar.txt b/Documentation/config/tar.txt index de8ff48ea9d..c68e294bbc5 100644 --- a/Documentation/config/tar.txt +++ b/Documentation/config/tar.txt @@ -3,4 +3,6 @@ tar.umask:: tar archive entries. The default is 0002, which turns off the world write bit. The special value "user" indicates that the archiving user's umask will be used instead. See umask(2) and - linkgit:git-archive[1]. + linkgit:git-archive[1] for + details. If `--remote` is used then only the configuration of + the remote repository takes effect. diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt index 60c040988bb..bbb407d4975 100644 --- a/Documentation/git-archive.txt +++ b/Documentation/git-archive.txt @@ -131,13 +131,7 @@ tar CONFIGURATION ------------- -tar.umask:: - This variable can be used to restrict the permission bits of - tar archive entries. The default is 0002, which turns off the - world write bit. The special value "user" indicates that the - archiving user's umask will be used instead. See umask(2) for - details. If `--remote` is used then only the configuration of - the remote repository takes effect. +include::config/tar.txt[] tar..command:: This variable specifies a shell command through which the tar From patchwork Thu Feb 2 09:32:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13125603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 547F7C61DA4 for ; Thu, 2 Feb 2023 09:32:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232111AbjBBJcs (ORCPT ); Thu, 2 Feb 2023 04:32:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40488 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232079AbjBBJcp (ORCPT ); Thu, 2 Feb 2023 04:32:45 -0500 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2158B4AA4E for ; Thu, 2 Feb 2023 01:32:43 -0800 (PST) Received: by mail-wm1-x330.google.com with SMTP id l8so878815wms.3 for ; Thu, 02 Feb 2023 01:32:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v1qjNOAhIKvP4aOQnU3+1zqmYU20nn3ykG5nkJeXmzg=; b=SW7OvkVu75tDKluAovdEHAU4MAqc1/d0WLtwdBcp09XUKYdF5CU6SzUZzsIv8eqDIn Z/z0+clK7Q1r/1QV9D8tKlHBfuEZMLWdbQebVeA0pdgnWbtyqBWlmryjZRIEQa0Mvylw pkK/gxqfOlY5uyLKLBnzdrPH1mrA1wfh8iN/w1q24QUhRtW2AFYn1HchR+dzA4CcWt6q 7MH3kX90MAeaTiUdZ7Vz7c16JQF8jPXAUKwtDtX3C6mvcub+V5MPSd1JevchVCfAsdfK 223X6qra0gTSBpaRKD9XQf/TNNJ5/ikePE/UkRMkUWEB+GyiJSnceSpRpYhlcY2zegny /Q3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v1qjNOAhIKvP4aOQnU3+1zqmYU20nn3ykG5nkJeXmzg=; b=wQUGKlm7VfQLYFahnTBdd3mWpqUpfE43jrwxvHxR0DGCxFgxCEeGP0+UPZzUY4erEz 3VUnpt3Tvyy81F2OPG9ejyHnml6S67jT+7MSBoqDgRrEFf8YI1aWnSQwqZrCM0Q+mVb0 2nkl/awfUZa1DO33CDtgtKnIDEY47WJ0l/ySKAcsgIq0mhTIgXVdwydXqkRl5mGbo1sW rWQxDLxXSbR6I+QI3jWfwN2SlvH3GXYzc/yt7EV8rlbKRUmn0YfYx+Lk1x0uRelNkbhk ulR/jRugG9iOyn+Z8TMkvdVoEhTv+0x70QzSLKlQn25hf/Qh6UtxQ9ofi/N0fyMxlmqH /eSQ== X-Gm-Message-State: AO0yUKXncgvqRaK7FnkHw53jlJeoU+VFKnI7u1/uAksIJqPlcf8+JSKE MlyccEuPA01hC3+iDCrRT6yx038c23XlMtbC X-Google-Smtp-Source: AK7set/0tw94VLjj9hi9jFRTHITLhd4ePetGqAkWfeiBZ07Yw2oVS5MfzGen3sNd9aKHnYYdcIRoWg== X-Received: by 2002:a05:600c:5406:b0:3dc:5614:919f with SMTP id he6-20020a05600c540600b003dc5614919fmr5178734wmb.23.1675330361339; Thu, 02 Feb 2023 01:32:41 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id l19-20020a05600c2cd300b003cf71b1f66csm4238753wmc.0.2023.02.02.01.32.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 01:32:40 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Eli Schwartz , =?utf-8?q?Ren=C3=A9_Scharfe?= , "brian m . carlson" , Konstantin Ryabitsev , =?utf-8?q?Michal_Suc?= =?utf-8?q?h=C3=A1nek?= , "Raymond E . Pasco" , demerphq , Theodore Ts'o , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFz?= =?utf-8?b?b24=?= Subject: [PATCH 2/9] git config docs: document "tar..{command,remote}" Date: Thu, 2 Feb 2023 10:32:22 +0100 Message-Id: X-Mailer: git-send-email 2.39.1.1392.g63e6d408230 In-Reply-To: References: <230131.86357rrtsg.gmgdl@evledraar.gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Since the "tar..command" and "tar..remote" configuration was added in [1] and [2], we have not included it in the "git-config(1)" docs themselves. Since we're including "Documentation/config/tar.txt" in "Documentation/config/git-archive.txt" as of the preceding commit, let's move this documentation to the former, to be included in the latter. This is a move-only change, aside from changing the mention of "`git archive`" to "linkgit:git-archive[1]", for consistency with other such mentions. 1. 767cf4579f0 (archive: implement configurable tar filters, 2011-06-21) 2. 7b97730b764 (upload-archive: allow user to turn off filters, 2011-06-21) Signed-off-by: Ævar Arnfjörð Bjarmason --- Documentation/config/tar.txt | 18 ++++++++++++++++++ Documentation/git-archive.txt | 18 ------------------ 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/Documentation/config/tar.txt b/Documentation/config/tar.txt index c68e294bbc5..894c1163bb9 100644 --- a/Documentation/config/tar.txt +++ b/Documentation/config/tar.txt @@ -6,3 +6,21 @@ tar.umask:: linkgit:git-archive[1] for details. If `--remote` is used then only the configuration of the remote repository takes effect. + +tar..command:: + This variable specifies a shell command through which the tar + output generated by linkgit:git-archive[1] should be piped. The command + is executed using the shell with the generated tar file on its + standard input, and should produce the final output on its + standard output. Any compression-level options will be passed + to the command (e.g., `-9`). ++ +The `tar.gz` and `tgz` formats are defined automatically and use the +magic command `git archive gzip` by default, which invokes an internal +implementation of gzip. + +tar..remote:: + If true, enable the format for use by remote clients via + linkgit:git-upload-archive[1]. Defaults to false for + user-defined formats, but true for the `tar.gz` and `tgz` + formats. diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt index bbb407d4975..268e797f03a 100644 --- a/Documentation/git-archive.txt +++ b/Documentation/git-archive.txt @@ -133,24 +133,6 @@ CONFIGURATION include::config/tar.txt[] -tar..command:: - This variable specifies a shell command through which the tar - output generated by `git archive` should be piped. The command - is executed using the shell with the generated tar file on its - standard input, and should produce the final output on its - standard output. Any compression-level options will be passed - to the command (e.g., `-9`). -+ -The `tar.gz` and `tgz` formats are defined automatically and use the -magic command `git archive gzip` by default, which invokes an internal -implementation of gzip. - -tar..remote:: - If true, enable the format for use by remote clients via - linkgit:git-upload-archive[1]. Defaults to false for - user-defined formats, but true for the `tar.gz` and `tgz` - formats. - [[ATTRIBUTES]] ATTRIBUTES ---------- From patchwork Thu Feb 2 09:32:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13125604 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4B90C61DA4 for ; Thu, 2 Feb 2023 09:32:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232198AbjBBJcx (ORCPT ); Thu, 2 Feb 2023 04:32:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40498 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232096AbjBBJcq (ORCPT ); Thu, 2 Feb 2023 04:32:46 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 26FC95AA70 for ; Thu, 2 Feb 2023 01:32:44 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id o36so891962wms.1 for ; Thu, 02 Feb 2023 01:32:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=a6Z4S3JBJLdX+kCju7Fvvw8qYOVc1/ogfhUYTn6PSZY=; b=LtZc+GWsMI0tszErV8xnggDPkJdCDXbdR8eRCCSucZkGgPzidUGbhNKITK65jRa/BT wlVQ2wN/XpQtOz+4QfzUILwjq3kEp9DlbUakGxWrfc+arAM8Bq/r1+Fm+3izqL3mt3JT e4St8ge/064R5ynwJ4N0o2s+NzZijnYvKepZihxzDPFkQ2zjy7n5lH2wWQGmmybmCjSE pHTPjnlYsB15N6d2hf8ENCvQuJT5UsGq6uF7PlEt+8w4uC4qSA2fhrQExKMtetm8pycU ReDbxs2qu2/q7X8oafT3U/Uwrc29u/61XCBmbh5SonOj2LDQvf93R3FEhNVxhO8wd5nX 1O5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a6Z4S3JBJLdX+kCju7Fvvw8qYOVc1/ogfhUYTn6PSZY=; b=D+rcgljRSbDc2b/rsFe7xQfRvP1yAWmISr+Opo7VUfbM81f9jzmRORm5G/La2iPmkk TQIrZM8c9YvnWD9Wm0qxkgwA582QntJuYByF1Ked8I//WitIAiezXxZQLNnrnttLVzSn YupwdAdS7kEOMswl1JjkORD/VdnEg3I/l0h81VlztbdbhIgaiP9LvvLP44NcL4+TVaOp bTowiGtBqHMZJf6vuMp/lC2h3dmW1WeNTmH//A3vuyXA/h9R9HFy9POqiWUG7qpRx5id DBc15UPcoVJdvd+6XIa//wa8Zr4ZZtTLTkPXytBdM/qUH6XRp9n89S6aReZnN0mZAaV7 H2VA== X-Gm-Message-State: AO0yUKXlD/JKxx7naxvMV9XPd2EejVkzu9pcQwjN1HFbPxgX5syxxJY4 TbCPvVLZzaJxLWHyhUsZTSiDjtiSNzvY6zuN X-Google-Smtp-Source: AK7set/iYMX2IVQn/2R/5bi3aoaq8TyUka2m8ta19F7Cnh6BUGFeJNgspHiJAgISsZpukxiRSgcJRg== X-Received: by 2002:a05:600c:524d:b0:3cf:7197:e68a with SMTP id fc13-20020a05600c524d00b003cf7197e68amr5192486wmb.18.1675330362386; Thu, 02 Feb 2023 01:32:42 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id l19-20020a05600c2cd300b003cf71b1f66csm4238753wmc.0.2023.02.02.01.32.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 01:32:41 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Eli Schwartz , =?utf-8?q?Ren=C3=A9_Scharfe?= , "brian m . carlson" , Konstantin Ryabitsev , =?utf-8?q?Michal_Suc?= =?utf-8?q?h=C3=A1nek?= , "Raymond E . Pasco" , demerphq , Theodore Ts'o , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFz?= =?utf-8?b?b24=?= Subject: [PATCH 3/9] archiver API: make the "flags" in "struct archiver" an enum Date: Thu, 2 Feb 2023 10:32:23 +0100 Message-Id: X-Mailer: git-send-email 2.39.1.1392.g63e6d408230 In-Reply-To: References: <230131.86357rrtsg.gmgdl@evledraar.gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Refactor the "#define" pattern in the archiver.h to use a new "enum archiver_flags". This isn't a functional change, but will make adding new flags in a subsequent commit easier to reason about. Signed-off-by: Ævar Arnfjörð Bjarmason --- archive.h | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/archive.h b/archive.h index 08bed3ed3af..6b51288c2ed 100644 --- a/archive.h +++ b/archive.h @@ -36,13 +36,15 @@ const char *archive_format_from_filename(const char *filename); /* archive backend stuff */ -#define ARCHIVER_WANT_COMPRESSION_LEVELS 1 -#define ARCHIVER_REMOTE 2 -#define ARCHIVER_HIGH_COMPRESSION_LEVELS 4 +enum archiver_flags { + ARCHIVER_WANT_COMPRESSION_LEVELS = 1<<0, + ARCHIVER_REMOTE = 1<<1, + ARCHIVER_HIGH_COMPRESSION_LEVELS = 1<<2, +}; struct archiver { const char *name; int (*write_archive)(const struct archiver *, struct archiver_args *); - unsigned flags; + enum archiver_flags flags; char *filter_command; }; void register_archiver(struct archiver *); From patchwork Thu Feb 2 09:32:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13125605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15877C636D4 for ; Thu, 2 Feb 2023 09:32:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232148AbjBBJcy (ORCPT ); Thu, 2 Feb 2023 04:32:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232110AbjBBJcq (ORCPT ); Thu, 2 Feb 2023 04:32:46 -0500 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2EDE270986 for ; Thu, 2 Feb 2023 01:32:45 -0800 (PST) Received: by mail-wm1-x336.google.com with SMTP id d4-20020a05600c3ac400b003db1de2aef0so864365wms.2 for ; Thu, 02 Feb 2023 01:32:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CNN083NT/OPQSIBpBbCdga9bcnGj4pF4fQn4m39t7Xk=; b=Fspw4fA5Pwj6I1Xhilr+wnxk0YOeXeiv/yzCdc3HdPL391/I1fGzJlwxnx30FPtc8T YW1waQc9nHdpmv06FP6bQR+m+cqdOdAQJhrCRpcVLjIiFLI+V4+04oKBV70P8zhsRUf/ Vqbb1AojsKKAYlKzgVxbaAjWrQJ3GlVyBC0K9y/2Fg+6CI5j4AnYU6U09geDmM6daqKb CaHRiU4i0qidqATgd3rbJZ/zuxm1/xWKdKwwT4VV7qFM5rE0n1lXA3O/KqROtiZnWc1O NoixStHm9DYceW2glT/0WuLtMYqMTk7buRjS/fZPoGIpfOvEbr6v6V/hsZCYLxnLoZ2T z4kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CNN083NT/OPQSIBpBbCdga9bcnGj4pF4fQn4m39t7Xk=; b=M2M33Ai6aCZSq6qDYEDNd2YnjoXduVsphHZoy48Fg866YtjGsIasLYc0jfcSu1RrSv pce1syPn4wd6Yp29qtws4w11h89MtBl4l7dnsoE5hrjGgaZoTRPWecaRuuAiqGlj5Zq7 n2o1brHOguU0dArginMHHREtFZXzNMsKOmenvyFjVL1w5a/DMz4+vQKUinbnDPBY7ezY 8Vl85fA1ecQXXkFHY5bWkpw5X/H7145zTxVKRozrVCa0nAtTd5YeAbuPb0Lz9+1q4HZA e20d7b81kUpMwE3GAM9ybLqYRTSxIogGCSAt5OfwzlXHkrhRI9NJM01lnJRfKZpossMh l7yQ== X-Gm-Message-State: AO0yUKVt8GAzBFmjkhDCdR0W/w8YPnZyABNAG2iurDHdeEp4oTlEr892 VNKbTM2pjwdSzwyf54q7uUNbdcEJd3gK+vl4 X-Google-Smtp-Source: AK7set+BumkwGhT4BCORCTQ/gIO+uuqBeac7/I83ADzWafR8uBNQPThNW69d5FwNYbxbQYXe7Fezgw== X-Received: by 2002:a7b:ca54:0:b0:3dc:2b62:ee9f with SMTP id m20-20020a7bca54000000b003dc2b62ee9fmr5170554wml.3.1675330363409; Thu, 02 Feb 2023 01:32:43 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id l19-20020a05600c2cd300b003cf71b1f66csm4238753wmc.0.2023.02.02.01.32.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 01:32:42 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Eli Schwartz , =?utf-8?q?Ren=C3=A9_Scharfe?= , "brian m . carlson" , Konstantin Ryabitsev , =?utf-8?q?Michal_Suc?= =?utf-8?q?h=C3=A1nek?= , "Raymond E . Pasco" , demerphq , Theodore Ts'o , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFz?= =?utf-8?b?b24=?= Subject: [PATCH 4/9] archive: omit the shell for built-in "command" filters Date: Thu, 2 Feb 2023 10:32:24 +0100 Message-Id: X-Mailer: git-send-email 2.39.1.1392.g63e6d408230 In-Reply-To: References: <230131.86357rrtsg.gmgdl@evledraar.gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Since the "tar..command=" where "" is equivalent to our hardcoded "" the same as when the same "" is specified in the config. If the user has configured e.g. "gzip -cn" they may be relying on what the shell gives them over a direct execve() of "gzip". This makes us marginally faster, but the real point is to make the error handling easier to deal with. When we're using the shell we don't know if e.g. the "gzip" we spawned fails as easily, i.e. "start_command()" won't fail, because we can find the "sh". A subsequent commit will tweak the default that [3] introduced to be a fallback instead, at which point we'll need this for correctness. 1. 767cf4579f0 (archive: implement configurable tar filters, 2011-06-21) 2. 0e804e09938 (archive: provide builtin .tar.gz filter, 2011-06-21) 3. 4f4be00d302 (archive-tar: use internal gzip by default, 2022-06-15) Signed-off-by: Ævar Arnfjörð Bjarmason --- Documentation/config/tar.txt | 3 +++ archive-tar.c | 17 +++++++++++++---- archive.h | 1 + 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/Documentation/config/tar.txt b/Documentation/config/tar.txt index 894c1163bb9..5456fc617a2 100644 --- a/Documentation/config/tar.txt +++ b/Documentation/config/tar.txt @@ -18,6 +18,9 @@ tar..command:: The `tar.gz` and `tgz` formats are defined automatically and use the magic command `git archive gzip` by default, which invokes an internal implementation of gzip. ++ +The automatically defined commands do not invoke the shell, avoiding +the minor overhead of an extra sh(1) process. tar..remote:: If true, enable the format for use by remote clients via diff --git a/archive-tar.c b/archive-tar.c index f8fad2946ef..8c5de949c64 100644 --- a/archive-tar.c +++ b/archive-tar.c @@ -367,12 +367,13 @@ static struct archiver *find_tar_filter(const char *name, size_t len) } static int tar_filter_config(const char *var, const char *value, - void *data UNUSED) + void *data) { struct archiver *ar; const char *name; const char *type; size_t namelen; + int *configured = data; if (parse_config_key(var, "tar", &name, &namelen, &type) < 0 || !name) return 0; @@ -388,6 +389,9 @@ static int tar_filter_config(const char *var, const char *value, tar_filters[nr_tar_filters++] = ar; } + if (configured && *configured) + ar->flags |= ARCHIVER_COMMAND_FROM_CONFIG; + if (!strcmp(type, "command")) { if (!value) return config_error_nonbool(var); @@ -495,8 +499,12 @@ static int write_tar_filter_archive(const struct archiver *ar, if (args->compression_level >= 0) strbuf_addf(&cmd, " -%d", args->compression_level); - strvec_push(&filter.args, cmd.buf); - filter.use_shell = 1; + if (ar->flags & ARCHIVER_COMMAND_FROM_CONFIG) { + strvec_push(&filter.args, cmd.buf); + filter.use_shell = 1; + } else { + strvec_split(&filter.args, cmd.buf); + } filter.in = -1; filter.silent_exec_failure = 1; @@ -526,13 +534,14 @@ static struct archiver tar_archiver = { void init_tar_archiver(void) { int i; + int configured = 1; register_archiver(&tar_archiver); tar_filter_config("tar.tgz.command", internal_gzip_command, NULL); tar_filter_config("tar.tgz.remote", "true", NULL); tar_filter_config("tar.tar.gz.command", internal_gzip_command, NULL); tar_filter_config("tar.tar.gz.remote", "true", NULL); - git_config(git_tar_config, NULL); + git_config(git_tar_config, &configured); for (i = 0; i < nr_tar_filters; i++) { /* omit any filters that never had a command configured */ if (tar_filters[i]->filter_command) diff --git a/archive.h b/archive.h index 6b51288c2ed..9686b3b5cc1 100644 --- a/archive.h +++ b/archive.h @@ -40,6 +40,7 @@ enum archiver_flags { ARCHIVER_WANT_COMPRESSION_LEVELS = 1<<0, ARCHIVER_REMOTE = 1<<1, ARCHIVER_HIGH_COMPRESSION_LEVELS = 1<<2, + ARCHIVER_COMMAND_FROM_CONFIG = 1<<3, }; struct archiver { const char *name; From patchwork Thu Feb 2 09:32:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13125606 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A6DDC61DA4 for ; Thu, 2 Feb 2023 09:32:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232243AbjBBJcz (ORCPT ); Thu, 2 Feb 2023 04:32:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40542 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232014AbjBBJcr (ORCPT ); Thu, 2 Feb 2023 04:32:47 -0500 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 555E974C3D for ; Thu, 2 Feb 2023 01:32:46 -0800 (PST) Received: by mail-wm1-x32c.google.com with SMTP id c10-20020a05600c0a4a00b003db0636ff84so883574wmq.0 for ; Thu, 02 Feb 2023 01:32:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UFtOhchuQF+QU6XMHWMrTv/NX3VDwdnvSVuj3x6KWCQ=; b=BBCzaq91OaIo5cmYU/3gcPRe411ZqTMaCs6TH1d+tWywHDrd6JN0fcAGddWxLFCDaN 2IimMrK+G38DmSfdRqe1ksRBwat9F4+OUA3KIzn81kCdcr+GUHQBXxpl5FN9+viC2fdf J45kt3I2sgQkl9GjOa2f+IZ899n1vT6tJR+raBduKQiJZWEqbhd98bNFjzTXVk1XJD4k +SKdoEr/QLB3wsJXwOesnc60UhAjVQYN+dvWkgV6xCzd2HHK9R6Ty2ZY7bB6Po8avxdt 997KuqI+p5edJX+flGAJSQXFQi/gC0oKR7ePk3pnhTOF6fu41Ce3wIIyg3s/d8UuPq2J Z24Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UFtOhchuQF+QU6XMHWMrTv/NX3VDwdnvSVuj3x6KWCQ=; b=ZTnpSQipa6ST5MD7srz1idHhNl6vxAQZWZrWFhRf0t+cyuGyQt67TU8RhO2EajnI13 LwWdJSr6L3aOasI/BUdgHIQMojIkJbeigz5j0aEjMRnmbRM4U5BOtv/7h+b/SF8vQJFL /P+nwhykk5L0fo+YwXTWCmooz8Pv46UWjLLJYNgxeGmENBSNKmxPUZVKG10mRVtleYv4 efuY3YU28BRbs74eEiIvXxjpEITYDeTiSJnhDrf+Op7UcGfC1Vt+J0yGur8neO5MWPSa 9YSJwKBGgxgXq9ExDe+AYjgs2JDsCUFlMne5MZ5Ej1wHMJIFjBig+2UxzGhhJFDGfsnW fFVQ== X-Gm-Message-State: AO0yUKU/7ocRWyB/5xSZzVC9VkVQo50594g7fscw3dMbF4m0LZzUOvOU WUPJFsu0XRpqgIqud9WQdBpnGFN6pvljG3Zc X-Google-Smtp-Source: AK7set+HZMIJJNBt0OT4jNRpEgbpd7Ms1IYxjyHvp3MZA1fHltko14GJSrxftSownajPjhvVwIDNyQ== X-Received: by 2002:a05:600c:46c9:b0:3db:1d8:9f25 with SMTP id q9-20020a05600c46c900b003db01d89f25mr4839835wmo.2.1675330364491; Thu, 02 Feb 2023 01:32:44 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id l19-20020a05600c2cd300b003cf71b1f66csm4238753wmc.0.2023.02.02.01.32.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 01:32:43 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Eli Schwartz , =?utf-8?q?Ren=C3=A9_Scharfe?= , "brian m . carlson" , Konstantin Ryabitsev , =?utf-8?q?Michal_Suc?= =?utf-8?q?h=C3=A1nek?= , "Raymond E . Pasco" , demerphq , Theodore Ts'o , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFz?= =?utf-8?b?b24=?= Subject: [PATCH 5/9] archive-tar.c: move internal gzip implementation to a function Date: Thu, 2 Feb 2023 10:32:25 +0100 Message-Id: X-Mailer: git-send-email 2.39.1.1392.g63e6d408230 In-Reply-To: References: <230131.86357rrtsg.gmgdl@evledraar.gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Refactor the code added in 76d7602631a (archive-tar: add internal gzip implementation, 2022-06-15) to call the magic "git archive gzip" command as a function. A subsequent commit will start using this as a fallback, but for now there's no functional changes here. Signed-off-by: Ævar Arnfjörð Bjarmason --- archive-tar.c | 43 +++++++++++++++++++++++++------------------ 1 file changed, 25 insertions(+), 18 deletions(-) diff --git a/archive-tar.c b/archive-tar.c index 8c5de949c64..dfc133deac7 100644 --- a/archive-tar.c +++ b/archive-tar.c @@ -465,12 +465,33 @@ static void tgz_write_block(const void *data) static const char internal_gzip_command[] = "git archive gzip"; -static int write_tar_filter_archive(const struct archiver *ar, - struct archiver_args *args) +static int gzip_internally(const struct archiver *ar, + struct archiver_args *args) { #if ZLIB_VERNUM >= 0x1221 struct gz_header_s gzhead = { .os = 3 }; /* Unix, for reproducibility */ #endif + int r; + + write_block = tgz_write_block; + git_deflate_init_gzip(&gzstream, args->compression_level); +#if ZLIB_VERNUM >= 0x1221 + if (deflateSetHeader(&gzstream.z, &gzhead) != Z_OK) + BUG("deflateSetHeader() called too late"); +#endif + gzstream.next_out = outbuf; + gzstream.avail_out = sizeof(outbuf); + + r = write_tar_archive(ar, args); + + tgz_deflate(Z_FINISH); + git_deflate_end(&gzstream); + return r; +} + +static int write_tar_filter_archive(const struct archiver *ar, + struct archiver_args *args) +{ struct strbuf cmd = STRBUF_INIT; struct child_process filter = CHILD_PROCESS_INIT; int r; @@ -478,22 +499,8 @@ static int write_tar_filter_archive(const struct archiver *ar, if (!ar->filter_command) BUG("tar-filter archiver called with no filter defined"); - if (!strcmp(ar->filter_command, internal_gzip_command)) { - write_block = tgz_write_block; - git_deflate_init_gzip(&gzstream, args->compression_level); -#if ZLIB_VERNUM >= 0x1221 - if (deflateSetHeader(&gzstream.z, &gzhead) != Z_OK) - BUG("deflateSetHeader() called too late"); -#endif - gzstream.next_out = outbuf; - gzstream.avail_out = sizeof(outbuf); - - r = write_tar_archive(ar, args); - - tgz_deflate(Z_FINISH); - git_deflate_end(&gzstream); - return r; - } + if (!strcmp(ar->filter_command, internal_gzip_command)) + return gzip_internally(ar, args); strbuf_addstr(&cmd, ar->filter_command); if (args->compression_level >= 0) From patchwork Thu Feb 2 09:32:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13125607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B739EC636D4 for ; Thu, 2 Feb 2023 09:32:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232258AbjBBJc4 (ORCPT ); Thu, 2 Feb 2023 04:32:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232115AbjBBJct (ORCPT ); Thu, 2 Feb 2023 04:32:49 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59CC570D7B for ; Thu, 2 Feb 2023 01:32:47 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id m5-20020a05600c4f4500b003db03b2559eso847432wmq.5 for ; Thu, 02 Feb 2023 01:32:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TdKHCyjlxNNREhrqbuQsPUv8vHWxJrelx/Q3orVxGRk=; b=Wi1cjiveXtmvtSonG5BVDwu3BCSI5Yhmyq540qbObk1uBGeru0P02oOhQmaRwH4x6c aVGYWizB0tqFW038Nc7LjLhXrKakeFWNhdjIAy9tnDBojPCHfPM0UWDylpY749daMcd2 o5QOPf5uLL59DjlV295sprYtWTXoqkzx6d4qfROlYOPxnGLxWBq9J1EZ9HAQhWBpTMUG GDjQUjewDSfhvzc/Cl1c3S87CBSyG24zGebGM9LbXwVdxO3wfiPDroE11+cD6TOIZ5t5 kOys2Q/ogu8BP3I9F3hsiREZsmRzOiVFrECzu5HEXUwLONR2qggi4DPQ7iR+fe1WDgjg 9oNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TdKHCyjlxNNREhrqbuQsPUv8vHWxJrelx/Q3orVxGRk=; b=A6TIIq6DrpeaJggG94FxLgwTULn30XYF7REjFkGl9NnvnEqT8agnTAyIQpmzlcJonn HHCcWeihoiuj4dFlHPtBJyxpaRUVr87SQaQdWjYtY/Tzz5rz4/9qOzSLuxS0ItmRwLC3 QZSpOh82LiS3ycNIHoXhqlzM2itzX4eVs2Doqleb7AQxe+d+3M39wc5NypdFep9ZTUpB 5WosiRPcejkCx8itakJ2KLvYb2jo87XtESlkngVcY4NGZRqWq5hSFc3vT1iTEmbyNatk 7mX1WFMsArcm8fFfkcOMnLzpq6rX8izWlOYppOn/x9bgFKkfZv/uxJjvA2Fb7c+FlxNw HZlQ== X-Gm-Message-State: AO0yUKUNGgjkRV2/KN0p+LjesCgO6i3cCRda9Jc+IyDex68EUKck5Wt6 YcQE+TmZEOjQbehY/RwTzM7M9ZPQ9xoXdQ1F X-Google-Smtp-Source: AK7set9FF0QF+ZcAxCPfGyI6GrK2QbtLGVt91MhSseiDmYp9iEYNOXglzvsO9Kh2yyHG44fchuaGOw== X-Received: by 2002:a05:600c:4e55:b0:3d9:e5d3:bf with SMTP id e21-20020a05600c4e5500b003d9e5d300bfmr5148421wmq.32.1675330365492; Thu, 02 Feb 2023 01:32:45 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id l19-20020a05600c2cd300b003cf71b1f66csm4238753wmc.0.2023.02.02.01.32.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 01:32:44 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Eli Schwartz , =?utf-8?q?Ren=C3=A9_Scharfe?= , "brian m . carlson" , Konstantin Ryabitsev , =?utf-8?q?Michal_Suc?= =?utf-8?q?h=C3=A1nek?= , "Raymond E . Pasco" , demerphq , Theodore Ts'o , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFz?= =?utf-8?b?b24=?= Subject: [PATCH 6/9] archive: use "gzip -cn" for stability, not "git archive gzip" Date: Thu, 2 Feb 2023 10:32:26 +0100 Message-Id: X-Mailer: git-send-email 2.39.1.1392.g63e6d408230 In-Reply-To: References: <230131.86357rrtsg.gmgdl@evledraar.gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This reverts and amends [1] so that we don't use "git archive gzip" by default, but only fall back on it when we cannot invoke "gzip". As noted in the discussion at [2] that commit first released with v2.38.0 caused widespread breakage in the wild: Hosting sites like GitHub tend to offer a feature to download tagged releases as archives, which are generated by some variant of "git archive --format=tgz". Downstream distributors then tend to (re-)download those archives as-is, hardcoding their known hash their packaging systems. See [3], [4] etc. for reports of those systems breaking in conjunction with [1]. The reason for "why" is entirely missing from the commit message for [1], but as seen in the question about that in [5] and reply at [6] at the time it was to "avoid a run[time] dependency; the build/test dependency remains.". It's not immediately apparent what the second part of that is referring to, as [1] also removed the "GZIP" prerequisite from some tests. The answer is that we still have other tests that need "GZIP", but those are invoking "gzip(1)" explicitly. In any case, whatever promises we make in the future about the stability and non-stability of "git archive" output (or the derived compressed artifact), this amount of fallout isn't worth it to get to the stated goal in [1]. Let's instead default to "gzip -cn" again, but if we can't find it fall back on "git archive gzip". Note that we'll only fallback if that "gzip -cn" is ours, not if it comes from the user's own "tar..command" configuration. If we do need the fallback we'll warn about it. No such warning will be emitted if the user has explicitly asked for "git archive gzip". 1. 4f4be00d302 (archive-tar: use internal gzip by default, 2022-06-15) 2. https://lore.kernel.org/git/a812a664-67ea-c0ba-599f-cb79e2d96694@gmail.com/ 3. https://github.com/Homebrew/homebrew-core/issues/121877 4. https://github.com/bazel-contrib/SIG-rules-authors/issues/11 5. https://lore.kernel.org/git/220615.86wndhwt9a.gmgdl@evledraar.gmail.com/ 6. https://lore.kernel.org/git/3ed80afd-34b3-afd8-5ffb-0187a4475ee1@web.de/ Signed-off-by: Ævar Arnfjörð Bjarmason --- Documentation/config/tar.txt | 8 ++++++-- archive-tar.c | 20 +++++++++++++++----- 2 files changed, 21 insertions(+), 7 deletions(-) diff --git a/Documentation/config/tar.txt b/Documentation/config/tar.txt index 5456fc617a2..37f24baa73a 100644 --- a/Documentation/config/tar.txt +++ b/Documentation/config/tar.txt @@ -16,8 +16,12 @@ tar..command:: to the command (e.g., `-9`). + The `tar.gz` and `tgz` formats are defined automatically and use the -magic command `git archive gzip` by default, which invokes an internal -implementation of gzip. +command `gzip -cn` by default. An internal gzip implementation can be +used by specifying the value `git archive gzip`. ++ +If 'gzip -cn' cannot be executed we'll fall back on `git archive gzip` +with a warning, if you don't have a gzip(1) and would like to use the +internal `git archive gzip` without warning, configure it explicitly. + The automatically defined commands do not invoke the shell, avoiding the minor overhead of an extra sh(1) process. diff --git a/archive-tar.c b/archive-tar.c index dfc133deac7..26efb911ebc 100644 --- a/archive-tar.c +++ b/archive-tar.c @@ -464,6 +464,7 @@ static void tgz_write_block(const void *data) } static const char internal_gzip_command[] = "git archive gzip"; +static const char gzip_cn_command[] = "gzip -cn"; static int gzip_internally(const struct archiver *ar, struct archiver_args *args) @@ -494,12 +495,15 @@ static int write_tar_filter_archive(const struct archiver *ar, { struct strbuf cmd = STRBUF_INIT; struct child_process filter = CHILD_PROCESS_INIT; + int filter_is_gzip_cn = 0; int r; if (!ar->filter_command) BUG("tar-filter archiver called with no filter defined"); - if (!strcmp(ar->filter_command, internal_gzip_command)) + if (!strcmp(ar->filter_command, gzip_cn_command)) + filter_is_gzip_cn = 1; + else if (!strcmp(ar->filter_command, internal_gzip_command)) return gzip_internally(ar, args); strbuf_addstr(&cmd, ar->filter_command); @@ -515,8 +519,14 @@ static int write_tar_filter_archive(const struct archiver *ar, filter.in = -1; filter.silent_exec_failure = 1; - if (start_command(&filter) < 0) - die_errno(_("unable to start '%s' filter"), cmd.buf); + if (start_command(&filter) < 0) { + if (!filter_is_gzip_cn) + die_errno(_("unable to start '%s' filter"), cmd.buf); + + warning_errno(_("unable to start '%s' filter, falling back to '%s'"), + cmd.buf, internal_gzip_command); + return gzip_internally(ar, args); + } close(1); if (dup2(filter.in, 1) < 0) die_errno(_("unable to redirect descriptor")); @@ -544,9 +554,9 @@ void init_tar_archiver(void) int configured = 1; register_archiver(&tar_archiver); - tar_filter_config("tar.tgz.command", internal_gzip_command, NULL); + tar_filter_config("tar.tgz.command", gzip_cn_command, NULL); tar_filter_config("tar.tgz.remote", "true", NULL); - tar_filter_config("tar.tar.gz.command", internal_gzip_command, NULL); + tar_filter_config("tar.tar.gz.command", gzip_cn_command, NULL); tar_filter_config("tar.tar.gz.remote", "true", NULL); git_config(git_tar_config, &configured); for (i = 0; i < nr_tar_filters; i++) { From patchwork Thu Feb 2 09:32:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13125608 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F3AAC05027 for ; Thu, 2 Feb 2023 09:32:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232277AbjBBJc6 (ORCPT ); Thu, 2 Feb 2023 04:32:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40660 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232124AbjBBJcw (ORCPT ); Thu, 2 Feb 2023 04:32:52 -0500 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7D84F79CA9 for ; Thu, 2 Feb 2023 01:32:48 -0800 (PST) Received: by mail-wm1-x329.google.com with SMTP id n28-20020a05600c3b9c00b003ddca7a2bcbso854184wms.3 for ; Thu, 02 Feb 2023 01:32:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=32jNOFJECkUz0FpaN53jVi+vA32XqMMuDypnVR6fleE=; b=TsskHnxlnhxh6cGLB3x4UznF7P/OPKg6JeKW/52btvDfcsfOxtMMw3ySDPSha+CZSJ wI42vANL6m5UDVLx1fnIfswm5lsk9IRWhQG3AMPIjSbi8/6Qt+pH7yycl13BWBb9Bxtv y7Ry/TszA90gc3MrevKdSEqGnvyY7+CDBhjACoj1zg5cNAkITtnIzCmlQxtZfc8PTEH/ bzRPonX0zMb9DrvpWsnnBMOGCK6TZCJ/sv3LpFFKpwDJJ8YDsWODMZMnB+UWW6QqBUDe F28LJZe42/1lH0fWpxNTo82s3CQ3xFKKpCf2O07p8cJPUbbHqQ5+1O0bJRfkxv/6IL1V qQ1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=32jNOFJECkUz0FpaN53jVi+vA32XqMMuDypnVR6fleE=; b=mOWRTRRLhhdacEk/yVtP6IYLqhZ6crEGdAsuxhp+W2EKPbnp3x0UCCyXDd3EC8W/sA EXSLUKIBpaL3EX02BDwiLt0uoNF/R7GYoMn8Cq/q6eyQhniBVrgAizQ4I5nD6plhMLeq /oLsvvcMMIf26xMxnM9xT9w9HLQD8yLu43JNBG5uUI3yF4KtRj9CNuu7BZKf9Na6izSU JPSi8XsesZISWV7tNr4pQa4RLI6WkX3qWdVH+cqXR54FDiXc33ZNHEbcT639V5rXMeK0 zx4kJEZaVXKzMWUOzI+XPiACLXem9yLVLA8k4dG3sZkMqJ2Qo6Ip6TOXwD0iD4KKdtGN Q0DQ== X-Gm-Message-State: AO0yUKXeGter5xxnqovmG2EHnNxKpT+I0NGRF/szwU0qmZrX+/mcK4Xs xUOPFktNLTiAgQ0oJhv3/I9FH6FgqB5rQg8O X-Google-Smtp-Source: AK7set901V/Z4s6b41dfC7xHGEiafxs8IftNjGDejpIsPa5fr0vCoHYswLeDTBFmT1Tl5t+f5wmZHw== X-Received: by 2002:a05:600c:44c8:b0:3dc:5009:bc74 with SMTP id f8-20020a05600c44c800b003dc5009bc74mr5002710wmo.7.1675330366616; Thu, 02 Feb 2023 01:32:46 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id l19-20020a05600c2cd300b003cf71b1f66csm4238753wmc.0.2023.02.02.01.32.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 01:32:46 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Eli Schwartz , =?utf-8?q?Ren=C3=A9_Scharfe?= , "brian m . carlson" , Konstantin Ryabitsev , =?utf-8?q?Michal_Suc?= =?utf-8?q?h=C3=A1nek?= , "Raymond E . Pasco" , demerphq , Theodore Ts'o , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFz?= =?utf-8?b?b24=?= Subject: [PATCH 7/9] test-lib.sh: add a lazy GZIP prerequisite Date: Thu, 2 Feb 2023 10:32:27 +0100 Message-Id: X-Mailer: git-send-email 2.39.1.1392.g63e6d408230 In-Reply-To: References: <230131.86357rrtsg.gmgdl@evledraar.gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Move the "gzip --version" lazy prerequisite added in [1] and copy/pasted to another test in [2] to test-lib.sh. A subsequent commit will add a third user, let's first stop duplicating it. 1. 96174145fc3 (t5000: simplify gzip prerequisite checks, 2013-12-03) 2. 6c213e863ae (http-backend: respect CONTENT_LENGTH for receive-pack, 2018-07-27) Signed-off-by: Ævar Arnfjörð Bjarmason --- t/t5000-tar-tree.sh | 2 -- t/t5562-http-backend-content-length.sh | 2 -- t/test-lib.sh | 4 ++++ 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/t/t5000-tar-tree.sh b/t/t5000-tar-tree.sh index d4730481384..e1fa34bb828 100755 --- a/t/t5000-tar-tree.sh +++ b/t/t5000-tar-tree.sh @@ -38,8 +38,6 @@ test_lazy_prereq TAR_NEEDS_PAX_FALLBACK ' ) ' -test_lazy_prereq GZIP 'gzip --version' - get_pax_header() { file=$1 header=$2= diff --git a/t/t5562-http-backend-content-length.sh b/t/t5562-http-backend-content-length.sh index b68ec22d3fd..e83aa336fa8 100755 --- a/t/t5562-http-backend-content-length.sh +++ b/t/t5562-http-backend-content-length.sh @@ -3,8 +3,6 @@ test_description='test git-http-backend respects CONTENT_LENGTH' . ./test-lib.sh -test_lazy_prereq GZIP 'gzip --version' - verify_http_result() { # some fatal errors still produce status 200 # so check if there is the error message diff --git a/t/test-lib.sh b/t/test-lib.sh index 01e88781dd2..33bb9fe991f 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1922,6 +1922,10 @@ test_lazy_prereq LONG_IS_64BIT ' test_lazy_prereq TIME_IS_64BIT 'test-tool date is64bit' test_lazy_prereq TIME_T_IS_64BIT 'test-tool date time_t-is64bit' +test_lazy_prereq GZIP ' + gzip --version +' + test_lazy_prereq CURL ' curl --version ' From patchwork Thu Feb 2 09:32:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13125609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00F09C636D4 for ; Thu, 2 Feb 2023 09:33:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232165AbjBBJdA (ORCPT ); Thu, 2 Feb 2023 04:33:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40608 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232079AbjBBJct (ORCPT ); Thu, 2 Feb 2023 04:32:49 -0500 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 710BA761F8 for ; Thu, 2 Feb 2023 01:32:48 -0800 (PST) Received: by mail-wm1-x332.google.com with SMTP id c4-20020a1c3504000000b003d9e2f72093so3249736wma.1 for ; Thu, 02 Feb 2023 01:32:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=p6f+5TOhdJNpIQCXeYi/I2WdZRWIvNxUTzunIlw3WlI=; b=Xc/LHc8BMEBKF1U/jJzI8JP3w7eNSFJ7S5B83eYhYUSustuAYF++3wsCe/rVs2UOL3 DqgJk6kl1aNISwgnyyX8UwSvM9OXwbEJQGFly5qfH3EEfEn3SXDSNscpb1keNyOyt6mb Pu2AuAXLSfLmAXTH1UIeaaH97idrp/c3C3fKOoEuEZ6sqNGyQ6IzfVQU76X21oRfqJ5f +Z4tpqGhJlPOT52erfw3lZ3cBZiZ9fKYHE09f36eD8VpC9FzGTq3KXM17h2FcKyykAdw zqopVuMigxpH7g+lkF7x6xcZG5BZc6Z6CCsFz6Qhncy74bkeRnblo7TkkRL+b7NkKARS rCUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p6f+5TOhdJNpIQCXeYi/I2WdZRWIvNxUTzunIlw3WlI=; b=fhshctdqMqE7OJWSwBcZDyMtco5CEMmSTgpilde+sO7d6ukiEuU49fHPbA2Io9g8bz ZWwKYIbxnVZ0lqVhEX2FjoSzCl+nqTXM6fmXuGsbLOYYSBOphkmAglEVB9G6AGDvdGcW hejQgUwVUon7/awJBwcM15JZWk8HjXaKE6GMW4unTbbgLAPcT6Fi3KDyKoxkApuRy8T6 /Zej4ao2fkYTgUopS5tfDXekCNWjBGfeAwFV4ZEfBPwpMoZzSxu6wBAaSmguc/uudP5B noFqB+QlQIRg07Qdld7yy0DkNfcjRMW8JBIw3mts2pUA++XI2nDJoD2zDrby+O1tywof VUSQ== X-Gm-Message-State: AO0yUKUzONZJkolV1U+DbpADVty2tlyNL/1NaqA/ToLXtSjgr1yziz0D 6dKZV2sKyJ86+FlCQtTlAWnIdQ8eFCNH7kyt X-Google-Smtp-Source: AK7set9JYTUsBJ6hWv3NVws7uZyU4l2IbG0kWqY6+FavugTDvg4Gq/MsgogU9uko+4tcY9VPXCM7WA== X-Received: by 2002:a05:600c:2141:b0:3de:d52:2cd2 with SMTP id v1-20020a05600c214100b003de0d522cd2mr5292681wml.4.1675330367631; Thu, 02 Feb 2023 01:32:47 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id l19-20020a05600c2cd300b003cf71b1f66csm4238753wmc.0.2023.02.02.01.32.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 01:32:46 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Eli Schwartz , =?utf-8?q?Ren=C3=A9_Scharfe?= , "brian m . carlson" , Konstantin Ryabitsev , =?utf-8?q?Michal_Suc?= =?utf-8?q?h=C3=A1nek?= , "Raymond E . Pasco" , demerphq , Theodore Ts'o , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFz?= =?utf-8?b?b24=?= Subject: [PATCH 8/9] archive tests: test for "gzip -cn" and "git archive gzip" stability Date: Thu, 2 Feb 2023 10:32:28 +0100 Message-Id: X-Mailer: git-send-email 2.39.1.1392.g63e6d408230 In-Reply-To: References: <230131.86357rrtsg.gmgdl@evledraar.gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org If our test suite is instrumented to run the first "test_cmp_bin" in "test_done" it'll mostly pass, but fail on a few tests, such as "t5319-multi-pack-index.sh". Those tests reveal edge cases where the output of "gzip -cn" is different than that of "git archive gzip" for the same input. Let's extract a minimal version of the part of "t5319-multi-pack-index.sh" which triggers it, and add a test for archival stability. Whatever we ultimately decide to promise when it comes to this stability (see [1]) it'll be better to go into any behavior difference knowing that's what we're about to do, rather than discover widespread breakage due to already released Git versions. The "GZIP_TRIVIALLY_STABLE" code here is added because on OSX even a trivial *.tgz generated by the two methods will be different. 1. https://lore.kernel.org/git/a812a664-67ea-c0ba-599f-cb79e2d96694@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason --- t/t5005-archive-stability.sh | 70 ++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100755 t/t5005-archive-stability.sh diff --git a/t/t5005-archive-stability.sh b/t/t5005-archive-stability.sh new file mode 100755 index 00000000000..c7532886920 --- /dev/null +++ b/t/t5005-archive-stability.sh @@ -0,0 +1,70 @@ +#!/bin/sh + +test_description='git archive stabilty' + +TEST_PASSES_SANITIZE_LEAK=true +. ./test-lib.sh + +create_archive_file_with_config () { + local file="$1" && + local config="$2" && + shift 2 && + + test_when_finished "rm -rf \"$file\"" && + git -c tar.tgz.command="$config" archive -o "$file" HEAD +} + +setup_gzip_vs_git_archive_gzip () { + create_archive_file_with_config "expect.tgz" "gzip -cn" && + create_archive_file_with_config "actual.tgz" "git archive gzip" +} + +test_lazy_prereq GZIP_TRIVIALLY_STABLE ' + git clone "$TRASH_DIRECTORY" . && + test_commit P && + setup_gzip_vs_git_archive_gzip && + test_cmp_bin expect.tgz actual.tgz +' + +if ! test_have_prereq GZIP_TRIVIALLY_STABLE +then + skip_all='skipping gzip v.s. git archive gzip tests, even trivial content differs' + test_done +fi + +# The first test_expect_success is after the "skip_all" so we'll get +# the skip summary in prove(1) output. +test_expect_success 'setup' ' + test_commit A +' + +test_expect_success GZIP '"gzip -cn" and v.s. "git archive gzip" produce the same output still' ' + setup_gzip_vs_git_archive_gzip && + test_cmp_bin expect.tgz actual.tgz +' + +generate_objects () { + i=$1 + iii=$(printf '%03i' $i) + { + echo $iii && + test-tool genrandom "$iii" 8192 + } >file_$iii && + git update-index --add file_$iii +} + +test_expect_success 'create objects with (stable) random data' ' + test_commit initial && + for i in $(test_seq 1 5) + do + generate_objects $i || return 1 + done && + git commit -m"add objects" +' + +test_expect_success GZIP '"gzip -cn" and v.s. "git archive gzip" have differing output' ' + setup_gzip_vs_git_archive_gzip && + ! test_cmp_bin expect.tgz actual.tgz +' + +test_done From patchwork Thu Feb 2 09:32:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13125610 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6ACCCC61DA4 for ; Thu, 2 Feb 2023 09:33:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232190AbjBBJdB (ORCPT ); Thu, 2 Feb 2023 04:33:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231718AbjBBJcw (ORCPT ); Thu, 2 Feb 2023 04:32:52 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB4057640D for ; Thu, 2 Feb 2023 01:32:49 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id m5-20020a05600c4f4500b003db03b2559eso847574wmq.5 for ; Thu, 02 Feb 2023 01:32:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MUh03QpswINkXXr7X0GLdDUu5VKOJburhspjvva+Cv4=; b=YtGCGh6oiOniD3aMRr7nkIn8bxlX8/l29xw9kCrvdbPhtiEZk70nSt2srXHY9QTxEx ljfl2QQ6jkexIZ41otdMroMmp+zjRZPKW1CuWlYlrD19P8rdVoXU/t5IQ8e5B+YVT35l i9IVWgM/7GSgBM8z1KbaPJzDXM4qZJkE3k0TmQpPESYLiYtzSgZ75jfkzLRO9eNq+g7O qGr3it5PfKkREy0f1P4iVtdWbu0Vgbf4bqlQKYHjaBFwT51fPOl4KxGMmurOE2X33tCU sIJZHtguGRxJWBYbHCPoT7uYYAUb3x08bQ1e4lYrKX4qRjcJRUmYimqT0l6HdjP8XuMB 556w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MUh03QpswINkXXr7X0GLdDUu5VKOJburhspjvva+Cv4=; b=yADcM9Vrmu7DeuT3iY33VQmUywyYbrBad/GaBflUInELxF9yyDrNlJRjd2mBxWoJWY m2V3mf3itijSKg9M6cn1wyqA1nBD7pherogsCJayyHJBtGdds/oKDrL+Ob287pNqSnyQ jDbEwe83QIfswofqwfV4mxKArXrOfdRKgdTIfgFS2qKn4Z/FftpoeRSH+A8CwsKhsPlh KOYwDwl2g+z6ng7QtwkVnJkdewv6nlwmKT4zILcnKKaGGUNlyXsC63zzrCviN2ZXzneB ZlAnbWXFi/IaogScSZe2RunMiV9/ZSTlaDzOOLAyTexsRWaefA3GsFWRH4E7bpBfiAD4 tUIQ== X-Gm-Message-State: AO0yUKU8WX4i5GXqqtnwpPpX4dEmEI66vMNN+slhQvz/hQAj8XAti5Oh CagJm4spmUQreIVNE1DC+OGcBoPDRbHqxvmo X-Google-Smtp-Source: AK7set9euMg7JbyUvM6WubRoukLbu7KyspeKdqqGpWcmhNcJ6hZwxdksHl73nMNIWTGmc0FGFEpExg== X-Received: by 2002:a1c:4b15:0:b0:3dd:1b6f:4f30 with SMTP id y21-20020a1c4b15000000b003dd1b6f4f30mr5798113wma.3.1675330368792; Thu, 02 Feb 2023 01:32:48 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id l19-20020a05600c2cd300b003cf71b1f66csm4238753wmc.0.2023.02.02.01.32.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 01:32:48 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Eli Schwartz , =?utf-8?q?Ren=C3=A9_Scharfe?= , "brian m . carlson" , Konstantin Ryabitsev , =?utf-8?q?Michal_Suc?= =?utf-8?q?h=C3=A1nek?= , "Raymond E . Pasco" , demerphq , Theodore Ts'o , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFz?= =?utf-8?b?b24=?= Subject: [PATCH 9/9] git archive docs: document output non-stability Date: Thu, 2 Feb 2023 10:32:29 +0100 Message-Id: X-Mailer: git-send-email 2.39.1.1392.g63e6d408230 In-Reply-To: References: <230131.86357rrtsg.gmgdl@evledraar.gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org There's an ongoing discussion about the output stability of "git archive"[1] as a follow-up to the incident GitHub experienced when upgrading to v2.38.0[2]. In a preceding commit we reverted the immediate cause of that incident, which was that we'd moved away from "gzip -cn" as the default compression method in favor of the internal "git archive gzip" in [3]. Let's follow that up by documenting the non-promises we've always maintained with regards to "git archive"'s output stability. We may want to make stronger promises in this area, but this change avoids addressing that question. Instead we're discussing that we've changed this in the past, aren't changing it willy-nilly, but it may change again in the future. The only new promise here that we haven't explicitly maintained historically is that we're promising to forever shell out to the system's "gzip" by default. Whether it produces stable output once that happens we leave up to the "gzip" tool. We're also discussing the caveats & differences in output with with SHA-1 and SHA-256 repositories, and trying to steer users towards more stable alternatives. First by using "git verify-tag" and the like to verify releases, and if they really must checksum generated output, to encourage them to at least checksum the "tar" output contained within the compressed output, not the compressed output itself. 1. https://lore.kernel.org/git/a812a664-67ea-c0ba-599f-cb79e2d96694@gmail.com/ 2. https://github.com/orgs/community/discussions/45830 3. 4f4be00d302 (archive-tar: use internal gzip by default, 2022-06-15) Signed-off-by: Ævar Arnfjörð Bjarmason --- Documentation/git-archive.txt | 70 ++++++++++++++++++++++++++++++++++- 1 file changed, 69 insertions(+), 1 deletion(-) diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt index 268e797f03a..78f1b033cb7 100644 --- a/Documentation/git-archive.txt +++ b/Documentation/git-archive.txt @@ -14,6 +14,7 @@ SYNOPSIS [--remote= [--exec=]] [...] +[[DESCRIPTION]] DESCRIPTION ----------- Creates an archive of the specified format containing the tree @@ -28,7 +29,7 @@ case the commit time as recorded in the referenced commit object is used instead. Additionally the commit ID is stored in a global extended pax header if the tar format is used; it can be extracted using 'git get-tar-commit-id'. In ZIP files it is stored as a file -comment. +comment. See the <> section below. OPTIONS ------- @@ -202,6 +203,73 @@ EXAMPLES You can use it specifying `--format=tar.xz`, or by creating an output file like `-o foo.tar.xz`. +[[STABILITY]] +OUTPUT STABILITY +---------------- + +The output of 'git archive' is not guaranteed to be stable, and may +change between versions. + +There are many valid ways to encode the same data in the tar format +itself. For non-`tar` arguments to the `--format` option we rely on +external tools (or libraries) for compressing the output we generate. + +The `tar` format contains the commit ID in the pax header (see the +<> section above). A repository that's been migrated from +SHA-1 to SHA-256 will therefore have different `tar` output for the +"same" commit. See `extension.objectFormat` in linkgit:git-config[1]. + +Instead of relying on the output of `git archive`, you should prefer +to stick to git's own transport protocols, and e.g. validate releases +with linkgit:git-tag[1]'s `--verify` option. + +Despite the output of `git archive` having never been promised to be +stable, various users in the wild have come to rely on that being the +case. + +Most notably, large hosting providers provide a way to download a +given tagged release as a `git archive`. Some downstream tools then +expect the content of that archive to be stable. When that's changed +widespread breakage has been observed, see +https://github.com/orgs/community/discussions/45830 for one such case. + +While we won't promise that the output won't change in the future, we +are aware of these users, and will try to avoid changing it +willy-nilly. Furthermore, we make the following promises: + +* The default gzip compression tool will continue to be gzip(1). If + you rely on this being e.g. GNU gzip for the purposes of stability, + it's up to you to ensure that its output is stable across + versions. ++ + +We in turn promise to not e.g. make the internal "git archive gzip" +implementation the default, as it produces different ouput than +gzip(1) in some case. + +* We will do our best not to change the "tar" output itself, but won't + promise that we're never going to change it. ++ +If you must avoid using "git" itself for the tree validation, you +should be checksumming the uncompressed "tar" output, not e.g. the +compressed "tgz" output. ++ + +This ensures that you're only relying on the output emitted by git +itself, and avoiding the additional dependency on external +compression. ++ +See +https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/get-verified-tarball +for an implementation of that workflow. + +* We promise that a given version of git will emit stable "tar" output + for the same tree ID (but not commit ID, see the discussion in the + <> section above). ++ +While you shouldn't assume that different versions of git will emit +the same output, you can assume (e.g. for the purposes of caching) +that a given version's output is stable. SEE ALSO --------