From patchwork Thu May 19 18:17:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12855899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E5EFC433F5 for ; Thu, 19 May 2022 18:18:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243832AbiESSST (ORCPT ); Thu, 19 May 2022 14:18:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243761AbiESSSK (ORCPT ); Thu, 19 May 2022 14:18:10 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2B87F8CB0E for ; Thu, 19 May 2022 11:18:08 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id s28so8203648wrb.7 for ; Thu, 19 May 2022 11:18:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=jyVAB3KM4yaayVBWUoDOwmy2rGzt9YVcaNOYJYx1hbw=; b=G/pA8fDvkP+qz8NCxMgTqhe/6Ov+cclphKmsR4qW4RiMYok8VKcbmB4OVghA5BUHNS QCFuvXK0EM1UKdt3pCLSg6FU7i2EBDyALu/ktV42cD5o2P6RKFoj7hNQEAiECDyEx06V F6WzQDAN3SfSVh6APhT8L4O582XqIpKMqWU639lqdTJrv1JGTHHACNg3qNQott/lrU8Q 9CaJw8LAVbWuVxYuIICDaNtNlCbzaLKlD6kUu/vxEX3FWAFQzD0AiPddkZz1yDvGLqOx Xs5OGfxQoUsWIoY7+AEY7dqNKOuvgWkP088CgJ/qnBnpIvA4X4g/rkfQ4ZsaaIAy0zRE iOVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=jyVAB3KM4yaayVBWUoDOwmy2rGzt9YVcaNOYJYx1hbw=; b=JvoZbza6FO5nf10PIuaSRLMssHVweuqF8Iqr5RhJ9idB6qJqaNALcTfu4QSuThGaLi EgaIq/1Wu1767GOAvD6HBwsW8OEJ+DrCl1IisGueptKPMO3+s+oL/b66njA2VKTa7yu/ uoeOgdNnJhqJ3HnJJixuZOsf1rSJMowWoIrSl1Hb52XMpf9p/Fs9z5NISPv/KbYrFd7c JMS945sE+VCJYA0IjwByqIKF/3D41b2nZRx/DXDXlQdMW9P0Lw8Z3Ppsemf66or9vicD OZCkjn7IJhGoV/xHGoc8rBU78AQUD9PPgoTuLUeP62A49mEHeN647ucZ/XST8sbMcLw5 fvmw== X-Gm-Message-State: AOAM5320XKDDJF/DjoCg1XXpuM1FIqRjXEqTMtbXOc1fe98dXHbmFH70 6pxtL1lOyITYTCIAOd6L9+gHzY1efSk= X-Google-Smtp-Source: ABdhPJzRMCTaP+Y/4ENpxFT5eGVXGrhqhcPAgax17iPANlX2Zxgjpk27U08VjkzRIdflXrtts6MW2w== X-Received: by 2002:a5d:6950:0:b0:20d:8e2:ccf4 with SMTP id r16-20020a5d6950000000b0020d08e2ccf4mr5210040wrw.222.1652984286280; Thu, 19 May 2022 11:18:06 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id az16-20020a05600c601000b0039729309a84sm216220wmb.20.2022.05.19.11.18.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 May 2022 11:18:05 -0700 (PDT) Message-Id: <42e73fb0aaca1f2498ed817c517859103d72d32b.1652984283.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 19 May 2022 18:17:57 +0000 Subject: [PATCH v5 1/7] archive: optionally add "virtual" files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , rsbecker@nexbridge.com, =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsA==?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin With the `--add-file-with-content=:` option, `git archive` now supports use cases where relatively trivial files need to be added that do not exist on disk. This will allow us to generate `.zip` files with generated content, without having to add said content to the object database and without having to write it out to disk. Signed-off-by: Johannes Schindelin --- Documentation/git-archive.txt | 11 ++++++++ archive.c | 53 +++++++++++++++++++++++++++++------ t/t5003-archive-zip.sh | 12 ++++++++ 3 files changed, 68 insertions(+), 8 deletions(-) diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt index bc4e76a7834..893cb1075bf 100644 --- a/Documentation/git-archive.txt +++ b/Documentation/git-archive.txt @@ -61,6 +61,17 @@ OPTIONS by concatenating the value for `--prefix` (if any) and the basename of . +--add-virtual-file=::: + Add the specified contents to the archive. Can be repeated to add + multiple files. The path of the file in the archive is built + by concatenating the value for `--prefix` (if any) and the + basename of . ++ +The `` cannot contain any colon, the file mode is limited to +a regular file, and the option may be subject to platform-dependent +command-line limits. For non-trivial cases, write an untracked file +and use `--add-file` instead. + --worktree-attributes:: Look for attributes in .gitattributes files in the working tree as well (see <>). diff --git a/archive.c b/archive.c index a3bbb091256..d20e16fa819 100644 --- a/archive.c +++ b/archive.c @@ -263,6 +263,7 @@ static int queue_or_write_archive_entry(const struct object_id *oid, struct extra_file_info { char *base; struct stat stat; + void *content; }; int write_archive_entries(struct archiver_args *args, @@ -337,7 +338,13 @@ int write_archive_entries(struct archiver_args *args, strbuf_addstr(&path_in_archive, basename(path)); strbuf_reset(&content); - if (strbuf_read_file(&content, path, info->stat.st_size) < 0) + if (info->content) + err = write_entry(args, &fake_oid, path_in_archive.buf, + path_in_archive.len, + info->stat.st_mode, + info->content, info->stat.st_size); + else if (strbuf_read_file(&content, path, + info->stat.st_size) < 0) err = error_errno(_("could not read '%s'"), path); else err = write_entry(args, &fake_oid, path_in_archive.buf, @@ -493,6 +500,7 @@ static void extra_file_info_clear(void *util, const char *str) { struct extra_file_info *info = util; free(info->base); + free(info->content); free(info); } @@ -514,14 +522,40 @@ static int add_file_cb(const struct option *opt, const char *arg, int unset) if (!arg) return -1; - path = prefix_filename(args->prefix, arg); - item = string_list_append_nodup(&args->extra_files, path); - item->util = info = xmalloc(sizeof(*info)); + info = xmalloc(sizeof(*info)); info->base = xstrdup_or_null(base); - if (stat(path, &info->stat)) - die(_("File not found: %s"), path); - if (!S_ISREG(info->stat.st_mode)) - die(_("Not a regular file: %s"), path); + + if (!strcmp(opt->long_name, "add-file")) { + path = prefix_filename(args->prefix, arg); + if (stat(path, &info->stat)) + die(_("File not found: %s"), path); + if (!S_ISREG(info->stat.st_mode)) + die(_("Not a regular file: %s"), path); + info->content = NULL; /* read the file later */ + } else if (!strcmp(opt->long_name, "add-virtual-file")) { + const char *colon = strchr(arg, ':'); + char *p; + + if (!colon) + die(_("missing colon: '%s'"), arg); + + p = xstrndup(arg, colon - arg); + if (!args->prefix) + path = p; + else { + path = prefix_filename(args->prefix, p); + free(p); + } + memset(&info->stat, 0, sizeof(info->stat)); + info->stat.st_mode = S_IFREG | 0644; + info->content = xstrdup(colon + 1); + info->stat.st_size = strlen(info->content); + } else { + BUG("add_file_cb() called for %s", opt->long_name); + } + item = string_list_append_nodup(&args->extra_files, path); + item->util = info; + return 0; } @@ -554,6 +588,9 @@ static int parse_archive_args(int argc, const char **argv, { OPTION_CALLBACK, 0, "add-file", args, N_("file"), N_("add untracked file to archive"), 0, add_file_cb, (intptr_t)&base }, + { OPTION_CALLBACK, 0, "add-virtual-file", args, + N_("path:content"), N_("add untracked file to archive"), 0, + add_file_cb, (intptr_t)&base }, OPT_STRING('o', "output", &output, N_("file"), N_("write the archive to this file")), OPT_BOOL(0, "worktree-attributes", &worktree_attributes, diff --git a/t/t5003-archive-zip.sh b/t/t5003-archive-zip.sh index 1e6d18b140e..ebc26e89a9b 100755 --- a/t/t5003-archive-zip.sh +++ b/t/t5003-archive-zip.sh @@ -206,6 +206,18 @@ test_expect_success 'git archive --format=zip --add-file' ' check_zip with_untracked check_added with_untracked untracked untracked +test_expect_success UNZIP 'git archive --format=zip --add-virtual-file' ' + git archive --format=zip >with_file_with_content.zip \ + --add-virtual-file=hello:world $EMPTY_TREE && + test_when_finished "rm -rf tmp-unpack" && + mkdir tmp-unpack && ( + cd tmp-unpack && + "$GIT_UNZIP" ../with_file_with_content.zip && + test_path_is_file hello && + test world = $(cat hello) + ) +' + test_expect_success 'git archive --format=zip --add-file twice' ' echo untracked >untracked && git archive --format=zip --prefix=one/ --add-file=untracked \ From patchwork Thu May 19 18:17:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12855900 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5714C433EF for ; Thu, 19 May 2022 18:18:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243765AbiESSSX (ORCPT ); Thu, 19 May 2022 14:18:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243764AbiESSSK (ORCPT ); Thu, 19 May 2022 14:18:10 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 40B6FD0291 for ; Thu, 19 May 2022 11:18:09 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id u27so7503386wru.8 for ; Thu, 19 May 2022 11:18:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=R+TqfgqTPCf5Rm+tpY0Ass55F0F9IxWLmjxroeuUXKY=; b=HNx1JR0uZg2AGWjxNYWmuhVdjeAYjcXazE44HVCRPDnIdd+DJ9HZYi+0y8I7ZjZ1SD Xu/ACP7jLVHRXk+AbOVcGKyuy16NCdZyZqTSA/xBShTdCOi8xPinF4E8ksIllRYLxVYw u3mC73xrTmCE5UVV0HhiqKLVEjU5aE4BWfb2vR8LobIv0KKJeFpiDwK9qePzkVP2IU4t WeGlvwXi464sh8GyZATdgKW2hOIXze9tG29h6RIS28WtAyo+TSRQCDVVT5+ZPDUmj1be TZ/zJs7iaro3ZHUVbIL0p7Ivcy2IV8nOxig7719mCBY3/2QYoTYwJdIdlZ/Ngl38xARy naxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=R+TqfgqTPCf5Rm+tpY0Ass55F0F9IxWLmjxroeuUXKY=; b=1pySikrCLDO7hvvxSuGOuCAy9Zkm/bj6wkb2E1CphgalLni19Tb5GpccP8ZHJfro+H Z39HPoaz2o98W1oW8AWFnaR8IevE/ctmRyTPPB37b3QpNQp7ntF6aWNnFSac7YbgvZlb pTrZBtB9AyVYjNhcIlPLUKM9Dfqdf6PzmMs7vSskgZQnUm5o6j50Di1+KX/LE+jMyT6H osq18GpqqDZ01ztFbZ4fu7NN9rOSj1PcdeGb7h17Z6hARB2PGmobr2mOc+w5P4cQRAaA ++bCE9cEABUlOnBrwhEDNTDZL8vPwGvihOL2sdO7M+8+BpZ842teftUmtPptmk0XT/2i 57DA== X-Gm-Message-State: AOAM530MIcppiCGgk1jC6uncg3zfwl+kYu1tpAvxQV/A+FY+aKohAaRe S6Uo4gGK1YnYri9CotQvovL26qWl/14= X-Google-Smtp-Source: ABdhPJzRvrZavzEq/erCJ4imJve9fAlgA0oXxdTJpZ0qmZOoP2zqFxq+DOVsdZ52wF1+CnUP2Fin9Q== X-Received: by 2002:a05:6000:1889:b0:20e:67e6:dffb with SMTP id a9-20020a056000188900b0020e67e6dffbmr4121627wri.691.1652984287518; Thu, 19 May 2022 11:18:07 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t11-20020adfba4b000000b0020c6fa5a797sm248321wrg.91.2022.05.19.11.18.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 May 2022 11:18:06 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 19 May 2022 18:17:58 +0000 Subject: [PATCH v5 2/7] archive --add-file-with-contents: allow paths containing colons Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , rsbecker@nexbridge.com, =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsA==?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin By allowing the path to be enclosed in double-quotes, we can avoid the limitation that paths cannot contain colons. Signed-off-by: Johannes Schindelin --- Documentation/git-archive.txt | 14 ++++++++++---- archive.c | 30 ++++++++++++++++++++---------- t/t5003-archive-zip.sh | 8 ++++++++ 3 files changed, 38 insertions(+), 14 deletions(-) diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt index 893cb1075bf..54de945a84e 100644 --- a/Documentation/git-archive.txt +++ b/Documentation/git-archive.txt @@ -67,10 +67,16 @@ OPTIONS by concatenating the value for `--prefix` (if any) and the basename of . + -The `` cannot contain any colon, the file mode is limited to -a regular file, and the option may be subject to platform-dependent -command-line limits. For non-trivial cases, write an untracked file -and use `--add-file` instead. +The `` argument can start and end with a literal double-quote +character; The contained file name is interpreted as a C-style string, +i.e. the backslash is interpreted as escape character. The path must +be quoted if it contains a colon, to avoid the colon from being +misinterpreted as the separator between the path and the contents, or +if the path begins or ends with a double-quote character. ++ +The file mode is limited to a regular file, and the option may be +subject to platform-dependent command-line limits. For non-trivial +cases, write an untracked file and use `--add-file` instead. --worktree-attributes:: Look for attributes in .gitattributes files in the working tree diff --git a/archive.c b/archive.c index d20e16fa819..b7756b91200 100644 --- a/archive.c +++ b/archive.c @@ -9,6 +9,7 @@ #include "parse-options.h" #include "unpack-trees.h" #include "dir.h" +#include "quote.h" static char const * const archive_usage[] = { N_("git archive [] [...]"), @@ -533,22 +534,31 @@ static int add_file_cb(const struct option *opt, const char *arg, int unset) die(_("Not a regular file: %s"), path); info->content = NULL; /* read the file later */ } else if (!strcmp(opt->long_name, "add-virtual-file")) { - const char *colon = strchr(arg, ':'); - char *p; + struct strbuf buf = STRBUF_INIT; + const char *p = arg; + + if (*p != '"') + p = strchr(p, ':'); + else if (unquote_c_style(&buf, p, &p) < 0) + die(_("unclosed quote: '%s'"), arg); - if (!colon) + if (!p || *p != ':') die(_("missing colon: '%s'"), arg); - p = xstrndup(arg, colon - arg); - if (!args->prefix) - path = p; - else { - path = prefix_filename(args->prefix, p); - free(p); + if (p == arg) + die(_("empty file name: '%s'"), arg); + + path = buf.len ? + strbuf_detach(&buf, NULL) : xstrndup(arg, p - arg); + + if (args->prefix) { + char *save = path; + path = prefix_filename(args->prefix, path); + free(save); } memset(&info->stat, 0, sizeof(info->stat)); info->stat.st_mode = S_IFREG | 0644; - info->content = xstrdup(colon + 1); + info->content = xstrdup(p + 1); info->stat.st_size = strlen(info->content); } else { BUG("add_file_cb() called for %s", opt->long_name); diff --git a/t/t5003-archive-zip.sh b/t/t5003-archive-zip.sh index ebc26e89a9b..50932a866c9 100755 --- a/t/t5003-archive-zip.sh +++ b/t/t5003-archive-zip.sh @@ -207,13 +207,21 @@ check_zip with_untracked check_added with_untracked untracked untracked test_expect_success UNZIP 'git archive --format=zip --add-virtual-file' ' + if test_have_prereq FUNNYNAMES + then + QUOTED=quoted:colon + else + QUOTED=quoted + fi && git archive --format=zip >with_file_with_content.zip \ + --add-virtual-file=\"$QUOTED\": \ --add-virtual-file=hello:world $EMPTY_TREE && test_when_finished "rm -rf tmp-unpack" && mkdir tmp-unpack && ( cd tmp-unpack && "$GIT_UNZIP" ../with_file_with_content.zip && test_path_is_file hello && + test_path_is_file $QUOTED && test world = $(cat hello) ) ' From patchwork Thu May 19 18:17:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12855902 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EDB8C433FE for ; Thu, 19 May 2022 18:18:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243843AbiESSSg (ORCPT ); Thu, 19 May 2022 14:18:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33910 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243781AbiESSSL (ORCPT ); Thu, 19 May 2022 14:18:11 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ADF79DE31F for ; Thu, 19 May 2022 11:18:10 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id r9-20020a1c4409000000b00397345f2c6fso797332wma.4 for ; Thu, 19 May 2022 11:18:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Uo6VeR463A8bpTnDe4txbUadyp7c2hS7RwvOwDAMeOI=; b=jr/kfFl6+YEIPlnrX3/LC4o12aGClr/jSv5wkOAearLbm4+bFXMAF+9oJvakIaUM8s oMK/zeHMVera1DrInjM3TawVbm1WxZPnGWXxmIrgQLYXkdD4TJNfHyAXze1vieJcq7/o Sx42vWI2zRMR4SB+Z2/4bPaBMLyQ8R1pwa6G7amk6IrPhMINux6ImQg1UB9KvipNaL9K 5n0HN9QjIHKBr5rt8xomF+6FuTz8qp8yTRrK5qo+XeVgM2JZxVaZRRDTZBpwkXdRnPYh CGgB8sKSwGjg7WX/dJrWl5CD9KY4ElCb9R/D+Ibhn8n3d7eEYntXsZtYyxbEPT6QRDD1 GcTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Uo6VeR463A8bpTnDe4txbUadyp7c2hS7RwvOwDAMeOI=; b=3tA765fwlSrpBPff2lIqi8i0XmkvvDp74XQlxpQGLrRJhlQ/KxbbRnYSKR/Xnq4GgW dkD892BHQ9z3a2u0Fe0fQQSPQRITUK/cDt7f/RLgctRd0+DBuaToUc3sW6AqlA1daD4g Xq4yaJBd7sfTM6D/YV0e5Nvu566ZFbmqOj14Iv7N+93bmvEpSn/3JmX2LOoKrdEfNI6y wiAsVUEd0M1bic6E78LYBcEF4vUi0qDb0ssbUW0FYJ6ZTkWFkyckfXkvmT7a6/GYmh2I ebDiXZH2VOWyAxZnI07QIx3LIh32k1dPQ1vMxw8qDvL8tUvM4mxFSyoMZA3gaaU5MHJt qmvA== X-Gm-Message-State: AOAM5336AJQy40m6fgRbCORATfUeN+09BKqX+xe5ofQ/TYjYWJllZ8ay PP3XToqsLJWevTP6oW0qXDRCs52NFUI= X-Google-Smtp-Source: ABdhPJwslcmeIZWXzN4YdrZH6wW4ctcDNuvvHYQUu1Yp10pN3QAE52/w08Qe0l+bfymHOxYFpSGHIg== X-Received: by 2002:a7b:c5cd:0:b0:38c:8b1b:d220 with SMTP id n13-20020a7bc5cd000000b0038c8b1bd220mr4858188wmk.118.1652984288620; Thu, 19 May 2022 11:18:08 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o21-20020adfa115000000b0020c5253d8bdsm340377wro.9.2022.05.19.11.18.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 May 2022 11:18:08 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 19 May 2022 18:17:59 +0000 Subject: [PATCH v5 3/7] scalar: validate the optional enlistment argument Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , rsbecker@nexbridge.com, =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsA==?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin The `scalar` command needs a Scalar enlistment for many subcommands, and looks in the current directory for such an enlistment (traversing the parent directories until it finds one). These is subcommands can also be called with an optional argument specifying the enlistment. Here, too, we traverse parent directories as needed, until we find an enlistment. However, if the specified directory does not even exist, or is not a directory, we should stop right there, with an error message. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 6 ++++-- contrib/scalar/t/t9099-scalar.sh | 5 +++++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 1ce9c2b00e8..00dcd4b50ef 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -43,9 +43,11 @@ static void setup_enlistment_directory(int argc, const char **argv, usage_with_options(usagestr, options); /* find the worktree, determine its corresponding root */ - if (argc == 1) + if (argc == 1) { strbuf_add_absolute_path(&path, argv[0]); - else if (strbuf_getcwd(&path) < 0) + if (!is_directory(path.buf)) + die(_("'%s' does not exist"), path.buf); + } else if (strbuf_getcwd(&path) < 0) die(_("need a working directory")); strbuf_trim_trailing_dir_sep(&path); diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 2e1502ad45e..9d83fdf25e8 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -85,4 +85,9 @@ test_expect_success 'scalar delete with enlistment' ' test_path_is_missing cloned ' +test_expect_success '`scalar [...] ` errors out when dir is missing' ' + ! scalar run config cloned 2>err && + grep "cloned. does not exist" err +' + test_done From patchwork Thu May 19 18:18:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12855903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E207DC433F5 for ; Thu, 19 May 2022 18:18:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243836AbiESSSd (ORCPT ); Thu, 19 May 2022 14:18:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243784AbiESSSR (ORCPT ); Thu, 19 May 2022 14:18:17 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CEBE0EBA89 for ; Thu, 19 May 2022 11:18:11 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id a14-20020a7bc1ce000000b00393fb52a386so5516709wmj.1 for ; Thu, 19 May 2022 11:18:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=KOhCLOG4xQM7GXYpo7HgIfpmlRXN1CU5iK5b1yz4gpM=; b=duV1wzheyCYMYBqMxb6ZX5IZdTrjS6x+pGlbxcPbJoZzq/3irka7V5d3hyAcPx3345 DeC/WlO4aXPlNIAOo4GrztNAtjYXzXS5pb+C/smz4LW/ZztptZFbCMoC9H6kRRhdqRrO Bkg5IAuIUYYwPoTyuCyFsZvSPzPz9anHw4UxMC3jRzLuUq0hfgylRGG2/bGs1vj+WUPC EjH+GuJWbBSrWPtRdmRHCVkJCZUvao8Br75IjubJsHQ8KW7VV/5dqRq4HBUHbwf6+3Bs K5SdOLa/oBSFLyHmW0yPFOLMa1MesH/Zcd9bSS/JvoRy5EiegiKMcMb910RwNatAHi6K B9GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=KOhCLOG4xQM7GXYpo7HgIfpmlRXN1CU5iK5b1yz4gpM=; b=agVEmCqES+M4ndaZDP8G+Anmdzq7PsY0eAWngUQ5gksehpDW2YpulC9KCy7Pe2qmyM chivwOAGQroaLV+Bv9KqqzIO/udQ2ssktLVM3REoUri2x6sYAgHUT2XUcFkVFlpDsXIv P3zuIowACYeqIFMFHRvuLvXSCFKmFTU71YH3X/qBkC5J4V7NHuKlXEVYdR5aYNU0zocC OI3y/1MtTfCAz3ffcIwhlCv1W3gppXiuFMj1LqVCxpnQeUX/WIiv5uv3JWo80Ro45LVM 8kL/iHzwGxTbpfikCSi4Fed8cuOJzB9FMwKB3rDXolgA/I+swKuQj22CoQ5fO4V2atEO FN4g== X-Gm-Message-State: AOAM53250REfluADJtsFrdFkTeznoDBmhN7o6bIyrvZEF/CeVhr9yy+8 5hxP/9ZTGX8rQzF/Z553qhvLCDU/T6U= X-Google-Smtp-Source: ABdhPJxHQI0VkPJcMDbKGTx0HcY28gjtox2lliupieUyWKjcoBqUtVrcgXlEKafACuLV290LcqvuaA== X-Received: by 2002:a05:600c:19cf:b0:394:96c4:8959 with SMTP id u15-20020a05600c19cf00b0039496c48959mr5612694wmq.9.1652984289845; Thu, 19 May 2022 11:18:09 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h2-20020a1ccc02000000b0039466988f6csm179228wmb.31.2022.05.19.11.18.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 May 2022 11:18:09 -0700 (PDT) Message-Id: <3fb90194744998e5dd8f2e5b46610a7cc773a09b.1652984283.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 19 May 2022 18:18:00 +0000 Subject: [PATCH v5 4/7] Implement `scalar diagnose` Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , rsbecker@nexbridge.com, =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsA==?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin Over the course of Scalar's development, it became obvious that there is a need for a command that can gather all kinds of useful information that can help identify the most typical problems with large worktrees/repositories. The `diagnose` command is the culmination of this hard-won knowledge: it gathers the installed hooks, the config, a couple statistics describing the data shape, among other pieces of information, and then wraps everything up in a tidy, neat `.zip` archive. Note: originally, Scalar was implemented in C# using the .NET API, where we had the luxury of a comprehensive standard library that includes basic functionality such as writing a `.zip` file. In the C version, we lack such a commodity. Rather than introducing a dependency on, say, libzip, we slightly abuse Git's `archive` machinery: we write out a `.zip` of the empty try, augmented by a couple files that are added via the `--add-file*` options. We are careful trying not to modify the current repository in any way lest the very circumstances that required `scalar diagnose` to be run are changed by the `diagnose` run itself. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 144 +++++++++++++++++++++++++++++++ contrib/scalar/scalar.txt | 12 +++ contrib/scalar/t/t9099-scalar.sh | 14 +++ 3 files changed, 170 insertions(+) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 00dcd4b50ef..53213f9a3b9 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -11,6 +11,7 @@ #include "dir.h" #include "packfile.h" #include "help.h" +#include "archive.h" /* * Remove the deepest subdirectory in the provided path string. Path must not @@ -261,6 +262,47 @@ static int unregister_dir(void) return res; } +static int add_directory_to_archiver(struct strvec *archiver_args, + const char *path, int recurse) +{ + int at_root = !*path; + DIR *dir = opendir(at_root ? "." : path); + struct dirent *e; + struct strbuf buf = STRBUF_INIT; + size_t len; + int res = 0; + + if (!dir) + return error_errno(_("could not open directory '%s'"), path); + + if (!at_root) + strbuf_addf(&buf, "%s/", path); + len = buf.len; + strvec_pushf(archiver_args, "--prefix=%s", buf.buf); + + while (!res && (e = readdir(dir))) { + if (!strcmp(".", e->d_name) || !strcmp("..", e->d_name)) + continue; + + strbuf_setlen(&buf, len); + strbuf_addstr(&buf, e->d_name); + + if (e->d_type == DT_REG) + strvec_pushf(archiver_args, "--add-file=%s", buf.buf); + else if (e->d_type != DT_DIR) + warning(_("skipping '%s', which is neither file nor " + "directory"), buf.buf); + else if (recurse && + add_directory_to_archiver(archiver_args, + buf.buf, recurse) < 0) + res = -1; + } + + closedir(dir); + strbuf_release(&buf); + return res; +} + /* printf-style interface, expects `=` argument */ static int set_config(const char *fmt, ...) { @@ -501,6 +543,107 @@ cleanup: return res; } +static int cmd_diagnose(int argc, const char **argv) +{ + struct option options[] = { + OPT_END(), + }; + const char * const usage[] = { + N_("scalar diagnose []"), + NULL + }; + struct strbuf zip_path = STRBUF_INIT; + struct strvec archiver_args = STRVEC_INIT; + char **argv_copy = NULL; + int stdout_fd = -1, archiver_fd = -1; + time_t now = time(NULL); + struct tm tm; + struct strbuf path = STRBUF_INIT, buf = STRBUF_INIT; + int res = 0; + + argc = parse_options(argc, argv, NULL, options, + usage, 0); + + setup_enlistment_directory(argc, argv, usage, options, &zip_path); + + strbuf_addstr(&zip_path, "/.scalarDiagnostics/scalar_"); + strbuf_addftime(&zip_path, + "%Y%m%d_%H%M%S", localtime_r(&now, &tm), 0, 0); + strbuf_addstr(&zip_path, ".zip"); + switch (safe_create_leading_directories(zip_path.buf)) { + case SCLD_EXISTS: + case SCLD_OK: + break; + default: + error_errno(_("could not create directory for '%s'"), + zip_path.buf); + goto diagnose_cleanup; + } + stdout_fd = dup(1); + if (stdout_fd < 0) { + res = error_errno(_("could not duplicate stdout")); + goto diagnose_cleanup; + } + + archiver_fd = xopen(zip_path.buf, O_CREAT | O_WRONLY | O_TRUNC, 0666); + if (archiver_fd < 0 || dup2(archiver_fd, 1) < 0) { + res = error_errno(_("could not redirect output")); + goto diagnose_cleanup; + } + + init_zip_archiver(); + strvec_pushl(&archiver_args, "scalar-diagnose", "--format=zip", NULL); + + strbuf_reset(&buf); + strbuf_addstr(&buf, "Collecting diagnostic info\n\n"); + get_version_info(&buf, 1); + + strbuf_addf(&buf, "Enlistment root: %s\n", the_repository->worktree); + write_or_die(stdout_fd, buf.buf, buf.len); + strvec_pushf(&archiver_args, + "--add-virtual-file=diagnostics.log:%.*s", + (int)buf.len, buf.buf); + + if ((res = add_directory_to_archiver(&archiver_args, ".git", 0)) || + (res = add_directory_to_archiver(&archiver_args, ".git/hooks", 0)) || + (res = add_directory_to_archiver(&archiver_args, ".git/info", 0)) || + (res = add_directory_to_archiver(&archiver_args, ".git/logs", 1)) || + (res = add_directory_to_archiver(&archiver_args, ".git/objects/info", 0))) + goto diagnose_cleanup; + + strvec_pushl(&archiver_args, "--prefix=", + oid_to_hex(the_hash_algo->empty_tree), "--", NULL); + + /* `write_archive()` modifies the `argv` passed to it. Let it. */ + argv_copy = xmemdupz(archiver_args.v, + sizeof(char *) * archiver_args.nr); + res = write_archive(archiver_args.nr, (const char **)argv_copy, NULL, + the_repository, NULL, 0); + if (res) { + error(_("failed to write archive")); + goto diagnose_cleanup; + } + + if (!res) + fprintf(stderr, "\n" + "Diagnostics complete.\n" + "All of the gathered info is captured in '%s'\n", + zip_path.buf); + +diagnose_cleanup: + if (archiver_fd >= 0) { + close(1); + dup2(stdout_fd, 1); + } + free(argv_copy); + strvec_clear(&archiver_args); + strbuf_release(&zip_path); + strbuf_release(&path); + strbuf_release(&buf); + + return res; +} + static int cmd_list(int argc, const char **argv) { if (argc != 1) @@ -802,6 +945,7 @@ static struct { { "reconfigure", cmd_reconfigure }, { "delete", cmd_delete }, { "version", cmd_version }, + { "diagnose", cmd_diagnose }, { NULL, NULL}, }; diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt index f416d637289..22583fe046e 100644 --- a/contrib/scalar/scalar.txt +++ b/contrib/scalar/scalar.txt @@ -14,6 +14,7 @@ scalar register [] scalar unregister [] scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [] scalar reconfigure [ --all | ] +scalar diagnose [] scalar delete DESCRIPTION @@ -129,6 +130,17 @@ reconfigure the enlistment. With the `--all` option, all enlistments currently registered with Scalar will be reconfigured. Use this option after each Scalar upgrade. +Diagnose +~~~~~~~~ + +diagnose []:: + When reporting issues with Scalar, it is often helpful to provide the + information gathered by this command, including logs and certain + statistics describing the data shape of the current enlistment. ++ +The output of this command is a `.zip` file that is written into +a directory adjacent to the worktree in the `src` directory. + Delete ~~~~~~ diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 9d83fdf25e8..6802d317258 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -90,4 +90,18 @@ test_expect_success '`scalar [...] ` errors out when dir is missing' ' grep "cloned. does not exist" err ' +SQ="'" +test_expect_success UNZIP 'scalar diagnose' ' + scalar clone "file://$(pwd)" cloned --single-branch && + scalar diagnose cloned >out 2>err && + sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && + zip_path=$(cat zip_path) && + test -n "$zip_path" && + unzip -v "$zip_path" && + folder=${zip_path%.zip} && + test_path_is_missing "$folder" && + unzip -p "$zip_path" diagnostics.log >out && + test_file_not_empty out +' + test_done From patchwork Thu May 19 18:18:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12855901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 884A4C433EF for ; Thu, 19 May 2022 18:18:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243827AbiESSS0 (ORCPT ); Thu, 19 May 2022 14:18:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243789AbiESSSR (ORCPT ); Thu, 19 May 2022 14:18:17 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DB9C6EBA9D for ; Thu, 19 May 2022 11:18:12 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id r23so8355777wrr.2 for ; Thu, 19 May 2022 11:18:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=sXZbh0MamYACiDTr6HvklkpnU4A4eZMx4voLUuNMvpg=; b=G5DoKSlVFslUU7jbeQnIGBLc6yl8wzceirTENODVQtDmC5nI9TnuKS7L93eUZ1rMUt fnFIHexfU3Cmai2WD7dPU9X7GIt9Yn3wfalBYuQluliFcStA3ZxFgnrwlv13DU9qrTmz Hj4RZf3vryUpFAYiKCMKqARLL21DXPEjxNwkCyMBFnOZ2myXVsKU5KlDd8XecZIfDaVh kR9FLprY67WZr7krxqEzZaP3CB6vkQaRTB+46WxpJomYgTUWXio2MU5Cj6fg4Rp4JSzq BEktmRQL4VBJQtW/stcvOLTAzTWn6o/9Xmgn5tBxIR+g/mDh4J2Q4DsnBzFhfL7CV6QO SoOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=sXZbh0MamYACiDTr6HvklkpnU4A4eZMx4voLUuNMvpg=; b=Za6WS3dBTyH5k7HiiT3QCpZBpejnD0xH10/yzNJ+dG5lLy6YYGddLBQIC+nFFExCfY lF9Q4skyT8tofs/sgMy0esHMZ6GSD+VD1R7pFdPpOgVtFdv6IN2pJitNT3PdyIPaJth3 H+FpfjWItm4jHvXSkI9/RZrjyQeU/ZGDSMw8u/AS4NYha3wlNObdPrBHogL1335Pqznh M3uvYEfLwIx/hb5lnbXAYgZsTpM0r+/yd3obZvYe1jOQLIXqTQuHvCKibUJ0072/fFev iHcmsjcHsSv2xZJmQlXxHpd1QekYACKN4xZWezenSnT7ohcMXuL3tCkvdlx305TGt+2Q cbFg== X-Gm-Message-State: AOAM532moju6fRuqPSVHgDg47kTCV5MA1dJBHbEHMXvr23IxN/lDSic+ VRlVZKN4w8lKtz6Rju0daIiMCbwKtQU= X-Google-Smtp-Source: ABdhPJxcGU4Fcf8epFbzHLP+68MwuW6g9ZJzoO2w2EuhjqMtg/lwNZebHC8PFtA4IDHIbsqY+aWi+A== X-Received: by 2002:a5d:52ce:0:b0:20c:e030:b05a with SMTP id r14-20020a5d52ce000000b0020ce030b05amr5003081wrv.174.1652984291130; Thu, 19 May 2022 11:18:11 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u16-20020a05600c035000b00396fbf6f524sm282950wmd.1.2022.05.19.11.18.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 May 2022 11:18:10 -0700 (PDT) Message-Id: <2e645b08a9e3c8505d4c525dadddd1f3eac77201.1652984283.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 19 May 2022 18:18:01 +0000 Subject: [PATCH v5 5/7] scalar diagnose: include disk space information Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , rsbecker@nexbridge.com, =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsA==?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin When analyzing problems with large worktrees/repositories, it is useful to know how close to a "full disk" situation Scalar/Git operates. Let's include this information. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 53 ++++++++++++++++++++++++++++++++ contrib/scalar/t/t9099-scalar.sh | 1 + 2 files changed, 54 insertions(+) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 53213f9a3b9..0a9e25a57f8 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -303,6 +303,58 @@ static int add_directory_to_archiver(struct strvec *archiver_args, return res; } +#ifndef WIN32 +#include +#endif + +static int get_disk_info(struct strbuf *out) +{ +#ifdef WIN32 + struct strbuf buf = STRBUF_INIT; + char volume_name[MAX_PATH], fs_name[MAX_PATH]; + DWORD serial_number, component_length, flags; + ULARGE_INTEGER avail2caller, total, avail; + + strbuf_realpath(&buf, ".", 1); + if (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) { + error(_("could not determine free disk size for '%s'"), + buf.buf); + strbuf_release(&buf); + return -1; + } + + strbuf_setlen(&buf, offset_1st_component(buf.buf)); + if (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name), + &serial_number, &component_length, &flags, + fs_name, sizeof(fs_name))) { + error(_("could not get info for '%s'"), buf.buf); + strbuf_release(&buf); + return -1; + } + strbuf_addf(out, "Available space on '%s': ", buf.buf); + strbuf_humanise_bytes(out, avail2caller.QuadPart); + strbuf_addch(out, '\n'); + strbuf_release(&buf); +#else + struct strbuf buf = STRBUF_INIT; + struct statvfs stat; + + strbuf_realpath(&buf, ".", 1); + if (statvfs(buf.buf, &stat) < 0) { + error_errno(_("could not determine free disk size for '%s'"), + buf.buf); + strbuf_release(&buf); + return -1; + } + + strbuf_addf(out, "Available space on '%s': ", buf.buf); + strbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail)); + strbuf_addf(out, " (mount flags 0x%lx)\n", stat.f_flag); + strbuf_release(&buf); +#endif + return 0; +} + /* printf-style interface, expects `=` argument */ static int set_config(const char *fmt, ...) { @@ -599,6 +651,7 @@ static int cmd_diagnose(int argc, const char **argv) get_version_info(&buf, 1); strbuf_addf(&buf, "Enlistment root: %s\n", the_repository->worktree); + get_disk_info(&buf); write_or_die(stdout_fd, buf.buf, buf.len); strvec_pushf(&archiver_args, "--add-virtual-file=diagnostics.log:%.*s", diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 6802d317258..934b2485d91 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -94,6 +94,7 @@ SQ="'" test_expect_success UNZIP 'scalar diagnose' ' scalar clone "file://$(pwd)" cloned --single-branch && scalar diagnose cloned >out 2>err && + grep "Available space" out && sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && zip_path=$(cat zip_path) && test -n "$zip_path" && From patchwork Thu May 19 18:18:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 12855904 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07A0CC433EF for ; Thu, 19 May 2022 18:18:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243796AbiESSSg (ORCPT ); Thu, 19 May 2022 14:18:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243803AbiESSSR (ORCPT ); Thu, 19 May 2022 14:18:17 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 204F9EBAA2 for ; Thu, 19 May 2022 11:18:14 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id k126-20020a1ca184000000b003943fd07180so3126094wme.3 for ; Thu, 19 May 2022 11:18:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=q4Nq0XKv3sAz87ZRO+MCbKnQgUW64Vx3dbiERrPUd8U=; b=EvnVBEOJ4Jmyf69+PNIiXo+BW/tNcX9n9OSGjMeIsDqbn9bb66hPHfXTL0PgFci7Rr 52Z122Q+bD+bVBz3EIFm7KSlKY7DDlPpa7Au6yfuuSuDz+825T70wfmHbIJLM97caaJy WtcLMtNEgkdJFrHbD6n3/ayGnqBuAqC9LrOVnMoOXCcKXN0E8p1s2ep/QQ5BBs7s7uB9 twH5jWqWBcQyuMg5rWW95ae+B5tVaQl0Gpek1WuCcoyLESQ0oS8l5FjM5yVSMLTNcuml nnawgvFFnEGTkJW4jox6yPVnwyIQUxIr3uWVhBEIMOhaMrUU2H7WU7px3yzyadJj4ZTU gBhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=q4Nq0XKv3sAz87ZRO+MCbKnQgUW64Vx3dbiERrPUd8U=; b=JscRgLkF/RqBOg56X+txK4OzjShpccIpZMYxrkyfaLTCOO/yiSlgSXbTuT85QWG9Eu tKVTTdeBnC4QRCtrm1mAVXJPpeXp5tB/SqtZx/uswkl9bqLMdwg+lOd3WO4f+6DVYVpD Mgr9Ct2xvrBgKAAy/FYFRj1tfc+yvP+g4hAtlZzvEZ7XK1QSmLd+x5Wh8pSGXOsL/J70 Dnlry38WYJY+xCEib5F/blhc/WolkKWBSwNensWLWHtGfL5JJwfEeyR7E+CapCytt8Wb DaElUAqjBk/yIHkYMbjaTHQX2pBw3qg+pKYqRDBaWwqQdQQ6kv59iczMOrqpHhgX9PUL zEIg== X-Gm-Message-State: AOAM532W29qK8uOjYeIKgLEOsZX1F8900QKqKY8NeMelUn/1VwU778sq RxxGDkYDIjYPojKY0sD/1+8swZJXa1g= X-Google-Smtp-Source: ABdhPJw/l8jY33z8WGVNn/cEudQsF9/dit6iF0qln6SVi5A9wd+uCZCfVrDTrrJQvlN4bOSVAvMUIg== X-Received: by 2002:a05:600c:5112:b0:394:55bd:5f9d with SMTP id o18-20020a05600c511200b0039455bd5f9dmr5603731wms.188.1652984292343; Thu, 19 May 2022 11:18:12 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t11-20020adfba4b000000b0020c6fa5a797sm248478wrg.91.2022.05.19.11.18.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 May 2022 11:18:11 -0700 (PDT) Message-Id: <0fa20d7375009999d978da682c3f98a604cda9e4.1652984283.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 19 May 2022 18:18:02 +0000 Subject: [PATCH v5 6/7] scalar: teach `diagnose` to gather packfile info Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , rsbecker@nexbridge.com, =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsA==?= Bjarmason , Johannes Schindelin , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham It's helpful to see if there are other crud files in the pack directory. Let's teach the `scalar diagnose` command to gather file size information about pack files. While at it, also enumerate the pack files in the alternate object directories, if any are registered. Signed-off-by: Matthew John Cheetham Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 30 ++++++++++++++++++++++++++++++ contrib/scalar/t/t9099-scalar.sh | 6 +++++- 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 0a9e25a57f8..d302c27e114 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -12,6 +12,7 @@ #include "packfile.h" #include "help.h" #include "archive.h" +#include "object-store.h" /* * Remove the deepest subdirectory in the provided path string. Path must not @@ -595,6 +596,29 @@ cleanup: return res; } +static void dir_file_stats_objects(const char *full_path, size_t full_path_len, + const char *file_name, void *data) +{ + struct strbuf *buf = data; + struct stat st; + + if (!stat(full_path, &st)) + strbuf_addf(buf, "%-70s %16" PRIuMAX "\n", file_name, + (uintmax_t)st.st_size); +} + +static int dir_file_stats(struct object_directory *object_dir, void *data) +{ + struct strbuf *buf = data; + + strbuf_addf(buf, "Contents of %s:\n", object_dir->path); + + for_each_file_in_pack_dir(object_dir->path, dir_file_stats_objects, + data); + + return 0; +} + static int cmd_diagnose(int argc, const char **argv) { struct option options[] = { @@ -657,6 +681,12 @@ static int cmd_diagnose(int argc, const char **argv) "--add-virtual-file=diagnostics.log:%.*s", (int)buf.len, buf.buf); + strbuf_reset(&buf); + strbuf_addstr(&buf, "--add-virtual-file=packs-local.txt:"); + dir_file_stats(the_repository->objects->odb, &buf); + foreach_alt_odb(dir_file_stats, &buf); + strvec_push(&archiver_args, buf.buf); + if ((res = add_directory_to_archiver(&archiver_args, ".git", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/hooks", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/info", 0)) || diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 934b2485d91..3dd5650cceb 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -93,6 +93,8 @@ test_expect_success '`scalar [...] ` errors out when dir is missing' ' SQ="'" test_expect_success UNZIP 'scalar diagnose' ' scalar clone "file://$(pwd)" cloned --single-branch && + git repack && + echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates && scalar diagnose cloned >out 2>err && grep "Available space" out && sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && @@ -102,7 +104,9 @@ test_expect_success UNZIP 'scalar diagnose' ' folder=${zip_path%.zip} && test_path_is_missing "$folder" && unzip -p "$zip_path" diagnostics.log >out && - test_file_not_empty out + test_file_not_empty out && + unzip -p "$zip_path" packs-local.txt >out && + grep "$(pwd)/.git/objects" out ' test_done From patchwork Thu May 19 18:18:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 12855905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C91A7C433F5 for ; Thu, 19 May 2022 18:18:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243849AbiESSSj (ORCPT ); Thu, 19 May 2022 14:18:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34094 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243826AbiESSSS (ORCPT ); Thu, 19 May 2022 14:18:18 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 471D0EBA97 for ; Thu, 19 May 2022 11:18:15 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id p189so3330331wmp.3 for ; Thu, 19 May 2022 11:18:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tFFHzUyw+itlS2o1LNy+a7mqx3/e2BvK13aAU/vxYfE=; b=Ym7QbPffadr9NKxcfyAacV2ivN8P71VbW3eXlbgp5DyKpMR/9beh3w0HVt3ZXfGE+W dW3PdbC4ZrT0B5WBcYPHXlIw7soZksygYQm0ErxiPdQIMfLq8/gecX16n3zg1IV2hFOb 8o7t/NKuJE+IKqcuLnmgduPJ6/+wHn/UqoMQiPMesLrg1P5uzQo4/hK3kPbSYXHw8unQ 6UniIunPNMa/eRqPTAk96rIxi0oAM9DC6DQKtHENrnE+rkuMgtuOMWlHnBOTxuO9dtpH J5kSOmyj97W/E8n877Yo5tER0erg+hjPOs0eFocb1cV8iKM6lflGvIx+HXmcgJSu8uJe JjQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tFFHzUyw+itlS2o1LNy+a7mqx3/e2BvK13aAU/vxYfE=; b=SAQH3F1RUf8v27dV/bPR/xe1ysSm3wWeuLtnP353eXrW5pdpyohLJMrIx0AYs+H9jD Y34Fgfshx7GCn8k7EIM45ibMiuzd+38OncMPHVW+wFiWr4tvIMhM4b4AOdZe+zwHdZuh GMNQ+0IRdqx3ShpPE0j8wJK6hCdAyiq3hOrAh0woWucsTXLfZ9ePB3DY4M0tTxGshbNa 6E/+60BHto3SmeB/huJgP8q4DQnXqwx5nhxl8wibr0AcWSqdoooposcYs1vbFjYuoSYx sS863Jqh5bWoLED/dXO8IbPjFpmoe2J3UrUzqGIf5rUFwe8Em3s9UGrkH/ZG2xHNuiQB w70A== X-Gm-Message-State: AOAM533IN8hBkioNRTB/60SDhAVR1so2JyB53y/WsGVnc/3Bqr/7PKCS mYNGk4lfCVZuqHs0k6JkFkLbQC270Lg= X-Google-Smtp-Source: ABdhPJz0ZozQQEvJKLs/77LHdPL4w3jTqD7CiFHhVURyax0YUl+Z+bqyG1MDUYd74xDSMRCdUFkISQ== X-Received: by 2002:a7b:c8c7:0:b0:394:9e02:baa with SMTP id f7-20020a7bc8c7000000b003949e020baamr5353590wml.26.1652984293470; Thu, 19 May 2022 11:18:13 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b9-20020adf9b09000000b0020c5253d8dfsm257932wrc.43.2022.05.19.11.18.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 May 2022 11:18:12 -0700 (PDT) Message-Id: <62e173b47cf2550a8e458c4d2cf667402b0c8ee3.1652984283.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 19 May 2022 18:18:03 +0000 Subject: [PATCH v5 7/7] scalar: teach `diagnose` to gather loose objects information Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , rsbecker@nexbridge.com, =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsA==?= Bjarmason , Johannes Schindelin , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham When operating at the scale that Scalar wants to support, certain data shapes are more likely to cause undesirable performance issues, such as large numbers of loose objects. By including statistics about this, `scalar diagnose` now makes it easier to identify such scenarios. Signed-off-by: Matthew John Cheetham Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 59 ++++++++++++++++++++++++++++++++ contrib/scalar/t/t9099-scalar.sh | 5 ++- 2 files changed, 63 insertions(+), 1 deletion(-) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index d302c27e114..0c278681758 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -619,6 +619,60 @@ static int dir_file_stats(struct object_directory *object_dir, void *data) return 0; } +static int count_files(char *path) +{ + DIR *dir = opendir(path); + struct dirent *e; + int count = 0; + + if (!dir) + return 0; + + while ((e = readdir(dir)) != NULL) + if (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG) + count++; + + closedir(dir); + return count; +} + +static void loose_objs_stats(struct strbuf *buf, const char *path) +{ + DIR *dir = opendir(path); + struct dirent *e; + int count; + int total = 0; + unsigned char c; + struct strbuf count_path = STRBUF_INIT; + size_t base_path_len; + + if (!dir) + return; + + strbuf_addstr(buf, "Object directory stats for "); + strbuf_add_absolute_path(buf, path); + strbuf_addstr(buf, ":\n"); + + strbuf_add_absolute_path(&count_path, path); + strbuf_addch(&count_path, '/'); + base_path_len = count_path.len; + + while ((e = readdir(dir)) != NULL) + if (!is_dot_or_dotdot(e->d_name) && + e->d_type == DT_DIR && strlen(e->d_name) == 2 && + !hex_to_bytes(&c, e->d_name, 1)) { + strbuf_setlen(&count_path, base_path_len); + strbuf_addstr(&count_path, e->d_name); + total += (count = count_files(count_path.buf)); + strbuf_addf(buf, "%s : %7d files\n", e->d_name, count); + } + + strbuf_addf(buf, "Total: %d loose objects", total); + + strbuf_release(&count_path); + closedir(dir); +} + static int cmd_diagnose(int argc, const char **argv) { struct option options[] = { @@ -687,6 +741,11 @@ static int cmd_diagnose(int argc, const char **argv) foreach_alt_odb(dir_file_stats, &buf); strvec_push(&archiver_args, buf.buf); + strbuf_reset(&buf); + strbuf_addstr(&buf, "--add-virtual-file=objects-local.txt:"); + loose_objs_stats(&buf, ".git/objects"); + strvec_push(&archiver_args, buf.buf); + if ((res = add_directory_to_archiver(&archiver_args, ".git", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/hooks", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/info", 0)) || diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 3dd5650cceb..72023a1ca1d 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -95,6 +95,7 @@ test_expect_success UNZIP 'scalar diagnose' ' scalar clone "file://$(pwd)" cloned --single-branch && git repack && echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates && + test_commit -C cloned/src loose && scalar diagnose cloned >out 2>err && grep "Available space" out && sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && @@ -106,7 +107,9 @@ test_expect_success UNZIP 'scalar diagnose' ' unzip -p "$zip_path" diagnostics.log >out && test_file_not_empty out && unzip -p "$zip_path" packs-local.txt >out && - grep "$(pwd)/.git/objects" out + grep "$(pwd)/.git/objects" out && + unzip -p "$zip_path" objects-local.txt >out && + grep "^Total: [1-9]" out ' test_done