From patchwork Sun Feb 6 22:39:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12736766 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B398C433F5 for ; Sun, 6 Feb 2022 22:39:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240865AbiBFWjN (ORCPT ); Sun, 6 Feb 2022 17:39:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239624AbiBFWjL (ORCPT ); Sun, 6 Feb 2022 17:39:11 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F253C061348 for ; Sun, 6 Feb 2022 14:39:10 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id w11so21691402wra.4 for ; Sun, 06 Feb 2022 14:39:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=aBKUUrI4uvsCfdz7CTI77NcsvhLomTqrbD8OMq5shmg=; b=KPYIIIQJQvQy9PelkbGuZXZxwashhVnrrDgQ8ZN72SF3vfOVPcA8uYyuBdYUvKOAQd +rQG67c4hx1pUWncefGk6/MX0HWi4e6ZK061qewBFozBr1qllMqMGKQkj5hg8oU8/D/1 JgDvmJlwj+voCtnxnmsJQ8svRASjoZl5eL4+DVyuBc4wjkWHr1SUA4fkS5D+3rqVA4KS a7yQ4t6FgKW0uSf8b+vYSsvfXVlYXupltcuSOzO2qaGSRYry7kaZHu0K8KCV/JoAIeqM MlYFMX3LZg2IrDeX045kYrq3qNv4YL1bsI03u+WK7F2aehxVv3AE8AwvC3JoEfS8TvOc Cr/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=aBKUUrI4uvsCfdz7CTI77NcsvhLomTqrbD8OMq5shmg=; b=lBuGZAVYOWfbg+LFxtqlfNq//5kphkIRDhf5Y1iWLktyZNL5Mf0V8+/ZmqYatuvANq p18aCxq/FyBtes7RmAMj+3yszN3eRSjMGMPjV+3C8gLnifzlRD4kCmOqbqLL7dhCd35H axWgtW/vUjsE3+YGV4dPUS1yq+NVtKQStgA998DqpBNKfV3cYAhIEnekQaBE4RQq2nWu 7HCJAFp666QqunTXWaGeVcR5Ib68ulOkspvUVGaqIwgG/UsZx3auda/AkphcHctLy8o8 Y8eQvk130kMOecQ66zSd5tjp2J+HJiy/nsFV1UDwwI9DhI0cVMY9IqmozxM2yNrcDmxX 5u+Q== X-Gm-Message-State: AOAM5312W8iwPvzXVE2yxvY37i7OfntQ6AFXJC1jYe1u0XvX+ud3hUJM Uc8P5OCG719GliQSdHH+cebwu5KcUXg= X-Google-Smtp-Source: ABdhPJzb4dwlgvSrLc9o1228esc89OBp8RMI49Vjd2P8OJDCofFGkE1dh9pAJUlpKEstvAihKrbveQ== X-Received: by 2002:a05:6000:156d:: with SMTP id 13mr7805387wrz.16.1644187148242; Sun, 06 Feb 2022 14:39:08 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j19sm17696867wmq.17.2022.02.06.14.39.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 06 Feb 2022 14:39:07 -0800 (PST) Message-Id: <49ff3c1f2b32b16df2b4216aa016d715b6de46bc.1644187146.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sun, 06 Feb 2022 22:39:01 +0000 Subject: [PATCH v2 1/6] archive: optionally add "virtual" files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin With the `--add-file-with-content=:` option, `git archive` now supports use cases where relatively trivial files need to be added that do not exist on disk. Signed-off-by: Johannes Schindelin Signed-off-by: René Scharfe --- Documentation/git-archive.txt | 11 ++++++++ archive.c | 51 +++++++++++++++++++++++++++++------ t/t5003-archive-zip.sh | 12 +++++++++ 3 files changed, 66 insertions(+), 8 deletions(-) diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt index bc4e76a7834..1b52a0a65a1 100644 --- a/Documentation/git-archive.txt +++ b/Documentation/git-archive.txt @@ -61,6 +61,17 @@ OPTIONS by concatenating the value for `--prefix` (if any) and the basename of . +--add-file-with-content=::: + Add the specified contents to the archive. Can be repeated to add + multiple files. The path of the file in the archive is built + by concatenating the value for `--prefix` (if any) and the + basename of . ++ +The `` cannot contain any colon, the file mode is limited to +a regular file, and the option may be subject platform-dependent +command-line limits. For non-trivial cases, write an untracked file +and use `--add-file` instead. + --worktree-attributes:: Look for attributes in .gitattributes files in the working tree as well (see <>). diff --git a/archive.c b/archive.c index a3bbb091256..172efd690c3 100644 --- a/archive.c +++ b/archive.c @@ -263,6 +263,7 @@ static int queue_or_write_archive_entry(const struct object_id *oid, struct extra_file_info { char *base; struct stat stat; + void *content; }; int write_archive_entries(struct archiver_args *args, @@ -337,7 +338,13 @@ int write_archive_entries(struct archiver_args *args, strbuf_addstr(&path_in_archive, basename(path)); strbuf_reset(&content); - if (strbuf_read_file(&content, path, info->stat.st_size) < 0) + if (info->content) + err = write_entry(args, &fake_oid, path_in_archive.buf, + path_in_archive.len, + info->stat.st_mode, + info->content, info->stat.st_size); + else if (strbuf_read_file(&content, path, + info->stat.st_size) < 0) err = error_errno(_("could not read '%s'"), path); else err = write_entry(args, &fake_oid, path_in_archive.buf, @@ -493,6 +500,7 @@ static void extra_file_info_clear(void *util, const char *str) { struct extra_file_info *info = util; free(info->base); + free(info->content); free(info); } @@ -514,14 +522,38 @@ static int add_file_cb(const struct option *opt, const char *arg, int unset) if (!arg) return -1; - path = prefix_filename(args->prefix, arg); - item = string_list_append_nodup(&args->extra_files, path); - item->util = info = xmalloc(sizeof(*info)); + info = xmalloc(sizeof(*info)); info->base = xstrdup_or_null(base); - if (stat(path, &info->stat)) - die(_("File not found: %s"), path); - if (!S_ISREG(info->stat.st_mode)) - die(_("Not a regular file: %s"), path); + + if (strcmp(opt->long_name, "add-file-with-content")) { + path = prefix_filename(args->prefix, arg); + if (stat(path, &info->stat)) + die(_("File not found: %s"), path); + if (!S_ISREG(info->stat.st_mode)) + die(_("Not a regular file: %s"), path); + info->content = NULL; /* read the file later */ + } else { + const char *colon = strchr(arg, ':'); + char *p; + + if (!colon) + die(_("missing colon: '%s'"), arg); + + p = xstrndup(arg, colon - arg); + if (!args->prefix) + path = p; + else { + path = prefix_filename(args->prefix, p); + free(p); + } + memset(&info->stat, 0, sizeof(info->stat)); + info->stat.st_mode = S_IFREG | 0644; + info->content = xstrdup(colon + 1); + info->stat.st_size = strlen(info->content); + } + item = string_list_append_nodup(&args->extra_files, path); + item->util = info; + return 0; } @@ -554,6 +586,9 @@ static int parse_archive_args(int argc, const char **argv, { OPTION_CALLBACK, 0, "add-file", args, N_("file"), N_("add untracked file to archive"), 0, add_file_cb, (intptr_t)&base }, + { OPTION_CALLBACK, 0, "add-file-with-content", args, + N_("file"), N_("add untracked file to archive"), 0, + add_file_cb, (intptr_t)&base }, OPT_STRING('o', "output", &output, N_("file"), N_("write the archive to this file")), OPT_BOOL(0, "worktree-attributes", &worktree_attributes, diff --git a/t/t5003-archive-zip.sh b/t/t5003-archive-zip.sh index 1e6d18b140e..8ff1257f1a0 100755 --- a/t/t5003-archive-zip.sh +++ b/t/t5003-archive-zip.sh @@ -206,6 +206,18 @@ test_expect_success 'git archive --format=zip --add-file' ' check_zip with_untracked check_added with_untracked untracked untracked +test_expect_success UNZIP 'git archive --format=zip --add-file-with-content' ' + git archive --format=zip >with_file_with_content.zip \ + --add-file-with-content=hello:world $EMPTY_TREE && + test_when_finished "rm -rf tmp-unpack" && + mkdir tmp-unpack && ( + cd tmp-unpack && + "$GIT_UNZIP" ../with_file_with_content.zip && + test_path_is_file hello && + test world = $(cat hello) + ) +' + test_expect_success 'git archive --format=zip --add-file twice' ' echo untracked >untracked && git archive --format=zip --prefix=one/ --add-file=untracked \ From patchwork Sun Feb 6 22:39:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12736765 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E5A7C433F5 for ; Sun, 6 Feb 2022 22:39:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240197AbiBFWjM (ORCPT ); Sun, 6 Feb 2022 17:39:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41704 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239688AbiBFWjL (ORCPT ); Sun, 6 Feb 2022 17:39:11 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94FCBC061355 for ; Sun, 6 Feb 2022 14:39:10 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id m14so21608024wrg.12 for ; Sun, 06 Feb 2022 14:39:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Uo6VeR463A8bpTnDe4txbUadyp7c2hS7RwvOwDAMeOI=; b=dCNGMXNIMdLaT/YxRzx8sRuXGOYQc9S+i6T284x0aa672lVRKLq2iNwwtY7cTNUNCO kzUAjBqOUFwLt67fmZpRUhyL1zzUJWZCfI6yCV8zCL0EELfChL7ijqz+gNWXm4UJhVFq dnSJK+ZROs4qYU08oIThS/OoXcaEcwuROanhv35tAUdZbCUVhlHl17a9cDvZZEz8SaV3 o1c1CHkbbOIZKIF6Y8pW0Fpg0/vi1AbdUCgKDzN0GgMvgUyl5eP9wq3IeQS8ymQeXP2D vSDCSK8KmE+lZ8cDa0qeSpuEoHGUW5LEO+BVzZu3mEZD3dx7JZHwru2oVBYvUqsAiWKg ieTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Uo6VeR463A8bpTnDe4txbUadyp7c2hS7RwvOwDAMeOI=; b=6fr6EDUlmyx4GuFeamyScIm4P2ETnzIzgYsKfGKMlgg5+vxkH2vv49a9LPlCNVUMF2 rgjEjqwpsiqxRKB5+B1wNt/N51r2l1neMDeQl93dbumjh94mCzoVoE3aExA8Zbq+AFfl CBprpnuzccSAne8aVOyLGXl0nv+l97ALchVjDJjC+mGDnHwkxgWSoImpv45g5LyZSKE+ YB5y7fOO5cLTinNDkS5qtvrV9M3uSJwZJfsxrVB1ZIoxj+DMeU9zCLhVMdHfAmofms1t wMYc0mVc1RIVrlUT40L9vEPfbqwGTMi+G7sE68wwut2HCFaEgBTovTEvz31Fq0JE2zqM oUiA== X-Gm-Message-State: AOAM533rhQHmP4ZyHhbJhg1vCRIetKmqWgzE0KVEcU2xVvd+Cste1FMS IHfc+AM/z9yNG9MDuYq94usTqRvYH+0= X-Google-Smtp-Source: ABdhPJxt70YPjEp9YZ7ZqU/2QqDB5cdlOk5D7YxRtC7GfXiSPdShJoJEbGZGUYKtx2bG5xzZA+Xa+w== X-Received: by 2002:a5d:610f:: with SMTP id v15mr1493074wrt.602.1644187149105; Sun, 06 Feb 2022 14:39:09 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b10sm5647136wrd.8.2022.02.06.14.39.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 06 Feb 2022 14:39:08 -0800 (PST) Message-Id: <600da8d465ef8e83c20cdf6fec85aff08eb65551.1644187146.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sun, 06 Feb 2022 22:39:02 +0000 Subject: [PATCH v2 2/6] scalar: validate the optional enlistment argument Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin The `scalar` command needs a Scalar enlistment for many subcommands, and looks in the current directory for such an enlistment (traversing the parent directories until it finds one). These is subcommands can also be called with an optional argument specifying the enlistment. Here, too, we traverse parent directories as needed, until we find an enlistment. However, if the specified directory does not even exist, or is not a directory, we should stop right there, with an error message. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 6 ++++-- contrib/scalar/t/t9099-scalar.sh | 5 +++++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 1ce9c2b00e8..00dcd4b50ef 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -43,9 +43,11 @@ static void setup_enlistment_directory(int argc, const char **argv, usage_with_options(usagestr, options); /* find the worktree, determine its corresponding root */ - if (argc == 1) + if (argc == 1) { strbuf_add_absolute_path(&path, argv[0]); - else if (strbuf_getcwd(&path) < 0) + if (!is_directory(path.buf)) + die(_("'%s' does not exist"), path.buf); + } else if (strbuf_getcwd(&path) < 0) die(_("need a working directory")); strbuf_trim_trailing_dir_sep(&path); diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 2e1502ad45e..9d83fdf25e8 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -85,4 +85,9 @@ test_expect_success 'scalar delete with enlistment' ' test_path_is_missing cloned ' +test_expect_success '`scalar [...] ` errors out when dir is missing' ' + ! scalar run config cloned 2>err && + grep "cloned. does not exist" err +' + test_done From patchwork Sun Feb 6 22:39:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12736768 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC448C433F5 for ; Sun, 6 Feb 2022 22:39:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241365AbiBFWjR (ORCPT ); Sun, 6 Feb 2022 17:39:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229564AbiBFWjM (ORCPT ); Sun, 6 Feb 2022 17:39:12 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 692FFC0612A4 for ; Sun, 6 Feb 2022 14:39:11 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id u15so21724867wrt.3 for ; Sun, 06 Feb 2022 14:39:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=VWd48JVPenDDsYO1Af79wypuAyYJsrjtXWbFsDP1ERc=; b=Oz1KpgTycartwLez0EcQIl9PXQh/Y59GUHSdSciRy9/TuTJJiiuHiIiL3gG+VtArLV K20mKT5lJRMrfeaXczUkRfd1UXHSfGGqRa7P1M8RncJkwjWJuykPn0UWFjfSC+FN23NK 541uG1wwCNhjItdeYqyH4/4iMi76+F564OZ89W0zXcDqO5BFXi6M/6uYAY7WryOxgen+ Vs1WzWVTRmjpEWfl+ZY9MmI24adPJx0Y5yPGesyEOqzC0ahH584O23UaBVngCfzG/lra teeOSnFvlKs/5vlD+8QQ5oZGtBu5GsLxD5lOatF1MDjk1FXTtBIAm0AUjdeFylwjFxKK MbOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=VWd48JVPenDDsYO1Af79wypuAyYJsrjtXWbFsDP1ERc=; b=DfU9alUGI17MraxGZKrBXV72FFRG9xyuwb10oAHvQ7Ubaay3kx5C7Y/OoGVagXA0Z/ qxxjrqGFDFAcmMxrtlKmQ6L/mfFjWKI/UlKJqXh/x3nzMKpl9EiNGoQHB+I/xYQPFUjd wBy+oZ/yRd+6O/s5Cz9YqpAtE6MDKrVAga/Lj0pSWYdOhuF4aut4rPJ0rW8LNOTFfRec 3K1FJBl2RLT/t/5XLNfs1EsDpBXftyiKjozv+lHJUCQ1I3cjHUd/1LYMMzHOjEthwFVM VZPDI6H1lzGnCHSJHF+ST+s8DCzIGjR6dWsqFNzWBIRgrueAqU7NujX1OabIOdC74xaM CVkg== X-Gm-Message-State: AOAM531mVNsGwTsge3wgsXiNwadfZbVir3PKnXX5erwyHfFUw5dkqH3q jlgfaP4bu2NskkC0WaSXW5EiS8NVwfo= X-Google-Smtp-Source: ABdhPJzdDvOruixNzWikoSfftWqieEJ/Q9R+9H1YPbkhoVeXlcjdg9J7rp9vILSw3/oL16PrmrdjFA== X-Received: by 2002:a5d:498a:: with SMTP id r10mr2561588wrq.368.1644187149827; Sun, 06 Feb 2022 14:39:09 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r2sm8442122wmq.24.2022.02.06.14.39.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 06 Feb 2022 14:39:09 -0800 (PST) Message-Id: <0d570137bb6aef675f4f5d74d140ace1dfba5eb7.1644187146.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sun, 06 Feb 2022 22:39:03 +0000 Subject: [PATCH v2 3/6] Implement `scalar diagnose` Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin Over the course of Scalar's development, it became obvious that there is a need for a command that can gather all kinds of useful information that can help identify the most typical problems with large worktrees/repositories. The `diagnose` command is the culmination of this hard-won knowledge: it gathers the installed hooks, the config, a couple statistics describing the data shape, among other pieces of information, and then wraps everything up in a tidy, neat `.zip` archive. Note: originally, Scalar was implemented in C# using the .NET API, where we had the luxury of a comprehensive standard library that includes basic functionality such as writing a `.zip` file. In the C version, we lack such a commodity. Rather than introducing a dependency on, say, libzip, we slightly abuse Git's `archive` command: Instead of writing the `.zip` file directly, we stage the file contents in a Git index of a temporary, bare repository, only to let `git archive` have at it, and finally removing the temporary repository. Also note: Due to the frequently-spawned `git hash-object` processes, this command is quite a bit slow on Windows. Should it turn out to be a big problem, the lack of a batch mode of the `hash-object` command could potentially be worked around via using `git fast-import` with a crafted `stdin`. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 143 +++++++++++++++++++++++++++++++ contrib/scalar/scalar.txt | 12 +++ contrib/scalar/t/t9099-scalar.sh | 14 +++ 3 files changed, 169 insertions(+) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 00dcd4b50ef..30ce0799c7a 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -11,6 +11,7 @@ #include "dir.h" #include "packfile.h" #include "help.h" +#include "archive.h" /* * Remove the deepest subdirectory in the provided path string. Path must not @@ -261,6 +262,44 @@ static int unregister_dir(void) return res; } +static int add_directory_to_archiver(struct strvec *archiver_args, + const char *path, int recurse) +{ + int at_root = !*path; + DIR *dir = opendir(at_root ? "." : path); + struct dirent *e; + struct strbuf buf = STRBUF_INIT; + size_t len; + int res = 0; + + if (!dir) + return error(_("could not open directory '%s'"), path); + + if (!at_root) + strbuf_addf(&buf, "%s/", path); + len = buf.len; + strvec_pushf(archiver_args, "--prefix=%s", buf.buf); + + while (!res && (e = readdir(dir))) { + if (!strcmp(".", e->d_name) || !strcmp("..", e->d_name)) + continue; + + strbuf_setlen(&buf, len); + strbuf_addstr(&buf, e->d_name); + + if (e->d_type == DT_REG) + strvec_pushf(archiver_args, "--add-file=%s", buf.buf); + else if (e->d_type != DT_DIR) + res = -1; + else if (recurse) + add_directory_to_archiver(archiver_args, buf.buf, recurse); + } + + closedir(dir); + strbuf_release(&buf); + return res; +} + /* printf-style interface, expects `=` argument */ static int set_config(const char *fmt, ...) { @@ -501,6 +540,109 @@ cleanup: return res; } +static int cmd_diagnose(int argc, const char **argv) +{ + struct option options[] = { + OPT_END(), + }; + const char * const usage[] = { + N_("scalar diagnose []"), + NULL + }; + struct strbuf zip_path = STRBUF_INIT; + struct strvec archiver_args = STRVEC_INIT; + char **argv_copy = NULL; + int stdout_fd = -1, archiver_fd = -1; + time_t now = time(NULL); + struct tm tm; + struct strbuf path = STRBUF_INIT, buf = STRBUF_INIT; + size_t off; + int res = 0; + + argc = parse_options(argc, argv, NULL, options, + usage, 0); + + setup_enlistment_directory(argc, argv, usage, options, &zip_path); + + strbuf_addstr(&zip_path, "/.scalarDiagnostics/scalar_"); + strbuf_addftime(&zip_path, + "%Y%m%d_%H%M%S", localtime_r(&now, &tm), 0, 0); + strbuf_addstr(&zip_path, ".zip"); + switch (safe_create_leading_directories(zip_path.buf)) { + case SCLD_EXISTS: + case SCLD_OK: + break; + default: + error_errno(_("could not create directory for '%s'"), + zip_path.buf); + goto diagnose_cleanup; + } + stdout_fd = dup(1); + if (stdout_fd < 0) { + res = error_errno(_("could not duplicate stdout")); + goto diagnose_cleanup; + } + + archiver_fd = xopen(zip_path.buf, O_CREAT | O_WRONLY | O_TRUNC, 0666); + if (archiver_fd < 0 || dup2(archiver_fd, 1) < 0) { + res = error_errno(_("could not redirect output")); + goto diagnose_cleanup; + } + + init_zip_archiver(); + strvec_pushl(&archiver_args, "scalar-diagnose", "--format=zip", NULL); + + strbuf_reset(&buf); + strbuf_addstr(&buf, + "--add-file-with-content=diagnostics.log:" + "Collecting diagnostic info\n\n"); + get_version_info(&buf, 1); + + strbuf_addf(&buf, "Enlistment root: %s\n", the_repository->worktree); + off = strchr(buf.buf, ':') + 1 - buf.buf; + write_or_die(stdout_fd, buf.buf + off, buf.len - off); + strvec_push(&archiver_args, buf.buf); + + if ((res = add_directory_to_archiver(&archiver_args, ".git", 0)) || + (res = add_directory_to_archiver(&archiver_args, ".git/hooks", 0)) || + (res = add_directory_to_archiver(&archiver_args, ".git/info", 0)) || + (res = add_directory_to_archiver(&archiver_args, ".git/logs", 1)) || + (res = add_directory_to_archiver(&archiver_args, ".git/objects/info", 0))) + goto diagnose_cleanup; + + strvec_pushl(&archiver_args, "--prefix=", + oid_to_hex(the_hash_algo->empty_tree), "--", NULL); + + /* `write_archive()` modifies the `argv` passed to it. Let it. */ + argv_copy = xmemdupz(archiver_args.v, + sizeof(char *) * archiver_args.nr); + res = write_archive(archiver_args.nr, (const char **)argv_copy, NULL, + the_repository, NULL, 0); + if (res) { + error(_("failed to write archive")); + goto diagnose_cleanup; + } + + if (!res) + printf("\n" + "Diagnostics complete.\n" + "All of the gathered info is captured in '%s'\n", + zip_path.buf); + +diagnose_cleanup: + if (archiver_fd >= 0) { + close(1); + dup2(stdout_fd, 1); + } + free(argv_copy); + strvec_clear(&archiver_args); + strbuf_release(&zip_path); + strbuf_release(&path); + strbuf_release(&buf); + + return res; +} + static int cmd_list(int argc, const char **argv) { if (argc != 1) @@ -802,6 +944,7 @@ static struct { { "reconfigure", cmd_reconfigure }, { "delete", cmd_delete }, { "version", cmd_version }, + { "diagnose", cmd_diagnose }, { NULL, NULL}, }; diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt index f416d637289..22583fe046e 100644 --- a/contrib/scalar/scalar.txt +++ b/contrib/scalar/scalar.txt @@ -14,6 +14,7 @@ scalar register [] scalar unregister [] scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [] scalar reconfigure [ --all | ] +scalar diagnose [] scalar delete DESCRIPTION @@ -129,6 +130,17 @@ reconfigure the enlistment. With the `--all` option, all enlistments currently registered with Scalar will be reconfigured. Use this option after each Scalar upgrade. +Diagnose +~~~~~~~~ + +diagnose []:: + When reporting issues with Scalar, it is often helpful to provide the + information gathered by this command, including logs and certain + statistics describing the data shape of the current enlistment. ++ +The output of this command is a `.zip` file that is written into +a directory adjacent to the worktree in the `src` directory. + Delete ~~~~~~ diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 9d83fdf25e8..bbd07a44426 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -90,4 +90,18 @@ test_expect_success '`scalar [...] ` errors out when dir is missing' ' grep "cloned. does not exist" err ' +SQ="'" +test_expect_success UNZIP 'scalar diagnose' ' + scalar clone "file://$(pwd)" cloned --single-branch && + scalar diagnose cloned >out && + sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && + zip_path=$(cat zip_path) && + test -n "$zip_path" && + unzip -v "$zip_path" && + folder=${zip_path%.zip} && + test_path_is_missing "$folder" && + unzip -p "$zip_path" diagnostics.log >out && + test_file_not_empty out +' + test_done From patchwork Sun Feb 6 22:39:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12736769 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3445EC433EF for ; Sun, 6 Feb 2022 22:39:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241643AbiBFWjS (ORCPT ); Sun, 6 Feb 2022 17:39:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41766 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240600AbiBFWjM (ORCPT ); Sun, 6 Feb 2022 17:39:12 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E98A0C061348 for ; Sun, 6 Feb 2022 14:39:11 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id v19so6771679wrv.6 for ; Sun, 06 Feb 2022 14:39:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=9tcq3SbDQrWR6k402xQD96ozw8uzjJFlE7nSNFTp9lY=; b=RVvhjqfJa4dzQqdfEEOw7O60NL68Qrpz9vN9GiJNbAERMkRpaoO8w0jnhtz+qPACsF H1LASj4fN6lkkk+WrAVtxq1qP5CT9dDuyERHodacT+r2tB8tRVSdsJv/oinQqzf4+CIp bpvx6s6WCDCbl0on6+v/uzwcwvkRABgfWG/roLJKx0T55jZbA3AASdb1gABa4DJRpWf2 qBmS88M3N5LttpkdWGTEEI0EZgeY5ES64MRTtjIdSFz8bQ1N6YGzhNf6ivwomukTUNn2 i+xutbtTuHT02YYbAghREvdPs3XkHlsay9XOBL2Xm8nCQKRfQ/ET4bos/qZZVV/fRAIQ p00Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=9tcq3SbDQrWR6k402xQD96ozw8uzjJFlE7nSNFTp9lY=; b=owJO5qNdx4a64JgidsHOEL0FTWcCvYcKl8TDhfxGzm/OMlxP4lsXHylqiX/+L7NZ62 PRQWCmAjfFnN0YBqMP6sBUb7LiqVpGyYDWrJi7zWxpucUO3lZ3fJioHZG7fgNp4sctUK yTYfyghrSI6xPgZRo+UFW2eRZuw9pYyIyk6aLZveku7mQNSFAvi77P0YBR7G5FwLr1wK ypAWv00vJKplBa3Z7TU12r5yPHrDRoYk+djryqOqJjWIzlEiHU7YkmdVntXrq9JMTEqs TRMQBwuf1S/oFifvoiCfKzi9bCkKDh082KkslhHURjuSOmOKx1DsvipV2fimubfg/Mmg rWmA== X-Gm-Message-State: AOAM5311z9C/uNlknE2WsjlJ3sYjVvUGI9bA6xcPmJTd9ZHVWQ1euVY4 8ldrXXJV0cU3AZGMNV4NnA0EW1Fck2g= X-Google-Smtp-Source: ABdhPJx3WL7R5u+EzgayAYXJhTCM970UrcWcejEaOJ7jPDwI2NjSqAL0MQBCS2JuSlJL/MbT3MMzXA== X-Received: by 2002:adf:e34c:: with SMTP id n12mr7721650wrj.263.1644187150437; Sun, 06 Feb 2022 14:39:10 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y14sm8912963wrd.91.2022.02.06.14.39.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 06 Feb 2022 14:39:10 -0800 (PST) Message-Id: <938e38b5a09c7cf1cef4faca577e969b455e1ea5.1644187146.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sun, 06 Feb 2022 22:39:04 +0000 Subject: [PATCH v2 4/6] scalar diagnose: include disk space information Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin When analyzing problems with large worktrees/repositories, it is useful to know how close to a "full disk" situation Scalar/Git operates. Let's include this information. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 53 ++++++++++++++++++++++++++++++++ contrib/scalar/t/t9099-scalar.sh | 1 + 2 files changed, 54 insertions(+) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 30ce0799c7a..fd666376109 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -300,6 +300,58 @@ static int add_directory_to_archiver(struct strvec *archiver_args, return res; } +#ifndef WIN32 +#include +#endif + +static int get_disk_info(struct strbuf *out) +{ +#ifdef WIN32 + struct strbuf buf = STRBUF_INIT; + char volume_name[MAX_PATH], fs_name[MAX_PATH]; + DWORD serial_number, component_length, flags; + ULARGE_INTEGER avail2caller, total, avail; + + strbuf_realpath(&buf, ".", 1); + if (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) { + error(_("could not determine free disk size for '%s'"), + buf.buf); + strbuf_release(&buf); + return -1; + } + + strbuf_setlen(&buf, offset_1st_component(buf.buf)); + if (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name), + &serial_number, &component_length, &flags, + fs_name, sizeof(fs_name))) { + error(_("could not get info for '%s'"), buf.buf); + strbuf_release(&buf); + return -1; + } + strbuf_addf(out, "Available space on '%s': ", buf.buf); + strbuf_humanise_bytes(out, avail2caller.QuadPart); + strbuf_addch(out, '\n'); + strbuf_release(&buf); +#else + struct strbuf buf = STRBUF_INIT; + struct statvfs stat; + + strbuf_realpath(&buf, ".", 1); + if (statvfs(buf.buf, &stat) < 0) { + error_errno(_("could not determine free disk size for '%s'"), + buf.buf); + strbuf_release(&buf); + return -1; + } + + strbuf_addf(out, "Available space on '%s': ", buf.buf); + strbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail)); + strbuf_addf(out, " (mount flags 0x%lx)\n", stat.f_flag); + strbuf_release(&buf); +#endif + return 0; +} + /* printf-style interface, expects `=` argument */ static int set_config(const char *fmt, ...) { @@ -599,6 +651,7 @@ static int cmd_diagnose(int argc, const char **argv) get_version_info(&buf, 1); strbuf_addf(&buf, "Enlistment root: %s\n", the_repository->worktree); + get_disk_info(&buf); off = strchr(buf.buf, ':') + 1 - buf.buf; write_or_die(stdout_fd, buf.buf + off, buf.len - off); strvec_push(&archiver_args, buf.buf); diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index bbd07a44426..f3d037823c8 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -94,6 +94,7 @@ SQ="'" test_expect_success UNZIP 'scalar diagnose' ' scalar clone "file://$(pwd)" cloned --single-branch && scalar diagnose cloned >out && + grep "Available space" out && sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && zip_path=$(cat zip_path) && test -n "$zip_path" && From patchwork Sun Feb 6 22:39:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 12736770 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1B21C433F5 for ; Sun, 6 Feb 2022 22:39:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347388AbiBFWj3 (ORCPT ); Sun, 6 Feb 2022 17:39:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240925AbiBFWjN (ORCPT ); Sun, 6 Feb 2022 17:39:13 -0500 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B3DD2C06173B for ; Sun, 6 Feb 2022 14:39:12 -0800 (PST) Received: by mail-wm1-x32b.google.com with SMTP id c192so8643163wma.4 for ; Sun, 06 Feb 2022 14:39:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Q/qPpODfOQGps3fZ0eUt0L/gO8hAtKPjrLL6IgJUDZ0=; b=N5besYk0QSuieDq6qAAoZbvbmc0lRvg4LjEXiKXlkH2+lT8R2WJ0p7QlidnJiFLZju hOBNZ/5DiadoVlwARdZwG/XZVKvxgdXhjAaMLtDS3ZcRbJub1daHaia1UADbecjBmwPQ o11Q8cJ5rAQQ65iHnfZYBxEwOQV6qBmHRvpueU11ePrREpKi9cETzRFkc5wE+U/HSXEZ SdsmuQGZ91RLT09LzqWlxMESvlZcZVjxXZhRn5E3JzXD7LnQXBMRup/pu6I8JySoCcA6 sMpkWLicAVt4M71Rua1PHT1aC4ywfyoDTVY0YGQ3kHwAZrsKXXCZm6mO+UhQNxqdrwOb mDWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Q/qPpODfOQGps3fZ0eUt0L/gO8hAtKPjrLL6IgJUDZ0=; b=M21e2+m1kth/3KZt+U2SYGTNUdCQ49ijrJfUlBh9U15kby3EF2417YIhIPsDeamSJx ErqBZKZDoGMOlnj6jAmTwInP9XXQYXPhs1GJyGBkVdG2AFOS183qG+OspJjsz2IaZgBN w6KCVD4j4dIaRKdbQ9882qFwNti+kA5N7eJUPfugWc3U3Z4eiMdw5fVoFcbFaf5CLh54 gAek/686eP16IM1Wf8VoSQzXFfgPDKQ3nzGJIvxC/sRygJfXCDRgl3OX85TWe/MCANc+ mb7yqlx66PeLvQQ87Y1MbLXm0r6GqD97WGSFH0tE9ia/5g0tRJ1Ox6a/Tze38975zHls iNwA== X-Gm-Message-State: AOAM532B3xcxvRsPnPSUFP3huzZBKBlcH9SIt4cRqe3AC6iEGws6UO1i CvHdjDAr2WsWzzsgvEpjgI3ft3Vr/9g= X-Google-Smtp-Source: ABdhPJwbItm0xxbY2YUm7AqfGoK3pPsB9YuqQ8Z7I6Mm0UyRwBJWUMWoj/Hz0EKN9sb2XbTGk5Bbpg== X-Received: by 2002:a05:600c:2b91:: with SMTP id j17mr4472874wmc.19.1644187151184; Sun, 06 Feb 2022 14:39:11 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l28sm5552434wrz.90.2022.02.06.14.39.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 06 Feb 2022 14:39:10 -0800 (PST) Message-Id: In-Reply-To: References: Date: Sun, 06 Feb 2022 22:39:05 +0000 Subject: [PATCH v2 5/6] scalar: teach `diagnose` to gather packfile info Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , Johannes Schindelin , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham It's helpful to see if there are other crud files in the pack directory. Let's teach the `scalar diagnose` command to gather file size information about pack files. While at it, also enumerate the pack files in the alternate object directories, if any are registered. Signed-off-by: Matthew John Cheetham Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 30 ++++++++++++++++++++++++++++++ contrib/scalar/t/t9099-scalar.sh | 6 +++++- 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index fd666376109..331d48b2a80 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -12,6 +12,7 @@ #include "packfile.h" #include "help.h" #include "archive.h" +#include "object-store.h" /* * Remove the deepest subdirectory in the provided path string. Path must not @@ -592,6 +593,29 @@ cleanup: return res; } +static void dir_file_stats_objects(const char *full_path, size_t full_path_len, + const char *file_name, void *data) +{ + struct strbuf *buf = data; + struct stat st; + + if (!stat(full_path, &st)) + strbuf_addf(buf, "%-70s %16" PRIuMAX "\n", file_name, + (uintmax_t)st.st_size); +} + +static int dir_file_stats(struct object_directory *object_dir, void *data) +{ + struct strbuf *buf = data; + + strbuf_addf(buf, "Contents of %s:\n", object_dir->path); + + for_each_file_in_pack_dir(object_dir->path, dir_file_stats_objects, + data); + + return 0; +} + static int cmd_diagnose(int argc, const char **argv) { struct option options[] = { @@ -656,6 +680,12 @@ static int cmd_diagnose(int argc, const char **argv) write_or_die(stdout_fd, buf.buf + off, buf.len - off); strvec_push(&archiver_args, buf.buf); + strbuf_reset(&buf); + strbuf_addstr(&buf, "--add-file-with-content=packs-local.txt:"); + dir_file_stats(the_repository->objects->odb, &buf); + foreach_alt_odb(dir_file_stats, &buf); + strvec_push(&archiver_args, buf.buf); + if ((res = add_directory_to_archiver(&archiver_args, ".git", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/hooks", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/info", 0)) || diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index f3d037823c8..e049221609d 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -93,6 +93,8 @@ test_expect_success '`scalar [...] ` errors out when dir is missing' ' SQ="'" test_expect_success UNZIP 'scalar diagnose' ' scalar clone "file://$(pwd)" cloned --single-branch && + git repack && + echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates && scalar diagnose cloned >out && grep "Available space" out && sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && @@ -102,7 +104,9 @@ test_expect_success UNZIP 'scalar diagnose' ' folder=${zip_path%.zip} && test_path_is_missing "$folder" && unzip -p "$zip_path" diagnostics.log >out && - test_file_not_empty out + test_file_not_empty out && + unzip -p "$zip_path" packs-local.txt >out && + grep "$(pwd)/.git/objects" out ' test_done From patchwork Sun Feb 6 22:39:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 12736771 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A965C433EF for ; Sun, 6 Feb 2022 22:39:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347355AbiBFWja (ORCPT ); Sun, 6 Feb 2022 17:39:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240994AbiBFWjO (ORCPT ); Sun, 6 Feb 2022 17:39:14 -0500 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 66EE7C061353 for ; Sun, 6 Feb 2022 14:39:13 -0800 (PST) Received: by mail-wr1-x42a.google.com with SMTP id h6so2956294wrb.9 for ; Sun, 06 Feb 2022 14:39:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=vqFeBZv0w8pn4i/0FkUVFoI6BFfYtcspoLuca1bT42w=; b=ocbhsOP0ZtwhxLcofvSxTWeIhdifR3/hwPA+5Q9IGZLDwbwYBMxtVdNXAypmPdN+lQ +CEMRC0yA3WA/bAzAvRixEqzvklQorTKcPNthTZYEDKd7RLzkJabB2M+JCUAniDK7jPg NcazJ8AG17qHcKirtk+c2DYcOfDkSZJoI1EAcRNBex3dKgHqfz+O1yNdtxRTOcgqZ3oK TK4wt6dS8/4Pe7UWiYdmXempZa6v8iJSUMJz1YclojQ+KveoIfMnrrcfY4ZsAJHr9JDn yKSUJ234ajBKBv93wgr8522hoo4JKffjwHGUSV+WKp5XWBxJGzHrASntl48lpLswbArX 4lTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=vqFeBZv0w8pn4i/0FkUVFoI6BFfYtcspoLuca1bT42w=; b=1zfsW5IitTS3Gh49lCGLKtekZ8KwOToY4kGEQ4PifKs9h8DRUopaegC6DfKJupSYfQ rMFHr84ebYdqwcoLmRht3BCzpcNdy86y7LGNTEaqmv+OYKfeW3G5Hcq/I98a4awhOxih 72JJAu+F8AbJFHn18X07BX3AJIUtZsMADGr4sKyFttocSrCrThEZXTC5WBj8HQgaCUsT fsTv6kc63gRiBxmhDkSncyEebRlXn51/3/cSbiTBq3Gz8oQ9aD8IE5c5c9tkmLWwOoZi Rl2RUFdW+E5Lzd2vMQpBU2rrPsttACbAwi1vDDpodktuuyyVTbiL+8SMMpuRL25cmzGt UsnA== X-Gm-Message-State: AOAM533VRjN8ts9Tme+h60YPT8tGcmeKmuiIR8nJE0GYwf2S3vhS+ViG aRHebalwTFW3a6/zSHWU/zabA0m0cSk= X-Google-Smtp-Source: ABdhPJwyQLwoppERSryk0IL4782YrkLQxu4Lllmwwce6mLwr879uQnEfSVXX0GkSxAftJ4wv52tpDA== X-Received: by 2002:a05:6000:1884:: with SMTP id a4mr7378975wri.509.1644187151889; Sun, 06 Feb 2022 14:39:11 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z6sm15363325wmf.37.2022.02.06.14.39.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 06 Feb 2022 14:39:11 -0800 (PST) Message-Id: <7a8875be425b272becd6c08f4cd5b23c41304ae3.1644187146.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sun, 06 Feb 2022 22:39:06 +0000 Subject: [PATCH v2 6/6] scalar: teach `diagnose` to gather loose objects information Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , Johannes Schindelin , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham When operating at the scale that Scalar wants to support, certain data shapes are more likely to cause undesirable performance issues, such as large numbers of loose objects. By including statistics about this, `scalar diagnose` now makes it easier to identify such scenarios. Signed-off-by: Matthew John Cheetham Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 59 ++++++++++++++++++++++++++++++++ contrib/scalar/t/t9099-scalar.sh | 5 ++- 2 files changed, 63 insertions(+), 1 deletion(-) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 331d48b2a80..537b97ae734 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -616,6 +616,60 @@ static int dir_file_stats(struct object_directory *object_dir, void *data) return 0; } +static int count_files(char *path) +{ + DIR *dir = opendir(path); + struct dirent *e; + int count = 0; + + if (!dir) + return 0; + + while ((e = readdir(dir)) != NULL) + if (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG) + count++; + + closedir(dir); + return count; +} + +static void loose_objs_stats(struct strbuf *buf, const char *path) +{ + DIR *dir = opendir(path); + struct dirent *e; + int count; + int total = 0; + unsigned char c; + struct strbuf count_path = STRBUF_INIT; + size_t base_path_len; + + if (!dir) + return; + + strbuf_addstr(buf, "Object directory stats for "); + strbuf_add_absolute_path(buf, path); + strbuf_addstr(buf, ":\n"); + + strbuf_add_absolute_path(&count_path, path); + strbuf_addch(&count_path, '/'); + base_path_len = count_path.len; + + while ((e = readdir(dir)) != NULL) + if (!is_dot_or_dotdot(e->d_name) && + e->d_type == DT_DIR && strlen(e->d_name) == 2 && + !hex_to_bytes(&c, e->d_name, 1)) { + strbuf_setlen(&count_path, base_path_len); + strbuf_addstr(&count_path, e->d_name); + total += (count = count_files(count_path.buf)); + strbuf_addf(buf, "%s : %7d files\n", e->d_name, count); + } + + strbuf_addf(buf, "Total: %d loose objects", total); + + strbuf_release(&count_path); + closedir(dir); +} + static int cmd_diagnose(int argc, const char **argv) { struct option options[] = { @@ -686,6 +740,11 @@ static int cmd_diagnose(int argc, const char **argv) foreach_alt_odb(dir_file_stats, &buf); strvec_push(&archiver_args, buf.buf); + strbuf_reset(&buf); + strbuf_addstr(&buf, "--add-file-with-content=objects-local.txt:"); + loose_objs_stats(&buf, ".git/objects"); + strvec_push(&archiver_args, buf.buf); + if ((res = add_directory_to_archiver(&archiver_args, ".git", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/hooks", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/info", 0)) || diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index e049221609d..9b4eedbb0aa 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -95,6 +95,7 @@ test_expect_success UNZIP 'scalar diagnose' ' scalar clone "file://$(pwd)" cloned --single-branch && git repack && echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates && + test_commit -C cloned/src loose && scalar diagnose cloned >out && grep "Available space" out && sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && @@ -106,7 +107,9 @@ test_expect_success UNZIP 'scalar diagnose' ' unzip -p "$zip_path" diagnostics.log >out && test_file_not_empty out && unzip -p "$zip_path" packs-local.txt >out && - grep "$(pwd)/.git/objects" out + grep "$(pwd)/.git/objects" out && + unzip -p "$zip_path" objects-local.txt >out && + grep "^Total: [1-9]" out ' test_done