From patchwork Wed Jan 23 21:59:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777941 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7CF5F13B4 for ; Wed, 23 Jan 2019 21:59:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6CE1E2B40B for ; Wed, 23 Jan 2019 21:59:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 603ED2DA43; Wed, 23 Jan 2019 21:59:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 57FF92B40B for ; Wed, 23 Jan 2019 21:59:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726251AbfAWV7T (ORCPT ); Wed, 23 Jan 2019 16:59:19 -0500 Received: from mail-ed1-f65.google.com ([209.85.208.65]:36248 "EHLO mail-ed1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726125AbfAWV7S (ORCPT ); Wed, 23 Jan 2019 16:59:18 -0500 Received: by mail-ed1-f65.google.com with SMTP id f23so2966200edb.3 for ; Wed, 23 Jan 2019 13:59:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=49q4r2bpeOJWd4xwUd3JrFtbK9sQA8Ri7lDj3msQCEI=; b=FmFvhT1zaf7E0T+L4czZxB5SK+WxnhRA2mAVemWGoabu3/wlNBXIiQFEHomRNr3c2W YME8B4ke5jktY1Njl5kBMHY0Hn5uwP06fnlnH2gKDIpgEtGGSd2r1HvZS7djCVE27xi6 dnqvAMpHNH0P9otPaGXv43lxW2ETVmWUs1xHVrq17I93RC02yO2qtDNCOGfqvQRFDrzG 0z3XruyAIS0+gieiSh2abye7g97/Qdpw1VwLVTDHlIY7cKo1C/ScLfKu79tCSTnX9c0I A2XfQt5suB7fid9bFO+S8Qt+Xmvhn+KCWu47FyKqVD2c//MVvM9zGz4OI4nk4Fw9isVP Wf5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=49q4r2bpeOJWd4xwUd3JrFtbK9sQA8Ri7lDj3msQCEI=; b=RmbkPtqBGm9ogJW2w0iN/d5FqATLaiX8jYsDvCjGwMNU4Vb3KmiVHorXLVLypZbW5c ZIgTpsLbrBPVLSmNKt63lxuY16Uazb4khOBga3PNQJ5J7sOrQkA+bfZxh+24kIvV2YiT K+iQAFjb4F5Hi4+bg2tUiifbWFcjb+lA06Due8YKfHTN44EABG7sHmrNX87uMf5cOXag 1wKf+B3qTHs2lWPIKHrs3L7lZtcXfMDPWWtQOKpqXXw3IcSzKI/SAY5rtBcWDZHlGtQR Ld8Dok0ATUUKpk4OzrrgMgvv5VSeFQZ7Tyw9y/Zvms/uopf/zURoBo7gMURZyzli0N73 LkgQ== X-Gm-Message-State: AJcUukdRFXOb+Hsl8Jb3HtHR4JUHvZDEHaCAX/orcDFdulBh/vBKQNb1 ijwdqQHLxUCwGrN7GY7CY4JkHmSR X-Google-Smtp-Source: ALg8bN6ZSt0GdQfHTwOFbWHdYl3ujC8Hp8Sba5BxtgHfNuQGnkvHAVM2scHnsyu6085H6T8sy1U7RQ== X-Received: by 2002:a50:ef18:: with SMTP id m24mr4211191eds.136.1548280755980; Wed, 23 Jan 2019 13:59:15 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v14sm10050122edq.74.2019.01.23.13.59.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 13:59:15 -0800 (PST) Date: Wed, 23 Jan 2019 13:59:15 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 21:59:08 GMT Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 1/6] commit-graph: return with errors during write Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, avarab@gmail.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee The write_commit_graph() method uses die() to report failure and exit when confronted with an unexpected condition. This use of die() in a library function is incorrect and is now replaced by error() statements and an int return type. Now that we use 'goto cleanup' to jump to the terminal condition on an error, we have new paths that could lead to uninitialized values. New initializers are added to correct for this. The builtins 'commit-graph', 'gc', and 'commit' call these methods, so update them to check the return value. Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 19 +++++++------ builtin/commit.c | 5 ++-- builtin/gc.c | 7 ++--- commit-graph.c | 60 +++++++++++++++++++++++++++++------------- commit-graph.h | 10 +++---- 5 files changed, 62 insertions(+), 39 deletions(-) diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c index 4ae502754c..b12d46fdc8 100644 --- a/builtin/commit-graph.c +++ b/builtin/commit-graph.c @@ -126,6 +126,7 @@ static int graph_write(int argc, const char **argv) struct string_list *pack_indexes = NULL; struct string_list *commit_hex = NULL; struct string_list lines; + int result; static struct option builtin_commit_graph_write_options[] = { OPT_STRING(0, "object-dir", &opts.obj_dir, @@ -153,10 +154,8 @@ static int graph_write(int argc, const char **argv) read_replace_refs = 0; - if (opts.reachable) { - write_commit_graph_reachable(opts.obj_dir, opts.append, 1); - return 0; - } + if (opts.reachable) + return write_commit_graph_reachable(opts.obj_dir, opts.append, 1); string_list_init(&lines, 0); if (opts.stdin_packs || opts.stdin_commits) { @@ -173,14 +172,14 @@ static int graph_write(int argc, const char **argv) UNLEAK(buf); } - write_commit_graph(opts.obj_dir, - pack_indexes, - commit_hex, - opts.append, - 1); + result = write_commit_graph(opts.obj_dir, + pack_indexes, + commit_hex, + opts.append, + 1); UNLEAK(lines); - return 0; + return result; } int cmd_commit_graph(int argc, const char **argv, const char *prefix) diff --git a/builtin/commit.c b/builtin/commit.c index 004b816635..04b0717b35 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1667,8 +1667,9 @@ int cmd_commit(int argc, const char **argv, const char *prefix) "new_index file. Check that disk is not full and quota is\n" "not exceeded, and then \"git reset HEAD\" to recover.")); - if (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0)) - write_commit_graph_reachable(get_object_directory(), 0, 0); + if (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) && + write_commit_graph_reachable(get_object_directory(), 0, 0)) + return 1; repo_rerere(the_repository, 0); run_command_v_opt(argv_gc_auto, RUN_GIT_CMD); diff --git a/builtin/gc.c b/builtin/gc.c index 7696017cd4..9c6c9c9007 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -662,9 +662,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix) if (pack_garbage.nr > 0) clean_pack_garbage(); - if (gc_write_commit_graph) - write_commit_graph_reachable(get_object_directory(), 0, - !quiet && !daemonized); + if (gc_write_commit_graph && + write_commit_graph_reachable(get_object_directory(), 0, + !quiet && !daemonized)) + return 1; if (auto_gc && too_many_loose_objects()) warning(_("There are too many unreachable loose objects; " diff --git a/commit-graph.c b/commit-graph.c index 0f8274d15d..162b9f2a85 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -755,27 +755,30 @@ static int add_ref_to_list(const char *refname, return 0; } -void write_commit_graph_reachable(const char *obj_dir, int append, - int report_progress) +int write_commit_graph_reachable(const char *obj_dir, int append, + int report_progress) { struct string_list list = STRING_LIST_INIT_DUP; + int result; for_each_ref(add_ref_to_list, &list); - write_commit_graph(obj_dir, NULL, &list, append, report_progress); + result = write_commit_graph(obj_dir, NULL, &list, + append, report_progress); string_list_clear(&list, 0); + return result; } -void write_commit_graph(const char *obj_dir, - struct string_list *pack_indexes, - struct string_list *commit_hex, - int append, int report_progress) +int write_commit_graph(const char *obj_dir, + struct string_list *pack_indexes, + struct string_list *commit_hex, + int append, int report_progress) { struct packed_oid_list oids; struct packed_commit_list commits; struct hashfile *f; uint32_t i, count_distinct = 0; - char *graph_name; + char *graph_name = NULL; struct lock_file lk = LOCK_INIT; uint32_t chunk_ids[5]; uint64_t chunk_offsets[5]; @@ -787,15 +790,17 @@ void write_commit_graph(const char *obj_dir, uint64_t progress_cnt = 0; struct strbuf progress_title = STRBUF_INIT; unsigned long approx_nr_objects; + int res = 0; if (!commit_graph_compatible(the_repository)) - return; + return 0; oids.nr = 0; approx_nr_objects = approximate_object_count(); oids.alloc = approx_nr_objects / 32; oids.progress = NULL; oids.progress_done = 0; + commits.list = NULL; if (append) { prepare_commit_graph_one(the_repository, obj_dir); @@ -836,10 +841,16 @@ void write_commit_graph(const char *obj_dir, strbuf_setlen(&packname, dirlen); strbuf_addstr(&packname, pack_indexes->items[i].string); p = add_packed_git(packname.buf, packname.len, 1); - if (!p) - die(_("error adding pack %s"), packname.buf); - if (open_pack_index(p)) - die(_("error opening index for %s"), packname.buf); + if (!p) { + error(_("error adding pack %s"), packname.buf); + res = 1; + goto cleanup; + } + if (open_pack_index(p)) { + error(_("error opening index for %s"), packname.buf); + res = 1; + goto cleanup; + } for_each_object_in_pack(p, add_packed_commits, &oids, FOR_EACH_OBJECT_PACK_ORDER); close_pack(p); @@ -910,8 +921,11 @@ void write_commit_graph(const char *obj_dir, } stop_progress(&progress); - if (count_distinct >= GRAPH_PARENT_MISSING) - die(_("the commit graph format cannot write %d commits"), count_distinct); + if (count_distinct >= GRAPH_PARENT_MISSING) { + error(_("the commit graph format cannot write %d commits"), count_distinct); + res = 1; + goto cleanup; + } commits.nr = 0; commits.alloc = count_distinct; @@ -943,16 +957,21 @@ void write_commit_graph(const char *obj_dir, num_chunks = num_extra_edges ? 4 : 3; stop_progress(&progress); - if (commits.nr >= GRAPH_PARENT_MISSING) - die(_("too many commits to write graph")); + if (commits.nr >= GRAPH_PARENT_MISSING) { + error(_("too many commits to write graph")); + res = 1; + goto cleanup; + } compute_generation_numbers(&commits, report_progress); graph_name = get_commit_graph_filename(obj_dir); if (safe_create_leading_directories(graph_name)) { UNLEAK(graph_name); - die_errno(_("unable to create leading directories of %s"), - graph_name); + error(_("unable to create leading directories of %s"), + graph_name); + res = errno; + goto cleanup; } hold_lock_file_for_update(&lk, graph_name, LOCK_DIE_ON_ERROR); @@ -1011,9 +1030,12 @@ void write_commit_graph(const char *obj_dir, finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC); commit_lock_file(&lk); +cleanup: free(graph_name); free(commits.list); free(oids.list); + + return res; } #define VERIFY_COMMIT_GRAPH_ERROR_HASH 2 diff --git a/commit-graph.h b/commit-graph.h index e6aff2c2e1..cd333a0cd0 100644 --- a/commit-graph.h +++ b/commit-graph.h @@ -60,12 +60,12 @@ struct commit_graph *load_commit_graph_one(const char *graph_file); */ int generation_numbers_enabled(struct repository *r); -void write_commit_graph_reachable(const char *obj_dir, int append, +int write_commit_graph_reachable(const char *obj_dir, int append, int report_progress); -void write_commit_graph(const char *obj_dir, - struct string_list *pack_indexes, - struct string_list *commit_hex, - int append, int report_progress); +int write_commit_graph(const char *obj_dir, + struct string_list *pack_indexes, + struct string_list *commit_hex, + int append, int report_progress); int verify_commit_graph(struct repository *r, struct commit_graph *g); From patchwork Wed Jan 23 21:59:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777943 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CE85C1390 for ; Wed, 23 Jan 2019 21:59:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BEEF42B40B for ; Wed, 23 Jan 2019 21:59:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B37EB2DA43; Wed, 23 Jan 2019 21:59:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 230392B40B for ; Wed, 23 Jan 2019 21:59:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726367AbfAWV7U (ORCPT ); Wed, 23 Jan 2019 16:59:20 -0500 Received: from mail-ed1-f66.google.com ([209.85.208.66]:40367 "EHLO mail-ed1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726120AbfAWV7T (ORCPT ); Wed, 23 Jan 2019 16:59:19 -0500 Received: by mail-ed1-f66.google.com with SMTP id g22so2952344edr.7 for ; Wed, 23 Jan 2019 13:59:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=J+I/sPsiAe8pwAU/XMKnP57lQQiWdAm7KX3gL17IkpQ=; b=j3K+30QOy1ipe6xd6JRjZ7FFcgmyEn4LlLmXyLFMv8XI5/z1qE8LYRTXZvXwh9GyZx /q5a+yBolHgkc28g0ce+PntPFux+b2AOJGOSeA3nBfqptABs6YKCjiPgSNxfx2S3Bxup mchvD55m+Nkta/q+3HTJgNNOoxaP7uyfmDaCb718gDMu8s3fM4yDB5hJDBU0giC8ezhs deQ0y81ks0EJp35mH90+1ITQE/XiTJh6Dj+9PaRkRIrtbNjCgay4W6X8n2vRAMR39RxL 7s2VehNg7BpAPoK+l72nm+c09EEfFtOXdNaQP4EH51HXBp521335g6tMTJBeLvD1iRE8 v5NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=J+I/sPsiAe8pwAU/XMKnP57lQQiWdAm7KX3gL17IkpQ=; b=Cv2bE0Su8P/ZUot2GPHHRUq91cRf3jkEkMa9BbUTxC681woTtCybiAVvUJnEDc2DdH NSy/ofV3P1J+UPM+xp31cWZUcrUFoBZElghfbxTnajaxY/+TYP1EJB1KmyHPcCacYEBR Y8IqxonR2ofjaQ11YIdd/pYJg+Tx1O/l73Xbr8E2XAgutMhEDWlw1K5nTsKt/lmwWDI7 OyZFXTSoQo+Os2jMtwGUP2sj9yU4Z6W/l72Xg15QAZKojwlFCv1pWR0iqXL2ta50M3J0 6zuksnryq/MR3JnbZBCwg1BCI6+sPKc+sMiO7fA3r5AciI7Gik0Mv4pAZ8QOoeGJPSBk 7SZQ== X-Gm-Message-State: AJcUukdYSAUz26p3bCDAMJZI0ga3TwcKw5R2q06Lo2UmT9dNYoIIbrss YAgXf+CQjPpNkkeF8i6c5O9FjP54 X-Google-Smtp-Source: ALg8bN7TTWUt8d6Z8IN2v5ovwm1V6oRY/LGiPszQCHNrWX6F0QikhR7magUJlB3CVBhxmOaTtdYuUw== X-Received: by 2002:a17:906:a4d5:: with SMTP id cc21mr3794390ejb.12.1548280756860; Wed, 23 Jan 2019 13:59:16 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d56sm10757756ede.76.2019.01.23.13.59.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 13:59:16 -0800 (PST) Date: Wed, 23 Jan 2019 13:59:16 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 21:59:09 GMT Message-Id: <43a40d0c43a1402a047b8c329ae8448643bedcce.1548280753.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 2/6] commit-graph: collapse parameters into flags Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, avarab@gmail.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee The write_commit_graph() and write_commit_graph_reachable() methods currently take two boolean parameters: 'append' and 'report_progress'. We will soon expand the possible options to send to these methods, so instead of complicating the parameter list, first simplify it. Collapse these parameters into a 'flags' parameter, and adjust the callers to provide flags as necessary. Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 8 +++++--- builtin/commit.c | 2 +- builtin/gc.c | 4 ++-- commit-graph.c | 9 +++++---- commit-graph.h | 8 +++++--- 5 files changed, 18 insertions(+), 13 deletions(-) diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c index b12d46fdc8..0c92421f75 100644 --- a/builtin/commit-graph.c +++ b/builtin/commit-graph.c @@ -127,6 +127,7 @@ static int graph_write(int argc, const char **argv) struct string_list *commit_hex = NULL; struct string_list lines; int result; + int flags = COMMIT_GRAPH_PROGRESS; static struct option builtin_commit_graph_write_options[] = { OPT_STRING(0, "object-dir", &opts.obj_dir, @@ -151,11 +152,13 @@ static int graph_write(int argc, const char **argv) die(_("use at most one of --reachable, --stdin-commits, or --stdin-packs")); if (!opts.obj_dir) opts.obj_dir = get_object_directory(); + if (opts.append) + flags |= COMMIT_GRAPH_APPEND; read_replace_refs = 0; if (opts.reachable) - return write_commit_graph_reachable(opts.obj_dir, opts.append, 1); + return write_commit_graph_reachable(opts.obj_dir, flags); string_list_init(&lines, 0); if (opts.stdin_packs || opts.stdin_commits) { @@ -175,8 +178,7 @@ static int graph_write(int argc, const char **argv) result = write_commit_graph(opts.obj_dir, pack_indexes, commit_hex, - opts.append, - 1); + flags); UNLEAK(lines); return result; diff --git a/builtin/commit.c b/builtin/commit.c index 04b0717b35..3228de4e3c 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1668,7 +1668,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix) "not exceeded, and then \"git reset HEAD\" to recover.")); if (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) && - write_commit_graph_reachable(get_object_directory(), 0, 0)) + write_commit_graph_reachable(get_object_directory(), 0)) return 1; repo_rerere(the_repository, 0); diff --git a/builtin/gc.c b/builtin/gc.c index 9c6c9c9007..198872206b 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -663,8 +663,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix) clean_pack_garbage(); if (gc_write_commit_graph && - write_commit_graph_reachable(get_object_directory(), 0, - !quiet && !daemonized)) + write_commit_graph_reachable(get_object_directory(), + !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0)) return 1; if (auto_gc && too_many_loose_objects()) diff --git a/commit-graph.c b/commit-graph.c index 162b9f2a85..28fe2378be 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -755,15 +755,14 @@ static int add_ref_to_list(const char *refname, return 0; } -int write_commit_graph_reachable(const char *obj_dir, int append, - int report_progress) +int write_commit_graph_reachable(const char *obj_dir, int flags) { struct string_list list = STRING_LIST_INIT_DUP; int result; for_each_ref(add_ref_to_list, &list); result = write_commit_graph(obj_dir, NULL, &list, - append, report_progress); + flags); string_list_clear(&list, 0); return result; @@ -772,7 +771,7 @@ int write_commit_graph_reachable(const char *obj_dir, int append, int write_commit_graph(const char *obj_dir, struct string_list *pack_indexes, struct string_list *commit_hex, - int append, int report_progress) + int flags) { struct packed_oid_list oids; struct packed_commit_list commits; @@ -791,6 +790,8 @@ int write_commit_graph(const char *obj_dir, struct strbuf progress_title = STRBUF_INIT; unsigned long approx_nr_objects; int res = 0; + int append = flags & COMMIT_GRAPH_APPEND; + int report_progress = flags & COMMIT_GRAPH_PROGRESS; if (!commit_graph_compatible(the_repository)) return 0; diff --git a/commit-graph.h b/commit-graph.h index cd333a0cd0..83fa548138 100644 --- a/commit-graph.h +++ b/commit-graph.h @@ -60,12 +60,14 @@ struct commit_graph *load_commit_graph_one(const char *graph_file); */ int generation_numbers_enabled(struct repository *r); -int write_commit_graph_reachable(const char *obj_dir, int append, - int report_progress); +#define COMMIT_GRAPH_APPEND (1 << 0) +#define COMMIT_GRAPH_PROGRESS (1 << 1) + +int write_commit_graph_reachable(const char *obj_dir, int flags); int write_commit_graph(const char *obj_dir, struct string_list *pack_indexes, struct string_list *commit_hex, - int append, int report_progress); + int flags); int verify_commit_graph(struct repository *r, struct commit_graph *g); From patchwork Wed Jan 23 21:59:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777951 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 065E61390 for ; Wed, 23 Jan 2019 21:59:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EBB5F2B40B for ; Wed, 23 Jan 2019 21:59:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E06802DA43; Wed, 23 Jan 2019 21:59:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6C69D2B40B for ; Wed, 23 Jan 2019 21:59:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726627AbfAWV7V (ORCPT ); Wed, 23 Jan 2019 16:59:21 -0500 Received: from mail-ed1-f66.google.com ([209.85.208.66]:45530 "EHLO mail-ed1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726157AbfAWV7U (ORCPT ); Wed, 23 Jan 2019 16:59:20 -0500 Received: by mail-ed1-f66.google.com with SMTP id d39so2925496edb.12 for ; Wed, 23 Jan 2019 13:59:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=te36t15TrASYJqeyEOeqASgjlyBGrxQZs5+2iwyGOBU=; b=L6Lpd1GTiG7W9VUegmHLcLs0UGJCBM0AAf4SnMbpKuuuYZeGa2WWQr/Z2xAv4q9wfJ CwWMkWnFuZmKwD9i7PDQl5gIHnW8LSVG+SJPBQMWiBqg9dqjaHk2+/vxds+dnQWMOaA9 vODYoZCOdCqthniB1Xdx/uT+gt3eeJSUfPr3GR/EmAn9ZuyZYxIT/IO0dGWI3fpk6+U2 Z9LmnBJbF9pVh0VoRsQ4f6vFBdqTuxEguYHfbgt8z3wW+CzqOo+8+Jiv/Q3pZBm2fu0t VCquYN8DFBqk6y2rnQnMSDPoXhrrpH3UHfgznHpKeEMxyrx+7Momit+AtMfXdmw+Tn1o 9TxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=te36t15TrASYJqeyEOeqASgjlyBGrxQZs5+2iwyGOBU=; b=nnway7dbNyBaCBfKwIZsHnO0B85hjbG+bEkl7Df5+U0i6VlSArliZCAJuYx9r7b80J ZxlD641GF+FynQ6EhcgFuFpdSlgsg1Dj5IrN5TNbVG1ipQ4YfqTvSXNdjZzfx9oPiBAP IMqX9ehM7woEW5SZ4LljN2zrmwtIrPABHi9cmok11Db4bH09zqvUTdBGGbytzlwyjrih gh+cgQnJUJGQtuiw+DRJuInYDc/KCSlJQMJRSavV3/4z4vY3w1/MPEEchqNoN2VuuDSA IH9jCbw1XXhTctBB/xtYMKDOUtIPj6Lm81x6yWUL7MFsNMdDkpA3F0lcUFMtGIGh98lF rn5Q== X-Gm-Message-State: AJcUukfwjC1Nj3zUN/oUS3ChFaX39LnJ5hajABA6aqoo7u0U690Sfazx 7b+imHPvsn8LIIT1F0yjgOKN54Yr X-Google-Smtp-Source: ALg8bN6PLFjBH7Evsxi7E9IyyBkQHnoGPjX6MMIL4/wcQnbTSRhCoErhjw7dHXaoTahcvr2AjTahUQ== X-Received: by 2002:a50:8689:: with SMTP id r9mr4029140eda.227.1548280757770; Wed, 23 Jan 2019 13:59:17 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t10-v6sm5616066ejg.41.2019.01.23.13.59.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 13:59:17 -0800 (PST) Date: Wed, 23 Jan 2019 13:59:17 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 21:59:10 GMT Message-Id: <39319e36bc35dc89aff7640a11c0819c3bce8b2f.1548280753.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 3/6] commit-graph: create new version flags Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, avarab@gmail.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee In anticipation of a new commit-graph file format version, create a flag for the write_commit_graph() and write_commit_graph_reachable() methods to take a version number. When there is no specified version, the implementation selects a default value. Currently, the only valid value is 1. The file format will change the header information, so place the existing header logic inside a switch statement with only one case. Signed-off-by: Derrick Stolee --- commit-graph.c | 58 +++++++++++++++++++++++++++++++++----------------- commit-graph.h | 1 + 2 files changed, 40 insertions(+), 19 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index 28fe2378be..f7f45893fd 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -25,9 +25,6 @@ #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16) -#define GRAPH_VERSION_1 0x1 -#define GRAPH_VERSION GRAPH_VERSION_1 - #define GRAPH_EXTRA_EDGES_NEEDED 0x80000000 #define GRAPH_PARENT_MISSING 0x7fffffff #define GRAPH_EDGE_LAST_MASK 0x7fffffff @@ -118,30 +115,35 @@ struct commit_graph *load_commit_graph_one(const char *graph_file) } graph_version = *(unsigned char*)(data + 4); - if (graph_version != GRAPH_VERSION) { + if (graph_version != 1) { error(_("graph version %X does not match version %X"), - graph_version, GRAPH_VERSION); - goto cleanup_fail; - } - - hash_version = *(unsigned char*)(data + 5); - if (hash_version != oid_version()) { - error(_("hash version %X does not match version %X"), - hash_version, oid_version()); + graph_version, 1); goto cleanup_fail; } graph = alloc_commit_graph(); + switch (graph_version) { + case 1: + hash_version = *(unsigned char*)(data + 5); + if (hash_version != oid_version()) { + error(_("hash version %X does not match version %X"), + hash_version, oid_version()); + goto cleanup_fail; + } + + graph->num_chunks = *(unsigned char*)(data + 6); + chunk_lookup = data + 8; + break; + } + graph->hash_len = the_hash_algo->rawsz; - graph->num_chunks = *(unsigned char*)(data + 6); graph->graph_fd = fd; graph->data = graph_map; graph->data_len = graph_size; last_chunk_id = 0; last_chunk_offset = 8; - chunk_lookup = data + 8; for (i = 0; i < graph->num_chunks; i++) { uint32_t chunk_id = get_be32(chunk_lookup + 0); uint64_t chunk_offset = get_be64(chunk_lookup + 4); @@ -792,10 +794,22 @@ int write_commit_graph(const char *obj_dir, int res = 0; int append = flags & COMMIT_GRAPH_APPEND; int report_progress = flags & COMMIT_GRAPH_PROGRESS; + int version = 0; + int header_size = 0; if (!commit_graph_compatible(the_repository)) return 0; + if (flags & COMMIT_GRAPH_VERSION_1) + version = 1; + if (!version) + version = 1; + if (version != 1) { + error(_("unsupported commit-graph version %d"), + version); + return 1; + } + oids.nr = 0; approx_nr_objects = approximate_object_count(); oids.alloc = approx_nr_objects / 32; @@ -980,10 +994,16 @@ int write_commit_graph(const char *obj_dir, hashwrite_be32(f, GRAPH_SIGNATURE); - hashwrite_u8(f, GRAPH_VERSION); - hashwrite_u8(f, oid_version()); - hashwrite_u8(f, num_chunks); - hashwrite_u8(f, 0); /* unused padding byte */ + hashwrite_u8(f, version); + + switch (version) { + case 1: + hashwrite_u8(f, oid_version()); + hashwrite_u8(f, num_chunks); + hashwrite_u8(f, 0); /* unused padding byte */ + header_size = 8; + break; + } chunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT; chunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP; @@ -994,7 +1014,7 @@ int write_commit_graph(const char *obj_dir, chunk_ids[3] = 0; chunk_ids[4] = 0; - chunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH; + chunk_offsets[0] = header_size + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH; chunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE; chunk_offsets[2] = chunk_offsets[1] + hashsz * commits.nr; chunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * commits.nr; diff --git a/commit-graph.h b/commit-graph.h index 83fa548138..e03df54e33 100644 --- a/commit-graph.h +++ b/commit-graph.h @@ -62,6 +62,7 @@ int generation_numbers_enabled(struct repository *r); #define COMMIT_GRAPH_APPEND (1 << 0) #define COMMIT_GRAPH_PROGRESS (1 << 1) +#define COMMIT_GRAPH_VERSION_1 (1 << 2) int write_commit_graph_reachable(const char *obj_dir, int flags); int write_commit_graph(const char *obj_dir, From patchwork Wed Jan 23 21:59:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777947 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4B9D51515 for ; Wed, 23 Jan 2019 21:59:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3C5A52B40B for ; Wed, 23 Jan 2019 21:59:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3047A2DA5E; Wed, 23 Jan 2019 21:59:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BF7E92DA42 for ; Wed, 23 Jan 2019 21:59:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726744AbfAWV7W (ORCPT ); Wed, 23 Jan 2019 16:59:22 -0500 Received: from mail-ed1-f65.google.com ([209.85.208.65]:37956 "EHLO mail-ed1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726125AbfAWV7U (ORCPT ); Wed, 23 Jan 2019 16:59:20 -0500 Received: by mail-ed1-f65.google.com with SMTP id h50so2954619ede.5 for ; Wed, 23 Jan 2019 13:59:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=WK4pOERtgzeysttqPNBR+dwND7YX0Bo1nbAcBF7zGoI=; b=j+0BegA5HuHtSgbH5/3iR8xyc6YSqnpHwU414zkPfh4sxw/KNkrr+PqwAhrmA07CSp FG1zBdrKkqL09mawnTIKv25JyhmEbyjtwFa8z8ifXlyLVH660Z6dHNf6u7isScA71zmK u4nlD+beZR5vJK3OHlQhuHkrVBIHDhTAHkMroP+PjTsokXewVL5T0pLbqbiudrQTA4H8 EvZt3mPqtMbCw5KlcqBUR0XnajycujkBvj9NjJ97fb4XNUi/3D/4mIlQaIns29IqOR8g Zn2vr5ZLJai7hFzoLQGr2P19+4kCrAsR5VfWHSPYbEHmppTBy9ynqcV5XieN+AR8saET HjlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=WK4pOERtgzeysttqPNBR+dwND7YX0Bo1nbAcBF7zGoI=; b=SUtJ4bsrErmtddGOrHInV9HxnIeF+vJjSbvtDSjMRYLYBi75QN7h31yg1gmw1reNIP 5ShfLfG50hQ1nJrWCSh+ZB/ZqrREV7q5KipsU7ElIaDKdwh6rvdqIsQk25H/SgicO1ZN PI9x8dSNAQu7HEhi9B/8JNo5YY7gyQyjn8xxnGiQAICgFcUR6mGcTnAlFFnUVNy21zle AKu9ow8xI0rtmM56ihp+UD9eGNLO10ZJ7HB7UJcPXbAcphLQpc0UgoLfVnRFc5Rw+p/v 5m/NOCozhNWvP3D1xybLO2CzOS7ZACIQqeQCq5K5xvIv8a1xxZnIqMCujas5QSOblp3F Ux1A== X-Gm-Message-State: AJcUukfFItrmS+AhMAlnltLijQLVLiMWMVYpkd23iFhv7ROtRromkIeS mHQ/Nl4B23BYymp+lgIy9B4a7/DX X-Google-Smtp-Source: ALg8bN4avwg/P4rNsQJVhP032bnGQjkaHMRRbTuC89CqHMv3A4ExnuF5yrzfuyZHMGAFfugIzW/n4w== X-Received: by 2002:a50:a395:: with SMTP id s21mr4344413edb.1.1548280758913; Wed, 23 Jan 2019 13:59:18 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m44sm11065247edm.54.2019.01.23.13.59.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 13:59:18 -0800 (PST) Date: Wed, 23 Jan 2019 13:59:18 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 21:59:11 GMT Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 4/6] commit-graph: add --version= option Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, avarab@gmail.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee Allo the commit-graph builtin to specify the file format version using the '--version=' option. Specify the version exactly in the verification tests as using a different version would change the offsets used in those tests. Signed-off-by: Derrick Stolee --- Documentation/git-commit-graph.txt | 3 +++ builtin/commit-graph.c | 13 +++++++++++-- t/t5318-commit-graph.sh | 2 +- 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt index 624470e198..1d1cc70de4 100644 --- a/Documentation/git-commit-graph.txt +++ b/Documentation/git-commit-graph.txt @@ -51,6 +51,9 @@ or `--stdin-packs`.) + With the `--append` option, include all commits that are present in the existing commit-graph file. ++ +With the `--version=` option, specify the file format version. Used +only for testing. 'read':: diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c index 0c92421f75..b1bed84260 100644 --- a/builtin/commit-graph.c +++ b/builtin/commit-graph.c @@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = { N_("git commit-graph [--object-dir ]"), N_("git commit-graph read [--object-dir ]"), N_("git commit-graph verify [--object-dir ]"), - N_("git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]"), + N_("git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits] [--version=]"), NULL }; @@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = { }; static const char * const builtin_commit_graph_write_usage[] = { - N_("git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]"), + N_("git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits] [--version=]"), NULL }; @@ -35,6 +35,7 @@ static struct opts_commit_graph { int stdin_packs; int stdin_commits; int append; + int version; } opts; @@ -141,6 +142,8 @@ static int graph_write(int argc, const char **argv) N_("start walk at commits listed by stdin")), OPT_BOOL(0, "append", &opts.append, N_("include all commits already in the commit-graph file")), + OPT_INTEGER(0, "version", &opts.version, + N_("specify the file format version")), OPT_END(), }; @@ -155,6 +158,12 @@ static int graph_write(int argc, const char **argv) if (opts.append) flags |= COMMIT_GRAPH_APPEND; + switch (opts.version) { + case 1: + flags |= COMMIT_GRAPH_VERSION_1; + break; + } + read_replace_refs = 0; if (opts.reachable) diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh index f4deb13b1d..b79d6263e9 100755 --- a/t/t5318-commit-graph.sh +++ b/t/t5318-commit-graph.sh @@ -328,7 +328,7 @@ test_expect_success 'replace-objects invalidates commit-graph' ' test_expect_success 'git commit-graph verify' ' cd "$TRASH_DIRECTORY/full" && - git rev-parse commits/8 | git commit-graph write --stdin-commits && + git rev-parse commits/8 | git commit-graph write --stdin-commits --version=1 && git commit-graph verify >output ' From patchwork Wed Jan 23 21:59:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777945 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BE3391390 for ; Wed, 23 Jan 2019 21:59:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AC5792B40B for ; Wed, 23 Jan 2019 21:59:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A0F802DA5F; Wed, 23 Jan 2019 21:59:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,LOTS_OF_MONEY,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ED8032B40B for ; Wed, 23 Jan 2019 21:59:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726785AbfAWV7X (ORCPT ); Wed, 23 Jan 2019 16:59:23 -0500 Received: from mail-ed1-f65.google.com ([209.85.208.65]:44675 "EHLO mail-ed1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726473AbfAWV7V (ORCPT ); Wed, 23 Jan 2019 16:59:21 -0500 Received: by mail-ed1-f65.google.com with SMTP id y56so2930621edd.11 for ; Wed, 23 Jan 2019 13:59:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=NivCbGUJB6WuorAbciQo5yG9ezlDC6l30RyY88tvGkg=; b=R899/X827Q0PCNtTxm1O0ApEPolCBpW6JfV87/0xtLFze6t3Q99BlM3qUc6BZwg1Jl 0rFTns4Zmku158efp1El95BaVQbeDCD6zOmZhfSTfWe67492YRk5G+wov6esD8kDjPqX qBN7jlr1C6+h2O+LbJEFnfHUdjeo2JeusKxBVb3Lb0U+jZ5Naup1sGfa+BdnjMoFTqGM NWTdBpzt9WC/ZAF5u3qB3Ino0X3x9aZSBrKWuwt2mhtBpBQSMX7GgW945X7uBdgKqlb5 obpj6eJLPNlgZyz/9y3tkz4Y1K2lGLEykXj10j0mlxo1n71EPnwCrKaeqmHfptMQKBYv z/sA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=NivCbGUJB6WuorAbciQo5yG9ezlDC6l30RyY88tvGkg=; b=hSzSPvCVAkKSgKK30A+y+BNX8O9DCX9sgF4+iIXlc+fE4Xqg2xtdItW62mXVkiARoi MFtijj+uGbg3QagTQQOw0Etb7P2F5SFIa2RyJapX+MMrazu6YillXfw96j2l5tdx4HEa LPXhfNlctygZmqW5Z3C9aw7j0TSpRCEplLQhcZPng0uJElzpYy5Uv8tXTkgqs21oP3Ml IHOEX37Z/W0HPQbCmgx91vomrbQfFbzXKJyaz3dnSEmf3U1P/X5cBjyalkHUnyt4SjUY ESKmwEXKse3fDi5wDE3/wsWwbs1LUil4ly0V1mqE9tLwLuWe0lX6FB5GXR7ibQwbQEVd 58Tg== X-Gm-Message-State: AJcUukcRdqSk0ZPmPl96ki3skrlDhtCgZElMrX5n9CdO18FqdvzXVhxT a1Rbu6jP261kbJWLEi0+hsQajJUc X-Google-Smtp-Source: ALg8bN56pE6mzZgCDVKqjLcVIEjWafVRjPV06Fa4tJlzvXleYvMlP9ON0gGRThiecZiN01LnrvX8Ng== X-Received: by 2002:a50:87d9:: with SMTP id 25mr4161714edz.259.1548280759815; Wed, 23 Jan 2019 13:59:19 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j8sm925182ejr.17.2019.01.23.13.59.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 13:59:19 -0800 (PST) Date: Wed, 23 Jan 2019 13:59:19 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 21:59:12 GMT Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 5/6] commit-graph: implement file format version 2 Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, avarab@gmail.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee The commit-graph file format had some shortcomings which we now correct: 1. The hash algorithm was determined by a single byte, instead of the 4-byte format identifier. 2. There was no way to update the reachability index we used. We currently only support generation numbers, but that will change in the future. 3. Git did not fail with error if the unused eighth byte was non-zero, so we could not use that to indicate an incremental file format without breaking compatibility across versions. The new format modifies the header of the commit-graph to solve these problems. We use the 4-byte hash format id, freeing up a byte in our 32-bit alignment to introduce a reachability index version. We can also fail to read the commit-graph if the eighth byte is non-zero. The 'git commit-graph read' subcommand needs updating to show the new data. Set the default file format version to 2, and adjust the tests to expect the new 'git commit-graph read' output. Signed-off-by: Derrick Stolee --- .../technical/commit-graph-format.txt | 26 +++++++++- builtin/commit-graph.c | 9 ++++ commit-graph.c | 47 ++++++++++++++++--- commit-graph.h | 1 + t/t5318-commit-graph.sh | 9 +++- 5 files changed, 83 insertions(+), 9 deletions(-) diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt index 16452a0504..e367aa94b1 100644 --- a/Documentation/technical/commit-graph-format.txt +++ b/Documentation/technical/commit-graph-format.txt @@ -31,13 +31,22 @@ and hash type. All 4-byte numbers are in network order. +There are two versions available, 1 and 2. These currently differ only in +the header. + HEADER: +All commit-graph files use the first five bytes for the same purpose. + 4-byte signature: The signature is: {'C', 'G', 'P', 'H'} 1-byte version number: - Currently, the only valid version is 1. + Currently, the valid version numbers are 1 and 2. + +The remainder of the header changes depending on the version. + +Version 1: 1-byte Hash Version (1 = SHA-1) We infer the hash length (H) from this value. @@ -47,6 +56,21 @@ HEADER: 1-byte (reserved for later use) Current clients should ignore this value. +Version 2: + + 1-byte number (C) of "chunks" + + 1-byte reachability index version number: + Currently, the only valid number is 1. + + 1-byte (reserved for later use) + Current clients expect this value to be zero, and will not + try to read the commit-graph file if it is non-zero. + + 4-byte format identifier for the hash algorithm: + If this identifier does not agree with the repository's current + hash algorithm, then the client will not read the commit graph. + CHUNK LOOKUP: (C + 1) * 12 bytes listing the table of contents for the chunks: diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c index b1bed84260..28787d0c9c 100644 --- a/builtin/commit-graph.c +++ b/builtin/commit-graph.c @@ -102,6 +102,11 @@ static int graph_read(int argc, const char **argv) *(unsigned char*)(graph->data + 5), *(unsigned char*)(graph->data + 6), *(unsigned char*)(graph->data + 7)); + + if (*(unsigned char *)(graph->data + 4) == 2) + printf("hash algorithm: %X\n", + get_be32(graph->data + 8)); + printf("num_commits: %u\n", graph->num_commits); printf("chunks:"); @@ -162,6 +167,10 @@ static int graph_write(int argc, const char **argv) case 1: flags |= COMMIT_GRAPH_VERSION_1; break; + + case 2: + flags |= COMMIT_GRAPH_VERSION_2; + break; } read_replace_refs = 0; diff --git a/commit-graph.c b/commit-graph.c index f7f45893fd..aeb6cae656 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -90,7 +90,8 @@ struct commit_graph *load_commit_graph_one(const char *graph_file) uint64_t last_chunk_offset; uint32_t last_chunk_id; uint32_t graph_signature; - unsigned char graph_version, hash_version; + unsigned char graph_version, hash_version, reach_index_version; + uint32_t hash_id; if (fd < 0) return NULL; @@ -115,9 +116,9 @@ struct commit_graph *load_commit_graph_one(const char *graph_file) } graph_version = *(unsigned char*)(data + 4); - if (graph_version != 1) { - error(_("graph version %X does not match version %X"), - graph_version, 1); + if (!graph_version || graph_version > 2) { + error(_("unsupported graph version %X"), + graph_version); goto cleanup_fail; } @@ -135,6 +136,30 @@ struct commit_graph *load_commit_graph_one(const char *graph_file) graph->num_chunks = *(unsigned char*)(data + 6); chunk_lookup = data + 8; break; + + case 2: + graph->num_chunks = *(unsigned char *)(data + 5); + + reach_index_version = *(unsigned char *)(data + 6); + if (reach_index_version != 1) { + error(_("unsupported reachability index version %d"), + reach_index_version); + goto cleanup_fail; + } + + if (*(unsigned char*)(data + 7)) { + error(_("unsupported value in commit-graph header")); + goto cleanup_fail; + } + + hash_id = get_be32(data + 8); + if (hash_id != the_hash_algo->format_id) { + error(_("commit-graph hash algorithm does not match current algorithm")); + goto cleanup_fail; + } + + chunk_lookup = data + 12; + break; } graph->hash_len = the_hash_algo->rawsz; @@ -802,9 +827,11 @@ int write_commit_graph(const char *obj_dir, if (flags & COMMIT_GRAPH_VERSION_1) version = 1; + if (flags & COMMIT_GRAPH_VERSION_2) + version = 2; if (!version) - version = 1; - if (version != 1) { + version = 2; + if (version <= 0 || version > 2) { error(_("unsupported commit-graph version %d"), version); return 1; @@ -1003,6 +1030,14 @@ int write_commit_graph(const char *obj_dir, hashwrite_u8(f, 0); /* unused padding byte */ header_size = 8; break; + + case 2: + hashwrite_u8(f, num_chunks); + hashwrite_u8(f, 1); /* reachability index version */ + hashwrite_u8(f, 0); /* unused padding byte */ + hashwrite_be32(f, the_hash_algo->format_id); + header_size = 12; + break; } chunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT; diff --git a/commit-graph.h b/commit-graph.h index e03df54e33..050137063b 100644 --- a/commit-graph.h +++ b/commit-graph.h @@ -63,6 +63,7 @@ int generation_numbers_enabled(struct repository *r); #define COMMIT_GRAPH_APPEND (1 << 0) #define COMMIT_GRAPH_PROGRESS (1 << 1) #define COMMIT_GRAPH_VERSION_1 (1 << 2) +#define COMMIT_GRAPH_VERSION_2 (1 << 3) int write_commit_graph_reachable(const char *obj_dir, int flags); int write_commit_graph(const char *obj_dir, diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh index b79d6263e9..3ff5e3b48d 100755 --- a/t/t5318-commit-graph.sh +++ b/t/t5318-commit-graph.sh @@ -65,7 +65,8 @@ graph_read_expect() { NUM_CHUNKS=$((3 + $(echo "$2" | wc -w))) fi cat >expect <<- EOF - header: 43475048 1 1 $NUM_CHUNKS 0 + header: 43475048 2 $NUM_CHUNKS 1 0 + hash algorithm: 73686131 num_commits: $1 chunks: oid_fanout oid_lookup commit_metadata$OPTIONAL EOF @@ -390,10 +391,14 @@ test_expect_success 'detect bad signature' ' ' test_expect_success 'detect bad version' ' - corrupt_graph_and_verify $GRAPH_BYTE_VERSION "\02" \ + corrupt_graph_and_verify $GRAPH_BYTE_VERSION "\03" \ "graph version" ' +test_expect_success 'detect version 2 with version 1 data' ' + corrupt_graph_and_verify $GRAPH_BYTE_VERSION "\02" \ + "reachability index version" +' test_expect_success 'detect bad hash version' ' corrupt_graph_and_verify $GRAPH_BYTE_HASH "\02" \ "hash version" From patchwork Wed Jan 23 21:59:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin via GitGitGadget X-Patchwork-Id: 10777949 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DCC9A1390 for ; Wed, 23 Jan 2019 21:59:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CE4FC2B40B for ; Wed, 23 Jan 2019 21:59:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C28162DA43; Wed, 23 Jan 2019 21:59:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 703AF2B40B for ; Wed, 23 Jan 2019 21:59:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726792AbfAWV7Y (ORCPT ); Wed, 23 Jan 2019 16:59:24 -0500 Received: from mail-ed1-f68.google.com ([209.85.208.68]:44678 "EHLO mail-ed1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726120AbfAWV7W (ORCPT ); Wed, 23 Jan 2019 16:59:22 -0500 Received: by mail-ed1-f68.google.com with SMTP id y56so2930635edd.11 for ; Wed, 23 Jan 2019 13:59:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IEyNyfk8QKwTAdJfBQdQRyBKmNz+9c/mmb3rMHAN2RU=; b=nAwsYj0lnMu/7JURGcIvjhMbsVVgcCgRhwxoI8auJS9t0DH2hsT8k0iB6V5UR8xJ0C NR30VsIt9NGB2RvjmDs6yfa40kerqKJf+YEbZ8hk87WfqCJqvooVaA6eh0SvvYifwoaz NMGee13rkHPt5jh2mWwlVt9saP/RDdDdiHej0lM7hzdiKp7EcpjKjNy3WD/icngeszME n2PjiftWLja559yyDwCb8Ha+mwkQva/UKvyH1X8K0Du4TnKT3+XYNMdMekEq/RBypP3f Sj0WOKen+d95hYx7fX+rV94Op5NcgEFLf7dZ53fJ9jbaLWEbzekFmvXxaZFRbL/O+i/m N09w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IEyNyfk8QKwTAdJfBQdQRyBKmNz+9c/mmb3rMHAN2RU=; b=tr2jZD3smQ9J/Jzz9F4DlUYn8vCgi6LvKSpEAYX+JpHCkJmIavfSu9Mcrq/NYX0Pd+ 4f4W02lKTzCRbCIynqTldaIGPc3X5oq888e4Axm1jCmFVtOJelAai889iRxDiFqPhEN9 gfcHyPBa/mI7n3JGF36VqNTl/3uB9CVA2X1levRK26+88T3rMEVj/BIRuUITRWBOML52 1KSii6YuZubJnrPMI4zUp9JNBODqBbylorznk47+WUFBNZrUEKQ+MH8bwUpd3ASv2YUI nZNI1dGuVjl76ior678r3vMxsd9pFafSl3aaxQ3wU2ArAGjrTzNzCuT38gtbHlV74Ahx 4tIQ== X-Gm-Message-State: AJcUukfclqzA0E6Ruhm7GH1FdgpLDW6TIeLYtp45IIzzAwedt1a9CYN2 r5EFyKAKdSPnjDasopAZeZMoftdD X-Google-Smtp-Source: ALg8bN44IgrOv1wDwN5MymTcYVJKPkD7lp3NGbjYnu73y633gL6ypoToIz7ulLhGYR/71y7X0CLyQA== X-Received: by 2002:a50:e045:: with SMTP id g5mr4358667edl.152.1548280760490; Wed, 23 Jan 2019 13:59:20 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e26-v6sm5836358ejb.29.2019.01.23.13.59.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jan 2019 13:59:20 -0800 (PST) Date: Wed, 23 Jan 2019 13:59:20 -0800 (PST) X-Google-Original-Date: Wed, 23 Jan 2019 21:59:13 GMT Message-Id: <693900b4c5af9adb98d98549b9340ad9f9d09562.1548280754.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 6/6] commit-graph: test verifying a corrupt v2 header Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, avarab@gmail.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee The commit-graph file format v2 changes the v1 data only in the header information. Add tests that check the 'verify' subcommand catches corruption in the v2 header. Signed-off-by: Derrick Stolee --- t/t5318-commit-graph.sh | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh index 3ff5e3b48d..be7bbf911a 100755 --- a/t/t5318-commit-graph.sh +++ b/t/t5318-commit-graph.sh @@ -497,6 +497,37 @@ test_expect_success 'git fsck (checks commit-graph)' ' test_must_fail git fsck ' +test_expect_success 'rewrite commmit-graph with version 2' ' + rm -f .git/objects/info/commit-graph && + git commit-graph write --reachable --version=2 && + git commit-graph verify +' + +GRAPH_BYTE_CHUNK_COUNT=5 +GRAPH_BYTE_REACH_INDEX=6 +GRAPH_BYTE_UNUSED=7 +GRAPH_BYTE_HASH=8 + +test_expect_success 'detect low chunk count (v2)' ' + corrupt_graph_and_verify $GRAPH_CHUNK_COUNT "\02" \ + "missing the .* chunk" +' + +test_expect_success 'detect incorrect reachability index' ' + corrupt_graph_and_verify $GRAPH_REACH_INDEX "\03" \ + "reachability index version" +' + +test_expect_success 'detect non-zero unused byte' ' + corrupt_graph_and_verify $GRAPH_BYTE_UNUSED "\01" \ + "unsupported value" +' + +test_expect_success 'detect bad hash version (v2)' ' + corrupt_graph_and_verify $GRAPH_BYTE_HASH "\00" \ + "hash algorithm" +' + test_expect_success 'setup non-the_repository tests' ' rm -rf repo && git init repo &&