| Message ID | cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | progress: assert "global_progress" + test fixes / cleanup | expand |
On Thu, Oct 14, 2021 at 12:28:16AM +0200, Ævar Arnfjörð Bjarmason wrote: > > This series fixes various issues in and related to progress.c, and > adds a BUG() assertion for us not starting two progress bars at the > same time. Those changes are needed for subsequent changes that do > more interesting things with this new global progress bar. > > This v3 hopefully addresses all the feedback on the v2, thanks > all. Changes: > > * Fix a memory leak in 1/10, and make the progress tests use the > SANITIZE=leak test mode. > > * Simplified some of the test-progress.c code (no more "start" > handling, the "total" count is mandatory now. > > * Split out a formatting change into 2/10 to make 3/10 easier to > read. > > * A new 9/10 makes an ad-hoc test recipie in 10/10 easier to explain > (in response to Emily's comment). > > * The BUG() assertion in 10/10 now has a much better message, we dump > the title of the two progress bars in play if we have a bug where > we started two at the same time. One thing I noticed (and so did SZEDER) was the addition of string_list to the helper; I made a comment accordingly in the patch where it was added. It kind of seemed like an oversight - but I remember you were wanting to head in that direction (multiple progress bar support) so maybe it was paving the way for that? Anyways, with or without it the series looks OK to me in this round too. - Emily
On Thu, Oct 21 2021, Emily Shaffer wrote: > On Thu, Oct 14, 2021 at 12:28:16AM +0200, Ævar Arnfjörð Bjarmason wrote: >> >> This series fixes various issues in and related to progress.c, and >> adds a BUG() assertion for us not starting two progress bars at the >> same time. Those changes are needed for subsequent changes that do >> more interesting things with this new global progress bar. >> >> This v3 hopefully addresses all the feedback on the v2, thanks >> all. Changes: >> >> * Fix a memory leak in 1/10, and make the progress tests use the >> SANITIZE=leak test mode. >> >> * Simplified some of the test-progress.c code (no more "start" >> handling, the "total" count is mandatory now. >> >> * Split out a formatting change into 2/10 to make 3/10 easier to >> read. >> >> * A new 9/10 makes an ad-hoc test recipie in 10/10 easier to explain >> (in response to Emily's comment). >> >> * The BUG() assertion in 10/10 now has a much better message, we dump >> the title of the two progress bars in play if we have a bug where >> we started two at the same time. > > One thing I noticed (and so did SZEDER) was the addition of string_list > to the helper; I made a comment accordingly in the patch where it was > added. It kind of seemed like an oversight - but I remember you were > wanting to head in that direction (multiple progress bar support) so > maybe it was paving the way for that? > > Anyways, with or without it the series looks OK to me in this round too. Replied to in the thread with SZEDER at: https://lore.kernel.org/git/211022.864k99kune.gmgdl@evledraar.gmail.com/ I.e. it's not for multi-progress-bar support, but because the API doesn't xstrdup() the title. The loop handing it the title needs to keep the string it gave to it alive.
This series fixes various issues in and related to progress.c, and adds a BUG() assertion for us not starting two progress bars at the same time. Those changes are needed for subsequent changes that do more interesting things with this new global progress bar. This v3 hopefully addresses all the feedback on the v2, thanks all. Changes: * Fix a memory leak in 1/10, and make the progress tests use the SANITIZE=leak test mode. * Simplified some of the test-progress.c code (no more "start" handling, the "total" count is mandatory now. * Split out a formatting change into 2/10 to make 3/10 easier to read. * A new 9/10 makes an ad-hoc test recipie in 10/10 easier to explain (in response to Emily's comment). * The BUG() assertion in 10/10 now has a much better message, we dump the title of the two progress bars in play if we have a bug where we started two at the same time. Ævar Arnfjörð Bjarmason (10): leak tests: fix a memory leaks in "test-progress" helper progress.c test helper: add missing braces progress.c tests: make start/stop verbs on stdin progress.c tests: test some invalid usage progress.c: move signal handler functions lower progress.c: call progress_interval() from progress_test_force_update() progress.c: add temporary variable from progress struct pack-bitmap-write.c: don't return without stop_progress() various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO) progress.c: add & assert a "global_progress" variable builtin/bisect--helper.c | 2 +- builtin/bundle.c | 2 +- compat/mingw.c | 2 +- pack-bitmap-write.c | 6 +- progress.c | 111 ++++++++++++++++++++---------------- t/helper/test-progress.c | 43 +++++++++----- t/t0500-progress-display.sh | 105 +++++++++++++++++++++++++++------- 7 files changed, 183 insertions(+), 88 deletions(-) Range-diff against v2: -: ----------- > 1: 40f7c438a1e leak tests: fix a memory leaks in "test-progress" helper -: ----------- > 2: ee177d253a8 progress.c test helper: add missing braces 1: e0a294eb479 ! 3: 045d58d8201 progress.c tests: make start/stop verbs on stdin @@ Commit message This makes for tests that are easier to read, since the recipe will mirror the API usage, and allows for easily testing invalid usage that would yield (or should yield) a BUG(), e.g. providing two "start" - calls in a row. A subsequent commit will add such stress tests. + calls in a row. A subsequent commit will add such tests. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> @@ t/helper/test-progress.c * * Reads instructions from standard input, one instruction per line: * -+ * "start[ <total>[ <title>]]" - Call start_progress(title, total), -+ * when "start" use a title of -+ * "Working hard" with a total of 0. ++ * "start <total>[ <title>]" - Call start_progress(title, total), ++ * Uses the default title of "Working hard" ++ * if the " <title>" is omitted. * "progress <items>" - Call display_progress() with the given item count * as parameter. * "throughput <bytes> <millis> - Call display_throughput() with the given @@ t/helper/test-progress.c * See 't0500-progress-display.sh' for examples. */ @@ + #include "parse-options.h" + #include "progress.h" + #include "strbuf.h" ++#include "string-list.h" int cmd__progress(int argc, const char **argv) { - int total = 0; - const char *title; -+ const char *default_title = "Working hard"; -+ char *detached_title = NULL; ++ const char *const default_title = "Working hard"; ++ struct string_list list = STRING_LIST_INIT_DUP; ++ const struct string_list_item *item; struct strbuf line = STRBUF_INIT; - struct progress *progress; + struct progress *progress = NULL; @@ t/helper/test-progress.c char *end; - if (skip_prefix(line.buf, "progress ", (const char **) &end)) { -+ if (!strcmp(line.buf, "start")) { -+ progress = start_progress(default_title, 0); -+ } else if (skip_prefix(line.buf, "start ", (const char **) &end)) { ++ if (skip_prefix(line.buf, "start ", (const char **) &end)) { + uint64_t total = strtoull(end, &end, 10); + if (*end == '\0') { + progress = start_progress(default_title, total); + } else if (*end == ' ') { -+ free(detached_title); -+ detached_title = strbuf_detach(&line, NULL); -+ progress = start_progress(end + 1, total); ++ item = string_list_insert(&list, end + 1); ++ progress = start_progress(item->string, total); + } else { + die("invalid input: '%s'\n", line.buf); + } @@ t/helper/test-progress.c if (*end != '\0') die("invalid input: '%s'\n", line.buf); @@ t/helper/test-progress.c: int cmd__progress(int argc, const char **argv) - die("invalid input: '%s'\n", line.buf); - progress_test_ns = test_ms * 1000 * 1000; display_throughput(progress, byte_count); -- } else if (!strcmp(line.buf, "update")) -+ } else if (!strcmp(line.buf, "update")) { + } else if (!strcmp(line.buf, "update")) { progress_test_force_update(); -- else + } else if (!strcmp(line.buf, "stop")) { + stop_progress(&progress); -+ } else { + } else { die("invalid input: '%s'\n", line.buf); -+ } + } } - stop_progress(&progress); -+ free(detached_title); + strbuf_release(&line); ++ string_list_clear(&list, 0); return 0; } @@ t/t0500-progress-display.sh: Working hard.......2.........3.........4.........5. EOF cat >in <<-\EOF && -- update + start 100000 Working hard.......2.........3.........4.........5.........6 + update progress 1 update progress 2 @@ t/t0500-progress-display.sh: test_expect_success 'progress display with throughp EOF cat >in <<-\EOF && -+ start ++ start 0 throughput 102400 1000 update progress 10 @@ t/t0500-progress-display.sh: test_expect_success 'cover up after throughput shor EOF cat >in <<-\EOF && -+ start ++ start 0 throughput 409600 1000 update progress 1 @@ t/t0500-progress-display.sh: test_expect_success 'cover up after throughput shor EOF cat >in <<-\EOF && -+ start ++ start 0 throughput 1 1000 update progress 1 2: 7b1220b641e ! 4: efc0ec360cc progress.c tests: test some invalid usage @@ Commit message extends the trace2 tests added in 98a13647408 (trace2: log progress time and throughput, 2020-05-12). + These tests are not merely testing the helper, but invalid API usage + that can happen if the progress.c API is misused. + + The "without stop" test will leak under SANITIZE=leak, since this + buggy use of the API will leak memory. But let's not skip it entirely, + or use the "!SANITIZE_LEAK" prerequisite check as we'd do with tests + that we're skipping due to leaks we haven't fixed yet. Instead + annotate the specific command that should skip leak checking with + custom $LSAN_OPTIONS[1]. + + 1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer + Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> ## t/t0500-progress-display.sh ## @@ t/t0500-progress-display.sh: test_expect_success 'progress generates traces' ' +test_expect_success 'progress generates traces: stop / start' ' + cat >in <<-\EOF && -+ start ++ start 0 + stop + EOF + @@ t/t0500-progress-display.sh: test_expect_success 'progress generates traces' ' + +test_expect_success 'progress generates traces: start without stop' ' + cat >in <<-\EOF && -+ start ++ start 0 + EOF + -+ GIT_TRACE2_EVENT="$(pwd)/trace-start.event" test-tool progress \ ++ GIT_TRACE2_EVENT="$(pwd)/trace-start.event" \ ++ LSAN_OPTIONS=detect_leaks=0 \ ++ test-tool progress \ + <in 2>stderr && + grep region_enter.*progress trace-start.event && + ! grep region_leave.*progress trace-start.event 3: f1b8bf1dbde = 5: 9e36f03de46 progress.c: move signal handler functions lower 4: 74057b0046a = 6: c7c3843564e progress.c: call progress_interval() from progress_test_force_update() 5: 250e50667c2 < -: ----------- progress.c: stop eagerly fflush(stderr) when not a terminal 6: d4e9ff1de73 = 7: cd2d27b1626 progress.c: add temporary variable from progress struct 7: a3f133ca7ad ! 8: e0a3510dd88 pack-bitmap-write.c: add a missing stop_progress() @@ Metadata Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com> ## Commit message ## - pack-bitmap-write.c: add a missing stop_progress() + pack-bitmap-write.c: don't return without stop_progress() Fix a bug that's been here since 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21), we did not call stop_progress() if we - reached the early exit in this function. This will matter in a - subsequent commit where we BUG(...) out if this happens, and matters - now e.g. because we don't have a corresponding "region_end" for the - progress trace2 event. + reached the early exit in this function. + We could call stop_progress() before we return, but better yet is to + defer calling start_progress() until we need it. + + This will matter in a subsequent commit where we BUG(...) out if this + happens, and matters now e.g. because we don't have a corresponding + "region_end" for the progress trace2 event. + + Suggested-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> ## pack-bitmap-write.c ## @@ pack-bitmap-write.c: void bitmap_writer_select_commits(struct commit **indexed_commits, + + QSORT(indexed_commits, indexed_commits_nr, date_compare); + +- if (writer.show_progress) +- writer.progress = start_progress("Selecting bitmap commits", 0); +- if (indexed_commits_nr < 100) { for (i = 0; i < indexed_commits_nr; ++i) push_bitmapped_commit(indexed_commits[i]); -+ stop_progress(&writer.progress); return; } ++ if (writer.show_progress) ++ writer.progress = start_progress("Selecting bitmap commits", 0); ++ + for (;;) { + struct commit *chosen = NULL; + -: ----------- > 9: 2cf14881ecf various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO) 8: 1bd285eba0d ! 10: 01d5bbfce76 progress.c: add & assert a "global_progress" variable @@ Commit message 3. I've likewise done an ad-hoc test to force progress bars to be displayed with: - perl -pi -e 's[isatty\((?:STDERR_FILENO|2)\)][1]g' $(git grep -l 'isatty\((STDERR_FILENO|2)\)') + perl -pi -e 's[isatty\(2\)][1]g' $(git grep -l -F 'isatty(2)') I.e. to replace all checks (not just for progress) of checking whether STDERR is connected to a TTY, and then monkeypatching @@ progress.c: void progress_test_force_update(void) struct itimerval v; + if (global_progress) -+ BUG("should have no global_progress in set_progress_signal()"); ++ BUG("'%s' progress still active when trying to start '%s'", ++ global_progress->title, progress->title); + global_progress = progress; + if (progress_testing) @@ progress.c: static void set_progress_signal(void) struct itimerval v = {{0,},}; + if (!global_progress) -+ BUG("should have a global_progress in clear_progress_signal()"); ++ BUG("should have active global_progress when cleaning up"); + global_progress = NULL; + if (progress_testing) @@ t/t0500-progress-display.sh: test_expect_success 'cover up after throughput shor + + test_must_fail test-tool progress \ + <in 2>stderr && -+ grep -E "^BUG: .*: should have no global_progress in set_progress_signal\(\)$" stderr ++ grep "^BUG: .*'\''one'\'' progress still active when trying to start '\''two'\''$" stderr +' + test_expect_success 'progress generates traces' '