| Message ID | 20220922232947.631309-1-calvinwan@google.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | submodule: parallelize status | expand |
Calvin Wan <calvinwan@google.com> writes: > Future work: The reason why status is much slower on a cold cache vs > warm cache is mainly due to refreshing the index. It is worth > investigating whether this is entirely necessary. I suspect that the CI breakage https://github.com/git/git/actions/runs/3115673002/jobs/5052804463 is due to this topic. In "SANTIZE=leak GIT_TEST_PASSING_SANITIZE_LEAK=true make test" run locally, I see t7507 failing quite badly.
I'm getting the following error running that command locally: Bailout called. Further testing stopped: GIT_TEST_PASSING_SANITIZE_LEAK=true has no effect except when compiled with SANITIZE=leak
When running 'git status' in a superproject, git spawns a subprocess in series to run status for every submodule. For projects with many large submodules, parallelizing status subprocesses can significantly speed up the runtime for both cold and warm caches. Here are some timing tests from running status on the Android Open Source Project (AOSP). My machine has an SSD and 48 cores. Warm Cache: git 2.37 Time (mean ± σ): 17.685 s ± 2.040 s [User: 5.041 s, System: 22.799 s] Range (min … max): 16.168 s … 22.804 s 10 runs this patch (status.parallelSubmodules=1) Time (mean ± σ): 13.102 s ± 0.500 s [User: 4.894 s, System: 19.533 s] Range (min … max): 12.841 s … 14.447 s 10 runs this patch (status.parallelSubmodules=5) Time (mean ± σ): 3.994 s ± 0.152 s [User: 4.998 s, System: 20.805 s] Range (min … max): 3.744 s … 4.163 s 10 runs this patch (status.parallelSubmodules=10) Time (mean ± σ): 3.445 s ± 0.085 s [User: 5.151 s, System: 20.208 s] Range (min … max): 3.319 s … 3.586 s 10 runs this patch (status.parallelSubmodules=20) Time (mean ± σ): 3.626 s ± 0.109 s [User: 5.087 s, System: 20.366 s] Range (min … max): 3.438 s … 3.763 s 10 runs We can see that there are diminishing returns and even slightly worse performance after a certain number of max processes, but optimally there is a speed up factor of around 5. Cold Cache: git 2.37 mean of 3 runs: 6m32s this patch (status.parallelSubmodules=1) mean of 3 runs: 5m34s this patch (status.parallelSubmodules=5) mean of 3 runs: 2m23s this patch (status.parallelSubmodules=10) mean of 3 runs: 2m45s this patch (status.parallelSubmodules=20) mean of 3 runs: 3m23s We can witness the same phenomenon as above and optimally there is a speed up factor of around 2.7. Patch 1 adds output piping to run_processes_parallel so the output from each submodule can be parsed. Patches 2 and 3 move preexisting functionality into separate functions and refactor code to prepare for patch 4 to implement parallelization. Future work: The reason why status is much slower on a cold cache vs warm cache is mainly due to refreshing the index. It is worth investigating whether this is entirely necessary. Calvin Wan (4): run-command: add pipe_output to run_processes_parallel submodule: move status parsing into function diff-lib: refactor functions diff-lib: parallelize run_diff_files for submodules Documentation/config/status.txt | 6 + add-interactive.c | 2 +- builtin/add.c | 4 +- builtin/commit.c | 6 + builtin/diff-files.c | 2 +- builtin/diff.c | 2 +- builtin/merge.c | 2 +- builtin/stash.c | 2 +- builtin/submodule--helper.c | 4 +- diff-lib.c | 120 +++++++++++++----- diff.h | 2 +- run-command.c | 6 +- run-command.h | 9 ++ submodule.c | 213 +++++++++++++++++++++++++++----- submodule.h | 9 ++ t/helper/test-run-command.c | 31 ++++- t/t0061-run-command.sh | 26 ++++ wt-status.c | 6 +- wt-status.h | 1 + 19 files changed, 372 insertions(+), 81 deletions(-) base-commit: 5502f77b6944eda8e26813d8f542cffe7d110aea