| Message ID | 20180926195442.1380-1-benpeart@microsoft.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | speed up index load through parallelization | expand |
Ben Peart <peartben@gmail.com> writes: > Base Ref: master > Web-Diff: https://github.com/benpeart/git/commit/a0300882d4 > Checkout: git fetch https://github.com/benpeart/git read-index-multithread-v6 && git checkout a0300882d4 > > > This iteration brings back the Index Entry Offset Table (IEOT) extension > which enables us to multi-thread the cache entry parsing without having > the primary thread have to scan all the entries first. In cases where the > cache entry parsing is the most expensive part, this yields some additional > savings. Nice. > Test w/100,000 files Baseline Optimize V4 Extensions Entries > ---------------------------------------------------------------------------- > 0002.1: read_cache 22.36 18.74 -16.2% 18.64 -16.6% 12.63 -43.5% > > Test w/1,000,000 files Baseline Optimize V4 Extensions Entries > ----------------------------------------------------------------------------- > 0002.1: read_cache 304.40 270.70 -11.1% 195.50 -35.8% 204.82 -32.7% > > Note that on the 1,000,000 files case, multi-threading the cache entry parsing > does not yield a performance win. This is because the cost to parse the > index extensions in this repo, far outweigh the cost of loading the cache > entries. > ... > The high cost of parsing the index extensions is driven by the cache tree > and the untracked cache extensions. As this is currently the longest pole, > any reduction in this time will reduce the overall index load times so is > worth further investigation in another patch series. Interesting. > One option would be to load each extension on a separate thread but I > believe that is overkill for the vast majority of repos. Instead, some > optimization of the loading code for these two extensions is probably worth > looking into as a quick examination shows that the bulk of the time for both > of them is spent in xcalloc(). Thanks. Looking forward to block some quality time off to read this through, but from the cursory look (read: diff between the previous round), this looks quite promising.
On Wed, Sep 26, 2018 at 9:54 PM Ben Peart <peartben@gmail.com> wrote: > The high cost of parsing the index extensions is driven by the cache tree > and the untracked cache extensions. As this is currently the longest pole, > any reduction in this time will reduce the overall index load times so is > worth further investigation in another patch series. > > Name First Last Elapsed > | + git!read_index_extension 684.052 870.244 186.192 > | + git!cache_tree_read 684.052 797.801 113.749 > | + git!read_untracked_extension 797.801 870.244 72.443 > > One option would be to load each extension on a separate thread but I > believe that is overkill for the vast majority of repos. They both grow proportional to the number of trees in worktree, which probably also scales to the worktree size. Frankly I think the parallel index loading is already overkill for the majority of repos, so speeding up further of the 1% giant repos does not sound that bad. And I think you already lay the foundation for loading index stuff in parallel with this series. > Instead, some > optimization of the loading code for these two extensions is probably worth > looking into as a quick examination shows that the bulk of the time for both > of them is spent in xcalloc(). Another easy "optimization" is delaying loading these until we need them (or load them in background, read_index() returns even before these extensions are finished, but this is of course trickier). UNTR extension for example is only useful for "git status" (and maybe one or two other use cases). Not having to load them all the time is likely a win. The role of TREE extension has grown bigger these days so it's still maybe worth putting more effort into making it load it faster rather than just hiding the cost.