[v6,00/10] Make submodules work if .gitmodules is not checked out
mbox series

Message ID 20181005130601.15879-1-ao2@ao2.it
Headers show
Series
  • Make submodules work if .gitmodules is not checked out
Related show

Message

Antonio Ospite Oct. 5, 2018, 1:05 p.m. UTC
Hi,

this series teaches git to try and read the .gitmodules file from the
index (:.gitmodules) or from the current branch (HEAD:.gitmodules) when
the file is not readily available in the working tree.

This can be used, together with sparse checkouts, to enable submodule
usage with programs like vcsh[1] which manage multiple repositories with
their working trees sharing the same path.

[1] https://github.com/RichiH/vcsh


Thanks to SZEDER Gábor we found out that the changes in patch 9 could
allow to access the object store in an inconsistent way when using
multi-threading in "git grep --recurse-submodules", this has been dealt
with in this revision.

BTW, with Stefan Beller we also identified some unneeded code which
could have been removed to alleviate the issue, but that would not have
solved it completely; so, I am not removing the unnecessary call to
repo_read_gitmodules() builtin/grep.c in this series, possibly this can
become a stand-alone change.

The problems from above also made me investigate what implications the
current use of a global object store had on my new addition, and now
I know that there is one case which I cannot support just yet: nested
submodules without .gitmodules in their working tree.

This case has been documented with a warning and test_expect_failure
items in tests, and hitting the unsupported case does not alter the
current behavior of git.

Apart form patch 9 and 10 there are no major changes to the previous
iteration.

IMHO we are in a place where the problem has been analyzed with enough
depth, the limitations have been identified and dealt with in a way that
should not affect current users nor diminish the usefulness of the new
code.

v5 of the series is here:
https://public-inbox.org/git/20180917140940.3839-1-ao2@ao2.it/

v4 of the series is here:
https://public-inbox.org/git/20180824132951.8000-1-ao2@ao2.it/

v3 of the series is here:
https://public-inbox.org/git/20180814110525.17801-1-ao2@ao2.it/

v2 of the series is here:
https://public-inbox.org/git/20180802134634.10300-1-ao2@ao2.it/

v1 of the series, with some background info, is here:
https://public-inbox.org/git/20180514105823.8378-1-ao2@ao2.it/

Changes since v5:

  * print_config_from_gitmodules() in patch 1 now accepts a struct
    repository argument.

  * submodule--helper config in patch 5 has been adjusted to use the new
    signature of print_config_from_gitmodules().

  * In patch 9 the grep read lock in builtin/grep.c now covers all code
    paths involving config_from_gitmodules().
    
    FTR git-grep is the only place where config_from_gitmodules() is
    called from multi-threaded code.

  * Patch 9 also documents the rare case that cannot be supported just
    yet, and adds a warning to the user.

  * In patch 9, config_from_gitmodules() now does not read any config
    when the config source is not specified.(I added a catch-all "else"
    block) This match more closely the behavior of the old code using
    git_config_from_file.

  * Added a new test tool in patch 10 to exercise config_read_config()
    in a more direct way, passing an arbitrary repository.
    
    Admittedly, patch 10 performs a similar test to the one added to
    t7814 in patch 9, so I'd be OK with dropping patch 10 if it is too
    specific.

Thank you,
   Antonio

Antonio Ospite (10):
  submodule: add a print_config_from_gitmodules() helper
  submodule: factor out a config_set_in_gitmodules_file_gently function
  t7411: merge tests 5 and 6
  t7411: be nicer to future tests and really clean things up
  submodule--helper: add a new 'config' subcommand
  submodule: use the 'submodule--helper config' command
  t7506: clean up .gitmodules properly before setting up new scenario
  submodule: add a helper to check if it is safe to write to .gitmodules
  submodule: support reading .gitmodules when it's not in the working
    tree
  t/helper: add test-submodule-nested-repo-config

 Makefile                                     |   1 +
 builtin/grep.c                               |  17 ++-
 builtin/submodule--helper.c                  |  40 ++++++
 cache.h                                      |   2 +
 git-submodule.sh                             |  13 +-
 submodule-config.c                           |  68 ++++++++-
 submodule-config.h                           |   2 +
 submodule.c                                  |  28 +++-
 submodule.h                                  |   1 +
 t/helper/test-submodule-nested-repo-config.c |  30 ++++
 t/helper/test-tool.c                         |   1 +
 t/helper/test-tool.h                         |   1 +
 t/t7411-submodule-config.sh                  | 141 +++++++++++++++++--
 t/t7416-submodule-sparse-gitmodules.sh       |  78 ++++++++++
 t/t7506-status-submodule.sh                  |   3 +-
 t/t7814-grep-recurse-submodules.sh           |  16 +++
 16 files changed, 410 insertions(+), 32 deletions(-)
 create mode 100644 t/helper/test-submodule-nested-repo-config.c
 create mode 100755 t/t7416-submodule-sparse-gitmodules.sh

Comments

Antonio Ospite Oct. 6, 2018, 9:20 a.m. UTC | #1
On Fri,  5 Oct 2018 15:05:51 +0200
Antonio Ospite <ao2@ao2.it> wrote:

[...]
>  t/t7416-submodule-sparse-gitmodules.sh       |  78 ++++++++++
>  16 files changed, 410 insertions(+), 32 deletions(-)
>  create mode 100755 t/t7416-submodule-sparse-gitmodules.sh

I just saw that t7416 and t7417 have been added in the latest stable
release, I'll wait some days before sending a v7 which renames the newly
added test to t/t7418-submodule-sparse-gitmodules.sh

Thanks,
   Antonio
Junio C Hamano Oct. 25, 2018, 8:40 a.m. UTC | #2
Antonio Ospite <ao2@ao2.it> writes:

> this series teaches git to try and read the .gitmodules file from the
> index (:.gitmodules) or from the current branch (HEAD:.gitmodules) when
> the file is not readily available in the working tree.

What you said in [*1*] the discussion on [09/10] sounded like you
are preparing an update of the series, so the topic is marked as
"Expecting a reroll" in the recent "What's cooking" report.  At
least one topic now depends on the enhancement this topic makes, so
I'd like to know what the current status and ETA of the reroll would
be, in order to sort-of act as a traffic cop.

Your answer could even be "I have been too busy, and I do not think
an update will come for some time"---in other words, I do not mean
to tell you to drop other things and work on this instead.

If you are too busy, I can even see if other stakeholders
(e.g. Stefan, whose topic now depends on this series) can take it
over and update it after re-reading the discussion on the latest
round.

Thanks.


[Reference]

*1* http://public-inbox.org/git/20181010205645.e1529eff9099805029b1d6ef@ao2.it/
Antonio Ospite Oct. 25, 2018, 1:20 p.m. UTC | #3
On Thu, 25 Oct 2018 17:40:47 +0900
Junio C Hamano <gitster@pobox.com> wrote:

> Antonio Ospite <ao2@ao2.it> writes:
> 
> > this series teaches git to try and read the .gitmodules file from the
> > index (:.gitmodules) or from the current branch (HEAD:.gitmodules) when
> > the file is not readily available in the working tree.
> 
> What you said in [*1*] the discussion on [09/10] sounded like you
> are preparing an update of the series, so the topic is marked as
> "Expecting a reroll" in the recent "What's cooking" report.  At
> least one topic now depends on the enhancement this topic makes, so
> I'd like to know what the current status and ETA of the reroll would
> be, in order to sort-of act as a traffic cop.
> 

Hi Junio,

I can send a v7 later today.

It will only contain the improvements to
7416-submodule-sparse-gitmodules.sh as discussed in [*1*], it won't
contain changes to patch 8 as motivated in
https://public-inbox.org/git/20181008143709.dfcc845ab393c9caea66035e@ao2.it/

I will also leave patch 10 unchanged, improvements can be made in
follow-up patches.

BTW, what is the new topic which depends on this one?

Thank you,
   Antonio

> [Reference]
> 
> *1* http://public-inbox.org/git/20181010205645.e1529eff9099805029b1d6ef@ao2.it/