| Message ID | 20220203123724.47529-1-fs@gigacodes.de (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | gpg-interface: fix for gpgsm v2.3 | expand |
Fabian Stelzer <fs@gigacodes.de> writes: > gpgsm v2.3 changed some details about its output: > - instead of displaying `fingerprint:` for keys it will print `sha1 > fpr:` and `sha2 fpr:` > - some wording of errors has changed > - signing will omit an extra debug output line before the [GNUPG]: tag > > This change adjusts the gpgsm test prerequisite to work with v2.3 as > well by accepting `sha1 fpr:` as well as `fingerprint:` and allows both > variants of errors for unknown certs. OK, so the change is meant to add support for the new behaviour without deprecating/removing the support for the older one. Good. > Checking for successful signature creation will omit the leading NL in > its search string. I think this is to adjust for "will omit an extra debug output"; as long as we still ensure that the "[GNUPG:] SIG_CREATED" comes at the beginning of a line with some other means, I think that is a good change. > I am not a user of gpgsm but noticed that the GPGSM test prereq was disabled > on my runs so i investigated. The `fix` seems rather trivial and I tried to > test this as thorough as possible. I ran the test suite on machines > available to me (fedora35, centos8) and did a full CI run on github without > any issues. > But if you actually use gpgsm with git please give this a go and let me know > if I missed anything. Yup, thanks for a call for help. I am not gpgsm user either and testing by actual users is very much appreciated. > @@ -939,7 +939,7 @@ static int sign_buffer_gpg(struct strbuf *buffer, struct strbuf *signature, > signature, 1024, &gpg_status, 0); > sigchain_pop(SIGPIPE); > > - ret |= !strstr(gpg_status.buf, "\n[GNUPG:] SIG_CREATED "); > + ret |= !strstr(gpg_status.buf, "[GNUPG:] SIG_CREATED "); This I am not sure about. I understand that the intention is to allow this at the beginning of gpg_status.buf, but not to allow the substring appear in the middle of an otherwise unrelated line. I am afraid that the new check is a bit too lose for that. Totally untested but just to illustrate the idea... gpg-interface.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git c/gpg-interface.c w/gpg-interface.c index b52eb0e2e0..4238e60dfa 100644 --- c/gpg-interface.c +++ w/gpg-interface.c @@ -920,6 +920,7 @@ static int sign_buffer_gpg(struct strbuf *buffer, struct strbuf *signature, struct child_process gpg = CHILD_PROCESS_INIT; int ret; size_t bottom; + const char *cp; struct strbuf gpg_status = STRBUF_INIT; strvec_pushl(&gpg.args, @@ -939,7 +940,13 @@ static int sign_buffer_gpg(struct strbuf *buffer, struct strbuf *signature, signature, 1024, &gpg_status, 0); sigchain_pop(SIGPIPE); - ret |= !strstr(gpg_status.buf, "\n[GNUPG:] SIG_CREATED "); + for (cp = gpg_status.buf; + cp && (cp = strstr(cp, "[GNUPG:] SIG_CREATED ")); + cp++) { + if (cp == gpg_status.buf || cp[-1] == '\n') + break; /* found */ + } + ret |= !cp; strbuf_release(&gpg_status); if (ret) return error(_("gpg failed to sign the data"));
Hi Fabian, Fabian Stelzer wrote: > gpgsm v2.3 changed some details about its output: > - instead of displaying `fingerprint:` for keys it will print `sha1 > fpr:` and `sha2 fpr:` > - some wording of errors has changed > - signing will omit an extra debug output line before the [GNUPG]: tag Thanks for sending this. I noticed these as well, as Fedora started shipping gnupg-2.3 a few months back. I have been trying (and failing) to make time to submit (when I know I won't be too distracted to actually converse about them). The commits I made for the tests in Fedora are all here: https://src.fedoraproject.org/rpms/git/c/a7d2f7e53 I don't intend that as something anyone here should feel the need to chase down. But since they provide some additional context on the changes I made in the same area, it might help if anyone's curious. > diff --git a/gpg-interface.c b/gpg-interface.c > index b52eb0e2e0..299e7f588a 100644 > --- a/gpg-interface.c > +++ b/gpg-interface.c > @@ -939,7 +939,7 @@ static int sign_buffer_gpg(struct strbuf *buffer, struct strbuf *signature, > signature, 1024, &gpg_status, 0); > sigchain_pop(SIGPIPE); > > - ret |= !strstr(gpg_status.buf, "\n[GNUPG:] SIG_CREATED "); > + ret |= !strstr(gpg_status.buf, "[GNUPG:] SIG_CREATED "); > strbuf_release(&gpg_status); > if (ret) > return error(_("gpg failed to sign the data")); As Junio noted, this loosens the GPG parsing a good bit. I worried it could lead to security issues as well. The simple fix I made in Fedora was to add a newline to the gpg_status string buffer before adding the command output to it: diff --git a/gpg-interface.c b/gpg-interface.c index 3e7255a2a9..d179dfb3ab 100644 --- a/gpg-interface.c +++ b/gpg-interface.c @@ -859,6 +859,12 @@ static int sign_buffer_gpg(struct strbuf *buffer, struct strbuf *signature, bottom = signature->len; + /* + * Ensure gpg_status begins with a newline or we'll fail to match if + * the SIG_CREATED line is at the start of the gpg output. + */ + strbuf_addch(&gpg_status, '\n'); + /* * When the username signingkey is bad, program could be terminated * because gpg exits without reading and then write gets SIGPIPE. https://src.fedoraproject.org/rpms/git/blob/a7d2f7e53/f/0005-gpg-interface-match-SIG_CREATED-if-it-s-the-first-li.patch But that seemed like a bit of a hack. What I had queued up to submit for discussion (as I'm not sure that it isn't entirely horrible) used the string-list API to parse the gpg output: diff --git a/gpg-interface.c b/gpg-interface.c index b52eb0e2e0..e63ccdcb11 100644 --- a/gpg-interface.c +++ b/gpg-interface.c @@ -921,6 +921,7 @@ static int sign_buffer_gpg(struct strbuf *buffer, struct strbuf *signature, int ret; size_t bottom; struct strbuf gpg_status = STRBUF_INIT; + struct string_list lines = { .cmp = starts_with }; strvec_pushl(&gpg.args, use_format->program, @@ -939,8 +940,11 @@ static int sign_buffer_gpg(struct strbuf *buffer, struct strbuf *signature, signature, 1024, &gpg_status, 0); sigchain_pop(SIGPIPE); - ret |= !strstr(gpg_status.buf, "\n[GNUPG:] SIG_CREATED "); + string_list_split_in_place(&lines, gpg_status.buf, '\n', -1); + ret |= !unsorted_string_list_has_string(&lines, "[GNUPG:] SIG_CREATED "); strbuf_release(&gpg_status); + string_list_clear(&lines, 0); + if (ret) return error(_("gpg failed to sign the data")); That's the commit I was most in doubt about though, as my C "skills" are close to non-existant. I'd rather have something ugly and clear (like the `strbuf_addch(...)` above) than clever and wrong in gpg-interface.c. (To be clear, I mean "clever and wrong" in regard to my use of the string list API, not anyone else's code.) > diff --git a/t/lib-gpg.sh b/t/lib-gpg.sh > index 3e7ee1386a..6c2dd4b14b 100644 > --- a/t/lib-gpg.sh > +++ b/t/lib-gpg.sh > @@ -73,8 +73,8 @@ test_lazy_prereq GPGSM ' > --import "$TEST_DIRECTORY"/lib-gpg/gpgsm_cert.p12 && > > gpgsm --homedir "${GNUPGHOME}" -K | > - grep fingerprint: | > - cut -d" " -f4 | > + grep -E "(fingerprint|sha1 fpr):" | > + cut -d":" -f2- | tr -d " " | > tr -d "\\n" >"${GNUPGHOME}/trustlist.txt" && I think this whole thing can (and should) be simplified by using gpg's --with-colons output which is intended to be machine parsable. If we'd been using that previously, we wouldn't need to make any further changes here. I think we're making our lives difficult by screen scraping here wher we don't need to do so. The change I made for the Fedora package to fix this does it like this: --- a/t/lib-gpg.sh +++ b/t/lib-gpg.sh @@ -72,12 +72,10 @@ test_lazy_prereq GPGSM ' --passphrase-fd 0 --pinentry-mode loopback \ --import "$TEST_DIRECTORY"/lib-gpg/gpgsm_cert.p12 && - gpgsm --homedir "${GNUPGHOME}" -K | - grep fingerprint: | - cut -d" " -f4 | - tr -d "\\n" >"${GNUPGHOME}/trustlist.txt" && + gpgsm --homedir "${GNUPGHOME}" -K --with-colons | + awk -F ":" "/^fpr:/ {printf \"%s S relax\\n\", \$10}" \ + >"${GNUPGHOME}/trustlist.txt" && - echo " S relax" >>"${GNUPGHOME}/trustlist.txt" && echo hello | gpgsm --homedir "${GNUPGHOME}" >/dev/null \ -u committer@example.com -o /dev/null --sign - ' With a commit message: https://src.fedoraproject.org/rpms/git/blob/a7d2f7e53/f/0001-t-lib-gpg-use-with-colons-when-parsing-gpgsm-output.patch I was hoping to submit that small series in the next day or two, while I've got a few days away from $work. If doing it that way is appealing, I can submit them. But only if that looks like a reasonable improvement to you and others. > echo " S relax" >>"${GNUPGHOME}/trustlist.txt" && > diff --git a/t/t4202-log.sh b/t/t4202-log.sh > index 5049559861..08556493ce 100755 > --- a/t/t4202-log.sh > +++ b/t/t4202-log.sh > @@ -1931,7 +1931,10 @@ test_expect_success GPGSM 'log --graph --show-signature for merged tag x509 miss > git merge --no-ff -m msg signed_tag_x509_nokey && > GNUPGHOME=. git log --graph --show-signature -n1 plain-x509-nokey >actual && > grep "^|\\\ merged tag" actual && > - grep "^| | gpgsm: certificate not found" actual > + ( > + grep "^| | gpgsm: certificate not found" actual || > + grep "^| | gpgsm: failed to find the certificate: Not found" actual > + ) > ' > > test_expect_success GPGSM 'log --graph --show-signature for merged tag x509 bad signature' ' Can we make this simpler by adjusting the grep pattern? It's certainly a slight trade-off in ease of reading, but it saves a subshell and an extra command: - grep "^| | gpgsm: certificate not found" actual + grep -Ei "^| | gpgsm:( failed to find the)? certificate:? not found" actual I did that in a separate patch: https://src.fedoraproject.org/rpms/git/blob/a7d2f7e53/f/0004-t4202-match-gpgsm-output-from-GnuPG-2.3.patch IMO, this is a bug in gnupg-2.3. I submitted a patch to resolve it back in November, but have not gotten any response as yet. :( https://lists.gnupg.org/pipermail/gnupg-devel/2021-November/034991.html Not that it will preclude us from having to fix this for the test suite, but it's worth noting why the change is needed (and when it will no longer be relevant -- if/when we don't care to support the few early gnupg-2.3.x releases). Thanks,
Todd Zullinger <tmz@pobox.com> writes: > - ret |= !strstr(gpg_status.buf, "\n[GNUPG:] SIG_CREATED "); > + string_list_split_in_place(&lines, gpg_status.buf, '\n', -1); > + ret |= !unsorted_string_list_has_string(&lines, "[GNUPG:] SIG_CREATED "); Is "SIG_CREATED " supposed to be at the end of that line? I thought that has_string() asks for an exact match, and unfortunately(?) there is not the string_list_has_string_that_has_this_prefix() function. So...
Junio C Hamano wrote: > Todd Zullinger <tmz@pobox.com> writes: > >> - ret |= !strstr(gpg_status.buf, "\n[GNUPG:] SIG_CREATED "); >> + string_list_split_in_place(&lines, gpg_status.buf, '\n', -1); >> + ret |= !unsorted_string_list_has_string(&lines, "[GNUPG:] SIG_CREATED "); > > Is "SIG_CREATED " supposed to be at the end of that line? I thought > that has_string() asks for an exact match, and unfortunately(?) > there is not the string_list_has_string_that_has_this_prefix() > function. So... By default, yes. The string_list struct uses strcmp() if no cmp function is given. That's why the previous chunk has: struct string_list lines = { .cmp = starts_with }; There aren't any similar uses in the code, which is just one of the reasons I wasn't confident that it was a good idea or even a good implementation.
Todd Zullinger <tmz@pobox.com> writes: > By default, yes. The string_list struct uses strcmp() if no > cmp function is given. That's why the previous chunk has: > > struct string_list lines = { .cmp = starts_with }; Ahh, I missed that part. Sounds correct to me, then. Splitting only to iterate over these lines sounds wasteful to me, though, since we do not need access to these lines only one at a time and there is no need to keep an array of them. Thanks.
On 03.02.2022 15:01, Todd Zullinger wrote: >Hi Fabian, > >Fabian Stelzer wrote: >> gpgsm v2.3 changed some details about its output: >> - instead of displaying `fingerprint:` for keys it will print `sha1 >> fpr:` and `sha2 fpr:` >> - some wording of errors has changed >> - signing will omit an extra debug output line before the [GNUPG]: tag > >Thanks for sending this. I noticed these as well, as Fedora >started shipping gnupg-2.3 a few months back. I have been >trying (and failing) to make time to submit (when I know I >won't be too distracted to actually converse about them). >The commits I made for the tests in Fedora are all here: > > https://src.fedoraproject.org/rpms/git/c/a7d2f7e53 > >I don't intend that as something anyone here should feel the >need to chase down. But since they provide some additional >context on the changes I made in the same area, it might >help if anyone's curious. > >> diff --git a/gpg-interface.c b/gpg-interface.c >> index b52eb0e2e0..299e7f588a 100644 >> --- a/gpg-interface.c >> +++ b/gpg-interface.c >> @@ -939,7 +939,7 @@ static int sign_buffer_gpg(struct strbuf *buffer, struct strbuf *signature, >> signature, 1024, &gpg_status, 0); >> sigchain_pop(SIGPIPE); >> >> - ret |= !strstr(gpg_status.buf, "\n[GNUPG:] SIG_CREATED "); >> + ret |= !strstr(gpg_status.buf, "[GNUPG:] SIG_CREATED "); >> strbuf_release(&gpg_status); >> if (ret) >> return error(_("gpg failed to sign the data")); > >As Junio noted, this loosens the GPG parsing a good bit. I >worried it could lead to security issues as well. The >simple fix I made in Fedora was to add a newline to the >gpg_status string buffer before adding the command output to >it: > > diff --git a/gpg-interface.c b/gpg-interface.c > index 3e7255a2a9..d179dfb3ab 100644 > --- a/gpg-interface.c > +++ b/gpg-interface.c > @@ -859,6 +859,12 @@ static int sign_buffer_gpg(struct strbuf *buffer, struct strbuf *signature, > > bottom = signature->len; > > + /* > + * Ensure gpg_status begins with a newline or we'll fail to match if > + * the SIG_CREATED line is at the start of the gpg output. > + */ > + strbuf_addch(&gpg_status, '\n'); > + > /* > * When the username signingkey is bad, program could be terminated > * because gpg exits without reading and then write gets SIGPIPE. > >https://src.fedoraproject.org/rpms/git/blob/a7d2f7e53/f/0005-gpg-interface-match-SIG_CREATED-if-it-s-the-first-li.patch > >But that seemed like a bit of a hack. What I had queued up >to submit for discussion (as I'm not sure that it isn't >entirely horrible) used the string-list API to parse the gpg >output: > > diff --git a/gpg-interface.c b/gpg-interface.c > index b52eb0e2e0..e63ccdcb11 100644 > --- a/gpg-interface.c > +++ b/gpg-interface.c > @@ -921,6 +921,7 @@ static int sign_buffer_gpg(struct strbuf *buffer, struct strbuf *signature, > int ret; > size_t bottom; > struct strbuf gpg_status = STRBUF_INIT; > + struct string_list lines = { .cmp = starts_with }; > > strvec_pushl(&gpg.args, > use_format->program, > @@ -939,8 +940,11 @@ static int sign_buffer_gpg(struct strbuf *buffer, struct strbuf *signature, > signature, 1024, &gpg_status, 0); > sigchain_pop(SIGPIPE); > > - ret |= !strstr(gpg_status.buf, "\n[GNUPG:] SIG_CREATED "); > + string_list_split_in_place(&lines, gpg_status.buf, '\n', -1); > + ret |= !unsorted_string_list_has_string(&lines, "[GNUPG:] SIG_CREATED "); > strbuf_release(&gpg_status); > + string_list_clear(&lines, 0); > + > if (ret) > return error(_("gpg failed to sign the data")); > >That's the commit I was most in doubt about though, as my C >"skills" are close to non-existant. I'd rather have >something ugly and clear (like the `strbuf_addch(...)` >above) than clever and wrong in gpg-interface.c. > >(To be clear, I mean "clever and wrong" in regard to my use >of the string list API, not anyone else's code.) string_list_split seems a bit like overkill. My first thought was actually sth like: cp = strstr(gpg_status.buf, "[GNUPG]: SIG_CREATED"); if (cp == gpg_status.buf || --cp == '\n') // found But that would fail in case the string came up before the actual success message at the beginning of a line. So Junios variant of using the for() loop is more robust and would normally find the correct string on its first iteration anyway. > >> diff --git a/t/lib-gpg.sh b/t/lib-gpg.sh >> index 3e7ee1386a..6c2dd4b14b 100644 >> --- a/t/lib-gpg.sh >> +++ b/t/lib-gpg.sh >> @@ -73,8 +73,8 @@ test_lazy_prereq GPGSM ' >> --import "$TEST_DIRECTORY"/lib-gpg/gpgsm_cert.p12 && >> >> gpgsm --homedir "${GNUPGHOME}" -K | >> - grep fingerprint: | >> - cut -d" " -f4 | >> + grep -E "(fingerprint|sha1 fpr):" | >> + cut -d":" -f2- | tr -d " " | >> tr -d "\\n" >"${GNUPGHOME}/trustlist.txt" && > >I think this whole thing can (and should) be simplified by >using gpg's --with-colons output which is intended to be >machine parsable. I looked for sth like this but gpgs --help does not list it so i didn't dig deeper. I've checked the blame and it seems like this was introduced >19 years ago. So i guess we can probably use this ^^ > >If we'd been using that previously, we wouldn't need to make >any further changes here. > >I think we're making our lives difficult by screen scraping >here wher we don't need to do so. > >The change I made for the Fedora package to fix this does >it like this: > > --- a/t/lib-gpg.sh > +++ b/t/lib-gpg.sh > @@ -72,12 +72,10 @@ test_lazy_prereq GPGSM ' > --passphrase-fd 0 --pinentry-mode loopback \ > --import "$TEST_DIRECTORY"/lib-gpg/gpgsm_cert.p12 && > > - gpgsm --homedir "${GNUPGHOME}" -K | > - grep fingerprint: | > - cut -d" " -f4 | > - tr -d "\\n" >"${GNUPGHOME}/trustlist.txt" && > + gpgsm --homedir "${GNUPGHOME}" -K --with-colons | > + awk -F ":" "/^fpr:/ {printf \"%s S relax\\n\", \$10}" \ > + >"${GNUPGHOME}/trustlist.txt" && This does not quite work for me. It will add the fingerprint without the colons into the trustlist which is not valid :/ It would need sth like: + gpgsm --with-colons --homedir "${GNUPGHOME}" -K | + awk -F ":" "/^(fpr|fingerprint):/ {gsub(/.{2}/, \"&:\", \$10); printf \"%s S relax\\n\", substr(\$10, 1, length(\$10)-1)}" \ + >"${GNUPGHOME}/trustlist.txt" && which looks needlessly complicated. There is probably some better way to do this with/without awk. > > - echo " S relax" >>"${GNUPGHOME}/trustlist.txt" && > echo hello | gpgsm --homedir "${GNUPGHOME}" >/dev/null \ > -u committer@example.com -o /dev/null --sign - > ' > >With a commit message: > > https://src.fedoraproject.org/rpms/git/blob/a7d2f7e53/f/0001-t-lib-gpg-use-with-colons-when-parsing-gpgsm-output.patch > >I was hoping to submit that small series in the next day or >two, while I've got a few days away from $work. If doing it >that way is appealing, I can submit them. But only if that >looks like a reasonable improvement to you and others. > >> echo " S relax" >>"${GNUPGHOME}/trustlist.txt" && >> diff --git a/t/t4202-log.sh b/t/t4202-log.sh >> index 5049559861..08556493ce 100755 >> --- a/t/t4202-log.sh >> +++ b/t/t4202-log.sh >> @@ -1931,7 +1931,10 @@ test_expect_success GPGSM 'log --graph --show-signature for merged tag x509 miss >> git merge --no-ff -m msg signed_tag_x509_nokey && >> GNUPGHOME=. git log --graph --show-signature -n1 plain-x509-nokey >actual && >> grep "^|\\\ merged tag" actual && >> - grep "^| | gpgsm: certificate not found" actual >> + ( >> + grep "^| | gpgsm: certificate not found" actual || >> + grep "^| | gpgsm: failed to find the certificate: Not found" actual >> + ) >> ' >> >> test_expect_success GPGSM 'log --graph --show-signature for merged tag x509 bad signature' ' > >Can we make this simpler by adjusting the grep pattern? >It's certainly a slight trade-off in ease of reading, but it >saves a subshell and an extra command: > > - grep "^| | gpgsm: certificate not found" actual > + grep -Ei "^| | gpgsm:( failed to find the)? certificate:? not found" actual Thanks, will do. > >I did that in a separate patch: > > https://src.fedoraproject.org/rpms/git/blob/a7d2f7e53/f/0004-t4202-match-gpgsm-output-from-GnuPG-2.3.patch > >IMO, this is a bug in gnupg-2.3. I submitted a patch to >resolve it back in November, but have not gotten any >response as yet. :( > > https://lists.gnupg.org/pipermail/gnupg-devel/2021-November/034991.html > >Not that it will preclude us from having to fix this for the >test suite, but it's worth noting why the change is needed >(and when it will no longer be relevant -- if/when we don't >care to support the few early gnupg-2.3.x releases). > >Thanks, > >-- >Todd
Hi Fabien, Fabian Stelzer wrote: > On 03.02.2022 15:01, Todd Zullinger wrote: >> (To be clear, I mean "clever and wrong" in regard to my use >> of the string list API, not anyone else's code.) > > string_list_split seems a bit like overkill. I have little doubt that the string_list_split() method is far from ideal. :) > I looked for sth like this but gpgs --help does not list it so i didn't dig > deeper. I've checked the blame and it seems like this was introduced >19 > years ago. So i guess we can probably use this ^^ Indeed, the --with-colons output goes much further back in the GnuPG history than Git will ever have to care about. >> --- a/t/lib-gpg.sh >> +++ b/t/lib-gpg.sh >> @@ -72,12 +72,10 @@ test_lazy_prereq GPGSM ' >> --passphrase-fd 0 --pinentry-mode loopback \ >> --import "$TEST_DIRECTORY"/lib-gpg/gpgsm_cert.p12 && >> >> - gpgsm --homedir "${GNUPGHOME}" -K | >> - grep fingerprint: | >> - cut -d" " -f4 | >> - tr -d "\\n" >"${GNUPGHOME}/trustlist.txt" && >> + gpgsm --homedir "${GNUPGHOME}" -K --with-colons | >> + awk -F ":" "/^fpr:/ {printf \"%s S relax\\n\", \$10}" \ >> + >"${GNUPGHOME}/trustlist.txt" && > > This does not quite work for me. It will add the fingerprint without the > colons into the trustlist which is not valid :/ The colons are optional, and have been documented as such since cb1840720 ((Agent Configuration): New section., 2005-04-20). The text in the gpg-agent docs from GnuPG 2.2 say: Colons may optionally be used to separate the bytes of a fingerprint; this enables cutting and pasting the fingerprint from a key listing output. Source: https://dev.gnupg.org/source/gnupg/browse/STABLE-BRANCH-2-2/doc/gpg-agent.texi;8021fe7670c79d5c698ec3fb600b02a9e5afb415$756?as=source&blame=off How did it fail for you? It passes all the tests when I've run it against Fedora and RHEL-based hosts. If it's flaky on other systems, that would put a damper on doing it this way. Though it _should_ work. [Note to myself] We don't just generate the key data, trustlist, etc. and store it in t/lib-gpg like we do with some other files per b41a36e635 (tests: create gpg homedir on the fly, 2014-12-12). That was because the gnupg home directory layout changed a bit between 2.0 and 2.1. Thanks,
On 07.02.2022 11:38, Todd Zullinger wrote: >Hi Fabien, > >Fabian Stelzer wrote: >> On 03.02.2022 15:01, Todd Zullinger wrote: >>> (To be clear, I mean "clever and wrong" in regard to my use >>> of the string list API, not anyone else's code.) >> >> string_list_split seems a bit like overkill. > >I have little doubt that the string_list_split() method is >far from ideal. :) > >> I looked for sth like this but gpgs --help does not list it so i didn't dig >> deeper. I've checked the blame and it seems like this was introduced >19 >> years ago. So i guess we can probably use this ^^ > >Indeed, the --with-colons output goes much further back in >the GnuPG history than Git will ever have to care about. > >>> --- a/t/lib-gpg.sh >>> +++ b/t/lib-gpg.sh >>> @@ -72,12 +72,10 @@ test_lazy_prereq GPGSM ' >>> --passphrase-fd 0 --pinentry-mode loopback \ >>> --import "$TEST_DIRECTORY"/lib-gpg/gpgsm_cert.p12 && >>> >>> - gpgsm --homedir "${GNUPGHOME}" -K | >>> - grep fingerprint: | >>> - cut -d" " -f4 | >>> - tr -d "\\n" >"${GNUPGHOME}/trustlist.txt" && >>> + gpgsm --homedir "${GNUPGHOME}" -K --with-colons | >>> + awk -F ":" "/^fpr:/ {printf \"%s S relax\\n\", \$10}" \ >>> + >"${GNUPGHOME}/trustlist.txt" && >> >> This does not quite work for me. It will add the fingerprint without the >> colons into the trustlist which is not valid :/ > >The colons are optional, and have been documented as such >since cb1840720 ((Agent Configuration): New section., >2005-04-20). The text in the gpg-agent docs from GnuPG 2.2 >say: > > Colons may optionally be used to separate the bytes of a > fingerprint; this enables cutting and pasting the > fingerprint from a key listing output. > >Source: https://dev.gnupg.org/source/gnupg/browse/STABLE-BRANCH-2-2/doc/gpg-agent.texi;8021fe7670c79d5c698ec3fb600b02a9e5afb415$756?as=source&blame=off > >How did it fail for you? It passes all the tests when I've >run it against Fedora and RHEL-based hosts. If it's flaky >on other systems, that would put a damper on doing it this >way. Though it _should_ work. Sorry for the delays, I'm a bit busy with other things at the moment. I did get an interactive popup asking if I would like to trust the key when I ran the t4202 test. This never happened with the old variant. > >[Note to myself] We don't just generate the key data, >trustlist, etc. and store it in t/lib-gpg like we do with >some other files per b41a36e635 (tests: create gpg homedir >on the fly, 2014-12-12). That was because the gnupg home >directory layout changed a bit between 2.0 and 2.1. > >Thanks, > >-- >Todd
Fabian Stelzer wrote: > On 07.02.2022 11:38, Todd Zullinger wrote: >> How did it fail for you? It passes all the tests when I've >> run it against Fedora and RHEL-based hosts. If it's flaky >> on other systems, that would put a damper on doing it this >> way. Though it _should_ work. > > Sorry for the delays, I'm a bit busy with other things at the moment. No apologies needed. This is something I worked on back in November and had yet to send to the list, so I'm the last person to rush another. :) > I did get an interactive popup asking if I would like to > trust the key when I ran the t4202 test. This never > happened with the old variant. Interesting. I do have a patch in my gnupg-2.3 series to reload the gpg agent after changing the trustlist, as the changes were not picked up prior to that. In my case, I was running the tests in an environment where gpg could not prompt me. (It also seems like we should try harder to have the test suite reject such prompts). --- 8< --- Subject: [PATCH] t/lib-gpg: reload gpg components after updating trustlist With gpgsm from gnupg-2.3, the changes to the trustlist.txt do not appear to be picked up without refreshing the gpg-agent. Use the 'all' keyword to reload all of the gpg components. The scdaemon is started as a child of gpg-agent, for example. We used to have a --kill at this spot, but I removed it in 2e285e7803 (t/lib-gpg: drop redundant killing of gpg-agent, 2019-02-07). It seems like it might be necessary (again) for 2.3. Signed-off-by: Todd Zullinger <tmz@pobox.com> --- Notes: An alternative to doing this dance with the trustlist.txt and having to kill and/or reload the gpg-agent to pick up the change right after the import in each test might be to make this part of the steps used when adding/updating/removing certificates in t/lib-gpg. If not as a one-time affair when a cert is added/update/removed, then perhaps as a step taken by/in t/lib-gpg.sh only once. It could populate a gpghome to be copied into the trash dir for each test which used gpg/gpgsm. I haven't measured the effect of the extra reload precisely, but I'm sure it's not free. (For what it's worth, it didn't add any noticeable amount of time to the full builds/test runs I made while working on this, so it's a seemingly small cost, at least.) Also, hello Henning, Way back, in February 2019¹, when I submitted 2e285e7803 to remove the "redundant" killing of the gpg-agent, you said: > Killing the agent once should be enough, i remember manually killing > it many times as i was looking for a way to generate certs and trust > (configure gpgsm for the test). That is probably why i copied it over > in the first place. As I wrote this patch to partially restore the gpg-agent killing (now just a reload), I thought this might have been the sort of issue that you hit while testing. It could be unrelated, but it sounds quite similar to what I found with gnupg-2.3 when trying to get it to pick up the trustlist.txt changes. I thought you might at least enjoy seeing it come back around. :) ¹ <20190208093324.7b17f270@md1za8fc.ad001.siemens.net> t/lib-gpg.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/t/lib-gpg.sh b/t/lib-gpg.sh index 6bc083ca77..38e2c0f4fb 100644 --- a/t/lib-gpg.sh +++ b/t/lib-gpg.sh @@ -75,6 +75,7 @@ test_lazy_prereq GPGSM ' gpgsm --homedir "${GNUPGHOME}" -K --with-colons | awk -F ":" "/^fpr:/ {printf \"%s S relax\\n\", \$10}" \ >"${GNUPGHOME}/trustlist.txt" && + (gpgconf --reload all || : ) && echo hello | gpgsm --homedir "${GNUPGHOME}" >/dev/null \ -u committer@example.com -o /dev/null --sign - --- 8< --- I have another patch which changes the earlier gpgconf call which kills gpg-agent to kill all gpg daemons, as there are some others which could potentially interfere with the tests: --- 8< --- Subject: [PATCH] t/lib-gpg: kill all gpg components, not just gpg-agent The gpg-agent is one of several processes that newer releases of GnuPG start automatically. Issue a kill to each of them to ensure they do not affect separate tests. (Yes, the separate GNUPGHOME should do that already. If we find that is case, we could drop the --kill entirely.) In terms of compatibility, the 'all' keyword was added to the --kill & --reload options in GnuPG 2.1.18. Debian and RHEL are often used as indicators of how a change might affect older systems we often try to support. - Debian Strech (old old stable), which has limited security support until June 2022, has GnuPG 2.1.18 (or 2.2.x in backports). - CentOS/RHEL 7, which is supported until June 2024, has GnuPG 2.0.22, which lacks the --kill option, so the change won't have any impact. Signed-off-by: Todd Zullinger <tmz@pobox.com> --- t/lib-gpg.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/lib-gpg.sh b/t/lib-gpg.sh index d675698a2d..2bb309a8c1 100644 --- a/t/lib-gpg.sh +++ b/t/lib-gpg.sh @@ -40,7 +40,7 @@ test_lazy_prereq GPG ' # > lib-gpg/ownertrust mkdir "$GNUPGHOME" && chmod 0700 "$GNUPGHOME" && - (gpgconf --kill gpg-agent || : ) && + (gpgconf --kill all || : ) && gpg --homedir "${GNUPGHOME}" --import \ "$TEST_DIRECTORY"/lib-gpg/keyring.gpg && gpg --homedir "${GNUPGHOME}" --import-ownertrust \ --- 8< --- I have the series in the gpg-misc-fixes branch: https://github.com/tmzullinger/git/commits/gpg-misc-fixes (That series does differ in that it has `string_list_split()` instead of the simpler `strbuf_addch(&gpg_status, '\n');` to fix the gpgsm parsing.) Iff you think it would be useful or helpful, I can post that series. Thanks,
On 09.02.2022 11:20, Todd Zullinger wrote: >Fabian Stelzer wrote: >> On 07.02.2022 11:38, Todd Zullinger wrote: >>> How did it fail for you? It passes all the tests when I've >>> run it against Fedora and RHEL-based hosts. If it's flaky >>> on other systems, that would put a damper on doing it this >>> way. Though it _should_ work. >> >> Sorry for the delays, I'm a bit busy with other things at the moment. > >No apologies needed. This is something I worked on back in >November and had yet to send to the list, so I'm the last >person to rush another. :) > >> I did get an interactive popup asking if I would like to >> trust the key when I ran the t4202 test. This never >> happened with the old variant. > >Interesting. I do have a patch in my gnupg-2.3 series to >reload the gpg agent after changing the trustlist, as the >changes were not picked up prior to that. In my case, I was >running the tests in an environment where gpg could not >prompt me. (It also seems like we should try harder to have >the test suite reject such prompts). > Yes, gpg-agent in general can be problematic for the tests. I'm not familiar enough with gpg but I don't know if we can get by without it? >--- 8< --- >Subject: [PATCH] t/lib-gpg: reload gpg components after updating trustlist > >With gpgsm from gnupg-2.3, the changes to the trustlist.txt do not >appear to be picked up without refreshing the gpg-agent. Use the 'all' >keyword to reload all of the gpg components. The scdaemon is started as >a child of gpg-agent, for example. > >We used to have a --kill at this spot, but I removed it in 2e285e7803 >(t/lib-gpg: drop redundant killing of gpg-agent, 2019-02-07). It seems >like it might be necessary (again) for 2.3. > >Signed-off-by: Todd Zullinger <tmz@pobox.com> >--- > >Notes: > An alternative to doing this dance with the trustlist.txt and having to > kill and/or reload the gpg-agent to pick up the change right after the > import in each test might be to make this part of the steps used when > adding/updating/removing certificates in t/lib-gpg. > > If not as a one-time affair when a cert is added/update/removed, then > perhaps as a step taken by/in t/lib-gpg.sh only once. It could populate > a gpghome to be copied into the trash dir for each test which used > gpg/gpgsm. I haven't measured the effect of the extra reload precisely, > but I'm sure it's not free. > > (For what it's worth, it didn't add any noticeable amount of time to the > full builds/test runs I made while working on this, so it's a seemingly > small cost, at least.) > > Also, hello Henning, > > Way back, in February 2019¹, when I submitted 2e285e7803 to remove the > "redundant" killing of the gpg-agent, you said: > > > Killing the agent once should be enough, i remember manually killing > > it many times as i was looking for a way to generate certs and trust > > (configure gpgsm for the test). That is probably why i copied it over > > in the first place. > > As I wrote this patch to partially restore the gpg-agent killing (now > just a reload), I thought this might have been the sort of issue that > you hit while testing. > > It could be unrelated, but it sounds quite similar to what I found with > gnupg-2.3 when trying to get it to pick up the trustlist.txt changes. I > thought you might at least enjoy seeing it come back around. :) > > ¹ <20190208093324.7b17f270@md1za8fc.ad001.siemens.net> > > t/lib-gpg.sh | 1 + > 1 file changed, 1 insertion(+) > >diff --git a/t/lib-gpg.sh b/t/lib-gpg.sh >index 6bc083ca77..38e2c0f4fb 100644 >--- a/t/lib-gpg.sh >+++ b/t/lib-gpg.sh >@@ -75,6 +75,7 @@ test_lazy_prereq GPGSM ' > gpgsm --homedir "${GNUPGHOME}" -K --with-colons | > awk -F ":" "/^fpr:/ {printf \"%s S relax\\n\", \$10}" \ > >"${GNUPGHOME}/trustlist.txt" && >+ (gpgconf --reload all || : ) && > > echo hello | gpgsm --homedir "${GNUPGHOME}" >/dev/null \ > -u committer@example.com -o /dev/null --sign - > >--- 8< --- This patch fixes it for me. > >I have another patch which changes the earlier gpgconf call >which kills gpg-agent to kill all gpg daemons, as there are >some others which could potentially interfere with the >tests: > >--- 8< --- >Subject: [PATCH] t/lib-gpg: kill all gpg components, not just gpg-agent > >The gpg-agent is one of several processes that newer releases of GnuPG >start automatically. Issue a kill to each of them to ensure they do not >affect separate tests. (Yes, the separate GNUPGHOME should do that >already. If we find that is case, we could drop the --kill entirely.) > >In terms of compatibility, the 'all' keyword was added to the --kill & >--reload options in GnuPG 2.1.18. Debian and RHEL are often used as >indicators of how a change might affect older systems we often try to >support. > > - Debian Strech (old old stable), which has limited security support > until June 2022, has GnuPG 2.1.18 (or 2.2.x in backports). > > - CentOS/RHEL 7, which is supported until June 2024, has GnuPG > 2.0.22, which lacks the --kill option, so the change won't have > any impact. > >Signed-off-by: Todd Zullinger <tmz@pobox.com> >--- > t/lib-gpg.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > >diff --git a/t/lib-gpg.sh b/t/lib-gpg.sh >index d675698a2d..2bb309a8c1 100644 >--- a/t/lib-gpg.sh >+++ b/t/lib-gpg.sh >@@ -40,7 +40,7 @@ test_lazy_prereq GPG ' > # > lib-gpg/ownertrust > mkdir "$GNUPGHOME" && > chmod 0700 "$GNUPGHOME" && >- (gpgconf --kill gpg-agent || : ) && >+ (gpgconf --kill all || : ) && > gpg --homedir "${GNUPGHOME}" --import \ > "$TEST_DIRECTORY"/lib-gpg/keyring.gpg && > gpg --homedir "${GNUPGHOME}" --import-ownertrust \ >--- 8< --- > >I have the series in the gpg-misc-fixes branch: > > https://github.com/tmzullinger/git/commits/gpg-misc-fixes > >(That series does differ in that it has `string_list_split()` >instead of the simpler `strbuf_addch(&gpg_status, '\n');` to >fix the gpgsm parsing.) > >Iff you think it would be useful or helpful, I can post that >series. I have prepared the patch with the simple strstr() matching I can post in a bit. I would add your two gpg test lib patches to it if thats ok? Thanks > >Thanks, > >-- >Todd
Fabian Stelzer wrote: > On 09.02.2022 11:20, Todd Zullinger wrote: >> Interesting. I do have a patch in my gnupg-2.3 series to >> reload the gpg agent after changing the trustlist, as the >> changes were not picked up prior to that. In my case, I was >> running the tests in an environment where gpg could not >> prompt me. (It also seems like we should try harder to have >> the test suite reject such prompts). >> > > Yes, gpg-agent in general can be problematic for the tests. I'm not familiar > enough with gpg but I don't know if we can get by without it? With modern gnupg, the secret keyring access is handled by gpg-agent. So it's no longer optional, which is mildly unfortunate for automated tests.. >> diff --git a/t/lib-gpg.sh b/t/lib-gpg.sh >> index 6bc083ca77..38e2c0f4fb 100644 >> --- a/t/lib-gpg.sh >> +++ b/t/lib-gpg.sh >> @@ -75,6 +75,7 @@ test_lazy_prereq GPGSM ' >> gpgsm --homedir "${GNUPGHOME}" -K --with-colons | >> awk -F ":" "/^fpr:/ {printf \"%s S relax\\n\", \$10}" \ >> >"${GNUPGHOME}/trustlist.txt" && >> + (gpgconf --reload all || : ) && >> >> echo hello | gpgsm --homedir "${GNUPGHOME}" >/dev/null \ >> -u committer@example.com -o /dev/null --sign - >> >> --- 8< --- > > This patch fixes it for me. Excellent. > I have prepared the patch with the simple strstr() matching I can post in a > bit. I would add your two gpg test lib patches to it if thats ok? Absolutely. Thank you for working on this and pulling it together. Cheers,
diff --git a/gpg-interface.c b/gpg-interface.c index b52eb0e2e0..299e7f588a 100644 --- a/gpg-interface.c +++ b/gpg-interface.c @@ -939,7 +939,7 @@ static int sign_buffer_gpg(struct strbuf *buffer, struct strbuf *signature, signature, 1024, &gpg_status, 0); sigchain_pop(SIGPIPE); - ret |= !strstr(gpg_status.buf, "\n[GNUPG:] SIG_CREATED "); + ret |= !strstr(gpg_status.buf, "[GNUPG:] SIG_CREATED "); strbuf_release(&gpg_status); if (ret) return error(_("gpg failed to sign the data")); diff --git a/t/lib-gpg.sh b/t/lib-gpg.sh index 3e7ee1386a..6c2dd4b14b 100644 --- a/t/lib-gpg.sh +++ b/t/lib-gpg.sh @@ -73,8 +73,8 @@ test_lazy_prereq GPGSM ' --import "$TEST_DIRECTORY"/lib-gpg/gpgsm_cert.p12 && gpgsm --homedir "${GNUPGHOME}" -K | - grep fingerprint: | - cut -d" " -f4 | + grep -E "(fingerprint|sha1 fpr):" | + cut -d":" -f2- | tr -d " " | tr -d "\\n" >"${GNUPGHOME}/trustlist.txt" && echo " S relax" >>"${GNUPGHOME}/trustlist.txt" && diff --git a/t/t4202-log.sh b/t/t4202-log.sh index 5049559861..08556493ce 100755 --- a/t/t4202-log.sh +++ b/t/t4202-log.sh @@ -1931,7 +1931,10 @@ test_expect_success GPGSM 'log --graph --show-signature for merged tag x509 miss git merge --no-ff -m msg signed_tag_x509_nokey && GNUPGHOME=. git log --graph --show-signature -n1 plain-x509-nokey >actual && grep "^|\\\ merged tag" actual && - grep "^| | gpgsm: certificate not found" actual + ( + grep "^| | gpgsm: certificate not found" actual || + grep "^| | gpgsm: failed to find the certificate: Not found" actual + ) ' test_expect_success GPGSM 'log --graph --show-signature for merged tag x509 bad signature' '