| Message ID | xmqqo8ds8k6r.fsf_-_@gitster.g (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | 768071c55445e087f32af8e794a2ca1aa5f61e47 |
| Headers | show |
| Series | [v2] CodingGuidelines: explicitly allow "local" for test scripts | expand |
On Mon, May 03 2021, Junio C Hamano wrote: > 01d3a526 (t0000: check whether the shell supports the "local" > keyword, 2017-10-26) raised a test balloon to see if those who build > and test Git use a platform with a shell that lacks support for the > "local" keyword. After two years, 7f0b5908 (t0000: reword comments > for "local" test, 2019-08-08) documented that "local" keyword, even > though is outside POSIX, is allowed in our test scripts. > > Let's write it in the CodingGuidelines, too. It might be tempting > to allow it in scripted Porcelains (we have avoided getting them > contaminiated by "local" so far), but they are on their way out and > getting rewritten in C. > > Signed-off-by: Junio C Hamano <gitster@pobox.com> > --- > Documentation/CodingGuidelines | 5 +++++ > 1 file changed, 5 insertions(+) > > diff --git a/Documentation/CodingGuidelines b/Documentation/CodingGuidelines > index 45465bc0c9..ea70676a30 100644 > --- a/Documentation/CodingGuidelines > +++ b/Documentation/CodingGuidelines > @@ -175,6 +175,11 @@ For shell scripts specifically (not exhaustive): > > does not have such a problem. > > + - Even though "local" is not part of POSIX, we make heavy use of it > + in our test suite. We do not use it in scripted Porcelains, and > + hopefully nobody starts using "local" before they are reimplemented > + in C ;-) > + Is there any portability reason to avoid "local" in the porcelains? I don't have any plans for using it, but I don't see why we'd explicitly forbid it.
Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > Is there any portability reason to avoid "local" in the porcelains? I > don't have any plans for using it, but I don't see why we'd explicitly > forbid it. Things that are not even in POSIX are forbidden unless explicitly allowed. In general, he way we encouraged people to think has been "don't use it, it is not even in POSIX" and "even if it is in POSIX, we know the support by platform/implementation X is broken, so let's not use it". It has been successfully helped us to stay out of portability troubles. There may be a few tiny cases where we said "practically everybody we care about has it, even though it is not in POSIX, and it makes our life so vastly be better" to explicitly allow some feature, though. And "local"? Not absolutely essential, unless you are doing a library that you want to avoid stepping on users' toes. Besides, we are no longer adding scripted Porcelains left and right---rather, people are actively rewriting them. Thanks.
On Tue, May 04 2021, Junio C Hamano wrote: > Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > >> Is there any portability reason to avoid "local" in the porcelains? I >> don't have any plans for using it, but I don't see why we'd explicitly >> forbid it. > > Things that are not even in POSIX are forbidden unless explicitly > allowed. > > In general, he way we encouraged people to think has been "don't use > it, it is not even in POSIX" and "even if it is in POSIX, we know > the support by platform/implementation X is broken, so let's not use > it". It has been successfully helped us to stay out of portability > troubles. > > There may be a few tiny cases where we said "practically everybody > we care about has it, even though it is not in POSIX, and it makes > our life so vastly be better" to explicitly allow some feature, > though. > > And "local"? Not absolutely essential, unless you are doing a > library that you want to avoid stepping on users' toes. Besides, we > are no longer adding scripted Porcelains left and right---rather, > people are actively rewriting them. [You mailed me off-list with "Did I forget to say it is not even POSIX?", I had this reply to that, also applicable here]: BEGIN QUOTE I'm aware that it's not in POSIX. What I'm getting at is that anyone who doesn't support this must have been failing t0000-basic.sh since 01d3a526ad9 (t0000: check whether the shell supports the "local" keyword, 2017-10-26), released with v2.16.0. And all tests of any kind since 78dc08875cd (test-lib: allow short options to be bundled, 2020-03-22), perhaps earlier, I didn't trace the full includes, but that's the first use in test-lib.sh itself. Released with v2.27.0. From my own cross-platform testing and us not having any reports about this I very much suspect that this is one of those not-in-POSIX but in practice supported everywhere, or at least anywhere Git has been ported to. So just saying it's OK to use unconditionally of "in t/?" should be fine, like e.g. some of the C89-plus-XYZ features. END QUOTE Afterwards it occurred to me that your stance on in-tree *.sh goes against e.g. brian's opinion on it expressed in reply to [1], i.e. that we might justify future new *.sh built-ins. In any case, as noted above I really don't care much about using "local" in any *.sh built-in. I just don't think this "We do not use it in scripted Porcelains[...]" clause is needed at all. It's effectively synonymous with saying "we still want to support git on platforms that are so broken they can't even run a single test in our test suite". 1. https://lore.kernel.org/git/patch-1.1-83266f30b67-20210417T084346Z-avarab@gmail.com/
Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > It's effectively synonymous with saying "we still want to support git on > platforms that are so broken they can't even run a single test in our > test suite". Not really. Those on such a platform would (rightly) say that it is the test suite that is broken and out of compliance.
On Tue, May 04 2021, Junio C Hamano wrote: > Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > >> It's effectively synonymous with saying "we still want to support git on >> platforms that are so broken they can't even run a single test in our >> test suite". > > Not really. Those on such a platform would (rightly) say that it is > the test suite that is broken and out of compliance. Indeed. But the lack of any reports about that suggests that in practice this is universally supported enough to be a hard dependency. In any case, it's clear you don't agree and you manage the patch queue. So I'll leave it at that. My aim here was to discover if we had any reason to think that "local" was less universally implemented than other POSIX/C89-plus features we rely on. It seems that it's not.
Ævar Arnfjörð Bjarmason wrote: > On Tue, May 04 2021, Junio C Hamano wrote: > > Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > > > >> It's effectively synonymous with saying "we still want to support git on > >> platforms that are so broken they can't even run a single test in our > >> test suite". > > > > Not really. Those on such a platform would (rightly) say that it is > > the test suite that is broken and out of compliance. > > Indeed. But the lack of any reports about that suggests that in practice > this is universally supported enough to be a hard dependency. It is universally supported but not always in the same way, which is why it hasn't managed to become part of POSIX. I for example stumbled upon an issue where 'local x' does not work the same in bash and zsh, triggering a huge discussion on the zsh mailing list. However, if you set a value (e.g. 'local x=') then you shouldn't have any problems. f () { local x; echo ${x-unset}; }; f # this behavior varies For a full discussion check [1]. [1] https://www.austingroupbugs.net/bug_view_page.php?bug_id=767
On 2021-05-04 at 15:09:54, Ævar Arnfjörð Bjarmason wrote: > > On Tue, May 04 2021, Junio C Hamano wrote: > > > Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > > > >> It's effectively synonymous with saying "we still want to support git on > >> platforms that are so broken they can't even run a single test in our > >> test suite". > > > > Not really. Those on such a platform would (rightly) say that it is > > the test suite that is broken and out of compliance. > > Indeed. But the lack of any reports about that suggests that in practice > this is universally supported enough to be a hard dependency. > > In any case, it's clear you don't agree and you manage the patch > queue. So I'll leave it at that. I don't feel very strongly, but I would be fine with requiring local. One of the main reasons the Austin Group is having trouble standardizing it is because some shells implement it with lexical scoping and some use dynamic scoping, but if we try not to make too many assumptions, we'll probably be okay. > My aim here was to discover if we had any reason to think that "local" > was less universally implemented than other POSIX/C89-plus features we > rely on. It seems that it's not. "local" is missing in AT&T ksh. As far as I'm aware, all of the other major open source shells support it: bash, dash, mksh, posh, and zsh, plus the default shells on most BSDs, so there are options for people who would like to use Git on systems with a less capable shell. As a practical matter, that means someone on a proprietary Unix or possibly a non-Unix system. In the latter case, we've only seen Plan 9, I believe, which is so devoid of reasonably functional basic Unix tools that it's probably hopeless, and therefore we really only need to consider Windows and Unix systems. I should point out that we also make several non-POSIX assumptions about shell behavior in our testsuite. I fixed one in c64368e3a2 ("t9001: avoid including non-trailing NUL bytes in variables", 2019-11-27), but the other one we make is that all components of a pipeline are run in subshells, which is not true of AT&T ksh or zsh (in zsh mode), which run the last item in the main shell. This assumption used to break running zsh on our testsuite, but the developers recently accepted a patch to make zsh in sh mode emulate what all other sh implementations do because this assumption is so widespread that, as a practical matter, many things break in such a case (the Git testsuite being one of them, but also things like Debian's debconf). So I'm okay with requiring a little more than POSIX behavior here because as a practical matter we already do and POSIX permits a wide variety of behavior which is never implemented (e.g., running something _other_ than the last element in a pipeline in the main shell) and which we could not practically test. I agree that we should aim for targets which provide excellent compatibility and that when in doubt, we should look to POSIX.
"brian m. carlson" <sandals@crustytoothpaste.net> writes: > One of the main reasons the Austin Group is having trouble standardizing > it is because some shells implement it with lexical scoping and some use > dynamic scoping, but if we try not to make too many assumptions, we'll > probably be okay. Thanks for raising that one. As long as we make sure that our script would work with anybody's "local", I do not care very much. I just didn't think we want to spend the effort to explain that in the documentation and enforce it---compared to the effort, just saying "no" is certainly easier. But if somebody wants an incremental patch on top to spell out how "local" is allowed to be used in our scripts, that's fine by me.
brian m. carlson wrote: > On 2021-05-04 at 15:09:54, Ævar Arnfjörð Bjarmason wrote: > > My aim here was to discover if we had any reason to think that "local" > > was less universally implemented than other POSIX/C89-plus features we > > rely on. It seems that it's not. > > "local" is missing in AT&T ksh. It's not missing, it's supported only in "functions", which have a different syntax in ksh: function f { local x="foo"; echo $x; }; f
On 2021-05-05 at 00:08:17, Felipe Contreras wrote: > brian m. carlson wrote: > > On 2021-05-04 at 15:09:54, Ævar Arnfjörð Bjarmason wrote: > > > My aim here was to discover if we had any reason to think that "local" > > > was less universally implemented than other POSIX/C89-plus features we > > > rely on. It seems that it's not. > > > > "local" is missing in AT&T ksh. > > It's not missing, it's supported only in "functions", which have a > different syntax in ksh: > > function f { local x="foo"; echo $x; }; f I believe there's a bash compatibility mode that enables this, but I don't see support for it in the version Debian ships[0], which doesn't enable that mode: ksh -c 'function f { local x="foo"; echo $x; }; f' ksh: f[1]: local: not found [No such file or directory] It's also possible that it may be aliased to "typeset" on some systems, in which case it may indeed work. Regardless of the specifics, I think we can both agree that it doesn't work in sh-style functions, so for Git's purposes, AT&T ksh does not meet our needs in terms of support for local. [0] I believe Debian's philosophy on this is that they ship bash, so programs that need bashisms can just use bash.
brian m. carlson wrote: > On 2021-05-05 at 00:08:17, Felipe Contreras wrote: > > brian m. carlson wrote: > > > On 2021-05-04 at 15:09:54, Ævar Arnfjörð Bjarmason wrote: > > > > My aim here was to discover if we had any reason to think that "local" > > > > was less universally implemented than other POSIX/C89-plus features we > > > > rely on. It seems that it's not. > > > > > > "local" is missing in AT&T ksh. > > > > It's not missing, it's supported only in "functions", which have a > > different syntax in ksh: > > > > function f { local x="foo"; echo $x; }; f > > I believe there's a bash compatibility mode that enables this, but I > don't see support for it in the version Debian ships[0], which doesn't > enable that mode: > > ksh -c 'function f { local x="foo"; echo $x; }; f' > ksh: f[1]: local: not found [No such file or directory] I tried to build ksh myself to make sure, and apparently there's a mess of different versions of it. I manually applied some patches and tried three times before giving up. It's clear the Debian version and the Arch Linux version are completely different. > Regardless of the specifics, I think we can both agree that it doesn't > work in sh-style functions, so for Git's purposes, AT&T ksh does not > meet our needs in terms of support for local. Indeed. But it also probably doesn't meet many other needs. A quick check shows 54 failed tests just on t0000-basic.sh. I don't think it's something we should even consider. Cheers.
On Tue, May 04, 2021 at 10:59:02PM -0500, Felipe Contreras wrote: > > I believe there's a bash compatibility mode that enables this, but I > > don't see support for it in the version Debian ships[0], which doesn't > > enable that mode: > > > > ksh -c 'function f { local x="foo"; echo $x; }; f' > > ksh: f[1]: local: not found [No such file or directory] > > I tried to build ksh myself to make sure, and apparently there's a mess > of different versions of it. I manually applied some patches and tried > three times before giving up. > > It's clear the Debian version and the Arch Linux version are completely > different. > > > Regardless of the specifics, I think we can both agree that it doesn't > > work in sh-style functions, so for Git's purposes, AT&T ksh does not > > meet our needs in terms of support for local. > > Indeed. But it also probably doesn't meet many other needs. A quick > check shows 54 failed tests just on t0000-basic.sh. I don't think it's > something we should even consider. Yeah, we have visited this off and on over the years (links below). There are some modernized ksh variants that work OK (like mksh), but I think we've declared ksh93 as not worth it. -Peff [0] https://lore.kernel.org/git/31108626.20150508231514@gmail.com/ [1] https://lore.kernel.org/git/xmqqinxt3kwq.fsf@gitster.mtv.corp.google.com/
Jeff King wrote: > On Tue, May 04, 2021 at 10:59:02PM -0500, Felipe Contreras wrote: > > Indeed. But it also probably doesn't meet many other needs. A quick > > check shows 54 failed tests just on t0000-basic.sh. I don't think it's > > something we should even consider. > > Yeah, we have visited this off and on over the years (links below). > There are some modernized ksh variants that work OK (like mksh), but I > think we've declared ksh93 as not worth it. Just to clarify what was said on that thread: ksh93 is not a version. There is at least ksh93u+ and ksh93v-, which are *completely* different.
diff --git a/Documentation/CodingGuidelines b/Documentation/CodingGuidelines index 45465bc0c9..ea70676a30 100644 --- a/Documentation/CodingGuidelines +++ b/Documentation/CodingGuidelines @@ -175,6 +175,11 @@ For shell scripts specifically (not exhaustive): does not have such a problem. + - Even though "local" is not part of POSIX, we make heavy use of it + in our test suite. We do not use it in scripted Porcelains, and + hopefully nobody starts using "local" before they are reimplemented + in C ;-) + For C programs:
01d3a526 (t0000: check whether the shell supports the "local" keyword, 2017-10-26) raised a test balloon to see if those who build and test Git use a platform with a shell that lacks support for the "local" keyword. After two years, 7f0b5908 (t0000: reword comments for "local" test, 2019-08-08) documented that "local" keyword, even though is outside POSIX, is allowed in our test scripts. Let's write it in the CodingGuidelines, too. It might be tempting to allow it in scripted Porcelains (we have avoided getting them contaminiated by "local" so far), but they are on their way out and getting rewritten in C. Signed-off-by: Junio C Hamano <gitster@pobox.com> --- Documentation/CodingGuidelines | 5 +++++ 1 file changed, 5 insertions(+)