| Message ID | CAH8yC8nnFYt0raL+twrG_v8XZn_FDJWyM=tH=QL13Z70_bHRiQ@mail.gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Solaris and sed: Too many commands, last: s/\n// | expand |
[cc:+Ævar] On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote: > I enabled self tests for Solaris. Solaris has some anemic utilities so > I put /usr/gnu/bin first on-path. The first question is if you are really running GNU 'sed'? My guess is "no, it's still picking up Solaris's 'sed'". > gmake -C templates SHELL_PATH='/bin/bash' PERL_PATH='/usr/bin/perl' > sed: Too many commands, last: s/\n// > --- chainlint/arithmetic-expansion.expect 2019-02-24 Ævar had taken a stab at getting this to work on Solaris back in September 2018. He did fix one or two issues, but the one you report here was not addressed. He ultimately abandoned the attempt, I think, after Junio's reminder that the project had long ago declared Solaris's tools as a lost cause. See the relevant messages: http://public-inbox.org/git/87bm9cs5y6.fsf@evledraar.gmail.com/ http://public-inbox.org/git/CAPig+cRrwW662Hzp8V1sH51x8qYxPyqG=VHpOkfd-=RwUSSZbw@mail.gmail.com http://public-inbox.org/git/877ek0rymz.fsf@evledraar.gmail.com http://public-inbox.org/git/xmqqzhwv51wy.fsf@gitster-ct.c.googlers.com > Solaris in a VM sucks. I can provide SSH access to the hardware if > anyone is interested. It is just an Solaris i86pc on an older Ivy > Bridge. I wouldn't mind taking a look at it, though I don't promise anything, and I suspect the only way forward is by ensuring that the GNU or XPG tools are used instead of the Solaris ones.
On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote: > > [cc:+Ævar] > > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote: > > I enabled self tests for Solaris. Solaris has some anemic utilities so > > I put /usr/gnu/bin first on-path. > > The first question is if you are really running GNU 'sed'? My guess is > "no, it's still picking up Solaris's 'sed'". I believe so. After modifying PATH, command -v returns: Solaris tools: sed: /usr/gnu/bin/sed awk: /usr/gnu/bin/awk grep: /usr/gnu/bin/grep (This was added to my scripts to confirm). Maybe Git would benefit from SED, AWK and GREP variables like PERL. > ... > > Solaris in a VM sucks. I can provide SSH access to the hardware if > > anyone is interested. It is just an Solaris i86pc on an older Ivy > > Bridge. > > I wouldn't mind taking a look at it, though I don't promise anything, > and I suspect the only way forward is by ensuring that the GNU or XPG > tools are used instead of the Solaris ones. Send over your authorized_keys. You will connect with: esunshine@151.196.22.177 -p 1523 I'm in a dynamic IP address block. You will have to ping me on occasion to get the updated IP address. Some other machines you may be interested in: * PowerMac G5, PPC big-endian with OS X 10.5 (port 1522) * MacBook late 2012, x86_64 with OS X 10.9 (port 1524) * Intel Goldmont with SHA extensions (port 1526) Andy Polyakov uses the PowerMac for tuning his ASM used in OpenSSL. If you want to speedup SHA-1 (re: unaligned accesses) then try the Goldmont machine. SHA-1 runs at 1.8 cycles per byte on Goldmont. Here's the compression function ready for copy/paste: https://github.com/noloader/SHA-Intrinsics Jeff
On Mon, Mar 11, 2019 at 05:43:55PM -0400, Jeffrey Walton wrote: > On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote: > > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote: > > > I enabled self tests for Solaris. Solaris has some anemic utilities so > > > I put /usr/gnu/bin first on-path. > > > > The first question is if you are really running GNU 'sed'? My guess is > > "no, it's still picking up Solaris's 'sed'". > > I believe so. After modifying PATH, command -v returns: > > Solaris tools: > sed: /usr/gnu/bin/sed > awk: /usr/gnu/bin/awk > grep: /usr/gnu/bin/grep > > (This was added to my scripts to confirm). Can you doubly confirm by having t/Makefile tell us itself? I ask because that error message in your original email is exactly the error spit out by Solaris 'sed'. Perhaps apply this patch: diff --git a/t/Makefile b/t/Makefile index c83fd18861..d22eff4463 100644 --- a/t/Makefile +++ b/t/Makefile @@ -70,6 +70,7 @@ clean-chainlint: check-chainlint: @mkdir -p '$(CHAINLINTTMP_SQ)' && \ + printf 'SED VERSION: ' && sed --version && \ err=0 && \ for i in $(CHAINLINTTESTS); do \ $(CHAINLINT) <chainlint/$$i.test | \ If it reports "SED VERSION: sed (GNU sed) x.y", then we can feel reasonably assured that it's using GNU 'sed', however, if it errors out or prints something else, then it's not picking up your PATH for some reason. > Maybe Git would benefit from SED, AWK and GREP variables like PERL. Very possibly.
On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote: > > [cc:+Ævar] > > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote: > > I enabled self tests for Solaris. Solaris has some anemic utilities so > > I put /usr/gnu/bin first on-path. > > The first question is if you are really running GNU 'sed'? My guess is > "no, it's still picking up Solaris's 'sed'". Yeah, you're right: $ make test V=1 ... gmake -C t/ all gmake[1]: Entering directory `/export/home/jwalton/Build-Scripts/git-2.21.0/t' rm -f -r 'test-results' SED VERSION: sed: illegal option -- version I wonder why ... Let me check if I am exporting PATH. Jeff
On Mon, Mar 11, 2019 at 7:52 PM Jeffrey Walton <noloader@gmail.com> wrote: > > On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote: > > > > [cc:+Ævar] > > > > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote: > > > I enabled self tests for Solaris. Solaris has some anemic utilities so > > > I put /usr/gnu/bin first on-path. > > > > The first question is if you are really running GNU 'sed'? My guess is > > "no, it's still picking up Solaris's 'sed'". > > Yeah, you're right: > > $ make test V=1 > ... > > gmake -C t/ all > gmake[1]: Entering directory `/export/home/jwalton/Build-Scripts/git-2.21.0/t' > rm -f -r 'test-results' > SED VERSION: sed: illegal option -- version > > I wonder why ... Let me check if I am exporting PATH. I was not exporting PATH, so that was my mistake. However, PATH is now exported but I get the same result. Is the build system clearing the environment? Jeff
On Mon, Mar 11, 2019 at 6:28 PM Eric Sunshine <sunshine@sunshineco.com> wrote: > > On Mon, Mar 11, 2019 at 05:43:55PM -0400, Jeffrey Walton wrote: > > On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote: > > > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote: > > > > I enabled self tests for Solaris. Solaris has some anemic utilities so > > > > I put /usr/gnu/bin first on-path. > > > > > > The first question is if you are really running GNU 'sed'? My guess is > > > "no, it's still picking up Solaris's 'sed'". > > > > I believe so. After modifying PATH, command -v returns: > > > > Solaris tools: > > sed: /usr/gnu/bin/sed > > awk: /usr/gnu/bin/awk > > grep: /usr/gnu/bin/grep > > > > (This was added to my scripts to confirm). > ... > > > Maybe Git would benefit from SED, AWK and GREP variables like PERL. > > Very possibly. Another potential workaround is to use Perl. Perl is already a prerequisite, it get passed to the test gear through PERL_PATH, and it avoids Solaris' anemic tools. Sadly my Perl sucks. You would be disappointed if I tried to whip up a patch. Jeff
On Mon, Mar 11, 2019 at 9:07 PM Jeffrey Walton <noloader@gmail.com> wrote: > > On Mon, Mar 11, 2019 at 6:28 PM Eric Sunshine <sunshine@sunshineco.com> wrote: > > > > On Mon, Mar 11, 2019 at 05:43:55PM -0400, Jeffrey Walton wrote: > > > On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote: > > > > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote: > > > > > I enabled self tests for Solaris. Solaris has some anemic utilities so > > > > > I put /usr/gnu/bin first on-path. > > > > > > > > The first question is if you are really running GNU 'sed'? My guess is > > > > "no, it's still picking up Solaris's 'sed'". > > > > > > I believe so. After modifying PATH, command -v returns: > > > > > > Solaris tools: > > > sed: /usr/gnu/bin/sed > > > awk: /usr/gnu/bin/awk > > > grep: /usr/gnu/bin/grep > > > > > > (This was added to my scripts to confirm). > > ... > > > > > Maybe Git would benefit from SED, AWK and GREP variables like PERL. > > > > Very possibly. > > Another potential workaround is to use Perl. Perl is already a > prerequisite, it get passed to the test gear through PERL_PATH, and it > avoids Solaris' anemic tools. > > Sadly my Perl sucks. You would be disappointed if I tried to whip up a patch. Oh man, you're using GNU make. I thought Git was using that anemic Posix Make. See attached. I think Solaris provides an older gawk. Is this an easier problem: awk: chainlint.sed:88: :squash awk: chainlint.sed:88: ^ syntax error awk: chainlint.sed:91: s/\\\n// awk: chainlint.sed:91: ^ backslash not last character on line Usage: awk [POSIX or GNU style options] -f progfile [--] file ... Usage: awk [POSIX or GNU style options] [--] 'program' file ... Jeff
On Mon, Mar 11, 2019 at 9:55 PM Jeffrey Walton <noloader@gmail.com> wrote: > > On Mon, Mar 11, 2019 at 9:07 PM Jeffrey Walton <noloader@gmail.com> wrote: > > > > On Mon, Mar 11, 2019 at 6:28 PM Eric Sunshine <sunshine@sunshineco.com> wrote: > > > > > > On Mon, Mar 11, 2019 at 05:43:55PM -0400, Jeffrey Walton wrote: > > > > On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote: > > > > > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote: > > > > > > I enabled self tests for Solaris. Solaris has some anemic utilities so > > > > > > I put /usr/gnu/bin first on-path. > > > > > > > > > > The first question is if you are really running GNU 'sed'? My guess is > > > > > "no, it's still picking up Solaris's 'sed'". > > > > > > > > I believe so. After modifying PATH, command -v returns: > > > > > > > > Solaris tools: > > > > sed: /usr/gnu/bin/sed > > > > awk: /usr/gnu/bin/awk > > > > grep: /usr/gnu/bin/grep > > > > > > > > (This was added to my scripts to confirm). > > > ... > > > > > > > Maybe Git would benefit from SED, AWK and GREP variables like PERL. > > > > > > Very possibly. > > > > Another potential workaround is to use Perl. Perl is already a > > prerequisite, it get passed to the test gear through PERL_PATH, and it > > avoids Solaris' anemic tools. > > > > Sadly my Perl sucks. You would be disappointed if I tried to whip up a patch. > > Oh man, you're using GNU make. I thought Git was using that anemic > Posix Make. See attached. > > I think Solaris provides an older gawk. Is this an easier problem: > > awk: chainlint.sed:88: :squash > awk: chainlint.sed:88: ^ syntax error > awk: chainlint.sed:91: s/\\\n// > awk: chainlint.sed:91: ^ backslash not last character on line > Usage: awk [POSIX or GNU style options] -f progfile [--] file ... > Usage: awk [POSIX or GNU style options] [--] 'program' file ... My bad , there was a typo... 'awk' got assigned to SED variable. This patch works as expected. Jeff
On Mon, Mar 11, 2019 at 10:11 PM Jeffrey Walton <noloader@gmail.com> wrote: > > On Mon, Mar 11, 2019 at 9:55 PM Jeffrey Walton <noloader@gmail.com> wrote: > > > > On Mon, Mar 11, 2019 at 9:07 PM Jeffrey Walton <noloader@gmail.com> wrote: > > > > > > On Mon, Mar 11, 2019 at 6:28 PM Eric Sunshine <sunshine@sunshineco.com> wrote: > > > > > > > > On Mon, Mar 11, 2019 at 05:43:55PM -0400, Jeffrey Walton wrote: > > > > > On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote: > > > > > > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote: > > > > > > > I enabled self tests for Solaris. Solaris has some anemic utilities so > > > > > > > I put /usr/gnu/bin first on-path. > > > > > > > > > > > > The first question is if you are really running GNU 'sed'? My guess is > > > > > > "no, it's still picking up Solaris's 'sed'". > > > > > > > > > > I believe so. After modifying PATH, command -v returns: > > > > > > > > > > Solaris tools: > > > > > sed: /usr/gnu/bin/sed > > > > > awk: /usr/gnu/bin/awk > > > > > grep: /usr/gnu/bin/grep > > > > > > > > > > (This was added to my scripts to confirm). > > > > ... > > > > > > > > > Maybe Git would benefit from SED, AWK and GREP variables like PERL. > > > > > > > > Very possibly. > > > > > > Another potential workaround is to use Perl. Perl is already a > > > prerequisite, it get passed to the test gear through PERL_PATH, and it > > > avoids Solaris' anemic tools. > > > > > > Sadly my Perl sucks. You would be disappointed if I tried to whip up a patch. > > > > Oh man, you're using GNU make. I thought Git was using that anemic > > Posix Make. See attached. > > > > I think Solaris provides an older gawk. Is this an easier problem: > > > > awk: chainlint.sed:88: :squash > > awk: chainlint.sed:88: ^ syntax error > > awk: chainlint.sed:91: s/\\\n// > > awk: chainlint.sed:91: ^ backslash not last character on line > > Usage: awk [POSIX or GNU style options] -f progfile [--] file ... > > Usage: awk [POSIX or GNU style options] [--] 'program' file ... > > My bad , there was a typo... 'awk' got assigned to SED variable. > > This patch works as expected. https://github.com/git/git/pull/582
On Mon, Mar 11, 2019 at 10:11:41PM -0400, Jeffrey Walton wrote: > On Mon, Mar 11, 2019 at 9:55 PM Jeffrey Walton <noloader@gmail.com> wrote: > > Oh man, you're using GNU make. I thought Git was using that anemic > > Posix Make. See attached. > > > > I think Solaris provides an older gawk. Is this an easier problem: > > > > awk: chainlint.sed:88: :squash > > awk: chainlint.sed:88: ^ syntax error > > awk: chainlint.sed:91: s/\\\n// > > awk: chainlint.sed:91: ^ backslash not last character on line > > Usage: awk [POSIX or GNU style options] -f progfile [--] file ... > > Usage: awk [POSIX or GNU style options] [--] 'program' file ... > > My bad , there was a typo... 'awk' got assigned to SED variable. > > This patch works as expected. > > diff --git a/t/Makefile b/t/Makefile > @@ -11,11 +11,25 @@ SHELL_PATH ?= $(SHELL) > TAR ?= $(TAR) > +AWK ?= $(AWK) > +SED ?= $(SED) > +GREP ?= $(GREP) > > +# Fix Solaris tools. These are Posix. GNU tools located at /usr/gnu/bin. > +ifneq ($(wildcard /usr/gnu/bin/grep),) > + GREP := /usr/gnu/bin/grep > +endif > +ifneq ($(wildcard /usr/gnu/bin/sed),) > + SED := /usr/gnu/bin/sed > +endif > +ifneq ($(wildcard /usr/gnu/bin/awk),) > + SED := /usr/gnu/bin/awk > +endif I think the last assignment ought to be "AWK := ...". Anyhow, this sort of platform-specific tool customization is typically done by config.mak.uname in the top-level Git directory. In fact, there's already a section for SunOS: ifeq ($(uname_S),SunOS) ... SANE_TOOL_PATH = /usr/xpg6/bin:/usr/xpg4/bin ... Prepending /usr/gnu/bin to SANE_TOOL_PATH might be a good idea as a first step toward fixing the problem you're seeing on Solaris, however, as Ævar mentioned in [1], SANE_TOOL_PATH isn't presently consulted when running tests. But, as he suggested, fixing the tests to respect SANE_TOOL_PATH might be a good solution overall. So, rather than making platform-specific customizations to t/Makefile, an arguably better solution would be to update config.mak.uname to add /usr/gnu/bin to SANE_TOOL_PATH and then update the test system to respect that value (thus, these GREP, SED, AWK specializations can be avoided). [1]: http://public-inbox.org/git/877ek0rymz.fsf@evledraar.gmail.com/
On Mon, Mar 11, 2019 at 10:45 PM Eric Sunshine <sunshine@sunshineco.com> wrote: > > On Mon, Mar 11, 2019 at 10:11:41PM -0400, Jeffrey Walton wrote: > > On Mon, Mar 11, 2019 at 9:55 PM Jeffrey Walton <noloader@gmail.com> wrote: > > > Oh man, you're using GNU make. I thought Git was using that anemic > > > Posix Make. See attached. > > > > > > I think Solaris provides an older gawk. Is this an easier problem: > > > > > > awk: chainlint.sed:88: :squash > > > awk: chainlint.sed:88: ^ syntax error > > > awk: chainlint.sed:91: s/\\\n// > > > awk: chainlint.sed:91: ^ backslash not last character on line > > > Usage: awk [POSIX or GNU style options] -f progfile [--] file ... > > > Usage: awk [POSIX or GNU style options] [--] 'program' file ... > > > > My bad , there was a typo... 'awk' got assigned to SED variable. > > > > This patch works as expected. > > > > diff --git a/t/Makefile b/t/Makefile > > @@ -11,11 +11,25 @@ SHELL_PATH ?= $(SHELL) > > TAR ?= $(TAR) > > +AWK ?= $(AWK) > > +SED ?= $(SED) > > +GREP ?= $(GREP) > > > > +# Fix Solaris tools. These are Posix. GNU tools located at /usr/gnu/bin. > > +ifneq ($(wildcard /usr/gnu/bin/grep),) > > + GREP := /usr/gnu/bin/grep > > +endif > > +ifneq ($(wildcard /usr/gnu/bin/sed),) > > + SED := /usr/gnu/bin/sed > > +endif > > +ifneq ($(wildcard /usr/gnu/bin/awk),) > > + SED := /usr/gnu/bin/awk > > +endif > > I think the last assignment ought to be "AWK := ...". > > Anyhow, this sort of platform-specific tool customization is typically done by config.mak.uname in the top-level Git directory. In fact, there's already a section for SunOS: > > ifeq ($(uname_S),SunOS) > ... > SANE_TOOL_PATH = /usr/xpg6/bin:/usr/xpg4/bin > ... > > Prepending /usr/gnu/bin to SANE_TOOL_PATH might be a good idea as a first step toward fixing the problem you're seeing on Solaris, however, as Ćvar mentioned in [1], SANE_TOOL_PATH isn't presently consulted when running tests. But, as he suggested, fixing the tests to respect SANE_TOOL_PATH might be a good solution overall. > > So, rather than making platform-specific customizations to t/Makefile, an arguably better solution would be to update config.mak.uname to add /usr/gnu/bin to SANE_TOOL_PATH and then update the test system to respect that value (thus, these GREP, SED, AWK specializations can be avoided). I may have been blowing away SANE_BUILD_PATH. My build script calls: SANE_BUILD_PATH="<my updated tool path> \ PKG_CONFIG_PATH="..." \ ... \ ./configure \ --prefix=<my new tool location> \ --libdir="..." \ ... I set SANE_BUILD_PATH in the environment because I want Git to use the new tools being built for testing. Or, if Git is blowing away my SANE_BUILD_PATH , then that might explain the encoding failures. The updated tools, like new iConv and Unistring, are not on PATH. The updated tools are located at /var/sanitize/bin, and are passed through SANE_BUILD_PATH . This might also explain the intermittent crashes I see. Old tools from /usr/bin are used, but new libraries from /var/sanitize/lib are used because of LD_LIBRARY_PATH. Gotta love those fucking path problems that have plagued Linux for the last 30 years. The idiot who figured it was a good idea to compile and link against one library, and then load the wrong library at runtime, should get a platinum Darwin award. It may be better to set SANE_TOOL_PATH something like: SANE_TOOL_PATH = "$SANE_TOOL_PATH:/usr/xpg6/bin:/usr/xpg4/bin" Jeff
On Mon, Mar 11 2019, Eric Sunshine wrote: > [cc:+Ævar] > > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote: >> I enabled self tests for Solaris. Solaris has some anemic utilities so >> I put /usr/gnu/bin first on-path. > > The first question is if you are really running GNU 'sed'? My guess is > "no, it's still picking up Solaris's 'sed'". > >> gmake -C templates SHELL_PATH='/bin/bash' PERL_PATH='/usr/bin/perl' >> sed: Too many commands, last: s/\n// >> --- chainlint/arithmetic-expansion.expect 2019-02-24 > > Ævar had taken a stab at getting this to work on Solaris back in > September 2018. He did fix one or two issues, but the one you report > here was not addressed. He ultimately abandoned the attempt, I think, > after Junio's reminder that the project had long ago declared > Solaris's tools as a lost cause. See the relevant messages: > > http://public-inbox.org/git/87bm9cs5y6.fsf@evledraar.gmail.com/ > http://public-inbox.org/git/CAPig+cRrwW662Hzp8V1sH51x8qYxPyqG=VHpOkfd-=RwUSSZbw@mail.gmail.com > http://public-inbox.org/git/877ek0rymz.fsf@evledraar.gmail.com > http://public-inbox.org/git/xmqqzhwv51wy.fsf@gitster-ct.c.googlers.com > >> Solaris in a VM sucks. I can provide SSH access to the hardware if >> anyone is interested. It is just an Solaris i86pc on an older Ivy >> Bridge. > > I wouldn't mind taking a look at it, though I don't promise anything, > and I suspect the only way forward is by ensuring that the GNU or XPG > tools are used instead of the Solaris ones. FWIW I gave up on Solaris sed after trying to fix it the last time around. If you need another box to test it on you can apply for a GCC Farm account: https://cfarm.tetaneutral.net/machines/list/
--- chainlint/bash-array.expect 2019-02-24 11:31:46.000000000 -0500 +++ chainlinttmp/bash-array.actual 2019-03-11 12:20:16.902186323 -0400 @@ -1,10 +0,0 @@ -( - foo && - bar=(gumbo stumbo wumbo) && - baz ->) && -( - foo && - bar=${#bar[@]} && - baz ->)