dash bug: double-quoted "\" breaks glob protection for next char

On 3/4/18 9:08 PM, Martijn Dekker wrote:
> Op 04-03-18 om 16:46 schreef Harald van Dijk:
>> FreeBSD sh also prints a blank line here.
> [...]
>> Like above, FreeBSD sh behaves like ksh.
> 
> I stand corrected.
> 
> Is there any port of FreeBSD sh to other operating systems? It would be
> much more convenient for me to include it in my tests if I didn't have
> to launch a FreeBSD VM and rsync & run the test scripts separately.

None that I know of. Running the test script over ssh might be slightly 
less difficult, but nothing as easy as a port. The source code contains 
several very much FreeBSD-specific bits.

>> Yes, the inconsistency should be fixed. Either it should be treated as
>> quoted or as unquoted, but not quoted-unless-it-comes-from-a-variable. I
>> have no strong feelings on which it should be.
> 
> Neither do I, so I would default to the behaviour that both pre-exists
> in dash and corresponds with the majority of other shells.

I went for the behaviour that required the fewest changes for now, which 
is to treat them as unquoted. If it is agreed that it should be quoted, 
it requires some additional (minor) complications in the parser, because 
the existing state would no longer be sufficient to determine whether } 
should end the substitution. But yes, I agree that given how long dash 
has treated this as quoted, it makes sense to keep that, unless there's 
a compelling reason not to.

> [...]
>>> $ src/dash -c 'printf "%s\n" "${$+\}}"'
>>> \}
>>>
>>> Expected output: }  (no backslash), as in bash 4, yash, ksh93, pdksh,
>>> mksh, zsh. In other words: it should be possible to escape a '}' with a
>>> backslash within the parameter expansion, even if the expansion is
>>> quoted.
>>>
>>> POSIX ref.: 2.6.2 Parameter Expansion
>>> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02
>>>
>>> | Any '}' escaped by a <backslash> or within a quoted string, and
>>> | characters in embedded arithmetic expansions, command substitutions,
>>> | and variable expansions, shall not be examined in determining the
>>> | matching '}'.
>>
>> I believe this actually requires dash's behaviour. This says the first }
>> isn't examined in determining the matching '}', but only that: it just
>> says the parameter expansion expression is $+\}. It doesn't say the
>> backslash is removed.
> 
> I believe the word "escaped" implies that removal. If a '}' is escaped
> by a backslash, it's implied that the backslash is removed as this
> escaping is parsed, just as it's implied that quotes are removed from a
> quoted string.

That's not implied, that's stated:

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_07

> The quote characters ( <backslash>, single-quote, and double-quote) that were present in the original word shall be removed unless they have themselves been quoted.

In this case, the backslash was quoted, so this doesn't apply.

>> I agree that it would be much better to print } here though.
> 
> All other current shells except bosh (schilytools sh) agree, too -- even
> FreeBSD sh, and I checked it this time.

Shells agree on the simple cases to remove the backslash:

   ${x+\}}
   "${x+\}}"
   <<EOF / ${x+\}} / EOF

And with this patch, dash behaves the same for the simple cases.

They aren't in agreement on the more complicated cases though:

In ${x+"\}"}, most shells keep the backslash. ksh and FreeBSD sh remove 
it. I think it makes most sense to keep it, because the general rule for 
\ in double-quoted strings is that it's only removed if the following 
character would have been special. This patch removes it.

In "${x+"\}"}", of the shells that treat the \} as quoted, most shells 
keep the backslash. bash removes it. I think it makes most sense to keep 
the backslash for the same reason as above.

In <<EOF / ${x+"\}"} / EOF, bash and ksh remove the backslash, posh and 
zsh keep it, and FreeBSD sh treats the " as a literal. Again I think it 
makes most sense to keep the backslash (but remove the "). This patch 
removes it.

Another pre-existing dash parsing bug that I encountered now is $(( ( 
$(( 1 )) ) )). This should expand to 1, but gives a hard error in dash, 
again due to the non-recursive nature of dash's parser. A small 
generalisation of what I've been doing so far could handle this, so it 
makes sense to me to try to achieve this at the same time.

Cheers,
Harald van Dijk

Message ID	041881f9-9084-4083-345a-8f85792b48ef@gigawatt.nl (mailing list archive)
State	Superseded
Delegated to:	Herbert Xu
Headers	show Return-Path: <dash-owner@kernel.org> Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 4FDD760365 for <patchwork-dash@patchwork.kernel.org>; Sun, 4 Mar 2018 21:28:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2F3D228833 for <patchwork-dash@patchwork.kernel.org>; Sun, 4 Mar 2018 21:28:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 23BF228843; Sun, 4 Mar 2018 21:28:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI, T_TVD_MIME_EPI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 464772883C for <patchwork-dash@patchwork.kernel.org>; Sun, 4 Mar 2018 21:28:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932083AbeCDV2a (ORCPT <rfc822;patchwork-dash@patchwork.kernel.org>); Sun, 4 Mar 2018 16:28:30 -0500 Received: from home.gigawatt.nl ([83.163.3.213]:51402 "EHLO home.gigawatt.nl" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752018AbeCDV22 (ORCPT <rfc822;dash@vger.kernel.org>); Sun, 4 Mar 2018 16:28:28 -0500 Received: from [IPv6:2001:980:4809:1:e045:1301:c405:78bf] (unknown [IPv6:2001:980:4809:1:e045:1301:c405:78bf]) by home.gigawatt.nl (Postfix) with ESMTPSA id 940355402947; Sun, 4 Mar 2018 21:28:25 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 home.gigawatt.nl 940355402947 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gigawatt.nl; s=default; t=1520198906; bh=/iBJBKfqa62B+YkGq4CxpVx95xdoXuPnaygUWrctO/g=; l=29438; h=Subject:To:Cc:References:From:Date:In-Reply-To:From; b=CzZOlv+AXtSVdB2DBK6WNzK40/CIc7tCsexGG+JfKAZb2drT4uXcjOL1OYu7NyU7O 6s/Q+z138NLJqYMTVDXYgR2rnet++AHL+wNtiTQfiyw1XW2VzY5iIu0EYLT3V8OLYV MF+GonnLmHUFuQE2FSLErOB2bGqN8XCLIzWT17Rk= Subject: Re: dash bug: double-quoted "\" breaks glob protection for next char To: Martijn Dekker <martijn@inlv.org>, Herbert Xu <herbert@gondor.apana.org.au> Cc: Denys Vlasenko <vda.linux@googlemail.com>, dash@vger.kernel.org References: <ecb3af24-cdf1-8b59-523d-8ab2673dbd5c@gigawatt.nl> <9f37ae19-6f74-f527-aa49-dd04c3c010f6@gigawatt.nl> <CAK1hOcMA0NqJ6ajfmNsSh619Nxqmu3oFyY=q7pY4DEhkKp+i-Q@mail.gmail.com> <73e4ad51-1c3b-3173-429f-401296244869@gigawatt.nl> <20180224003344.GA3354@gondor.apana.org.au> <32935756-b1c4-70bc-2e72-4d2b0cb2a835@gigawatt.nl> <20180224165224.GA3864@gondor.apana.org.au> <CAK1hOcOha+c0CkK+5QVs3Vttcrw1z7kKy+pZ_fX5fSN-+W_VKA@mail.gmail.com> <a8ec6fb3-2882-0057-c9f8-96b8f01abba7@gigawatt.nl> <86692fea-c33f-d26d-3b26-6e43bc22a0ee@gigawatt.nl> <20180302074922.GA19418@gondor.apana.org.au> <4242819b-4aee-1238-203f-ec08d001be05@gigawatt.nl> <e8d17ec3-8bd1-a502-816b-638996dca9ab@gigawatt.nl> <ae008292-cbe3-969d-9002-6ab74acacf81@inlv.org> <b06a377a-4467-5b30-f648-12305ab8f714@gigawatt.nl> <7dac7df9-4093-095e-dd71-2d7383edd8c3@inlv.org> From: Harald van Dijk <harald@gigawatt.nl> Message-ID: <041881f9-9084-4083-345a-8f85792b48ef@gigawatt.nl> Date: Sun, 4 Mar 2018 22:29:25 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Thunderbird/58.0 MIME-Version: 1.0 In-Reply-To: <7dac7df9-4093-095e-dd71-2d7383edd8c3@inlv.org> Content-Type: multipart/mixed; boundary="------------906993C7AED20A5A15C4997F" Content-Language: en-GB Sender: dash-owner@vger.kernel.org Precedence: bulk List-ID: <dash.vger.kernel.org> X-Mailing-List: dash@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP

dash bug: double-quoted "\" breaks glob protection for next char

Commit Message

Patch