[v4] scripts/symlink-install-tree.py: Fix MESONINTROSPECT parsing

Commit Message

Peter Maydell Oct. 18, 2024, 1:08 p.m. UTC

From: Akihiko Odaki <[e-mail]>

The arguments in MESONINTROSPECT are quoted with shlex.quote() so it
must be parsed with shlex.split().  Otherwise the script will fail if
the build directory has a character like "~" in it.

Note: this fix cannot be backported directly to any stable branch
that doesn't require Meson version 1.4.0 or better; otherwise it will
work OK on Linux but will break on Windows hosts.

(Unfortunately, Meson prior to version 1.4.0 was inconsistent between
host OSes about how it quoted arguments, and used a different quoting
process on Windows hosts.  Our current git trunk already requires
1.5.0 as of commit 07f0d32641e ("Require meson version 1.5.0"), but
the stable branches are still on older Meson.)

Fixes: cf60ccc330 ("cutils: Introduce bundle mechanism")
Reported-by: Michael Tokarev <[e-mail]>
Signed-off-by: Akihiko Odaki <[e-mail]>
[PMM: Updated commit message to give all the detail about the
Meson version compability requirements.]
Signed-off-by: Peter Maydell <[e-mail]>
---
This is essentially back to version 1 of Akihiko's patch, now we
have a new enough Meson; I just updated the commit message.
 https://patchew.org/QEMU/[e-mail]/
(I have dropped the various reviewed-by and tested-by headers because
I figured the passage of time was enough to make them moot.)

 scripts/symlink-install-tree.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Comments

Pierrick Bouvier Oct. 21, 2024, 5:27 p.m. UTC | #1

On 10/18/24 06:08, Peter Maydell wrote:
> From: Akihiko Odaki <[e-mail]>
> 
> The arguments in MESONINTROSPECT are quoted with shlex.quote() so it
> must be parsed with shlex.split().  Otherwise the script will fail if
> the build directory has a character like "~" in it.
> 
> Note: this fix cannot be backported directly to any stable branch
> that doesn't require Meson version 1.4.0 or better; otherwise it will
> work OK on Linux but will break on Windows hosts.
> 
> (Unfortunately, Meson prior to version 1.4.0 was inconsistent between
> host OSes about how it quoted arguments, and used a different quoting
> process on Windows hosts.  Our current git trunk already requires
> 1.5.0 as of commit 07f0d32641e ("Require meson version 1.5.0"), but
> the stable branches are still on older Meson.)
> 
> Fixes: cf60ccc330 ("cutils: Introduce bundle mechanism")
> Reported-by: Michael Tokarev <[e-mail]>
> Signed-off-by: Akihiko Odaki <[e-mail]>
> [PMM: Updated commit message to give all the detail about the
> Meson version compability requirements.]
> Signed-off-by: Peter Maydell <[e-mail]>
> ---
> This is essentially back to version 1 of Akihiko's patch, now we
> have a new enough Meson; I just updated the commit message.
>   https://patchew.org/QEMU/[e-mail]/
> (I have dropped the various reviewed-by and tested-by headers because
> I figured the passage of time was enough to make them moot.)
> 
>   scripts/symlink-install-tree.py | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/symlink-install-tree.py b/scripts/symlink-install-tree.py
> index 8ed97e3c943..b72563895c5 100644
> --- a/scripts/symlink-install-tree.py
> +++ b/scripts/symlink-install-tree.py
> @@ -4,6 +4,7 @@
>   import errno
>   import json
>   import os
> +import shlex
>   import subprocess
>   import sys
>   
> @@ -14,7 +15,7 @@ def destdir_join(d1: str, d2: str) -> str:
>       return str(PurePath(d1, *PurePath(d2).parts[1:]))
>   
>   introspect = os.environ.get('MESONINTROSPECT')
> -out = subprocess.run([*introspect.split(' '), '--installed'],
> +out = subprocess.run([*shlex.split(introspect), '--installed'],
>                        stdout=subprocess.PIPE, check=True).stdout
>   for source, dest in json.loads(out).items():
>       bundle_dest = destdir_join('qemu-bundle', dest)

Reviewed-by: Pierrick Bouvier <[e-mail]>
Tested-by: Pierrick Bouvier <[e-mail]>

I confirm this fixes the error when the path to qemu src tree contains '~'.

Christophe Lyon Oct. 21, 2024, 5:48 p.m. UTC | #2

Hi,



Le lun. 21 oct. 2024, 19:27, Pierrick Bouvier <[e-mail]>
a écrit :

> On 10/18/24 06:08, Peter Maydell wrote:
> > From: Akihiko Odaki <[e-mail]>
> >
> > The arguments in MESONINTROSPECT are quoted with shlex.quote() so it
> > must be parsed with shlex.split().  Otherwise the script will fail if
> > the build directory has a character like "~" in it.
> >
> > Note: this fix cannot be backported directly to any stable branch
> > that doesn't require Meson version 1.4.0 or better; otherwise it will
> > work OK on Linux but will break on Windows hosts.
> >
> > (Unfortunately, Meson prior to version 1.4.0 was inconsistent between
> > host OSes about how it quoted arguments, and used a different quoting
> > process on Windows hosts.  Our current git trunk already requires
> > 1.5.0 as of commit 07f0d32641e ("Require meson version 1.5.0"), but
> > the stable branches are still on older Meson.)
> >
> > Fixes: cf60ccc330 ("cutils: Introduce bundle mechanism")
> > Reported-by: Michael Tokarev <[e-mail]>
> > Signed-off-by: Akihiko Odaki <[e-mail]>
> > [PMM: Updated commit message to give all the detail about the
> > Meson version compability requirements.]
> > Signed-off-by: Peter Maydell <[e-mail]>
> > ---
> > This is essentially back to version 1 of Akihiko's patch, now we
> > have a new enough Meson; I just updated the commit message.
> >
> https://patchew.org/QEMU/[e-mail]/
> > (I have dropped the various reviewed-by and tested-by headers because
> > I figured the passage of time was enough to make them moot.)
> >
> >   scripts/symlink-install-tree.py | 3 ++-
> >   1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/scripts/symlink-install-tree.py
> b/scripts/symlink-install-tree.py
> > index 8ed97e3c943..b72563895c5 100644
> > --- a/scripts/symlink-install-tree.py
> > +++ b/scripts/symlink-install-tree.py
> > @@ -4,6 +4,7 @@
> >   import errno
> >   import json
> >   import os
> > +import shlex
> >   import subprocess
> >   import sys
> >
> > @@ -14,7 +15,7 @@ def destdir_join(d1: str, d2: str) -> str:
> >       return str(PurePath(d1, *PurePath(d2).parts[1:]))
> >
> >   introspect = os.environ.get('MESONINTROSPECT')
> > -out = subprocess.run([*introspect.split(' '), '--installed'],
> > +out = subprocess.run([*shlex.split(introspect), '--installed'],
> >                        stdout=subprocess.PIPE, check=True).stdout
> >   for source, dest in json.loads(out).items():
> >       bundle_dest = destdir_join('qemu-bundle', dest)
>
> Reviewed-by: Pierrick Bouvier <[e-mail]>
> Tested-by: Pierrick Bouvier <[e-mail]>
>
> I confirm this fixes the error when the path to qemu src tree contains '~'.
>

I confirm this fixes the problem we detected in our CI (where some build
dirnames have '~'

Thanks

>

Pierrick Bouvier Oct. 28, 2024, 7:43 p.m. UTC | #3

On 10/18/24 06:08, Peter Maydell wrote:
> From: Akihiko Odaki <[e-mail]>
> 
> The arguments in MESONINTROSPECT are quoted with shlex.quote() so it
> must be parsed with shlex.split().  Otherwise the script will fail if
> the build directory has a character like "~" in it.
> 
> Note: this fix cannot be backported directly to any stable branch
> that doesn't require Meson version 1.4.0 or better; otherwise it will
> work OK on Linux but will break on Windows hosts.
> 
> (Unfortunately, Meson prior to version 1.4.0 was inconsistent between
> host OSes about how it quoted arguments, and used a different quoting
> process on Windows hosts.  Our current git trunk already requires
> 1.5.0 as of commit 07f0d32641e ("Require meson version 1.5.0"), but
> the stable branches are still on older Meson.)
> 
> Fixes: cf60ccc330 ("cutils: Introduce bundle mechanism")
> Reported-by: Michael Tokarev <[e-mail]>
> Signed-off-by: Akihiko Odaki <[e-mail]>
> [PMM: Updated commit message to give all the detail about the
> Meson version compability requirements.]
> Signed-off-by: Peter Maydell <[e-mail]>
> ---
> This is essentially back to version 1 of Akihiko's patch, now we
> have a new enough Meson; I just updated the commit message.
>   https://patchew.org/QEMU/[e-mail]/
> (I have dropped the various reviewed-by and tested-by headers because
> I figured the passage of time was enough to make them moot.)
> 
>   scripts/symlink-install-tree.py | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/symlink-install-tree.py b/scripts/symlink-install-tree.py
> index 8ed97e3c943..b72563895c5 100644
> --- a/scripts/symlink-install-tree.py
> +++ b/scripts/symlink-install-tree.py
> @@ -4,6 +4,7 @@
>   import errno
>   import json
>   import os
> +import shlex
>   import subprocess
>   import sys
>   
> @@ -14,7 +15,7 @@ def destdir_join(d1: str, d2: str) -> str:
>       return str(PurePath(d1, *PurePath(d2).parts[1:]))
>   
>   introspect = os.environ.get('MESONINTROSPECT')
> -out = subprocess.run([*introspect.split(' '), '--installed'],
> +out = subprocess.run([*shlex.split(introspect), '--installed'],
>                        stdout=subprocess.PIPE, check=True).stdout
>   for source, dest in json.loads(out).items():
>       bundle_dest = destdir_join('qemu-bundle', dest)

Hi,

would that be possible to pull this patch please?
It's currently blocking the devs who reported it initially.

Thanks,
Pierrick

Peter Maydell Oct. 29, 2024, 10:04 a.m. UTC | #4

On Mon, 28 Oct 2024 at 19:43, Pierrick Bouvier
<[e-mail]> wrote:
>
> On 10/18/24 06:08, Peter Maydell wrote:
> > From: Akihiko Odaki <[e-mail]>
> >
> > The arguments in MESONINTROSPECT are quoted with shlex.quote() so it
> > must be parsed with shlex.split().  Otherwise the script will fail if
> > the build directory has a character like "~" in it.
> >
> > Note: this fix cannot be backported directly to any stable branch
> > that doesn't require Meson version 1.4.0 or better; otherwise it will
> > work OK on Linux but will break on Windows hosts.
> >
> > (Unfortunately, Meson prior to version 1.4.0 was inconsistent between
> > host OSes about how it quoted arguments, and used a different quoting
> > process on Windows hosts.  Our current git trunk already requires
> > 1.5.0 as of commit 07f0d32641e ("Require meson version 1.5.0"), but
> > the stable branches are still on older Meson.)
> >
> > Fixes: cf60ccc330 ("cutils: Introduce bundle mechanism")
> > Reported-by: Michael Tokarev <[e-mail]>
> > Signed-off-by: Akihiko Odaki <[e-mail]>
> > [PMM: Updated commit message to give all the detail about the
> > Meson version compability requirements.]
> > Signed-off-by: Peter Maydell <[e-mail]>
> > ---
> > This is essentially back to version 1 of Akihiko's patch, now we
> > have a new enough Meson; I just updated the commit message.
> >   https://patchew.org/QEMU/[e-mail]/
> > (I have dropped the various reviewed-by and tested-by headers because
> > I figured the passage of time was enough to make them moot.)
> >
> >   scripts/symlink-install-tree.py | 3 ++-
> >   1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/scripts/symlink-install-tree.py b/scripts/symlink-install-tree.py
> > index 8ed97e3c943..b72563895c5 100644
> > --- a/scripts/symlink-install-tree.py
> > +++ b/scripts/symlink-install-tree.py
> > @@ -4,6 +4,7 @@
> >   import errno
> >   import json
> >   import os
> > +import shlex
> >   import subprocess
> >   import sys
> >
> > @@ -14,7 +15,7 @@ def destdir_join(d1: str, d2: str) -> str:
> >       return str(PurePath(d1, *PurePath(d2).parts[1:]))
> >
> >   introspect = os.environ.get('MESONINTROSPECT')
> > -out = subprocess.run([*introspect.split(' '), '--installed'],
> > +out = subprocess.run([*shlex.split(introspect), '--installed'],
> >                        stdout=subprocess.PIPE, check=True).stdout
> >   for source, dest in json.loads(out).items():
> >       bundle_dest = destdir_join('qemu-bundle', dest)
>
> Hi,
>
> would that be possible to pull this patch please?
> It's currently blocking the devs who reported it initially.

Yes, I'll put it in my upcoming pullreq.

thanks
-- PMM

Pierrick Bouvier Oct. 29, 2024, 3:50 p.m. UTC | #5

On 10/29/24 03:04, Peter Maydell wrote:
> On Mon, 28 Oct 2024 at 19:43, Pierrick Bouvier
> <[e-mail]> wrote:
>>
>> On 10/18/24 06:08, Peter Maydell wrote:
>>> From: Akihiko Odaki <[e-mail]>
>>>
>>> The arguments in MESONINTROSPECT are quoted with shlex.quote() so it
>>> must be parsed with shlex.split().  Otherwise the script will fail if
>>> the build directory has a character like "~" in it.
>>>
>>> Note: this fix cannot be backported directly to any stable branch
>>> that doesn't require Meson version 1.4.0 or better; otherwise it will
>>> work OK on Linux but will break on Windows hosts.
>>>
>>> (Unfortunately, Meson prior to version 1.4.0 was inconsistent between
>>> host OSes about how it quoted arguments, and used a different quoting
>>> process on Windows hosts.  Our current git trunk already requires
>>> 1.5.0 as of commit 07f0d32641e ("Require meson version 1.5.0"), but
>>> the stable branches are still on older Meson.)
>>>
>>> Fixes: cf60ccc330 ("cutils: Introduce bundle mechanism")
>>> Reported-by: Michael Tokarev <[e-mail]>
>>> Signed-off-by: Akihiko Odaki <[e-mail]>
>>> [PMM: Updated commit message to give all the detail about the
>>> Meson version compability requirements.]
>>> Signed-off-by: Peter Maydell <[e-mail]>
>>> ---
>>> This is essentially back to version 1 of Akihiko's patch, now we
>>> have a new enough Meson; I just updated the commit message.
>>>    https://patchew.org/QEMU/[e-mail]/
>>> (I have dropped the various reviewed-by and tested-by headers because
>>> I figured the passage of time was enough to make them moot.)
>>>
>>>    scripts/symlink-install-tree.py | 3 ++-
>>>    1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/scripts/symlink-install-tree.py b/scripts/symlink-install-tree.py
>>> index 8ed97e3c943..b72563895c5 100644
>>> --- a/scripts/symlink-install-tree.py
>>> +++ b/scripts/symlink-install-tree.py
>>> @@ -4,6 +4,7 @@
>>>    import errno
>>>    import json
>>>    import os
>>> +import shlex
>>>    import subprocess
>>>    import sys
>>>
>>> @@ -14,7 +15,7 @@ def destdir_join(d1: str, d2: str) -> str:
>>>        return str(PurePath(d1, *PurePath(d2).parts[1:]))
>>>
>>>    introspect = os.environ.get('MESONINTROSPECT')
>>> -out = subprocess.run([*introspect.split(' '), '--installed'],
>>> +out = subprocess.run([*shlex.split(introspect), '--installed'],
>>>                         stdout=subprocess.PIPE, check=True).stdout
>>>    for source, dest in json.loads(out).items():
>>>        bundle_dest = destdir_join('qemu-bundle', dest)
>>
>> Hi,
>>
>> would that be possible to pull this patch please?
>> It's currently blocking the devs who reported it initially.
> 
> Yes, I'll put it in my upcoming pullreq.
> 
> thanks
> -- PMM

Thanks a lot Peter :).

Patch

diff --git a/scripts/symlink-install-tree.py b/scripts/symlink-install-tree.py
index 8ed97e3c943..b72563895c5 100644
--- a/scripts/symlink-install-tree.py
+++ b/scripts/symlink-install-tree.py
@@ -4,6 +4,7 @@ 
 import errno
 import json
 import os
+import shlex
 import subprocess
 import sys
 
@@ -14,7 +15,7 @@  def destdir_join(d1: str, d2: str) -> str:
     return str(PurePath(d1, *PurePath(d2).parts[1:]))
 
 introspect = os.environ.get('MESONINTROSPECT')
-out = subprocess.run([*introspect.split(' '), '--installed'],
+out = subprocess.run([*shlex.split(introspect), '--installed'],
                      stdout=subprocess.PIPE, check=True).stdout
 for source, dest in json.loads(out).items():
     bundle_dest = destdir_join('qemu-bundle', dest)