Message ID | 20250303-pks-meson-cross-compiling-v1-1-73002ef6432e@pks.im (mailing list archive) |
---|---|
State | New |
Headers | show |
Series | meson: distinguish build and target host binaries | expand |
Hello Patrick, On Mon, 03 Mar 2025 13:10:59 +0100, Patrick Steinhardt <ps@pks.im> wrote: > Almost all of the tools we discover during the build process need to be > native programs. There are only a handful of exceptions, which typically > are programs whose paths we need to embed into the resulting executable > so that they can be found on the target system when Git executes. While > this distinction typically doesn't matter, it does start to matter when > considering cross-compilation where the build and target machines are > different. > > Meson supports cross-compilation via so-called machine files. These > machine files allow the user to override parameters for the build > machine, but also for the target machine when cross-compiling. Part of > the machine file is a section that allows the user to override the > location where binaries are to be found in the target system. The > following machine file would for example override the path of the POSIX > shell: > > [binaries] > sh = '/usr/xpg4/bin/sh' > > It can be handed over to Meson via `meson setup --cross-file`. > > We do not handle this correctly right now though because we don't know > to distinguish binaries for the build and target hosts at all. Address > this by explicitly passing the `native:` parameter to `find_program()`: > > - When set to `true`, we get binaries discovered on the build host. > > - When set to `false`, we get either the path specified in the > machine file. Or, if no machine file exists or it doesn't specify > the binary path, then we fall back to the binary discovered on the > build host. > > As mentioned, only a handful of binaries are not native: only the system > shell, Python and Perl need to be treated specially here. > > Signed-off-by: Patrick Steinhardt <ps@pks.im> > --- > Hi, > > this patch addresses the issue reported at [1], where it is impossible > to specify the shell, Python and Perl paths during cross-compilation > when using Meson. I still believe that is is a 'misuse' of the cross-file (as stated already here [1]) the given programs in cross-file are to be meant to run while cross-compiling (at compile time) and not on the target (e.g. it would be impossible to find a program (as the name find_program indicates) at compile/configure where the target layout is yet unknown.... I believe the correct solution is an extra configure option for cross-compile and a sane default (or find_program) in case of native build... Regards, Peter [1] https://public-inbox.org/git/20250210122603.5130e309@gmx.net/ > > The series is built on top of master at cb0ae672aea (A bit more post > -rc0, 2025-02-27) with junio/ps/build-meson-fixes-0130 at 9350423982a > (gitlab-ci: restrict maximum number of link jobs on Windows, 2025-02-26) > merged into it. > > Thanks! > > Patrick > > [1]: <20250209133027.64a865aa@gmx.net> > --- > Documentation/meson.build | 12 ++++----- > gitweb/meson.build | 2 +- > meson.build | 66 ++++++++++++++++++++++++++++++++++++----------- > templates/meson.build | 4 +-- > 4 files changed, 60 insertions(+), 24 deletions(-) > > diff --git a/Documentation/meson.build b/Documentation/meson.build > index 0a0f2bfa14a..63891fb4455 100644 > --- a/Documentation/meson.build > +++ b/Documentation/meson.build > @@ -207,9 +207,9 @@ manpages = { > > docs_backend = get_option('docs_backend') > if docs_backend == 'auto' > - if find_program('asciidoc', dirs: program_path, required: false).found() > + if find_program('asciidoc', dirs: program_path, native: true, required: false).found() > docs_backend = 'asciidoc' > - elif find_program('asciidoctor', dirs: program_path, required: false).found() > + elif find_program('asciidoctor', dirs: program_path, native: true, required: false).found() > docs_backend = 'asciidoctor' > else > error('Neither asciidoc nor asciidoctor were found.') > @@ -217,7 +217,7 @@ if docs_backend == 'auto' > endif > > if docs_backend == 'asciidoc' > - asciidoc = find_program('asciidoc', dirs: program_path) > + asciidoc = find_program('asciidoc', dirs: program_path, native: true) > asciidoc_html = 'xhtml11' > asciidoc_docbook = 'docbook' > xmlto_extra = [ ] > @@ -246,7 +246,7 @@ if docs_backend == 'asciidoc' > asciidoc_conf, > ] > elif docs_backend == 'asciidoctor' > - asciidoctor = find_program('asciidoctor', dirs: program_path) > + asciidoctor = find_program('asciidoctor', dirs: program_path, native: true) > asciidoc_html = 'xhtml5' > asciidoc_docbook = 'docbook5' > xmlto_extra = [ > @@ -284,7 +284,7 @@ elif docs_backend == 'asciidoctor' > ] > endif > > -xmlto = find_program('xmlto', dirs: program_path) > +xmlto = find_program('xmlto', dirs: program_path, native: true) > > cmd_lists = [ > 'cmds-ancillaryinterrogators.adoc', > @@ -405,7 +405,7 @@ if get_option('docs').contains('html') > pointing_to: 'git.html', > ) > > - xsltproc = find_program('xsltproc', dirs: program_path) > + xsltproc = find_program('xsltproc', dirs: program_path, native: true) > > user_manual_xml = custom_target( > command: asciidoc_common_options + [ > diff --git a/gitweb/meson.build b/gitweb/meson.build > index 89b403dc9de..88a54b4dc99 100644 > --- a/gitweb/meson.build > +++ b/gitweb/meson.build > @@ -1,5 +1,5 @@ > gitweb_config = configuration_data() > -gitweb_config.set_quoted('PERL_PATH', perl.full_path()) > +gitweb_config.set_quoted('PERL_PATH', target_perl.full_path()) > gitweb_config.set_quoted('CSSMIN', '') > gitweb_config.set_quoted('JSMIN', '') > gitweb_config.set_quoted('GIT_BINDIR', get_option('prefix') / get_option('bindir')) > diff --git a/meson.build b/meson.build > index 516207f9cfe..0a229f44199 100644 > --- a/meson.build > +++ b/meson.build > @@ -155,6 +155,37 @@ > # These machine files can be passed to `meson setup` via the `--native-file` > # option. > # > +# Cross compilation > +# ================= > +# > +# Machine files can also be used in the context of cross-compilation to > +# describe the target machine as well as the cross-compiler toolchain that > +# shall be used. An example machine file could look like the following: > +# > +# [binaries] > +# c = 'x86_64-w64-mingw32-gcc' > +# cpp = 'x86_64-w64-mingw32-g++' > +# ar = 'x86_64-w64-mingw32-ar' > +# windres = 'x86_64-w64-mingw32-windres' > +# strip = 'x86_64-w64-mingw32-strip' > +# exe_wrapper = 'wine64' > +# sh = 'C:/Program Files/Git for Windows/usr/bin/sh.exe' > +# > +# [host_machine] > +# system = 'windows' > +# cpu_family = 'x86_64' > +# cpu = 'x86_64' > +# endian = 'little' > +# > +# These machine files can be passed to `meson setup` via the `--cross-file` > +# option. > +# > +# Note that next to the cross-compiler toolchain, the `[binaries]` section is > +# also used to locate a couple of binaries that will be built into Git. This > +# includes `sh`, `python` and `perl`, so when cross-compiling Git you likely > +# want to set these binary paths in addition to the cross-compiler toolchain > +# binaries. > +# > # Subproject wrappers > # =================== > # > @@ -173,7 +204,7 @@ project('git', 'c', > # The version is only of cosmetic nature, so if we cannot find a shell yet we > # simply don't set up a version at all. This may be the case for example on > # Windows systems, where we first have to bootstrap the host environment. > - version: find_program('sh', required: false).found() ? run_command( > + version: find_program('sh', native: true, required: false).found() ? run_command( > 'GIT-VERSION-GEN', meson.current_source_dir(), '--format=@GIT_VERSION@', > capture: true, > check: true, > @@ -198,16 +229,18 @@ elif host_machine.system() == 'windows' > program_path = [ 'C:/Program Files/Git/bin', 'C:/Program Files/Git/usr/bin' ] > endif > > -cygpath = find_program('cygpath', dirs: program_path, required: false) > -diff = find_program('diff', dirs: program_path) > -git = find_program('git', dirs: program_path, required: false) > -sed = find_program('sed', dirs: program_path) > -shell = find_program('sh', dirs: program_path) > -tar = find_program('tar', dirs: program_path) > +cygpath = find_program('cygpath', dirs: program_path, native: true, required: false) > +diff = find_program('diff', dirs: program_path, native: true) > +git = find_program('git', dirs: program_path, native: true, required: false) > +sed = find_program('sed', dirs: program_path, native: true) > +shell = find_program('sh', dirs: program_path, native: true) > +tar = find_program('tar', dirs: program_path, native: true) > + > +target_shell = find_program('sh', dirs: program_path, native: false) > > # Sanity-check that programs required for the build exist. > foreach tool : ['cat', 'cut', 'grep', 'sort', 'tr', 'uname'] > - find_program(tool, dirs: program_path) > + find_program(tool, dirs: program_path, native: true) > endforeach > > script_environment = environment() > @@ -706,7 +739,7 @@ libgit_c_args = [ > '-DGIT_LOCALE_PATH="' + get_option('localedir') + '"', > '-DGIT_MAN_PATH="' + get_option('mandir') + '"', > '-DPAGER_ENV="' + get_option('pager_environment') + '"', > - '-DSHELL_PATH="' + fs.as_posix(shell.full_path()) + '"', > + '-DSHELL_PATH="' + fs.as_posix(target_shell.full_path()) + '"', > ] > libgit_include_directories = [ '.' ] > libgit_dependencies = [ ] > @@ -761,6 +794,7 @@ endif > build_options_config.set_quoted('X', executable_suffix) > > python = import('python').find_installation('python3', required: get_option('python')) > +target_python = find_program('python3', native: false, required: python.found()) > if python.found() > build_options_config.set('NO_PYTHON', '') > else > @@ -790,9 +824,11 @@ endif > # which we can do starting with Meson 1.5.0 and newer, or we have to > # match against the minor version. > if meson.version().version_compare('>=1.5.0') > - perl = find_program('perl', dirs: program_path, required: perl_required, version: '>=5.26.0', version_argument: '-V:version') > + perl = find_program('perl', dirs: program_path, native: true, required: perl_required, version: '>=5.26.0', version_argument: '-V:version') > + target_perl = find_program('perl', dirs: program_path, native: false, required: perl.found(), version: '>=5.26.0', version_argument: '-V:version') > else > - perl = find_program('perl', dirs: program_path, required: perl_required, version: '>=26') > + perl = find_program('perl', dirs: program_path, native: true, required: perl_required, version: '>=26') > + target_perl = find_program('perl', dirs: program_path, native: false, required: perl.found(), version: '>=26') > endif > perl_features_enabled = perl.found() and get_option('perl').allowed() > if perl_features_enabled > @@ -843,7 +879,7 @@ else > build_options_config.set('NO_PTHREADS', '1') > endif > > -msgfmt = find_program('msgfmt', dirs: program_path, required: false) > +msgfmt = find_program('msgfmt', dirs: program_path, native: true, required: false) > gettext_option = get_option('gettext').disable_auto_if(not msgfmt.found()) > if not msgfmt.found() and gettext_option.enabled() > error('Internationalization via libintl requires msgfmt') > @@ -1975,9 +2011,9 @@ foreach key, value : { > 'GIT_TEST_TEMPLATE_DIR': meson.project_build_root() / 'templates', > 'GIT_TEST_TEXTDOMAINDIR': meson.project_build_root() / 'po', > 'PAGER_ENV': get_option('pager_environment'), > - 'PERL_PATH': perl.found() ? perl.full_path() : '', > - 'PYTHON_PATH': python.found () ? python.full_path() : '', > - 'SHELL_PATH': shell.full_path(), > + 'PERL_PATH': target_perl.found() ? target_perl.full_path() : '', > + 'PYTHON_PATH': target_python.found () ? target_python.full_path() : '', > + 'SHELL_PATH': target_shell.full_path(), > 'TAR': tar.full_path(), > 'TEST_OUTPUT_DIRECTORY': test_output_directory, > 'TEST_SHELL_PATH': shell.full_path(), > diff --git a/templates/meson.build b/templates/meson.build > index 1faf9a44cea..02e6eebe80b 100644 > --- a/templates/meson.build > +++ b/templates/meson.build > @@ -1,6 +1,6 @@ > template_config = configuration_data() > -template_config.set('PERL_PATH', perl.found() ? fs.as_posix(perl.full_path()) : '') > -template_config.set('SHELL_PATH', fs.as_posix(shell.full_path())) > +template_config.set('PERL_PATH', target_perl.found() ? fs.as_posix(target_perl.full_path()) : '') > +template_config.set('SHELL_PATH', fs.as_posix(target_shell.full_path())) > template_config.set('GITWEBDIR', fs.as_posix(get_option('prefix') / get_option('datadir') / 'gitweb')) > > configure_file( > > --- > base-commit: af208620eecbe9b4655e06a28d5146a40738150e > change-id: 20250303-pks-meson-cross-compiling-7bad3078b9e8 >
diff --git a/Documentation/meson.build b/Documentation/meson.build index 0a0f2bfa14a..63891fb4455 100644 --- a/Documentation/meson.build +++ b/Documentation/meson.build @@ -207,9 +207,9 @@ manpages = { docs_backend = get_option('docs_backend') if docs_backend == 'auto' - if find_program('asciidoc', dirs: program_path, required: false).found() + if find_program('asciidoc', dirs: program_path, native: true, required: false).found() docs_backend = 'asciidoc' - elif find_program('asciidoctor', dirs: program_path, required: false).found() + elif find_program('asciidoctor', dirs: program_path, native: true, required: false).found() docs_backend = 'asciidoctor' else error('Neither asciidoc nor asciidoctor were found.') @@ -217,7 +217,7 @@ if docs_backend == 'auto' endif if docs_backend == 'asciidoc' - asciidoc = find_program('asciidoc', dirs: program_path) + asciidoc = find_program('asciidoc', dirs: program_path, native: true) asciidoc_html = 'xhtml11' asciidoc_docbook = 'docbook' xmlto_extra = [ ] @@ -246,7 +246,7 @@ if docs_backend == 'asciidoc' asciidoc_conf, ] elif docs_backend == 'asciidoctor' - asciidoctor = find_program('asciidoctor', dirs: program_path) + asciidoctor = find_program('asciidoctor', dirs: program_path, native: true) asciidoc_html = 'xhtml5' asciidoc_docbook = 'docbook5' xmlto_extra = [ @@ -284,7 +284,7 @@ elif docs_backend == 'asciidoctor' ] endif -xmlto = find_program('xmlto', dirs: program_path) +xmlto = find_program('xmlto', dirs: program_path, native: true) cmd_lists = [ 'cmds-ancillaryinterrogators.adoc', @@ -405,7 +405,7 @@ if get_option('docs').contains('html') pointing_to: 'git.html', ) - xsltproc = find_program('xsltproc', dirs: program_path) + xsltproc = find_program('xsltproc', dirs: program_path, native: true) user_manual_xml = custom_target( command: asciidoc_common_options + [ diff --git a/gitweb/meson.build b/gitweb/meson.build index 89b403dc9de..88a54b4dc99 100644 --- a/gitweb/meson.build +++ b/gitweb/meson.build @@ -1,5 +1,5 @@ gitweb_config = configuration_data() -gitweb_config.set_quoted('PERL_PATH', perl.full_path()) +gitweb_config.set_quoted('PERL_PATH', target_perl.full_path()) gitweb_config.set_quoted('CSSMIN', '') gitweb_config.set_quoted('JSMIN', '') gitweb_config.set_quoted('GIT_BINDIR', get_option('prefix') / get_option('bindir')) diff --git a/meson.build b/meson.build index 516207f9cfe..0a229f44199 100644 --- a/meson.build +++ b/meson.build @@ -155,6 +155,37 @@ # These machine files can be passed to `meson setup` via the `--native-file` # option. # +# Cross compilation +# ================= +# +# Machine files can also be used in the context of cross-compilation to +# describe the target machine as well as the cross-compiler toolchain that +# shall be used. An example machine file could look like the following: +# +# [binaries] +# c = 'x86_64-w64-mingw32-gcc' +# cpp = 'x86_64-w64-mingw32-g++' +# ar = 'x86_64-w64-mingw32-ar' +# windres = 'x86_64-w64-mingw32-windres' +# strip = 'x86_64-w64-mingw32-strip' +# exe_wrapper = 'wine64' +# sh = 'C:/Program Files/Git for Windows/usr/bin/sh.exe' +# +# [host_machine] +# system = 'windows' +# cpu_family = 'x86_64' +# cpu = 'x86_64' +# endian = 'little' +# +# These machine files can be passed to `meson setup` via the `--cross-file` +# option. +# +# Note that next to the cross-compiler toolchain, the `[binaries]` section is +# also used to locate a couple of binaries that will be built into Git. This +# includes `sh`, `python` and `perl`, so when cross-compiling Git you likely +# want to set these binary paths in addition to the cross-compiler toolchain +# binaries. +# # Subproject wrappers # =================== # @@ -173,7 +204,7 @@ project('git', 'c', # The version is only of cosmetic nature, so if we cannot find a shell yet we # simply don't set up a version at all. This may be the case for example on # Windows systems, where we first have to bootstrap the host environment. - version: find_program('sh', required: false).found() ? run_command( + version: find_program('sh', native: true, required: false).found() ? run_command( 'GIT-VERSION-GEN', meson.current_source_dir(), '--format=@GIT_VERSION@', capture: true, check: true, @@ -198,16 +229,18 @@ elif host_machine.system() == 'windows' program_path = [ 'C:/Program Files/Git/bin', 'C:/Program Files/Git/usr/bin' ] endif -cygpath = find_program('cygpath', dirs: program_path, required: false) -diff = find_program('diff', dirs: program_path) -git = find_program('git', dirs: program_path, required: false) -sed = find_program('sed', dirs: program_path) -shell = find_program('sh', dirs: program_path) -tar = find_program('tar', dirs: program_path) +cygpath = find_program('cygpath', dirs: program_path, native: true, required: false) +diff = find_program('diff', dirs: program_path, native: true) +git = find_program('git', dirs: program_path, native: true, required: false) +sed = find_program('sed', dirs: program_path, native: true) +shell = find_program('sh', dirs: program_path, native: true) +tar = find_program('tar', dirs: program_path, native: true) + +target_shell = find_program('sh', dirs: program_path, native: false) # Sanity-check that programs required for the build exist. foreach tool : ['cat', 'cut', 'grep', 'sort', 'tr', 'uname'] - find_program(tool, dirs: program_path) + find_program(tool, dirs: program_path, native: true) endforeach script_environment = environment() @@ -706,7 +739,7 @@ libgit_c_args = [ '-DGIT_LOCALE_PATH="' + get_option('localedir') + '"', '-DGIT_MAN_PATH="' + get_option('mandir') + '"', '-DPAGER_ENV="' + get_option('pager_environment') + '"', - '-DSHELL_PATH="' + fs.as_posix(shell.full_path()) + '"', + '-DSHELL_PATH="' + fs.as_posix(target_shell.full_path()) + '"', ] libgit_include_directories = [ '.' ] libgit_dependencies = [ ] @@ -761,6 +794,7 @@ endif build_options_config.set_quoted('X', executable_suffix) python = import('python').find_installation('python3', required: get_option('python')) +target_python = find_program('python3', native: false, required: python.found()) if python.found() build_options_config.set('NO_PYTHON', '') else @@ -790,9 +824,11 @@ endif # which we can do starting with Meson 1.5.0 and newer, or we have to # match against the minor version. if meson.version().version_compare('>=1.5.0') - perl = find_program('perl', dirs: program_path, required: perl_required, version: '>=5.26.0', version_argument: '-V:version') + perl = find_program('perl', dirs: program_path, native: true, required: perl_required, version: '>=5.26.0', version_argument: '-V:version') + target_perl = find_program('perl', dirs: program_path, native: false, required: perl.found(), version: '>=5.26.0', version_argument: '-V:version') else - perl = find_program('perl', dirs: program_path, required: perl_required, version: '>=26') + perl = find_program('perl', dirs: program_path, native: true, required: perl_required, version: '>=26') + target_perl = find_program('perl', dirs: program_path, native: false, required: perl.found(), version: '>=26') endif perl_features_enabled = perl.found() and get_option('perl').allowed() if perl_features_enabled @@ -843,7 +879,7 @@ else build_options_config.set('NO_PTHREADS', '1') endif -msgfmt = find_program('msgfmt', dirs: program_path, required: false) +msgfmt = find_program('msgfmt', dirs: program_path, native: true, required: false) gettext_option = get_option('gettext').disable_auto_if(not msgfmt.found()) if not msgfmt.found() and gettext_option.enabled() error('Internationalization via libintl requires msgfmt') @@ -1975,9 +2011,9 @@ foreach key, value : { 'GIT_TEST_TEMPLATE_DIR': meson.project_build_root() / 'templates', 'GIT_TEST_TEXTDOMAINDIR': meson.project_build_root() / 'po', 'PAGER_ENV': get_option('pager_environment'), - 'PERL_PATH': perl.found() ? perl.full_path() : '', - 'PYTHON_PATH': python.found () ? python.full_path() : '', - 'SHELL_PATH': shell.full_path(), + 'PERL_PATH': target_perl.found() ? target_perl.full_path() : '', + 'PYTHON_PATH': target_python.found () ? target_python.full_path() : '', + 'SHELL_PATH': target_shell.full_path(), 'TAR': tar.full_path(), 'TEST_OUTPUT_DIRECTORY': test_output_directory, 'TEST_SHELL_PATH': shell.full_path(), diff --git a/templates/meson.build b/templates/meson.build index 1faf9a44cea..02e6eebe80b 100644 --- a/templates/meson.build +++ b/templates/meson.build @@ -1,6 +1,6 @@ template_config = configuration_data() -template_config.set('PERL_PATH', perl.found() ? fs.as_posix(perl.full_path()) : '') -template_config.set('SHELL_PATH', fs.as_posix(shell.full_path())) +template_config.set('PERL_PATH', target_perl.found() ? fs.as_posix(target_perl.full_path()) : '') +template_config.set('SHELL_PATH', fs.as_posix(target_shell.full_path())) template_config.set('GITWEBDIR', fs.as_posix(get_option('prefix') / get_option('datadir') / 'gitweb')) configure_file(
Almost all of the tools we discover during the build process need to be native programs. There are only a handful of exceptions, which typically are programs whose paths we need to embed into the resulting executable so that they can be found on the target system when Git executes. While this distinction typically doesn't matter, it does start to matter when considering cross-compilation where the build and target machines are different. Meson supports cross-compilation via so-called machine files. These machine files allow the user to override parameters for the build machine, but also for the target machine when cross-compiling. Part of the machine file is a section that allows the user to override the location where binaries are to be found in the target system. The following machine file would for example override the path of the POSIX shell: [binaries] sh = '/usr/xpg4/bin/sh' It can be handed over to Meson via `meson setup --cross-file`. We do not handle this correctly right now though because we don't know to distinguish binaries for the build and target hosts at all. Address this by explicitly passing the `native:` parameter to `find_program()`: - When set to `true`, we get binaries discovered on the build host. - When set to `false`, we get either the path specified in the machine file. Or, if no machine file exists or it doesn't specify the binary path, then we fall back to the binary discovered on the build host. As mentioned, only a handful of binaries are not native: only the system shell, Python and Perl need to be treated specially here. Signed-off-by: Patrick Steinhardt <ps@pks.im> --- Hi, this patch addresses the issue reported at [1], where it is impossible to specify the shell, Python and Perl paths during cross-compilation when using Meson. The series is built on top of master at cb0ae672aea (A bit more post -rc0, 2025-02-27) with junio/ps/build-meson-fixes-0130 at 9350423982a (gitlab-ci: restrict maximum number of link jobs on Windows, 2025-02-26) merged into it. Thanks! Patrick [1]: <20250209133027.64a865aa@gmx.net> --- Documentation/meson.build | 12 ++++----- gitweb/meson.build | 2 +- meson.build | 66 ++++++++++++++++++++++++++++++++++++----------- templates/meson.build | 4 +-- 4 files changed, 60 insertions(+), 24 deletions(-) --- base-commit: af208620eecbe9b4655e06a28d5146a40738150e change-id: 20250303-pks-meson-cross-compiling-7bad3078b9e8