| Message ID | 20240711202337.3128473-1-andrew.cooper3@citrix.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | CI: part 3 (slimline, and PPC/RISCV fixes) | expand |
On 11/07/2024 9:23 pm, Andrew Cooper wrote: > Existing as suse:opensuse-tumbleweed is a historical quirk, and adjusted for > consistency with all the other containers. > > Make it non-root, use heredocs for legibility, and use the zypper long names > for the benefit of those wondering what was being referenced or duplicated. > > Trim the dependencies substantially. Testing docs isn't very interesting and > saves a lot of space (pandoc pulls in all of Haskel). Other savings come from > removing a huge pile of optional QEMU dependencies (QEMU just needs to build > the Xen parts to be useful here, not have a full GUI environment). > > Finally, there where some packages such as bc, libssh2-devel, libtasn1-devel > and nasm that I'm not aware of any reason to have had, even historically. > > This shaves 1G off the size of the container: > > registry.gitlab.com/xen-project/xen/opensuse tumbleweed-x86_64 1.39GB > registry.gitlab.com/xen-project/xen/suse opensuse-tumbleweed 2.35GB > > Furthermore, identify which components of the build use which dependencies, > which will help managing them in the future. > > Thanks to Olaf Hering for dependency fixes that have been subsumed into this > total overhaul. > > Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> > --- > CC: Anthony PERARD <anthony.perard@vates.tech> > CC: Juergen Gross <jgross@suse.com> > CC: Roger Pau Monné <roger.pau@citrix.com> > CC: Jan Beulich <JBeulich@suse.com> > CC: Stefano Stabellini <sstabellini@kernel.org> > CC: Julien Grall <julien@xen.org> > CC: Oleksii Kurochko <oleksii.kurochko@gmail.com> > CC: Shawn Anastasio <sanastasio@raptorengineering.com> > CC: Olaf Hering <olaf@aepfle.de> > > This includes both of Olaf's recent changes, python3-packaging (now required > for a rebuilt container), and findutils (not required yet; still gets pulled > in transitively), but added for good measure. > > In terms of package delta, newly included are: > > findutils # As suggested > ocaml-findlib # Replaces ocaml-findlib-devel > perl # Transitive previously > python3 # Transitive previously > python3-packaging # As suggested > > while removed are: > > bc # ? > cmake # vtpm stubdom, optional > discount # docs > ghostscript # docs > glibc-devel # Not really optional, given a toolchain > hostname # Optional, not interesting > libaio-devel # Xen < 4.13, perhaps Qemu still but optional > libext2fs-devel # Libfsimage, off by default, optional > libgnutls-devel # Qemu ? > libjpeg62-devel # Qemu ? > libnl3-devel # Libxl COLO(?), option > libnuma-devel # ? > libpng16-devel # Qemu ? > libssh2-devel # ? > libtasn1-devel # ? > nasm # ? > ocaml-findlib-devel # Swapped for non-devel > ocaml-ocamlbuild # ? > ocaml-ocamldoc # ? > pandoc # docs > 'pkgconfig(libpci)' # Qemu > 'pkgconfig(sdl)' # Qemu > 'pkgconfig(sdl2)' # Qemu > systemd-devel # Xen < 4.19, optional > transfig # docs > valgrind-devel # really only for developers > which # Xen < 4.17 > > I've tested this on staging-4.17 and staging, both of which build fine. > staging-4.16 does not, because of SeaBIOS which we've decided not to fix, > hence it's safe to drop `which`. For good measure, runs with this container: 4.17: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322567620 4.19: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322493465 and the expected failure: 4.16: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322499885 ~Andrew
Thu, 11 Jul 2024 21:23:37 +0100 Andrew Cooper <andrew.cooper3@citrix.com>:
> use the zypper long names for the benefit of those wondering what was being referenced or duplicated.
Names of rpm packages do change over time, configure already searches for "something" where it is available, instead of some library or header.
If it is touched anyway, I would use more instead of less 'pkgconfig(something)' references:
libbz2-devel
pkgconfig(bzip2) = 1.0.8
libzstd-devel
pkgconfig(libzstd) = 1.5.6
lzo-devel
pkgconfig(lzo2) = 2.10
xz-devel
pkgconfig(liblzma) = 5.6.2
zlib-devel
pkgconfig(zlib) = 1.3.1
libuuid-devel
pkgconfig(uuid) = 2.40.1
libyajl-devel
pkgconfig(yajl) = 2.1.0
ncurses-devel
pkgconfig(form) = 6.5.20240629
pkgconfig(formw) = 6.5.20240629
pkgconfig(menu) = 6.5.20240629
pkgconfig(menuw) = 6.5.20240629
pkgconfig(ncurses) = 6.5.20240629
pkgconfig(ncurses++) = 6.5.20240629
pkgconfig(ncurses++w) = 6.5.20240629
pkgconfig(ncursesw) = 6.5.20240629
pkgconfig(panel) = 6.5.20240629
pkgconfig(panelw) = 6.5.20240629
pkgconfig(tic) = 6.5.20240629
pkgconfig(tinfo) = 6.5.20240629
glib2-devel
pkgconfig(gio-2.0) = 2.80.3
pkgconfig(gio-unix-2.0) = 2.80.3
pkgconfig(girepository-2.0) = 2.80.3
pkgconfig(glib-2.0) = 2.80.3
pkgconfig(gmodule-2.0) = 2.80.3
pkgconfig(gmodule-export-2.0) = 2.80.3
pkgconfig(gmodule-no-export-2.0) = 2.80.3
pkgconfig(gobject-2.0) = 2.80.3
pkgconfig(gthread-2.0) = 2.80.3
libpixman-1-0-devel
pkgconfig(pixman-1) = 0.43.4
python3-devel is "special", can be used as is.
'git' could be just 'git-core'
Olaf
On 11/07/2024 10:49 pm, Olaf Hering wrote: > Thu, 11 Jul 2024 21:23:37 +0100 Andrew Cooper <andrew.cooper3@citrix.com>: > >> use the zypper long names for the benefit of those wondering what was being referenced or duplicated. > Names of rpm packages do change over time, configure already searches for "something" where it is available, configure has to cope with distros taking different naming decisions, and while I appreciate that packages may change over time even within a distro ... > instead of some library or header. > If it is touched anyway, I would use more instead of less 'pkgconfig(something)' references: > > libbz2-devel > pkgconfig(bzip2) = 1.0.8 > libzstd-devel > pkgconfig(libzstd) = 1.5.6 > lzo-devel > pkgconfig(lzo2) = 2.10 > xz-devel > pkgconfig(liblzma) = 5.6.2 > zlib-devel > pkgconfig(zlib) = 1.3.1 > libuuid-devel > pkgconfig(uuid) = 2.40.1 > libyajl-devel > pkgconfig(yajl) = 2.1.0 > ncurses-devel > pkgconfig(form) = 6.5.20240629 > pkgconfig(formw) = 6.5.20240629 > pkgconfig(menu) = 6.5.20240629 > pkgconfig(menuw) = 6.5.20240629 > pkgconfig(ncurses) = 6.5.20240629 > pkgconfig(ncurses++) = 6.5.20240629 > pkgconfig(ncurses++w) = 6.5.20240629 > pkgconfig(ncursesw) = 6.5.20240629 > pkgconfig(panel) = 6.5.20240629 > pkgconfig(panelw) = 6.5.20240629 > pkgconfig(tic) = 6.5.20240629 > pkgconfig(tinfo) = 6.5.20240629 > glib2-devel > pkgconfig(gio-2.0) = 2.80.3 > pkgconfig(gio-unix-2.0) = 2.80.3 > pkgconfig(girepository-2.0) = 2.80.3 > pkgconfig(glib-2.0) = 2.80.3 > pkgconfig(gmodule-2.0) = 2.80.3 > pkgconfig(gmodule-export-2.0) = 2.80.3 > pkgconfig(gmodule-no-export-2.0) = 2.80.3 > pkgconfig(gobject-2.0) = 2.80.3 > pkgconfig(gthread-2.0) = 2.80.3 > libpixman-1-0-devel > pkgconfig(pixman-1) = 0.43.4 ... this is just a differently-inconsistent list of names, with some very dubious aliases such as `form` and `menu`. What makes pkgconfig() any more stable than the main package name? Why do we want to be treating libraries different to other packages? > python3-devel is "special", can be used as is. As an aside, I looked for a python3-minimal or equivalent, and couldn't find one. I take it there isn't one? > 'git' could be just 'git-core' Thanks. Will fix locally, for Tumbleweed and Leap. ~Andrew
Thu, 11 Jul 2024 23:39:42 +0100 Andrew Cooper <andrew.cooper3@citrix.com>: > configure has to cope with distros taking different naming decisions, > and while I appreciate that packages may change over time even within a > distro ... configure does "pkgconfig --exists something" to probe for some API. I'm sure noone will rename the pkgconfig files, "something.pc" in this case. > ... this is just a differently-inconsistent list of names, with some > very dubious aliases such as `form` and `menu`. I should have truncated that list of names, 'pkgconfig(ncurses)' and 'pkgconfig(glib-2.0)' are relevant. > What makes pkgconfig() any more stable than the main package name? They are supposed to describe some API, and in some cases some functionality. It does not matter much which package provides the API. > Why do we want to be treating libraries different to other packages? Because there is no other namespace to describe functionality. It was implemented for pkg-config, it exists for some cmake and perl modules. > > python3-devel is "special", can be used as is. > As an aside, I looked for a python3-minimal or equivalent, and couldn't > find one. I take it there isn't one? I think there are libraries compiled. There is indeed a 'pkgconfig(python3)', my search just failed to find it. Otherwise, if there are really just .py files in the tree, 'python3-base' is the string provided by 'python3N-base'. Olaf
diff --git a/automation/build/opensuse/tumbleweed-x86_64.dockerfile b/automation/build/opensuse/tumbleweed-x86_64.dockerfile new file mode 100644 index 000000000000..8c5ff56cdc88 --- /dev/null +++ b/automation/build/opensuse/tumbleweed-x86_64.dockerfile @@ -0,0 +1,76 @@ +# syntax=docker/dockerfile:1 +FROM --platform=linux/amd64 opensuse/tumbleweed +LABEL maintainer.name="The Xen Project" +LABEL maintainer.email="xen-devel@lists.xenproject.org" + +ENV XEN_TARGET_ARCH=x86_64 + +RUN <<EOF +#!/bin/bash + set -e + + useradd --create-home user + + zypper refresh + zypper dist-upgrade -y --no-recommends + + DEPS=( + # Xen + bison + checkpolicy + clang + diffutils + findutils + flex + gawk + gcc + make + python3 + + # Tools (general) + git + gzip + patch + perl + pkg-config + wget + # libxenguest dombuilder + libbz2-devel + libzstd-devel + lzo-devel + xz-devel + zlib-devel + # libacpi + acpica + # libxl + libuuid-devel + libyajl-devel + # Header Check + gcc-c++ + # xentop + ncurses-devel + # Python bindings + python3-devel + python3-setuptools + # Ocaml bindings/oxenstored + ocaml + ocaml-findlib + + # Stubdom (download/extract) + bzip2 + tar + + # Qemu build + glib2-devel + libpixman-1-0-devel + meson + ninja + python3-packaging + ) + + zypper install -y --no-recommends "${DEPS[@]}" + zypper clean -a +EOF + +USER user +WORKDIR /build diff --git a/automation/build/suse/opensuse-tumbleweed.dockerfile b/automation/build/suse/opensuse-tumbleweed.dockerfile deleted file mode 100644 index 077fed22260c..000000000000 --- a/automation/build/suse/opensuse-tumbleweed.dockerfile +++ /dev/null @@ -1,74 +0,0 @@ -# syntax=docker/dockerfile:1 -FROM --platform=linux/amd64 opensuse/tumbleweed -LABEL maintainer.name="The Xen Project" \ - maintainer.email="xen-devel@lists.xenproject.org" - -ENV USER root - -RUN mkdir /build -WORKDIR /build - -RUN zypper ref && zypper dup -y --no-recommends -RUN zypper install -y --no-recommends \ - acpica \ - bc \ - bison \ - bzip2 \ - checkpolicy \ - clang \ - cmake \ - diffutils \ - discount \ - flex \ - gawk \ - gcc \ - gcc-c++ \ - git \ - ghostscript \ - glib2-devel \ - glibc-devel \ - gzip \ - hostname \ - libaio-devel \ - libbz2-devel \ - libext2fs-devel \ - libgnutls-devel \ - libjpeg62-devel \ - libnl3-devel \ - libnuma-devel \ - libpixman-1-0-devel \ - libpng16-devel \ - libssh2-devel \ - libtasn1-devel \ - libuuid-devel \ - libyajl-devel \ - libzstd-devel \ - lzo-devel \ - make \ - meson \ - nasm \ - ncurses-devel \ - ninja \ - ocaml \ - ocaml-findlib-devel \ - ocaml-ocamlbuild \ - ocaml-ocamldoc \ - pandoc \ - patch \ - pkg-config \ - 'pkgconfig(libpci)' \ - 'pkgconfig(sdl)' \ - 'pkgconfig(sdl2)' \ - python3-devel \ - python3-setuptools \ - # systemd-devel for Xen < 4.19 - systemd-devel \ - tar \ - transfig \ - valgrind-devel \ - wget \ - which \ - xz-devel \ - zlib-devel \ - && \ - zypper clean -a diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml index b4139414bc8e..fd6ca1270e4e 100644 --- a/automation/gitlab-ci/build.yaml +++ b/automation/gitlab-ci/build.yaml @@ -668,25 +668,25 @@ opensuse-leap-gcc-debug: opensuse-tumbleweed-clang: extends: .clang-x86-64-build variables: - CONTAINER: suse:opensuse-tumbleweed + CONTAINER: opensuse:tumbleweed-x86_64 allow_failure: true opensuse-tumbleweed-clang-debug: extends: .clang-x86-64-build-debug variables: - CONTAINER: suse:opensuse-tumbleweed + CONTAINER: opensuse:tumbleweed-x86_64 allow_failure: true opensuse-tumbleweed-gcc: extends: .gcc-x86-64-build variables: - CONTAINER: suse:opensuse-tumbleweed + CONTAINER: opensuse:tumbleweed-x86_64 allow_failure: true opensuse-tumbleweed-gcc-debug: extends: .gcc-x86-64-build-debug variables: - CONTAINER: suse:opensuse-tumbleweed + CONTAINER: opensuse:tumbleweed-x86_64 allow_failure: true # PowerPC builds (x86 cross) diff --git a/automation/scripts/containerize b/automation/scripts/containerize index 4d5669c5ad01..c4f013702122 100755 --- a/automation/scripts/containerize +++ b/automation/scripts/containerize @@ -43,7 +43,7 @@ case "_${CONTAINER}" in _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;; _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;; _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;; - _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/suse:opensuse-tumbleweed" ;; + _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;; esac # Use this variable to control whether root should be used
Existing as suse:opensuse-tumbleweed is a historical quirk, and adjusted for consistency with all the other containers. Make it non-root, use heredocs for legibility, and use the zypper long names for the benefit of those wondering what was being referenced or duplicated. Trim the dependencies substantially. Testing docs isn't very interesting and saves a lot of space (pandoc pulls in all of Haskel). Other savings come from removing a huge pile of optional QEMU dependencies (QEMU just needs to build the Xen parts to be useful here, not have a full GUI environment). Finally, there where some packages such as bc, libssh2-devel, libtasn1-devel and nasm that I'm not aware of any reason to have had, even historically. This shaves 1G off the size of the container: registry.gitlab.com/xen-project/xen/opensuse tumbleweed-x86_64 1.39GB registry.gitlab.com/xen-project/xen/suse opensuse-tumbleweed 2.35GB Furthermore, identify which components of the build use which dependencies, which will help managing them in the future. Thanks to Olaf Hering for dependency fixes that have been subsumed into this total overhaul. Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> --- CC: Anthony PERARD <anthony.perard@vates.tech> CC: Juergen Gross <jgross@suse.com> CC: Roger Pau Monné <roger.pau@citrix.com> CC: Jan Beulich <JBeulich@suse.com> CC: Stefano Stabellini <sstabellini@kernel.org> CC: Julien Grall <julien@xen.org> CC: Oleksii Kurochko <oleksii.kurochko@gmail.com> CC: Shawn Anastasio <sanastasio@raptorengineering.com> CC: Olaf Hering <olaf@aepfle.de> This includes both of Olaf's recent changes, python3-packaging (now required for a rebuilt container), and findutils (not required yet; still gets pulled in transitively), but added for good measure. In terms of package delta, newly included are: findutils # As suggested ocaml-findlib # Replaces ocaml-findlib-devel perl # Transitive previously python3 # Transitive previously python3-packaging # As suggested while removed are: bc # ? cmake # vtpm stubdom, optional discount # docs ghostscript # docs glibc-devel # Not really optional, given a toolchain hostname # Optional, not interesting libaio-devel # Xen < 4.13, perhaps Qemu still but optional libext2fs-devel # Libfsimage, off by default, optional libgnutls-devel # Qemu ? libjpeg62-devel # Qemu ? libnl3-devel # Libxl COLO(?), option libnuma-devel # ? libpng16-devel # Qemu ? libssh2-devel # ? libtasn1-devel # ? nasm # ? ocaml-findlib-devel # Swapped for non-devel ocaml-ocamlbuild # ? ocaml-ocamldoc # ? pandoc # docs 'pkgconfig(libpci)' # Qemu 'pkgconfig(sdl)' # Qemu 'pkgconfig(sdl2)' # Qemu systemd-devel # Xen < 4.19, optional transfig # docs valgrind-devel # really only for developers which # Xen < 4.17 I've tested this on staging-4.17 and staging, both of which build fine. staging-4.16 does not, because of SeaBIOS which we've decided not to fix, hence it's safe to drop `which`. --- .../opensuse/tumbleweed-x86_64.dockerfile | 76 +++++++++++++++++++ .../build/suse/opensuse-tumbleweed.dockerfile | 74 ------------------ automation/gitlab-ci/build.yaml | 8 +- automation/scripts/containerize | 2 +- 4 files changed, 81 insertions(+), 79 deletions(-) create mode 100644 automation/build/opensuse/tumbleweed-x86_64.dockerfile delete mode 100644 automation/build/suse/opensuse-tumbleweed.dockerfile