| Message ID | 20220319003221.3012314-2-sstabellini@kernel.org (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | automation: qemu32 smoke test | expand |
On Fri, 18 Mar 2022, Stefano Stabellini wrote: > Add a minimal ARM32 smoke test based on qemu-system-arm, as provided by > the test-artifacts qemu container. The minimal test simply boots Xen > (built from previous build stages) and Dom0. The test is fetching the > Dom0 kernel and initrd from Debian Jessie: they work just fine and this > way we don't have to maintain a build for them too. Thanks to the Xen fix recently submitted (https://marc.info/?l=xen-devel&m=164774063802402) I'll be able to update this script to use Debian Bullseye. I am thinking of merging the below directly with this patch. --- automation: upgrade Debian to Bullseye for testing Xen aarch32 Also change initrd. As the new netboot initrd from Debian Bullseye is huge (22MB), use a tiny initrd from Alpine Linux instead (only 2.5MB). Also note that the huge Debian Bullseye initrd would cause QEMU to crash due to the -device loader parameter. Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com> diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh index 162922ace5..d554de7939 100755 --- a/automation/scripts/qemu-smoke-arm32.sh +++ b/automation/scripts/qemu-smoke-arm32.sh @@ -5,11 +5,20 @@ set -ex export DEBIAN_FRONTENT=noninteractive apt-get -qy update apt-get -qy install --no-install-recommends device-tree-compiler \ - curl + curl \ + cpio cd binaries -curl --fail --silent --show-error --location --output vmlinuz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz -curl --fail --silent --show-error --location --output initrd.gz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz +# Use the kernel from Debian +curl --fail --silent --show-error --location --output vmlinuz http://http.us.debian.org/debian/dists/bullseye/main/installer-armhf/current/images/netboot/vmlinuz +# Use a tiny initrd based on busybox from Alpine Linux +curl --fail --silent --show-error --location --output initrd.tar.gz https://dl-cdn.alpinelinux.org/alpine/v3.15/releases/armhf/alpine-minirootfs-3.15.1-armhf.tar.gz + +mkdir rootfs +cd rootfs +tar xvzf ../initrd.tar.gz +find . | cpio -H newc -o | gzip > ../initrd.gz +cd .. kernel=`stat -L --printf="%s" vmlinuz` initrd=`stat -L --printf="%s" initrd.gz` @@ -68,5 +77,5 @@ timeout -k 1 240 \ -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial set -e -(grep -q "^BusyBox" smoke.serial) || exit 1 +(grep -q "^/ #" smoke.serial) || exit 1 exit 0
Hi, On 20/03/2022 01:46, Stefano Stabellini wrote: > On Fri, 18 Mar 2022, Stefano Stabellini wrote: >> Add a minimal ARM32 smoke test based on qemu-system-arm, as provided by >> the test-artifacts qemu container. The minimal test simply boots Xen >> (built from previous build stages) and Dom0. The test is fetching the >> Dom0 kernel and initrd from Debian Jessie: they work just fine and this >> way we don't have to maintain a build for them too. > > > Thanks to the Xen fix recently submitted > (https://marc.info/?l=xen-devel&m=164774063802402) I'll be able to > update this script to use Debian Bullseye. I am thinking of merging the > below directly with this patch. > > > --- > > automation: upgrade Debian to Bullseye for testing Xen aarch32 > > Also change initrd. As the new netboot initrd from Debian Bullseye is > huge (22MB), use a tiny initrd from Alpine Linux instead (only 2.5MB). This is sounds odd to me. So we are going to use Bullseye but not really because we want to use a different initrd. Why can't you get everything from the same place? > > Also note that the huge Debian Bullseye initrd would cause QEMU to > crash due to the -device loader parameter. Can you provide more details? Was this reported to QEMU? > > Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com> > > diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh > index 162922ace5..d554de7939 100755 > --- a/automation/scripts/qemu-smoke-arm32.sh > +++ b/automation/scripts/qemu-smoke-arm32.sh > @@ -5,11 +5,20 @@ set -ex > export DEBIAN_FRONTENT=noninteractive > apt-get -qy update > apt-get -qy install --no-install-recommends device-tree-compiler \ > - curl > + curl \ > + cpio > > cd binaries > -curl --fail --silent --show-error --location --output vmlinuz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz > -curl --fail --silent --show-error --location --output initrd.gz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz > +# Use the kernel from Debian > +curl --fail --silent --show-error --location --output vmlinuz http://http.us.debian.org/debian/dists/bullseye/main/installer-armhf/current/images/netboot/vmlinuz > +# Use a tiny initrd based on busybox from Alpine Linux > +curl --fail --silent --show-error --location --output initrd.tar.gz https://dl-cdn.alpinelinux.org/alpine/v3.15/releases/armhf/alpine-minirootfs-3.15.1-armhf.tar.gz > + > +mkdir rootfs > +cd rootfs > +tar xvzf ../initrd.tar.gz > +find . | cpio -H newc -o | gzip > ../initrd.gz > +cd .. > > kernel=`stat -L --printf="%s" vmlinuz` > initrd=`stat -L --printf="%s" initrd.gz` > @@ -68,5 +77,5 @@ timeout -k 1 240 \ > -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial > > set -e > -(grep -q "^BusyBox" smoke.serial) || exit 1 > +(grep -q "^/ #" smoke.serial) || exit 1 > exit 0 > Cheers,
On Sun, 20 Mar 2022, Julien Grall wrote: > On 20/03/2022 01:46, Stefano Stabellini wrote: > > On Fri, 18 Mar 2022, Stefano Stabellini wrote: > > > Add a minimal ARM32 smoke test based on qemu-system-arm, as provided by > > > the test-artifacts qemu container. The minimal test simply boots Xen > > > (built from previous build stages) and Dom0. The test is fetching the > > > Dom0 kernel and initrd from Debian Jessie: they work just fine and this > > > way we don't have to maintain a build for them too. > > > > > > Thanks to the Xen fix recently submitted > > (https://marc.info/?l=xen-devel&m=164774063802402) I'll be able to > > update this script to use Debian Bullseye. I am thinking of merging the > > below directly with this patch. > > > > > > --- > > > > automation: upgrade Debian to Bullseye for testing Xen aarch32 > > > > Also change initrd. As the new netboot initrd from Debian Bullseye is > > huge (22MB), use a tiny initrd from Alpine Linux instead (only 2.5MB). > > This is sounds odd to me. So we are going to use Bullseye but not really > because we want to use a different initrd. > > Why can't you get everything from the same place? Because it doesn't work :-( > > Also note that the huge Debian Bullseye initrd would cause QEMU to > > crash due to the -device loader parameter. > > Can you provide more details? Was this reported to QEMU? QEMU core dumps when provided with the Debian Bullseye initrd binary to load. I guessed it was due to the size and tried with a smaller size. Everything worked with a smaller initrd. I also think that it is not a good idea to use a 22MB initrd anyway so decided against the Debian Bullseye initrd. (For reference 22MB is basically the size of a fully featured Yocto-build rootfs.) I did not file a bug to qemu-devel yet and didn't investigate further on the QEMU side as I ran out of time. Alpine Linux provides a very nice 2.5MB initrd. I tried to use both kernel and initrd from Alpine Linux but unfortunately the Alpine Linux kernel doesn't boot. I don't know why but I think it is because it might be missing the console driver. I am not sure. There are a lot of combinations that don't work and it is time consuming to investigate them all. I have been trying to investigate only the most critical things -- they are too many! I should add that the Debian initrd is not the ideal initrd because it is made to start the Debian installer. Here we just want a tiny busybox initrd. In general, I think it would be better if we could use the kernel and initrd from the same source but I couldn't find one that works. I could build one myself but it would be one more thing to maintain in gitlab-ci. Also using u-boot might solve the problem of loading the binary but again we would have to maintain a u-boot arm32 build in gitlab-ci. So in order of preference best to worst in my opinion: 1) kernel and initrd from the same source 2) kernel and initrd from different sources 3) build your own kernel/initrd/u-boot So I ended up doing 2). I tested it and it is sufficient to get the test up and running. > > Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com> > > > > diff --git a/automation/scripts/qemu-smoke-arm32.sh > > b/automation/scripts/qemu-smoke-arm32.sh > > index 162922ace5..d554de7939 100755 > > --- a/automation/scripts/qemu-smoke-arm32.sh > > +++ b/automation/scripts/qemu-smoke-arm32.sh > > @@ -5,11 +5,20 @@ set -ex > > export DEBIAN_FRONTENT=noninteractive > > apt-get -qy update > > apt-get -qy install --no-install-recommends device-tree-compiler \ > > - curl > > + curl \ > > + cpio > > cd binaries > > -curl --fail --silent --show-error --location --output vmlinuz > > http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz > > -curl --fail --silent --show-error --location --output initrd.gz > > http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz > > +# Use the kernel from Debian > > +curl --fail --silent --show-error --location --output vmlinuz > > http://http.us.debian.org/debian/dists/bullseye/main/installer-armhf/current/images/netboot/vmlinuz > > +# Use a tiny initrd based on busybox from Alpine Linux > > +curl --fail --silent --show-error --location --output initrd.tar.gz > > https://dl-cdn.alpinelinux.org/alpine/v3.15/releases/armhf/alpine-minirootfs-3.15.1-armhf.tar.gz > > + > > +mkdir rootfs > > +cd rootfs > > +tar xvzf ../initrd.tar.gz > > +find . | cpio -H newc -o | gzip > ../initrd.gz > > +cd .. > > kernel=`stat -L --printf="%s" vmlinuz` > > initrd=`stat -L --printf="%s" initrd.gz` > > @@ -68,5 +77,5 @@ timeout -k 1 240 \ > > -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial > > set -e > > -(grep -q "^BusyBox" smoke.serial) || exit 1 > > +(grep -q "^/ #" smoke.serial) || exit 1 > > exit 0
Hi Stefano, On 21/03/2022 19:47, Stefano Stabellini wrote: > On Sun, 20 Mar 2022, Julien Grall wrote: >> On 20/03/2022 01:46, Stefano Stabellini wrote: >>> On Fri, 18 Mar 2022, Stefano Stabellini wrote: >>>> Add a minimal ARM32 smoke test based on qemu-system-arm, as provided by >>>> the test-artifacts qemu container. The minimal test simply boots Xen >>>> (built from previous build stages) and Dom0. The test is fetching the >>>> Dom0 kernel and initrd from Debian Jessie: they work just fine and this >>>> way we don't have to maintain a build for them too. >>> >>> >>> Thanks to the Xen fix recently submitted >>> (https://marc.info/?l=xen-devel&m=164774063802402) I'll be able to >>> update this script to use Debian Bullseye. I am thinking of merging the >>> below directly with this patch. >>> >>> >>> --- >>> >>> automation: upgrade Debian to Bullseye for testing Xen aarch32 >>> >>> Also change initrd. As the new netboot initrd from Debian Bullseye is >>> huge (22MB), use a tiny initrd from Alpine Linux instead (only 2.5MB). >> >> This is sounds odd to me. So we are going to use Bullseye but not really >> because we want to use a different initrd. >> >> Why can't you get everything from the same place? > > Because it doesn't work :-( > > >>> Also note that the huge Debian Bullseye initrd would cause QEMU to >>> crash due to the -device loader parameter. >> >> Can you provide more details? Was this reported to QEMU? > > QEMU core dumps when provided with the Debian Bullseye initrd binary to > load. I guessed it was due to the size and tried with a smaller size. > Everything worked with a smaller initrd. I also think that it is not a > good idea to use a 22MB initrd anyway so decided against the Debian > Bullseye initrd. Why is it a bad idea? In general, bigger file allows us to test corner cases. > (For reference 22MB is basically the size of a fully > featured Yocto-build rootfs.) I did not file a bug to qemu-devel yet and > didn't investigate further on the QEMU side as I ran out of time. > > Alpine Linux provides a very nice 2.5MB initrd. I tried to use both > kernel and initrd from Alpine Linux but unfortunately the Alpine Linux > kernel doesn't boot. I don't know why but I think it is because it might > be missing the console driver. I am not sure. There are a lot of > combinations that don't work and it is time consuming to investigate > them all. I have been trying to investigate only the most critical > things -- they are too many! > > I should add that the Debian initrd is not the ideal initrd because it > is made to start the Debian installer. Here we just want a tiny busybox > initrd. > > In general, I think it would be better if we could use the kernel and > initrd from the same source but I couldn't find one that works. I could > build one myself but it would be one more thing to maintain in > gitlab-ci. Also using u-boot might solve the problem of loading the > binary but again we would have to maintain a u-boot arm32 build in > gitlab-ci. > > So in order of preference best to worst in my opinion: > > 1) kernel and initrd from the same source > 2) kernel and initrd from different sources > 3) build your own kernel/initrd/u-boot > > So I ended up doing 2). I tested it and it is sufficient to get the test > up and running. Thanks for the explanation. So I think we should not call that an "Upgrade to Bullseye" because you are not using Debian. Instead, you borrowed a working kernel that happens to have everything you need built-in. Also, can you update the commit message and provide a summary of this discussion? Cheers,
On Mon, 21 Mar 2022, Julien Grall wrote: > On 21/03/2022 19:47, Stefano Stabellini wrote: > > On Sun, 20 Mar 2022, Julien Grall wrote: > > > On 20/03/2022 01:46, Stefano Stabellini wrote: > > > > On Fri, 18 Mar 2022, Stefano Stabellini wrote: > > > > > Add a minimal ARM32 smoke test based on qemu-system-arm, as provided > > > > > by > > > > > the test-artifacts qemu container. The minimal test simply boots Xen > > > > > (built from previous build stages) and Dom0. The test is fetching the > > > > > Dom0 kernel and initrd from Debian Jessie: they work just fine and > > > > > this > > > > > way we don't have to maintain a build for them too. > > > > > > > > > > > > Thanks to the Xen fix recently submitted > > > > (https://marc.info/?l=xen-devel&m=164774063802402) I'll be able to > > > > update this script to use Debian Bullseye. I am thinking of merging the > > > > below directly with this patch. > > > > > > > > > > > > --- > > > > > > > > automation: upgrade Debian to Bullseye for testing Xen aarch32 > > > > > > > > Also change initrd. As the new netboot initrd from Debian Bullseye is > > > > huge (22MB), use a tiny initrd from Alpine Linux instead (only 2.5MB). > > > > > > This is sounds odd to me. So we are going to use Bullseye but not really > > > because we want to use a different initrd. > > > > > > Why can't you get everything from the same place? > > > > Because it doesn't work :-( > > > > > > > > Also note that the huge Debian Bullseye initrd would cause QEMU to > > > > crash due to the -device loader parameter. > > > > > > Can you provide more details? Was this reported to QEMU? > > > > QEMU core dumps when provided with the Debian Bullseye initrd binary to > > load. I guessed it was due to the size and tried with a smaller size. > > Everything worked with a smaller initrd. I also think that it is not a > > good idea to use a 22MB initrd anyway so decided against the Debian > > Bullseye initrd. > > Why is it a bad idea? In general, bigger file allows us to test corner cases. That is also true. This test is minimal, there is only dom0 booting, no domUs. To me, it makes sense that also the initrd is small. In general 20-25MB is the regular full size of a Linux arm64 rootfs. I think it makes sense to stay below 10-15MB for arm32 if we can. We could go up in size if we added the Xen tools and a domU to the initrd and tested xl create. There is a test like that for arm64. We can add more tests like that. > > (For reference 22MB is basically the size of a fully > > featured Yocto-build rootfs.) I did not file a bug to qemu-devel yet and > > didn't investigate further on the QEMU side as I ran out of time. > > > > Alpine Linux provides a very nice 2.5MB initrd. I tried to use both > > kernel and initrd from Alpine Linux but unfortunately the Alpine Linux > > kernel doesn't boot. I don't know why but I think it is because it might > > be missing the console driver. I am not sure. There are a lot of > > combinations that don't work and it is time consuming to investigate > > them all. I have been trying to investigate only the most critical > > things -- they are too many! > > > > I should add that the Debian initrd is not the ideal initrd because it > > is made to start the Debian installer. Here we just want a tiny busybox > > initrd. > > > > In general, I think it would be better if we could use the kernel and > > initrd from the same source but I couldn't find one that works. I could > > build one myself but it would be one more thing to maintain in > > gitlab-ci. Also using u-boot might solve the problem of loading the > > binary but again we would have to maintain a u-boot arm32 build in > > gitlab-ci. > > > > So in order of preference best to worst in my opinion: > > > > 1) kernel and initrd from the same source > > 2) kernel and initrd from different sources > > 3) build your own kernel/initrd/u-boot > > > > So I ended up doing 2). I tested it and it is sufficient to get the test > > up and running. > > Thanks for the explanation. So I think we should not call that an "Upgrade to > Bullseye" because you are not using Debian. Instead, you borrowed a working > kernel that happens to have everything you need built-in. > > Also, can you update the commit message and provide a summary of this > discussion? Yes I can do that, thanks for the review!
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml index ec2a2e1607..42cd725a12 100644 --- a/automation/gitlab-ci/test.yaml +++ b/automation/gitlab-ci/test.yaml @@ -95,6 +95,29 @@ qemu-smoke-arm64-gcc: - /^coverity-tested\/.*/ - /^stable-.*/ +qemu-smoke-arm32-gcc: + stage: test + image: registry.gitlab.com/xen-project/xen/${CONTAINER} + variables: + CONTAINER: debian:unstable-arm64v8 + script: + - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log + dependencies: + - debian-unstable-gcc-arm32 + - qemu-system-aarch64-6.0.0-arm32-export + artifacts: + paths: + - smoke.serial + - '*.log' + when: always + tags: + - arm64 + except: + - master + - smoke + - /^coverity-tested\/.*/ + - /^stable-.*/ + qemu-smoke-x86-64-gcc: stage: test image: registry.gitlab.com/xen-project/xen/${CONTAINER} diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh new file mode 100755 index 0000000000..162922ace5 --- /dev/null +++ b/automation/scripts/qemu-smoke-arm32.sh @@ -0,0 +1,72 @@ +#!/bin/bash + +set -ex + +export DEBIAN_FRONTENT=noninteractive +apt-get -qy update +apt-get -qy install --no-install-recommends device-tree-compiler \ + curl + +cd binaries +curl --fail --silent --show-error --location --output vmlinuz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz +curl --fail --silent --show-error --location --output initrd.gz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz + +kernel=`stat -L --printf="%s" vmlinuz` +initrd=`stat -L --printf="%s" initrd.gz` + +# For Xen, we need a couple of more node. Dump the DT from QEMU and add them +# XXX QEMU looks for "efi-virtio.rom" even if it is unneeded +curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom +./qemu-system-arm \ + -machine virt-6.0 \ + -machine virtualization=true \ + -smp 4 \ + -m 1024 \ + -serial stdio \ + -monitor none \ + -display none \ + -machine dumpdtb=virt.dtb + +dtc -I dtb -O dts virt.dtb > virt.dts + +cat >> virt.dts << EOF +/ { + chosen { + #address-cells = <0x2>; + #size-cells = <0x2>; + stdout-path = "/pl011@9000000"; + xen,xen-bootargs = "console=dtuart dtuart=/pl011@9000000 dom0_mem=512M bootscrub=0"; + xen,dom0-bootargs = "console=tty0 console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh init=/bin/sh"; + dom0 { + compatible = "xen,linux-zimage", "xen,multiboot-module"; + reg = <0x0 0x1000000 0x0 $kernel>; + }; + dom0-ramdisk { + compatible = "xen,linux-initrd", "xen,multiboot-module"; + reg = <0x0 0x3200000 0x0 $initrd>; + }; + }; +}; +EOF +dtc -I dts -O dtb virt.dts > virt.dtb + +rm -f smoke.serial +set +e +timeout -k 1 240 \ +./qemu-system-arm \ + -machine virt-6.0 \ + -machine virtualization=true \ + -smp 4 \ + -m 1024 \ + -serial stdio \ + -monitor none \ + -display none \ + -dtb virt.dtb \ + -no-reboot \ + -kernel ./xen \ + -device loader,file=./vmlinuz,addr=0x1000000 \ + -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial + +set -e +(grep -q "^BusyBox" smoke.serial) || exit 1 +exit 0
Add a minimal ARM32 smoke test based on qemu-system-arm, as provided by the test-artifacts qemu container. The minimal test simply boots Xen (built from previous build stages) and Dom0. The test is fetching the Dom0 kernel and initrd from Debian Jessie: they work just fine and this way we don't have to maintain a build for them too. Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com> --- Changes in v3: - use long options for curl - use curl --output Changes in v2: - improve comments - don't limit dom0 cpus - decrease mem to 1024M and dom0_mem=512M - use Debian Bullseye instead of Jessie --- automation/gitlab-ci/test.yaml | 23 ++++++++ automation/scripts/qemu-smoke-arm32.sh | 72 ++++++++++++++++++++++++++ 2 files changed, 95 insertions(+) create mode 100755 automation/scripts/qemu-smoke-arm32.sh