| Message ID | b01494665d1a8cce5c426be70beca2c519215eca.1682369736.git-series.marmarek@invisiblethingslab.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | automation: add PCI passthrough tests on x86 | expand |
On Mon, 24 Apr 2023, Marek Marczykowski-Górecki wrote: > Add simple PCI passthrough test to both PV and HVM domU. It passes > through a network adapter (the only one in the system), gets an IP via > DHCP (first basic test) and then ping the gateway (second basic test). > Finally, if device is supposed to use MSI or MSI-X (as set in the > PCIDEV_INTR test variable), check if it's in use via /proc/interrupts. > > On the current runner, the device in question is this: > 03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03) > Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d25] > Flags: bus master, fast devsel, latency 0, IRQ 18 > Memory at 50400000 (32-bit, non-prefetchable) [size=1M] > Memory at 50500000 (32-bit, non-prefetchable) [size=16K] > Capabilities: [40] Power Management version 3 > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > Capabilities: [a0] Express Endpoint, MSI 00 > Capabilities: [100] Advanced Error Reporting > Capabilities: [140] Device Serial Number ... > Capabilities: [1c0] Latency Tolerance Reporting > Capabilities: [1f0] Precision Time Measurement > Capabilities: [1e0] L1 PM Substates > Kernel driver in use: igc > Kernel modules: igc > > With the current Xen version, it uses MSI-X under PV and MSI under HVM. > > This patch moves domU config to a variable, to make it configurable on > per-test basis. Add also a few comments for visual separation of tests. > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > --- > automation/gitlab-ci/test.yaml | 20 ++++++++- > automation/scripts/qubes-x86-64.sh | 80 ++++++++++++++++++++++++++----- > 2 files changed, 89 insertions(+), 11 deletions(-) > > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > index d68c584269dd..1ce083e6cd88 100644 > --- a/automation/gitlab-ci/test.yaml > +++ b/automation/gitlab-ci/test.yaml > @@ -94,6 +94,8 @@ > # the test controller runs on RPi4 > CONTAINER: alpine:3.12-arm64v8 > LOGFILE: smoke-test.log > + PCIDEV: "03:00.0" > + PCIDEV_INTR: "MSI-X" This is minor but I would move PCIDEV_INTR to adl-pci-pv-x86-64-gcc-debug given that adl-pci-hvm-x86-64-gcc-debug already redefines it. I would also move PCIDEV to adl-pci-pv-x86-64-gcc-debug and adl-pci-hvm-x86-64-gcc-debug, but I am fine either way. However the two new tests failed for me: https://gitlab.com/xen-project/people/sstabellini/xen/-/pipelines/847157948 + grep '^Welcome to Alpine Linux' smoke.serial + '[' 0 -le 0 ] + '[' 0 -le 0 ] + echo 'ERROR: test timeout, aborting' ERROR: test timeout, aborting The Welcome to Alpine Linux message is missing > artifacts: > paths: > - smoke.serial > @@ -147,6 +149,24 @@ adl-suspend-x86-64-gcc-debug: > - *x86-64-test-needs > - alpine-3.12-gcc-debug > > +adl-pci-pv-x86-64-gcc-debug: > + extends: .adl-x86-64 > + script: > + - ./automation/scripts/qubes-x86-64.sh pci-pv 2>&1 | tee ${LOGFILE} > + needs: > + - *x86-64-test-needs > + - alpine-3.12-gcc-debug > + > +adl-pci-hvm-x86-64-gcc-debug: > + extends: .adl-x86-64 > + variables: > + PCIDEV_INTR: "MSI" > + script: > + - ./automation/scripts/qubes-x86-64.sh pci-hvm 2>&1 | tee ${LOGFILE} > + needs: > + - *x86-64-test-needs > + - alpine-3.12-gcc-debug > + > qemu-smoke-dom0-arm64-gcc: > extends: .qemu-arm64 > script: > diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh > index c0bc71764f73..6442f7dda515 100755 > --- a/automation/scripts/qubes-x86-64.sh > +++ b/automation/scripts/qubes-x86-64.sh > @@ -4,8 +4,21 @@ set -ex > > test_variant=$1 > > +### defaults > wait_and_wakeup= > timeout=120 > +domU_config=' > +type = "pvh" > +name = "domU" > +kernel = "/boot/vmlinuz" > +ramdisk = "/boot/initrd-domU" > +extra = "root=/dev/ram0 console=hvc0" > +memory = 512 > +vif = [ "bridge=xenbr0", ] > +disk = [ ] > +' > + > +### test: smoke test > if [ -z "${test_variant}" ]; then > passed="ping test passed" > domU_check=" > @@ -23,6 +36,8 @@ done > tail -n 100 /var/log/xen/console/guest-domU.log > echo \"${passed}\" > " > + > +### test: S3 > elif [ "${test_variant}" = "s3" ]; then > passed="suspend test passed" > wait_and_wakeup="started, suspending" > @@ -48,6 +63,59 @@ xl dmesg | grep 'Finishing wakeup from ACPI S3 state' || exit 1 > ping -c 10 192.168.0.2 || exit 1 > echo \"${passed}\" > " > + > +### test: pci-pv, pci-hvm > +elif [ "${test_variant}" = "pci-pv" ] || [ "${test_variant}" = "pci-hvm" ]; then > + > + if [ -z "$PCIDEV" ]; then > + echo "Please set 'PCIDEV' variable with BDF of test network adapter" >&2 > + echo "Optionally set also 'PCIDEV_INTR' to 'MSI' or 'MSI-X'" >&2 > + exit 1 > + fi > + > + passed="pci test passed" > + > + domU_config=' > +type = "'${test_variant#pci-}'" > +name = "domU" > +kernel = "/boot/vmlinuz" > +ramdisk = "/boot/initrd-domU" > +extra = "root=/dev/ram0 console=hvc0" > +memory = 512 > +vif = [ ] > +disk = [ ] > +pci = [ "'$PCIDEV',seize=1" ] > +on_reboot = "destroy" > +' > + > + domU_check=" > +set -x -e > +ip link set eth0 up > +timeout 30s udhcpc -i eth0 > +pingip=\$(ip -o -4 r show default|cut -f 3 -d ' ') > +ping -c 10 \"\$pingip\" > +echo domU started > +cat /proc/interrupts > +" > + if [ "$PCIDEV_INTR" = "MSI-X" ]; then > + domU_check="$domU_check > +grep -- '\\(-msi-x\\|PCI-MSI-X\\).*eth0' /proc/interrupts > +" > + elif [ "$PCIDEV_INTR" = "MSI" ]; then > + # depending on the kernel version and domain type, the MSI can be > + # marked as '-msi', 'PCI-MSI', or 'PCI-MSI-<SBDF>'; be careful to not match > + # -msi-x nor PCI-MSI-X > + domU_check="$domU_check > +grep -- '\\(-msi\\|PCI-MSI\\( \\|-[^X]\\)\\).*eth0' /proc/interrupts > +" > + fi > + domU_check="$domU_check > +echo \"${passed}\" > +" > + > + dom0_check=" > +tail -n 100 -F /var/log/xen/console/guest-domU.log > +" > fi > > # DomU > @@ -97,17 +165,7 @@ xl create /etc/xen/domU.cfg > ${dom0_check} > " > etc/local.d/xen.start > chmod +x etc/local.d/xen.start > -# just PVH for now > -echo ' > -type = "pvh" > -name = "domU" > -kernel = "/boot/vmlinuz" > -ramdisk = "/boot/initrd-domU" > -extra = "root=/dev/ram0 console=hvc0" > -memory = 512 > -vif = [ "bridge=xenbr0", ] > -disk = [ ] > -' > etc/xen/domU.cfg > +echo "$domU_config" > etc/xen/domU.cfg > > echo "rc_verbose=yes" >> etc/rc.conf > echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons > -- > git-series 0.9.1 >
Hi, Marek, On Mon, Apr 24, 2023 at 4:57 PM Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> wrote: > + elif [ "$PCIDEV_INTR" = "MSI" ]; then > + # depending on the kernel version and domain type, the MSI can be > + # marked as '-msi', 'PCI-MSI', or 'PCI-MSI-<SBDF>'; be careful to not match > + # -msi-x nor PCI-MSI-X > + domU_check="$domU_check > +grep -- '\\(-msi\\|PCI-MSI\\( \\|-[^X]\\)\\).*eth0' /proc/interrupts > +" This will match -msi-x. Do you want to make the first part "-msi "? Regards, Jason
On Mon, Apr 24, 2023 at 08:01:46PM -0700, Stefano Stabellini wrote: > On Mon, 24 Apr 2023, Marek Marczykowski-Górecki wrote: > > Add simple PCI passthrough test to both PV and HVM domU. It passes > > through a network adapter (the only one in the system), gets an IP via > > DHCP (first basic test) and then ping the gateway (second basic test). > > Finally, if device is supposed to use MSI or MSI-X (as set in the > > PCIDEV_INTR test variable), check if it's in use via /proc/interrupts. > > > > On the current runner, the device in question is this: > > 03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03) > > Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d25] > > Flags: bus master, fast devsel, latency 0, IRQ 18 > > Memory at 50400000 (32-bit, non-prefetchable) [size=1M] > > Memory at 50500000 (32-bit, non-prefetchable) [size=16K] > > Capabilities: [40] Power Management version 3 > > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ > > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > > Capabilities: [a0] Express Endpoint, MSI 00 > > Capabilities: [100] Advanced Error Reporting > > Capabilities: [140] Device Serial Number ... > > Capabilities: [1c0] Latency Tolerance Reporting > > Capabilities: [1f0] Precision Time Measurement > > Capabilities: [1e0] L1 PM Substates > > Kernel driver in use: igc > > Kernel modules: igc > > > > With the current Xen version, it uses MSI-X under PV and MSI under HVM. > > > > This patch moves domU config to a variable, to make it configurable on > > per-test basis. Add also a few comments for visual separation of tests. > > > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > > --- > > automation/gitlab-ci/test.yaml | 20 ++++++++- > > automation/scripts/qubes-x86-64.sh | 80 ++++++++++++++++++++++++++----- > > 2 files changed, 89 insertions(+), 11 deletions(-) > > > > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > > index d68c584269dd..1ce083e6cd88 100644 > > --- a/automation/gitlab-ci/test.yaml > > +++ b/automation/gitlab-ci/test.yaml > > @@ -94,6 +94,8 @@ > > # the test controller runs on RPi4 > > CONTAINER: alpine:3.12-arm64v8 > > LOGFILE: smoke-test.log > > + PCIDEV: "03:00.0" > > + PCIDEV_INTR: "MSI-X" > > This is minor but I would move PCIDEV_INTR to > adl-pci-pv-x86-64-gcc-debug given that adl-pci-hvm-x86-64-gcc-debug > already redefines it. The device is MSI-X capable and I'd expect Linux should use MSI-X. My guess is it's using MSI in HVM because MSI-X doesn't work (and I hope my other series will fix that), but I have _not_ verified this theory. If I'm right, I hope the PCIDEV_INTR="MSI" will go away from the HVM test, and it will be testing if MSI-X didn't regressed. > I would also move PCIDEV to adl-pci-pv-x86-64-gcc-debug and > adl-pci-hvm-x86-64-gcc-debug, but I am fine either way. My idea is those both are more of a test target properties, not an individual test properties. > However the two new tests failed for me: > > https://gitlab.com/xen-project/people/sstabellini/xen/-/pipelines/847157948 > > > + grep '^Welcome to Alpine Linux' smoke.serial > + '[' 0 -le 0 ] > + '[' 0 -le 0 ] > + echo 'ERROR: test timeout, aborting' > ERROR: test timeout, aborting > > The Welcome to Alpine Linux message is missing Ah, I forgot to remove debug shell... I'll post v2 once it passes test run (waiting in the queue...).
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml index d68c584269dd..1ce083e6cd88 100644 --- a/automation/gitlab-ci/test.yaml +++ b/automation/gitlab-ci/test.yaml @@ -94,6 +94,8 @@ # the test controller runs on RPi4 CONTAINER: alpine:3.12-arm64v8 LOGFILE: smoke-test.log + PCIDEV: "03:00.0" + PCIDEV_INTR: "MSI-X" artifacts: paths: - smoke.serial @@ -147,6 +149,24 @@ adl-suspend-x86-64-gcc-debug: - *x86-64-test-needs - alpine-3.12-gcc-debug +adl-pci-pv-x86-64-gcc-debug: + extends: .adl-x86-64 + script: + - ./automation/scripts/qubes-x86-64.sh pci-pv 2>&1 | tee ${LOGFILE} + needs: + - *x86-64-test-needs + - alpine-3.12-gcc-debug + +adl-pci-hvm-x86-64-gcc-debug: + extends: .adl-x86-64 + variables: + PCIDEV_INTR: "MSI" + script: + - ./automation/scripts/qubes-x86-64.sh pci-hvm 2>&1 | tee ${LOGFILE} + needs: + - *x86-64-test-needs + - alpine-3.12-gcc-debug + qemu-smoke-dom0-arm64-gcc: extends: .qemu-arm64 script: diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh index c0bc71764f73..6442f7dda515 100755 --- a/automation/scripts/qubes-x86-64.sh +++ b/automation/scripts/qubes-x86-64.sh @@ -4,8 +4,21 @@ set -ex test_variant=$1 +### defaults wait_and_wakeup= timeout=120 +domU_config=' +type = "pvh" +name = "domU" +kernel = "/boot/vmlinuz" +ramdisk = "/boot/initrd-domU" +extra = "root=/dev/ram0 console=hvc0" +memory = 512 +vif = [ "bridge=xenbr0", ] +disk = [ ] +' + +### test: smoke test if [ -z "${test_variant}" ]; then passed="ping test passed" domU_check=" @@ -23,6 +36,8 @@ done tail -n 100 /var/log/xen/console/guest-domU.log echo \"${passed}\" " + +### test: S3 elif [ "${test_variant}" = "s3" ]; then passed="suspend test passed" wait_and_wakeup="started, suspending" @@ -48,6 +63,59 @@ xl dmesg | grep 'Finishing wakeup from ACPI S3 state' || exit 1 ping -c 10 192.168.0.2 || exit 1 echo \"${passed}\" " + +### test: pci-pv, pci-hvm +elif [ "${test_variant}" = "pci-pv" ] || [ "${test_variant}" = "pci-hvm" ]; then + + if [ -z "$PCIDEV" ]; then + echo "Please set 'PCIDEV' variable with BDF of test network adapter" >&2 + echo "Optionally set also 'PCIDEV_INTR' to 'MSI' or 'MSI-X'" >&2 + exit 1 + fi + + passed="pci test passed" + + domU_config=' +type = "'${test_variant#pci-}'" +name = "domU" +kernel = "/boot/vmlinuz" +ramdisk = "/boot/initrd-domU" +extra = "root=/dev/ram0 console=hvc0" +memory = 512 +vif = [ ] +disk = [ ] +pci = [ "'$PCIDEV',seize=1" ] +on_reboot = "destroy" +' + + domU_check=" +set -x -e +ip link set eth0 up +timeout 30s udhcpc -i eth0 +pingip=\$(ip -o -4 r show default|cut -f 3 -d ' ') +ping -c 10 \"\$pingip\" +echo domU started +cat /proc/interrupts +" + if [ "$PCIDEV_INTR" = "MSI-X" ]; then + domU_check="$domU_check +grep -- '\\(-msi-x\\|PCI-MSI-X\\).*eth0' /proc/interrupts +" + elif [ "$PCIDEV_INTR" = "MSI" ]; then + # depending on the kernel version and domain type, the MSI can be + # marked as '-msi', 'PCI-MSI', or 'PCI-MSI-<SBDF>'; be careful to not match + # -msi-x nor PCI-MSI-X + domU_check="$domU_check +grep -- '\\(-msi\\|PCI-MSI\\( \\|-[^X]\\)\\).*eth0' /proc/interrupts +" + fi + domU_check="$domU_check +echo \"${passed}\" +" + + dom0_check=" +tail -n 100 -F /var/log/xen/console/guest-domU.log +" fi # DomU @@ -97,17 +165,7 @@ xl create /etc/xen/domU.cfg ${dom0_check} " > etc/local.d/xen.start chmod +x etc/local.d/xen.start -# just PVH for now -echo ' -type = "pvh" -name = "domU" -kernel = "/boot/vmlinuz" -ramdisk = "/boot/initrd-domU" -extra = "root=/dev/ram0 console=hvc0" -memory = 512 -vif = [ "bridge=xenbr0", ] -disk = [ ] -' > etc/xen/domU.cfg +echo "$domU_config" > etc/xen/domU.cfg echo "rc_verbose=yes" >> etc/rc.conf echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons
Add simple PCI passthrough test to both PV and HVM domU. It passes through a network adapter (the only one in the system), gets an IP via DHCP (first basic test) and then ping the gateway (second basic test). Finally, if device is supposed to use MSI or MSI-X (as set in the PCIDEV_INTR test variable), check if it's in use via /proc/interrupts. On the current runner, the device in question is this: 03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03) Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d25] Flags: bus master, fast devsel, latency 0, IRQ 18 Memory at 50400000 (32-bit, non-prefetchable) [size=1M] Memory at 50500000 (32-bit, non-prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=5 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number ... Capabilities: [1c0] Latency Tolerance Reporting Capabilities: [1f0] Precision Time Measurement Capabilities: [1e0] L1 PM Substates Kernel driver in use: igc Kernel modules: igc With the current Xen version, it uses MSI-X under PV and MSI under HVM. This patch moves domU config to a variable, to make it configurable on per-test basis. Add also a few comments for visual separation of tests. Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> --- automation/gitlab-ci/test.yaml | 20 ++++++++- automation/scripts/qubes-x86-64.sh | 80 ++++++++++++++++++++++++++----- 2 files changed, 89 insertions(+), 11 deletions(-)