[RFC,v2] drm/msm: Add initial ci/ subdirectory

Message ID	20220510141329.54414-1-tomeu.vizoso@collabora.com (mailing list archive)
State	New, archived
Headers	show Return-Path: <dri-devel-bounces@lists.freedesktop.org> From: Tomeu Vizoso <tomeu.vizoso@collabora.com> To: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>, Maxime Ripard <mripard@kernel.org>, Thomas Zimmermann <tzimmermann@suse.de>, David Airlie <airlied@linux.ie>, Daniel Vetter <daniel@ffwll.ch>, Jonathan Corbet <corbet@lwn.net>, Rob Clark <robdclark@gmail.com>, Sean Paul <sean@poorly.run>, Abhinav Kumar <quic_abhinavk@quicinc.com> Subject: [RFC v2] drm/msm: Add initial ci/ subdirectory Date: Tue, 10 May 2022 16:13:28 +0200 Message-Id: <20220510141329.54414-1-tomeu.vizoso@collabora.com> In-Reply-To: <20220510070140.45407-1-tomeu.vizoso@collabora.com> References: <20220510070140.45407-1-tomeu.vizoso@collabora.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: list Cc: Tomeu Vizoso <tomeu.vizoso@collabora.com>, linux-doc@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, freedreno@lists.freedesktop.org Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" <dri-devel-bounces@lists.freedesktop.org>
Series	[RFC,v2] drm/msm: Add initial ci/ subdirectory \| expand [RFC,v2] drm/msm: Add initial ci/ subdirectory

Tomeu Vizoso May 10, 2022, 2:13 p.m. UTC

And use it to store expectations about what the drm/msm driver is
supposed to pass in the IGT test suite.

Also include a configuration file that points to the out-of-tree CI
scripts.

By storing the test expectations along the code we can make sure both
stay in sync with each other, and so we can know when a code change
breaks those expectations.

This will allow all contributors to drm/msm to reuse the infrastructure
already in gitlab.freedesktop.org to test the driver on several
generations of the hardware.

v2:
  - Fix names of result expectation files to match SoC
  - Don't execute tests that are going to skip on all boards

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
---
 Documentation/gpu/msm_automated_testing.rst   |  70 +++++++++
 drivers/gpu/drm/msm/ci/gitlab-ci.yml          |  11 ++
 drivers/gpu/drm/msm/ci/msm.testlist           | 148 ++++++++++++++++++
 .../gpu/drm/msm/ci/msm_apq8016_results.txt    | 140 +++++++++++++++++
 .../gpu/drm/msm/ci/msm_apq8096_results.txt    | 140 +++++++++++++++++
 drivers/gpu/drm/msm/ci/msm_sc7180_results.txt | 141 +++++++++++++++++
 drivers/gpu/drm/msm/ci/msm_sdm845_results.txt | 141 +++++++++++++++++
 7 files changed, 791 insertions(+)
 create mode 100644 Documentation/gpu/msm_automated_testing.rst
 create mode 100644 drivers/gpu/drm/msm/ci/gitlab-ci.yml
 create mode 100644 drivers/gpu/drm/msm/ci/msm.testlist
 create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
 create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
 create mode 100644 drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
 create mode 100644 drivers/gpu/drm/msm/ci/msm_sdm845_results.txt

Jessica Zhang May 10, 2022, 7:39 p.m. UTC | #1

On 5/10/2022 7:13 AM, Tomeu Vizoso wrote:
> And use it to store expectations about what the drm/msm driver is
> supposed to pass in the IGT test suite.
> 
> Also include a configuration file that points to the out-of-tree CI
> scripts.
> 
> By storing the test expectations along the code we can make sure both
> stay in sync with each other, and so we can know when a code change
> breaks those expectations.
> 
> This will allow all contributors to drm/msm to reuse the infrastructure
> already in gitlab.freedesktop.org to test the driver on several
> generations of the hardware.
> 
> v2:
>    - Fix names of result expectation files to match SoC
>    - Don't execute tests that are going to skip on all boards
> 
> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> ---
>   Documentation/gpu/msm_automated_testing.rst   |  70 +++++++++
>   drivers/gpu/drm/msm/ci/gitlab-ci.yml          |  11 ++
>   drivers/gpu/drm/msm/ci/msm.testlist           | 148 ++++++++++++++++++
>   .../gpu/drm/msm/ci/msm_apq8016_results.txt    | 140 +++++++++++++++++
>   .../gpu/drm/msm/ci/msm_apq8096_results.txt    | 140 +++++++++++++++++
>   drivers/gpu/drm/msm/ci/msm_sc7180_results.txt | 141 +++++++++++++++++
>   drivers/gpu/drm/msm/ci/msm_sdm845_results.txt | 141 +++++++++++++++++
>   7 files changed, 791 insertions(+)
>   create mode 100644 Documentation/gpu/msm_automated_testing.rst
>   create mode 100644 drivers/gpu/drm/msm/ci/gitlab-ci.yml
>   create mode 100644 drivers/gpu/drm/msm/ci/msm.testlist
>   create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
>   create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
>   create mode 100644 drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
>   create mode 100644 drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> 
> diff --git a/Documentation/gpu/msm_automated_testing.rst b/Documentation/gpu/msm_automated_testing.rst
> new file mode 100644
> index 000000000000..fe59474569c2
> --- /dev/null
> +++ b/Documentation/gpu/msm_automated_testing.rst
> @@ -0,0 +1,70 @@
> +.. SPDX-License-Identifier: GPL-2.0+
> +
> +=========================================
> +Automated testing of MSM DRM driver
> +=========================================

Hey Tomeu,

I like the overall idea of having this in as part of the repo, but just 
have a few questions on some details.

> +
> +
> +Introduction
> +============
> +
> +Making sure that changes to a driver don't introduce regressions can be very time consuming when lots of different hardware configurations need to be tested. Moreover, it isn't practical for each person interested in this testing to have to acquire and maintain what can be a considerable amount of hardware.
> +
> +Also, it is desirable for developers to check for regressions in their code by themselves, instead of relying on the maintainer finding them and then reporting back.
> +
> +There are facilities in gitlab.freedesktop.org to automatically test Mesa that can be used as well for testing DRM drivers such as drm/msm. This document explains how people interested in testing the drm/msm driver can use this shared infrastructure to save quite some time and effort.
> +
> +
> +Relevant files
> +==============
> +
> +drivers/gpu/drm/msm/ci/gitlab-ci.yml
> +------------------------------------
> +
> +Specifies the specific version of the scripts to be used. GitLab CI will use the values defined in this file to fetch the right scripts.
> +
> +
> +drivers/gpu/drm/msm/ci/msm.testlist
> +-----------------------------------
> +
> +Specifies the tests that the current code is expected to be able to reliably run. These tests are expected to not hang the DUT (device under testing) when running on the revision they belong to, and to give consistent results.
> +
> +
> +drivers/gpu/drm/msm/ci/msm_*_results.txt
> +----------------------------------------
> +
> +Specifies the expected results of running this specific kernel revision on a given hardware configuration.
> +
> +
> +How to enable automated testing on your tree
> +============================================
> +
> +1. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one yet
> +
> +2. In your kernel repo's configuration (eg. https://gitlab.freedesktop.org/tomeu/linux/-/settings/ci_cd), change the CI/CD configuration file from .gitlab-ci.yml to drivers/gpu/drm/msm/ci/gitlab-ci.yml.
> +
> +3. Next time you push to this repository, you will see a CI pipeline being created (eg. https://gitlab.freedesktop.org/tomeu/linux/-/pipelines)
> +
> +4. The various jobs will be run and when the pipeline is finished, all jobs should be green unless a regression has been found.
> +
> +
> +How to update test expectations
> +===============================
> +
> +If your changes to the code fix any tests, you will have to update one or more of the files in drivers/gpu/drm/msm/ci/msm_*_results.txt, for each of the test platforms affected by the change.
> +
> +If you have run a pipeline and it failed because of a mismatched test result, you will have been offered a patch at the end of the run that you can apply after reviewing it.
> +
> +
> +How to expand coverage
> +======================
> +
> +If your code changes makes it possible to run more tests (by solving reliability issues, for example), more tests can be added to this list, and then the expected results updated with the patch file that will be printed at the end of each job.
> +
> +If there is a need for updating the version of IGT being used (maybe you have added more tests to it), follow the instructions at https://gitlab.freedesktop.org/gfx-ci/drm-ci/-/tree/msm#upreving-igt.
> +
> +
> +How to test your changes to the scripts
> +==========================================
> +
> +For testing changes to the scripts in the drm-ci repo, change the DRM_CI_PROJECT_PATH variable in drivers/gpu/drm/msm/ci/gitlab-ci.yml to match your fork of the project (eg. tomeu/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/.
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/msm/ci/gitlab-ci.yml b/drivers/gpu/drm/msm/ci/gitlab-ci.yml
> new file mode 100644
> index 000000000000..9b7caa7fcab2
> --- /dev/null
> +++ b/drivers/gpu/drm/msm/ci/gitlab-ci.yml
> @@ -0,0 +1,11 @@
> +variables:
> +  # Change this to use your fork of drm-ci
> +  DRM_CI_PROJECT_PATH: &drm-ci-project-path gfx-ci/drm-ci
> +  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha msm
> +
> +include:
> +  - project: *drm-ci-project-path
> +    ref: *drm-ci-commit-sha
> +    file:
> +      - '.msm-gitlab-ci.yml'
> +      - '.gitlab-ci.yml'
> diff --git a/drivers/gpu/drm/msm/ci/msm.testlist b/drivers/gpu/drm/msm/ci/msm.testlist
> new file mode 100644
> index 000000000000..8805a3bc7316
> --- /dev/null
> +++ b/drivers/gpu/drm/msm/ci/msm.testlist
> @@ -0,0 +1,148 @@
> +# Keep alphabetically sorted by default
> +
> +igt@core_auth@getclient-simple
> +igt@core_auth@getclient-master-drop
> +igt@core_auth@basic-auth
> +igt@core_auth@many-magics
> +igt@core_getclient
> +igt@core_getstats
> +igt@core_getversion
> +igt@core_setmaster_vs_auth
> +igt@drm_read@invalid-buffer
> +igt@drm_read@fault-buffer
> +igt@drm_read@empty-block
> +igt@drm_read@empty-nonblock
> +igt@drm_read@short-buffer-block
> +igt@drm_read@short-buffer-nonblock
> +igt@drm_read@short-buffer-wakeup
> +igt@kms_addfb_basic@unused-handle
> +igt@kms_addfb_basic@unused-pitches
> +igt@kms_addfb_basic@unused-offsets
> +igt@kms_addfb_basic@unused-modifier
> +igt@kms_addfb_basic@legacy-format
> +igt@kms_addfb_basic@no-handle
> +igt@kms_addfb_basic@basic
> +igt@kms_addfb_basic@bad-pitch-0
> +igt@kms_addfb_basic@bad-pitch-32
> +igt@kms_addfb_basic@bad-pitch-63
> +igt@kms_addfb_basic@bad-pitch-128
> +igt@kms_addfb_basic@bad-pitch-256
> +igt@kms_addfb_basic@bad-pitch-1024
> +igt@kms_addfb_basic@bad-pitch-999
> +igt@kms_addfb_basic@bad-pitch-65536
> +igt@kms_addfb_basic@size-max
> +igt@kms_addfb_basic@too-wide
> +igt@kms_addfb_basic@too-high
> +igt@kms_addfb_basic@bo-too-small
> +igt@kms_addfb_basic@small-bo
> +igt@kms_addfb_basic@addfb25-modifier-no-flag
> +igt@kms_addfb_basic@addfb25-bad-modifier
> +igt@kms_addfb_basic@invalid-get-prop-any
> +igt@kms_addfb_basic@invalid-get-prop
> +igt@kms_addfb_basic@invalid-set-prop-any
> +igt@kms_addfb_basic@invalid-set-prop
> +igt@kms_addfb_basic@master-rmfb
> +igt@kms_atomic@plane-overlay-legacy
> +igt@kms_atomic@plane-primary-legacy
> +igt@kms_atomic@plane-primary-overlay-mutable-zpos
> +igt@kms_atomic@plane-immutable-zpos
> +igt@kms_atomic@test-only
> +igt@kms_atomic@plane-cursor-legacy
> +igt@kms_atomic@plane-invalid-params
> +igt@kms_atomic@crtc-invalid-params
> +igt@kms_atomic@atomic-invalid-params
> +igt@kms_atomic@atomic_plane_damage
> +igt@kms_atomic_interruptible@legacy-setmode@pipe-a-edp-1
> +igt@kms_atomic_interruptible@atomic-setmode@pipe-a-edp-1
> +igt@kms_atomic_interruptible@legacy-dpms@pipe-a-edp-1
> +igt@kms_atomic_interruptible@legacy-pageflip@pipe-a-edp-1
> +igt@kms_atomic_interruptible@legacy-cursor@pipe-a-edp-1
> +igt@kms_atomic_interruptible@universal-setplane-primary@pipe-a-edp-1
> +igt@kms_atomic_interruptible@universal-setplane-cursor@pipe-a-edp-1

I see that there are some KMS subtests/tests are missing such as 
kms_atomic_transition -- are they unstable across all boards?

 From what I've seen, kms_atomic_transition is unstable only on Trogdor, 
but has stable results on other boards.

> +igt@kms_content_protection@LIC > +igt@kms_flip_event_leak
> +igt@kms_getfb@getfb-handle-zero
> +igt@kms_getfb@getfb-handle-valid
> +igt@kms_getfb@getfb-handle-closed
> +igt@kms_getfb@getfb-handle-not-fb
> +igt@kms_getfb@getfb-addfb-different-handles
> +igt@kms_getfb@getfb-repeated-different-handles
> +igt@kms_getfb@getfb2-handle-zero
> +igt@kms_getfb@getfb2-handle-closed
> +igt@kms_getfb@getfb2-handle-not-fb
> +igt@kms_getfb@getfb2-into-addfb2
> +igt@kms_getfb@getfb-handle-protection
> +igt@kms_getfb@getfb2-handle-protection
> +igt@kms_hdmi_inject@inject-4k > +igt@kms_multipipe_modeset@basic-max-pipe-crc-check
> +igt@kms_pipe_crc_basic@bad-source
> +igt@kms_pipe_crc_basic@read-crc-pipe-a
> +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a
> +igt@kms_pipe_crc_basic@read-crc-pipe-b
> +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b
> +igt@kms_prop_blob@basic
> +igt@kms_prop_blob@blob-prop-core
> +igt@kms_prop_blob@blob-prop-validate
> +igt@kms_prop_blob@blob-prop-lifetime
> +igt@kms_prop_blob@blob-multiple
> +igt@kms_prop_blob@invalid-get-prop-any
> +igt@kms_prop_blob@invalid-get-prop
> +igt@kms_prop_blob@invalid-set-prop-any
> +igt@kms_prop_blob@invalid-set-prop
> +igt@kms_rmfb@rmfb-ioctl
> +igt@kms_rmfb@close-fd
> +igt@kms_setmode@basic@pipe-a-edp-1
> +igt@kms_setmode@basic@pipe-b-edp-1
> +
> +# Flakes on a530 with:
> +# arm-smmu b40000.iommu: Unhandled context fault: fsr=0x408, iova=0x01003000, fsynr=0x13, cbfrsynra=0x0, cb=0
> +# igt@msm_mapping@ring
> +
> +igt@msm_mapping@sqefw
> +igt@msm_mapping@shadow
> +igt@msm_recovery@hangcheck
> +igt@msm_recovery@gpu-fault
> +igt@msm_recovery@iova-fault
> +igt@msm_submit@empty-submit
> +igt@msm_submit@invalid-queue-submit
> +igt@msm_submit@invalid-flags-submit
> +igt@msm_submit@invalid-in-fence-submit
> +igt@msm_submit@invalid-duplicate-bo-submit
> +igt@msm_submit@invalid-cmd-idx-submit
> +igt@msm_submit@invalid-cmd-type-submit
> +igt@msm_submit@valid-submit
> +igt@kms_sysfs_edid_timing
> +igt@kms_universal_plane@universal-plane-pipe-a-functional
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b
> +igt@kms_vblank@invalid
> +igt@kms_vblank@crtc-id
> +igt@kms_vblank@pipe-a-query-idle
> +igt@kms_vblank@pipe-a-query-forked
> +igt@kms_vblank@pipe-a-query-busy
> +igt@kms_vblank@pipe-a-query-forked-busy
> +igt@kms_vblank@pipe-a-wait-idle
> +igt@kms_vblank@pipe-a-wait-forked
> +igt@kms_vblank@pipe-a-wait-busy
> +igt@kms_vblank@pipe-a-wait-forked-busy
> +igt@kms_vblank@pipe-a-ts-continuation-idle
> +igt@kms_vblank@pipe-a-ts-continuation-modeset
> +igt@kms_vblank@pipe-b-accuracy-idle
> +igt@kms_vblank@pipe-b-query-idle
> +igt@kms_vblank@pipe-b-query-forked
> +igt@kms_vblank@pipe-b-query-busy
> +igt@kms_vblank@pipe-b-query-forked-busy
> +igt@kms_vblank@pipe-b-wait-idle
> +igt@kms_vblank@pipe-b-wait-forked
> +igt@kms_vblank@pipe-b-wait-busy
> +igt@kms_vblank@pipe-b-wait-forked-busy
> +igt@kms_vblank@pipe-b-ts-continuation-idle
> +igt@kms_vblank@pipe-b-ts-continuation-modeset
> diff --git a/drivers/gpu/drm/msm/ci/msm_apq8016_results.txt b/drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> new file mode 100644
> index 000000000000..2cad61d89faf
> --- /dev/null
> +++ b/drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> @@ -0,0 +1,140 @@
> +igt@core_auth@getclient-simple,pass
> +igt@core_auth@getclient-master-drop,pass
> +igt@core_auth@basic-auth,pass
> +igt@core_auth@many-magics,pass
> +igt@core_getclient,pass
> +igt@core_getstats,pass
> +igt@core_getversion,pass
> +igt@core_setmaster_vs_auth,pass
> +igt@drm_read@invalid-buffer,skip
> +igt@drm_read@fault-buffer,skip
> +igt@drm_read@empty-block,skip
> +igt@drm_read@empty-nonblock,skip
> +igt@drm_read@short-buffer-block,skip
> +igt@drm_read@short-buffer-nonblock,skip
> +igt@drm_read@short-buffer-wakeup,skip
> +igt@kms_addfb_basic@unused-handle,pass
> +igt@kms_addfb_basic@unused-pitches,pass
> +igt@kms_addfb_basic@unused-offsets,pass
> +igt@kms_addfb_basic@unused-modifier,pass
> +igt@kms_addfb_basic@legacy-format,dmesg-warn
> +igt@kms_addfb_basic@no-handle,pass
> +igt@kms_addfb_basic@basic,pass
> +igt@kms_addfb_basic@bad-pitch-0,pass
> +igt@kms_addfb_basic@bad-pitch-32,pass
> +igt@kms_addfb_basic@bad-pitch-63,pass
> +igt@kms_addfb_basic@bad-pitch-128,pass
> +igt@kms_addfb_basic@bad-pitch-256,pass
> +igt@kms_addfb_basic@bad-pitch-1024,pass
> +igt@kms_addfb_basic@bad-pitch-999,pass
> +igt@kms_addfb_basic@bad-pitch-65536,pass
> +igt@kms_addfb_basic@size-max,pass
> +igt@kms_addfb_basic@too-wide,pass
> +igt@kms_addfb_basic@too-high,dmesg-warn
> +igt@kms_addfb_basic@bo-too-small,pass
> +igt@kms_addfb_basic@small-bo,pass
> +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> +igt@kms_addfb_basic@addfb25-bad-modifier,fail
> +igt@kms_addfb_basic@invalid-get-prop-any,pass
> +igt@kms_addfb_basic@invalid-get-prop,pass
> +igt@kms_addfb_basic@invalid-set-prop-any,pass
> +igt@kms_addfb_basic@invalid-set-prop,pass
> +igt@kms_addfb_basic@master-rmfb,pass
> +igt@kms_atomic@plane-overlay-legacy,skip
> +igt@kms_atomic@plane-primary-legacy,skip
> +igt@kms_atomic@plane-primary-overlay-mutable-zpos,skip
> +igt@kms_atomic@plane-immutable-zpos,skip
> +igt@kms_atomic@test-only,skip
> +igt@kms_atomic@plane-cursor-legacy,skip
> +igt@kms_atomic@plane-invalid-params,skip
> +igt@kms_atomic@crtc-invalid-params,skip
> +igt@kms_atomic@atomic-invalid-params,skip
> +igt@kms_atomic@atomic_plane_damage,skip
> +igt@kms_atomic_interruptible@legacy-setmode,skip
> +igt@kms_atomic_interruptible@atomic-setmode,skip
> +igt@kms_atomic_interruptible@legacy-dpms,skip
> +igt@kms_atomic_interruptible@legacy-pageflip,skip
> +igt@kms_atomic_interruptible@legacy-cursor,skip
> +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> +igt@kms_content_protection@lic,skip
> +igt@kms_flip_event_leak,skip
> +igt@kms_getfb@getfb-handle-zero,pass
> +igt@kms_getfb@getfb-handle-valid,pass
> +igt@kms_getfb@getfb-handle-closed,pass
> +igt@kms_getfb@getfb-handle-not-fb,pass
> +igt@kms_getfb@getfb-addfb-different-handles,pass
> +igt@kms_getfb@getfb-repeated-different-handles,pass
> +igt@kms_getfb@getfb2-handle-zero,pass
> +igt@kms_getfb@getfb2-handle-closed,pass
> +igt@kms_getfb@getfb2-handle-not-fb,pass
> +igt@kms_getfb@getfb2-into-addfb2,pass
> +igt@kms_getfb@getfb-handle-protection,pass
> +igt@kms_getfb@getfb2-handle-protection,pass
> +igt@kms_hdmi_inject@inject-4k,fail
> +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,skip
> +igt@kms_pipe_crc_basic@bad-source,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-a,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,skip
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,skip
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-b,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,skip
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,skip
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,skip
> +igt@kms_prop_blob@basic,pass
> +igt@kms_prop_blob@blob-prop-core,pass
> +igt@kms_prop_blob@blob-prop-validate,pass
> +igt@kms_prop_blob@blob-prop-lifetime,pass
> +igt@kms_prop_blob@blob-multiple,pass
> +igt@kms_prop_blob@invalid-get-prop-any,pass
> +igt@kms_prop_blob@invalid-get-prop,pass
> +igt@kms_prop_blob@invalid-set-prop-any,pass
> +igt@kms_prop_blob@invalid-set-prop,pass
> +igt@kms_rmfb@rmfb-ioctl,skip
> +igt@kms_rmfb@close-fd,skip
> +igt@kms_setmode@basic,skip
> +igt@msm_mapping@sqefw,skip
> +igt@msm_mapping@shadow,skip
> +igt@msm_recovery@hangcheck,skip
> +igt@msm_recovery@gpu-fault,skip
> +igt@msm_recovery@iova-fault,skip
> +igt@msm_submit@empty-submit,pass
> +igt@msm_submit@invalid-queue-submit,pass
> +igt@msm_submit@invalid-flags-submit,pass
> +igt@msm_submit@invalid-in-fence-submit,pass
> +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> +igt@msm_submit@valid-submit,pass
> +igt@kms_sysfs_edid_timing,pass
> +igt@kms_universal_plane@universal-plane-pipe-a-functional,skip
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,skip
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,skip
> +igt@kms_vblank@invalid,skip
> +igt@kms_vblank@crtc-id,skip
> +igt@kms_vblank@pipe-a-query-idle,skip
> +igt@kms_vblank@pipe-a-query-forked,skip
> +igt@kms_vblank@pipe-a-query-busy,skip
> +igt@kms_vblank@pipe-a-query-forked-busy,skip
> +igt@kms_vblank@pipe-a-wait-idle,skip
> +igt@kms_vblank@pipe-a-wait-forked,skip
> +igt@kms_vblank@pipe-a-wait-busy,skip
> +igt@kms_vblank@pipe-a-wait-forked-busy,skip
> +igt@kms_vblank@pipe-a-ts-continuation-idle,skip
> +igt@kms_vblank@pipe-a-ts-continuation-modeset,skip
> +igt@kms_vblank@pipe-b-accuracy-idle,skip
> +igt@kms_vblank@pipe-b-query-idle,skip
> +igt@kms_vblank@pipe-b-query-forked,skip
> +igt@kms_vblank@pipe-b-query-busy,skip
> +igt@kms_vblank@pipe-b-query-forked-busy,skip
> +igt@kms_vblank@pipe-b-wait-idle,skip
> +igt@kms_vblank@pipe-b-wait-forked,skip
> +igt@kms_vblank@pipe-b-wait-busy,skip
> +igt@kms_vblank@pipe-b-wait-forked-busy,skip
> +igt@kms_vblank@pipe-b-ts-continuation-idle,skip
> +igt@kms_vblank@pipe-b-ts-continuation-modeset,skip
> diff --git a/drivers/gpu/drm/msm/ci/msm_apq8096_results.txt b/drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> new file mode 100644
> index 000000000000..6f10941e5626
> --- /dev/null
> +++ b/drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> @@ -0,0 +1,140 @@
> +igt@core_auth@getclient-simple,pass
> +igt@core_auth@getclient-master-drop,pass
> +igt@core_auth@basic-auth,pass
> +igt@core_auth@many-magics,pass
> +igt@core_getclient,pass
> +igt@core_getstats,pass
> +igt@core_getversion,pass
> +igt@core_setmaster_vs_auth,pass
> +igt@drm_read@invalid-buffer,skip
> +igt@drm_read@fault-buffer,skip
> +igt@drm_read@empty-block,skip
> +igt@drm_read@empty-nonblock,skip
> +igt@drm_read@short-buffer-block,skip
> +igt@drm_read@short-buffer-nonblock,skip
> +igt@drm_read@short-buffer-wakeup,skip
> +igt@kms_addfb_basic@unused-handle,pass
> +igt@kms_addfb_basic@unused-pitches,pass
> +igt@kms_addfb_basic@unused-offsets,pass
> +igt@kms_addfb_basic@unused-modifier,pass
> +igt@kms_addfb_basic@legacy-format,dmesg-warn
> +igt@kms_addfb_basic@no-handle,pass
> +igt@kms_addfb_basic@basic,pass
> +igt@kms_addfb_basic@bad-pitch-0,pass
> +igt@kms_addfb_basic@bad-pitch-32,pass
> +igt@kms_addfb_basic@bad-pitch-63,pass
> +igt@kms_addfb_basic@bad-pitch-128,pass
> +igt@kms_addfb_basic@bad-pitch-256,pass
> +igt@kms_addfb_basic@bad-pitch-1024,pass
> +igt@kms_addfb_basic@bad-pitch-999,pass
> +igt@kms_addfb_basic@bad-pitch-65536,pass
> +igt@kms_addfb_basic@size-max,pass
> +igt@kms_addfb_basic@too-wide,pass
> +igt@kms_addfb_basic@too-high,dmesg-warn
> +igt@kms_addfb_basic@bo-too-small,pass
> +igt@kms_addfb_basic@small-bo,pass
> +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> +igt@kms_addfb_basic@addfb25-bad-modifier,fail
> +igt@kms_addfb_basic@invalid-get-prop-any,pass
> +igt@kms_addfb_basic@invalid-get-prop,pass
> +igt@kms_addfb_basic@invalid-set-prop-any,pass
> +igt@kms_addfb_basic@invalid-set-prop,pass
> +igt@kms_addfb_basic@master-rmfb,pass
> +igt@kms_atomic@plane-overlay-legacy,skip
> +igt@kms_atomic@plane-primary-legacy,skip
> +igt@kms_atomic@plane-primary-overlay-mutable-zpos,skip
> +igt@kms_atomic@plane-immutable-zpos,skip
> +igt@kms_atomic@test-only,skip
> +igt@kms_atomic@plane-cursor-legacy,skip
> +igt@kms_atomic@plane-invalid-params,skip
> +igt@kms_atomic@crtc-invalid-params,skip
> +igt@kms_atomic@atomic-invalid-params,skip
> +igt@kms_atomic@atomic_plane_damage,skip
> +igt@kms_atomic_interruptible@legacy-setmode,skip
> +igt@kms_atomic_interruptible@atomic-setmode,skip
> +igt@kms_atomic_interruptible@legacy-dpms,skip
> +igt@kms_atomic_interruptible@legacy-pageflip,skip
> +igt@kms_atomic_interruptible@legacy-cursor,skip
> +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> +igt@kms_content_protection@lic,skip
> +igt@kms_flip_event_leak,skip
> +igt@kms_getfb@getfb-handle-zero,pass
> +igt@kms_getfb@getfb-handle-valid,pass
> +igt@kms_getfb@getfb-handle-closed,pass
> +igt@kms_getfb@getfb-handle-not-fb,pass
> +igt@kms_getfb@getfb-addfb-different-handles,pass
> +igt@kms_getfb@getfb-repeated-different-handles,pass
> +igt@kms_getfb@getfb2-handle-zero,pass
> +igt@kms_getfb@getfb2-handle-closed,pass
> +igt@kms_getfb@getfb2-handle-not-fb,pass
> +igt@kms_getfb@getfb2-into-addfb2,pass
> +igt@kms_getfb@getfb-handle-protection,pass
> +igt@kms_getfb@getfb2-handle-protection,pass
> +igt@kms_hdmi_inject@inject-4k,dmesg-warn
> +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,skip
> +igt@kms_pipe_crc_basic@bad-source,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-a,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,skip
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,skip
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-b,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,skip
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,skip
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,skip
> +igt@kms_prop_blob@basic,pass
> +igt@kms_prop_blob@blob-prop-core,pass
> +igt@kms_prop_blob@blob-prop-validate,pass
> +igt@kms_prop_blob@blob-prop-lifetime,pass
> +igt@kms_prop_blob@blob-multiple,pass
> +igt@kms_prop_blob@invalid-get-prop-any,pass
> +igt@kms_prop_blob@invalid-get-prop,pass
> +igt@kms_prop_blob@invalid-set-prop-any,pass
> +igt@kms_prop_blob@invalid-set-prop,pass
> +igt@kms_rmfb@rmfb-ioctl,skip
> +igt@kms_rmfb@close-fd,skip
> +igt@kms_setmode@basic,skip
> +igt@msm_mapping@sqefw,skip
> +igt@msm_mapping@shadow,skip
> +igt@msm_recovery@hangcheck,skip
> +igt@msm_recovery@gpu-fault,skip
> +igt@msm_recovery@iova-fault,skip
> +igt@msm_submit@empty-submit,pass
> +igt@msm_submit@invalid-queue-submit,pass
> +igt@msm_submit@invalid-flags-submit,pass
> +igt@msm_submit@invalid-in-fence-submit,pass
> +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> +igt@msm_submit@valid-submit,pass
> +igt@kms_sysfs_edid_timing,pass
> +igt@kms_universal_plane@universal-plane-pipe-a-functional,skip
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,skip
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,skip
> +igt@kms_vblank@invalid,skip
> +igt@kms_vblank@crtc-id,skip
> +igt@kms_vblank@pipe-a-query-idle,skip
> +igt@kms_vblank@pipe-a-query-forked,skip
> +igt@kms_vblank@pipe-a-query-busy,skip
> +igt@kms_vblank@pipe-a-query-forked-busy,skip
> +igt@kms_vblank@pipe-a-wait-idle,skip
> +igt@kms_vblank@pipe-a-wait-forked,skip
> +igt@kms_vblank@pipe-a-wait-busy,skip
> +igt@kms_vblank@pipe-a-wait-forked-busy,skip
> +igt@kms_vblank@pipe-a-ts-continuation-idle,skip
> +igt@kms_vblank@pipe-a-ts-continuation-modeset,skip
> +igt@kms_vblank@pipe-b-accuracy-idle,skip
> +igt@kms_vblank@pipe-b-query-idle,skip
> +igt@kms_vblank@pipe-b-query-forked,skip
> +igt@kms_vblank@pipe-b-query-busy,skip
> +igt@kms_vblank@pipe-b-query-forked-busy,skip
> +igt@kms_vblank@pipe-b-wait-idle,skip
> +igt@kms_vblank@pipe-b-wait-forked,skip
> +igt@kms_vblank@pipe-b-wait-busy,skip
> +igt@kms_vblank@pipe-b-wait-forked-busy,skip
> +igt@kms_vblank@pipe-b-ts-continuation-idle,skip
> +igt@kms_vblank@pipe-b-ts-continuation-modeset,skip
> diff --git a/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> new file mode 100644
> index 000000000000..01f7b4b399b5
> --- /dev/null
> +++ b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> @@ -0,0 +1,141 @@
> +igt@core_auth@getclient-simple,dmesg-warn
> +igt@core_auth@getclient-master-drop,pass
> +igt@core_auth@basic-auth,pass
> +igt@core_auth@many-magics,pass
> +igt@core_getclient,pass
> +igt@core_getstats,pass
> +igt@core_getversion,pass
> +igt@core_setmaster_vs_auth,pass
> +igt@drm_read@invalid-buffer,pass
> +igt@drm_read@fault-buffer,pass
> +igt@drm_read@empty-block,pass
> +igt@drm_read@empty-nonblock,pass
> +igt@drm_read@short-buffer-block,pass
> +igt@drm_read@short-buffer-nonblock,pass
> +igt@drm_read@short-buffer-wakeup,pass
> +igt@kms_addfb_basic@unused-handle,pass
> +igt@kms_addfb_basic@unused-pitches,pass
> +igt@kms_addfb_basic@unused-offsets,pass
> +igt@kms_addfb_basic@unused-modifier,pass
> +igt@kms_addfb_basic@legacy-format,dmesg-warn
> +igt@kms_addfb_basic@no-handle,pass
> +igt@kms_addfb_basic@basic,pass
> +igt@kms_addfb_basic@bad-pitch-0,pass
> +igt@kms_addfb_basic@bad-pitch-32,pass
> +igt@kms_addfb_basic@bad-pitch-63,pass
> +igt@kms_addfb_basic@bad-pitch-128,pass
> +igt@kms_addfb_basic@bad-pitch-256,pass
> +igt@kms_addfb_basic@bad-pitch-1024,pass
> +igt@kms_addfb_basic@bad-pitch-999,pass
> +igt@kms_addfb_basic@bad-pitch-65536,pass
> +igt@kms_addfb_basic@size-max,pass
> +igt@kms_addfb_basic@too-wide,pass
> +igt@kms_addfb_basic@too-high,dmesg-warn

For test results on Trogdor, is is possible to have them be 
success/fail/skip only?

Results such as dmesg-warn/dmesg-fail are igt_runner specific and 
because there isn't support for igt_runner on ChromeOS, they will be 
difficult to replicate and debug.

> +igt@kms_addfb_basic@bo-too-small,pass
> +igt@kms_addfb_basic@small-bo,pass
> +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> +igt@kms_addfb_basic@addfb25-bad-modifier,dmesg-warn
> +igt@kms_addfb_basic@invalid-get-prop-any,pass
> +igt@kms_addfb_basic@invalid-get-prop,pass
> +igt@kms_addfb_basic@invalid-set-prop-any,pass
> +igt@kms_addfb_basic@invalid-set-prop,pass
> +igt@kms_addfb_basic@master-rmfb,pass
> +igt@kms_atomic@plane-overlay-legacy,skip
> +igt@kms_atomic@plane-primary-legacy,pass
> +igt@kms_atomic@plane-primary-overlay-mutable-zpos,skip
> +igt@kms_atomic@plane-immutable-zpos,pass
> +igt@kms_atomic@test-only,pass
> +igt@kms_atomic@plane-cursor-legacy,pass
> +igt@kms_atomic@plane-invalid-params,pass
> +igt@kms_atomic@crtc-invalid-params,pass
> +igt@kms_atomic@atomic-invalid-params,pass
> +igt@kms_atomic@atomic_plane_damage,pass
> +igt@kms_atomic_interruptible@legacy-setmode,skip
> +igt@kms_atomic_interruptible@atomic-setmode,skip
> +igt@kms_atomic_interruptible@legacy-dpms,skip
> +igt@kms_atomic_interruptible@legacy-pageflip,skip
> +igt@kms_atomic_interruptible@legacy-cursor,skip
> +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> +igt@kms_content_protection@lic,skip
> +igt@kms_flip_event_leak,pass
> +igt@kms_getfb@getfb-handle-zero,pass
> +igt@kms_getfb@getfb-handle-valid,pass
> +igt@kms_getfb@getfb-handle-closed,pass
> +igt@kms_getfb@getfb-handle-not-fb,pass
> +igt@kms_getfb@getfb-addfb-different-handles,pass
> +igt@kms_getfb@getfb-repeated-different-handles,pass
> +igt@kms_getfb@getfb2-handle-zero,pass
> +igt@kms_getfb@getfb2-handle-closed,pass
> +igt@kms_getfb@getfb2-handle-not-fb,pass
> +igt@kms_getfb@getfb2-into-addfb2,pass
> +igt@kms_getfb@getfb-handle-protection,pass
> +igt@kms_getfb@getfb2-handle-protection,pass
> +igt@kms_hdmi_inject@inject-4k,skip
> +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,skip
> +igt@kms_pipe_crc_basic@bad-source,pass
> +igt@kms_pipe_crc_basic@read-crc-pipe-a,pass
> +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,pass
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,pass
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,pass
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,pass
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,pass
> +igt@kms_pipe_crc_basic@read-crc-pipe-b,pass
> +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,pass
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,pass
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,pass
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,pass
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,pass
> +igt@kms_prop_blob@basic,pass
> +igt@kms_prop_blob@blob-prop-core,pass
> +igt@kms_prop_blob@blob-prop-validate,pass
> +igt@kms_prop_blob@blob-prop-lifetime,pass
> +igt@kms_prop_blob@blob-multiple,pass
> +igt@kms_prop_blob@invalid-get-prop-any,pass
> +igt@kms_prop_blob@invalid-get-prop,pass
> +igt@kms_prop_blob@invalid-set-prop-any,pass
> +igt@kms_prop_blob@invalid-set-prop,pass
> +igt@kms_rmfb@rmfb-ioctl,pass
> +igt@kms_rmfb@close-fd,fail
> +igt@kms_setmode@basic@pipe-a-edp-1,pass
> +igt@kms_setmode@basic@pipe-b-edp-1,pass
> +igt@msm_mapping@sqefw,dmesg-fail
> +igt@msm_mapping@shadow,dmesg-fail
> +igt@msm_recovery@hangcheck,dmesg-warn
> +igt@msm_recovery@gpu-fault,dmesg-warn
> +igt@msm_recovery@iova-fault,dmesg-warn
> +igt@msm_submit@empty-submit,pass
> +igt@msm_submit@invalid-queue-submit,pass
> +igt@msm_submit@invalid-flags-submit,pass
> +igt@msm_submit@invalid-in-fence-submit,pass
> +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> +igt@msm_submit@valid-submit,pass
> +igt@kms_sysfs_edid_timing,pass
> +igt@kms_universal_plane@universal-plane-pipe-a-functional,skip
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,pass
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,fail

We could probably skip checking the results for kms_universal_plane on 
Trogdor for now, since this is a test affected by the hack regression. 
There is an IGT patch in the works for fixing the 
disable-primary-vs-flip-pipe-b failure, so it should be updated pretty 
soon too.

Thanks,
Jessica Zhang

> +igt@kms_vblank@invalid,pass
> +igt@kms_vblank@crtc-id,pass
> +igt@kms_vblank@pipe-a-query-idle,pass
> +igt@kms_vblank@pipe-a-query-forked,pass
> +igt@kms_vblank@pipe-a-query-busy,pass
> +igt@kms_vblank@pipe-a-query-forked-busy,pass
> +igt@kms_vblank@pipe-a-wait-idle,pass
> +igt@kms_vblank@pipe-a-wait-forked,pass
> +igt@kms_vblank@pipe-a-wait-busy,pass
> +igt@kms_vblank@pipe-a-wait-forked-busy,pass
> +igt@kms_vblank@pipe-a-ts-continuation-idle,pass
> +igt@kms_vblank@pipe-a-ts-continuation-modeset,pass
> +igt@kms_vblank@pipe-b-accuracy-idle,pass
> +igt@kms_vblank@pipe-b-query-idle,pass
> +igt@kms_vblank@pipe-b-query-forked,pass
> +igt@kms_vblank@pipe-b-query-busy,pass
> +igt@kms_vblank@pipe-b-query-forked-busy,pass
> +igt@kms_vblank@pipe-b-wait-idle,pass
> +igt@kms_vblank@pipe-b-wait-forked,pass
> +igt@kms_vblank@pipe-b-wait-busy,pass
> +igt@kms_vblank@pipe-b-wait-forked-busy,pass
> +igt@kms_vblank@pipe-b-ts-continuation-idle,pass
> +igt@kms_vblank@pipe-b-ts-continuation-modeset,pass
> diff --git a/drivers/gpu/drm/msm/ci/msm_sdm845_results.txt b/drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> new file mode 100644
> index 000000000000..3df1e5ef6d80
> --- /dev/null
> +++ b/drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> @@ -0,0 +1,141 @@
> +igt@core_auth@getclient-simple,dmesg-warn
> +igt@core_auth@getclient-master-drop,pass
> +igt@core_auth@basic-auth,pass
> +igt@core_auth@many-magics,pass
> +igt@core_getclient,pass
> +igt@core_getstats,pass
> +igt@core_getversion,pass
> +igt@core_setmaster_vs_auth,pass
> +igt@drm_read@invalid-buffer,pass
> +igt@drm_read@fault-buffer,pass
> +igt@drm_read@empty-block,pass
> +igt@drm_read@empty-nonblock,pass
> +igt@drm_read@short-buffer-block,pass
> +igt@drm_read@short-buffer-nonblock,pass
> +igt@drm_read@short-buffer-wakeup,pass
> +igt@kms_addfb_basic@unused-handle,pass
> +igt@kms_addfb_basic@unused-pitches,pass
> +igt@kms_addfb_basic@unused-offsets,pass
> +igt@kms_addfb_basic@unused-modifier,pass
> +igt@kms_addfb_basic@legacy-format,dmesg-warn
> +igt@kms_addfb_basic@no-handle,pass
> +igt@kms_addfb_basic@basic,pass
> +igt@kms_addfb_basic@bad-pitch-0,pass
> +igt@kms_addfb_basic@bad-pitch-32,pass
> +igt@kms_addfb_basic@bad-pitch-63,pass
> +igt@kms_addfb_basic@bad-pitch-128,pass
> +igt@kms_addfb_basic@bad-pitch-256,pass
> +igt@kms_addfb_basic@bad-pitch-1024,pass
> +igt@kms_addfb_basic@bad-pitch-999,pass
> +igt@kms_addfb_basic@bad-pitch-65536,pass
> +igt@kms_addfb_basic@size-max,pass
> +igt@kms_addfb_basic@too-wide,pass
> +igt@kms_addfb_basic@too-high,dmesg-warn
> +igt@kms_addfb_basic@bo-too-small,pass
> +igt@kms_addfb_basic@small-bo,pass
> +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> +igt@kms_addfb_basic@addfb25-bad-modifier,dmesg-warn
> +igt@kms_addfb_basic@invalid-get-prop-any,pass
> +igt@kms_addfb_basic@invalid-get-prop,pass
> +igt@kms_addfb_basic@invalid-set-prop-any,pass
> +igt@kms_addfb_basic@invalid-set-prop,pass
> +igt@kms_addfb_basic@master-rmfb,pass
> +igt@kms_atomic@plane-overlay-legacy,dmesg-warn
> +igt@kms_atomic@plane-primary-legacy,dmesg-warn
> +igt@kms_atomic@plane-primary-overlay-mutable-zpos,dmesg-warn
> +igt@kms_atomic@plane-immutable-zpos,dmesg-warn
> +igt@kms_atomic@test-only,dmesg-warn
> +igt@kms_atomic@plane-cursor-legacy,dmesg-warn
> +igt@kms_atomic@plane-invalid-params,dmesg-warn
> +igt@kms_atomic@crtc-invalid-params,dmesg-warn
> +igt@kms_atomic@atomic-invalid-params,dmesg-warn
> +igt@kms_atomic@atomic_plane_damage,dmesg-warn
> +igt@kms_atomic_interruptible@legacy-setmode,skip
> +igt@kms_atomic_interruptible@atomic-setmode,skip
> +igt@kms_atomic_interruptible@legacy-dpms,skip
> +igt@kms_atomic_interruptible@legacy-pageflip,skip
> +igt@kms_atomic_interruptible@legacy-cursor,skip
> +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> +igt@kms_content_protection@lic,skip
> +igt@kms_flip_event_leak,dmesg-warn
> +igt@kms_getfb@getfb-handle-zero,pass
> +igt@kms_getfb@getfb-handle-valid,pass
> +igt@kms_getfb@getfb-handle-closed,pass
> +igt@kms_getfb@getfb-handle-not-fb,pass
> +igt@kms_getfb@getfb-addfb-different-handles,pass
> +igt@kms_getfb@getfb-repeated-different-handles,pass
> +igt@kms_getfb@getfb2-handle-zero,pass
> +igt@kms_getfb@getfb2-handle-closed,pass
> +igt@kms_getfb@getfb2-handle-not-fb,pass
> +igt@kms_getfb@getfb2-into-addfb2,pass
> +igt@kms_getfb@getfb-handle-protection,pass
> +igt@kms_getfb@getfb2-handle-protection,pass
> +igt@kms_hdmi_inject@inject-4k,skip
> +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,pass
> +igt@kms_pipe_crc_basic@bad-source,pass
> +igt@kms_pipe_crc_basic@read-crc-pipe-a,pass
> +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,pass
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,pass
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,pass
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,dmesg-warn
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,dmesg-warn
> +igt@kms_pipe_crc_basic@read-crc-pipe-b,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,skip
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,skip
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,skip
> +igt@kms_prop_blob@basic,pass
> +igt@kms_prop_blob@blob-prop-core,pass
> +igt@kms_prop_blob@blob-prop-validate,pass
> +igt@kms_prop_blob@blob-prop-lifetime,pass
> +igt@kms_prop_blob@blob-multiple,pass
> +igt@kms_prop_blob@invalid-get-prop-any,pass
> +igt@kms_prop_blob@invalid-get-prop,pass
> +igt@kms_prop_blob@invalid-set-prop-any,pass
> +igt@kms_prop_blob@invalid-set-prop,pass
> +igt@kms_rmfb@rmfb-ioctl,pass
> +igt@kms_rmfb@close-fd,fail
> +igt@kms_setmode@basic@pipe-a-edp-1,dmesg-warn
> +igt@kms_setmode@basic,skip
> +igt@msm_mapping@sqefw,dmesg-fail
> +igt@msm_mapping@shadow,dmesg-fail
> +igt@msm_recovery@hangcheck,dmesg-warn
> +igt@msm_recovery@gpu-fault,dmesg-warn
> +igt@msm_recovery@iova-fault,dmesg-warn
> +igt@msm_submit@empty-submit,pass
> +igt@msm_submit@invalid-queue-submit,pass
> +igt@msm_submit@invalid-flags-submit,pass
> +igt@msm_submit@invalid-in-fence-submit,pass
> +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> +igt@msm_submit@valid-submit,pass
> +igt@kms_sysfs_edid_timing,pass
> +igt@kms_universal_plane@universal-plane-pipe-a-functional,dmesg-fail
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,dmesg-warn
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,skip
> +igt@kms_vblank@invalid,dmesg-warn
> +igt@kms_vblank@crtc-id,dmesg-warn
> +igt@kms_vblank@pipe-a-query-idle,dmesg-warn
> +igt@kms_vblank@pipe-a-query-forked,dmesg-warn
> +igt@kms_vblank@pipe-a-query-busy,dmesg-warn
> +igt@kms_vblank@pipe-a-query-forked-busy,dmesg-warn
> +igt@kms_vblank@pipe-a-wait-idle,dmesg-warn
> +igt@kms_vblank@pipe-a-wait-forked,dmesg-warn
> +igt@kms_vblank@pipe-a-wait-busy,dmesg-warn
> +igt@kms_vblank@pipe-a-wait-forked-busy,dmesg-warn
> +igt@kms_vblank@pipe-a-ts-continuation-idle,dmesg-warn
> +igt@kms_vblank@pipe-a-ts-continuation-modeset,dmesg-warn
> +igt@kms_vblank@pipe-b-accuracy-idle,skip
> +igt@kms_vblank@pipe-b-query-idle,skip
> +igt@kms_vblank@pipe-b-query-forked,skip
> +igt@kms_vblank@pipe-b-query-busy,skip
> +igt@kms_vblank@pipe-b-query-forked-busy,skip
> +igt@kms_vblank@pipe-b-wait-idle,skip
> +igt@kms_vblank@pipe-b-wait-forked,skip
> +igt@kms_vblank@pipe-b-wait-busy,skip
> +igt@kms_vblank@pipe-b-wait-forked-busy,skip
> +igt@kms_vblank@pipe-b-ts-continuation-idle,skip
> +igt@kms_vblank@pipe-b-ts-continuation-modeset,skip
> -- 
> 2.31.1
>

Rob Clark May 10, 2022, 8:25 p.m. UTC | #2

On Tue, May 10, 2022 at 12:39 PM Jessica Zhang
<quic_jesszhan@quicinc.com> wrote:
>
>
>
> On 5/10/2022 7:13 AM, Tomeu Vizoso wrote:
> > And use it to store expectations about what the drm/msm driver is
> > supposed to pass in the IGT test suite.
> >
> > Also include a configuration file that points to the out-of-tree CI
> > scripts.
> >
> > By storing the test expectations along the code we can make sure both
> > stay in sync with each other, and so we can know when a code change
> > breaks those expectations.
> >
> > This will allow all contributors to drm/msm to reuse the infrastructure
> > already in gitlab.freedesktop.org to test the driver on several
> > generations of the hardware.
> >
> > v2:
> >    - Fix names of result expectation files to match SoC
> >    - Don't execute tests that are going to skip on all boards
> >
> > Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> > ---
> >   Documentation/gpu/msm_automated_testing.rst   |  70 +++++++++
> >   drivers/gpu/drm/msm/ci/gitlab-ci.yml          |  11 ++
> >   drivers/gpu/drm/msm/ci/msm.testlist           | 148 ++++++++++++++++++
> >   .../gpu/drm/msm/ci/msm_apq8016_results.txt    | 140 +++++++++++++++++
> >   .../gpu/drm/msm/ci/msm_apq8096_results.txt    | 140 +++++++++++++++++
> >   drivers/gpu/drm/msm/ci/msm_sc7180_results.txt | 141 +++++++++++++++++
> >   drivers/gpu/drm/msm/ci/msm_sdm845_results.txt | 141 +++++++++++++++++
> >   7 files changed, 791 insertions(+)
> >   create mode 100644 Documentation/gpu/msm_automated_testing.rst
> >   create mode 100644 drivers/gpu/drm/msm/ci/gitlab-ci.yml
> >   create mode 100644 drivers/gpu/drm/msm/ci/msm.testlist
> >   create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> >   create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> >   create mode 100644 drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> >   create mode 100644 drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> >
> > diff --git a/Documentation/gpu/msm_automated_testing.rst b/Documentation/gpu/msm_automated_testing.rst
> > new file mode 100644
> > index 000000000000..fe59474569c2
> > --- /dev/null
> > +++ b/Documentation/gpu/msm_automated_testing.rst
> > @@ -0,0 +1,70 @@
> > +.. SPDX-License-Identifier: GPL-2.0+
> > +
> > +=========================================
> > +Automated testing of MSM DRM driver
> > +=========================================
>
> Hey Tomeu,
>
> I like the overall idea of having this in as part of the repo, but just
> have a few questions on some details.
>
> > +
> > +
> > +Introduction
> > +============
> > +
> > +Making sure that changes to a driver don't introduce regressions can be very time consuming when lots of different hardware configurations need to be tested. Moreover, it isn't practical for each person interested in this testing to have to acquire and maintain what can be a considerable amount of hardware.
> > +
> > +Also, it is desirable for developers to check for regressions in their code by themselves, instead of relying on the maintainer finding them and then reporting back.
> > +
> > +There are facilities in gitlab.freedesktop.org to automatically test Mesa that can be used as well for testing DRM drivers such as drm/msm. This document explains how people interested in testing the drm/msm driver can use this shared infrastructure to save quite some time and effort.
> > +
> > +
> > +Relevant files
> > +==============
> > +
> > +drivers/gpu/drm/msm/ci/gitlab-ci.yml
> > +------------------------------------
> > +
> > +Specifies the specific version of the scripts to be used. GitLab CI will use the values defined in this file to fetch the right scripts.
> > +
> > +
> > +drivers/gpu/drm/msm/ci/msm.testlist
> > +-----------------------------------
> > +
> > +Specifies the tests that the current code is expected to be able to reliably run. These tests are expected to not hang the DUT (device under testing) when running on the revision they belong to, and to give consistent results.
> > +
> > +
> > +drivers/gpu/drm/msm/ci/msm_*_results.txt
> > +----------------------------------------
> > +
> > +Specifies the expected results of running this specific kernel revision on a given hardware configuration.
> > +
> > +
> > +How to enable automated testing on your tree
> > +============================================
> > +
> > +1. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one yet
> > +
> > +2. In your kernel repo's configuration (eg. https://gitlab.freedesktop.org/tomeu/linux/-/settings/ci_cd), change the CI/CD configuration file from .gitlab-ci.yml to drivers/gpu/drm/msm/ci/gitlab-ci.yml.
> > +
> > +3. Next time you push to this repository, you will see a CI pipeline being created (eg. https://gitlab.freedesktop.org/tomeu/linux/-/pipelines)
> > +
> > +4. The various jobs will be run and when the pipeline is finished, all jobs should be green unless a regression has been found.
> > +
> > +
> > +How to update test expectations
> > +===============================
> > +
> > +If your changes to the code fix any tests, you will have to update one or more of the files in drivers/gpu/drm/msm/ci/msm_*_results.txt, for each of the test platforms affected by the change.
> > +
> > +If you have run a pipeline and it failed because of a mismatched test result, you will have been offered a patch at the end of the run that you can apply after reviewing it.
> > +
> > +
> > +How to expand coverage
> > +======================
> > +
> > +If your code changes makes it possible to run more tests (by solving reliability issues, for example), more tests can be added to this list, and then the expected results updated with the patch file that will be printed at the end of each job.
> > +
> > +If there is a need for updating the version of IGT being used (maybe you have added more tests to it), follow the instructions at https://gitlab.freedesktop.org/gfx-ci/drm-ci/-/tree/msm#upreving-igt.
> > +
> > +
> > +How to test your changes to the scripts
> > +==========================================
> > +
> > +For testing changes to the scripts in the drm-ci repo, change the DRM_CI_PROJECT_PATH variable in drivers/gpu/drm/msm/ci/gitlab-ci.yml to match your fork of the project (eg. tomeu/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/.
> > \ No newline at end of file
> > diff --git a/drivers/gpu/drm/msm/ci/gitlab-ci.yml b/drivers/gpu/drm/msm/ci/gitlab-ci.yml
> > new file mode 100644
> > index 000000000000..9b7caa7fcab2
> > --- /dev/null
> > +++ b/drivers/gpu/drm/msm/ci/gitlab-ci.yml
> > @@ -0,0 +1,11 @@
> > +variables:
> > +  # Change this to use your fork of drm-ci
> > +  DRM_CI_PROJECT_PATH: &drm-ci-project-path gfx-ci/drm-ci
> > +  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha msm
> > +
> > +include:
> > +  - project: *drm-ci-project-path
> > +    ref: *drm-ci-commit-sha
> > +    file:
> > +      - '.msm-gitlab-ci.yml'
> > +      - '.gitlab-ci.yml'
> > diff --git a/drivers/gpu/drm/msm/ci/msm.testlist b/drivers/gpu/drm/msm/ci/msm.testlist
> > new file mode 100644
> > index 000000000000..8805a3bc7316
> > --- /dev/null
> > +++ b/drivers/gpu/drm/msm/ci/msm.testlist
> > @@ -0,0 +1,148 @@
> > +# Keep alphabetically sorted by default
> > +
> > +igt@core_auth@getclient-simple
> > +igt@core_auth@getclient-master-drop
> > +igt@core_auth@basic-auth
> > +igt@core_auth@many-magics
> > +igt@core_getclient
> > +igt@core_getstats
> > +igt@core_getversion
> > +igt@core_setmaster_vs_auth
> > +igt@drm_read@invalid-buffer
> > +igt@drm_read@fault-buffer
> > +igt@drm_read@empty-block
> > +igt@drm_read@empty-nonblock
> > +igt@drm_read@short-buffer-block
> > +igt@drm_read@short-buffer-nonblock
> > +igt@drm_read@short-buffer-wakeup
> > +igt@kms_addfb_basic@unused-handle
> > +igt@kms_addfb_basic@unused-pitches
> > +igt@kms_addfb_basic@unused-offsets
> > +igt@kms_addfb_basic@unused-modifier
> > +igt@kms_addfb_basic@legacy-format
> > +igt@kms_addfb_basic@no-handle
> > +igt@kms_addfb_basic@basic
> > +igt@kms_addfb_basic@bad-pitch-0
> > +igt@kms_addfb_basic@bad-pitch-32
> > +igt@kms_addfb_basic@bad-pitch-63
> > +igt@kms_addfb_basic@bad-pitch-128
> > +igt@kms_addfb_basic@bad-pitch-256
> > +igt@kms_addfb_basic@bad-pitch-1024
> > +igt@kms_addfb_basic@bad-pitch-999
> > +igt@kms_addfb_basic@bad-pitch-65536
> > +igt@kms_addfb_basic@size-max
> > +igt@kms_addfb_basic@too-wide
> > +igt@kms_addfb_basic@too-high
> > +igt@kms_addfb_basic@bo-too-small
> > +igt@kms_addfb_basic@small-bo
> > +igt@kms_addfb_basic@addfb25-modifier-no-flag
> > +igt@kms_addfb_basic@addfb25-bad-modifier
> > +igt@kms_addfb_basic@invalid-get-prop-any
> > +igt@kms_addfb_basic@invalid-get-prop
> > +igt@kms_addfb_basic@invalid-set-prop-any
> > +igt@kms_addfb_basic@invalid-set-prop
> > +igt@kms_addfb_basic@master-rmfb
> > +igt@kms_atomic@plane-overlay-legacy
> > +igt@kms_atomic@plane-primary-legacy
> > +igt@kms_atomic@plane-primary-overlay-mutable-zpos
> > +igt@kms_atomic@plane-immutable-zpos
> > +igt@kms_atomic@test-only
> > +igt@kms_atomic@plane-cursor-legacy
> > +igt@kms_atomic@plane-invalid-params
> > +igt@kms_atomic@crtc-invalid-params
> > +igt@kms_atomic@atomic-invalid-params
> > +igt@kms_atomic@atomic_plane_damage
> > +igt@kms_atomic_interruptible@legacy-setmode@pipe-a-edp-1
> > +igt@kms_atomic_interruptible@atomic-setmode@pipe-a-edp-1
> > +igt@kms_atomic_interruptible@legacy-dpms@pipe-a-edp-1
> > +igt@kms_atomic_interruptible@legacy-pageflip@pipe-a-edp-1
> > +igt@kms_atomic_interruptible@legacy-cursor@pipe-a-edp-1
> > +igt@kms_atomic_interruptible@universal-setplane-primary@pipe-a-edp-1
> > +igt@kms_atomic_interruptible@universal-setplane-cursor@pipe-a-edp-1
>
> I see that there are some KMS subtests/tests are missing such as
> kms_atomic_transition -- are they unstable across all boards?
>
>  From what I've seen, kms_atomic_transition is unstable only on Trogdor,
> but has stable results on other boards.
>
> > +igt@kms_content_protection@LIC > +igt@kms_flip_event_leak
> > +igt@kms_getfb@getfb-handle-zero
> > +igt@kms_getfb@getfb-handle-valid
> > +igt@kms_getfb@getfb-handle-closed
> > +igt@kms_getfb@getfb-handle-not-fb
> > +igt@kms_getfb@getfb-addfb-different-handles
> > +igt@kms_getfb@getfb-repeated-different-handles
> > +igt@kms_getfb@getfb2-handle-zero
> > +igt@kms_getfb@getfb2-handle-closed
> > +igt@kms_getfb@getfb2-handle-not-fb
> > +igt@kms_getfb@getfb2-into-addfb2
> > +igt@kms_getfb@getfb-handle-protection
> > +igt@kms_getfb@getfb2-handle-protection
> > +igt@kms_hdmi_inject@inject-4k > +igt@kms_multipipe_modeset@basic-max-pipe-crc-check
> > +igt@kms_pipe_crc_basic@bad-source
> > +igt@kms_pipe_crc_basic@read-crc-pipe-a
> > +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence
> > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a
> > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a
> > +igt@kms_pipe_crc_basic@read-crc-pipe-b
> > +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence
> > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b
> > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b
> > +igt@kms_prop_blob@basic
> > +igt@kms_prop_blob@blob-prop-core
> > +igt@kms_prop_blob@blob-prop-validate
> > +igt@kms_prop_blob@blob-prop-lifetime
> > +igt@kms_prop_blob@blob-multiple
> > +igt@kms_prop_blob@invalid-get-prop-any
> > +igt@kms_prop_blob@invalid-get-prop
> > +igt@kms_prop_blob@invalid-set-prop-any
> > +igt@kms_prop_blob@invalid-set-prop
> > +igt@kms_rmfb@rmfb-ioctl
> > +igt@kms_rmfb@close-fd
> > +igt@kms_setmode@basic@pipe-a-edp-1
> > +igt@kms_setmode@basic@pipe-b-edp-1
> > +
> > +# Flakes on a530 with:
> > +# arm-smmu b40000.iommu: Unhandled context fault: fsr=0x408, iova=0x01003000, fsynr=0x13, cbfrsynra=0x0, cb=0
> > +# igt@msm_mapping@ring
> > +
> > +igt@msm_mapping@sqefw
> > +igt@msm_mapping@shadow
> > +igt@msm_recovery@hangcheck
> > +igt@msm_recovery@gpu-fault
> > +igt@msm_recovery@iova-fault
> > +igt@msm_submit@empty-submit
> > +igt@msm_submit@invalid-queue-submit
> > +igt@msm_submit@invalid-flags-submit
> > +igt@msm_submit@invalid-in-fence-submit
> > +igt@msm_submit@invalid-duplicate-bo-submit
> > +igt@msm_submit@invalid-cmd-idx-submit
> > +igt@msm_submit@invalid-cmd-type-submit
> > +igt@msm_submit@valid-submit
> > +igt@kms_sysfs_edid_timing
> > +igt@kms_universal_plane@universal-plane-pipe-a-functional
> > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a
> > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b
> > +igt@kms_vblank@invalid
> > +igt@kms_vblank@crtc-id
> > +igt@kms_vblank@pipe-a-query-idle
> > +igt@kms_vblank@pipe-a-query-forked
> > +igt@kms_vblank@pipe-a-query-busy
> > +igt@kms_vblank@pipe-a-query-forked-busy
> > +igt@kms_vblank@pipe-a-wait-idle
> > +igt@kms_vblank@pipe-a-wait-forked
> > +igt@kms_vblank@pipe-a-wait-busy
> > +igt@kms_vblank@pipe-a-wait-forked-busy
> > +igt@kms_vblank@pipe-a-ts-continuation-idle
> > +igt@kms_vblank@pipe-a-ts-continuation-modeset
> > +igt@kms_vblank@pipe-b-accuracy-idle
> > +igt@kms_vblank@pipe-b-query-idle
> > +igt@kms_vblank@pipe-b-query-forked
> > +igt@kms_vblank@pipe-b-query-busy
> > +igt@kms_vblank@pipe-b-query-forked-busy
> > +igt@kms_vblank@pipe-b-wait-idle
> > +igt@kms_vblank@pipe-b-wait-forked
> > +igt@kms_vblank@pipe-b-wait-busy
> > +igt@kms_vblank@pipe-b-wait-forked-busy
> > +igt@kms_vblank@pipe-b-ts-continuation-idle
> > +igt@kms_vblank@pipe-b-ts-continuation-modeset
> > diff --git a/drivers/gpu/drm/msm/ci/msm_apq8016_results.txt b/drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> > new file mode 100644
> > index 000000000000..2cad61d89faf
> > --- /dev/null
> > +++ b/drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> > @@ -0,0 +1,140 @@
> > +igt@core_auth@getclient-simple,pass
> > +igt@core_auth@getclient-master-drop,pass
> > +igt@core_auth@basic-auth,pass
> > +igt@core_auth@many-magics,pass
> > +igt@core_getclient,pass
> > +igt@core_getstats,pass
> > +igt@core_getversion,pass
> > +igt@core_setmaster_vs_auth,pass
> > +igt@drm_read@invalid-buffer,skip
> > +igt@drm_read@fault-buffer,skip
> > +igt@drm_read@empty-block,skip
> > +igt@drm_read@empty-nonblock,skip
> > +igt@drm_read@short-buffer-block,skip
> > +igt@drm_read@short-buffer-nonblock,skip
> > +igt@drm_read@short-buffer-wakeup,skip
> > +igt@kms_addfb_basic@unused-handle,pass
> > +igt@kms_addfb_basic@unused-pitches,pass
> > +igt@kms_addfb_basic@unused-offsets,pass
> > +igt@kms_addfb_basic@unused-modifier,pass
> > +igt@kms_addfb_basic@legacy-format,dmesg-warn
> > +igt@kms_addfb_basic@no-handle,pass
> > +igt@kms_addfb_basic@basic,pass
> > +igt@kms_addfb_basic@bad-pitch-0,pass
> > +igt@kms_addfb_basic@bad-pitch-32,pass
> > +igt@kms_addfb_basic@bad-pitch-63,pass
> > +igt@kms_addfb_basic@bad-pitch-128,pass
> > +igt@kms_addfb_basic@bad-pitch-256,pass
> > +igt@kms_addfb_basic@bad-pitch-1024,pass
> > +igt@kms_addfb_basic@bad-pitch-999,pass
> > +igt@kms_addfb_basic@bad-pitch-65536,pass
> > +igt@kms_addfb_basic@size-max,pass
> > +igt@kms_addfb_basic@too-wide,pass
> > +igt@kms_addfb_basic@too-high,dmesg-warn
> > +igt@kms_addfb_basic@bo-too-small,pass
> > +igt@kms_addfb_basic@small-bo,pass
> > +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> > +igt@kms_addfb_basic@addfb25-bad-modifier,fail
> > +igt@kms_addfb_basic@invalid-get-prop-any,pass
> > +igt@kms_addfb_basic@invalid-get-prop,pass
> > +igt@kms_addfb_basic@invalid-set-prop-any,pass
> > +igt@kms_addfb_basic@invalid-set-prop,pass
> > +igt@kms_addfb_basic@master-rmfb,pass
> > +igt@kms_atomic@plane-overlay-legacy,skip
> > +igt@kms_atomic@plane-primary-legacy,skip
> > +igt@kms_atomic@plane-primary-overlay-mutable-zpos,skip
> > +igt@kms_atomic@plane-immutable-zpos,skip
> > +igt@kms_atomic@test-only,skip
> > +igt@kms_atomic@plane-cursor-legacy,skip
> > +igt@kms_atomic@plane-invalid-params,skip
> > +igt@kms_atomic@crtc-invalid-params,skip
> > +igt@kms_atomic@atomic-invalid-params,skip
> > +igt@kms_atomic@atomic_plane_damage,skip
> > +igt@kms_atomic_interruptible@legacy-setmode,skip
> > +igt@kms_atomic_interruptible@atomic-setmode,skip
> > +igt@kms_atomic_interruptible@legacy-dpms,skip
> > +igt@kms_atomic_interruptible@legacy-pageflip,skip
> > +igt@kms_atomic_interruptible@legacy-cursor,skip
> > +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> > +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> > +igt@kms_content_protection@lic,skip
> > +igt@kms_flip_event_leak,skip
> > +igt@kms_getfb@getfb-handle-zero,pass
> > +igt@kms_getfb@getfb-handle-valid,pass
> > +igt@kms_getfb@getfb-handle-closed,pass
> > +igt@kms_getfb@getfb-handle-not-fb,pass
> > +igt@kms_getfb@getfb-addfb-different-handles,pass
> > +igt@kms_getfb@getfb-repeated-different-handles,pass
> > +igt@kms_getfb@getfb2-handle-zero,pass
> > +igt@kms_getfb@getfb2-handle-closed,pass
> > +igt@kms_getfb@getfb2-handle-not-fb,pass
> > +igt@kms_getfb@getfb2-into-addfb2,pass
> > +igt@kms_getfb@getfb-handle-protection,pass
> > +igt@kms_getfb@getfb2-handle-protection,pass
> > +igt@kms_hdmi_inject@inject-4k,fail
> > +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,skip
> > +igt@kms_pipe_crc_basic@bad-source,skip
> > +igt@kms_pipe_crc_basic@read-crc-pipe-a,skip
> > +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,skip
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,skip
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,skip
> > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,skip
> > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,skip
> > +igt@kms_pipe_crc_basic@read-crc-pipe-b,skip
> > +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,skip
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,skip
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,skip
> > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,skip
> > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,skip
> > +igt@kms_prop_blob@basic,pass
> > +igt@kms_prop_blob@blob-prop-core,pass
> > +igt@kms_prop_blob@blob-prop-validate,pass
> > +igt@kms_prop_blob@blob-prop-lifetime,pass
> > +igt@kms_prop_blob@blob-multiple,pass
> > +igt@kms_prop_blob@invalid-get-prop-any,pass
> > +igt@kms_prop_blob@invalid-get-prop,pass
> > +igt@kms_prop_blob@invalid-set-prop-any,pass
> > +igt@kms_prop_blob@invalid-set-prop,pass
> > +igt@kms_rmfb@rmfb-ioctl,skip
> > +igt@kms_rmfb@close-fd,skip
> > +igt@kms_setmode@basic,skip
> > +igt@msm_mapping@sqefw,skip
> > +igt@msm_mapping@shadow,skip
> > +igt@msm_recovery@hangcheck,skip
> > +igt@msm_recovery@gpu-fault,skip
> > +igt@msm_recovery@iova-fault,skip
> > +igt@msm_submit@empty-submit,pass
> > +igt@msm_submit@invalid-queue-submit,pass
> > +igt@msm_submit@invalid-flags-submit,pass
> > +igt@msm_submit@invalid-in-fence-submit,pass
> > +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> > +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> > +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> > +igt@msm_submit@valid-submit,pass
> > +igt@kms_sysfs_edid_timing,pass
> > +igt@kms_universal_plane@universal-plane-pipe-a-functional,skip
> > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,skip
> > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,skip
> > +igt@kms_vblank@invalid,skip
> > +igt@kms_vblank@crtc-id,skip
> > +igt@kms_vblank@pipe-a-query-idle,skip
> > +igt@kms_vblank@pipe-a-query-forked,skip
> > +igt@kms_vblank@pipe-a-query-busy,skip
> > +igt@kms_vblank@pipe-a-query-forked-busy,skip
> > +igt@kms_vblank@pipe-a-wait-idle,skip
> > +igt@kms_vblank@pipe-a-wait-forked,skip
> > +igt@kms_vblank@pipe-a-wait-busy,skip
> > +igt@kms_vblank@pipe-a-wait-forked-busy,skip
> > +igt@kms_vblank@pipe-a-ts-continuation-idle,skip
> > +igt@kms_vblank@pipe-a-ts-continuation-modeset,skip
> > +igt@kms_vblank@pipe-b-accuracy-idle,skip
> > +igt@kms_vblank@pipe-b-query-idle,skip
> > +igt@kms_vblank@pipe-b-query-forked,skip
> > +igt@kms_vblank@pipe-b-query-busy,skip
> > +igt@kms_vblank@pipe-b-query-forked-busy,skip
> > +igt@kms_vblank@pipe-b-wait-idle,skip
> > +igt@kms_vblank@pipe-b-wait-forked,skip
> > +igt@kms_vblank@pipe-b-wait-busy,skip
> > +igt@kms_vblank@pipe-b-wait-forked-busy,skip
> > +igt@kms_vblank@pipe-b-ts-continuation-idle,skip
> > +igt@kms_vblank@pipe-b-ts-continuation-modeset,skip
> > diff --git a/drivers/gpu/drm/msm/ci/msm_apq8096_results.txt b/drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> > new file mode 100644
> > index 000000000000..6f10941e5626
> > --- /dev/null
> > +++ b/drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> > @@ -0,0 +1,140 @@
> > +igt@core_auth@getclient-simple,pass
> > +igt@core_auth@getclient-master-drop,pass
> > +igt@core_auth@basic-auth,pass
> > +igt@core_auth@many-magics,pass
> > +igt@core_getclient,pass
> > +igt@core_getstats,pass
> > +igt@core_getversion,pass
> > +igt@core_setmaster_vs_auth,pass
> > +igt@drm_read@invalid-buffer,skip
> > +igt@drm_read@fault-buffer,skip
> > +igt@drm_read@empty-block,skip
> > +igt@drm_read@empty-nonblock,skip
> > +igt@drm_read@short-buffer-block,skip
> > +igt@drm_read@short-buffer-nonblock,skip
> > +igt@drm_read@short-buffer-wakeup,skip
> > +igt@kms_addfb_basic@unused-handle,pass
> > +igt@kms_addfb_basic@unused-pitches,pass
> > +igt@kms_addfb_basic@unused-offsets,pass
> > +igt@kms_addfb_basic@unused-modifier,pass
> > +igt@kms_addfb_basic@legacy-format,dmesg-warn
> > +igt@kms_addfb_basic@no-handle,pass
> > +igt@kms_addfb_basic@basic,pass
> > +igt@kms_addfb_basic@bad-pitch-0,pass
> > +igt@kms_addfb_basic@bad-pitch-32,pass
> > +igt@kms_addfb_basic@bad-pitch-63,pass
> > +igt@kms_addfb_basic@bad-pitch-128,pass
> > +igt@kms_addfb_basic@bad-pitch-256,pass
> > +igt@kms_addfb_basic@bad-pitch-1024,pass
> > +igt@kms_addfb_basic@bad-pitch-999,pass
> > +igt@kms_addfb_basic@bad-pitch-65536,pass
> > +igt@kms_addfb_basic@size-max,pass
> > +igt@kms_addfb_basic@too-wide,pass
> > +igt@kms_addfb_basic@too-high,dmesg-warn
> > +igt@kms_addfb_basic@bo-too-small,pass
> > +igt@kms_addfb_basic@small-bo,pass
> > +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> > +igt@kms_addfb_basic@addfb25-bad-modifier,fail
> > +igt@kms_addfb_basic@invalid-get-prop-any,pass
> > +igt@kms_addfb_basic@invalid-get-prop,pass
> > +igt@kms_addfb_basic@invalid-set-prop-any,pass
> > +igt@kms_addfb_basic@invalid-set-prop,pass
> > +igt@kms_addfb_basic@master-rmfb,pass
> > +igt@kms_atomic@plane-overlay-legacy,skip
> > +igt@kms_atomic@plane-primary-legacy,skip
> > +igt@kms_atomic@plane-primary-overlay-mutable-zpos,skip
> > +igt@kms_atomic@plane-immutable-zpos,skip
> > +igt@kms_atomic@test-only,skip
> > +igt@kms_atomic@plane-cursor-legacy,skip
> > +igt@kms_atomic@plane-invalid-params,skip
> > +igt@kms_atomic@crtc-invalid-params,skip
> > +igt@kms_atomic@atomic-invalid-params,skip
> > +igt@kms_atomic@atomic_plane_damage,skip
> > +igt@kms_atomic_interruptible@legacy-setmode,skip
> > +igt@kms_atomic_interruptible@atomic-setmode,skip
> > +igt@kms_atomic_interruptible@legacy-dpms,skip
> > +igt@kms_atomic_interruptible@legacy-pageflip,skip
> > +igt@kms_atomic_interruptible@legacy-cursor,skip
> > +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> > +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> > +igt@kms_content_protection@lic,skip
> > +igt@kms_flip_event_leak,skip
> > +igt@kms_getfb@getfb-handle-zero,pass
> > +igt@kms_getfb@getfb-handle-valid,pass
> > +igt@kms_getfb@getfb-handle-closed,pass
> > +igt@kms_getfb@getfb-handle-not-fb,pass
> > +igt@kms_getfb@getfb-addfb-different-handles,pass
> > +igt@kms_getfb@getfb-repeated-different-handles,pass
> > +igt@kms_getfb@getfb2-handle-zero,pass
> > +igt@kms_getfb@getfb2-handle-closed,pass
> > +igt@kms_getfb@getfb2-handle-not-fb,pass
> > +igt@kms_getfb@getfb2-into-addfb2,pass
> > +igt@kms_getfb@getfb-handle-protection,pass
> > +igt@kms_getfb@getfb2-handle-protection,pass
> > +igt@kms_hdmi_inject@inject-4k,dmesg-warn
> > +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,skip
> > +igt@kms_pipe_crc_basic@bad-source,skip
> > +igt@kms_pipe_crc_basic@read-crc-pipe-a,skip
> > +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,skip
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,skip
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,skip
> > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,skip
> > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,skip
> > +igt@kms_pipe_crc_basic@read-crc-pipe-b,skip
> > +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,skip
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,skip
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,skip
> > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,skip
> > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,skip
> > +igt@kms_prop_blob@basic,pass
> > +igt@kms_prop_blob@blob-prop-core,pass
> > +igt@kms_prop_blob@blob-prop-validate,pass
> > +igt@kms_prop_blob@blob-prop-lifetime,pass
> > +igt@kms_prop_blob@blob-multiple,pass
> > +igt@kms_prop_blob@invalid-get-prop-any,pass
> > +igt@kms_prop_blob@invalid-get-prop,pass
> > +igt@kms_prop_blob@invalid-set-prop-any,pass
> > +igt@kms_prop_blob@invalid-set-prop,pass
> > +igt@kms_rmfb@rmfb-ioctl,skip
> > +igt@kms_rmfb@close-fd,skip
> > +igt@kms_setmode@basic,skip
> > +igt@msm_mapping@sqefw,skip
> > +igt@msm_mapping@shadow,skip
> > +igt@msm_recovery@hangcheck,skip
> > +igt@msm_recovery@gpu-fault,skip
> > +igt@msm_recovery@iova-fault,skip
> > +igt@msm_submit@empty-submit,pass
> > +igt@msm_submit@invalid-queue-submit,pass
> > +igt@msm_submit@invalid-flags-submit,pass
> > +igt@msm_submit@invalid-in-fence-submit,pass
> > +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> > +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> > +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> > +igt@msm_submit@valid-submit,pass
> > +igt@kms_sysfs_edid_timing,pass
> > +igt@kms_universal_plane@universal-plane-pipe-a-functional,skip
> > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,skip
> > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,skip
> > +igt@kms_vblank@invalid,skip
> > +igt@kms_vblank@crtc-id,skip
> > +igt@kms_vblank@pipe-a-query-idle,skip
> > +igt@kms_vblank@pipe-a-query-forked,skip
> > +igt@kms_vblank@pipe-a-query-busy,skip
> > +igt@kms_vblank@pipe-a-query-forked-busy,skip
> > +igt@kms_vblank@pipe-a-wait-idle,skip
> > +igt@kms_vblank@pipe-a-wait-forked,skip
> > +igt@kms_vblank@pipe-a-wait-busy,skip
> > +igt@kms_vblank@pipe-a-wait-forked-busy,skip
> > +igt@kms_vblank@pipe-a-ts-continuation-idle,skip
> > +igt@kms_vblank@pipe-a-ts-continuation-modeset,skip
> > +igt@kms_vblank@pipe-b-accuracy-idle,skip
> > +igt@kms_vblank@pipe-b-query-idle,skip
> > +igt@kms_vblank@pipe-b-query-forked,skip
> > +igt@kms_vblank@pipe-b-query-busy,skip
> > +igt@kms_vblank@pipe-b-query-forked-busy,skip
> > +igt@kms_vblank@pipe-b-wait-idle,skip
> > +igt@kms_vblank@pipe-b-wait-forked,skip
> > +igt@kms_vblank@pipe-b-wait-busy,skip
> > +igt@kms_vblank@pipe-b-wait-forked-busy,skip
> > +igt@kms_vblank@pipe-b-ts-continuation-idle,skip
> > +igt@kms_vblank@pipe-b-ts-continuation-modeset,skip
> > diff --git a/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > new file mode 100644
> > index 000000000000..01f7b4b399b5
> > --- /dev/null
> > +++ b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > @@ -0,0 +1,141 @@
> > +igt@core_auth@getclient-simple,dmesg-warn
> > +igt@core_auth@getclient-master-drop,pass
> > +igt@core_auth@basic-auth,pass
> > +igt@core_auth@many-magics,pass
> > +igt@core_getclient,pass
> > +igt@core_getstats,pass
> > +igt@core_getversion,pass
> > +igt@core_setmaster_vs_auth,pass
> > +igt@drm_read@invalid-buffer,pass
> > +igt@drm_read@fault-buffer,pass
> > +igt@drm_read@empty-block,pass
> > +igt@drm_read@empty-nonblock,pass
> > +igt@drm_read@short-buffer-block,pass
> > +igt@drm_read@short-buffer-nonblock,pass
> > +igt@drm_read@short-buffer-wakeup,pass
> > +igt@kms_addfb_basic@unused-handle,pass
> > +igt@kms_addfb_basic@unused-pitches,pass
> > +igt@kms_addfb_basic@unused-offsets,pass
> > +igt@kms_addfb_basic@unused-modifier,pass
> > +igt@kms_addfb_basic@legacy-format,dmesg-warn
> > +igt@kms_addfb_basic@no-handle,pass
> > +igt@kms_addfb_basic@basic,pass
> > +igt@kms_addfb_basic@bad-pitch-0,pass
> > +igt@kms_addfb_basic@bad-pitch-32,pass
> > +igt@kms_addfb_basic@bad-pitch-63,pass
> > +igt@kms_addfb_basic@bad-pitch-128,pass
> > +igt@kms_addfb_basic@bad-pitch-256,pass
> > +igt@kms_addfb_basic@bad-pitch-1024,pass
> > +igt@kms_addfb_basic@bad-pitch-999,pass
> > +igt@kms_addfb_basic@bad-pitch-65536,pass
> > +igt@kms_addfb_basic@size-max,pass
> > +igt@kms_addfb_basic@too-wide,pass
> > +igt@kms_addfb_basic@too-high,dmesg-warn
>
> For test results on Trogdor, is is possible to have them be
> success/fail/skip only?
>
> Results such as dmesg-warn/dmesg-fail are igt_runner specific and
> because there isn't support for igt_runner on ChromeOS, they will be
> difficult to replicate and debug.

Actually, I wonder if it would be better to just treat
dmesg-warn/dmesg-fail as pass/fail?  I'd noticed some flakes on
rockchip which looked just like unrelated dmesg msg which just
happened to show up while the test was running.

Additionally, some of the tests, like msm_recovery, are *expected* to
generate some dmesg spam since they are intentionally triggering GPU
hangs to test the recovery mechanism.

BR,
-R

> > +igt@kms_addfb_basic@bo-too-small,pass
> > +igt@kms_addfb_basic@small-bo,pass
> > +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> > +igt@kms_addfb_basic@addfb25-bad-modifier,dmesg-warn
> > +igt@kms_addfb_basic@invalid-get-prop-any,pass
> > +igt@kms_addfb_basic@invalid-get-prop,pass
> > +igt@kms_addfb_basic@invalid-set-prop-any,pass
> > +igt@kms_addfb_basic@invalid-set-prop,pass
> > +igt@kms_addfb_basic@master-rmfb,pass
> > +igt@kms_atomic@plane-overlay-legacy,skip
> > +igt@kms_atomic@plane-primary-legacy,pass
> > +igt@kms_atomic@plane-primary-overlay-mutable-zpos,skip
> > +igt@kms_atomic@plane-immutable-zpos,pass
> > +igt@kms_atomic@test-only,pass
> > +igt@kms_atomic@plane-cursor-legacy,pass
> > +igt@kms_atomic@plane-invalid-params,pass
> > +igt@kms_atomic@crtc-invalid-params,pass
> > +igt@kms_atomic@atomic-invalid-params,pass
> > +igt@kms_atomic@atomic_plane_damage,pass
> > +igt@kms_atomic_interruptible@legacy-setmode,skip
> > +igt@kms_atomic_interruptible@atomic-setmode,skip
> > +igt@kms_atomic_interruptible@legacy-dpms,skip
> > +igt@kms_atomic_interruptible@legacy-pageflip,skip
> > +igt@kms_atomic_interruptible@legacy-cursor,skip
> > +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> > +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> > +igt@kms_content_protection@lic,skip
> > +igt@kms_flip_event_leak,pass
> > +igt@kms_getfb@getfb-handle-zero,pass
> > +igt@kms_getfb@getfb-handle-valid,pass
> > +igt@kms_getfb@getfb-handle-closed,pass
> > +igt@kms_getfb@getfb-handle-not-fb,pass
> > +igt@kms_getfb@getfb-addfb-different-handles,pass
> > +igt@kms_getfb@getfb-repeated-different-handles,pass
> > +igt@kms_getfb@getfb2-handle-zero,pass
> > +igt@kms_getfb@getfb2-handle-closed,pass
> > +igt@kms_getfb@getfb2-handle-not-fb,pass
> > +igt@kms_getfb@getfb2-into-addfb2,pass
> > +igt@kms_getfb@getfb-handle-protection,pass
> > +igt@kms_getfb@getfb2-handle-protection,pass
> > +igt@kms_hdmi_inject@inject-4k,skip
> > +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,skip
> > +igt@kms_pipe_crc_basic@bad-source,pass
> > +igt@kms_pipe_crc_basic@read-crc-pipe-a,pass
> > +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,pass
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,pass
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,pass
> > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,pass
> > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,pass
> > +igt@kms_pipe_crc_basic@read-crc-pipe-b,pass
> > +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,pass
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,pass
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,pass
> > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,pass
> > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,pass
> > +igt@kms_prop_blob@basic,pass
> > +igt@kms_prop_blob@blob-prop-core,pass
> > +igt@kms_prop_blob@blob-prop-validate,pass
> > +igt@kms_prop_blob@blob-prop-lifetime,pass
> > +igt@kms_prop_blob@blob-multiple,pass
> > +igt@kms_prop_blob@invalid-get-prop-any,pass
> > +igt@kms_prop_blob@invalid-get-prop,pass
> > +igt@kms_prop_blob@invalid-set-prop-any,pass
> > +igt@kms_prop_blob@invalid-set-prop,pass
> > +igt@kms_rmfb@rmfb-ioctl,pass
> > +igt@kms_rmfb@close-fd,fail
> > +igt@kms_setmode@basic@pipe-a-edp-1,pass
> > +igt@kms_setmode@basic@pipe-b-edp-1,pass
> > +igt@msm_mapping@sqefw,dmesg-fail
> > +igt@msm_mapping@shadow,dmesg-fail
> > +igt@msm_recovery@hangcheck,dmesg-warn
> > +igt@msm_recovery@gpu-fault,dmesg-warn
> > +igt@msm_recovery@iova-fault,dmesg-warn
> > +igt@msm_submit@empty-submit,pass
> > +igt@msm_submit@invalid-queue-submit,pass
> > +igt@msm_submit@invalid-flags-submit,pass
> > +igt@msm_submit@invalid-in-fence-submit,pass
> > +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> > +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> > +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> > +igt@msm_submit@valid-submit,pass
> > +igt@kms_sysfs_edid_timing,pass
> > +igt@kms_universal_plane@universal-plane-pipe-a-functional,skip
> > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,pass
> > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,fail
>
> We could probably skip checking the results for kms_universal_plane on
> Trogdor for now, since this is a test affected by the hack regression.
> There is an IGT patch in the works for fixing the
> disable-primary-vs-flip-pipe-b failure, so it should be updated pretty
> soon too.
>
> Thanks,
> Jessica Zhang
>
> > +igt@kms_vblank@invalid,pass
> > +igt@kms_vblank@crtc-id,pass
> > +igt@kms_vblank@pipe-a-query-idle,pass
> > +igt@kms_vblank@pipe-a-query-forked,pass
> > +igt@kms_vblank@pipe-a-query-busy,pass
> > +igt@kms_vblank@pipe-a-query-forked-busy,pass
> > +igt@kms_vblank@pipe-a-wait-idle,pass
> > +igt@kms_vblank@pipe-a-wait-forked,pass
> > +igt@kms_vblank@pipe-a-wait-busy,pass
> > +igt@kms_vblank@pipe-a-wait-forked-busy,pass
> > +igt@kms_vblank@pipe-a-ts-continuation-idle,pass
> > +igt@kms_vblank@pipe-a-ts-continuation-modeset,pass
> > +igt@kms_vblank@pipe-b-accuracy-idle,pass
> > +igt@kms_vblank@pipe-b-query-idle,pass
> > +igt@kms_vblank@pipe-b-query-forked,pass
> > +igt@kms_vblank@pipe-b-query-busy,pass
> > +igt@kms_vblank@pipe-b-query-forked-busy,pass
> > +igt@kms_vblank@pipe-b-wait-idle,pass
> > +igt@kms_vblank@pipe-b-wait-forked,pass
> > +igt@kms_vblank@pipe-b-wait-busy,pass
> > +igt@kms_vblank@pipe-b-wait-forked-busy,pass
> > +igt@kms_vblank@pipe-b-ts-continuation-idle,pass
> > +igt@kms_vblank@pipe-b-ts-continuation-modeset,pass
> > diff --git a/drivers/gpu/drm/msm/ci/msm_sdm845_results.txt b/drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> > new file mode 100644
> > index 000000000000..3df1e5ef6d80
> > --- /dev/null
> > +++ b/drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> > @@ -0,0 +1,141 @@
> > +igt@core_auth@getclient-simple,dmesg-warn
> > +igt@core_auth@getclient-master-drop,pass
> > +igt@core_auth@basic-auth,pass
> > +igt@core_auth@many-magics,pass
> > +igt@core_getclient,pass
> > +igt@core_getstats,pass
> > +igt@core_getversion,pass
> > +igt@core_setmaster_vs_auth,pass
> > +igt@drm_read@invalid-buffer,pass
> > +igt@drm_read@fault-buffer,pass
> > +igt@drm_read@empty-block,pass
> > +igt@drm_read@empty-nonblock,pass
> > +igt@drm_read@short-buffer-block,pass
> > +igt@drm_read@short-buffer-nonblock,pass
> > +igt@drm_read@short-buffer-wakeup,pass
> > +igt@kms_addfb_basic@unused-handle,pass
> > +igt@kms_addfb_basic@unused-pitches,pass
> > +igt@kms_addfb_basic@unused-offsets,pass
> > +igt@kms_addfb_basic@unused-modifier,pass
> > +igt@kms_addfb_basic@legacy-format,dmesg-warn
> > +igt@kms_addfb_basic@no-handle,pass
> > +igt@kms_addfb_basic@basic,pass
> > +igt@kms_addfb_basic@bad-pitch-0,pass
> > +igt@kms_addfb_basic@bad-pitch-32,pass
> > +igt@kms_addfb_basic@bad-pitch-63,pass
> > +igt@kms_addfb_basic@bad-pitch-128,pass
> > +igt@kms_addfb_basic@bad-pitch-256,pass
> > +igt@kms_addfb_basic@bad-pitch-1024,pass
> > +igt@kms_addfb_basic@bad-pitch-999,pass
> > +igt@kms_addfb_basic@bad-pitch-65536,pass
> > +igt@kms_addfb_basic@size-max,pass
> > +igt@kms_addfb_basic@too-wide,pass
> > +igt@kms_addfb_basic@too-high,dmesg-warn
> > +igt@kms_addfb_basic@bo-too-small,pass
> > +igt@kms_addfb_basic@small-bo,pass
> > +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> > +igt@kms_addfb_basic@addfb25-bad-modifier,dmesg-warn
> > +igt@kms_addfb_basic@invalid-get-prop-any,pass
> > +igt@kms_addfb_basic@invalid-get-prop,pass
> > +igt@kms_addfb_basic@invalid-set-prop-any,pass
> > +igt@kms_addfb_basic@invalid-set-prop,pass
> > +igt@kms_addfb_basic@master-rmfb,pass
> > +igt@kms_atomic@plane-overlay-legacy,dmesg-warn
> > +igt@kms_atomic@plane-primary-legacy,dmesg-warn
> > +igt@kms_atomic@plane-primary-overlay-mutable-zpos,dmesg-warn
> > +igt@kms_atomic@plane-immutable-zpos,dmesg-warn
> > +igt@kms_atomic@test-only,dmesg-warn
> > +igt@kms_atomic@plane-cursor-legacy,dmesg-warn
> > +igt@kms_atomic@plane-invalid-params,dmesg-warn
> > +igt@kms_atomic@crtc-invalid-params,dmesg-warn
> > +igt@kms_atomic@atomic-invalid-params,dmesg-warn
> > +igt@kms_atomic@atomic_plane_damage,dmesg-warn
> > +igt@kms_atomic_interruptible@legacy-setmode,skip
> > +igt@kms_atomic_interruptible@atomic-setmode,skip
> > +igt@kms_atomic_interruptible@legacy-dpms,skip
> > +igt@kms_atomic_interruptible@legacy-pageflip,skip
> > +igt@kms_atomic_interruptible@legacy-cursor,skip
> > +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> > +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> > +igt@kms_content_protection@lic,skip
> > +igt@kms_flip_event_leak,dmesg-warn
> > +igt@kms_getfb@getfb-handle-zero,pass
> > +igt@kms_getfb@getfb-handle-valid,pass
> > +igt@kms_getfb@getfb-handle-closed,pass
> > +igt@kms_getfb@getfb-handle-not-fb,pass
> > +igt@kms_getfb@getfb-addfb-different-handles,pass
> > +igt@kms_getfb@getfb-repeated-different-handles,pass
> > +igt@kms_getfb@getfb2-handle-zero,pass
> > +igt@kms_getfb@getfb2-handle-closed,pass
> > +igt@kms_getfb@getfb2-handle-not-fb,pass
> > +igt@kms_getfb@getfb2-into-addfb2,pass
> > +igt@kms_getfb@getfb-handle-protection,pass
> > +igt@kms_getfb@getfb2-handle-protection,pass
> > +igt@kms_hdmi_inject@inject-4k,skip
> > +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,pass
> > +igt@kms_pipe_crc_basic@bad-source,pass
> > +igt@kms_pipe_crc_basic@read-crc-pipe-a,pass
> > +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,pass
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,pass
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,pass
> > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,dmesg-warn
> > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,dmesg-warn
> > +igt@kms_pipe_crc_basic@read-crc-pipe-b,skip
> > +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,skip
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,skip
> > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,skip
> > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,skip
> > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,skip
> > +igt@kms_prop_blob@basic,pass
> > +igt@kms_prop_blob@blob-prop-core,pass
> > +igt@kms_prop_blob@blob-prop-validate,pass
> > +igt@kms_prop_blob@blob-prop-lifetime,pass
> > +igt@kms_prop_blob@blob-multiple,pass
> > +igt@kms_prop_blob@invalid-get-prop-any,pass
> > +igt@kms_prop_blob@invalid-get-prop,pass
> > +igt@kms_prop_blob@invalid-set-prop-any,pass
> > +igt@kms_prop_blob@invalid-set-prop,pass
> > +igt@kms_rmfb@rmfb-ioctl,pass
> > +igt@kms_rmfb@close-fd,fail
> > +igt@kms_setmode@basic@pipe-a-edp-1,dmesg-warn
> > +igt@kms_setmode@basic,skip
> > +igt@msm_mapping@sqefw,dmesg-fail
> > +igt@msm_mapping@shadow,dmesg-fail
> > +igt@msm_recovery@hangcheck,dmesg-warn
> > +igt@msm_recovery@gpu-fault,dmesg-warn
> > +igt@msm_recovery@iova-fault,dmesg-warn
> > +igt@msm_submit@empty-submit,pass
> > +igt@msm_submit@invalid-queue-submit,pass
> > +igt@msm_submit@invalid-flags-submit,pass
> > +igt@msm_submit@invalid-in-fence-submit,pass
> > +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> > +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> > +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> > +igt@msm_submit@valid-submit,pass
> > +igt@kms_sysfs_edid_timing,pass
> > +igt@kms_universal_plane@universal-plane-pipe-a-functional,dmesg-fail
> > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,dmesg-warn
> > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,skip
> > +igt@kms_vblank@invalid,dmesg-warn
> > +igt@kms_vblank@crtc-id,dmesg-warn
> > +igt@kms_vblank@pipe-a-query-idle,dmesg-warn
> > +igt@kms_vblank@pipe-a-query-forked,dmesg-warn
> > +igt@kms_vblank@pipe-a-query-busy,dmesg-warn
> > +igt@kms_vblank@pipe-a-query-forked-busy,dmesg-warn
> > +igt@kms_vblank@pipe-a-wait-idle,dmesg-warn
> > +igt@kms_vblank@pipe-a-wait-forked,dmesg-warn
> > +igt@kms_vblank@pipe-a-wait-busy,dmesg-warn
> > +igt@kms_vblank@pipe-a-wait-forked-busy,dmesg-warn
> > +igt@kms_vblank@pipe-a-ts-continuation-idle,dmesg-warn
> > +igt@kms_vblank@pipe-a-ts-continuation-modeset,dmesg-warn
> > +igt@kms_vblank@pipe-b-accuracy-idle,skip
> > +igt@kms_vblank@pipe-b-query-idle,skip
> > +igt@kms_vblank@pipe-b-query-forked,skip
> > +igt@kms_vblank@pipe-b-query-busy,skip
> > +igt@kms_vblank@pipe-b-query-forked-busy,skip
> > +igt@kms_vblank@pipe-b-wait-idle,skip
> > +igt@kms_vblank@pipe-b-wait-forked,skip
> > +igt@kms_vblank@pipe-b-wait-busy,skip
> > +igt@kms_vblank@pipe-b-wait-forked-busy,skip
> > +igt@kms_vblank@pipe-b-ts-continuation-idle,skip
> > +igt@kms_vblank@pipe-b-ts-continuation-modeset,skip
> > --
> > 2.31.1
> >

Tomeu Vizoso May 11, 2022, 4:25 a.m. UTC | #3

On 5/10/22 9:39 PM, Jessica Zhang wrote:
> 
> 
> On 5/10/2022 7:13 AM, Tomeu Vizoso wrote:
>> +igt@kms_atomic_interruptible@legacy-setmode@pipe-a-edp-1
>> +igt@kms_atomic_interruptible@atomic-setmode@pipe-a-edp-1
>> +igt@kms_atomic_interruptible@legacy-dpms@pipe-a-edp-1
>> +igt@kms_atomic_interruptible@legacy-pageflip@pipe-a-edp-1
>> +igt@kms_atomic_interruptible@legacy-cursor@pipe-a-edp-1
>> +igt@kms_atomic_interruptible@universal-setplane-primary@pipe-a-edp-1
>> +igt@kms_atomic_interruptible@universal-setplane-cursor@pipe-a-edp-1
> 
> I see that there are some KMS subtests/tests are missing such as 
> kms_atomic_transition -- are they unstable across all boards?
> 
>  From what I've seen, kms_atomic_transition is unstable only on Trogdor, 
> but has stable results on other boards.

I don't know of a good reason for not having the tests in this list. 
Developers can follow the instructions at "How to expand coverage" to 
add tests if the driver can consistently pass (or fail) them.

There is ongoing work to have a test runner that will better handle 
flaky tests, which will make it possible to expand the list without 
having to resort to per-soc test lists.

>> +igt@kms_addfb_basic@size-max,pass
>> +igt@kms_addfb_basic@too-wide,pass
>> +igt@kms_addfb_basic@too-high,dmesg-warn
> 
> For test results on Trogdor, is is possible to have them be 
> success/fail/skip only?
> 
> Results such as dmesg-warn/dmesg-fail are igt_runner specific and 
> because there isn't support for igt_runner on ChromeOS, they will be 
> difficult to replicate and debug.

As Rob said later, it's probably better to drop that distinction. Maybe 
at some future point we will want to track expected kmsg output but I 
think there are lots of work to do before that makes sense.

Will do that in a v3.

>> +igt@kms_universal_plane@universal-plane-pipe-a-functional,skip
>> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,pass
>> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,fail
> 
> We could probably skip checking the results for kms_universal_plane on 
> Trogdor for now, since this is a test affected by the hack regression. 
> There is an IGT patch in the works for fixing the 
> disable-primary-vs-flip-pipe-b failure, so it should be updated pretty 
> soon too.

So, how this should work is for the patch that fixes that bug to update 
also the expectation files (see "How to update test expectations" in the 
docs). In this case though, the fix is likely to land before this patch, 
so I will just update the expectations once I rebase and the fix is 
brought into my branch.

Thanks for the feedback!

Tomeu

Dave Airlie May 11, 2022, 5:06 a.m. UTC | #4

> And use it to store expectations about what the drm/msm driver is
> supposed to pass in the IGT test suite.

I wanted to loop in Linus/Greg to see if there are any issues raised
by adding CI results file to the tree in their minds, or if any other
subsystem has done this already, and it's all fine.

I think this is a good thing after our Mesa experience, but Mesa has a
lot tighter integration here, so I want to get some more opinions
outside the group.

Dave.

>
> Also include a configuration file that points to the out-of-tree CI
> scripts.
>
> By storing the test expectations along the code we can make sure both
> stay in sync with each other, and so we can know when a code change
> breaks those expectations.
>
> This will allow all contributors to drm/msm to reuse the infrastructure
> already in gitlab.freedesktop.org to test the driver on several
> generations of the hardware.
>
> v2:
>   - Fix names of result expectation files to match SoC
>   - Don't execute tests that are going to skip on all boards
>
> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> ---
>  Documentation/gpu/msm_automated_testing.rst   |  70 +++++++++
>  drivers/gpu/drm/msm/ci/gitlab-ci.yml          |  11 ++
>  drivers/gpu/drm/msm/ci/msm.testlist           | 148 ++++++++++++++++++
>  .../gpu/drm/msm/ci/msm_apq8016_results.txt    | 140 +++++++++++++++++
>  .../gpu/drm/msm/ci/msm_apq8096_results.txt    | 140 +++++++++++++++++
>  drivers/gpu/drm/msm/ci/msm_sc7180_results.txt | 141 +++++++++++++++++
>  drivers/gpu/drm/msm/ci/msm_sdm845_results.txt | 141 +++++++++++++++++
>  7 files changed, 791 insertions(+)
>  create mode 100644 Documentation/gpu/msm_automated_testing.rst
>  create mode 100644 drivers/gpu/drm/msm/ci/gitlab-ci.yml
>  create mode 100644 drivers/gpu/drm/msm/ci/msm.testlist
>  create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
>  create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
>  create mode 100644 drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
>  create mode 100644 drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
>
> diff --git a/Documentation/gpu/msm_automated_testing.rst b/Documentation/gpu/msm_automated_testing.rst
> new file mode 100644
> index 000000000000..fe59474569c2
> --- /dev/null
> +++ b/Documentation/gpu/msm_automated_testing.rst
> @@ -0,0 +1,70 @@
> +.. SPDX-License-Identifier: GPL-2.0+
> +
> +=========================================
> +Automated testing of MSM DRM driver
> +=========================================
> +
> +
> +Introduction
> +============
> +
> +Making sure that changes to a driver don't introduce regressions can be very time consuming when lots of different hardware configurations need to be tested. Moreover, it isn't practical for each person interested in this testing to have to acquire and maintain what can be a considerable amount of hardware.
> +
> +Also, it is desirable for developers to check for regressions in their code by themselves, instead of relying on the maintainer finding them and then reporting back.
> +
> +There are facilities in gitlab.freedesktop.org to automatically test Mesa that can be used as well for testing DRM drivers such as drm/msm. This document explains how people interested in testing the drm/msm driver can use this shared infrastructure to save quite some time and effort.
> +
> +
> +Relevant files
> +==============
> +
> +drivers/gpu/drm/msm/ci/gitlab-ci.yml
> +------------------------------------
> +
> +Specifies the specific version of the scripts to be used. GitLab CI will use the values defined in this file to fetch the right scripts.
> +
> +
> +drivers/gpu/drm/msm/ci/msm.testlist
> +-----------------------------------
> +
> +Specifies the tests that the current code is expected to be able to reliably run. These tests are expected to not hang the DUT (device under testing) when running on the revision they belong to, and to give consistent results.
> +
> +
> +drivers/gpu/drm/msm/ci/msm_*_results.txt
> +----------------------------------------
> +
> +Specifies the expected results of running this specific kernel revision on a given hardware configuration.
> +
> +
> +How to enable automated testing on your tree
> +============================================
> +
> +1. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one yet
> +
> +2. In your kernel repo's configuration (eg. https://gitlab.freedesktop.org/tomeu/linux/-/settings/ci_cd), change the CI/CD configuration file from .gitlab-ci.yml to drivers/gpu/drm/msm/ci/gitlab-ci.yml.
> +
> +3. Next time you push to this repository, you will see a CI pipeline being created (eg. https://gitlab.freedesktop.org/tomeu/linux/-/pipelines)
> +
> +4. The various jobs will be run and when the pipeline is finished, all jobs should be green unless a regression has been found.
> +
> +
> +How to update test expectations
> +===============================
> +
> +If your changes to the code fix any tests, you will have to update one or more of the files in drivers/gpu/drm/msm/ci/msm_*_results.txt, for each of the test platforms affected by the change.
> +
> +If you have run a pipeline and it failed because of a mismatched test result, you will have been offered a patch at the end of the run that you can apply after reviewing it.
> +
> +
> +How to expand coverage
> +======================
> +
> +If your code changes makes it possible to run more tests (by solving reliability issues, for example), more tests can be added to this list, and then the expected results updated with the patch file that will be printed at the end of each job.
> +
> +If there is a need for updating the version of IGT being used (maybe you have added more tests to it), follow the instructions at https://gitlab.freedesktop.org/gfx-ci/drm-ci/-/tree/msm#upreving-igt.
> +
> +
> +How to test your changes to the scripts
> +==========================================
> +
> +For testing changes to the scripts in the drm-ci repo, change the DRM_CI_PROJECT_PATH variable in drivers/gpu/drm/msm/ci/gitlab-ci.yml to match your fork of the project (eg. tomeu/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/.
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/msm/ci/gitlab-ci.yml b/drivers/gpu/drm/msm/ci/gitlab-ci.yml
> new file mode 100644
> index 000000000000..9b7caa7fcab2
> --- /dev/null
> +++ b/drivers/gpu/drm/msm/ci/gitlab-ci.yml
> @@ -0,0 +1,11 @@
> +variables:
> +  # Change this to use your fork of drm-ci
> +  DRM_CI_PROJECT_PATH: &drm-ci-project-path gfx-ci/drm-ci
> +  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha msm
> +
> +include:
> +  - project: *drm-ci-project-path
> +    ref: *drm-ci-commit-sha
> +    file:
> +      - '.msm-gitlab-ci.yml'
> +      - '.gitlab-ci.yml'
> diff --git a/drivers/gpu/drm/msm/ci/msm.testlist b/drivers/gpu/drm/msm/ci/msm.testlist
> new file mode 100644
> index 000000000000..8805a3bc7316
> --- /dev/null
> +++ b/drivers/gpu/drm/msm/ci/msm.testlist
> @@ -0,0 +1,148 @@
> +# Keep alphabetically sorted by default
> +
> +igt@core_auth@getclient-simple
> +igt@core_auth@getclient-master-drop
> +igt@core_auth@basic-auth
> +igt@core_auth@many-magics
> +igt@core_getclient
> +igt@core_getstats
> +igt@core_getversion
> +igt@core_setmaster_vs_auth
> +igt@drm_read@invalid-buffer
> +igt@drm_read@fault-buffer
> +igt@drm_read@empty-block
> +igt@drm_read@empty-nonblock
> +igt@drm_read@short-buffer-block
> +igt@drm_read@short-buffer-nonblock
> +igt@drm_read@short-buffer-wakeup
> +igt@kms_addfb_basic@unused-handle
> +igt@kms_addfb_basic@unused-pitches
> +igt@kms_addfb_basic@unused-offsets
> +igt@kms_addfb_basic@unused-modifier
> +igt@kms_addfb_basic@legacy-format
> +igt@kms_addfb_basic@no-handle
> +igt@kms_addfb_basic@basic
> +igt@kms_addfb_basic@bad-pitch-0
> +igt@kms_addfb_basic@bad-pitch-32
> +igt@kms_addfb_basic@bad-pitch-63
> +igt@kms_addfb_basic@bad-pitch-128
> +igt@kms_addfb_basic@bad-pitch-256
> +igt@kms_addfb_basic@bad-pitch-1024
> +igt@kms_addfb_basic@bad-pitch-999
> +igt@kms_addfb_basic@bad-pitch-65536
> +igt@kms_addfb_basic@size-max
> +igt@kms_addfb_basic@too-wide
> +igt@kms_addfb_basic@too-high
> +igt@kms_addfb_basic@bo-too-small
> +igt@kms_addfb_basic@small-bo
> +igt@kms_addfb_basic@addfb25-modifier-no-flag
> +igt@kms_addfb_basic@addfb25-bad-modifier
> +igt@kms_addfb_basic@invalid-get-prop-any
> +igt@kms_addfb_basic@invalid-get-prop
> +igt@kms_addfb_basic@invalid-set-prop-any
> +igt@kms_addfb_basic@invalid-set-prop
> +igt@kms_addfb_basic@master-rmfb
> +igt@kms_atomic@plane-overlay-legacy
> +igt@kms_atomic@plane-primary-legacy
> +igt@kms_atomic@plane-primary-overlay-mutable-zpos
> +igt@kms_atomic@plane-immutable-zpos
> +igt@kms_atomic@test-only
> +igt@kms_atomic@plane-cursor-legacy
> +igt@kms_atomic@plane-invalid-params
> +igt@kms_atomic@crtc-invalid-params
> +igt@kms_atomic@atomic-invalid-params
> +igt@kms_atomic@atomic_plane_damage
> +igt@kms_atomic_interruptible@legacy-setmode@pipe-a-edp-1
> +igt@kms_atomic_interruptible@atomic-setmode@pipe-a-edp-1
> +igt@kms_atomic_interruptible@legacy-dpms@pipe-a-edp-1
> +igt@kms_atomic_interruptible@legacy-pageflip@pipe-a-edp-1
> +igt@kms_atomic_interruptible@legacy-cursor@pipe-a-edp-1
> +igt@kms_atomic_interruptible@universal-setplane-primary@pipe-a-edp-1
> +igt@kms_atomic_interruptible@universal-setplane-cursor@pipe-a-edp-1
> +igt@kms_content_protection@LIC
> +igt@kms_flip_event_leak
> +igt@kms_getfb@getfb-handle-zero
> +igt@kms_getfb@getfb-handle-valid
> +igt@kms_getfb@getfb-handle-closed
> +igt@kms_getfb@getfb-handle-not-fb
> +igt@kms_getfb@getfb-addfb-different-handles
> +igt@kms_getfb@getfb-repeated-different-handles
> +igt@kms_getfb@getfb2-handle-zero
> +igt@kms_getfb@getfb2-handle-closed
> +igt@kms_getfb@getfb2-handle-not-fb
> +igt@kms_getfb@getfb2-into-addfb2
> +igt@kms_getfb@getfb-handle-protection
> +igt@kms_getfb@getfb2-handle-protection
> +igt@kms_hdmi_inject@inject-4k
> +igt@kms_multipipe_modeset@basic-max-pipe-crc-check
> +igt@kms_pipe_crc_basic@bad-source
> +igt@kms_pipe_crc_basic@read-crc-pipe-a
> +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a
> +igt@kms_pipe_crc_basic@read-crc-pipe-b
> +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b
> +igt@kms_prop_blob@basic
> +igt@kms_prop_blob@blob-prop-core
> +igt@kms_prop_blob@blob-prop-validate
> +igt@kms_prop_blob@blob-prop-lifetime
> +igt@kms_prop_blob@blob-multiple
> +igt@kms_prop_blob@invalid-get-prop-any
> +igt@kms_prop_blob@invalid-get-prop
> +igt@kms_prop_blob@invalid-set-prop-any
> +igt@kms_prop_blob@invalid-set-prop
> +igt@kms_rmfb@rmfb-ioctl
> +igt@kms_rmfb@close-fd
> +igt@kms_setmode@basic@pipe-a-edp-1
> +igt@kms_setmode@basic@pipe-b-edp-1
> +
> +# Flakes on a530 with:
> +# arm-smmu b40000.iommu: Unhandled context fault: fsr=0x408, iova=0x01003000, fsynr=0x13, cbfrsynra=0x0, cb=0
> +# igt@msm_mapping@ring
> +
> +igt@msm_mapping@sqefw
> +igt@msm_mapping@shadow
> +igt@msm_recovery@hangcheck
> +igt@msm_recovery@gpu-fault
> +igt@msm_recovery@iova-fault
> +igt@msm_submit@empty-submit
> +igt@msm_submit@invalid-queue-submit
> +igt@msm_submit@invalid-flags-submit
> +igt@msm_submit@invalid-in-fence-submit
> +igt@msm_submit@invalid-duplicate-bo-submit
> +igt@msm_submit@invalid-cmd-idx-submit
> +igt@msm_submit@invalid-cmd-type-submit
> +igt@msm_submit@valid-submit
> +igt@kms_sysfs_edid_timing
> +igt@kms_universal_plane@universal-plane-pipe-a-functional
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b
> +igt@kms_vblank@invalid
> +igt@kms_vblank@crtc-id
> +igt@kms_vblank@pipe-a-query-idle
> +igt@kms_vblank@pipe-a-query-forked
> +igt@kms_vblank@pipe-a-query-busy
> +igt@kms_vblank@pipe-a-query-forked-busy
> +igt@kms_vblank@pipe-a-wait-idle
> +igt@kms_vblank@pipe-a-wait-forked
> +igt@kms_vblank@pipe-a-wait-busy
> +igt@kms_vblank@pipe-a-wait-forked-busy
> +igt@kms_vblank@pipe-a-ts-continuation-idle
> +igt@kms_vblank@pipe-a-ts-continuation-modeset
> +igt@kms_vblank@pipe-b-accuracy-idle
> +igt@kms_vblank@pipe-b-query-idle
> +igt@kms_vblank@pipe-b-query-forked
> +igt@kms_vblank@pipe-b-query-busy
> +igt@kms_vblank@pipe-b-query-forked-busy
> +igt@kms_vblank@pipe-b-wait-idle
> +igt@kms_vblank@pipe-b-wait-forked
> +igt@kms_vblank@pipe-b-wait-busy
> +igt@kms_vblank@pipe-b-wait-forked-busy
> +igt@kms_vblank@pipe-b-ts-continuation-idle
> +igt@kms_vblank@pipe-b-ts-continuation-modeset
> diff --git a/drivers/gpu/drm/msm/ci/msm_apq8016_results.txt b/drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> new file mode 100644
> index 000000000000..2cad61d89faf
> --- /dev/null
> +++ b/drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> @@ -0,0 +1,140 @@
> +igt@core_auth@getclient-simple,pass
> +igt@core_auth@getclient-master-drop,pass
> +igt@core_auth@basic-auth,pass
> +igt@core_auth@many-magics,pass
> +igt@core_getclient,pass
> +igt@core_getstats,pass
> +igt@core_getversion,pass
> +igt@core_setmaster_vs_auth,pass
> +igt@drm_read@invalid-buffer,skip
> +igt@drm_read@fault-buffer,skip
> +igt@drm_read@empty-block,skip
> +igt@drm_read@empty-nonblock,skip
> +igt@drm_read@short-buffer-block,skip
> +igt@drm_read@short-buffer-nonblock,skip
> +igt@drm_read@short-buffer-wakeup,skip
> +igt@kms_addfb_basic@unused-handle,pass
> +igt@kms_addfb_basic@unused-pitches,pass
> +igt@kms_addfb_basic@unused-offsets,pass
> +igt@kms_addfb_basic@unused-modifier,pass
> +igt@kms_addfb_basic@legacy-format,dmesg-warn
> +igt@kms_addfb_basic@no-handle,pass
> +igt@kms_addfb_basic@basic,pass
> +igt@kms_addfb_basic@bad-pitch-0,pass
> +igt@kms_addfb_basic@bad-pitch-32,pass
> +igt@kms_addfb_basic@bad-pitch-63,pass
> +igt@kms_addfb_basic@bad-pitch-128,pass
> +igt@kms_addfb_basic@bad-pitch-256,pass
> +igt@kms_addfb_basic@bad-pitch-1024,pass
> +igt@kms_addfb_basic@bad-pitch-999,pass
> +igt@kms_addfb_basic@bad-pitch-65536,pass
> +igt@kms_addfb_basic@size-max,pass
> +igt@kms_addfb_basic@too-wide,pass
> +igt@kms_addfb_basic@too-high,dmesg-warn
> +igt@kms_addfb_basic@bo-too-small,pass
> +igt@kms_addfb_basic@small-bo,pass
> +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> +igt@kms_addfb_basic@addfb25-bad-modifier,fail
> +igt@kms_addfb_basic@invalid-get-prop-any,pass
> +igt@kms_addfb_basic@invalid-get-prop,pass
> +igt@kms_addfb_basic@invalid-set-prop-any,pass
> +igt@kms_addfb_basic@invalid-set-prop,pass
> +igt@kms_addfb_basic@master-rmfb,pass
> +igt@kms_atomic@plane-overlay-legacy,skip
> +igt@kms_atomic@plane-primary-legacy,skip
> +igt@kms_atomic@plane-primary-overlay-mutable-zpos,skip
> +igt@kms_atomic@plane-immutable-zpos,skip
> +igt@kms_atomic@test-only,skip
> +igt@kms_atomic@plane-cursor-legacy,skip
> +igt@kms_atomic@plane-invalid-params,skip
> +igt@kms_atomic@crtc-invalid-params,skip
> +igt@kms_atomic@atomic-invalid-params,skip
> +igt@kms_atomic@atomic_plane_damage,skip
> +igt@kms_atomic_interruptible@legacy-setmode,skip
> +igt@kms_atomic_interruptible@atomic-setmode,skip
> +igt@kms_atomic_interruptible@legacy-dpms,skip
> +igt@kms_atomic_interruptible@legacy-pageflip,skip
> +igt@kms_atomic_interruptible@legacy-cursor,skip
> +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> +igt@kms_content_protection@lic,skip
> +igt@kms_flip_event_leak,skip
> +igt@kms_getfb@getfb-handle-zero,pass
> +igt@kms_getfb@getfb-handle-valid,pass
> +igt@kms_getfb@getfb-handle-closed,pass
> +igt@kms_getfb@getfb-handle-not-fb,pass
> +igt@kms_getfb@getfb-addfb-different-handles,pass
> +igt@kms_getfb@getfb-repeated-different-handles,pass
> +igt@kms_getfb@getfb2-handle-zero,pass
> +igt@kms_getfb@getfb2-handle-closed,pass
> +igt@kms_getfb@getfb2-handle-not-fb,pass
> +igt@kms_getfb@getfb2-into-addfb2,pass
> +igt@kms_getfb@getfb-handle-protection,pass
> +igt@kms_getfb@getfb2-handle-protection,pass
> +igt@kms_hdmi_inject@inject-4k,fail
> +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,skip
> +igt@kms_pipe_crc_basic@bad-source,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-a,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,skip
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,skip
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-b,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,skip
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,skip
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,skip
> +igt@kms_prop_blob@basic,pass
> +igt@kms_prop_blob@blob-prop-core,pass
> +igt@kms_prop_blob@blob-prop-validate,pass
> +igt@kms_prop_blob@blob-prop-lifetime,pass
> +igt@kms_prop_blob@blob-multiple,pass
> +igt@kms_prop_blob@invalid-get-prop-any,pass
> +igt@kms_prop_blob@invalid-get-prop,pass
> +igt@kms_prop_blob@invalid-set-prop-any,pass
> +igt@kms_prop_blob@invalid-set-prop,pass
> +igt@kms_rmfb@rmfb-ioctl,skip
> +igt@kms_rmfb@close-fd,skip
> +igt@kms_setmode@basic,skip
> +igt@msm_mapping@sqefw,skip
> +igt@msm_mapping@shadow,skip
> +igt@msm_recovery@hangcheck,skip
> +igt@msm_recovery@gpu-fault,skip
> +igt@msm_recovery@iova-fault,skip
> +igt@msm_submit@empty-submit,pass
> +igt@msm_submit@invalid-queue-submit,pass
> +igt@msm_submit@invalid-flags-submit,pass
> +igt@msm_submit@invalid-in-fence-submit,pass
> +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> +igt@msm_submit@valid-submit,pass
> +igt@kms_sysfs_edid_timing,pass
> +igt@kms_universal_plane@universal-plane-pipe-a-functional,skip
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,skip
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,skip
> +igt@kms_vblank@invalid,skip
> +igt@kms_vblank@crtc-id,skip
> +igt@kms_vblank@pipe-a-query-idle,skip
> +igt@kms_vblank@pipe-a-query-forked,skip
> +igt@kms_vblank@pipe-a-query-busy,skip
> +igt@kms_vblank@pipe-a-query-forked-busy,skip
> +igt@kms_vblank@pipe-a-wait-idle,skip
> +igt@kms_vblank@pipe-a-wait-forked,skip
> +igt@kms_vblank@pipe-a-wait-busy,skip
> +igt@kms_vblank@pipe-a-wait-forked-busy,skip
> +igt@kms_vblank@pipe-a-ts-continuation-idle,skip
> +igt@kms_vblank@pipe-a-ts-continuation-modeset,skip
> +igt@kms_vblank@pipe-b-accuracy-idle,skip
> +igt@kms_vblank@pipe-b-query-idle,skip
> +igt@kms_vblank@pipe-b-query-forked,skip
> +igt@kms_vblank@pipe-b-query-busy,skip
> +igt@kms_vblank@pipe-b-query-forked-busy,skip
> +igt@kms_vblank@pipe-b-wait-idle,skip
> +igt@kms_vblank@pipe-b-wait-forked,skip
> +igt@kms_vblank@pipe-b-wait-busy,skip
> +igt@kms_vblank@pipe-b-wait-forked-busy,skip
> +igt@kms_vblank@pipe-b-ts-continuation-idle,skip
> +igt@kms_vblank@pipe-b-ts-continuation-modeset,skip
> diff --git a/drivers/gpu/drm/msm/ci/msm_apq8096_results.txt b/drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> new file mode 100644
> index 000000000000..6f10941e5626
> --- /dev/null
> +++ b/drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> @@ -0,0 +1,140 @@
> +igt@core_auth@getclient-simple,pass
> +igt@core_auth@getclient-master-drop,pass
> +igt@core_auth@basic-auth,pass
> +igt@core_auth@many-magics,pass
> +igt@core_getclient,pass
> +igt@core_getstats,pass
> +igt@core_getversion,pass
> +igt@core_setmaster_vs_auth,pass
> +igt@drm_read@invalid-buffer,skip
> +igt@drm_read@fault-buffer,skip
> +igt@drm_read@empty-block,skip
> +igt@drm_read@empty-nonblock,skip
> +igt@drm_read@short-buffer-block,skip
> +igt@drm_read@short-buffer-nonblock,skip
> +igt@drm_read@short-buffer-wakeup,skip
> +igt@kms_addfb_basic@unused-handle,pass
> +igt@kms_addfb_basic@unused-pitches,pass
> +igt@kms_addfb_basic@unused-offsets,pass
> +igt@kms_addfb_basic@unused-modifier,pass
> +igt@kms_addfb_basic@legacy-format,dmesg-warn
> +igt@kms_addfb_basic@no-handle,pass
> +igt@kms_addfb_basic@basic,pass
> +igt@kms_addfb_basic@bad-pitch-0,pass
> +igt@kms_addfb_basic@bad-pitch-32,pass
> +igt@kms_addfb_basic@bad-pitch-63,pass
> +igt@kms_addfb_basic@bad-pitch-128,pass
> +igt@kms_addfb_basic@bad-pitch-256,pass
> +igt@kms_addfb_basic@bad-pitch-1024,pass
> +igt@kms_addfb_basic@bad-pitch-999,pass
> +igt@kms_addfb_basic@bad-pitch-65536,pass
> +igt@kms_addfb_basic@size-max,pass
> +igt@kms_addfb_basic@too-wide,pass
> +igt@kms_addfb_basic@too-high,dmesg-warn
> +igt@kms_addfb_basic@bo-too-small,pass
> +igt@kms_addfb_basic@small-bo,pass
> +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> +igt@kms_addfb_basic@addfb25-bad-modifier,fail
> +igt@kms_addfb_basic@invalid-get-prop-any,pass
> +igt@kms_addfb_basic@invalid-get-prop,pass
> +igt@kms_addfb_basic@invalid-set-prop-any,pass
> +igt@kms_addfb_basic@invalid-set-prop,pass
> +igt@kms_addfb_basic@master-rmfb,pass
> +igt@kms_atomic@plane-overlay-legacy,skip
> +igt@kms_atomic@plane-primary-legacy,skip
> +igt@kms_atomic@plane-primary-overlay-mutable-zpos,skip
> +igt@kms_atomic@plane-immutable-zpos,skip
> +igt@kms_atomic@test-only,skip
> +igt@kms_atomic@plane-cursor-legacy,skip
> +igt@kms_atomic@plane-invalid-params,skip
> +igt@kms_atomic@crtc-invalid-params,skip
> +igt@kms_atomic@atomic-invalid-params,skip
> +igt@kms_atomic@atomic_plane_damage,skip
> +igt@kms_atomic_interruptible@legacy-setmode,skip
> +igt@kms_atomic_interruptible@atomic-setmode,skip
> +igt@kms_atomic_interruptible@legacy-dpms,skip
> +igt@kms_atomic_interruptible@legacy-pageflip,skip
> +igt@kms_atomic_interruptible@legacy-cursor,skip
> +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> +igt@kms_content_protection@lic,skip
> +igt@kms_flip_event_leak,skip
> +igt@kms_getfb@getfb-handle-zero,pass
> +igt@kms_getfb@getfb-handle-valid,pass
> +igt@kms_getfb@getfb-handle-closed,pass
> +igt@kms_getfb@getfb-handle-not-fb,pass
> +igt@kms_getfb@getfb-addfb-different-handles,pass
> +igt@kms_getfb@getfb-repeated-different-handles,pass
> +igt@kms_getfb@getfb2-handle-zero,pass
> +igt@kms_getfb@getfb2-handle-closed,pass
> +igt@kms_getfb@getfb2-handle-not-fb,pass
> +igt@kms_getfb@getfb2-into-addfb2,pass
> +igt@kms_getfb@getfb-handle-protection,pass
> +igt@kms_getfb@getfb2-handle-protection,pass
> +igt@kms_hdmi_inject@inject-4k,dmesg-warn
> +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,skip
> +igt@kms_pipe_crc_basic@bad-source,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-a,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,skip
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,skip
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-b,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,skip
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,skip
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,skip
> +igt@kms_prop_blob@basic,pass
> +igt@kms_prop_blob@blob-prop-core,pass
> +igt@kms_prop_blob@blob-prop-validate,pass
> +igt@kms_prop_blob@blob-prop-lifetime,pass
> +igt@kms_prop_blob@blob-multiple,pass
> +igt@kms_prop_blob@invalid-get-prop-any,pass
> +igt@kms_prop_blob@invalid-get-prop,pass
> +igt@kms_prop_blob@invalid-set-prop-any,pass
> +igt@kms_prop_blob@invalid-set-prop,pass
> +igt@kms_rmfb@rmfb-ioctl,skip
> +igt@kms_rmfb@close-fd,skip
> +igt@kms_setmode@basic,skip
> +igt@msm_mapping@sqefw,skip
> +igt@msm_mapping@shadow,skip
> +igt@msm_recovery@hangcheck,skip
> +igt@msm_recovery@gpu-fault,skip
> +igt@msm_recovery@iova-fault,skip
> +igt@msm_submit@empty-submit,pass
> +igt@msm_submit@invalid-queue-submit,pass
> +igt@msm_submit@invalid-flags-submit,pass
> +igt@msm_submit@invalid-in-fence-submit,pass
> +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> +igt@msm_submit@valid-submit,pass
> +igt@kms_sysfs_edid_timing,pass
> +igt@kms_universal_plane@universal-plane-pipe-a-functional,skip
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,skip
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,skip
> +igt@kms_vblank@invalid,skip
> +igt@kms_vblank@crtc-id,skip
> +igt@kms_vblank@pipe-a-query-idle,skip
> +igt@kms_vblank@pipe-a-query-forked,skip
> +igt@kms_vblank@pipe-a-query-busy,skip
> +igt@kms_vblank@pipe-a-query-forked-busy,skip
> +igt@kms_vblank@pipe-a-wait-idle,skip
> +igt@kms_vblank@pipe-a-wait-forked,skip
> +igt@kms_vblank@pipe-a-wait-busy,skip
> +igt@kms_vblank@pipe-a-wait-forked-busy,skip
> +igt@kms_vblank@pipe-a-ts-continuation-idle,skip
> +igt@kms_vblank@pipe-a-ts-continuation-modeset,skip
> +igt@kms_vblank@pipe-b-accuracy-idle,skip
> +igt@kms_vblank@pipe-b-query-idle,skip
> +igt@kms_vblank@pipe-b-query-forked,skip
> +igt@kms_vblank@pipe-b-query-busy,skip
> +igt@kms_vblank@pipe-b-query-forked-busy,skip
> +igt@kms_vblank@pipe-b-wait-idle,skip
> +igt@kms_vblank@pipe-b-wait-forked,skip
> +igt@kms_vblank@pipe-b-wait-busy,skip
> +igt@kms_vblank@pipe-b-wait-forked-busy,skip
> +igt@kms_vblank@pipe-b-ts-continuation-idle,skip
> +igt@kms_vblank@pipe-b-ts-continuation-modeset,skip
> diff --git a/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> new file mode 100644
> index 000000000000..01f7b4b399b5
> --- /dev/null
> +++ b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> @@ -0,0 +1,141 @@
> +igt@core_auth@getclient-simple,dmesg-warn
> +igt@core_auth@getclient-master-drop,pass
> +igt@core_auth@basic-auth,pass
> +igt@core_auth@many-magics,pass
> +igt@core_getclient,pass
> +igt@core_getstats,pass
> +igt@core_getversion,pass
> +igt@core_setmaster_vs_auth,pass
> +igt@drm_read@invalid-buffer,pass
> +igt@drm_read@fault-buffer,pass
> +igt@drm_read@empty-block,pass
> +igt@drm_read@empty-nonblock,pass
> +igt@drm_read@short-buffer-block,pass
> +igt@drm_read@short-buffer-nonblock,pass
> +igt@drm_read@short-buffer-wakeup,pass
> +igt@kms_addfb_basic@unused-handle,pass
> +igt@kms_addfb_basic@unused-pitches,pass
> +igt@kms_addfb_basic@unused-offsets,pass
> +igt@kms_addfb_basic@unused-modifier,pass
> +igt@kms_addfb_basic@legacy-format,dmesg-warn
> +igt@kms_addfb_basic@no-handle,pass
> +igt@kms_addfb_basic@basic,pass
> +igt@kms_addfb_basic@bad-pitch-0,pass
> +igt@kms_addfb_basic@bad-pitch-32,pass
> +igt@kms_addfb_basic@bad-pitch-63,pass
> +igt@kms_addfb_basic@bad-pitch-128,pass
> +igt@kms_addfb_basic@bad-pitch-256,pass
> +igt@kms_addfb_basic@bad-pitch-1024,pass
> +igt@kms_addfb_basic@bad-pitch-999,pass
> +igt@kms_addfb_basic@bad-pitch-65536,pass
> +igt@kms_addfb_basic@size-max,pass
> +igt@kms_addfb_basic@too-wide,pass
> +igt@kms_addfb_basic@too-high,dmesg-warn
> +igt@kms_addfb_basic@bo-too-small,pass
> +igt@kms_addfb_basic@small-bo,pass
> +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> +igt@kms_addfb_basic@addfb25-bad-modifier,dmesg-warn
> +igt@kms_addfb_basic@invalid-get-prop-any,pass
> +igt@kms_addfb_basic@invalid-get-prop,pass
> +igt@kms_addfb_basic@invalid-set-prop-any,pass
> +igt@kms_addfb_basic@invalid-set-prop,pass
> +igt@kms_addfb_basic@master-rmfb,pass
> +igt@kms_atomic@plane-overlay-legacy,skip
> +igt@kms_atomic@plane-primary-legacy,pass
> +igt@kms_atomic@plane-primary-overlay-mutable-zpos,skip
> +igt@kms_atomic@plane-immutable-zpos,pass
> +igt@kms_atomic@test-only,pass
> +igt@kms_atomic@plane-cursor-legacy,pass
> +igt@kms_atomic@plane-invalid-params,pass
> +igt@kms_atomic@crtc-invalid-params,pass
> +igt@kms_atomic@atomic-invalid-params,pass
> +igt@kms_atomic@atomic_plane_damage,pass
> +igt@kms_atomic_interruptible@legacy-setmode,skip
> +igt@kms_atomic_interruptible@atomic-setmode,skip
> +igt@kms_atomic_interruptible@legacy-dpms,skip
> +igt@kms_atomic_interruptible@legacy-pageflip,skip
> +igt@kms_atomic_interruptible@legacy-cursor,skip
> +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> +igt@kms_content_protection@lic,skip
> +igt@kms_flip_event_leak,pass
> +igt@kms_getfb@getfb-handle-zero,pass
> +igt@kms_getfb@getfb-handle-valid,pass
> +igt@kms_getfb@getfb-handle-closed,pass
> +igt@kms_getfb@getfb-handle-not-fb,pass
> +igt@kms_getfb@getfb-addfb-different-handles,pass
> +igt@kms_getfb@getfb-repeated-different-handles,pass
> +igt@kms_getfb@getfb2-handle-zero,pass
> +igt@kms_getfb@getfb2-handle-closed,pass
> +igt@kms_getfb@getfb2-handle-not-fb,pass
> +igt@kms_getfb@getfb2-into-addfb2,pass
> +igt@kms_getfb@getfb-handle-protection,pass
> +igt@kms_getfb@getfb2-handle-protection,pass
> +igt@kms_hdmi_inject@inject-4k,skip
> +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,skip
> +igt@kms_pipe_crc_basic@bad-source,pass
> +igt@kms_pipe_crc_basic@read-crc-pipe-a,pass
> +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,pass
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,pass
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,pass
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,pass
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,pass
> +igt@kms_pipe_crc_basic@read-crc-pipe-b,pass
> +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,pass
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,pass
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,pass
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,pass
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,pass
> +igt@kms_prop_blob@basic,pass
> +igt@kms_prop_blob@blob-prop-core,pass
> +igt@kms_prop_blob@blob-prop-validate,pass
> +igt@kms_prop_blob@blob-prop-lifetime,pass
> +igt@kms_prop_blob@blob-multiple,pass
> +igt@kms_prop_blob@invalid-get-prop-any,pass
> +igt@kms_prop_blob@invalid-get-prop,pass
> +igt@kms_prop_blob@invalid-set-prop-any,pass
> +igt@kms_prop_blob@invalid-set-prop,pass
> +igt@kms_rmfb@rmfb-ioctl,pass
> +igt@kms_rmfb@close-fd,fail
> +igt@kms_setmode@basic@pipe-a-edp-1,pass
> +igt@kms_setmode@basic@pipe-b-edp-1,pass
> +igt@msm_mapping@sqefw,dmesg-fail
> +igt@msm_mapping@shadow,dmesg-fail
> +igt@msm_recovery@hangcheck,dmesg-warn
> +igt@msm_recovery@gpu-fault,dmesg-warn
> +igt@msm_recovery@iova-fault,dmesg-warn
> +igt@msm_submit@empty-submit,pass
> +igt@msm_submit@invalid-queue-submit,pass
> +igt@msm_submit@invalid-flags-submit,pass
> +igt@msm_submit@invalid-in-fence-submit,pass
> +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> +igt@msm_submit@valid-submit,pass
> +igt@kms_sysfs_edid_timing,pass
> +igt@kms_universal_plane@universal-plane-pipe-a-functional,skip
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,pass
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,fail
> +igt@kms_vblank@invalid,pass
> +igt@kms_vblank@crtc-id,pass
> +igt@kms_vblank@pipe-a-query-idle,pass
> +igt@kms_vblank@pipe-a-query-forked,pass
> +igt@kms_vblank@pipe-a-query-busy,pass
> +igt@kms_vblank@pipe-a-query-forked-busy,pass
> +igt@kms_vblank@pipe-a-wait-idle,pass
> +igt@kms_vblank@pipe-a-wait-forked,pass
> +igt@kms_vblank@pipe-a-wait-busy,pass
> +igt@kms_vblank@pipe-a-wait-forked-busy,pass
> +igt@kms_vblank@pipe-a-ts-continuation-idle,pass
> +igt@kms_vblank@pipe-a-ts-continuation-modeset,pass
> +igt@kms_vblank@pipe-b-accuracy-idle,pass
> +igt@kms_vblank@pipe-b-query-idle,pass
> +igt@kms_vblank@pipe-b-query-forked,pass
> +igt@kms_vblank@pipe-b-query-busy,pass
> +igt@kms_vblank@pipe-b-query-forked-busy,pass
> +igt@kms_vblank@pipe-b-wait-idle,pass
> +igt@kms_vblank@pipe-b-wait-forked,pass
> +igt@kms_vblank@pipe-b-wait-busy,pass
> +igt@kms_vblank@pipe-b-wait-forked-busy,pass
> +igt@kms_vblank@pipe-b-ts-continuation-idle,pass
> +igt@kms_vblank@pipe-b-ts-continuation-modeset,pass
> diff --git a/drivers/gpu/drm/msm/ci/msm_sdm845_results.txt b/drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> new file mode 100644
> index 000000000000..3df1e5ef6d80
> --- /dev/null
> +++ b/drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> @@ -0,0 +1,141 @@
> +igt@core_auth@getclient-simple,dmesg-warn
> +igt@core_auth@getclient-master-drop,pass
> +igt@core_auth@basic-auth,pass
> +igt@core_auth@many-magics,pass
> +igt@core_getclient,pass
> +igt@core_getstats,pass
> +igt@core_getversion,pass
> +igt@core_setmaster_vs_auth,pass
> +igt@drm_read@invalid-buffer,pass
> +igt@drm_read@fault-buffer,pass
> +igt@drm_read@empty-block,pass
> +igt@drm_read@empty-nonblock,pass
> +igt@drm_read@short-buffer-block,pass
> +igt@drm_read@short-buffer-nonblock,pass
> +igt@drm_read@short-buffer-wakeup,pass
> +igt@kms_addfb_basic@unused-handle,pass
> +igt@kms_addfb_basic@unused-pitches,pass
> +igt@kms_addfb_basic@unused-offsets,pass
> +igt@kms_addfb_basic@unused-modifier,pass
> +igt@kms_addfb_basic@legacy-format,dmesg-warn
> +igt@kms_addfb_basic@no-handle,pass
> +igt@kms_addfb_basic@basic,pass
> +igt@kms_addfb_basic@bad-pitch-0,pass
> +igt@kms_addfb_basic@bad-pitch-32,pass
> +igt@kms_addfb_basic@bad-pitch-63,pass
> +igt@kms_addfb_basic@bad-pitch-128,pass
> +igt@kms_addfb_basic@bad-pitch-256,pass
> +igt@kms_addfb_basic@bad-pitch-1024,pass
> +igt@kms_addfb_basic@bad-pitch-999,pass
> +igt@kms_addfb_basic@bad-pitch-65536,pass
> +igt@kms_addfb_basic@size-max,pass
> +igt@kms_addfb_basic@too-wide,pass
> +igt@kms_addfb_basic@too-high,dmesg-warn
> +igt@kms_addfb_basic@bo-too-small,pass
> +igt@kms_addfb_basic@small-bo,pass
> +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> +igt@kms_addfb_basic@addfb25-bad-modifier,dmesg-warn
> +igt@kms_addfb_basic@invalid-get-prop-any,pass
> +igt@kms_addfb_basic@invalid-get-prop,pass
> +igt@kms_addfb_basic@invalid-set-prop-any,pass
> +igt@kms_addfb_basic@invalid-set-prop,pass
> +igt@kms_addfb_basic@master-rmfb,pass
> +igt@kms_atomic@plane-overlay-legacy,dmesg-warn
> +igt@kms_atomic@plane-primary-legacy,dmesg-warn
> +igt@kms_atomic@plane-primary-overlay-mutable-zpos,dmesg-warn
> +igt@kms_atomic@plane-immutable-zpos,dmesg-warn
> +igt@kms_atomic@test-only,dmesg-warn
> +igt@kms_atomic@plane-cursor-legacy,dmesg-warn
> +igt@kms_atomic@plane-invalid-params,dmesg-warn
> +igt@kms_atomic@crtc-invalid-params,dmesg-warn
> +igt@kms_atomic@atomic-invalid-params,dmesg-warn
> +igt@kms_atomic@atomic_plane_damage,dmesg-warn
> +igt@kms_atomic_interruptible@legacy-setmode,skip
> +igt@kms_atomic_interruptible@atomic-setmode,skip
> +igt@kms_atomic_interruptible@legacy-dpms,skip
> +igt@kms_atomic_interruptible@legacy-pageflip,skip
> +igt@kms_atomic_interruptible@legacy-cursor,skip
> +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> +igt@kms_content_protection@lic,skip
> +igt@kms_flip_event_leak,dmesg-warn
> +igt@kms_getfb@getfb-handle-zero,pass
> +igt@kms_getfb@getfb-handle-valid,pass
> +igt@kms_getfb@getfb-handle-closed,pass
> +igt@kms_getfb@getfb-handle-not-fb,pass
> +igt@kms_getfb@getfb-addfb-different-handles,pass
> +igt@kms_getfb@getfb-repeated-different-handles,pass
> +igt@kms_getfb@getfb2-handle-zero,pass
> +igt@kms_getfb@getfb2-handle-closed,pass
> +igt@kms_getfb@getfb2-handle-not-fb,pass
> +igt@kms_getfb@getfb2-into-addfb2,pass
> +igt@kms_getfb@getfb-handle-protection,pass
> +igt@kms_getfb@getfb2-handle-protection,pass
> +igt@kms_hdmi_inject@inject-4k,skip
> +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,pass
> +igt@kms_pipe_crc_basic@bad-source,pass
> +igt@kms_pipe_crc_basic@read-crc-pipe-a,pass
> +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,pass
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,pass
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,pass
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,dmesg-warn
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,dmesg-warn
> +igt@kms_pipe_crc_basic@read-crc-pipe-b,skip
> +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,skip
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,skip
> +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,skip
> +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,skip
> +igt@kms_prop_blob@basic,pass
> +igt@kms_prop_blob@blob-prop-core,pass
> +igt@kms_prop_blob@blob-prop-validate,pass
> +igt@kms_prop_blob@blob-prop-lifetime,pass
> +igt@kms_prop_blob@blob-multiple,pass
> +igt@kms_prop_blob@invalid-get-prop-any,pass
> +igt@kms_prop_blob@invalid-get-prop,pass
> +igt@kms_prop_blob@invalid-set-prop-any,pass
> +igt@kms_prop_blob@invalid-set-prop,pass
> +igt@kms_rmfb@rmfb-ioctl,pass
> +igt@kms_rmfb@close-fd,fail
> +igt@kms_setmode@basic@pipe-a-edp-1,dmesg-warn
> +igt@kms_setmode@basic,skip
> +igt@msm_mapping@sqefw,dmesg-fail
> +igt@msm_mapping@shadow,dmesg-fail
> +igt@msm_recovery@hangcheck,dmesg-warn
> +igt@msm_recovery@gpu-fault,dmesg-warn
> +igt@msm_recovery@iova-fault,dmesg-warn
> +igt@msm_submit@empty-submit,pass
> +igt@msm_submit@invalid-queue-submit,pass
> +igt@msm_submit@invalid-flags-submit,pass
> +igt@msm_submit@invalid-in-fence-submit,pass
> +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> +igt@msm_submit@valid-submit,pass
> +igt@kms_sysfs_edid_timing,pass
> +igt@kms_universal_plane@universal-plane-pipe-a-functional,dmesg-fail
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,dmesg-warn
> +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,skip
> +igt@kms_vblank@invalid,dmesg-warn
> +igt@kms_vblank@crtc-id,dmesg-warn
> +igt@kms_vblank@pipe-a-query-idle,dmesg-warn
> +igt@kms_vblank@pipe-a-query-forked,dmesg-warn
> +igt@kms_vblank@pipe-a-query-busy,dmesg-warn
> +igt@kms_vblank@pipe-a-query-forked-busy,dmesg-warn
> +igt@kms_vblank@pipe-a-wait-idle,dmesg-warn
> +igt@kms_vblank@pipe-a-wait-forked,dmesg-warn
> +igt@kms_vblank@pipe-a-wait-busy,dmesg-warn
> +igt@kms_vblank@pipe-a-wait-forked-busy,dmesg-warn
> +igt@kms_vblank@pipe-a-ts-continuation-idle,dmesg-warn
> +igt@kms_vblank@pipe-a-ts-continuation-modeset,dmesg-warn
> +igt@kms_vblank@pipe-b-accuracy-idle,skip
> +igt@kms_vblank@pipe-b-query-idle,skip
> +igt@kms_vblank@pipe-b-query-forked,skip
> +igt@kms_vblank@pipe-b-query-busy,skip
> +igt@kms_vblank@pipe-b-query-forked-busy,skip
> +igt@kms_vblank@pipe-b-wait-idle,skip
> +igt@kms_vblank@pipe-b-wait-forked,skip
> +igt@kms_vblank@pipe-b-wait-busy,skip
> +igt@kms_vblank@pipe-b-wait-forked-busy,skip
> +igt@kms_vblank@pipe-b-ts-continuation-idle,skip
> +igt@kms_vblank@pipe-b-ts-continuation-modeset,skip
> --
> 2.31.1
>

Greg KH May 11, 2022, 6:22 a.m. UTC | #5

On Wed, May 11, 2022 at 03:06:47PM +1000, Dave Airlie wrote:
> > And use it to store expectations about what the drm/msm driver is
> > supposed to pass in the IGT test suite.
> 
> I wanted to loop in Linus/Greg to see if there are any issues raised
> by adding CI results file to the tree in their minds, or if any other
> subsystem has done this already, and it's all fine.

Why does the results need to be added to the tree?  Shouldn't they be
either "all is good" or "constantly changing and a constant churn"?

> I think this is a good thing after our Mesa experience, but Mesa has a
> lot tighter integration here, so I want to get some more opinions
> outside the group.

For systems that have "tight integration" this might make sense as proof
that all is working for a specific commit, but I can't see how this will
help the kernel out much.

What are you going to do with these results being checked in all the
time?

thanks,

greg k-h

Michel Dänzer May 11, 2022, 10:26 a.m. UTC | #6

On 2022-05-11 08:22, Greg Kroah-Hartman wrote:
> On Wed, May 11, 2022 at 03:06:47PM +1000, Dave Airlie wrote:
>>> And use it to store expectations about what the drm/msm driver is
>>> supposed to pass in the IGT test suite.
>>
>> I wanted to loop in Linus/Greg to see if there are any issues raised
>> by adding CI results file to the tree in their minds, or if any other
>> subsystem has done this already, and it's all fine.
> 
> Why does the results need to be added to the tree?  Shouldn't they be
> either "all is good" or "constantly changing and a constant churn"?
> 
>> I think this is a good thing after our Mesa experience, but Mesa has a
>> lot tighter integration here, so I want to get some more opinions
>> outside the group.
> 
> For systems that have "tight integration" this might make sense as proof
> that all is working for a specific commit, but I can't see how this will
> help the kernel out much.
> 
> What are you going to do with these results being checked in all the
> time?

Having the expected results in the tree keeps them consistent with the driver code itself, and allows putting in place gating CI to prevent merging driver changes which make any of the tests deviate from the expected result.

Keeping them separate inevitably results in divergence between the driver code and the expected test results, which would result in spurious failures of such CI.

I expect the main complication for the kernel will be due to driver changes merged via different trees, e.g. for cross-subsystem reworks. Since those will not go through the same CI, they may accidentally introduce inconsistencies. The ideal solution for this IMO would be centralizing CI such that the same gating tests have to pass regardless of how the code is merged. But there's likely quite a long way to go until we get there. :)

Greg KH May 11, 2022, 11:50 a.m. UTC | #7

On Wed, May 11, 2022 at 12:26:05PM +0200, Michel Dänzer wrote:
> On 2022-05-11 08:22, Greg Kroah-Hartman wrote:
> > On Wed, May 11, 2022 at 03:06:47PM +1000, Dave Airlie wrote:
> >>> And use it to store expectations about what the drm/msm driver is
> >>> supposed to pass in the IGT test suite.
> >>
> >> I wanted to loop in Linus/Greg to see if there are any issues raised
> >> by adding CI results file to the tree in their minds, or if any other
> >> subsystem has done this already, and it's all fine.
> > 
> > Why does the results need to be added to the tree?  Shouldn't they be
> > either "all is good" or "constantly changing and a constant churn"?
> > 
> >> I think this is a good thing after our Mesa experience, but Mesa has a
> >> lot tighter integration here, so I want to get some more opinions
> >> outside the group.
> > 
> > For systems that have "tight integration" this might make sense as proof
> > that all is working for a specific commit, but I can't see how this will
> > help the kernel out much.
> > 
> > What are you going to do with these results being checked in all the
> > time?
> 
> Having the expected results in the tree keeps them consistent with the driver code itself, and allows putting in place gating CI to prevent merging driver changes which make any of the tests deviate from the expected result.

Shouldn't "expected result" always be "pass"?

If not, then the test should be changed to be "skipped" like we have
today in the kselftest tests.

And how about tieing this into the kselftest process as well, why would
this be somehow separate from the rest of the kernel tests?

> Keeping them separate inevitably results in divergence between the driver code and the expected test results, which would result in spurious failures of such CI.

Again, "pass" should be the expected results :)

> I expect the main complication for the kernel will be due to driver changes merged via different trees, e.g. for cross-subsystem reworks. Since those will not go through the same CI, they may accidentally introduce inconsistencies. The ideal solution for this IMO would be centralizing CI such that the same gating tests have to pass regardless of how the code is merged. But there's likely quite a long way to go until we get there. :)

We have in-kernel tests for the rest of the kernel, why can't you put
your testing stuff into there as well?

thanks,

greg k-h

Rob Clark May 11, 2022, 1:33 p.m. UTC | #8

On Wed, May 11, 2022 at 4:50 AM Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Wed, May 11, 2022 at 12:26:05PM +0200, Michel Dänzer wrote:
> > On 2022-05-11 08:22, Greg Kroah-Hartman wrote:
> > > On Wed, May 11, 2022 at 03:06:47PM +1000, Dave Airlie wrote:
> > >>> And use it to store expectations about what the drm/msm driver is
> > >>> supposed to pass in the IGT test suite.
> > >>
> > >> I wanted to loop in Linus/Greg to see if there are any issues raised
> > >> by adding CI results file to the tree in their minds, or if any other
> > >> subsystem has done this already, and it's all fine.
> > >
> > > Why does the results need to be added to the tree?  Shouldn't they be
> > > either "all is good" or "constantly changing and a constant churn"?
> > >
> > >> I think this is a good thing after our Mesa experience, but Mesa has a
> > >> lot tighter integration here, so I want to get some more opinions
> > >> outside the group.
> > >
> > > For systems that have "tight integration" this might make sense as proof
> > > that all is working for a specific commit, but I can't see how this will
> > > help the kernel out much.
> > >
> > > What are you going to do with these results being checked in all the
> > > time?
> >
> > Having the expected results in the tree keeps them consistent with the driver code itself, and allows putting in place gating CI to prevent merging driver changes which make any of the tests deviate from the expected result.
>
> Shouldn't "expected result" always be "pass"?
>
> If not, then the test should be changed to be "skipped" like we have
> today in the kselftest tests.

No, we want to run tests even if they are expected to fail.  This
prevents the scenario of a test getting fixed without being noticed
(for ex, developer was working on fixing test A and didn't notice that
the fix also fixed test B).  If a fix goes unnoticed, a later
regression would also go unnoticed ;-)

I was skeptical about this approach at first with mesa CI, but having
used mesa CI for a while, I am now a firm believer in the approach.

And ofc we want the expectations to be in the kernel tree because
there could be, for example, differences between -fixes and -next
branches.  (Or even stable kernel branches if/when we get to the point
of running CI on those.)

> And how about tieing this into the kselftest process as well, why would
> this be somehow separate from the rest of the kernel tests?
>
> > Keeping them separate inevitably results in divergence between the driver code and the expected test results, which would result in spurious failures of such CI.
>
> Again, "pass" should be the expected results :)
>
> > I expect the main complication for the kernel will be due to driver changes merged via different trees, e.g. for cross-subsystem reworks. Since those will not go through the same CI, they may accidentally introduce inconsistencies. The ideal solution for this IMO would be centralizing CI such that the same gating tests have to pass regardless of how the code is merged. But there's likely quite a long way to go until we get there. :)
>
> We have in-kernel tests for the rest of the kernel, why can't you put
> your testing stuff into there as well?

We could ofc put a lot more of the gitlab yml and scripts into the
kernel tree.  Probably all of i-g-t is a bit much to put in the kernel
tree.  Not to mention I'd like to see this expand to also run some
deqp and/or piglit tests, which is definitely too much to vendor into
the kernel tree.

The approach of this RFC was to put only what was absolutely required
in the kernel tree (such as expectations), and then link out to an
external drm-ci tree[1] which has all the necessary scripts and yml
for building and running tests, to avoid having to put a whole lot
more in the kernel tree. (We should be specifying exact commit-sha for
that tree, IMO, as it controls the version of i-g-t which gets used,
and we need to be able to update expectations in sync with an i-g-t
uprev, for example when new tests are added or if a test fix caused a
fail->pass transition.)

BR,
-R

[1] https://gitlab.freedesktop.org/gfx-ci/drm-ci

> thanks,
>
> greg k-h

Daniel Vetter May 11, 2022, 4:43 p.m. UTC | #9

On Wed, 11 May 2022 at 15:33, Rob Clark <robdclark@gmail.com> wrote:
> On Wed, May 11, 2022 at 4:50 AM Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Wed, May 11, 2022 at 12:26:05PM +0200, Michel Dänzer wrote:
> > > On 2022-05-11 08:22, Greg Kroah-Hartman wrote:
> > > > On Wed, May 11, 2022 at 03:06:47PM +1000, Dave Airlie wrote:
> > > >>> And use it to store expectations about what the drm/msm driver is
> > > >>> supposed to pass in the IGT test suite.
> > > >>
> > > >> I wanted to loop in Linus/Greg to see if there are any issues raised
> > > >> by adding CI results file to the tree in their minds, or if any other
> > > >> subsystem has done this already, and it's all fine.
> > > >
> > > > Why does the results need to be added to the tree?  Shouldn't they be
> > > > either "all is good" or "constantly changing and a constant churn"?
> > > >
> > > >> I think this is a good thing after our Mesa experience, but Mesa has a
> > > >> lot tighter integration here, so I want to get some more opinions
> > > >> outside the group.
> > > >
> > > > For systems that have "tight integration" this might make sense as proof
> > > > that all is working for a specific commit, but I can't see how this will
> > > > help the kernel out much.
> > > >
> > > > What are you going to do with these results being checked in all the
> > > > time?
> > >
> > > Having the expected results in the tree keeps them consistent with the driver code itself, and allows putting in place gating CI to prevent merging driver changes which make any of the tests deviate from the expected result.
> >
> > Shouldn't "expected result" always be "pass"?
> >
> > If not, then the test should be changed to be "skipped" like we have
> > today in the kselftest tests.
>
> No, we want to run tests even if they are expected to fail.  This
> prevents the scenario of a test getting fixed without being noticed
> (for ex, developer was working on fixing test A and didn't notice that
> the fix also fixed test B).  If a fix goes unnoticed, a later
> regression would also go unnoticed ;-)
>
> I was skeptical about this approach at first with mesa CI, but having
> used mesa CI for a while, I am now a firm believer in the approach.
>
> And ofc we want the expectations to be in the kernel tree because
> there could be, for example, differences between -fixes and -next
> branches.  (Or even stable kernel branches if/when we get to the point
> of running CI on those.)

Yeah result files in tree is kinda needed, even more so for the
kernel. A lot of the linux-next integration testing is only done after
patches have landed, and sometimes such breakage makes it to upstream
and then into the subsystem/driver tree. Annotating in the backmerge
what exactly broke and why helps a lot with tracking issues.

And expecting every subsystem to run every other subsystem's tests,
especially tests that run on hw, is just not going to scale. So there
will be all kinds of difference in test results.

> > And how about tieing this into the kselftest process as well, why would
> > this be somehow separate from the rest of the kernel tests?
> >
> > > Keeping them separate inevitably results in divergence between the driver code and the expected test results, which would result in spurious failures of such CI.
> >
> > Again, "pass" should be the expected results :)
> >
> > > I expect the main complication for the kernel will be due to driver changes merged via different trees, e.g. for cross-subsystem reworks. Since those will not go through the same CI, they may accidentally introduce inconsistencies. The ideal solution for this IMO would be centralizing CI such that the same gating tests have to pass regardless of how the code is merged. But there's likely quite a long way to go until we get there. :)
> >
> > We have in-kernel tests for the rest of the kernel, why can't you put
> > your testing stuff into there as well?
>
> We could ofc put a lot more of the gitlab yml and scripts into the
> kernel tree.  Probably all of i-g-t is a bit much to put in the kernel
> tree.  Not to mention I'd like to see this expand to also run some
> deqp and/or piglit tests, which is definitely too much to vendor into
> the kernel tree.
>
> The approach of this RFC was to put only what was absolutely required
> in the kernel tree (such as expectations), and then link out to an
> external drm-ci tree[1] which has all the necessary scripts and yml
> for building and running tests, to avoid having to put a whole lot
> more in the kernel tree. (We should be specifying exact commit-sha for
> that tree, IMO, as it controls the version of i-g-t which gets used,
> and we need to be able to update expectations in sync with an i-g-t
> uprev, for example when new tests are added or if a test fix caused a
> fail->pass transition.)

Yeah I think longer-term we should carry a lot more in upstream, at
least anything that's shared across drivers wrt the ci integration (or
build testing and running tests which are hw agnostic). Maybe even
igt, not sure (otoh xfs-tests isn't moving into the kernel either, and
there's lots more like that).

Personally I think long-term the only thing outside should be other
repos with tests or stuff you need to run them, and not really the
glue to make it all work in ci. But that's maybe a bit too much
wishful thinking if CI systems stay largely subsystem specific (which
they currently are in many ways, with some overlap).

But maybe there is enough random pieces to share here for a lot more
in-tree to make sense, and imo the fewer extra steps and indirection
CI testing and test updating has, the better.

But like Rob says, eventually there's a limit and when you put the
entire GL/vulkan stack + it's conformance testsuite (which is
maintained by khronos somewhere completely different than both
kernel.org and freedesktop.org) then it's definitely too much and wont
work. And eventually we do want to run these things too (e.g.
intel-gfx-ci does run mesa + piglit on every run).
-Daniel

Daniel Vetter May 11, 2022, 5:12 p.m. UTC | #10

On Tue, 10 May 2022 at 22:26, Rob Clark <robdclark@gmail.com> wrote:
>
> On Tue, May 10, 2022 at 12:39 PM Jessica Zhang
> <quic_jesszhan@quicinc.com> wrote:
> >
> >
> >
> > On 5/10/2022 7:13 AM, Tomeu Vizoso wrote:
> > > And use it to store expectations about what the drm/msm driver is
> > > supposed to pass in the IGT test suite.
> > >
> > > Also include a configuration file that points to the out-of-tree CI
> > > scripts.
> > >
> > > By storing the test expectations along the code we can make sure both
> > > stay in sync with each other, and so we can know when a code change
> > > breaks those expectations.
> > >
> > > This will allow all contributors to drm/msm to reuse the infrastructure
> > > already in gitlab.freedesktop.org to test the driver on several
> > > generations of the hardware.
> > >
> > > v2:
> > >    - Fix names of result expectation files to match SoC
> > >    - Don't execute tests that are going to skip on all boards
> > >
> > > Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> > > ---
> > >   Documentation/gpu/msm_automated_testing.rst   |  70 +++++++++
> > >   drivers/gpu/drm/msm/ci/gitlab-ci.yml          |  11 ++
> > >   drivers/gpu/drm/msm/ci/msm.testlist           | 148 ++++++++++++++++++
> > >   .../gpu/drm/msm/ci/msm_apq8016_results.txt    | 140 +++++++++++++++++
> > >   .../gpu/drm/msm/ci/msm_apq8096_results.txt    | 140 +++++++++++++++++
> > >   drivers/gpu/drm/msm/ci/msm_sc7180_results.txt | 141 +++++++++++++++++
> > >   drivers/gpu/drm/msm/ci/msm_sdm845_results.txt | 141 +++++++++++++++++
> > >   7 files changed, 791 insertions(+)
> > >   create mode 100644 Documentation/gpu/msm_automated_testing.rst
> > >   create mode 100644 drivers/gpu/drm/msm/ci/gitlab-ci.yml
> > >   create mode 100644 drivers/gpu/drm/msm/ci/msm.testlist
> > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> > >
> > > diff --git a/Documentation/gpu/msm_automated_testing.rst b/Documentation/gpu/msm_automated_testing.rst
> > > new file mode 100644
> > > index 000000000000..fe59474569c2
> > > --- /dev/null
> > > +++ b/Documentation/gpu/msm_automated_testing.rst
> > > @@ -0,0 +1,70 @@
> > > +.. SPDX-License-Identifier: GPL-2.0+
> > > +
> > > +=========================================
> > > +Automated testing of MSM DRM driver
> > > +=========================================
> >
> > Hey Tomeu,
> >
> > I like the overall idea of having this in as part of the repo, but just
> > have a few questions on some details.
> >
> > > +
> > > +
> > > +Introduction
> > > +============
> > > +
> > > +Making sure that changes to a driver don't introduce regressions can be very time consuming when lots of different hardware configurations need to be tested. Moreover, it isn't practical for each person interested in this testing to have to acquire and maintain what can be a considerable amount of hardware.
> > > +
> > > +Also, it is desirable for developers to check for regressions in their code by themselves, instead of relying on the maintainer finding them and then reporting back.
> > > +
> > > +There are facilities in gitlab.freedesktop.org to automatically test Mesa that can be used as well for testing DRM drivers such as drm/msm. This document explains how people interested in testing the drm/msm driver can use this shared infrastructure to save quite some time and effort.
> > > +
> > > +
> > > +Relevant files
> > > +==============
> > > +
> > > +drivers/gpu/drm/msm/ci/gitlab-ci.yml
> > > +------------------------------------
> > > +
> > > +Specifies the specific version of the scripts to be used. GitLab CI will use the values defined in this file to fetch the right scripts.
> > > +
> > > +
> > > +drivers/gpu/drm/msm/ci/msm.testlist
> > > +-----------------------------------
> > > +
> > > +Specifies the tests that the current code is expected to be able to reliably run. These tests are expected to not hang the DUT (device under testing) when running on the revision they belong to, and to give consistent results.
> > > +
> > > +
> > > +drivers/gpu/drm/msm/ci/msm_*_results.txt
> > > +----------------------------------------
> > > +
> > > +Specifies the expected results of running this specific kernel revision on a given hardware configuration.
> > > +
> > > +
> > > +How to enable automated testing on your tree
> > > +============================================
> > > +
> > > +1. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one yet
> > > +
> > > +2. In your kernel repo's configuration (eg. https://gitlab.freedesktop.org/tomeu/linux/-/settings/ci_cd), change the CI/CD configuration file from .gitlab-ci.yml to drivers/gpu/drm/msm/ci/gitlab-ci.yml.
> > > +
> > > +3. Next time you push to this repository, you will see a CI pipeline being created (eg. https://gitlab.freedesktop.org/tomeu/linux/-/pipelines)
> > > +
> > > +4. The various jobs will be run and when the pipeline is finished, all jobs should be green unless a regression has been found.
> > > +
> > > +
> > > +How to update test expectations
> > > +===============================
> > > +
> > > +If your changes to the code fix any tests, you will have to update one or more of the files in drivers/gpu/drm/msm/ci/msm_*_results.txt, for each of the test platforms affected by the change.
> > > +
> > > +If you have run a pipeline and it failed because of a mismatched test result, you will have been offered a patch at the end of the run that you can apply after reviewing it.
> > > +
> > > +
> > > +How to expand coverage
> > > +======================
> > > +
> > > +If your code changes makes it possible to run more tests (by solving reliability issues, for example), more tests can be added to this list, and then the expected results updated with the patch file that will be printed at the end of each job.
> > > +
> > > +If there is a need for updating the version of IGT being used (maybe you have added more tests to it), follow the instructions at https://gitlab.freedesktop.org/gfx-ci/drm-ci/-/tree/msm#upreving-igt.
> > > +
> > > +
> > > +How to test your changes to the scripts
> > > +==========================================
> > > +
> > > +For testing changes to the scripts in the drm-ci repo, change the DRM_CI_PROJECT_PATH variable in drivers/gpu/drm/msm/ci/gitlab-ci.yml to match your fork of the project (eg. tomeu/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/.
> > > \ No newline at end of file
> > > diff --git a/drivers/gpu/drm/msm/ci/gitlab-ci.yml b/drivers/gpu/drm/msm/ci/gitlab-ci.yml
> > > new file mode 100644
> > > index 000000000000..9b7caa7fcab2
> > > --- /dev/null
> > > +++ b/drivers/gpu/drm/msm/ci/gitlab-ci.yml
> > > @@ -0,0 +1,11 @@
> > > +variables:
> > > +  # Change this to use your fork of drm-ci
> > > +  DRM_CI_PROJECT_PATH: &drm-ci-project-path gfx-ci/drm-ci
> > > +  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha msm
> > > +
> > > +include:
> > > +  - project: *drm-ci-project-path
> > > +    ref: *drm-ci-commit-sha
> > > +    file:
> > > +      - '.msm-gitlab-ci.yml'
> > > +      - '.gitlab-ci.yml'
> > > diff --git a/drivers/gpu/drm/msm/ci/msm.testlist b/drivers/gpu/drm/msm/ci/msm.testlist
> > > new file mode 100644
> > > index 000000000000..8805a3bc7316
> > > --- /dev/null
> > > +++ b/drivers/gpu/drm/msm/ci/msm.testlist
> > > @@ -0,0 +1,148 @@
> > > +# Keep alphabetically sorted by default
> > > +
> > > +igt@core_auth@getclient-simple
> > > +igt@core_auth@getclient-master-drop
> > > +igt@core_auth@basic-auth
> > > +igt@core_auth@many-magics
> > > +igt@core_getclient
> > > +igt@core_getstats
> > > +igt@core_getversion
> > > +igt@core_setmaster_vs_auth
> > > +igt@drm_read@invalid-buffer
> > > +igt@drm_read@fault-buffer
> > > +igt@drm_read@empty-block
> > > +igt@drm_read@empty-nonblock
> > > +igt@drm_read@short-buffer-block
> > > +igt@drm_read@short-buffer-nonblock
> > > +igt@drm_read@short-buffer-wakeup
> > > +igt@kms_addfb_basic@unused-handle
> > > +igt@kms_addfb_basic@unused-pitches
> > > +igt@kms_addfb_basic@unused-offsets
> > > +igt@kms_addfb_basic@unused-modifier
> > > +igt@kms_addfb_basic@legacy-format
> > > +igt@kms_addfb_basic@no-handle
> > > +igt@kms_addfb_basic@basic
> > > +igt@kms_addfb_basic@bad-pitch-0
> > > +igt@kms_addfb_basic@bad-pitch-32
> > > +igt@kms_addfb_basic@bad-pitch-63
> > > +igt@kms_addfb_basic@bad-pitch-128
> > > +igt@kms_addfb_basic@bad-pitch-256
> > > +igt@kms_addfb_basic@bad-pitch-1024
> > > +igt@kms_addfb_basic@bad-pitch-999
> > > +igt@kms_addfb_basic@bad-pitch-65536
> > > +igt@kms_addfb_basic@size-max
> > > +igt@kms_addfb_basic@too-wide
> > > +igt@kms_addfb_basic@too-high
> > > +igt@kms_addfb_basic@bo-too-small
> > > +igt@kms_addfb_basic@small-bo
> > > +igt@kms_addfb_basic@addfb25-modifier-no-flag
> > > +igt@kms_addfb_basic@addfb25-bad-modifier
> > > +igt@kms_addfb_basic@invalid-get-prop-any
> > > +igt@kms_addfb_basic@invalid-get-prop
> > > +igt@kms_addfb_basic@invalid-set-prop-any
> > > +igt@kms_addfb_basic@invalid-set-prop
> > > +igt@kms_addfb_basic@master-rmfb
> > > +igt@kms_atomic@plane-overlay-legacy
> > > +igt@kms_atomic@plane-primary-legacy
> > > +igt@kms_atomic@plane-primary-overlay-mutable-zpos
> > > +igt@kms_atomic@plane-immutable-zpos
> > > +igt@kms_atomic@test-only
> > > +igt@kms_atomic@plane-cursor-legacy
> > > +igt@kms_atomic@plane-invalid-params
> > > +igt@kms_atomic@crtc-invalid-params
> > > +igt@kms_atomic@atomic-invalid-params
> > > +igt@kms_atomic@atomic_plane_damage
> > > +igt@kms_atomic_interruptible@legacy-setmode@pipe-a-edp-1
> > > +igt@kms_atomic_interruptible@atomic-setmode@pipe-a-edp-1
> > > +igt@kms_atomic_interruptible@legacy-dpms@pipe-a-edp-1
> > > +igt@kms_atomic_interruptible@legacy-pageflip@pipe-a-edp-1
> > > +igt@kms_atomic_interruptible@legacy-cursor@pipe-a-edp-1
> > > +igt@kms_atomic_interruptible@universal-setplane-primary@pipe-a-edp-1
> > > +igt@kms_atomic_interruptible@universal-setplane-cursor@pipe-a-edp-1
> >
> > I see that there are some KMS subtests/tests are missing such as
> > kms_atomic_transition -- are they unstable across all boards?
> >
> >  From what I've seen, kms_atomic_transition is unstable only on Trogdor,
> > but has stable results on other boards.
> >
> > > +igt@kms_content_protection@LIC > +igt@kms_flip_event_leak
> > > +igt@kms_getfb@getfb-handle-zero
> > > +igt@kms_getfb@getfb-handle-valid
> > > +igt@kms_getfb@getfb-handle-closed
> > > +igt@kms_getfb@getfb-handle-not-fb
> > > +igt@kms_getfb@getfb-addfb-different-handles
> > > +igt@kms_getfb@getfb-repeated-different-handles
> > > +igt@kms_getfb@getfb2-handle-zero
> > > +igt@kms_getfb@getfb2-handle-closed
> > > +igt@kms_getfb@getfb2-handle-not-fb
> > > +igt@kms_getfb@getfb2-into-addfb2
> > > +igt@kms_getfb@getfb-handle-protection
> > > +igt@kms_getfb@getfb2-handle-protection
> > > +igt@kms_hdmi_inject@inject-4k > +igt@kms_multipipe_modeset@basic-max-pipe-crc-check
> > > +igt@kms_pipe_crc_basic@bad-source
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-a
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence
> > > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a
> > > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-b
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence
> > > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b
> > > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b
> > > +igt@kms_prop_blob@basic
> > > +igt@kms_prop_blob@blob-prop-core
> > > +igt@kms_prop_blob@blob-prop-validate
> > > +igt@kms_prop_blob@blob-prop-lifetime
> > > +igt@kms_prop_blob@blob-multiple
> > > +igt@kms_prop_blob@invalid-get-prop-any
> > > +igt@kms_prop_blob@invalid-get-prop
> > > +igt@kms_prop_blob@invalid-set-prop-any
> > > +igt@kms_prop_blob@invalid-set-prop
> > > +igt@kms_rmfb@rmfb-ioctl
> > > +igt@kms_rmfb@close-fd
> > > +igt@kms_setmode@basic@pipe-a-edp-1
> > > +igt@kms_setmode@basic@pipe-b-edp-1
> > > +
> > > +# Flakes on a530 with:
> > > +# arm-smmu b40000.iommu: Unhandled context fault: fsr=0x408, iova=0x01003000, fsynr=0x13, cbfrsynra=0x0, cb=0
> > > +# igt@msm_mapping@ring
> > > +
> > > +igt@msm_mapping@sqefw
> > > +igt@msm_mapping@shadow
> > > +igt@msm_recovery@hangcheck
> > > +igt@msm_recovery@gpu-fault
> > > +igt@msm_recovery@iova-fault
> > > +igt@msm_submit@empty-submit
> > > +igt@msm_submit@invalid-queue-submit
> > > +igt@msm_submit@invalid-flags-submit
> > > +igt@msm_submit@invalid-in-fence-submit
> > > +igt@msm_submit@invalid-duplicate-bo-submit
> > > +igt@msm_submit@invalid-cmd-idx-submit
> > > +igt@msm_submit@invalid-cmd-type-submit
> > > +igt@msm_submit@valid-submit
> > > +igt@kms_sysfs_edid_timing
> > > +igt@kms_universal_plane@universal-plane-pipe-a-functional
> > > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a
> > > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b
> > > +igt@kms_vblank@invalid
> > > +igt@kms_vblank@crtc-id
> > > +igt@kms_vblank@pipe-a-query-idle
> > > +igt@kms_vblank@pipe-a-query-forked
> > > +igt@kms_vblank@pipe-a-query-busy
> > > +igt@kms_vblank@pipe-a-query-forked-busy
> > > +igt@kms_vblank@pipe-a-wait-idle
> > > +igt@kms_vblank@pipe-a-wait-forked
> > > +igt@kms_vblank@pipe-a-wait-busy
> > > +igt@kms_vblank@pipe-a-wait-forked-busy
> > > +igt@kms_vblank@pipe-a-ts-continuation-idle
> > > +igt@kms_vblank@pipe-a-ts-continuation-modeset
> > > +igt@kms_vblank@pipe-b-accuracy-idle
> > > +igt@kms_vblank@pipe-b-query-idle
> > > +igt@kms_vblank@pipe-b-query-forked
> > > +igt@kms_vblank@pipe-b-query-busy
> > > +igt@kms_vblank@pipe-b-query-forked-busy
> > > +igt@kms_vblank@pipe-b-wait-idle
> > > +igt@kms_vblank@pipe-b-wait-forked
> > > +igt@kms_vblank@pipe-b-wait-busy
> > > +igt@kms_vblank@pipe-b-wait-forked-busy
> > > +igt@kms_vblank@pipe-b-ts-continuation-idle
> > > +igt@kms_vblank@pipe-b-ts-continuation-modeset
> > > diff --git a/drivers/gpu/drm/msm/ci/msm_apq8016_results.txt b/drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> > > new file mode 100644
> > > index 000000000000..2cad61d89faf
> > > --- /dev/null
> > > +++ b/drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> > > @@ -0,0 +1,140 @@
> > > +igt@core_auth@getclient-simple,pass
> > > +igt@core_auth@getclient-master-drop,pass
> > > +igt@core_auth@basic-auth,pass
> > > +igt@core_auth@many-magics,pass
> > > +igt@core_getclient,pass
> > > +igt@core_getstats,pass
> > > +igt@core_getversion,pass
> > > +igt@core_setmaster_vs_auth,pass
> > > +igt@drm_read@invalid-buffer,skip
> > > +igt@drm_read@fault-buffer,skip
> > > +igt@drm_read@empty-block,skip
> > > +igt@drm_read@empty-nonblock,skip
> > > +igt@drm_read@short-buffer-block,skip
> > > +igt@drm_read@short-buffer-nonblock,skip
> > > +igt@drm_read@short-buffer-wakeup,skip
> > > +igt@kms_addfb_basic@unused-handle,pass
> > > +igt@kms_addfb_basic@unused-pitches,pass
> > > +igt@kms_addfb_basic@unused-offsets,pass
> > > +igt@kms_addfb_basic@unused-modifier,pass
> > > +igt@kms_addfb_basic@legacy-format,dmesg-warn
> > > +igt@kms_addfb_basic@no-handle,pass
> > > +igt@kms_addfb_basic@basic,pass
> > > +igt@kms_addfb_basic@bad-pitch-0,pass
> > > +igt@kms_addfb_basic@bad-pitch-32,pass
> > > +igt@kms_addfb_basic@bad-pitch-63,pass
> > > +igt@kms_addfb_basic@bad-pitch-128,pass
> > > +igt@kms_addfb_basic@bad-pitch-256,pass
> > > +igt@kms_addfb_basic@bad-pitch-1024,pass
> > > +igt@kms_addfb_basic@bad-pitch-999,pass
> > > +igt@kms_addfb_basic@bad-pitch-65536,pass
> > > +igt@kms_addfb_basic@size-max,pass
> > > +igt@kms_addfb_basic@too-wide,pass
> > > +igt@kms_addfb_basic@too-high,dmesg-warn
> > > +igt@kms_addfb_basic@bo-too-small,pass
> > > +igt@kms_addfb_basic@small-bo,pass
> > > +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> > > +igt@kms_addfb_basic@addfb25-bad-modifier,fail
> > > +igt@kms_addfb_basic@invalid-get-prop-any,pass
> > > +igt@kms_addfb_basic@invalid-get-prop,pass
> > > +igt@kms_addfb_basic@invalid-set-prop-any,pass
> > > +igt@kms_addfb_basic@invalid-set-prop,pass
> > > +igt@kms_addfb_basic@master-rmfb,pass
> > > +igt@kms_atomic@plane-overlay-legacy,skip
> > > +igt@kms_atomic@plane-primary-legacy,skip
> > > +igt@kms_atomic@plane-primary-overlay-mutable-zpos,skip
> > > +igt@kms_atomic@plane-immutable-zpos,skip
> > > +igt@kms_atomic@test-only,skip
> > > +igt@kms_atomic@plane-cursor-legacy,skip
> > > +igt@kms_atomic@plane-invalid-params,skip
> > > +igt@kms_atomic@crtc-invalid-params,skip
> > > +igt@kms_atomic@atomic-invalid-params,skip
> > > +igt@kms_atomic@atomic_plane_damage,skip
> > > +igt@kms_atomic_interruptible@legacy-setmode,skip
> > > +igt@kms_atomic_interruptible@atomic-setmode,skip
> > > +igt@kms_atomic_interruptible@legacy-dpms,skip
> > > +igt@kms_atomic_interruptible@legacy-pageflip,skip
> > > +igt@kms_atomic_interruptible@legacy-cursor,skip
> > > +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> > > +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> > > +igt@kms_content_protection@lic,skip
> > > +igt@kms_flip_event_leak,skip
> > > +igt@kms_getfb@getfb-handle-zero,pass
> > > +igt@kms_getfb@getfb-handle-valid,pass
> > > +igt@kms_getfb@getfb-handle-closed,pass
> > > +igt@kms_getfb@getfb-handle-not-fb,pass
> > > +igt@kms_getfb@getfb-addfb-different-handles,pass
> > > +igt@kms_getfb@getfb-repeated-different-handles,pass
> > > +igt@kms_getfb@getfb2-handle-zero,pass
> > > +igt@kms_getfb@getfb2-handle-closed,pass
> > > +igt@kms_getfb@getfb2-handle-not-fb,pass
> > > +igt@kms_getfb@getfb2-into-addfb2,pass
> > > +igt@kms_getfb@getfb-handle-protection,pass
> > > +igt@kms_getfb@getfb2-handle-protection,pass
> > > +igt@kms_hdmi_inject@inject-4k,fail
> > > +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,skip
> > > +igt@kms_pipe_crc_basic@bad-source,skip
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-a,skip
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,skip
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,skip
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,skip
> > > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,skip
> > > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,skip
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-b,skip
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,skip
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,skip
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,skip
> > > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,skip
> > > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,skip
> > > +igt@kms_prop_blob@basic,pass
> > > +igt@kms_prop_blob@blob-prop-core,pass
> > > +igt@kms_prop_blob@blob-prop-validate,pass
> > > +igt@kms_prop_blob@blob-prop-lifetime,pass
> > > +igt@kms_prop_blob@blob-multiple,pass
> > > +igt@kms_prop_blob@invalid-get-prop-any,pass
> > > +igt@kms_prop_blob@invalid-get-prop,pass
> > > +igt@kms_prop_blob@invalid-set-prop-any,pass
> > > +igt@kms_prop_blob@invalid-set-prop,pass
> > > +igt@kms_rmfb@rmfb-ioctl,skip
> > > +igt@kms_rmfb@close-fd,skip
> > > +igt@kms_setmode@basic,skip
> > > +igt@msm_mapping@sqefw,skip
> > > +igt@msm_mapping@shadow,skip
> > > +igt@msm_recovery@hangcheck,skip
> > > +igt@msm_recovery@gpu-fault,skip
> > > +igt@msm_recovery@iova-fault,skip
> > > +igt@msm_submit@empty-submit,pass
> > > +igt@msm_submit@invalid-queue-submit,pass
> > > +igt@msm_submit@invalid-flags-submit,pass
> > > +igt@msm_submit@invalid-in-fence-submit,pass
> > > +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> > > +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> > > +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> > > +igt@msm_submit@valid-submit,pass
> > > +igt@kms_sysfs_edid_timing,pass
> > > +igt@kms_universal_plane@universal-plane-pipe-a-functional,skip
> > > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,skip
> > > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,skip
> > > +igt@kms_vblank@invalid,skip
> > > +igt@kms_vblank@crtc-id,skip
> > > +igt@kms_vblank@pipe-a-query-idle,skip
> > > +igt@kms_vblank@pipe-a-query-forked,skip
> > > +igt@kms_vblank@pipe-a-query-busy,skip
> > > +igt@kms_vblank@pipe-a-query-forked-busy,skip
> > > +igt@kms_vblank@pipe-a-wait-idle,skip
> > > +igt@kms_vblank@pipe-a-wait-forked,skip
> > > +igt@kms_vblank@pipe-a-wait-busy,skip
> > > +igt@kms_vblank@pipe-a-wait-forked-busy,skip
> > > +igt@kms_vblank@pipe-a-ts-continuation-idle,skip
> > > +igt@kms_vblank@pipe-a-ts-continuation-modeset,skip
> > > +igt@kms_vblank@pipe-b-accuracy-idle,skip
> > > +igt@kms_vblank@pipe-b-query-idle,skip
> > > +igt@kms_vblank@pipe-b-query-forked,skip
> > > +igt@kms_vblank@pipe-b-query-busy,skip
> > > +igt@kms_vblank@pipe-b-query-forked-busy,skip
> > > +igt@kms_vblank@pipe-b-wait-idle,skip
> > > +igt@kms_vblank@pipe-b-wait-forked,skip
> > > +igt@kms_vblank@pipe-b-wait-busy,skip
> > > +igt@kms_vblank@pipe-b-wait-forked-busy,skip
> > > +igt@kms_vblank@pipe-b-ts-continuation-idle,skip
> > > +igt@kms_vblank@pipe-b-ts-continuation-modeset,skip
> > > diff --git a/drivers/gpu/drm/msm/ci/msm_apq8096_results.txt b/drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> > > new file mode 100644
> > > index 000000000000..6f10941e5626
> > > --- /dev/null
> > > +++ b/drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> > > @@ -0,0 +1,140 @@
> > > +igt@core_auth@getclient-simple,pass
> > > +igt@core_auth@getclient-master-drop,pass
> > > +igt@core_auth@basic-auth,pass
> > > +igt@core_auth@many-magics,pass
> > > +igt@core_getclient,pass
> > > +igt@core_getstats,pass
> > > +igt@core_getversion,pass
> > > +igt@core_setmaster_vs_auth,pass
> > > +igt@drm_read@invalid-buffer,skip
> > > +igt@drm_read@fault-buffer,skip
> > > +igt@drm_read@empty-block,skip
> > > +igt@drm_read@empty-nonblock,skip
> > > +igt@drm_read@short-buffer-block,skip
> > > +igt@drm_read@short-buffer-nonblock,skip
> > > +igt@drm_read@short-buffer-wakeup,skip
> > > +igt@kms_addfb_basic@unused-handle,pass
> > > +igt@kms_addfb_basic@unused-pitches,pass
> > > +igt@kms_addfb_basic@unused-offsets,pass
> > > +igt@kms_addfb_basic@unused-modifier,pass
> > > +igt@kms_addfb_basic@legacy-format,dmesg-warn
> > > +igt@kms_addfb_basic@no-handle,pass
> > > +igt@kms_addfb_basic@basic,pass
> > > +igt@kms_addfb_basic@bad-pitch-0,pass
> > > +igt@kms_addfb_basic@bad-pitch-32,pass
> > > +igt@kms_addfb_basic@bad-pitch-63,pass
> > > +igt@kms_addfb_basic@bad-pitch-128,pass
> > > +igt@kms_addfb_basic@bad-pitch-256,pass
> > > +igt@kms_addfb_basic@bad-pitch-1024,pass
> > > +igt@kms_addfb_basic@bad-pitch-999,pass
> > > +igt@kms_addfb_basic@bad-pitch-65536,pass
> > > +igt@kms_addfb_basic@size-max,pass
> > > +igt@kms_addfb_basic@too-wide,pass
> > > +igt@kms_addfb_basic@too-high,dmesg-warn
> > > +igt@kms_addfb_basic@bo-too-small,pass
> > > +igt@kms_addfb_basic@small-bo,pass
> > > +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> > > +igt@kms_addfb_basic@addfb25-bad-modifier,fail
> > > +igt@kms_addfb_basic@invalid-get-prop-any,pass
> > > +igt@kms_addfb_basic@invalid-get-prop,pass
> > > +igt@kms_addfb_basic@invalid-set-prop-any,pass
> > > +igt@kms_addfb_basic@invalid-set-prop,pass
> > > +igt@kms_addfb_basic@master-rmfb,pass
> > > +igt@kms_atomic@plane-overlay-legacy,skip
> > > +igt@kms_atomic@plane-primary-legacy,skip
> > > +igt@kms_atomic@plane-primary-overlay-mutable-zpos,skip
> > > +igt@kms_atomic@plane-immutable-zpos,skip
> > > +igt@kms_atomic@test-only,skip
> > > +igt@kms_atomic@plane-cursor-legacy,skip
> > > +igt@kms_atomic@plane-invalid-params,skip
> > > +igt@kms_atomic@crtc-invalid-params,skip
> > > +igt@kms_atomic@atomic-invalid-params,skip
> > > +igt@kms_atomic@atomic_plane_damage,skip
> > > +igt@kms_atomic_interruptible@legacy-setmode,skip
> > > +igt@kms_atomic_interruptible@atomic-setmode,skip
> > > +igt@kms_atomic_interruptible@legacy-dpms,skip
> > > +igt@kms_atomic_interruptible@legacy-pageflip,skip
> > > +igt@kms_atomic_interruptible@legacy-cursor,skip
> > > +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> > > +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> > > +igt@kms_content_protection@lic,skip
> > > +igt@kms_flip_event_leak,skip
> > > +igt@kms_getfb@getfb-handle-zero,pass
> > > +igt@kms_getfb@getfb-handle-valid,pass
> > > +igt@kms_getfb@getfb-handle-closed,pass
> > > +igt@kms_getfb@getfb-handle-not-fb,pass
> > > +igt@kms_getfb@getfb-addfb-different-handles,pass
> > > +igt@kms_getfb@getfb-repeated-different-handles,pass
> > > +igt@kms_getfb@getfb2-handle-zero,pass
> > > +igt@kms_getfb@getfb2-handle-closed,pass
> > > +igt@kms_getfb@getfb2-handle-not-fb,pass
> > > +igt@kms_getfb@getfb2-into-addfb2,pass
> > > +igt@kms_getfb@getfb-handle-protection,pass
> > > +igt@kms_getfb@getfb2-handle-protection,pass
> > > +igt@kms_hdmi_inject@inject-4k,dmesg-warn
> > > +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,skip
> > > +igt@kms_pipe_crc_basic@bad-source,skip
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-a,skip
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,skip
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,skip
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,skip
> > > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,skip
> > > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,skip
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-b,skip
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,skip
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,skip
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,skip
> > > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,skip
> > > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,skip
> > > +igt@kms_prop_blob@basic,pass
> > > +igt@kms_prop_blob@blob-prop-core,pass
> > > +igt@kms_prop_blob@blob-prop-validate,pass
> > > +igt@kms_prop_blob@blob-prop-lifetime,pass
> > > +igt@kms_prop_blob@blob-multiple,pass
> > > +igt@kms_prop_blob@invalid-get-prop-any,pass
> > > +igt@kms_prop_blob@invalid-get-prop,pass
> > > +igt@kms_prop_blob@invalid-set-prop-any,pass
> > > +igt@kms_prop_blob@invalid-set-prop,pass
> > > +igt@kms_rmfb@rmfb-ioctl,skip
> > > +igt@kms_rmfb@close-fd,skip
> > > +igt@kms_setmode@basic,skip
> > > +igt@msm_mapping@sqefw,skip
> > > +igt@msm_mapping@shadow,skip
> > > +igt@msm_recovery@hangcheck,skip
> > > +igt@msm_recovery@gpu-fault,skip
> > > +igt@msm_recovery@iova-fault,skip
> > > +igt@msm_submit@empty-submit,pass
> > > +igt@msm_submit@invalid-queue-submit,pass
> > > +igt@msm_submit@invalid-flags-submit,pass
> > > +igt@msm_submit@invalid-in-fence-submit,pass
> > > +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> > > +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> > > +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> > > +igt@msm_submit@valid-submit,pass
> > > +igt@kms_sysfs_edid_timing,pass
> > > +igt@kms_universal_plane@universal-plane-pipe-a-functional,skip
> > > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,skip
> > > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,skip
> > > +igt@kms_vblank@invalid,skip
> > > +igt@kms_vblank@crtc-id,skip
> > > +igt@kms_vblank@pipe-a-query-idle,skip
> > > +igt@kms_vblank@pipe-a-query-forked,skip
> > > +igt@kms_vblank@pipe-a-query-busy,skip
> > > +igt@kms_vblank@pipe-a-query-forked-busy,skip
> > > +igt@kms_vblank@pipe-a-wait-idle,skip
> > > +igt@kms_vblank@pipe-a-wait-forked,skip
> > > +igt@kms_vblank@pipe-a-wait-busy,skip
> > > +igt@kms_vblank@pipe-a-wait-forked-busy,skip
> > > +igt@kms_vblank@pipe-a-ts-continuation-idle,skip
> > > +igt@kms_vblank@pipe-a-ts-continuation-modeset,skip
> > > +igt@kms_vblank@pipe-b-accuracy-idle,skip
> > > +igt@kms_vblank@pipe-b-query-idle,skip
> > > +igt@kms_vblank@pipe-b-query-forked,skip
> > > +igt@kms_vblank@pipe-b-query-busy,skip
> > > +igt@kms_vblank@pipe-b-query-forked-busy,skip
> > > +igt@kms_vblank@pipe-b-wait-idle,skip
> > > +igt@kms_vblank@pipe-b-wait-forked,skip
> > > +igt@kms_vblank@pipe-b-wait-busy,skip
> > > +igt@kms_vblank@pipe-b-wait-forked-busy,skip
> > > +igt@kms_vblank@pipe-b-ts-continuation-idle,skip
> > > +igt@kms_vblank@pipe-b-ts-continuation-modeset,skip
> > > diff --git a/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > new file mode 100644
> > > index 000000000000..01f7b4b399b5
> > > --- /dev/null
> > > +++ b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > @@ -0,0 +1,141 @@
> > > +igt@core_auth@getclient-simple,dmesg-warn
> > > +igt@core_auth@getclient-master-drop,pass
> > > +igt@core_auth@basic-auth,pass
> > > +igt@core_auth@many-magics,pass
> > > +igt@core_getclient,pass
> > > +igt@core_getstats,pass
> > > +igt@core_getversion,pass
> > > +igt@core_setmaster_vs_auth,pass
> > > +igt@drm_read@invalid-buffer,pass
> > > +igt@drm_read@fault-buffer,pass
> > > +igt@drm_read@empty-block,pass
> > > +igt@drm_read@empty-nonblock,pass
> > > +igt@drm_read@short-buffer-block,pass
> > > +igt@drm_read@short-buffer-nonblock,pass
> > > +igt@drm_read@short-buffer-wakeup,pass
> > > +igt@kms_addfb_basic@unused-handle,pass
> > > +igt@kms_addfb_basic@unused-pitches,pass
> > > +igt@kms_addfb_basic@unused-offsets,pass
> > > +igt@kms_addfb_basic@unused-modifier,pass
> > > +igt@kms_addfb_basic@legacy-format,dmesg-warn
> > > +igt@kms_addfb_basic@no-handle,pass
> > > +igt@kms_addfb_basic@basic,pass
> > > +igt@kms_addfb_basic@bad-pitch-0,pass
> > > +igt@kms_addfb_basic@bad-pitch-32,pass
> > > +igt@kms_addfb_basic@bad-pitch-63,pass
> > > +igt@kms_addfb_basic@bad-pitch-128,pass
> > > +igt@kms_addfb_basic@bad-pitch-256,pass
> > > +igt@kms_addfb_basic@bad-pitch-1024,pass
> > > +igt@kms_addfb_basic@bad-pitch-999,pass
> > > +igt@kms_addfb_basic@bad-pitch-65536,pass
> > > +igt@kms_addfb_basic@size-max,pass
> > > +igt@kms_addfb_basic@too-wide,pass
> > > +igt@kms_addfb_basic@too-high,dmesg-warn
> >
> > For test results on Trogdor, is is possible to have them be
> > success/fail/skip only?
> >
> > Results such as dmesg-warn/dmesg-fail are igt_runner specific and
> > because there isn't support for igt_runner on ChromeOS, they will be
> > difficult to replicate and debug.
>
> Actually, I wonder if it would be better to just treat
> dmesg-warn/dmesg-fail as pass/fail?  I'd noticed some flakes on
> rockchip which looked just like unrelated dmesg msg which just
> happened to show up while the test was running.

This is kinda the reason behind standardizing on drm dmesg logging, so
that we have some chances at filtering stuff out. Not sure that's a
good idea, since when your entire box splats and lockdep is dead, then
continuing to run drm tests is still fairly pointless.

I think this is another reason why trying at least to standardize this
stuff over drivers would be pretty good idea.

> Additionally, some of the tests, like msm_recovery, are *expected* to
> generate some dmesg spam since they are intentionally triggering GPU
> hangs to test the recovery mechanism.

Uh I don't like that. It just allows userspace to spam dmesg, which
doesn't seem like a great idea. That's at least why i915 dumps these
at a lower level, and in the past had a special "I'm going to whack
the gpu real hard expect hangs" knob in debugfs.

Having tests which intentionally spam dmesg above info level isn't
really good since then you need endless amounts of test-specific
encoding of what is considered a success and what not. Like when a
backmerge breaks a testcases which is already at dmesg-fail, is that
bad or not? Probably bad, but was the situation before that really
good or already kinda on fire?
-Daniel

> BR,
> -R
>
> > > +igt@kms_addfb_basic@bo-too-small,pass
> > > +igt@kms_addfb_basic@small-bo,pass
> > > +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> > > +igt@kms_addfb_basic@addfb25-bad-modifier,dmesg-warn
> > > +igt@kms_addfb_basic@invalid-get-prop-any,pass
> > > +igt@kms_addfb_basic@invalid-get-prop,pass
> > > +igt@kms_addfb_basic@invalid-set-prop-any,pass
> > > +igt@kms_addfb_basic@invalid-set-prop,pass
> > > +igt@kms_addfb_basic@master-rmfb,pass
> > > +igt@kms_atomic@plane-overlay-legacy,skip
> > > +igt@kms_atomic@plane-primary-legacy,pass
> > > +igt@kms_atomic@plane-primary-overlay-mutable-zpos,skip
> > > +igt@kms_atomic@plane-immutable-zpos,pass
> > > +igt@kms_atomic@test-only,pass
> > > +igt@kms_atomic@plane-cursor-legacy,pass
> > > +igt@kms_atomic@plane-invalid-params,pass
> > > +igt@kms_atomic@crtc-invalid-params,pass
> > > +igt@kms_atomic@atomic-invalid-params,pass
> > > +igt@kms_atomic@atomic_plane_damage,pass
> > > +igt@kms_atomic_interruptible@legacy-setmode,skip
> > > +igt@kms_atomic_interruptible@atomic-setmode,skip
> > > +igt@kms_atomic_interruptible@legacy-dpms,skip
> > > +igt@kms_atomic_interruptible@legacy-pageflip,skip
> > > +igt@kms_atomic_interruptible@legacy-cursor,skip
> > > +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> > > +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> > > +igt@kms_content_protection@lic,skip
> > > +igt@kms_flip_event_leak,pass
> > > +igt@kms_getfb@getfb-handle-zero,pass
> > > +igt@kms_getfb@getfb-handle-valid,pass
> > > +igt@kms_getfb@getfb-handle-closed,pass
> > > +igt@kms_getfb@getfb-handle-not-fb,pass
> > > +igt@kms_getfb@getfb-addfb-different-handles,pass
> > > +igt@kms_getfb@getfb-repeated-different-handles,pass
> > > +igt@kms_getfb@getfb2-handle-zero,pass
> > > +igt@kms_getfb@getfb2-handle-closed,pass
> > > +igt@kms_getfb@getfb2-handle-not-fb,pass
> > > +igt@kms_getfb@getfb2-into-addfb2,pass
> > > +igt@kms_getfb@getfb-handle-protection,pass
> > > +igt@kms_getfb@getfb2-handle-protection,pass
> > > +igt@kms_hdmi_inject@inject-4k,skip
> > > +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,skip
> > > +igt@kms_pipe_crc_basic@bad-source,pass
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-a,pass
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,pass
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,pass
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,pass
> > > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,pass
> > > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,pass
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-b,pass
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,pass
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,pass
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,pass
> > > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,pass
> > > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,pass
> > > +igt@kms_prop_blob@basic,pass
> > > +igt@kms_prop_blob@blob-prop-core,pass
> > > +igt@kms_prop_blob@blob-prop-validate,pass
> > > +igt@kms_prop_blob@blob-prop-lifetime,pass
> > > +igt@kms_prop_blob@blob-multiple,pass
> > > +igt@kms_prop_blob@invalid-get-prop-any,pass
> > > +igt@kms_prop_blob@invalid-get-prop,pass
> > > +igt@kms_prop_blob@invalid-set-prop-any,pass
> > > +igt@kms_prop_blob@invalid-set-prop,pass
> > > +igt@kms_rmfb@rmfb-ioctl,pass
> > > +igt@kms_rmfb@close-fd,fail
> > > +igt@kms_setmode@basic@pipe-a-edp-1,pass
> > > +igt@kms_setmode@basic@pipe-b-edp-1,pass
> > > +igt@msm_mapping@sqefw,dmesg-fail
> > > +igt@msm_mapping@shadow,dmesg-fail
> > > +igt@msm_recovery@hangcheck,dmesg-warn
> > > +igt@msm_recovery@gpu-fault,dmesg-warn
> > > +igt@msm_recovery@iova-fault,dmesg-warn
> > > +igt@msm_submit@empty-submit,pass
> > > +igt@msm_submit@invalid-queue-submit,pass
> > > +igt@msm_submit@invalid-flags-submit,pass
> > > +igt@msm_submit@invalid-in-fence-submit,pass
> > > +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> > > +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> > > +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> > > +igt@msm_submit@valid-submit,pass
> > > +igt@kms_sysfs_edid_timing,pass
> > > +igt@kms_universal_plane@universal-plane-pipe-a-functional,skip
> > > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,pass
> > > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,fail
> >
> > We could probably skip checking the results for kms_universal_plane on
> > Trogdor for now, since this is a test affected by the hack regression.
> > There is an IGT patch in the works for fixing the
> > disable-primary-vs-flip-pipe-b failure, so it should be updated pretty
> > soon too.
> >
> > Thanks,
> > Jessica Zhang
> >
> > > +igt@kms_vblank@invalid,pass
> > > +igt@kms_vblank@crtc-id,pass
> > > +igt@kms_vblank@pipe-a-query-idle,pass
> > > +igt@kms_vblank@pipe-a-query-forked,pass
> > > +igt@kms_vblank@pipe-a-query-busy,pass
> > > +igt@kms_vblank@pipe-a-query-forked-busy,pass
> > > +igt@kms_vblank@pipe-a-wait-idle,pass
> > > +igt@kms_vblank@pipe-a-wait-forked,pass
> > > +igt@kms_vblank@pipe-a-wait-busy,pass
> > > +igt@kms_vblank@pipe-a-wait-forked-busy,pass
> > > +igt@kms_vblank@pipe-a-ts-continuation-idle,pass
> > > +igt@kms_vblank@pipe-a-ts-continuation-modeset,pass
> > > +igt@kms_vblank@pipe-b-accuracy-idle,pass
> > > +igt@kms_vblank@pipe-b-query-idle,pass
> > > +igt@kms_vblank@pipe-b-query-forked,pass
> > > +igt@kms_vblank@pipe-b-query-busy,pass
> > > +igt@kms_vblank@pipe-b-query-forked-busy,pass
> > > +igt@kms_vblank@pipe-b-wait-idle,pass
> > > +igt@kms_vblank@pipe-b-wait-forked,pass
> > > +igt@kms_vblank@pipe-b-wait-busy,pass
> > > +igt@kms_vblank@pipe-b-wait-forked-busy,pass
> > > +igt@kms_vblank@pipe-b-ts-continuation-idle,pass
> > > +igt@kms_vblank@pipe-b-ts-continuation-modeset,pass
> > > diff --git a/drivers/gpu/drm/msm/ci/msm_sdm845_results.txt b/drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> > > new file mode 100644
> > > index 000000000000..3df1e5ef6d80
> > > --- /dev/null
> > > +++ b/drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> > > @@ -0,0 +1,141 @@
> > > +igt@core_auth@getclient-simple,dmesg-warn
> > > +igt@core_auth@getclient-master-drop,pass
> > > +igt@core_auth@basic-auth,pass
> > > +igt@core_auth@many-magics,pass
> > > +igt@core_getclient,pass
> > > +igt@core_getstats,pass
> > > +igt@core_getversion,pass
> > > +igt@core_setmaster_vs_auth,pass
> > > +igt@drm_read@invalid-buffer,pass
> > > +igt@drm_read@fault-buffer,pass
> > > +igt@drm_read@empty-block,pass
> > > +igt@drm_read@empty-nonblock,pass
> > > +igt@drm_read@short-buffer-block,pass
> > > +igt@drm_read@short-buffer-nonblock,pass
> > > +igt@drm_read@short-buffer-wakeup,pass
> > > +igt@kms_addfb_basic@unused-handle,pass
> > > +igt@kms_addfb_basic@unused-pitches,pass
> > > +igt@kms_addfb_basic@unused-offsets,pass
> > > +igt@kms_addfb_basic@unused-modifier,pass
> > > +igt@kms_addfb_basic@legacy-format,dmesg-warn
> > > +igt@kms_addfb_basic@no-handle,pass
> > > +igt@kms_addfb_basic@basic,pass
> > > +igt@kms_addfb_basic@bad-pitch-0,pass
> > > +igt@kms_addfb_basic@bad-pitch-32,pass
> > > +igt@kms_addfb_basic@bad-pitch-63,pass
> > > +igt@kms_addfb_basic@bad-pitch-128,pass
> > > +igt@kms_addfb_basic@bad-pitch-256,pass
> > > +igt@kms_addfb_basic@bad-pitch-1024,pass
> > > +igt@kms_addfb_basic@bad-pitch-999,pass
> > > +igt@kms_addfb_basic@bad-pitch-65536,pass
> > > +igt@kms_addfb_basic@size-max,pass
> > > +igt@kms_addfb_basic@too-wide,pass
> > > +igt@kms_addfb_basic@too-high,dmesg-warn
> > > +igt@kms_addfb_basic@bo-too-small,pass
> > > +igt@kms_addfb_basic@small-bo,pass
> > > +igt@kms_addfb_basic@addfb25-modifier-no-flag,pass
> > > +igt@kms_addfb_basic@addfb25-bad-modifier,dmesg-warn
> > > +igt@kms_addfb_basic@invalid-get-prop-any,pass
> > > +igt@kms_addfb_basic@invalid-get-prop,pass
> > > +igt@kms_addfb_basic@invalid-set-prop-any,pass
> > > +igt@kms_addfb_basic@invalid-set-prop,pass
> > > +igt@kms_addfb_basic@master-rmfb,pass
> > > +igt@kms_atomic@plane-overlay-legacy,dmesg-warn
> > > +igt@kms_atomic@plane-primary-legacy,dmesg-warn
> > > +igt@kms_atomic@plane-primary-overlay-mutable-zpos,dmesg-warn
> > > +igt@kms_atomic@plane-immutable-zpos,dmesg-warn
> > > +igt@kms_atomic@test-only,dmesg-warn
> > > +igt@kms_atomic@plane-cursor-legacy,dmesg-warn
> > > +igt@kms_atomic@plane-invalid-params,dmesg-warn
> > > +igt@kms_atomic@crtc-invalid-params,dmesg-warn
> > > +igt@kms_atomic@atomic-invalid-params,dmesg-warn
> > > +igt@kms_atomic@atomic_plane_damage,dmesg-warn
> > > +igt@kms_atomic_interruptible@legacy-setmode,skip
> > > +igt@kms_atomic_interruptible@atomic-setmode,skip
> > > +igt@kms_atomic_interruptible@legacy-dpms,skip
> > > +igt@kms_atomic_interruptible@legacy-pageflip,skip
> > > +igt@kms_atomic_interruptible@legacy-cursor,skip
> > > +igt@kms_atomic_interruptible@universal-setplane-primary,skip
> > > +igt@kms_atomic_interruptible@universal-setplane-cursor,skip
> > > +igt@kms_content_protection@lic,skip
> > > +igt@kms_flip_event_leak,dmesg-warn
> > > +igt@kms_getfb@getfb-handle-zero,pass
> > > +igt@kms_getfb@getfb-handle-valid,pass
> > > +igt@kms_getfb@getfb-handle-closed,pass
> > > +igt@kms_getfb@getfb-handle-not-fb,pass
> > > +igt@kms_getfb@getfb-addfb-different-handles,pass
> > > +igt@kms_getfb@getfb-repeated-different-handles,pass
> > > +igt@kms_getfb@getfb2-handle-zero,pass
> > > +igt@kms_getfb@getfb2-handle-closed,pass
> > > +igt@kms_getfb@getfb2-handle-not-fb,pass
> > > +igt@kms_getfb@getfb2-into-addfb2,pass
> > > +igt@kms_getfb@getfb-handle-protection,pass
> > > +igt@kms_getfb@getfb2-handle-protection,pass
> > > +igt@kms_hdmi_inject@inject-4k,skip
> > > +igt@kms_multipipe_modeset@basic-max-pipe-crc-check,pass
> > > +igt@kms_pipe_crc_basic@bad-source,pass
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-a,pass
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence,pass
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a,pass
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence,pass
> > > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-a,dmesg-warn
> > > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-a,dmesg-warn
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-b,skip
> > > +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence,skip
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b,skip
> > > +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence,skip
> > > +igt@kms_pipe_crc_basic@disable-crc-after-crtc-pipe-b,skip
> > > +igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b,skip
> > > +igt@kms_prop_blob@basic,pass
> > > +igt@kms_prop_blob@blob-prop-core,pass
> > > +igt@kms_prop_blob@blob-prop-validate,pass
> > > +igt@kms_prop_blob@blob-prop-lifetime,pass
> > > +igt@kms_prop_blob@blob-multiple,pass
> > > +igt@kms_prop_blob@invalid-get-prop-any,pass
> > > +igt@kms_prop_blob@invalid-get-prop,pass
> > > +igt@kms_prop_blob@invalid-set-prop-any,pass
> > > +igt@kms_prop_blob@invalid-set-prop,pass
> > > +igt@kms_rmfb@rmfb-ioctl,pass
> > > +igt@kms_rmfb@close-fd,fail
> > > +igt@kms_setmode@basic@pipe-a-edp-1,dmesg-warn
> > > +igt@kms_setmode@basic,skip
> > > +igt@msm_mapping@sqefw,dmesg-fail
> > > +igt@msm_mapping@shadow,dmesg-fail
> > > +igt@msm_recovery@hangcheck,dmesg-warn
> > > +igt@msm_recovery@gpu-fault,dmesg-warn
> > > +igt@msm_recovery@iova-fault,dmesg-warn
> > > +igt@msm_submit@empty-submit,pass
> > > +igt@msm_submit@invalid-queue-submit,pass
> > > +igt@msm_submit@invalid-flags-submit,pass
> > > +igt@msm_submit@invalid-in-fence-submit,pass
> > > +igt@msm_submit@invalid-duplicate-bo-submit,dmesg-warn
> > > +igt@msm_submit@invalid-cmd-idx-submit,dmesg-warn
> > > +igt@msm_submit@invalid-cmd-type-submit,dmesg-warn
> > > +igt@msm_submit@valid-submit,pass
> > > +igt@kms_sysfs_edid_timing,pass
> > > +igt@kms_universal_plane@universal-plane-pipe-a-functional,dmesg-fail
> > > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-a,dmesg-warn
> > > +igt@kms_universal_plane@disable-primary-vs-flip-pipe-b,skip
> > > +igt@kms_vblank@invalid,dmesg-warn
> > > +igt@kms_vblank@crtc-id,dmesg-warn
> > > +igt@kms_vblank@pipe-a-query-idle,dmesg-warn
> > > +igt@kms_vblank@pipe-a-query-forked,dmesg-warn
> > > +igt@kms_vblank@pipe-a-query-busy,dmesg-warn
> > > +igt@kms_vblank@pipe-a-query-forked-busy,dmesg-warn
> > > +igt@kms_vblank@pipe-a-wait-idle,dmesg-warn
> > > +igt@kms_vblank@pipe-a-wait-forked,dmesg-warn
> > > +igt@kms_vblank@pipe-a-wait-busy,dmesg-warn
> > > +igt@kms_vblank@pipe-a-wait-forked-busy,dmesg-warn
> > > +igt@kms_vblank@pipe-a-ts-continuation-idle,dmesg-warn
> > > +igt@kms_vblank@pipe-a-ts-continuation-modeset,dmesg-warn
> > > +igt@kms_vblank@pipe-b-accuracy-idle,skip
> > > +igt@kms_vblank@pipe-b-query-idle,skip
> > > +igt@kms_vblank@pipe-b-query-forked,skip
> > > +igt@kms_vblank@pipe-b-query-busy,skip
> > > +igt@kms_vblank@pipe-b-query-forked-busy,skip
> > > +igt@kms_vblank@pipe-b-wait-idle,skip
> > > +igt@kms_vblank@pipe-b-wait-forked,skip
> > > +igt@kms_vblank@pipe-b-wait-busy,skip
> > > +igt@kms_vblank@pipe-b-wait-forked-busy,skip
> > > +igt@kms_vblank@pipe-b-ts-continuation-idle,skip
> > > +igt@kms_vblank@pipe-b-ts-continuation-modeset,skip
> > > --
> > > 2.31.1
> > >

Rob Clark May 11, 2022, 5:23 p.m. UTC | #11

On Wed, May 11, 2022 at 9:43 AM Daniel Vetter <daniel@ffwll.ch> wrote:
>
> On Wed, 11 May 2022 at 15:33, Rob Clark <robdclark@gmail.com> wrote:
> > On Wed, May 11, 2022 at 4:50 AM Greg Kroah-Hartman
> > <gregkh@linuxfoundation.org> wrote:
> > >
> > > On Wed, May 11, 2022 at 12:26:05PM +0200, Michel Dänzer wrote:
> > > > On 2022-05-11 08:22, Greg Kroah-Hartman wrote:
> > > > > On Wed, May 11, 2022 at 03:06:47PM +1000, Dave Airlie wrote:
> > > > >>> And use it to store expectations about what the drm/msm driver is
> > > > >>> supposed to pass in the IGT test suite.
> > > > >>
> > > > >> I wanted to loop in Linus/Greg to see if there are any issues raised
> > > > >> by adding CI results file to the tree in their minds, or if any other
> > > > >> subsystem has done this already, and it's all fine.
> > > > >
> > > > > Why does the results need to be added to the tree?  Shouldn't they be
> > > > > either "all is good" or "constantly changing and a constant churn"?
> > > > >
> > > > >> I think this is a good thing after our Mesa experience, but Mesa has a
> > > > >> lot tighter integration here, so I want to get some more opinions
> > > > >> outside the group.
> > > > >
> > > > > For systems that have "tight integration" this might make sense as proof
> > > > > that all is working for a specific commit, but I can't see how this will
> > > > > help the kernel out much.
> > > > >
> > > > > What are you going to do with these results being checked in all the
> > > > > time?
> > > >
> > > > Having the expected results in the tree keeps them consistent with the driver code itself, and allows putting in place gating CI to prevent merging driver changes which make any of the tests deviate from the expected result.
> > >
> > > Shouldn't "expected result" always be "pass"?
> > >
> > > If not, then the test should be changed to be "skipped" like we have
> > > today in the kselftest tests.
> >
> > No, we want to run tests even if they are expected to fail.  This
> > prevents the scenario of a test getting fixed without being noticed
> > (for ex, developer was working on fixing test A and didn't notice that
> > the fix also fixed test B).  If a fix goes unnoticed, a later
> > regression would also go unnoticed ;-)
> >
> > I was skeptical about this approach at first with mesa CI, but having
> > used mesa CI for a while, I am now a firm believer in the approach.
> >
> > And ofc we want the expectations to be in the kernel tree because
> > there could be, for example, differences between -fixes and -next
> > branches.  (Or even stable kernel branches if/when we get to the point
> > of running CI on those.)
>
> Yeah result files in tree is kinda needed, even more so for the
> kernel. A lot of the linux-next integration testing is only done after
> patches have landed, and sometimes such breakage makes it to upstream
> and then into the subsystem/driver tree. Annotating in the backmerge
> what exactly broke and why helps a lot with tracking issues.
>
> And expecting every subsystem to run every other subsystem's tests,
> especially tests that run on hw, is just not going to scale. So there
> will be all kinds of difference in test results.
>
> > > And how about tieing this into the kselftest process as well, why would
> > > this be somehow separate from the rest of the kernel tests?
> > >
> > > > Keeping them separate inevitably results in divergence between the driver code and the expected test results, which would result in spurious failures of such CI.
> > >
> > > Again, "pass" should be the expected results :)
> > >
> > > > I expect the main complication for the kernel will be due to driver changes merged via different trees, e.g. for cross-subsystem reworks. Since those will not go through the same CI, they may accidentally introduce inconsistencies. The ideal solution for this IMO would be centralizing CI such that the same gating tests have to pass regardless of how the code is merged. But there's likely quite a long way to go until we get there. :)
> > >
> > > We have in-kernel tests for the rest of the kernel, why can't you put
> > > your testing stuff into there as well?
> >
> > We could ofc put a lot more of the gitlab yml and scripts into the
> > kernel tree.  Probably all of i-g-t is a bit much to put in the kernel
> > tree.  Not to mention I'd like to see this expand to also run some
> > deqp and/or piglit tests, which is definitely too much to vendor into
> > the kernel tree.
> >
> > The approach of this RFC was to put only what was absolutely required
> > in the kernel tree (such as expectations), and then link out to an
> > external drm-ci tree[1] which has all the necessary scripts and yml
> > for building and running tests, to avoid having to put a whole lot
> > more in the kernel tree. (We should be specifying exact commit-sha for
> > that tree, IMO, as it controls the version of i-g-t which gets used,
> > and we need to be able to update expectations in sync with an i-g-t
> > uprev, for example when new tests are added or if a test fix caused a
> > fail->pass transition.)
>
> Yeah I think longer-term we should carry a lot more in upstream, at
> least anything that's shared across drivers wrt the ci integration (or
> build testing and running tests which are hw agnostic). Maybe even
> igt, not sure (otoh xfs-tests isn't moving into the kernel either, and
> there's lots more like that).

A lot of the drm-ci tree is the scripts/etc for things like power
control, booting, etc.. and a lot of that is identical to what we have
in the mesa tree (since the on-hw tests run on the same CI farms as
mesa-ci)

But ofc it can be re-used by other drivers via one line in toplevel
$driver/ci/gitlab-ci.yml, ie:

  DRM_CI_PROJECT_PATH: &drm-ci-project-path gfx-ci/drm-ci

BR,
-R

> Personally I think long-term the only thing outside should be other
> repos with tests or stuff you need to run them, and not really the
> glue to make it all work in ci. But that's maybe a bit too much
> wishful thinking if CI systems stay largely subsystem specific (which
> they currently are in many ways, with some overlap).
>
> But maybe there is enough random pieces to share here for a lot more
> in-tree to make sense, and imo the fewer extra steps and indirection
> CI testing and test updating has, the better.
>
> But like Rob says, eventually there's a limit and when you put the
> entire GL/vulkan stack + it's conformance testsuite (which is
> maintained by khronos somewhere completely different than both
> kernel.org and freedesktop.org) then it's definitely too much and wont
> work. And eventually we do want to run these things too (e.g.
> intel-gfx-ci does run mesa + piglit on every run).
> -Daniel
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch

Linus Torvalds May 11, 2022, 5:33 p.m. UTC | #12

On Tue, May 10, 2022 at 10:07 PM Dave Airlie <airlied@gmail.com> wrote:
>
> > And use it to store expectations about what the drm/msm driver is
> > supposed to pass in the IGT test suite.
>
> I wanted to loop in Linus/Greg to see if there are any issues raised
> by adding CI results file to the tree in their minds, or if any other
> subsystem has done this already, and it's all fine.
>
> I think this is a good thing after our Mesa experience, but Mesa has a
> lot tighter integration here, so I want to get some more opinions
> outside the group.

Honestly, my immediate reaction is that I think it might be ok, but

 (a) are these things going to absolutely balloon over time?

 (b) should these not be separated out?

Those two issues kind of interact.

If it's a small and targeted test-suite, by all means keep it in the
kernel, but why not make it part of "tools/testing/selftests"

But if people expect this to balloon and we end up having megabytes of
test output, then I really think it should be a separate git tree.

A diffstat like this:

>  7 files changed, 791 insertions(+)

is not a problem at all. But I get the feeling that this is just the
tip of the iceberg, and people will want to not just have the result
files, but start adding actual *input* files that may be largely
automated stuff and may be tens of megabytes in size.

Because the result files on their own aren't really self-contained,
and then people will want to keep them in sync with the test-files
themselves, and start adding those, and now it *really* is likely very
unwieldy.

Or if that doesn't happen, and the actual input test files stay in a
separate CI repo, and then you end up having random coherency issues
with that CI repo, and it all gets to be either horribly messy, or the
result files in the kernel end up really stale.

So honestly, I personally don't see a good end result here.  This
particular small patch? *This* one looks fine to me, except I really
think tools/testing/selftests/gpu would be a much more logical place
for it.

But I don't see a way forward that is sane.

Can somebody argue otherwise?

            Linus

Rob Clark May 11, 2022, 5:46 p.m. UTC | #13

On Wed, May 11, 2022 at 10:12 AM Daniel Vetter <daniel@ffwll.ch> wrote:
>
> On Tue, 10 May 2022 at 22:26, Rob Clark <robdclark@gmail.com> wrote:
> >
> > On Tue, May 10, 2022 at 12:39 PM Jessica Zhang
> > <quic_jesszhan@quicinc.com> wrote:
> > >
> > >
> > >
> > > On 5/10/2022 7:13 AM, Tomeu Vizoso wrote:
> > > > And use it to store expectations about what the drm/msm driver is
> > > > supposed to pass in the IGT test suite.
> > > >
> > > > Also include a configuration file that points to the out-of-tree CI
> > > > scripts.
> > > >
> > > > By storing the test expectations along the code we can make sure both
> > > > stay in sync with each other, and so we can know when a code change
> > > > breaks those expectations.
> > > >
> > > > This will allow all contributors to drm/msm to reuse the infrastructure
> > > > already in gitlab.freedesktop.org to test the driver on several
> > > > generations of the hardware.
> > > >
> > > > v2:
> > > >    - Fix names of result expectation files to match SoC
> > > >    - Don't execute tests that are going to skip on all boards
> > > >
> > > > Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> > > > ---
> > > >   Documentation/gpu/msm_automated_testing.rst   |  70 +++++++++
> > > >   drivers/gpu/drm/msm/ci/gitlab-ci.yml          |  11 ++
> > > >   drivers/gpu/drm/msm/ci/msm.testlist           | 148 ++++++++++++++++++
> > > >   .../gpu/drm/msm/ci/msm_apq8016_results.txt    | 140 +++++++++++++++++
> > > >   .../gpu/drm/msm/ci/msm_apq8096_results.txt    | 140 +++++++++++++++++
> > > >   drivers/gpu/drm/msm/ci/msm_sc7180_results.txt | 141 +++++++++++++++++
> > > >   drivers/gpu/drm/msm/ci/msm_sdm845_results.txt | 141 +++++++++++++++++
> > > >   7 files changed, 791 insertions(+)
> > > >   create mode 100644 Documentation/gpu/msm_automated_testing.rst
> > > >   create mode 100644 drivers/gpu/drm/msm/ci/gitlab-ci.yml
> > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm.testlist
> > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> > > >

[snip]

> > > > diff --git a/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > > new file mode 100644
> > > > index 000000000000..01f7b4b399b5
> > > > --- /dev/null
> > > > +++ b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > > @@ -0,0 +1,141 @@
> > > > +igt@core_auth@getclient-simple,dmesg-warn
> > > > +igt@core_auth@getclient-master-drop,pass
> > > > +igt@core_auth@basic-auth,pass
> > > > +igt@core_auth@many-magics,pass
> > > > +igt@core_getclient,pass
> > > > +igt@core_getstats,pass
> > > > +igt@core_getversion,pass
> > > > +igt@core_setmaster_vs_auth,pass
> > > > +igt@drm_read@invalid-buffer,pass
> > > > +igt@drm_read@fault-buffer,pass
> > > > +igt@drm_read@empty-block,pass
> > > > +igt@drm_read@empty-nonblock,pass
> > > > +igt@drm_read@short-buffer-block,pass
> > > > +igt@drm_read@short-buffer-nonblock,pass
> > > > +igt@drm_read@short-buffer-wakeup,pass
> > > > +igt@kms_addfb_basic@unused-handle,pass
> > > > +igt@kms_addfb_basic@unused-pitches,pass
> > > > +igt@kms_addfb_basic@unused-offsets,pass
> > > > +igt@kms_addfb_basic@unused-modifier,pass
> > > > +igt@kms_addfb_basic@legacy-format,dmesg-warn
> > > > +igt@kms_addfb_basic@no-handle,pass
> > > > +igt@kms_addfb_basic@basic,pass
> > > > +igt@kms_addfb_basic@bad-pitch-0,pass
> > > > +igt@kms_addfb_basic@bad-pitch-32,pass
> > > > +igt@kms_addfb_basic@bad-pitch-63,pass
> > > > +igt@kms_addfb_basic@bad-pitch-128,pass
> > > > +igt@kms_addfb_basic@bad-pitch-256,pass
> > > > +igt@kms_addfb_basic@bad-pitch-1024,pass
> > > > +igt@kms_addfb_basic@bad-pitch-999,pass
> > > > +igt@kms_addfb_basic@bad-pitch-65536,pass
> > > > +igt@kms_addfb_basic@size-max,pass
> > > > +igt@kms_addfb_basic@too-wide,pass
> > > > +igt@kms_addfb_basic@too-high,dmesg-warn
> > >
> > > For test results on Trogdor, is is possible to have them be
> > > success/fail/skip only?
> > >
> > > Results such as dmesg-warn/dmesg-fail are igt_runner specific and
> > > because there isn't support for igt_runner on ChromeOS, they will be
> > > difficult to replicate and debug.
> >
> > Actually, I wonder if it would be better to just treat
> > dmesg-warn/dmesg-fail as pass/fail?  I'd noticed some flakes on
> > rockchip which looked just like unrelated dmesg msg which just
> > happened to show up while the test was running.
>
> This is kinda the reason behind standardizing on drm dmesg logging, so
> that we have some chances at filtering stuff out. Not sure that's a
> good idea, since when your entire box splats and lockdep is dead, then
> continuing to run drm tests is still fairly pointless.

I'm not sure if we are using it yet for drm-ci, but for mesa-ci we
monitor dmesg (over serial port, from the controller) for splats, so
we already have the tech for restarting or aborting the CI run.  We
don't need igt-runner to tell us.

> I think this is another reason why trying at least to standardize this
> stuff over drivers would be pretty good idea.
>
> > Additionally, some of the tests, like msm_recovery, are *expected* to
> > generate some dmesg spam since they are intentionally triggering GPU
> > hangs to test the recovery mechanism.
>
> Uh I don't like that. It just allows userspace to spam dmesg, which
> doesn't seem like a great idea. That's at least why i915 dumps these
> at a lower level, and in the past had a special "I'm going to whack
> the gpu real hard expect hangs" knob in debugfs.
>
> Having tests which intentionally spam dmesg above info level isn't
> really good since then you need endless amounts of test-specific
> encoding of what is considered a success and what not. Like when a
> backmerge breaks a testcases which is already at dmesg-fail, is that
> bad or not? Probably bad, but was the situation before that really
> good or already kinda on fire?

I guess I could add some debugfs knobs to squelch the dmesg msgs on
gpu hangs.  In the normal case, I'd prefer that gpu hangs are not
silent.. since that is something we get in feedback reports if a user
(or dogfooder) reports a bug.

The rockchip case I mentioned was some unrelated dmesg about
linktraining failing.. presumably because there was no display
attached?  IDK, I didn't look too closely.  But my point is we could
be getting unrelated and asynchronous dmesg spam, even from other
kernel subsystems.  Letting that be part of the test results just
sounds like asking for flakes.

BR,
-R

> -Daniel
>
> > BR,
> > -R
> >

Rob Clark May 11, 2022, 6:39 p.m. UTC | #14

On Wed, May 11, 2022 at 10:33 AM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Tue, May 10, 2022 at 10:07 PM Dave Airlie <airlied@gmail.com> wrote:
> >
> > > And use it to store expectations about what the drm/msm driver is
> > > supposed to pass in the IGT test suite.
> >
> > I wanted to loop in Linus/Greg to see if there are any issues raised
> > by adding CI results file to the tree in their minds, or if any other
> > subsystem has done this already, and it's all fine.
> >
> > I think this is a good thing after our Mesa experience, but Mesa has a
> > lot tighter integration here, so I want to get some more opinions
> > outside the group.
>
> Honestly, my immediate reaction is that I think it might be ok, but
>
>  (a) are these things going to absolutely balloon over time?
>
>  (b) should these not be separated out?
>
> Those two issues kind of interact.
>
> If it's a small and targeted test-suite, by all means keep it in the
> kernel, but why not make it part of "tools/testing/selftests"
>
> But if people expect this to balloon and we end up having megabytes of
> test output, then I really think it should be a separate git tree.
>
> A diffstat like this:
>
> >  7 files changed, 791 insertions(+)
>
> is not a problem at all. But I get the feeling that this is just the
> tip of the iceberg, and people will want to not just have the result
> files, but start adding actual *input* files that may be largely
> automated stuff and may be tens of megabytes in size.
>
> Because the result files on their own aren't really self-contained,
> and then people will want to keep them in sync with the test-files
> themselves, and start adding those, and now it *really* is likely very
> unwieldy.

It is missing in this revision of the RFC, but the intention is to
have the gitlab-ci.yml point to a specific commit SHA in the
gfx-ci/drm-ci[1] tree, to solve the problem of keeping the results in
sync with the expectations.  Ie. a kernel commit would control moving
to a new version of i-g-t (and eventually deqp and/or piglit), and at
the same time make any necessary updates in the expectations files.

BR,
-R

[1] https://gitlab.freedesktop.org/gfx-ci/drm-ci

> Or if that doesn't happen, and the actual input test files stay in a
> separate CI repo, and then you end up having random coherency issues
> with that CI repo, and it all gets to be either horribly messy, or the
> result files in the kernel end up really stale.
>
> So honestly, I personally don't see a good end result here.  This
> particular small patch? *This* one looks fine to me, except I really
> think tools/testing/selftests/gpu would be a much more logical place
> for it.
>
> But I don't see a way forward that is sane.
>
> Can somebody argue otherwise?
>
>             Linus

Linus Torvalds May 11, 2022, 7:08 p.m. UTC | #15

On Wed, May 11, 2022 at 11:40 AM Rob Clark <robdclark@gmail.com> wrote:
>
> It is missing in this revision of the RFC, but the intention is to
> have the gitlab-ci.yml point to a specific commit SHA in the
> gfx-ci/drm-ci[1] tree, to solve the problem of keeping the results in
> sync with the expectations.  Ie. a kernel commit would control moving
> to a new version of i-g-t (and eventually deqp and/or piglit), and at
> the same time make any necessary updates in the expectations files.

Wouldn't it then be better to just have the expectation files in the
ci tree too?

The kernel tree might have just the expected *failures* listed, if
there are any. Presumably the ci tree has to have the expected results
anyway, so what's the advantage of listing non-failures?

                  Linus

Linus Torvalds May 11, 2022, 7:12 p.m. UTC | #16

On Wed, May 11, 2022 at 12:08 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> The kernel tree might have just the expected *failures* listed, if
> there are any. Presumably the ci tree has to have the expected results
> anyway, so what's the advantage of listing non-failures?

.. put another way: I think a list of "we are aware that these
currently fail" is quite reasonable for a development tree, maybe even
with a comment in the commit that created them about why they
currently fail.

That also ends up being very nice if you fix a problem, and the fix
commit might then remove the failure for the list, and that all makes
perfect sense.

But having just the raw output of "these are the expected CI results"
that is being done and specified by some other tree entirely - that
seems pointless and just noise to me. There's no actual reason to have
that kind of noise - and update that kind of noise - that I really
see.

                Linus

Daniel Vetter May 11, 2022, 7:14 p.m. UTC | #17

On Wed, 11 May 2022 at 19:46, Rob Clark <robdclark@gmail.com> wrote:
>
> On Wed, May 11, 2022 at 10:12 AM Daniel Vetter <daniel@ffwll.ch> wrote:
> >
> > On Tue, 10 May 2022 at 22:26, Rob Clark <robdclark@gmail.com> wrote:
> > >
> > > On Tue, May 10, 2022 at 12:39 PM Jessica Zhang
> > > <quic_jesszhan@quicinc.com> wrote:
> > > >
> > > >
> > > >
> > > > On 5/10/2022 7:13 AM, Tomeu Vizoso wrote:
> > > > > And use it to store expectations about what the drm/msm driver is
> > > > > supposed to pass in the IGT test suite.
> > > > >
> > > > > Also include a configuration file that points to the out-of-tree CI
> > > > > scripts.
> > > > >
> > > > > By storing the test expectations along the code we can make sure both
> > > > > stay in sync with each other, and so we can know when a code change
> > > > > breaks those expectations.
> > > > >
> > > > > This will allow all contributors to drm/msm to reuse the infrastructure
> > > > > already in gitlab.freedesktop.org to test the driver on several
> > > > > generations of the hardware.
> > > > >
> > > > > v2:
> > > > >    - Fix names of result expectation files to match SoC
> > > > >    - Don't execute tests that are going to skip on all boards
> > > > >
> > > > > Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> > > > > ---
> > > > >   Documentation/gpu/msm_automated_testing.rst   |  70 +++++++++
> > > > >   drivers/gpu/drm/msm/ci/gitlab-ci.yml          |  11 ++
> > > > >   drivers/gpu/drm/msm/ci/msm.testlist           | 148 ++++++++++++++++++
> > > > >   .../gpu/drm/msm/ci/msm_apq8016_results.txt    | 140 +++++++++++++++++
> > > > >   .../gpu/drm/msm/ci/msm_apq8096_results.txt    | 140 +++++++++++++++++
> > > > >   drivers/gpu/drm/msm/ci/msm_sc7180_results.txt | 141 +++++++++++++++++
> > > > >   drivers/gpu/drm/msm/ci/msm_sdm845_results.txt | 141 +++++++++++++++++
> > > > >   7 files changed, 791 insertions(+)
> > > > >   create mode 100644 Documentation/gpu/msm_automated_testing.rst
> > > > >   create mode 100644 drivers/gpu/drm/msm/ci/gitlab-ci.yml
> > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm.testlist
> > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> > > > >
>
> [snip]
>
> > > > > diff --git a/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > > > new file mode 100644
> > > > > index 000000000000..01f7b4b399b5
> > > > > --- /dev/null
> > > > > +++ b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > > > @@ -0,0 +1,141 @@
> > > > > +igt@core_auth@getclient-simple,dmesg-warn
> > > > > +igt@core_auth@getclient-master-drop,pass
> > > > > +igt@core_auth@basic-auth,pass
> > > > > +igt@core_auth@many-magics,pass
> > > > > +igt@core_getclient,pass
> > > > > +igt@core_getstats,pass
> > > > > +igt@core_getversion,pass
> > > > > +igt@core_setmaster_vs_auth,pass
> > > > > +igt@drm_read@invalid-buffer,pass
> > > > > +igt@drm_read@fault-buffer,pass
> > > > > +igt@drm_read@empty-block,pass
> > > > > +igt@drm_read@empty-nonblock,pass
> > > > > +igt@drm_read@short-buffer-block,pass
> > > > > +igt@drm_read@short-buffer-nonblock,pass
> > > > > +igt@drm_read@short-buffer-wakeup,pass
> > > > > +igt@kms_addfb_basic@unused-handle,pass
> > > > > +igt@kms_addfb_basic@unused-pitches,pass
> > > > > +igt@kms_addfb_basic@unused-offsets,pass
> > > > > +igt@kms_addfb_basic@unused-modifier,pass
> > > > > +igt@kms_addfb_basic@legacy-format,dmesg-warn
> > > > > +igt@kms_addfb_basic@no-handle,pass
> > > > > +igt@kms_addfb_basic@basic,pass
> > > > > +igt@kms_addfb_basic@bad-pitch-0,pass
> > > > > +igt@kms_addfb_basic@bad-pitch-32,pass
> > > > > +igt@kms_addfb_basic@bad-pitch-63,pass
> > > > > +igt@kms_addfb_basic@bad-pitch-128,pass
> > > > > +igt@kms_addfb_basic@bad-pitch-256,pass
> > > > > +igt@kms_addfb_basic@bad-pitch-1024,pass
> > > > > +igt@kms_addfb_basic@bad-pitch-999,pass
> > > > > +igt@kms_addfb_basic@bad-pitch-65536,pass
> > > > > +igt@kms_addfb_basic@size-max,pass
> > > > > +igt@kms_addfb_basic@too-wide,pass
> > > > > +igt@kms_addfb_basic@too-high,dmesg-warn
> > > >
> > > > For test results on Trogdor, is is possible to have them be
> > > > success/fail/skip only?
> > > >
> > > > Results such as dmesg-warn/dmesg-fail are igt_runner specific and
> > > > because there isn't support for igt_runner on ChromeOS, they will be
> > > > difficult to replicate and debug.
> > >
> > > Actually, I wonder if it would be better to just treat
> > > dmesg-warn/dmesg-fail as pass/fail?  I'd noticed some flakes on
> > > rockchip which looked just like unrelated dmesg msg which just
> > > happened to show up while the test was running.
> >
> > This is kinda the reason behind standardizing on drm dmesg logging, so
> > that we have some chances at filtering stuff out. Not sure that's a
> > good idea, since when your entire box splats and lockdep is dead, then
> > continuing to run drm tests is still fairly pointless.
>
> I'm not sure if we are using it yet for drm-ci, but for mesa-ci we
> monitor dmesg (over serial port, from the controller) for splats, so
> we already have the tech for restarting or aborting the CI run.  We
> don't need igt-runner to tell us.
>
> > I think this is another reason why trying at least to standardize this
> > stuff over drivers would be pretty good idea.
> >
> > > Additionally, some of the tests, like msm_recovery, are *expected* to
> > > generate some dmesg spam since they are intentionally triggering GPU
> > > hangs to test the recovery mechanism.
> >
> > Uh I don't like that. It just allows userspace to spam dmesg, which
> > doesn't seem like a great idea. That's at least why i915 dumps these
> > at a lower level, and in the past had a special "I'm going to whack
> > the gpu real hard expect hangs" knob in debugfs.
> >
> > Having tests which intentionally spam dmesg above info level isn't
> > really good since then you need endless amounts of test-specific
> > encoding of what is considered a success and what not. Like when a
> > backmerge breaks a testcases which is already at dmesg-fail, is that
> > bad or not? Probably bad, but was the situation before that really
> > good or already kinda on fire?
>
> I guess I could add some debugfs knobs to squelch the dmesg msgs on
> gpu hangs.  In the normal case, I'd prefer that gpu hangs are not
> silent.. since that is something we get in feedback reports if a user
> (or dogfooder) reports a bug.
>
> The rockchip case I mentioned was some unrelated dmesg about
> linktraining failing.. presumably because there was no display
> attached?  IDK, I didn't look too closely.  But my point is we could
> be getting unrelated and asynchronous dmesg spam, even from other
> kernel subsystems.  Letting that be part of the test results just
> sounds like asking for flakes.

That's why I think you need to filter dmesg, otherwise you'll just get
random stuff in there that causes flaps.

And then in another level you need different filtering to figure out
when your kernel is too busted and you need to reboot (like when
lockdep splats anywhere).

Maybe we should pull that part of igt_runner out so that it can be
used in places where the full thing cant? Maybe even in mesa as a "my
kernel is in really bad shape" kinda test. There's a lot of tea leaves
reading heuristics involved in this stuff ime.
-Daniel

>
> BR,
> -R
>
> > -Daniel
> >
> > > BR,
> > > -R
> > >

Daniel Vetter May 11, 2022, 7:39 p.m. UTC | #18

On Wed, May 11, 2022 at 10:33:06AM -0700, Linus Torvalds wrote:
> On Tue, May 10, 2022 at 10:07 PM Dave Airlie <airlied@gmail.com> wrote:
> >
> > > And use it to store expectations about what the drm/msm driver is
> > > supposed to pass in the IGT test suite.
> >
> > I wanted to loop in Linus/Greg to see if there are any issues raised
> > by adding CI results file to the tree in their minds, or if any other
> > subsystem has done this already, and it's all fine.
> >
> > I think this is a good thing after our Mesa experience, but Mesa has a
> > lot tighter integration here, so I want to get some more opinions
> > outside the group.
> 
> Honestly, my immediate reaction is that I think it might be ok, but
> 
>  (a) are these things going to absolutely balloon over time?
> 
>  (b) should these not be separated out?
> 
> Those two issues kind of interact.
> 
> If it's a small and targeted test-suite, by all means keep it in the
> kernel, but why not make it part of "tools/testing/selftests"
> 
> But if people expect this to balloon and we end up having megabytes of
> test output, then I really think it should be a separate git tree.
> 
> A diffstat like this:
> 
> >  7 files changed, 791 insertions(+)

Yeah I guess it's good to have some numbers for where this might go. Good
comparison is probably mesa3d, since it's the same-ish people doing the
same-ish ci on the same-ish infrastructure, just the userspace part of it.

mesa$ git ls-files | grep ci | xargs cat | wc -l
123077

mesa$ git ls-files | grep ci | wc -l
421

Compared to drivers/gpu it's really not much, and mesa is about the size
of drivers/gpu if you exclude the massive amount of register headers from
amd.

And I guess if we do stuff like result file compression like you mentioned
it should be quite a bit less even.

So yeah if this does take off it wil be substantially more, but I don't
think it'll ever get to a point where it'll swamp code changes. And if it
does that's kinda a solid indicator that something really wrong is going
on.

> is not a problem at all. But I get the feeling that this is just the
> tip of the iceberg, and people will want to not just have the result
> files, but start adding actual *input* files that may be largely
> automated stuff and may be tens of megabytes in size.
> 
> Because the result files on their own aren't really self-contained,
> and then people will want to keep them in sync with the test-files
> themselves, and start adding those, and now it *really* is likely very
> unwieldy.
> 
> Or if that doesn't happen, and the actual input test files stay in a
> separate CI repo, and then you end up having random coherency issues
> with that CI repo, and it all gets to be either horribly messy, or the
> result files in the kernel end up really stale.
> 
> So honestly, I personally don't see a good end result here.  This
> particular small patch? *This* one looks fine to me, except I really
> think tools/testing/selftests/gpu would be a much more logical place
> for it.
> 
> But I don't see a way forward that is sane.
> 
> Can somebody argue otherwise?

I do personally think we should add a bunch more things here, radically
putting everything into the drm-ci repo feels a bit much like appeasement
to get the foot in the door. Like some of the scripts are definitely
specific to the ci infra on freedesktop.org (or specific hw runners for
the drivers), and that makes sense to keep in that drm/fd ci repo. But
other scripts should probably migrate to scripts/ and at least start out
in a ci/ folder in the kernel.

igt itself might eventually move to tools/testing/selftests/gpu or
whatever, but that's kinda a huge discussion onto itself. And I haven't
seen a clear consensus yet among subsystem that these kind of tests (like
xfs-tests, and I think pretty much ever bigger subsystem that is old
enough to predate selftests has them somewhere) should all move into
tools/testing/selftest. Maybe they should, but feels like this is
orthogonal to ci integration.

Note that mesa3d has the exact same issue going that you're raising, and
some of those are unfixable because the opengl/vulkan conformance test
suites are maintained entirely externally by Khronos (and you have to use
those or you're not conformant to the spec, which renders the point of
having a shared spec a bit moot). It's messy but workable, and the CI you
get seems very much to be worth the price.

One idea I tossed out on irc is to move this all under drivers/gpu/ci.
There's driver specific stuff like the test result/fail lists, and maybe
those could eventually move out to drivers. But for starting out it might
be better to keep it all in one place so it's a bit better under control
and doesn't accidentally become a kranken of some kind. And then make sure
pieces move to scripts/ or tools/testing/ appropriately.

In general I think any mess this causes is a pretty good indicator that
something is amiss, like if this causes messy history due to tests
flipping too much and causing issues then that also indicates an issue
with the kernel or testcase quality itself. And it might be good to shine
more light on that stuff.

Cheers, Daniel

Rob Clark May 11, 2022, 8:06 p.m. UTC | #19

On Wed, May 11, 2022 at 12:08 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Wed, May 11, 2022 at 11:40 AM Rob Clark <robdclark@gmail.com> wrote:
> >
> > It is missing in this revision of the RFC, but the intention is to
> > have the gitlab-ci.yml point to a specific commit SHA in the
> > gfx-ci/drm-ci[1] tree, to solve the problem of keeping the results in
> > sync with the expectations.  Ie. a kernel commit would control moving
> > to a new version of i-g-t (and eventually deqp and/or piglit), and at
> > the same time make any necessary updates in the expectations files.
>
> Wouldn't it then be better to just have the expectation files in the
> ci tree too?

The main reason is that we would frequently have situations where both
-next and -fixes pointing at the same ci tree commit, but with
differing expectations.  If we kept the expectations in the ci tree,
we'd end up frequently updating the ci tree and then updating the
kernel tree to point to the appropriate ci tree version.

Additionally, on the mesa side, it has been useful to squash the
expectations update into the commit that fixed a bug or added a
feature.  It provides a connection in git history between code and
test results.

BR,
-R

Rob Clark May 11, 2022, 8:14 p.m. UTC | #20

On Wed, May 11, 2022 at 12:12 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Wed, May 11, 2022 at 12:08 PM Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > The kernel tree might have just the expected *failures* listed, if
> > there are any. Presumably the ci tree has to have the expected results
> > anyway, so what's the advantage of listing non-failures?
>
> .. put another way: I think a list of "we are aware that these
> currently fail" is quite reasonable for a development tree, maybe even
> with a comment in the commit that created them about why they
> currently fail.
>
> That also ends up being very nice if you fix a problem, and the fix
> commit might then remove the failure for the list, and that all makes
> perfect sense.
>
> But having just the raw output of "these are the expected CI results"
> that is being done and specified by some other tree entirely - that
> seems pointless and just noise to me. There's no actual reason to have
> that kind of noise - and update that kind of noise - that I really
> see.

Yeah, the only reason we have full results is that the current tool to
check for pass/fail of the entire CI job is 'diff' ;-)

It has the nice benefit of generating a patch for you to squash into
whatever commit to update the expectation files, I suppose.  But we
have something more clever on the mesa-ci side of things where we list
skips/flakes/expected-fails but not expected-passes.  To be fair, the
# of tests on the mesa side is something on the order of 750,000, I
don't expect to ever get close to that # on the kernel side.

BR,
-R

>
>                 Linus

Rob Clark May 11, 2022, 8:32 p.m. UTC | #21

On Wed, May 11, 2022 at 12:14 PM Daniel Vetter <daniel@ffwll.ch> wrote:
>
> On Wed, 11 May 2022 at 19:46, Rob Clark <robdclark@gmail.com> wrote:
> >
> > On Wed, May 11, 2022 at 10:12 AM Daniel Vetter <daniel@ffwll.ch> wrote:
> > >
> > > On Tue, 10 May 2022 at 22:26, Rob Clark <robdclark@gmail.com> wrote:
> > > >
> > > > On Tue, May 10, 2022 at 12:39 PM Jessica Zhang
> > > > <quic_jesszhan@quicinc.com> wrote:
> > > > >
> > > > >
> > > > >
> > > > > On 5/10/2022 7:13 AM, Tomeu Vizoso wrote:
> > > > > > And use it to store expectations about what the drm/msm driver is
> > > > > > supposed to pass in the IGT test suite.
> > > > > >
> > > > > > Also include a configuration file that points to the out-of-tree CI
> > > > > > scripts.
> > > > > >
> > > > > > By storing the test expectations along the code we can make sure both
> > > > > > stay in sync with each other, and so we can know when a code change
> > > > > > breaks those expectations.
> > > > > >
> > > > > > This will allow all contributors to drm/msm to reuse the infrastructure
> > > > > > already in gitlab.freedesktop.org to test the driver on several
> > > > > > generations of the hardware.
> > > > > >
> > > > > > v2:
> > > > > >    - Fix names of result expectation files to match SoC
> > > > > >    - Don't execute tests that are going to skip on all boards
> > > > > >
> > > > > > Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> > > > > > ---
> > > > > >   Documentation/gpu/msm_automated_testing.rst   |  70 +++++++++
> > > > > >   drivers/gpu/drm/msm/ci/gitlab-ci.yml          |  11 ++
> > > > > >   drivers/gpu/drm/msm/ci/msm.testlist           | 148 ++++++++++++++++++
> > > > > >   .../gpu/drm/msm/ci/msm_apq8016_results.txt    | 140 +++++++++++++++++
> > > > > >   .../gpu/drm/msm/ci/msm_apq8096_results.txt    | 140 +++++++++++++++++
> > > > > >   drivers/gpu/drm/msm/ci/msm_sc7180_results.txt | 141 +++++++++++++++++
> > > > > >   drivers/gpu/drm/msm/ci/msm_sdm845_results.txt | 141 +++++++++++++++++
> > > > > >   7 files changed, 791 insertions(+)
> > > > > >   create mode 100644 Documentation/gpu/msm_automated_testing.rst
> > > > > >   create mode 100644 drivers/gpu/drm/msm/ci/gitlab-ci.yml
> > > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm.testlist
> > > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> > > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> > > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> > > > > >
> >
> > [snip]
> >
> > > > > > diff --git a/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > > > > new file mode 100644
> > > > > > index 000000000000..01f7b4b399b5
> > > > > > --- /dev/null
> > > > > > +++ b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > > > > @@ -0,0 +1,141 @@
> > > > > > +igt@core_auth@getclient-simple,dmesg-warn
> > > > > > +igt@core_auth@getclient-master-drop,pass
> > > > > > +igt@core_auth@basic-auth,pass
> > > > > > +igt@core_auth@many-magics,pass
> > > > > > +igt@core_getclient,pass
> > > > > > +igt@core_getstats,pass
> > > > > > +igt@core_getversion,pass
> > > > > > +igt@core_setmaster_vs_auth,pass
> > > > > > +igt@drm_read@invalid-buffer,pass
> > > > > > +igt@drm_read@fault-buffer,pass
> > > > > > +igt@drm_read@empty-block,pass
> > > > > > +igt@drm_read@empty-nonblock,pass
> > > > > > +igt@drm_read@short-buffer-block,pass
> > > > > > +igt@drm_read@short-buffer-nonblock,pass
> > > > > > +igt@drm_read@short-buffer-wakeup,pass
> > > > > > +igt@kms_addfb_basic@unused-handle,pass
> > > > > > +igt@kms_addfb_basic@unused-pitches,pass
> > > > > > +igt@kms_addfb_basic@unused-offsets,pass
> > > > > > +igt@kms_addfb_basic@unused-modifier,pass
> > > > > > +igt@kms_addfb_basic@legacy-format,dmesg-warn
> > > > > > +igt@kms_addfb_basic@no-handle,pass
> > > > > > +igt@kms_addfb_basic@basic,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-0,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-32,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-63,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-128,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-256,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-1024,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-999,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-65536,pass
> > > > > > +igt@kms_addfb_basic@size-max,pass
> > > > > > +igt@kms_addfb_basic@too-wide,pass
> > > > > > +igt@kms_addfb_basic@too-high,dmesg-warn
> > > > >
> > > > > For test results on Trogdor, is is possible to have them be
> > > > > success/fail/skip only?
> > > > >
> > > > > Results such as dmesg-warn/dmesg-fail are igt_runner specific and
> > > > > because there isn't support for igt_runner on ChromeOS, they will be
> > > > > difficult to replicate and debug.
> > > >
> > > > Actually, I wonder if it would be better to just treat
> > > > dmesg-warn/dmesg-fail as pass/fail?  I'd noticed some flakes on
> > > > rockchip which looked just like unrelated dmesg msg which just
> > > > happened to show up while the test was running.
> > >
> > > This is kinda the reason behind standardizing on drm dmesg logging, so
> > > that we have some chances at filtering stuff out. Not sure that's a
> > > good idea, since when your entire box splats and lockdep is dead, then
> > > continuing to run drm tests is still fairly pointless.
> >
> > I'm not sure if we are using it yet for drm-ci, but for mesa-ci we
> > monitor dmesg (over serial port, from the controller) for splats, so
> > we already have the tech for restarting or aborting the CI run.  We
> > don't need igt-runner to tell us.
> >
> > > I think this is another reason why trying at least to standardize this
> > > stuff over drivers would be pretty good idea.
> > >
> > > > Additionally, some of the tests, like msm_recovery, are *expected* to
> > > > generate some dmesg spam since they are intentionally triggering GPU
> > > > hangs to test the recovery mechanism.
> > >
> > > Uh I don't like that. It just allows userspace to spam dmesg, which
> > > doesn't seem like a great idea. That's at least why i915 dumps these
> > > at a lower level, and in the past had a special "I'm going to whack
> > > the gpu real hard expect hangs" knob in debugfs.
> > >
> > > Having tests which intentionally spam dmesg above info level isn't
> > > really good since then you need endless amounts of test-specific
> > > encoding of what is considered a success and what not. Like when a
> > > backmerge breaks a testcases which is already at dmesg-fail, is that
> > > bad or not? Probably bad, but was the situation before that really
> > > good or already kinda on fire?
> >
> > I guess I could add some debugfs knobs to squelch the dmesg msgs on
> > gpu hangs.  In the normal case, I'd prefer that gpu hangs are not
> > silent.. since that is something we get in feedback reports if a user
> > (or dogfooder) reports a bug.
> >
> > The rockchip case I mentioned was some unrelated dmesg about
> > linktraining failing.. presumably because there was no display
> > attached?  IDK, I didn't look too closely.  But my point is we could
> > be getting unrelated and asynchronous dmesg spam, even from other
> > kernel subsystems.  Letting that be part of the test results just
> > sounds like asking for flakes.
>
> That's why I think you need to filter dmesg, otherwise you'll just get
> random stuff in there that causes flaps.

We've evolved a bit of that over the time, see:

https://gitlab.freedesktop.org/gfx-ci/drm-ci/-/blob/main/.gitlab-ci/bare-metal/cros_servo_run.py#L94

It tends to be a bit specific to the runners and their quirks.  But
mesa does a lot of CI runs, so once-in-a-blue-moon quirks quickly turn
frustrating if the CI machinery doesn't detect them, and (for ex)
reboot and restart the DUT

BR,
-R

> And then in another level you need different filtering to figure out
> when your kernel is too busted and you need to reboot (like when
> lockdep splats anywhere).
>
> Maybe we should pull that part of igt_runner out so that it can be
> used in places where the full thing cant? Maybe even in mesa as a "my
> kernel is in really bad shape" kinda test. There's a lot of tea leaves
> reading heuristics involved in this stuff ime.
> -Daniel
>
> >
> > BR,
> > -R
> >
> > > -Daniel
> > >
> > > > BR,
> > > > -R
> > > >
>
>
>
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch

Daniel Vetter May 11, 2022, 9:09 p.m. UTC | #22

On Wed, 11 May 2022 at 22:32, Rob Clark <robdclark@gmail.com> wrote:
>
> On Wed, May 11, 2022 at 12:14 PM Daniel Vetter <daniel@ffwll.ch> wrote:
> >
> > On Wed, 11 May 2022 at 19:46, Rob Clark <robdclark@gmail.com> wrote:
> > >
> > > On Wed, May 11, 2022 at 10:12 AM Daniel Vetter <daniel@ffwll.ch> wrote:
> > > >
> > > > On Tue, 10 May 2022 at 22:26, Rob Clark <robdclark@gmail.com> wrote:
> > > > >
> > > > > On Tue, May 10, 2022 at 12:39 PM Jessica Zhang
> > > > > <quic_jesszhan@quicinc.com> wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 5/10/2022 7:13 AM, Tomeu Vizoso wrote:
> > > > > > > And use it to store expectations about what the drm/msm driver is
> > > > > > > supposed to pass in the IGT test suite.
> > > > > > >
> > > > > > > Also include a configuration file that points to the out-of-tree CI
> > > > > > > scripts.
> > > > > > >
> > > > > > > By storing the test expectations along the code we can make sure both
> > > > > > > stay in sync with each other, and so we can know when a code change
> > > > > > > breaks those expectations.
> > > > > > >
> > > > > > > This will allow all contributors to drm/msm to reuse the infrastructure
> > > > > > > already in gitlab.freedesktop.org to test the driver on several
> > > > > > > generations of the hardware.
> > > > > > >
> > > > > > > v2:
> > > > > > >    - Fix names of result expectation files to match SoC
> > > > > > >    - Don't execute tests that are going to skip on all boards
> > > > > > >
> > > > > > > Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> > > > > > > ---
> > > > > > >   Documentation/gpu/msm_automated_testing.rst   |  70 +++++++++
> > > > > > >   drivers/gpu/drm/msm/ci/gitlab-ci.yml          |  11 ++
> > > > > > >   drivers/gpu/drm/msm/ci/msm.testlist           | 148 ++++++++++++++++++
> > > > > > >   .../gpu/drm/msm/ci/msm_apq8016_results.txt    | 140 +++++++++++++++++
> > > > > > >   .../gpu/drm/msm/ci/msm_apq8096_results.txt    | 140 +++++++++++++++++
> > > > > > >   drivers/gpu/drm/msm/ci/msm_sc7180_results.txt | 141 +++++++++++++++++
> > > > > > >   drivers/gpu/drm/msm/ci/msm_sdm845_results.txt | 141 +++++++++++++++++
> > > > > > >   7 files changed, 791 insertions(+)
> > > > > > >   create mode 100644 Documentation/gpu/msm_automated_testing.rst
> > > > > > >   create mode 100644 drivers/gpu/drm/msm/ci/gitlab-ci.yml
> > > > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm.testlist
> > > > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> > > > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> > > > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > > > > >   create mode 100644 drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> > > > > > >
> > >
> > > [snip]
> > >
> > > > > > > diff --git a/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > > > > > new file mode 100644
> > > > > > > index 000000000000..01f7b4b399b5
> > > > > > > --- /dev/null
> > > > > > > +++ b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > > > > > @@ -0,0 +1,141 @@
> > > > > > > +igt@core_auth@getclient-simple,dmesg-warn
> > > > > > > +igt@core_auth@getclient-master-drop,pass
> > > > > > > +igt@core_auth@basic-auth,pass
> > > > > > > +igt@core_auth@many-magics,pass
> > > > > > > +igt@core_getclient,pass
> > > > > > > +igt@core_getstats,pass
> > > > > > > +igt@core_getversion,pass
> > > > > > > +igt@core_setmaster_vs_auth,pass
> > > > > > > +igt@drm_read@invalid-buffer,pass
> > > > > > > +igt@drm_read@fault-buffer,pass
> > > > > > > +igt@drm_read@empty-block,pass
> > > > > > > +igt@drm_read@empty-nonblock,pass
> > > > > > > +igt@drm_read@short-buffer-block,pass
> > > > > > > +igt@drm_read@short-buffer-nonblock,pass
> > > > > > > +igt@drm_read@short-buffer-wakeup,pass
> > > > > > > +igt@kms_addfb_basic@unused-handle,pass
> > > > > > > +igt@kms_addfb_basic@unused-pitches,pass
> > > > > > > +igt@kms_addfb_basic@unused-offsets,pass
> > > > > > > +igt@kms_addfb_basic@unused-modifier,pass
> > > > > > > +igt@kms_addfb_basic@legacy-format,dmesg-warn
> > > > > > > +igt@kms_addfb_basic@no-handle,pass
> > > > > > > +igt@kms_addfb_basic@basic,pass
> > > > > > > +igt@kms_addfb_basic@bad-pitch-0,pass
> > > > > > > +igt@kms_addfb_basic@bad-pitch-32,pass
> > > > > > > +igt@kms_addfb_basic@bad-pitch-63,pass
> > > > > > > +igt@kms_addfb_basic@bad-pitch-128,pass
> > > > > > > +igt@kms_addfb_basic@bad-pitch-256,pass
> > > > > > > +igt@kms_addfb_basic@bad-pitch-1024,pass
> > > > > > > +igt@kms_addfb_basic@bad-pitch-999,pass
> > > > > > > +igt@kms_addfb_basic@bad-pitch-65536,pass
> > > > > > > +igt@kms_addfb_basic@size-max,pass
> > > > > > > +igt@kms_addfb_basic@too-wide,pass
> > > > > > > +igt@kms_addfb_basic@too-high,dmesg-warn
> > > > > >
> > > > > > For test results on Trogdor, is is possible to have them be
> > > > > > success/fail/skip only?
> > > > > >
> > > > > > Results such as dmesg-warn/dmesg-fail are igt_runner specific and
> > > > > > because there isn't support for igt_runner on ChromeOS, they will be
> > > > > > difficult to replicate and debug.
> > > > >
> > > > > Actually, I wonder if it would be better to just treat
> > > > > dmesg-warn/dmesg-fail as pass/fail?  I'd noticed some flakes on
> > > > > rockchip which looked just like unrelated dmesg msg which just
> > > > > happened to show up while the test was running.
> > > >
> > > > This is kinda the reason behind standardizing on drm dmesg logging, so
> > > > that we have some chances at filtering stuff out. Not sure that's a
> > > > good idea, since when your entire box splats and lockdep is dead, then
> > > > continuing to run drm tests is still fairly pointless.
> > >
> > > I'm not sure if we are using it yet for drm-ci, but for mesa-ci we
> > > monitor dmesg (over serial port, from the controller) for splats, so
> > > we already have the tech for restarting or aborting the CI run.  We
> > > don't need igt-runner to tell us.
> > >
> > > > I think this is another reason why trying at least to standardize this
> > > > stuff over drivers would be pretty good idea.
> > > >
> > > > > Additionally, some of the tests, like msm_recovery, are *expected* to
> > > > > generate some dmesg spam since they are intentionally triggering GPU
> > > > > hangs to test the recovery mechanism.
> > > >
> > > > Uh I don't like that. It just allows userspace to spam dmesg, which
> > > > doesn't seem like a great idea. That's at least why i915 dumps these
> > > > at a lower level, and in the past had a special "I'm going to whack
> > > > the gpu real hard expect hangs" knob in debugfs.
> > > >
> > > > Having tests which intentionally spam dmesg above info level isn't
> > > > really good since then you need endless amounts of test-specific
> > > > encoding of what is considered a success and what not. Like when a
> > > > backmerge breaks a testcases which is already at dmesg-fail, is that
> > > > bad or not? Probably bad, but was the situation before that really
> > > > good or already kinda on fire?
> > >
> > > I guess I could add some debugfs knobs to squelch the dmesg msgs on
> > > gpu hangs.  In the normal case, I'd prefer that gpu hangs are not
> > > silent.. since that is something we get in feedback reports if a user
> > > (or dogfooder) reports a bug.
> > >
> > > The rockchip case I mentioned was some unrelated dmesg about
> > > linktraining failing.. presumably because there was no display
> > > attached?  IDK, I didn't look too closely.  But my point is we could
> > > be getting unrelated and asynchronous dmesg spam, even from other
> > > kernel subsystems.  Letting that be part of the test results just
> > > sounds like asking for flakes.
> >
> > That's why I think you need to filter dmesg, otherwise you'll just get
> > random stuff in there that causes flaps.
>
> We've evolved a bit of that over the time, see:
>
> https://gitlab.freedesktop.org/gfx-ci/drm-ci/-/blob/main/.gitlab-ci/bare-metal/cros_servo_run.py#L94
>
> It tends to be a bit specific to the runners and their quirks.  But
> mesa does a lot of CI runs, so once-in-a-blue-moon quirks quickly turn
> frustrating if the CI machinery doesn't detect them, and (for ex)
> reboot and restart the DUT

Yeah that kind of stuff is probably best kept in a ci-farm adjacent
repo, since it's all very specific to the machine and ci setup.
-Daniel

>
> BR,
> -R
>
> > And then in another level you need different filtering to figure out
> > when your kernel is too busted and you need to reboot (like when
> > lockdep splats anywhere).
> >
> > Maybe we should pull that part of igt_runner out so that it can be
> > used in places where the full thing cant? Maybe even in mesa as a "my
> > kernel is in really bad shape" kinda test. There's a lot of tea leaves
> > reading heuristics involved in this stuff ime.
> > -Daniel
> >
> > >
> > > BR,
> > > -R
> > >
> > > > -Daniel
> > > >
> > > > > BR,
> > > > > -R
> > > > >
> >
> >
> >
> > --
> > Daniel Vetter
> > Software Engineer, Intel Corporation
> > http://blog.ffwll.ch

Theodore Ts'o May 12, 2022, 2:24 a.m. UTC | #23

On Wed, May 11, 2022 at 06:33:32AM -0700, Rob Clark wrote:
> 
> And ofc we want the expectations to be in the kernel tree because
> there could be, for example, differences between -fixes and -next
> branches.  (Or even stable kernel branches if/when we get to the point
> of running CI on those.)

There are tradeoffs both ways, whether the patches are kept separate,
opr in the kernel tree.

In the file system world, when we discover a bug, very often a test
case is found to test the fix, and to protect us against regressions.
It has one other benefit; since the tests (xfstests) are kept separate
from the kernel, it's a useful way to identify when some patch didn't
get automatically backported to a LTS or distro kernel.  (For example,
because the patch didn't cherry-pick cleanly and the manual backport
process fell through the cracks.)

It does make things annoying when we have bugs that can not be safely
backported (which results in tests that fail on the LTS kernel without
kernel-version exclude files), and/or when the expectations change
between versions.  (Although to be honest, for us, the more common
annoyance is when some userspace package --- e.g., bash or coreutils
or util-linux --- changes their output, and we have to add filter
functions to accomodate expected output differences.)

						- Ted

Tomeu Vizoso May 12, 2022, 1:28 p.m. UTC | #24

On 5/11/22 7:46 PM, Rob Clark wrote:
> On Wed, May 11, 2022 at 10:12 AM Daniel Vetter <daniel@ffwll.ch> wrote:
>>
>> On Tue, 10 May 2022 at 22:26, Rob Clark <robdclark@gmail.com> wrote:
>>>
>>> On Tue, May 10, 2022 at 12:39 PM Jessica Zhang
>>> <quic_jesszhan@quicinc.com> wrote:
>>>>
>>>>
>>>>
>>>> On 5/10/2022 7:13 AM, Tomeu Vizoso wrote:
>>>>> And use it to store expectations about what the drm/msm driver is
>>>>> supposed to pass in the IGT test suite.
>>>>>
>>>>> Also include a configuration file that points to the out-of-tree CI
>>>>> scripts.
>>>>>
>>>>> By storing the test expectations along the code we can make sure both
>>>>> stay in sync with each other, and so we can know when a code change
>>>>> breaks those expectations.
>>>>>
>>>>> This will allow all contributors to drm/msm to reuse the infrastructure
>>>>> already in gitlab.freedesktop.org to test the driver on several
>>>>> generations of the hardware.
>>>>>
>>>>> v2:
>>>>>     - Fix names of result expectation files to match SoC
>>>>>     - Don't execute tests that are going to skip on all boards
>>>>>
>>>>> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
>>>>> ---
>>>>>    Documentation/gpu/msm_automated_testing.rst   |  70 +++++++++
>>>>>    drivers/gpu/drm/msm/ci/gitlab-ci.yml          |  11 ++
>>>>>    drivers/gpu/drm/msm/ci/msm.testlist           | 148 ++++++++++++++++++
>>>>>    .../gpu/drm/msm/ci/msm_apq8016_results.txt    | 140 +++++++++++++++++
>>>>>    .../gpu/drm/msm/ci/msm_apq8096_results.txt    | 140 +++++++++++++++++
>>>>>    drivers/gpu/drm/msm/ci/msm_sc7180_results.txt | 141 +++++++++++++++++
>>>>>    drivers/gpu/drm/msm/ci/msm_sdm845_results.txt | 141 +++++++++++++++++
>>>>>    7 files changed, 791 insertions(+)
>>>>>    create mode 100644 Documentation/gpu/msm_automated_testing.rst
>>>>>    create mode 100644 drivers/gpu/drm/msm/ci/gitlab-ci.yml
>>>>>    create mode 100644 drivers/gpu/drm/msm/ci/msm.testlist
>>>>>    create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
>>>>>    create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
>>>>>    create mode 100644 drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
>>>>>    create mode 100644 drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
>>>>>
> 
> [snip]
> 
>>>>> diff --git a/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
>>>>> new file mode 100644
>>>>> index 000000000000..01f7b4b399b5
>>>>> --- /dev/null
>>>>> +++ b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
>>>>> @@ -0,0 +1,141 @@
>>>>> +igt@core_auth@getclient-simple,dmesg-warn
>>>>> +igt@core_auth@getclient-master-drop,pass
>>>>> +igt@core_auth@basic-auth,pass
>>>>> +igt@core_auth@many-magics,pass
>>>>> +igt@core_getclient,pass
>>>>> +igt@core_getstats,pass
>>>>> +igt@core_getversion,pass
>>>>> +igt@core_setmaster_vs_auth,pass
>>>>> +igt@drm_read@invalid-buffer,pass
>>>>> +igt@drm_read@fault-buffer,pass
>>>>> +igt@drm_read@empty-block,pass
>>>>> +igt@drm_read@empty-nonblock,pass
>>>>> +igt@drm_read@short-buffer-block,pass
>>>>> +igt@drm_read@short-buffer-nonblock,pass
>>>>> +igt@drm_read@short-buffer-wakeup,pass
>>>>> +igt@kms_addfb_basic@unused-handle,pass
>>>>> +igt@kms_addfb_basic@unused-pitches,pass
>>>>> +igt@kms_addfb_basic@unused-offsets,pass
>>>>> +igt@kms_addfb_basic@unused-modifier,pass
>>>>> +igt@kms_addfb_basic@legacy-format,dmesg-warn
>>>>> +igt@kms_addfb_basic@no-handle,pass
>>>>> +igt@kms_addfb_basic@basic,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-0,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-32,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-63,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-128,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-256,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-1024,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-999,pass
>>>>> +igt@kms_addfb_basic@bad-pitch-65536,pass
>>>>> +igt@kms_addfb_basic@size-max,pass
>>>>> +igt@kms_addfb_basic@too-wide,pass
>>>>> +igt@kms_addfb_basic@too-high,dmesg-warn
>>>>
>>>> For test results on Trogdor, is is possible to have them be
>>>> success/fail/skip only?
>>>>
>>>> Results such as dmesg-warn/dmesg-fail are igt_runner specific and
>>>> because there isn't support for igt_runner on ChromeOS, they will be
>>>> difficult to replicate and debug.
>>>
>>> Actually, I wonder if it would be better to just treat
>>> dmesg-warn/dmesg-fail as pass/fail?  I'd noticed some flakes on
>>> rockchip which looked just like unrelated dmesg msg which just
>>> happened to show up while the test was running.
>>
>> This is kinda the reason behind standardizing on drm dmesg logging, so
>> that we have some chances at filtering stuff out. Not sure that's a
>> good idea, since when your entire box splats and lockdep is dead, then
>> continuing to run drm tests is still fairly pointless.
> 
> I'm not sure if we are using it yet for drm-ci, but for mesa-ci we
> monitor dmesg (over serial port, from the controller) for splats, so
> we already have the tech for restarting or aborting the CI run.  We
> don't need igt-runner to tell us.

Yep, these scripts are currently being used as-is from Mesa, so we got 
that functionality for free.

>> I think this is another reason why trying at least to standardize this
>> stuff over drivers would be pretty good idea.
>>
>>> Additionally, some of the tests, like msm_recovery, are *expected* to
>>> generate some dmesg spam since they are intentionally triggering GPU
>>> hangs to test the recovery mechanism.
>>
>> Uh I don't like that. It just allows userspace to spam dmesg, which
>> doesn't seem like a great idea. That's at least why i915 dumps these
>> at a lower level, and in the past had a special "I'm going to whack
>> the gpu real hard expect hangs" knob in debugfs.
>>
>> Having tests which intentionally spam dmesg above info level isn't
>> really good since then you need endless amounts of test-specific
>> encoding of what is considered a success and what not. Like when a
>> backmerge breaks a testcases which is already at dmesg-fail, is that
>> bad or not? Probably bad, but was the situation before that really
>> good or already kinda on fire?
> 
> I guess I could add some debugfs knobs to squelch the dmesg msgs on
> gpu hangs.  In the normal case, I'd prefer that gpu hangs are not
> silent.. since that is something we get in feedback reports if a user
> (or dogfooder) reports a bug.
> 
> The rockchip case I mentioned was some unrelated dmesg about
> linktraining failing.. presumably because there was no display
> attached?  IDK, I didn't look too closely.  But my point is we could
> be getting unrelated and asynchronous dmesg spam, even from other
> kernel subsystems.  Letting that be part of the test results just
> sounds like asking for flakes.

I think some drivers are currently a bit too buggy to behave reliably 
under CI unless one reduces coverage (rockchip on rk3399, for example). 
And some other drivers (in other subsystems as well) could do with a 
review of what they print to the console. I guess these are things we 
could and probably should fix?

Cheers,

Tomeu

> BR,
> -R
> 
>> -Daniel
>>
>>> BR,
>>> -R
>>>

Daniel Vetter May 12, 2022, 2:02 p.m. UTC | #25

On Thu, May 12, 2022 at 03:28:16PM +0200, Tomeu Vizoso wrote:
> On 5/11/22 7:46 PM, Rob Clark wrote:
> > On Wed, May 11, 2022 at 10:12 AM Daniel Vetter <daniel@ffwll.ch> wrote:
> > > 
> > > On Tue, 10 May 2022 at 22:26, Rob Clark <robdclark@gmail.com> wrote:
> > > > 
> > > > On Tue, May 10, 2022 at 12:39 PM Jessica Zhang
> > > > <quic_jesszhan@quicinc.com> wrote:
> > > > > 
> > > > > 
> > > > > 
> > > > > On 5/10/2022 7:13 AM, Tomeu Vizoso wrote:
> > > > > > And use it to store expectations about what the drm/msm driver is
> > > > > > supposed to pass in the IGT test suite.
> > > > > > 
> > > > > > Also include a configuration file that points to the out-of-tree CI
> > > > > > scripts.
> > > > > > 
> > > > > > By storing the test expectations along the code we can make sure both
> > > > > > stay in sync with each other, and so we can know when a code change
> > > > > > breaks those expectations.
> > > > > > 
> > > > > > This will allow all contributors to drm/msm to reuse the infrastructure
> > > > > > already in gitlab.freedesktop.org to test the driver on several
> > > > > > generations of the hardware.
> > > > > > 
> > > > > > v2:
> > > > > >     - Fix names of result expectation files to match SoC
> > > > > >     - Don't execute tests that are going to skip on all boards
> > > > > > 
> > > > > > Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> > > > > > ---
> > > > > >    Documentation/gpu/msm_automated_testing.rst   |  70 +++++++++
> > > > > >    drivers/gpu/drm/msm/ci/gitlab-ci.yml          |  11 ++
> > > > > >    drivers/gpu/drm/msm/ci/msm.testlist           | 148 ++++++++++++++++++
> > > > > >    .../gpu/drm/msm/ci/msm_apq8016_results.txt    | 140 +++++++++++++++++
> > > > > >    .../gpu/drm/msm/ci/msm_apq8096_results.txt    | 140 +++++++++++++++++
> > > > > >    drivers/gpu/drm/msm/ci/msm_sc7180_results.txt | 141 +++++++++++++++++
> > > > > >    drivers/gpu/drm/msm/ci/msm_sdm845_results.txt | 141 +++++++++++++++++
> > > > > >    7 files changed, 791 insertions(+)
> > > > > >    create mode 100644 Documentation/gpu/msm_automated_testing.rst
> > > > > >    create mode 100644 drivers/gpu/drm/msm/ci/gitlab-ci.yml
> > > > > >    create mode 100644 drivers/gpu/drm/msm/ci/msm.testlist
> > > > > >    create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8016_results.txt
> > > > > >    create mode 100644 drivers/gpu/drm/msm/ci/msm_apq8096_results.txt
> > > > > >    create mode 100644 drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > > > >    create mode 100644 drivers/gpu/drm/msm/ci/msm_sdm845_results.txt
> > > > > > 
> > 
> > [snip]
> > 
> > > > > > diff --git a/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > > > > new file mode 100644
> > > > > > index 000000000000..01f7b4b399b5
> > > > > > --- /dev/null
> > > > > > +++ b/drivers/gpu/drm/msm/ci/msm_sc7180_results.txt
> > > > > > @@ -0,0 +1,141 @@
> > > > > > +igt@core_auth@getclient-simple,dmesg-warn
> > > > > > +igt@core_auth@getclient-master-drop,pass
> > > > > > +igt@core_auth@basic-auth,pass
> > > > > > +igt@core_auth@many-magics,pass
> > > > > > +igt@core_getclient,pass
> > > > > > +igt@core_getstats,pass
> > > > > > +igt@core_getversion,pass
> > > > > > +igt@core_setmaster_vs_auth,pass
> > > > > > +igt@drm_read@invalid-buffer,pass
> > > > > > +igt@drm_read@fault-buffer,pass
> > > > > > +igt@drm_read@empty-block,pass
> > > > > > +igt@drm_read@empty-nonblock,pass
> > > > > > +igt@drm_read@short-buffer-block,pass
> > > > > > +igt@drm_read@short-buffer-nonblock,pass
> > > > > > +igt@drm_read@short-buffer-wakeup,pass
> > > > > > +igt@kms_addfb_basic@unused-handle,pass
> > > > > > +igt@kms_addfb_basic@unused-pitches,pass
> > > > > > +igt@kms_addfb_basic@unused-offsets,pass
> > > > > > +igt@kms_addfb_basic@unused-modifier,pass
> > > > > > +igt@kms_addfb_basic@legacy-format,dmesg-warn
> > > > > > +igt@kms_addfb_basic@no-handle,pass
> > > > > > +igt@kms_addfb_basic@basic,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-0,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-32,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-63,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-128,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-256,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-1024,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-999,pass
> > > > > > +igt@kms_addfb_basic@bad-pitch-65536,pass
> > > > > > +igt@kms_addfb_basic@size-max,pass
> > > > > > +igt@kms_addfb_basic@too-wide,pass
> > > > > > +igt@kms_addfb_basic@too-high,dmesg-warn
> > > > > 
> > > > > For test results on Trogdor, is is possible to have them be
> > > > > success/fail/skip only?
> > > > > 
> > > > > Results such as dmesg-warn/dmesg-fail are igt_runner specific and
> > > > > because there isn't support for igt_runner on ChromeOS, they will be
> > > > > difficult to replicate and debug.
> > > > 
> > > > Actually, I wonder if it would be better to just treat
> > > > dmesg-warn/dmesg-fail as pass/fail?  I'd noticed some flakes on
> > > > rockchip which looked just like unrelated dmesg msg which just
> > > > happened to show up while the test was running.
> > > 
> > > This is kinda the reason behind standardizing on drm dmesg logging, so
> > > that we have some chances at filtering stuff out. Not sure that's a
> > > good idea, since when your entire box splats and lockdep is dead, then
> > > continuing to run drm tests is still fairly pointless.
> > 
> > I'm not sure if we are using it yet for drm-ci, but for mesa-ci we
> > monitor dmesg (over serial port, from the controller) for splats, so
> > we already have the tech for restarting or aborting the CI run.  We
> > don't need igt-runner to tell us.
> 
> Yep, these scripts are currently being used as-is from Mesa, so we got that
> functionality for free.
> 
> > > I think this is another reason why trying at least to standardize this
> > > stuff over drivers would be pretty good idea.
> > > 
> > > > Additionally, some of the tests, like msm_recovery, are *expected* to
> > > > generate some dmesg spam since they are intentionally triggering GPU
> > > > hangs to test the recovery mechanism.
> > > 
> > > Uh I don't like that. It just allows userspace to spam dmesg, which
> > > doesn't seem like a great idea. That's at least why i915 dumps these
> > > at a lower level, and in the past had a special "I'm going to whack
> > > the gpu real hard expect hangs" knob in debugfs.
> > > 
> > > Having tests which intentionally spam dmesg above info level isn't
> > > really good since then you need endless amounts of test-specific
> > > encoding of what is considered a success and what not. Like when a
> > > backmerge breaks a testcases which is already at dmesg-fail, is that
> > > bad or not? Probably bad, but was the situation before that really
> > > good or already kinda on fire?
> > 
> > I guess I could add some debugfs knobs to squelch the dmesg msgs on
> > gpu hangs.  In the normal case, I'd prefer that gpu hangs are not
> > silent.. since that is something we get in feedback reports if a user
> > (or dogfooder) reports a bug.
> > 
> > The rockchip case I mentioned was some unrelated dmesg about
> > linktraining failing.. presumably because there was no display
> > attached?  IDK, I didn't look too closely.  But my point is we could
> > be getting unrelated and asynchronous dmesg spam, even from other
> > kernel subsystems.  Letting that be part of the test results just
> > sounds like asking for flakes.
> 
> I think some drivers are currently a bit too buggy to behave reliably under
> CI unless one reduces coverage (rockchip on rk3399, for example). And some
> other drivers (in other subsystems as well) could do with a review of what
> they print to the console. I guess these are things we could and probably
> should fix?

I think the best way for these is to still run the tests, but supress the
results. At least if the flakiness is only in whether the test passes, and
not in the test run outright killing the kernel/machine somehow.

But yeah sometimes just having an explicit run list is the way to go, and
we have plenty of that in intel ci. But for i915 it's more a testament to
how terrible the i915-gem tests in particular have become, largely also
because people simply kept adding crap and never removed much at all (and
sometimes even removed important tests, oh well).
-Daniel

[RFC,v2] drm/msm: Add initial ci/ subdirectory

Commit Message

Comments

Patch