[v2,0/2] Annonate fstests with possible reasons for failure

Message

Amir Goldstein April 19, 2022, 12:56 p.m. UTC

Zorro,

This series adresses your review comments on v1.
Specifically, I dropped the controversial patch #3 and the
SKIP_KNOWN_ISSUES run option.

To clarify, multiple annotations could be used in each test,
for example:
 _fixed_by_kernel_commit 09889695864 xfs: foo
 _fixed_by_kernel_commit 46464565465 ext4: bar
 _fixed_by_git_commit e2fsprogs 62464626 e2fs: baz
 _fixed_in_version xfsprogs v5.15

On test failure, all the hints will be printed to output as *possible*
reason for failure.

These annotation and printed hints are inspired by a similar LTP feature.

These annotation could be used by testers to auto-generate expunge list
for testing stable kernels from the information in _fixed_by_kernel_commit
and _fixed_in_kernel_version. This trivial task is left is an excercise.

Thanks,
Amir.

Changes since v1:
- Rename s/_known_issue_before_kernel/_fixed_in_kernel_version (zlang)
- Add generic helpers for non-kernel fixes (zlang)
- Drop _known_issue_on_fs (zlang)
- Clarify multiple hints are allowed (tytso)

Amir Goldstein (2):
  common: support black listing fs in _supported_fs()
  common: print hints for reasons of test failures

 check             | 10 ++++++++-
 common/preamble   |  2 +-
 common/rc         | 57 +++++++++++++++++++++++++++++++++++++++--------
 tests/generic/500 |  3 +--
 tests/generic/631 |  3 +--
 tests/generic/679 |  3 +--
 tests/overlay/009 |  3 +++
 tests/overlay/010 |  3 +++
 tests/overlay/014 |  2 ++
 tests/overlay/016 |  4 +++-
 tests/overlay/017 |  4 +++-
 tests/overlay/018 |  4 +++-
 tests/overlay/020 |  3 +++
 tests/overlay/022 |  2 ++
 tests/overlay/029 |  2 ++
 tests/overlay/038 |  2 ++
 tests/overlay/041 |  2 ++
 tests/overlay/042 |  3 +++
 tests/overlay/043 |  4 +++-
 tests/overlay/044 |  4 +++-
 tests/overlay/054 |  3 +++
 tests/overlay/055 |  3 +++
 tests/overlay/063 |  3 +++
 tests/overlay/065 |  4 ++++
 tests/overlay/067 |  3 +++
 tests/overlay/070 |  4 +++-
 tests/overlay/071 |  4 +++-
 tests/overlay/072 |  2 ++
 tests/overlay/074 |  5 +++++
 tests/overlay/077 |  5 +++++
 tests/overlay/078 |  1 +
 31 files changed, 133 insertions(+), 24 deletions(-)

Comments

Zorro Lang May 2, 2022, 4:04 a.m. UTC | #1

On Tue, Apr 19, 2022 at 03:56:35PM +0300, Amir Goldstein wrote:
> Zorro,
> 
> This series adresses your review comments on v1.
> Specifically, I dropped the controversial patch #3 and the
> SKIP_KNOWN_ISSUES run option.
> 
> To clarify, multiple annotations could be used in each test,
> for example:
>  _fixed_by_kernel_commit 09889695864 xfs: foo
>  _fixed_by_kernel_commit 46464565465 ext4: bar
>  _fixed_by_git_commit e2fsprogs 62464626 e2fs: baz
>  _fixed_in_version xfsprogs v5.15
> 
> On test failure, all the hints will be printed to output as *possible*
> reason for failure.
> 
> These annotation and printed hints are inspired by a similar LTP feature.
> 
> These annotation could be used by testers to auto-generate expunge list
> for testing stable kernels from the information in _fixed_by_kernel_commit
> and _fixed_in_kernel_version. This trivial task is left is an excercise.

I've reviewed V1 patchset, and keep V2 2 weeks here to get more review points.
if there's not objection or suggestion, I'll merge this change in next fstests
release.

Thanks,
Zorro

> 
> Thanks,
> Amir.
> 
> Changes since v1:
> - Rename s/_known_issue_before_kernel/_fixed_in_kernel_version (zlang)
> - Add generic helpers for non-kernel fixes (zlang)
> - Drop _known_issue_on_fs (zlang)
> - Clarify multiple hints are allowed (tytso)
> 
> Amir Goldstein (2):
>   common: support black listing fs in _supported_fs()
>   common: print hints for reasons of test failures
> 
>  check             | 10 ++++++++-
>  common/preamble   |  2 +-
>  common/rc         | 57 +++++++++++++++++++++++++++++++++++++++--------
>  tests/generic/500 |  3 +--
>  tests/generic/631 |  3 +--
>  tests/generic/679 |  3 +--
>  tests/overlay/009 |  3 +++
>  tests/overlay/010 |  3 +++
>  tests/overlay/014 |  2 ++
>  tests/overlay/016 |  4 +++-
>  tests/overlay/017 |  4 +++-
>  tests/overlay/018 |  4 +++-
>  tests/overlay/020 |  3 +++
>  tests/overlay/022 |  2 ++
>  tests/overlay/029 |  2 ++
>  tests/overlay/038 |  2 ++
>  tests/overlay/041 |  2 ++
>  tests/overlay/042 |  3 +++
>  tests/overlay/043 |  4 +++-
>  tests/overlay/044 |  4 +++-
>  tests/overlay/054 |  3 +++
>  tests/overlay/055 |  3 +++
>  tests/overlay/063 |  3 +++
>  tests/overlay/065 |  4 ++++
>  tests/overlay/067 |  3 +++
>  tests/overlay/070 |  4 +++-
>  tests/overlay/071 |  4 +++-
>  tests/overlay/072 |  2 ++
>  tests/overlay/074 |  5 +++++
>  tests/overlay/077 |  5 +++++
>  tests/overlay/078 |  1 +
>  31 files changed, 133 insertions(+), 24 deletions(-)
> 
> -- 
> 2.25.1
>