| Message ID | 20240416103207.2754778-1-shinichiro.kawasaki@wdc.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | support test case repeat by different conditions | expand |
On Tue, Apr 16, 2024 at 07:31:56PM +0900, Shin'ichiro Kawasaki wrote: > In the recent discussion for nvme test group [1], two pain points were mentioned > regarding the test case runs. > > 1) Several test cases in nvme test group do exactly the same test except the > NVME transport backend set up condition difference (device vs. file). This > results in duplicate test script codes. It is desired to unify the test cases > and run them repeatedly with the different conditions. > > 2) NVME transport types can be specified with nvme_trtype parameter so that the > same tests can be run for various transport types. However, some test cases > do not depend on the transport types. They are repeated in multiple runs for > the various transport types under the exact same conditions. It is desired to > repeat the test cases only when such repetition is required. > > [1] https://lore.kernel.org/linux-block/w2eaegjopbah5qbjsvpnrwln2t5dr7mv3v4n2e63m5tjqiochm@uonrjm2i2g72/ > > One idea to address these pain points is to add the test repeat feature to the > nvme test group. However, Daniel questioned if the feature could be implemented > in the blktests framework. Actually, a similar feature has already been > implemented to repeat some test cases for non-zoned block devices and zoned > block devices. However, this feature is implemented only for the zoned and non- > zoned device conditions. It can not fulfill the desires for nvme test group. > > This series proposes to generalize the feature in the blktests framework to > repeat the test cases with different conditions. Introduce a new function > set_conditions() that each test case can define and instruct the framework to > repeat the test case. The first four patches introduce the feature and apply it > to the repetition for non-zoned and zoned block devices. The following seven > patches apply the feature to nvme test group so that the test cases can be > repeated for NVME transport types and backend types in the ideal way. Two of the > seven patches are reused from the work by Daniel. The all patches are listed in > the order that does not lose the test coverage with the default set up. > > This series introduces new config parameters NVMET_TRTYPES and > NVMET_BLKDEV_TYPES, which can take multiple values with space separators. When > they are defined in the config file as follows, > > NVMET_TRTYPES="loop rdma tcp" > NVMET_BLKDEV_TYPES="device file" > > the test cases which depend on these parameters are repeated 3 x 2 = 6 times. > For example, nvme/006 is repeated as follows. > > nvme/006 (nvmet bd=device tr=loop) (create an NVMeOF target) [passed] > runtime 0.148s ... 0.165s > nvme/006 (nvmet bd=device tr=rdma) (create an NVMeOF target) [passed] > runtime 0.273s ... 0.235s > nvme/006 (nvmet bd=device tr=tcp) (create an NVMeOF target) [passed] > runtime 0.162s ... 0.164s > nvme/006 (nvmet bd=file tr=loop) (create an NVMeOF target) [passed] > runtime 0.138s ... 0.134s > nvme/006 (nvmet bd=file tr=rdma) (create an NVMeOF target) [passed] > runtime 0.216s ... 0.201s > nvme/006 (nvmet bd=file tr=tcp) (create an NVMeOF target) [passed] > runtime 0.154s ... 0.146s Looks all good to me. I also run the nvme part of blktests. The framework works fine but as expected we have a bunch of new errors reported because of the additional test coverage this change brings. Thanks a lot for your work! Reviewed-by: Daniel Wagner <dwagner@suse.de>
On 4/16/24 03:31, Shin'ichiro Kawasaki wrote: > In the recent discussion for nvme test group [1], two pain points were mentioned > regarding the test case runs. > > 1) Several test cases in nvme test group do exactly the same test except the > NVME transport backend set up condition difference (device vs. file). This > results in duplicate test script codes. It is desired to unify the test cases > and run them repeatedly with the different conditions. > > 2) NVME transport types can be specified with nvme_trtype parameter so that the > same tests can be run for various transport types. However, some test cases > do not depend on the transport types. They are repeated in multiple runs for > the various transport types under the exact same conditions. It is desired to > repeat the test cases only when such repetition is required. > > [1] https://lore.kernel.org/linux-block/w2eaegjopbah5qbjsvpnrwln2t5dr7mv3v4n2e63m5tjqiochm@uonrjm2i2g72/ > > One idea to address these pain points is to add the test repeat feature to the > nvme test group. However, Daniel questioned if the feature could be implemented > in the blktests framework. Actually, a similar feature has already been > implemented to repeat some test cases for non-zoned block devices and zoned > block devices. However, this feature is implemented only for the zoned and non- > zoned device conditions. It can not fulfill the desires for nvme test group. > > This series proposes to generalize the feature in the blktests framework to > repeat the test cases with different conditions. Introduce a new function > set_conditions() that each test case can define and instruct the framework to > repeat the test case. The first four patches introduce the feature and apply it > to the repetition for non-zoned and zoned block devices. The following seven > patches apply the feature to nvme test group so that the test cases can be > repeated for NVME transport types and backend types in the ideal way. Two of the > seven patches are reused from the work by Daniel. The all patches are listed in > the order that does not lose the test coverage with the default set up. > > This series introduces new config parameters NVMET_TRTYPES and > NVMET_BLKDEV_TYPES, which can take multiple values with space separators. When > they are defined in the config file as follows, > > NVMET_TRTYPES="loop rdma tcp" > NVMET_BLKDEV_TYPES="device file" > > the test cases which depend on these parameters are repeated 3 x 2 = 6 times. > For example, nvme/006 is repeated as follows. > > nvme/006 (nvmet bd=device tr=loop) (create an NVMeOF target) [passed] > runtime 0.148s ... 0.165s > nvme/006 (nvmet bd=device tr=rdma) (create an NVMeOF target) [passed] > runtime 0.273s ... 0.235s > nvme/006 (nvmet bd=device tr=tcp) (create an NVMeOF target) [passed] > runtime 0.162s ... 0.164s > nvme/006 (nvmet bd=file tr=loop) (create an NVMeOF target) [passed] > runtime 0.138s ... 0.134s > nvme/006 (nvmet bd=file tr=rdma) (create an NVMeOF target) [passed] > runtime 0.216s ... 0.201s > nvme/006 (nvmet bd=file tr=tcp) (create an NVMeOF target) [passed] > runtime 0.154s ... 0.146s > Thanks for doing this, looks good. Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com> -ck
On 16/04/2024 13:31, Shin'ichiro Kawasaki wrote: > In the recent discussion for nvme test group [1], two pain points were mentioned > regarding the test case runs. > > 1) Several test cases in nvme test group do exactly the same test except the > NVME transport backend set up condition difference (device vs. file). This > results in duplicate test script codes. It is desired to unify the test cases > and run them repeatedly with the different conditions. > > 2) NVME transport types can be specified with nvme_trtype parameter so that the > same tests can be run for various transport types. However, some test cases > do not depend on the transport types. They are repeated in multiple runs for > the various transport types under the exact same conditions. It is desired to > repeat the test cases only when such repetition is required. > > [1] https://lore.kernel.org/linux-block/w2eaegjopbah5qbjsvpnrwln2t5dr7mv3v4n2e63m5tjqiochm@uonrjm2i2g72/ > > One idea to address these pain points is to add the test repeat feature to the > nvme test group. However, Daniel questioned if the feature could be implemented > in the blktests framework. Actually, a similar feature has already been > implemented to repeat some test cases for non-zoned block devices and zoned > block devices. However, this feature is implemented only for the zoned and non- > zoned device conditions. It can not fulfill the desires for nvme test group. > > This series proposes to generalize the feature in the blktests framework to > repeat the test cases with different conditions. Introduce a new function > set_conditions() that each test case can define and instruct the framework to > repeat the test case. The first four patches introduce the feature and apply it > to the repetition for non-zoned and zoned block devices. The following seven > patches apply the feature to nvme test group so that the test cases can be > repeated for NVME transport types and backend types in the ideal way. Two of the > seven patches are reused from the work by Daniel. The all patches are listed in > the order that does not lose the test coverage with the default set up. Nice idea! > This series introduces new config parameters NVMET_TRTYPES and > NVMET_BLKDEV_TYPES, which can take multiple values with space separators. When > they are defined in the config file as follows, > > NVMET_TRTYPES="loop rdma tcp" > NVMET_BLKDEV_TYPES="device file" > > the test cases which depend on these parameters are repeated 3 x 2 = 6 times. > For example, nvme/006 is repeated as follows. > > nvme/006 (nvmet bd=device tr=loop) (create an NVMeOF target) [passed] > runtime 0.148s ... 0.165s > nvme/006 (nvmet bd=device tr=rdma) (create an NVMeOF target) [passed] > runtime 0.273s ... 0.235s > nvme/006 (nvmet bd=device tr=tcp) (create an NVMeOF target) [passed] > runtime 0.162s ... 0.164s > nvme/006 (nvmet bd=file tr=loop) (create an NVMeOF target) [passed] > runtime 0.138s ... 0.134s > nvme/006 (nvmet bd=file tr=rdma) (create an NVMeOF target) [passed] > runtime 0.216s ... 0.201s > nvme/006 (nvmet bd=file tr=tcp) (create an NVMeOF target) [passed] > runtime 0.154s ... 0.146s > > > Changes from v1: > * Renamed NVMET_TR_TYPES to NVMET_TRTYPES > * 1st patch: reflected comments on the list and added Reviewed-by tag > * 5th patch: changed NVMET_TRTYPES from array to variable > * 7th patch: changed NVMET_BLKDEV_TYPES from array to variable > * Reflected other comments on the list > > > Daniel Wagner (3): > nvme/rc: add blkdev type environment variable > nvme/{007,009,011,013,015,020,024}: drop duplicate nvmet blkdev type > tests > nvme/{021,022,025,026,027,028}: do not hard code target blkdev type > > Shin'ichiro Kawasaki (8): > check: factor out _run_test() > check: support test case repeat by different conditions > check: use set_conditions() for the CAN_BE_ZONED test cases > meta/{016,017}: add test cases to check repeated test case runs > nvme/rc: introduce NVMET_TRTYPES > nvme/rc: introduce NVMET_BLKDEV_TYPES > nvme/{002-031,033-038,040-045,047,048}: support NMVET_TRTYPES > nvme/{006,008,010,012,014,019,023}: support NVMET_BLKDEV_TYPES > > Documentation/running-tests.md | 16 +++- > Makefile | 3 +- > check | 129 ++++++++++++++++++++++----------- > common/shellcheck | 2 +- > common/zoned | 22 ++++++ > new | 21 ++++++ > tests/meta/016 | 29 ++++++++ > tests/meta/016.out | 2 + > tests/meta/017 | 29 ++++++++ > tests/meta/017.out | 2 + > tests/nvme/002 | 4 + > tests/nvme/003 | 4 + > tests/nvme/004 | 4 + > tests/nvme/005 | 4 + > tests/nvme/006 | 9 ++- > tests/nvme/007 | 28 ------- > tests/nvme/007.out | 2 - > tests/nvme/008 | 8 +- > tests/nvme/009 | 36 --------- > tests/nvme/009.out | 3 - > tests/nvme/010 | 8 +- > tests/nvme/011 | 39 ---------- > tests/nvme/011.out | 3 - > tests/nvme/012 | 8 +- > tests/nvme/013 | 43 ----------- > tests/nvme/013.out | 3 - > tests/nvme/014 | 8 +- > tests/nvme/015 | 48 ------------ > tests/nvme/015.out | 4 - > tests/nvme/016 | 4 + > tests/nvme/017 | 4 + > tests/nvme/018 | 4 + > tests/nvme/019 | 8 +- > tests/nvme/020 | 40 ---------- > tests/nvme/020.out | 4 - > tests/nvme/021 | 10 ++- > tests/nvme/022 | 10 ++- > tests/nvme/023 | 8 +- > tests/nvme/024 | 40 ---------- > tests/nvme/024.out | 2 - > tests/nvme/025 | 10 ++- > tests/nvme/026 | 10 ++- > tests/nvme/027 | 10 ++- > tests/nvme/028 | 10 ++- > tests/nvme/029 | 4 + > tests/nvme/030 | 4 + > tests/nvme/031 | 4 + > tests/nvme/033 | 4 + > tests/nvme/034 | 4 + > tests/nvme/035 | 4 + > tests/nvme/036 | 4 + > tests/nvme/037 | 4 + > tests/nvme/038 | 4 + > tests/nvme/040 | 4 + > tests/nvme/041 | 3 + > tests/nvme/042 | 3 + > tests/nvme/043 | 3 + > tests/nvme/044 | 3 + > tests/nvme/045 | 3 + > tests/nvme/047 | 4 + > tests/nvme/048 | 4 + > tests/nvme/rc | 58 ++++++++++++++- > 62 files changed, 437 insertions(+), 379 deletions(-) > create mode 100644 common/zoned > create mode 100755 tests/meta/016 > create mode 100644 tests/meta/016.out > create mode 100755 tests/meta/017 > create mode 100644 tests/meta/017.out > delete mode 100755 tests/nvme/007 > delete mode 100644 tests/nvme/007.out > delete mode 100755 tests/nvme/009 > delete mode 100644 tests/nvme/009.out > delete mode 100755 tests/nvme/011 > delete mode 100644 tests/nvme/011.out > delete mode 100755 tests/nvme/013 > delete mode 100644 tests/nvme/013.out > delete mode 100755 tests/nvme/015 > delete mode 100644 tests/nvme/015.out > delete mode 100755 tests/nvme/020 > delete mode 100644 tests/nvme/020.out > delete mode 100755 tests/nvme/024 > delete mode 100644 tests/nvme/024.out >
On 16/04/24 07:31PM, Shin'ichiro Kawasaki wrote: >In the recent discussion for nvme test group [1], two pain points were mentioned >regarding the test case runs. > >1) Several test cases in nvme test group do exactly the same test except the > NVME transport backend set up condition difference (device vs. file). This > results in duplicate test script codes. It is desired to unify the test cases > and run them repeatedly with the different conditions. > >2) NVME transport types can be specified with nvme_trtype parameter so that the > same tests can be run for various transport types. However, some test cases > do not depend on the transport types. They are repeated in multiple runs for > the various transport types under the exact same conditions. It is desired to > repeat the test cases only when such repetition is required. > >[1] https://lore.kernel.org/linux-block/w2eaegjopbah5qbjsvpnrwln2t5dr7mv3v4n2e63m5tjqiochm@uonrjm2i2g72/ > >One idea to address these pain points is to add the test repeat feature to the >nvme test group. However, Daniel questioned if the feature could be implemented >in the blktests framework. Actually, a similar feature has already been >implemented to repeat some test cases for non-zoned block devices and zoned >block devices. However, this feature is implemented only for the zoned and non- >zoned device conditions. It can not fulfill the desires for nvme test group. > >This series proposes to generalize the feature in the blktests framework to >repeat the test cases with different conditions. Introduce a new function >set_conditions() that each test case can define and instruct the framework to >repeat the test case. The first four patches introduce the feature and apply it >to the repetition for non-zoned and zoned block devices. The following seven >patches apply the feature to nvme test group so that the test cases can be >repeated for NVME transport types and backend types in the ideal way. Two of the >seven patches are reused from the work by Daniel. The all patches are listed in >the order that does not lose the test coverage with the default set up. > >This series introduces new config parameters NVMET_TRTYPES and >NVMET_BLKDEV_TYPES, which can take multiple values with space separators. When >they are defined in the config file as follows, > > NVMET_TRTYPES="loop rdma tcp" > NVMET_BLKDEV_TYPES="device file" > >the test cases which depend on these parameters are repeated 3 x 2 = 6 times. >For example, nvme/006 is repeated as follows. > >nvme/006 (nvmet bd=device tr=loop) (create an NVMeOF target) [passed] > runtime 0.148s ... 0.165s >nvme/006 (nvmet bd=device tr=rdma) (create an NVMeOF target) [passed] > runtime 0.273s ... 0.235s >nvme/006 (nvmet bd=device tr=tcp) (create an NVMeOF target) [passed] > runtime 0.162s ... 0.164s >nvme/006 (nvmet bd=file tr=loop) (create an NVMeOF target) [passed] > runtime 0.138s ... 0.134s >nvme/006 (nvmet bd=file tr=rdma) (create an NVMeOF target) [passed] > runtime 0.216s ... 0.201s >nvme/006 (nvmet bd=file tr=tcp) (create an NVMeOF target) [passed] > runtime 0.154s ... 0.146s > > >Changes from v1: >* Renamed NVMET_TR_TYPES to NVMET_TRTYPES >* 1st patch: reflected comments on the list and added Reviewed-by tag >* 5th patch: changed NVMET_TRTYPES from array to variable >* 7th patch: changed NVMET_BLKDEV_TYPES from array to variable >* Reflected other comments on the list > > >Daniel Wagner (3): > nvme/rc: add blkdev type environment variable > nvme/{007,009,011,013,015,020,024}: drop duplicate nvmet blkdev type > tests > nvme/{021,022,025,026,027,028}: do not hard code target blkdev type > >Shin'ichiro Kawasaki (8): > check: factor out _run_test() > check: support test case repeat by different conditions > check: use set_conditions() for the CAN_BE_ZONED test cases > meta/{016,017}: add test cases to check repeated test case runs > nvme/rc: introduce NVMET_TRTYPES > nvme/rc: introduce NVMET_BLKDEV_TYPES > nvme/{002-031,033-038,040-045,047,048}: support NMVET_TRTYPES > nvme/{006,008,010,012,014,019,023}: support NVMET_BLKDEV_TYPES > Acked-by: Nitesh Shetty <nj.shetty@samsung.com>
In the recent discussion for nvme test group [1], two pain points were mentioned regarding the test case runs. 1) Several test cases in nvme test group do exactly the same test except the NVME transport backend set up condition difference (device vs. file). This results in duplicate test script codes. It is desired to unify the test cases and run them repeatedly with the different conditions. 2) NVME transport types can be specified with nvme_trtype parameter so that the same tests can be run for various transport types. However, some test cases do not depend on the transport types. They are repeated in multiple runs for the various transport types under the exact same conditions. It is desired to repeat the test cases only when such repetition is required. [1] https://lore.kernel.org/linux-block/w2eaegjopbah5qbjsvpnrwln2t5dr7mv3v4n2e63m5tjqiochm@uonrjm2i2g72/ One idea to address these pain points is to add the test repeat feature to the nvme test group. However, Daniel questioned if the feature could be implemented in the blktests framework. Actually, a similar feature has already been implemented to repeat some test cases for non-zoned block devices and zoned block devices. However, this feature is implemented only for the zoned and non- zoned device conditions. It can not fulfill the desires for nvme test group. This series proposes to generalize the feature in the blktests framework to repeat the test cases with different conditions. Introduce a new function set_conditions() that each test case can define and instruct the framework to repeat the test case. The first four patches introduce the feature and apply it to the repetition for non-zoned and zoned block devices. The following seven patches apply the feature to nvme test group so that the test cases can be repeated for NVME transport types and backend types in the ideal way. Two of the seven patches are reused from the work by Daniel. The all patches are listed in the order that does not lose the test coverage with the default set up. This series introduces new config parameters NVMET_TRTYPES and NVMET_BLKDEV_TYPES, which can take multiple values with space separators. When they are defined in the config file as follows, NVMET_TRTYPES="loop rdma tcp" NVMET_BLKDEV_TYPES="device file" the test cases which depend on these parameters are repeated 3 x 2 = 6 times. For example, nvme/006 is repeated as follows. nvme/006 (nvmet bd=device tr=loop) (create an NVMeOF target) [passed] runtime 0.148s ... 0.165s nvme/006 (nvmet bd=device tr=rdma) (create an NVMeOF target) [passed] runtime 0.273s ... 0.235s nvme/006 (nvmet bd=device tr=tcp) (create an NVMeOF target) [passed] runtime 0.162s ... 0.164s nvme/006 (nvmet bd=file tr=loop) (create an NVMeOF target) [passed] runtime 0.138s ... 0.134s nvme/006 (nvmet bd=file tr=rdma) (create an NVMeOF target) [passed] runtime 0.216s ... 0.201s nvme/006 (nvmet bd=file tr=tcp) (create an NVMeOF target) [passed] runtime 0.154s ... 0.146s Changes from v1: * Renamed NVMET_TR_TYPES to NVMET_TRTYPES * 1st patch: reflected comments on the list and added Reviewed-by tag * 5th patch: changed NVMET_TRTYPES from array to variable * 7th patch: changed NVMET_BLKDEV_TYPES from array to variable * Reflected other comments on the list Daniel Wagner (3): nvme/rc: add blkdev type environment variable nvme/{007,009,011,013,015,020,024}: drop duplicate nvmet blkdev type tests nvme/{021,022,025,026,027,028}: do not hard code target blkdev type Shin'ichiro Kawasaki (8): check: factor out _run_test() check: support test case repeat by different conditions check: use set_conditions() for the CAN_BE_ZONED test cases meta/{016,017}: add test cases to check repeated test case runs nvme/rc: introduce NVMET_TRTYPES nvme/rc: introduce NVMET_BLKDEV_TYPES nvme/{002-031,033-038,040-045,047,048}: support NMVET_TRTYPES nvme/{006,008,010,012,014,019,023}: support NVMET_BLKDEV_TYPES Documentation/running-tests.md | 16 +++- Makefile | 3 +- check | 129 ++++++++++++++++++++++----------- common/shellcheck | 2 +- common/zoned | 22 ++++++ new | 21 ++++++ tests/meta/016 | 29 ++++++++ tests/meta/016.out | 2 + tests/meta/017 | 29 ++++++++ tests/meta/017.out | 2 + tests/nvme/002 | 4 + tests/nvme/003 | 4 + tests/nvme/004 | 4 + tests/nvme/005 | 4 + tests/nvme/006 | 9 ++- tests/nvme/007 | 28 ------- tests/nvme/007.out | 2 - tests/nvme/008 | 8 +- tests/nvme/009 | 36 --------- tests/nvme/009.out | 3 - tests/nvme/010 | 8 +- tests/nvme/011 | 39 ---------- tests/nvme/011.out | 3 - tests/nvme/012 | 8 +- tests/nvme/013 | 43 ----------- tests/nvme/013.out | 3 - tests/nvme/014 | 8 +- tests/nvme/015 | 48 ------------ tests/nvme/015.out | 4 - tests/nvme/016 | 4 + tests/nvme/017 | 4 + tests/nvme/018 | 4 + tests/nvme/019 | 8 +- tests/nvme/020 | 40 ---------- tests/nvme/020.out | 4 - tests/nvme/021 | 10 ++- tests/nvme/022 | 10 ++- tests/nvme/023 | 8 +- tests/nvme/024 | 40 ---------- tests/nvme/024.out | 2 - tests/nvme/025 | 10 ++- tests/nvme/026 | 10 ++- tests/nvme/027 | 10 ++- tests/nvme/028 | 10 ++- tests/nvme/029 | 4 + tests/nvme/030 | 4 + tests/nvme/031 | 4 + tests/nvme/033 | 4 + tests/nvme/034 | 4 + tests/nvme/035 | 4 + tests/nvme/036 | 4 + tests/nvme/037 | 4 + tests/nvme/038 | 4 + tests/nvme/040 | 4 + tests/nvme/041 | 3 + tests/nvme/042 | 3 + tests/nvme/043 | 3 + tests/nvme/044 | 3 + tests/nvme/045 | 3 + tests/nvme/047 | 4 + tests/nvme/048 | 4 + tests/nvme/rc | 58 ++++++++++++++- 62 files changed, 437 insertions(+), 379 deletions(-) create mode 100644 common/zoned create mode 100755 tests/meta/016 create mode 100644 tests/meta/016.out create mode 100755 tests/meta/017 create mode 100644 tests/meta/017.out delete mode 100755 tests/nvme/007 delete mode 100644 tests/nvme/007.out delete mode 100755 tests/nvme/009 delete mode 100644 tests/nvme/009.out delete mode 100755 tests/nvme/011 delete mode 100644 tests/nvme/011.out delete mode 100755 tests/nvme/013 delete mode 100644 tests/nvme/013.out delete mode 100755 tests/nvme/015 delete mode 100644 tests/nvme/015.out delete mode 100755 tests/nvme/020 delete mode 100644 tests/nvme/020.out delete mode 100755 tests/nvme/024 delete mode 100644 tests/nvme/024.out