| Message ID | 20181004151429.7232-7-crosa@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Acceptance Tests: basic architecture support | expand |
On 10/04/18 17:14, Cleber Rosa wrote: > One of the Avocado features relevant to virtualization testing is the > ability to reuse tests in different scenarios, known as variants. > This adds a JSON based variants file, that can be used to run most > tests in a number of different architectures. It can be run with: > > $ avocado run \ > --json-variants-load=tests/acceptance/variants/arch.json \ > --filter-by-tags='-x86_64' -- tests/acceptance/ > > Currently this covers 5 architectures, resulting in the execution > of 25 different combinations. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > tests/acceptance/variants/arch.json | 1 + > 1 file changed, 1 insertion(+) > create mode 100644 tests/acceptance/variants/arch.json > > diff --git a/tests/acceptance/variants/arch.json b/tests/acceptance/variants/arch.json > new file mode 100644 > index 0000000000..a7a2570553 > --- /dev/null > +++ b/tests/acceptance/variants/arch.json > @@ -0,0 +1 @@ > +[{"paths":["/run/*"],"variant":[["/run/aarch64",[["/run/aarch64", "arch", "aarch64"]]]],"variant_id": "aarch64"},{"paths":["/run/*"],"variant":[["/run/ppc",[["/run/ppc", "arch", "ppc"]]]],"variant_id": "ppc"},{"paths":["/run/*"],"variant":[["/run/ppc64",[["/run/ppc64", "arch", "ppc64"]]]],"variant_id": "ppc64"},{"paths":["/run/*"],"variant":[["/run/s390x",[["/run/s390x", "arch", "s390x"]]]],"variant_id": "s390x"},{"paths":["/run/*"],"variant":[["/run/x86_64",[["/run/x86_64", "arch", "x86_64"]]]],"variant_id": "x86_64"}] > (Side comment: you can parse json? :) That's awesome. Then I *am* tempted to suggest that Philippe's work parse the firmware metadata format, in the long run.) Laszlo
Hi Cleber, On 04/10/2018 17:14, Cleber Rosa wrote: > One of the Avocado features relevant to virtualization testing is the > ability to reuse tests in different scenarios, known as variants. > This adds a JSON based variants file, that can be used to run most > tests in a number of different architectures. It can be run with: > > $ avocado run \ > --json-variants-load=tests/acceptance/variants/arch.json \ > --filter-by-tags='-x86_64' -- tests/acceptance/ > > Currently this covers 5 architectures, resulting in the execution > of 25 different combinations. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > tests/acceptance/variants/arch.json | 1 + > 1 file changed, 1 insertion(+) > create mode 100644 tests/acceptance/variants/arch.json > > diff --git a/tests/acceptance/variants/arch.json b/tests/acceptance/variants/arch.json > new file mode 100644 > index 0000000000..a7a2570553 > --- /dev/null > +++ b/tests/acceptance/variants/arch.json > @@ -0,0 +1 @@ > +[{"paths":["/run/*"],"variant":[["/run/aarch64",[["/run/aarch64", "arch", "aarch64"]]]],"variant_id": "aarch64"},{"paths":["/run/*"],"variant":[["/run/ppc",[["/run/ppc", "arch", "ppc"]]]],"variant_id": "ppc"},{"paths":["/run/*"],"variant":[["/run/ppc64",[["/run/ppc64", "arch", "ppc64"]]]],"variant_id": "ppc64"},{"paths":["/run/*"],"variant":[["/run/s390x",[["/run/s390x", "arch", "s390x"]]]],"variant_id": "s390x"},{"paths":["/run/*"],"variant":[["/run/x86_64",[["/run/x86_64", "arch", "x86_64"]]]],"variant_id": "x86_64"}] > Is this generated? (thinking about the other archs supported). You should use some linter ;)
On 10/5/18 11:24 AM, Philippe Mathieu-Daudé wrote: > Hi Cleber, > > On 04/10/2018 17:14, Cleber Rosa wrote: >> One of the Avocado features relevant to virtualization testing is the >> ability to reuse tests in different scenarios, known as variants. >> This adds a JSON based variants file, that can be used to run most >> tests in a number of different architectures. It can be run with: >> >> $ avocado run \ >> --json-variants-load=tests/acceptance/variants/arch.json \ >> --filter-by-tags='-x86_64' -- tests/acceptance/ >> +++ b/tests/acceptance/variants/arch.json >> @@ -0,0 +1 @@ >> +[{"paths":["/run/*"],"variant":[["/run/aarch64",[["/run/aarch64", "arch", "aarch64"]]]],"variant_id": "aarch64"},{"paths":["/run/*"],"variant":[["/run/ppc",[["/run/ppc", "arch", "ppc"]]]],"variant_id": "ppc"},{"paths":["/run/*"],"variant":[["/run/ppc64",[["/run/ppc64", "arch", "ppc64"]]]],"variant_id": "ppc64"},{"paths":["/run/*"],"variant":[["/run/s390x",[["/run/s390x", "arch", "s390x"]]]],"variant_id": "s390x"},{"paths":["/run/*"],"variant":[["/run/x86_64",[["/run/x86_64", "arch", "x86_64"]]]],"variant_id": "x86_64"}] >> > > Is this generated? (thinking about the other archs supported). > > You should use some linter ;) Also, that's a long line, which will probably get longer as more support is added. Beyond 990 bytes, it starts risking problems with corruption over email. It's also hard to view what changes incrementally if the single line changes. Is there a way to pretty-print things across multiple lines, for shorter lines and easier reading of future diffs?
On 10/5/18 12:32 PM, Eric Blake wrote: > On 10/5/18 11:24 AM, Philippe Mathieu-Daudé wrote: >> Hi Cleber, >> >> On 04/10/2018 17:14, Cleber Rosa wrote: >>> One of the Avocado features relevant to virtualization testing is the >>> ability to reuse tests in different scenarios, known as variants. >>> This adds a JSON based variants file, that can be used to run most >>> tests in a number of different architectures. It can be run with: >>> >>> $ avocado run \ >>> --json-variants-load=tests/acceptance/variants/arch.json \ >>> --filter-by-tags='-x86_64' -- tests/acceptance/ > >>> +++ b/tests/acceptance/variants/arch.json >>> @@ -0,0 +1 @@ >>> +[{"paths":["/run/*"],"variant":[["/run/aarch64",[["/run/aarch64", >>> "arch", "aarch64"]]]],"variant_id": >>> "aarch64"},{"paths":["/run/*"],"variant":[["/run/ppc",[["/run/ppc", >>> "arch", "ppc"]]]],"variant_id": >>> "ppc"},{"paths":["/run/*"],"variant":[["/run/ppc64",[["/run/ppc64", >>> "arch", "ppc64"]]]],"variant_id": >>> "ppc64"},{"paths":["/run/*"],"variant":[["/run/s390x",[["/run/s390x", >>> "arch", "s390x"]]]],"variant_id": >>> "s390x"},{"paths":["/run/*"],"variant":[["/run/x86_64",[["/run/x86_64", >>> "arch", "x86_64"]]]],"variant_id": "x86_64"}] >>> >> >> Is this generated? (thinking about the other archs supported). >> It's generated and kept on every job result (jobdata/variants.json). Basically, you'd use any varianter plugin on a job, and then you can reuse the JSON generated on other jobs. TBH, I tweaked this one a bit. >> You should use some linter ;) I missed your point here... do you mean the style is not ideal? > > Also, that's a long line, which will probably get longer as more support > is added. Beyond 990 bytes, it starts risking problems with corruption > over email. It's also hard to view what changes incrementally if the > single line changes. Is there a way to pretty-print things across > multiple lines, for shorter lines and easier reading of future diffs? > Yes, good point. I'll pretty print it. Just a disclaimer: I've chosen to use a JSON variants because it's a core Avocado feature (doesn't require any external plugin), and the results are 100% reproducible (the variants are static). In the future, we may consider also shipping (and depending) on other variants. One idea that is being maturing (and prototype) is a native QEMU varianter. There's some info here: https://trello.com/c/qW4kMw50/32-guest-abi-machine-type-cpu-model-test-cases Regards, - Cleber.
On 05/10/2018 19:07, Cleber Rosa wrote: > On 10/5/18 12:32 PM, Eric Blake wrote: >> On 10/5/18 11:24 AM, Philippe Mathieu-Daudé wrote: >>> Hi Cleber, >>> >>> On 04/10/2018 17:14, Cleber Rosa wrote: >>>> One of the Avocado features relevant to virtualization testing is the >>>> ability to reuse tests in different scenarios, known as variants. >>>> This adds a JSON based variants file, that can be used to run most >>>> tests in a number of different architectures. It can be run with: >>>> >>>> $ avocado run \ >>>> --json-variants-load=tests/acceptance/variants/arch.json \ >>>> --filter-by-tags='-x86_64' -- tests/acceptance/ >> >>>> +++ b/tests/acceptance/variants/arch.json >>>> @@ -0,0 +1 @@ >>>> +[{"paths":["/run/*"],"variant":[["/run/aarch64",[["/run/aarch64", >>>> "arch", "aarch64"]]]],"variant_id": >>>> "aarch64"},{"paths":["/run/*"],"variant":[["/run/ppc",[["/run/ppc", >>>> "arch", "ppc"]]]],"variant_id": >>>> "ppc"},{"paths":["/run/*"],"variant":[["/run/ppc64",[["/run/ppc64", >>>> "arch", "ppc64"]]]],"variant_id": >>>> "ppc64"},{"paths":["/run/*"],"variant":[["/run/s390x",[["/run/s390x", >>>> "arch", "s390x"]]]],"variant_id": >>>> "s390x"},{"paths":["/run/*"],"variant":[["/run/x86_64",[["/run/x86_64", >>>> "arch", "x86_64"]]]],"variant_id": "x86_64"}] >>>> >>> >>> Is this generated? (thinking about the other archs supported). >>> > > It's generated and kept on every job result (jobdata/variants.json). > Basically, you'd use any varianter plugin on a job, and then you can > reuse the JSON generated on other jobs. > > TBH, I tweaked this one a bit. > >>> You should use some linter ;) > > I missed your point here... do you mean the style is not ideal? As meant "pretty printer" :) [ { "paths": [ "/run/*" ], "variant": [ [ "/run/aarch64", [ [ "/run/aarch64", "arch", "aarch64" ] ] ] ], "variant_id": "aarch64" }, ... But since it is generated I'd rather generate it... >> >> Also, that's a long line, which will probably get longer as more support >> is added. Beyond 990 bytes, it starts risking problems with corruption >> over email. It's also hard to view what changes incrementally if the >> single line changes. Is there a way to pretty-print things across >> multiple lines, for shorter lines and easier reading of future diffs? >> > > Yes, good point. I'll pretty print it. > > Just a disclaimer: I've chosen to use a JSON variants because it's a > core Avocado feature (doesn't require any external plugin), and the > results are 100% reproducible (the variants are static). In the future, > we may consider also shipping (and depending) on other variants. > > One idea that is being maturing (and prototype) is a native QEMU > varianter. There's some info here: > > https://trello.com/c/qW4kMw50/32-guest-abi-machine-type-cpu-model-test-cases > > Regards, > - Cleber. >
On 10/5/18 1:30 PM, Philippe Mathieu-Daudé wrote: > On 05/10/2018 19:07, Cleber Rosa wrote: >> On 10/5/18 12:32 PM, Eric Blake wrote: >>> On 10/5/18 11:24 AM, Philippe Mathieu-Daudé wrote: >>>> Hi Cleber, >>>> >>>> On 04/10/2018 17:14, Cleber Rosa wrote: >>>>> One of the Avocado features relevant to virtualization testing is the >>>>> ability to reuse tests in different scenarios, known as variants. >>>>> This adds a JSON based variants file, that can be used to run most >>>>> tests in a number of different architectures. It can be run with: >>>>> >>>>> $ avocado run \ >>>>> --json-variants-load=tests/acceptance/variants/arch.json \ >>>>> --filter-by-tags='-x86_64' -- tests/acceptance/ >>> >>>>> +++ b/tests/acceptance/variants/arch.json >>>>> @@ -0,0 +1 @@ >>>>> +[{"paths":["/run/*"],"variant":[["/run/aarch64",[["/run/aarch64", >>>>> "arch", "aarch64"]]]],"variant_id": >>>>> "aarch64"},{"paths":["/run/*"],"variant":[["/run/ppc",[["/run/ppc", >>>>> "arch", "ppc"]]]],"variant_id": >>>>> "ppc"},{"paths":["/run/*"],"variant":[["/run/ppc64",[["/run/ppc64", >>>>> "arch", "ppc64"]]]],"variant_id": >>>>> "ppc64"},{"paths":["/run/*"],"variant":[["/run/s390x",[["/run/s390x", >>>>> "arch", "s390x"]]]],"variant_id": >>>>> "s390x"},{"paths":["/run/*"],"variant":[["/run/x86_64",[["/run/x86_64", >>>>> "arch", "x86_64"]]]],"variant_id": "x86_64"}] >>>>> >>>> >>>> Is this generated? (thinking about the other archs supported). >>>> >> >> It's generated and kept on every job result (jobdata/variants.json). >> Basically, you'd use any varianter plugin on a job, and then you can >> reuse the JSON generated on other jobs. >> >> TBH, I tweaked this one a bit. >> >>>> You should use some linter ;) >> >> I missed your point here... do you mean the style is not ideal? > > As meant "pretty printer" :) > > [ > { > "paths": [ > "/run/*" > ], > "variant": [ > [ > "/run/aarch64", > [ > [ > "/run/aarch64", > "arch", > "aarch64" > ] > ] > ] > ], > "variant_id": "aarch64" > }, > ... > > But since it is generated I'd rather generate it... Actually, I think it's a good idea indeed to pretty print it, "python -m json.tool" should do the job. Thanks, - Cleber. > >>> >>> Also, that's a long line, which will probably get longer as more support >>> is added. Beyond 990 bytes, it starts risking problems with corruption >>> over email. It's also hard to view what changes incrementally if the >>> single line changes. Is there a way to pretty-print things across >>> multiple lines, for shorter lines and easier reading of future diffs? >>> >> >> Yes, good point. I'll pretty print it. >> >> Just a disclaimer: I've chosen to use a JSON variants because it's a >> core Avocado feature (doesn't require any external plugin), and the >> results are 100% reproducible (the variants are static). In the future, >> we may consider also shipping (and depending) on other variants. >> >> One idea that is being maturing (and prototype) is a native QEMU >> varianter. There's some info here: >> >> https://trello.com/c/qW4kMw50/32-guest-abi-machine-type-cpu-model-test-cases >> >> Regards, >> - Cleber. >>
diff --git a/tests/acceptance/variants/arch.json b/tests/acceptance/variants/arch.json new file mode 100644 index 0000000000..a7a2570553 --- /dev/null +++ b/tests/acceptance/variants/arch.json @@ -0,0 +1 @@ +[{"paths":["/run/*"],"variant":[["/run/aarch64",[["/run/aarch64", "arch", "aarch64"]]]],"variant_id": "aarch64"},{"paths":["/run/*"],"variant":[["/run/ppc",[["/run/ppc", "arch", "ppc"]]]],"variant_id": "ppc"},{"paths":["/run/*"],"variant":[["/run/ppc64",[["/run/ppc64", "arch", "ppc64"]]]],"variant_id": "ppc64"},{"paths":["/run/*"],"variant":[["/run/s390x",[["/run/s390x", "arch", "s390x"]]]],"variant_id": "s390x"},{"paths":["/run/*"],"variant":[["/run/x86_64",[["/run/x86_64", "arch", "x86_64"]]]],"variant_id": "x86_64"}]
One of the Avocado features relevant to virtualization testing is the ability to reuse tests in different scenarios, known as variants. This adds a JSON based variants file, that can be used to run most tests in a number of different architectures. It can be run with: $ avocado run \ --json-variants-load=tests/acceptance/variants/arch.json \ --filter-by-tags='-x86_64' -- tests/acceptance/ Currently this covers 5 architectures, resulting in the execution of 25 different combinations. Signed-off-by: Cleber Rosa <crosa@redhat.com> --- tests/acceptance/variants/arch.json | 1 + 1 file changed, 1 insertion(+) create mode 100644 tests/acceptance/variants/arch.json