diff mbox series

[XEN,3/4] automation: Add ECLAIR pipelines

Message ID 5e0b4afa229e29914392ffee736129f5ec2462cd.1690294965.git.simone.ballarin@bugseng.com (mailing list archive)
State Superseded
Headers show
Series automation: Add ECLAIR pipelines | expand

Commit Message

Simone Ballarin July 25, 2023, 2:55 p.m. UTC 
Add two pipelines that analyze an ARM64 and a X86_64 build with the
ECLAIR static analyzer on the guidelines contained in Set1.

The analysis configuration is stored in automation/eclair_analysis.

All commits on the xen-project/xen:staging branch will be analyzed
and their artifacts will be stored indefinitely; the integration will
report differential information with respect to the previous analysis.

All commits on other branches or repositories will be analyzed and
only the last ten artifacts will be kept; the integration will report
differential information with respect to the analysis done on the common
ancestor with xen-project/xen:staging (if available).

Currently the pipeline variable ENABLE_ECLAIR_BOT is set to "n".
Doing so disables the generation of comments with the analysis summary
on the commit threads. The variable can be set to "y" if the a masked
variable named ECLAIR_BOT_TOKEN is set with the impersonation token of
an account with enough privileges to write on all repositories.

Additionaly any repository should be able to read a masked variable
named WTOKEN with the token provided by BUGSENG.

The analysis fails if it contains violations of guidelines tagged as
clean:added. The list of clean guidelines are maintained in
automation/eclair_analysis/ECLAIR/tagging.ecl.

Signed-off-by: Simone Ballarin <simone.ballarin@bugseng.com>

--
Changes in v3:
- split definitions of the ECLAIR pipelines in a separate patch;
- if the WTOKEN variable is missing now the analysis fails immediately.

Changes in v2:
- add ECLAIR configuration files (before they were fetched from a separate
    repository);
- now the pipeline fails if there are new violations of guidelines tagged
    with clean:added.
---
 .gitlab-ci.yml                    |  2 ++
 automation/gitlab-ci/analyze.yaml | 38 +++++++++++++++++++++++++++++++
 automation/gitlab-ci/build.yaml   |  1 +
 automation/scripts/eclair         | 34 +++++++++++++++++++++++++++
 4 files changed, 75 insertions(+)
 create mode 100644 automation/gitlab-ci/analyze.yaml
 create mode 100755 automation/scripts/eclair

Comments

Stefano Stabellini July 25, 2023, 8:04 p.m. UTC | #1 
On Tue, 25 Jul 2023, Simone Ballarin wrote:
> Add two pipelines that analyze an ARM64 and a X86_64 build with the
> ECLAIR static analyzer on the guidelines contained in Set1.
> 
> The analysis configuration is stored in automation/eclair_analysis.
> 
> All commits on the xen-project/xen:staging branch will be analyzed
> and their artifacts will be stored indefinitely; the integration will
> report differential information with respect to the previous analysis.
> 
> All commits on other branches or repositories will be analyzed and
> only the last ten artifacts will be kept; the integration will report
> differential information with respect to the analysis done on the common
> ancestor with xen-project/xen:staging (if available).
> 
> Currently the pipeline variable ENABLE_ECLAIR_BOT is set to "n".
> Doing so disables the generation of comments with the analysis summary
> on the commit threads. The variable can be set to "y" if the a masked
> variable named ECLAIR_BOT_TOKEN is set with the impersonation token of
> an account with enough privileges to write on all repositories.
> 
> Additionaly any repository should be able to read a masked variable
> named WTOKEN with the token provided by BUGSENG.
> 
> The analysis fails if it contains violations of guidelines tagged as
> clean:added. The list of clean guidelines are maintained in
> automation/eclair_analysis/ECLAIR/tagging.ecl.
> 
> Signed-off-by: Simone Ballarin <simone.ballarin@bugseng.com>

This patch looks good to me, just one question before I give my
acked-by.


> --
> Changes in v3:
> - split definitions of the ECLAIR pipelines in a separate patch;
> - if the WTOKEN variable is missing now the analysis fails immediately.
> 
> Changes in v2:
> - add ECLAIR configuration files (before they were fetched from a separate
>     repository);
> - now the pipeline fails if there are new violations of guidelines tagged
>     with clean:added.
> ---
>  .gitlab-ci.yml                    |  2 ++
>  automation/gitlab-ci/analyze.yaml | 38 +++++++++++++++++++++++++++++++
>  automation/gitlab-ci/build.yaml   |  1 +
>  automation/scripts/eclair         | 34 +++++++++++++++++++++++++++
>  4 files changed, 75 insertions(+)
>  create mode 100644 automation/gitlab-ci/analyze.yaml
>  create mode 100755 automation/scripts/eclair
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index c8bd7519d5..ee5430b8b7 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,7 +1,9 @@
>  stages:
> +  - analyze
>    - build
>    - test
>  
>  include:
> +  - 'automation/gitlab-ci/analyze.yaml'
>    - 'automation/gitlab-ci/build.yaml'
>    - 'automation/gitlab-ci/test.yaml'
> diff --git a/automation/gitlab-ci/analyze.yaml b/automation/gitlab-ci/analyze.yaml
> new file mode 100644
> index 0000000000..3d8166572b
> --- /dev/null
> +++ b/automation/gitlab-ci/analyze.yaml
> @@ -0,0 +1,38 @@
> +.eclair-analysis:
> +  stage: analyze
> +  tags:
> +    - eclair-analysis
> +  variables:
> +    ECLAIR_OUTPUT_DIR: "ECLAIR_out"
> +    ANALYSIS_KIND: "normal"
> +    ENABLE_ECLAIR_BOT: "n"
> +    AUTO_PR_BRANCH: "staging"
> +    AUTO_PR_REPOSITORY: "xen-project/xen"
> +  artifacts:
> +    when: always
> +    paths:
> +      - "${ECLAIR_OUTPUT_DIR}/*.log"
> +      - "${ECLAIR_OUTPUT_DIR}/*.txt"
> +      - '*.log'
> +    reports:
> +      codequality: gl-code-quality-report.json

How do I access "gl-code-quality-report.json" or otherwise any other
meaningful ECLAIR output? If I browse the job artifacts I see all the
various logs but no gl-code-quality-report.json.

Scrolling up from the bottom of the job console output I see:

Browse analysis: https://saas.eclairit.com:3787/fs/var/local/eclair/xen-project.ecdf/xen-project/people/sstabellini/xen/ECLAIR_normal/ppp2/ARM64/4732041018/index.html

And if I click on the link, I can access a web interface with the
results. Is that the intended way to access the job output?

If so, would it be possible to print out the message "Browse
analysis:..." as the very last message to make it easier to spot? After
it at the moment I can see:

From https://gitlab.com:443/xen-project/xen
 * [new branch]            4.10.0-shim-comet   -> autoPRRemote/4.10.0-shim-comet
 [...]

The long list of branch names hides the "Browse analysis" link.


BTW I really like the graphics output, e.g.:
https://saas.eclairit.com:3787/fs/var/local/eclair/xen-project.ecdf/xen-project/people/sstabellini/xen/ECLAIR_normal/ppp2/ARM64/4732041018/PROJECT.ecd;/by_service.html#service/first_file&kind

Very nice and clear!


> +eclair-x86_64:
> +  extends: .eclair-analysis
> +  variables:
> +    LOGFILE: "eclair-x86_64.log"
> +    VARIANT: "X86_64"
> +    RULESET: "Set1"
> +  script:
> +    - ./automation/scripts/eclair 2>&1 | tee "${LOGFILE}"
> +  allow_failure: true
> +
> +eclair-ARM64:
> +  extends: .eclair-analysis
> +  variables:
> +    LOGFILE: "eclair-ARM64.log"
> +    VARIANT: "ARM64"
> +    RULESET: "Set1"
> +  script:
> +    - ./automation/scripts/eclair 2>&1 | tee "${LOGFILE}"
> +  allow_failure: true
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index c401f62d61..f01e2c32bb 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -11,6 +11,7 @@
>        - '*.log'
>        - '*/*.log'
>      when: always
> +  needs: []
>    except:
>      - master
>      - smoke
> diff --git a/automation/scripts/eclair b/automation/scripts/eclair
> new file mode 100755
> index 0000000000..55888617b3
> --- /dev/null
> +++ b/automation/scripts/eclair
> @@ -0,0 +1,34 @@
> +#!/bin/sh -eu
> +
> +ECLAIR_ANALYSIS_DIR=automation/eclair_analysis
> +ECLAIR_DIR="${ECLAIR_ANALYSIS_DIR}/ECLAIR"
> +ECLAIR_OUTPUT_DIR=$(realpath "${ECLAIR_OUTPUT_DIR}")
> +
> +if [ -z "${WTOKEN:-}" ]; then
> +    echo "Failure: the WTOKEN variable is not defined." >&2
> +    exit 1
> +fi
> +
> +"${ECLAIR_ANALYSIS_DIR}/prepare.sh" "${VARIANT}"
> +
> +ex=0
> +"${ECLAIR_DIR}/analyze.sh" "${VARIANT}" "${RULESET}" || ex=$?
> +"${ECLAIR_DIR}/action_log.sh" ANALYSIS_LOG \
> +                             "ECLAIR analysis log" \
> +                             "${ECLAIR_OUTPUT_DIR}/ANALYSIS.log" \
> +                             "${ex}"
> +"${ECLAIR_DIR}/action_log.sh" REPORT_LOG \
> +                             "ECLAIR report log" \
> +                             "${ECLAIR_OUTPUT_DIR}/REPORT.log" \
> +                             "${ex}"
> +[ "${ex}" = 0 ] || exit "${ex}"
> +"${ECLAIR_DIR}/action_push.sh" "${WTOKEN}" "${ECLAIR_OUTPUT_DIR}"
> +
> +# Fail in case of new reports
> +"${ECLAIR_DIR}/action_clean_added.sh" "${ECLAIR_OUTPUT_DIR}" || ex=$?
> +"${ECLAIR_DIR}/action_log.sh" DIFF_CHECK_LOG \
> +                             "ECLAIR diff check" \
> +                             "${ECLAIR_OUTPUT_DIR}/clean_added.log" \
> +                             "${ex}"
> +
> +[ "${ex}" = 0 ] || exit "${ex}"
> -- 
> 2.34.1
>
Simone Ballarin July 26, 2023, 12:34 a.m. UTC | #2 
Il giorno mar 25 lug 2023 alle ore 22:04 Stefano Stabellini <
sstabellini@kernel.org> ha scritto:

> How do I access "gl-code-quality-report.json" or otherwise any other
> meaningful ECLAIR output? If I browse the job artifacts I see all the
> various logs but no gl-code-quality-report.json.
>

gl-code-quality-report.json is a GitLab-specific artifact that GitLab
exploits
to provide some features called Code Quality (
https://docs.gitlab.com/ee/ci/testing/code_quality.html).
The file is not supposed to be used outside of the context of the Code
Quality
features.

ECLAIR can produce stand-alone artifacts in various formats and
we can decide to store some of them in the job artifacts (see
https://www.bugseng.com/eclair/reports for an exhaustive list).

Scrolling up from the bottom of the job console output I see:
>
> Browse analysis:
> https://saas.eclairit.com:3787/fs/var/local/eclair/xen-project.ecdf/xen-project/people/sstabellini/xen/ECLAIR_normal/ppp2/ARM64/4732041018/index.html
>
> And if I click on the link, I can access a web interface with the
> results. Is that the intended way to access the job output?
>

The link in the console is just a way to access the analysis results.
Typically the most
convenient one is the message written by the integration in the commit
thread,
see here an example:
https://eclairit.com:8444/swquality/eclair_demo/-/commit/0d312f8ebca6c4e98eabbeaf9b0fcb8b4a4344d9
.
To enable this feature you have to provide an impersonation token to the
integration,
you can find more information on the commit message.

If so, would it be possible to print out the message "Browse
> analysis:..." as the very last message to make it easier to spot? After
> it at the moment I can see:
>
> From https://gitlab.com:443/xen-project/xen
>  * [new branch]            4.10.0-shim-comet   ->
> autoPRRemote/4.10.0-shim-comet
>  [...]
>
> The long list of branch names hides the "Browse analysis" link.
>
> Ok. I will try also to remove the warnings.

>
> BTW I really like the graphics output, e.g.:
>
> https://saas.eclairit.com:3787/fs/var/local/eclair/xen-project.ecdf/xen-project/people/sstabellini/xen/ECLAIR_normal/ppp2/ARM64/4732041018/PROJECT.ecd;/by_service.html#service/first_file&kind
>
> Very nice and clear!
>
>
diff mbox series

Patch

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index c8bd7519d5..ee5430b8b7 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,7 +1,9 @@ 
 stages:
+  - analyze
   - build
   - test
 
 include:
+  - 'automation/gitlab-ci/analyze.yaml'
   - 'automation/gitlab-ci/build.yaml'
   - 'automation/gitlab-ci/test.yaml'
diff --git a/automation/gitlab-ci/analyze.yaml b/automation/gitlab-ci/analyze.yaml
new file mode 100644
index 0000000000..3d8166572b
--- /dev/null
+++ b/automation/gitlab-ci/analyze.yaml
@@ -0,0 +1,38 @@ 
+.eclair-analysis:
+  stage: analyze
+  tags:
+    - eclair-analysis
+  variables:
+    ECLAIR_OUTPUT_DIR: "ECLAIR_out"
+    ANALYSIS_KIND: "normal"
+    ENABLE_ECLAIR_BOT: "n"
+    AUTO_PR_BRANCH: "staging"
+    AUTO_PR_REPOSITORY: "xen-project/xen"
+  artifacts:
+    when: always
+    paths:
+      - "${ECLAIR_OUTPUT_DIR}/*.log"
+      - "${ECLAIR_OUTPUT_DIR}/*.txt"
+      - '*.log'
+    reports:
+      codequality: gl-code-quality-report.json
+
+eclair-x86_64:
+  extends: .eclair-analysis
+  variables:
+    LOGFILE: "eclair-x86_64.log"
+    VARIANT: "X86_64"
+    RULESET: "Set1"
+  script:
+    - ./automation/scripts/eclair 2>&1 | tee "${LOGFILE}"
+  allow_failure: true
+
+eclair-ARM64:
+  extends: .eclair-analysis
+  variables:
+    LOGFILE: "eclair-ARM64.log"
+    VARIANT: "ARM64"
+    RULESET: "Set1"
+  script:
+    - ./automation/scripts/eclair 2>&1 | tee "${LOGFILE}"
+  allow_failure: true
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index c401f62d61..f01e2c32bb 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -11,6 +11,7 @@ 
       - '*.log'
       - '*/*.log'
     when: always
+  needs: []
   except:
     - master
     - smoke
diff --git a/automation/scripts/eclair b/automation/scripts/eclair
new file mode 100755
index 0000000000..55888617b3
--- /dev/null
+++ b/automation/scripts/eclair
@@ -0,0 +1,34 @@ 
+#!/bin/sh -eu
+
+ECLAIR_ANALYSIS_DIR=automation/eclair_analysis
+ECLAIR_DIR="${ECLAIR_ANALYSIS_DIR}/ECLAIR"
+ECLAIR_OUTPUT_DIR=$(realpath "${ECLAIR_OUTPUT_DIR}")
+
+if [ -z "${WTOKEN:-}" ]; then
+    echo "Failure: the WTOKEN variable is not defined." >&2
+    exit 1
+fi
+
+"${ECLAIR_ANALYSIS_DIR}/prepare.sh" "${VARIANT}"
+
+ex=0
+"${ECLAIR_DIR}/analyze.sh" "${VARIANT}" "${RULESET}" || ex=$?
+"${ECLAIR_DIR}/action_log.sh" ANALYSIS_LOG \
+                             "ECLAIR analysis log" \
+                             "${ECLAIR_OUTPUT_DIR}/ANALYSIS.log" \
+                             "${ex}"
+"${ECLAIR_DIR}/action_log.sh" REPORT_LOG \
+                             "ECLAIR report log" \
+                             "${ECLAIR_OUTPUT_DIR}/REPORT.log" \
+                             "${ex}"
+[ "${ex}" = 0 ] || exit "${ex}"
+"${ECLAIR_DIR}/action_push.sh" "${WTOKEN}" "${ECLAIR_OUTPUT_DIR}"
+
+# Fail in case of new reports
+"${ECLAIR_DIR}/action_clean_added.sh" "${ECLAIR_OUTPUT_DIR}" || ex=$?
+"${ECLAIR_DIR}/action_log.sh" DIFF_CHECK_LOG \
+                             "ECLAIR diff check" \
+                             "${ECLAIR_OUTPUT_DIR}/clean_added.log" \
+                             "${ex}"
+
+[ "${ex}" = 0 ] || exit "${ex}"