From patchwork Thu Jan 18 17:59:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sai.Sathujoda@toshiba-tsip.com X-Patchwork-Id: 13523082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84623C47DAF for ; Thu, 18 Jan 2024 17:59:56 +0000 (UTC) Received: from mo-csw.securemx.jp (mo-csw.securemx.jp [210.130.202.152]) by mx.groups.io with SMTP id smtpd.web10.20567.1705600788084060852 for ; Thu, 18 Jan 2024 09:59:48 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: toshiba-tsip.com, ip: 210.130.202.152, mailfrom: sai.sathujoda@toshiba-tsip.com) Received: by mo-csw.securemx.jp (mx-mo-csw1802) id 40IHxkhO675931; Fri, 19 Jan 2024 02:59:46 +0900 X-Iguazu-Qid: 2yAaI9nTHBGfkzB1jW X-Iguazu-QSIG: v=2; s=0; t=1705600785; q=2yAaI9nTHBGfkzB1jW; m=Bh5EjLfdHPa1PGXsEzEiqTjVSJfni7zww/fZFzJjHtU= Received: from imx12-a.toshiba.co.jp ([38.106.60.135]) by relay.securemx.jp (mx-mr1802) id 40IHxjle3868984 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 19 Jan 2024 02:59:45 +0900 From: Sai.Sathujoda@toshiba-tsip.com To: cip-dev@lists.cip-project.org, jan.kiszka@siemens.com Cc: Sai Sathujoda , dinesh.kumar@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp Subject: [isar-cip-core v2 1/3] scripts/run-cve-checks.sh: Add script to generate CVE report Date: Thu, 18 Jan 2024 23:29:40 +0530 X-TSB-HOP2: ON Message-Id: <20240118175942.1052089-2-Sai.Sathujoda@toshiba-tsip.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240118175942.1052089-1-Sai.Sathujoda@toshiba-tsip.com> References: <20240118175942.1052089-1-Sai.Sathujoda@toshiba-tsip.com> MIME-Version: 1.0 X-OriginalArrivalTime: 18 Jan 2024 17:59:41.0773 (UTC) FILETIME=[1C8197D0:01DA4A38] List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 18 Jan 2024 17:59:56 -0000 X-Groupsio-URL: https://lists.cip-project.org/g/cip-dev/message/14401 From: Sai Sathujoda This script will extract latest dpkg-status files for all the deployed targets and generate their CVE reports using the cve_checker.py script in [1] and these report shall be uploaded back to cve-reports sub-directory under cip-project.org in the s3 bucket. [1] https://gitlab.com/cip-playground/debian-cve-checker Signed-off-by: Sai Sathujoda --- scripts/run-cve-checks.sh | 40 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 scripts/run-cve-checks.sh diff --git a/scripts/run-cve-checks.sh b/scripts/run-cve-checks.sh new file mode 100755 index 0000000..15a2bd8 --- /dev/null +++ b/scripts/run-cve-checks.sh @@ -0,0 +1,40 @@ +#!/bin/sh +# +# CIP Core, generic profile +# +# Copyright (c) Toshiba Corp., 2023 +# +# Authors: +# Daniel Sangorrin +# +# SPDX-License-Identifier: MIT +# + +# This script is used in .gitlab-ci.yml to create +# CVE reports in CSV format for each deployed build target. +# It uses the dpkg status files generated during the +# build stages and saved as gitlab-ci artifacts. + +set -e + +# Install AWS CLI +if ! which aws 2>&1 >/dev/null; then + echo "Installing awscli..." + apt update + apt install -y python3-wheel + apt install -y awscli +fi + +# Retrieve the latest dpkg status files from AWS +aws s3 cp --no-progress --recursive s3://download.cip-project.org/cip-core/cve-checks/dpkg-status/ ./ + +# Create new CVE reports +mkdir cve-reports +for i in *.dpkg_status; do + echo "Checking $i" + filename=${i%.dpkg_status} + cve_checker.py --status $i --output ./cve-reports/$filename.csv +done + +# Synchronize the CVE reports to AWS (it will delete old reports) +aws s3 sync --no-progress --delete --acl public-read cve-reports s3://download.cip-project.org/cip-core/cve-checks/cve-reports From patchwork Thu Jan 18 17:59:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sai.Sathujoda@toshiba-tsip.com X-Patchwork-Id: 13523080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82B45C47DB3 for ; Thu, 18 Jan 2024 17:59:56 +0000 (UTC) Received: from mo-csw.securemx.jp (mo-csw.securemx.jp [210.130.202.135]) by mx.groups.io with SMTP id smtpd.web11.20770.1705600788895109867 for ; Thu, 18 Jan 2024 09:59:49 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: toshiba-tsip.com, ip: 210.130.202.135, mailfrom: sai.sathujoda@toshiba-tsip.com) Received: by mo-csw.securemx.jp (mx-mo-csw1801) id 40IHxkSY672812; Fri, 19 Jan 2024 02:59:46 +0900 X-Iguazu-Qid: 2yAaI9nTHBGfkzB1jX X-Iguazu-QSIG: v=2; s=0; t=1705600785; q=2yAaI9nTHBGfkzB1jX; m=JCLZv73fR2LukKoKBngvG4NIuQqgNXxlaKg8H2GHES4= Received: from imx2-a.toshiba.co.jp (imx2-a.toshiba.co.jp [106.186.93.35]) by relay.securemx.jp (mx-mr1803) id 40IHxjZ13638379 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 19 Jan 2024 02:59:45 +0900 From: Sai.Sathujoda@toshiba-tsip.com To: cip-dev@lists.cip-project.org, jan.kiszka@siemens.com Cc: Sai Sathujoda , dinesh.kumar@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp Subject: [isar-cip-core v2 2/3] scripts/deploy-cip-core.sh: Upload dpkg-status files to gitlab CI artifacts Date: Thu, 18 Jan 2024 23:29:41 +0530 X-TSB-HOP2: ON Message-Id: <20240118175942.1052089-3-Sai.Sathujoda@toshiba-tsip.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240118175942.1052089-1-Sai.Sathujoda@toshiba-tsip.com> References: <20240118175942.1052089-1-Sai.Sathujoda@toshiba-tsip.com> MIME-Version: 1.0 X-OriginalArrivalTime: 18 Jan 2024 17:59:41.0867 (UTC) FILETIME=[1C8FEFB0:01DA4A38] List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 18 Jan 2024 17:59:56 -0000 X-Groupsio-URL: https://lists.cip-project.org/g/cip-dev/message/14403 From: Sai Sathujoda dpkg-status files are also uploaded along with other artifacts during deploy stage for marked targets since packages can be added or deleted from the system in the development phase which can alter the resultant cve-reports. Signed-off-by: Sai Sathujoda --- scripts/deploy-cip-core.sh | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/scripts/deploy-cip-core.sh b/scripts/deploy-cip-core.sh index 5855acb..d018341 100755 --- a/scripts/deploy-cip-core.sh +++ b/scripts/deploy-cip-core.sh @@ -53,3 +53,15 @@ else aws s3 cp --no-progress --acl public-read build/tmp/deploy/images/*/"$DTB" "${S3_TARGET}" fi fi + +# Deploy the dpkg status file (/var/lib/dpkg/status) to AWS for the CIP CVE checker (debian-cve-checker) +# to periodically extract and report pending CVEs. +# * CI builds will override the previous dpkg status files +# * AWS S3 bucket structure +# * download.cip-project.org/cip-core/cve-checks/ +# * dpkg-status: folder to store the dpkg status files generated by the CI and released images +# * cve-reports: folder to store the cve-reports generated from the dpkg status files +# * they can be linked from the release website +DPKG_STATUS="cip-core-image-*.dpkg_status" +DPKG_STATUS_FILENAME=${CI_JOB_NAME#build:}.dpkg_status +aws s3 cp --no-progress build/tmp/deploy/images/$TARGET/$DPKG_STATUS s3://download.cip-project.org/cip-core/cve-checks/dpkg-status/$DPKG_STATUS_FILENAME From patchwork Thu Jan 18 17:59:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sai.Sathujoda@toshiba-tsip.com X-Patchwork-Id: 13523079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FFF6C47DD6 for ; Thu, 18 Jan 2024 17:59:56 +0000 (UTC) Received: from mo-csw.securemx.jp (mo-csw.securemx.jp [210.130.202.135]) by mx.groups.io with SMTP id smtpd.web10.20568.1705600789376367132 for ; Thu, 18 Jan 2024 09:59:49 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: toshiba-tsip.com, ip: 210.130.202.135, mailfrom: sai.sathujoda@toshiba-tsip.com) Received: by mo-csw.securemx.jp (mx-mo-csw1801) id 40IHxlFM672856; Fri, 19 Jan 2024 02:59:48 +0900 X-Iguazu-Qid: 2yAa86qROgT3rA6aky X-Iguazu-QSIG: v=2; s=0; t=1705600787; q=2yAa86qROgT3rA6aky; m=mS+AE5UmBZ8cI4i0vwRj16vbSp5HPFUtCXypdLNun7M= Received: from imx12-a.toshiba.co.jp ([38.106.60.135]) by relay.securemx.jp (mx-mr1801) id 40IHxkpf3702282 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 19 Jan 2024 02:59:47 +0900 From: Sai.Sathujoda@toshiba-tsip.com To: cip-dev@lists.cip-project.org, jan.kiszka@siemens.com Cc: Sai Sathujoda , dinesh.kumar@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp Subject: [isar-cip-core v2 3/3] .gitlab-ci.yml: Run cve-checks job only when it is manually triggered in the pipeline Date: Thu, 18 Jan 2024 23:29:42 +0530 X-TSB-HOP2: ON Message-Id: <20240118175942.1052089-4-Sai.Sathujoda@toshiba-tsip.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240118175942.1052089-1-Sai.Sathujoda@toshiba-tsip.com> References: <20240118175942.1052089-1-Sai.Sathujoda@toshiba-tsip.com> MIME-Version: 1.0 X-OriginalArrivalTime: 18 Jan 2024 17:59:42.0008 (UTC) FILETIME=[1CA57380:01DA4A38] List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 18 Jan 2024 17:59:56 -0000 X-Groupsio-URL: https://lists.cip-project.org/g/cip-dev/message/14404 From: Sai Sathujoda The cve-checks job generates CVE reports from latest copy of dpkg status files using cve_checker.py script in debian-cve-checker. This job can only be triggered manually with no dependency on build jobs running in build stage. This dependency is removed so that one does not have to wait until all the jobs running in build stage are passed or is unable to run the cve-checks at all because some job has failed in the build stage. Signed-off-by: Sai Sathujoda --- .gitlab-ci.yml | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 1de6570..2527427 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -29,6 +29,7 @@ variables: stages: - build - test + - cve-check default: before_script: @@ -341,4 +342,17 @@ build:qemu-riscv64: deploy: disable allow_failure: true +cve-checks: + stage: cve-check + needs: [] + image: registry.gitlab.com/cip-playground/debian-cve-checker:latest + script: + - scripts/run-cve-checks.sh + when: manual + allow_failure: true + artifacts: + expire_in: 1 day + paths: + - cve-reports + include: '.reproducible-check-ci.yml'