From patchwork Thu Dec 21 12:04:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sai.Sathujoda@toshiba-tsip.com X-Patchwork-Id: 13501772 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75697C35274 for ; Thu, 21 Dec 2023 12:04:39 +0000 (UTC) Received: from mo-csw.securemx.jp (mo-csw.securemx.jp [210.130.202.134]) by mx.groups.io with SMTP id smtpd.web11.49476.1703160269539387852 for ; Thu, 21 Dec 2023 04:04:30 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: toshiba-tsip.com, ip: 210.130.202.134, mailfrom: sai.sathujoda@toshiba-tsip.com) Received: by mo-csw.securemx.jp (mx-mo-csw1800) id 3BLC4RJv1561226; Thu, 21 Dec 2023 21:04:27 +0900 X-Iguazu-Qid: 2yAao8uS6A6nkerYXY X-Iguazu-QSIG: v=2; s=0; t=1703160267; q=2yAao8uS6A6nkerYXY; m=EEmEyoqf97hTM63yKtPNL48aUVGZyvt5SWhcuZ/cr0A= Received: from imx2-a.toshiba.co.jp (imx2-a.toshiba.co.jp [106.186.93.35]) by relay.securemx.jp (mx-mr1803) id 3BLC4Quo1689984 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Thu, 21 Dec 2023 21:04:27 +0900 From: Sai.Sathujoda@toshiba-tsip.com To: cip-dev@lists.cip-project.org, jan.kiszka@siemens.com Cc: Sai Sathujoda , dinesh.kumar@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp Subject: [isar-cip-core 1/3] scripts/run-cve-checks.sh: Add script to generate CVE report Date: Thu, 21 Dec 2023 17:34:21 +0530 X-TSB-HOP2: ON Message-Id: <20231221120423.2388639-2-Sai.Sathujoda@toshiba-tsip.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20231221120423.2388639-1-Sai.Sathujoda@toshiba-tsip.com> References: <20231221120423.2388639-1-Sai.Sathujoda@toshiba-tsip.com> MIME-Version: 1.0 X-OriginalArrivalTime: 21 Dec 2023 12:04:23.0911 (UTC) FILETIME=[D680FF70:01DA3405] List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 21 Dec 2023 12:04:39 -0000 X-Groupsio-URL: https://lists.cip-project.org/g/cip-dev/message/14184 From: Sai Sathujoda This script will extract latest dpkg-status files for all the deployed targets and generate their CVE reports using the cve_checker.py script in [1] and these report shall be uploaded back to cve-reports sub-directory under cip-project.org in the s3 bucket. [1] https://gitlab.com/cip-playground/debian-cve-checker Signed-off-by: Sai Sathujoda --- scripts/run-cve-checks.sh | 40 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 scripts/run-cve-checks.sh diff --git a/scripts/run-cve-checks.sh b/scripts/run-cve-checks.sh new file mode 100755 index 0000000..b8da81d --- /dev/null +++ b/scripts/run-cve-checks.sh @@ -0,0 +1,40 @@ +#!/bin/sh +# +# CIP Core, generic profile +# +# Copyright (c) Toshiba Corp., 2023 +# +# Authors: +# Daniel Sangorrin +# +# SPDX-License-Identifier: MIT +# + +# This script is used in .gitlab-ci.yml to create +# CVE reports in CSV format for each deployed build target. +# It uses the dpkg status files generated during the +#build stages and saved as gitlab-ci artifacts. + +set -e + +# Install AWS CLI +if ! which aws 2>&1 >/dev/null; then + echo "Installing awscli..." + apt update + apt install -y python3-wheel + apt install -y awscli +fi + +# Retrieve the latest dpkg status files from AWS +aws s3 cp --no-progress --recursive s3://download.cip-project.org/cip-core/cve-checks/dpkg-status/ ./ + +# Create new CVE reports +mkdir cve-reports +for i in *.dpkg_status; do + echo "Checking $i" + filename=${i%.dpkg_status} + cve_checker.py --status $i --output ./cve-reports/$filename.csv +done + +# Synchronize the CVE reports to AWS (it will delete old reports) +aws s3 sync --no-progress --delete --acl public-read cve-reports s3://download.cip-project.org/cip-core/cve-checks/cve-reports From patchwork Thu Dec 21 12:04:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sai.Sathujoda@toshiba-tsip.com X-Patchwork-Id: 13501771 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76B0DC46CD8 for ; Thu, 21 Dec 2023 12:04:39 +0000 (UTC) Received: from mo-csw.securemx.jp (mo-csw.securemx.jp [210.130.202.152]) by mx.groups.io with SMTP id smtpd.web11.49478.1703160270875831414 for ; Thu, 21 Dec 2023 04:04:31 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: toshiba-tsip.com, ip: 210.130.202.152, mailfrom: sai.sathujoda@toshiba-tsip.com) Received: by mo-csw.securemx.jp (mx-mo-csw1802) id 3BLC4Trd1413258; Thu, 21 Dec 2023 21:04:29 +0900 X-Iguazu-Qid: 2yAaWUSdETorSG68UH X-Iguazu-QSIG: v=2; s=0; t=1703160268; q=2yAaWUSdETorSG68UH; m=BkdTIxxue0Ibp1l/lcgBYgObOfOhQwO9RqmdRI+SEo0= Received: from imx2-a.toshiba.co.jp (imx2-a.toshiba.co.jp [106.186.93.35]) by relay.securemx.jp (mx-mr1803) id 3BLC4RnH1690000 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Thu, 21 Dec 2023 21:04:28 +0900 From: Sai.Sathujoda@toshiba-tsip.com To: cip-dev@lists.cip-project.org, jan.kiszka@siemens.com Cc: Sai Sathujoda , dinesh.kumar@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp Subject: [isar-cip-core 2/3] scripts/deploy-cip-core.sh: Upload dpkg-status files to aws s3 bucket Date: Thu, 21 Dec 2023 17:34:22 +0530 X-TSB-HOP2: ON Message-Id: <20231221120423.2388639-3-Sai.Sathujoda@toshiba-tsip.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20231221120423.2388639-1-Sai.Sathujoda@toshiba-tsip.com> References: <20231221120423.2388639-1-Sai.Sathujoda@toshiba-tsip.com> MIME-Version: 1.0 X-OriginalArrivalTime: 21 Dec 2023 12:04:23.0989 (UTC) FILETIME=[D68CE650:01DA3405] List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 21 Dec 2023 12:04:39 -0000 X-Groupsio-URL: https://lists.cip-project.org/g/cip-dev/message/14186 From: Sai Sathujoda For all the deployed targets in CI, the dpkg-status file shall be uploaded to a respective directory [1] in aws s3 bucket. [1] s3://download.cip-project.org/cip-core/cve-checks/dpkg-status Signed-off-by: Sai Sathujoda --- scripts/deploy-cip-core.sh | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/scripts/deploy-cip-core.sh b/scripts/deploy-cip-core.sh index 5855acb..d018341 100755 --- a/scripts/deploy-cip-core.sh +++ b/scripts/deploy-cip-core.sh @@ -53,3 +53,15 @@ else aws s3 cp --no-progress --acl public-read build/tmp/deploy/images/*/"$DTB" "${S3_TARGET}" fi fi + +# Deploy the dpkg status file (/var/lib/dpkg/status) to AWS for the CIP CVE checker (debian-cve-checker) +# to periodically extract and report pending CVEs. +# * CI builds will override the previous dpkg status files +# * AWS S3 bucket structure +# * download.cip-project.org/cip-core/cve-checks/ +# * dpkg-status: folder to store the dpkg status files generated by the CI and released images +# * cve-reports: folder to store the cve-reports generated from the dpkg status files +# * they can be linked from the release website +DPKG_STATUS="cip-core-image-*.dpkg_status" +DPKG_STATUS_FILENAME=${CI_JOB_NAME#build:}.dpkg_status +aws s3 cp --no-progress build/tmp/deploy/images/$TARGET/$DPKG_STATUS s3://download.cip-project.org/cip-core/cve-checks/dpkg-status/$DPKG_STATUS_FILENAME From patchwork Thu Dec 21 12:04:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sai.Sathujoda@toshiba-tsip.com X-Patchwork-Id: 13501774 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 761A2C46CD4 for ; Thu, 21 Dec 2023 12:04:39 +0000 (UTC) Received: from mo-csw.securemx.jp (mo-csw.securemx.jp [210.130.202.132]) by mx.groups.io with SMTP id smtpd.web11.49479.1703160272765097407 for ; Thu, 21 Dec 2023 04:04:33 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: toshiba-tsip.com, ip: 210.130.202.132, mailfrom: sai.sathujoda@toshiba-tsip.com) Received: by mo-csw.securemx.jp (mx-mo-csw1121) id 3BLC4Uuw041571; Thu, 21 Dec 2023 21:04:31 +0900 X-Iguazu-Qid: 2rWgKwcnFBzk4wZMqC X-Iguazu-QSIG: v=2; s=0; t=1703160270; q=2rWgKwcnFBzk4wZMqC; m=Ex68g1XdMfIQ5Jdm3qqkK40zhOvBOUvpmCUosJ9wFvE= Received: from imx2-a.toshiba.co.jp (imx2-a.toshiba.co.jp [106.186.93.35]) by relay.securemx.jp (mx-mr1120) id 3BLC4Sx94120509 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Thu, 21 Dec 2023 21:04:30 +0900 From: Sai.Sathujoda@toshiba-tsip.com To: cip-dev@lists.cip-project.org, jan.kiszka@siemens.com Cc: Sai Sathujoda , dinesh.kumar@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp Subject: [isar-cip-core 3/3] .gitlab-ci.yml: Add cve-checks job which runs when a tag is pushed Date: Thu, 21 Dec 2023 17:34:23 +0530 X-TSB-HOP2: ON Message-Id: <20231221120423.2388639-4-Sai.Sathujoda@toshiba-tsip.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20231221120423.2388639-1-Sai.Sathujoda@toshiba-tsip.com> References: <20231221120423.2388639-1-Sai.Sathujoda@toshiba-tsip.com> MIME-Version: 1.0 X-OriginalArrivalTime: 21 Dec 2023 12:04:24.0130 (UTC) FILETIME=[D6A26A20:01DA3405] List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 21 Dec 2023 12:04:39 -0000 X-Groupsio-URL: https://lists.cip-project.org/g/cip-dev/message/14187 From: Sai Sathujoda The cve-checks job generate latest CVE reports using the cve_checker.py script in debian-cve-checker container. Signed-off-by: Sai Sathujoda --- .gitlab-ci.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 1de6570..b8422be 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -63,6 +63,13 @@ default: - if [ "${deploy}" = "enable" ]; then scripts/deploy-cip-core.sh ${release} ${target} ${extension} ${dtb} ${CI_COMMIT_REF_SLUG}; fi - if [ "${deploy_kernelci}" = "enable" ]; then scripts/deploy-kernelci.py ${release} ${target} ${extension} ${dtb}; fi +cve-checks: + image: registry.gitlab.com/cip-playground/debian-cve-checker:latest + script: + - scripts/run-cve-checks.sh + only: + - tags + # base image build:simatic-ipc227e-base: extends: