From patchwork Thu Jun 20 05:51:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Sangorrin X-Patchwork-Id: 11005893 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C20961580 for ; Thu, 20 Jun 2019 05:52:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B29E3284F1 for ; Thu, 20 Jun 2019 05:52:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 92406284E8; Thu, 20 Jun 2019 05:52:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 68989284F5 for ; Thu, 20 Jun 2019 05:52:04 +0000 (UTC) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 01A15C03; Thu, 20 Jun 2019 05:52:04 +0000 (UTC) X-Original-To: cip-dev@lists.cip-project.org Delivered-To: cip-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 88BA6B88 for ; Thu, 20 Jun 2019 05:52:02 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mo-csw.securemx.jp (mo-csw1514.securemx.jp [210.130.202.153]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 02D74108 for ; Thu, 20 Jun 2019 05:52:00 +0000 (UTC) Received: by mo-csw.securemx.jp (mx-mo-csw1514) id x5K5pl32015463; Thu, 20 Jun 2019 14:51:48 +0900 X-Iguazu-Qid: 34trw7wxXc1O0TPMo8 X-Iguazu-QSIG: v=2; s=0; t=1561009907; q=34trw7wxXc1O0TPMo8; m=NBdPguIMGUEqVBaLJlM9aikGybJBca+46/e8NELAfbc= Received: from imx2.toshiba.co.jp (imx2.toshiba.co.jp [106.186.93.51]) by relay.securemx.jp (mx-mr1513) id x5K5pkGw002320; Thu, 20 Jun 2019 14:51:47 +0900 Received: from enc01.localdomain ([106.186.93.100]) by imx2.toshiba.co.jp with ESMTP id x5K5pkbt029045; Thu, 20 Jun 2019 14:51:46 +0900 (JST) Received: from hop001.toshiba.co.jp ([133.199.164.63]) by enc01.localdomain with ESMTP id x5K5pkEJ003263; Thu, 20 Jun 2019 14:51:46 +0900 From: Daniel Sangorrin To: ben.hutchings@codethink.co.uk Date: Thu, 20 Jun 2019 14:51:43 +0900 X-TSB-HOP: ON Message-Id: <20190620055143.341-3-daniel.sangorrin@toshiba.co.jp> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190620055143.341-1-daniel.sangorrin@toshiba.co.jp> References: <20190620055143.341-1-daniel.sangorrin@toshiba.co.jp> Cc: cip-dev@lists.cip-project.org Subject: [cip-dev] [cip-kernel-sec][PATCH v2 3/3] remotes: check that remotes exist X-BeenThere: cip-dev@lists.cip-project.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: cip-dev-bounces@lists.cip-project.org Errors-To: cip-dev-bounces@lists.cip-project.org X-Virus-Scanned: ClamAV using ClamSMTP This patch introduces some checks for the local git repository containing the remotes configured in conf/remotes.yml These checks occur at the beginning of import_stable.py, report_affected.py and webview.py, to make sure that the local repository is consistent with the configuration yaml files. We also add a new script, prepare_remotes.py, that is useful to bootstrap the local git repository with the configured remote branches. Expert users may want to use their own, but the script is helpful for new users or a quickstart. Both the checks and the new script, take repository name mappings into account. Signed-off-by: Daniel Sangorrin --- README.md | 5 ++++ conf/remotes.yml | 3 +++ scripts/import_stable.py | 9 +++---- scripts/kernel_sec/branch.py | 42 ++++++++++++++++++++++++++++++++ scripts/prepare_remotes.py | 46 ++++++++++++++++++++++++++++++++++++ scripts/report_affected.py | 1 + scripts/webview.py | 1 + 7 files changed, 101 insertions(+), 6 deletions(-) create mode 100755 scripts/prepare_remotes.py diff --git a/README.md b/README.md index 4c5808f..576cc75 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,10 @@ this is assumed to be in `../kernel`, with remotes configured in stable and cip repositories. These can be overridden by command-line options or configuration (`~/.config/kernel-sec/remotes.yml`). +* `scripts/prepare_remotes.py` - creates the local git repository +and adds all configured remotes. You may prefer to skip this script +and configure the repository by hand. + * `scripts/import_debian.py` - import information from Debian's `kernel_sec` project. It includes all issues that Debian considers active or that are already tracked here. @@ -81,6 +85,7 @@ with the keys: branch from this remote. * `git_name`: (optional) The name actually used for this git remote, if it's different from the default. +* `git_repo_url`: URL of the remote git repository. ## Contributions diff --git a/conf/remotes.yml b/conf/remotes.yml index 51c523d..cfaa35f 100644 --- a/conf/remotes.yml +++ b/conf/remotes.yml @@ -1,6 +1,9 @@ torvalds: commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id= + git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git stable: commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit?id= + git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git cip: commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git/commit?id= + git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git diff --git a/scripts/import_stable.py b/scripts/import_stable.py index 2c88b07..194219d 100755 --- a/scripts/import_stable.py +++ b/scripts/import_stable.py @@ -31,11 +31,6 @@ BACKPORT_COMMIT_BOTTOM_RE = re.compile( .format(**RE_USE)) -def update(git_repo, remote_name): - subprocess.check_call(['git', 'remote', 'update', remote_name], - cwd=git_repo) - - def get_backports(git_repo, remotes, branches, debug=False): backports = {} @@ -140,7 +135,8 @@ def main(git_repo, remotes, debug=False): remote_names = set(branch['git_remote'] for branch in branches) for remote_name in remote_names: - update(git_repo, remotes[remote_name]['git_name']) + kernel_sec.branch.remote_update( + git_repo, remotes[remote_name]['git_name']) backports = get_backports(git_repo, remotes, branches, debug) c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches) @@ -190,4 +186,5 @@ if __name__ == '__main__': remotes = kernel_sec.branch.get_remotes(args.remote_name, mainline=args.mainline_remote_name, stable=args.stable_remote_name) + kernel_sec.branch.check_git_repo(args.git_repo, remotes) main(args.git_repo, remotes, args.debug) diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py index 10f3339..fb8785c 100644 --- a/scripts/kernel_sec/branch.py +++ b/scripts/kernel_sec/branch.py @@ -4,10 +4,12 @@ # Public License, Version 3 or later. See http://www.gnu.org/copyleft/gpl.html # for details. +import argparse import io import os import re import subprocess +import sys import time import urllib.error import urllib.request @@ -219,3 +221,43 @@ def get_remotes(mappings, mainline=None, stable=None): if stable: remotes['stable']['git_name'] = stable return remotes + + +def remote_update(git_repo, remote_name): + subprocess.check_call(['git', 'remote', 'update', remote_name], + cwd=git_repo) + + +def remote_add(git_repo, remote_name, remote_url): + subprocess.check_call(['git', 'remote', 'add', remote_name, remote_url], + cwd=git_repo) + + +def check_git_repo(git_repo, remotes, create=False): + if create: + if os.path.isdir(git_repo): + msg = "directory %r already exists" % git_repo + raise argparse.ArgumentTypeError(msg) + else: + os.mkdir(git_repo) + subprocess.check_call(['git', 'init', '.'], cwd=git_repo) + + for key in remotes.keys(): + remote = remotes[key] # __getitem__ will add git_name + remote_add(git_repo, remote['git_name'], remote['git_repo_url']) + else: + if not os.path.isdir(git_repo): + msg = "directory %r not present" % git_repo + raise argparse.ArgumentTypeError(msg) + if not os.path.isdir(os.path.join(git_repo, '.git')): + msg = "directory %r is not a git repository" % git_repo + raise argparse.ArgumentTypeError(msg) + + current_remotes = subprocess.check_output( + ['git', 'remote', 'show'], cwd=git_repo).decode( + sys.stdout.encoding).strip().split('\n') + for key in remotes.keys(): + remote = remotes[key] # __getitem__ will add git_name + if remote['git_name'] not in current_remotes: + msg = "remote %r not in git repository" % remote['git_name'] + raise argparse.ArgumentTypeError(msg) diff --git a/scripts/prepare_remotes.py b/scripts/prepare_remotes.py new file mode 100755 index 0000000..004519d --- /dev/null +++ b/scripts/prepare_remotes.py @@ -0,0 +1,46 @@ +#!/usr/bin/python3 + +# Copyright 2019 Toshiba corp. +# Based on import_stable.py by Codethink Ltd. +# +# This script is distributed under the terms and conditions of the GNU General +# Public License, Version 3 or later. See http://www.gnu.org/copyleft/gpl.html +# for details. + +# Helper script that prepares the local git repository with the configured +# remote branches + +import argparse + +import kernel_sec.branch + + +def main(git_repo, remotes): + kernel_sec.branch.check_git_repo(git_repo, remotes, create=True) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser( + description=('Prepare local git repository with configured remotes.')) + parser.add_argument('--git-repo', + dest='git_repo', default='../kernel', + help=('local git repository location ' + '(default: ../kernel)'), + metavar='DIRECTORY') + parser.add_argument('--remote-name', + dest='remote_name', action='append', default=[], + help='git remote name mappings, e.g. stable:korg-stable', + metavar='NAME:OTHER-NAME') + parser.add_argument('--mainline-remote', + dest='mainline_remote_name', + help="git remote name to use instead of 'torvalds'", + metavar='OTHER-NAME') + parser.add_argument('--stable-remote', + dest='stable_remote_name', + help="git remote name to use instead of 'stable'", + metavar='OTHER-NAME') + args = parser.parse_args() + remotes = kernel_sec.branch.get_remotes(args.remote_name, + mainline=args.mainline_remote_name, + stable=args.stable_remote_name) + main(args.git_repo, remotes) diff --git a/scripts/report_affected.py b/scripts/report_affected.py index 879c021..d2f1f22 100755 --- a/scripts/report_affected.py +++ b/scripts/report_affected.py @@ -100,5 +100,6 @@ if __name__ == '__main__': remotes = kernel_sec.branch.get_remotes(args.remote_name, mainline=args.mainline_remote_name, stable=args.stable_remote_name) + kernel_sec.branch.check_git_repo(args.git_repo, remotes) main(args.git_repo, remotes, args.only_fixed_upstream, args.include_ignored, *args.branches) diff --git a/scripts/webview.py b/scripts/webview.py index a3a643b..e4b36f6 100755 --- a/scripts/webview.py +++ b/scripts/webview.py @@ -219,6 +219,7 @@ if __name__ == '__main__': remotes = kernel_sec.branch.get_remotes(args.remote_name, mainline=args.mainline_remote_name, stable=args.stable_remote_name) + kernel_sec.branch.check_git_repo(git_repo, remotes) conf = { '/static/style.css': {