From patchwork Thu May 2 17:42:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 10927431 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 38D6E1390 for ; Thu, 2 May 2019 17:55:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 27D6C1FF13 for ; Thu, 2 May 2019 17:55:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1C68A1FF87; Thu, 2 May 2019 17:55:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 784F11FF13 for ; Thu, 2 May 2019 17:55:15 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hMFtU-0006i1-3c; Thu, 02 May 2019 17:53:16 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hMFtS-0006hT-Ao for xen-devel@lists.xenproject.org; Thu, 02 May 2019 17:53:14 +0000 X-Inumbo-ID: 27ab44f2-6d03-11e9-843c-bc764e045a96 Received: from SMTP03.CITRIX.COM (unknown [162.221.156.55]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTPS id 27ab44f2-6d03-11e9-843c-bc764e045a96; Thu, 02 May 2019 17:53:13 +0000 (UTC) X-IronPort-AV: E=Sophos;i="5.60,422,1549929600"; d="scan'208";a="84983442" From: Ian Jackson To: Date: Thu, 2 May 2019 18:42:38 +0100 Message-ID: <20190502174238.23848-10-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190502174238.23848-1-ian.jackson@eu.citrix.com> References: <20190502174238.23848-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [OSSTEST PATCH 9/9] mg-repro-flight: Provide --rebuild to make variant build jobs too X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP This allows a single command to repro a particular job with a variety of different source code. The implementation technique is: - run the build job in a separate flight, so that it can run with a separate task which gives its host up after the build - do much of the heavy lifting of runvar fiddling etc. in a new helper routine in cs-adjust-flight Signed-off-by: Ian Jackson --- cs-adjust-flight | 109 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ mg-repro-setup | 84 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 192 insertions(+), 1 deletion(-) diff --git a/cs-adjust-flight b/cs-adjust-flight index cc1684b4..5be5af38 100755 --- a/cs-adjust-flight +++ b/cs-adjust-flight @@ -469,6 +469,115 @@ END $runvarq->execute($newjob, $dstflight, $oldjob); }) } + +sub change__repro_buildjobs { + # helper for mg-repro-setup + # arguments to this change are + # EXAMPLE-FLIGHT EXAMPLE-JOB THING... + # where THING is + # .BUILDJOB =BUILDJOBRUNVAR +TREE=URL#REVISION + # (we eat all until it doesn't start with . = +) + # stdout output is + # list of runvar settings for repro job + die unless @changes >= 2; + my $eflight = shift @changes; + my $ejob = shift @changes; + my @refspecs; + my @varspecs; + my %treespecs; + while (@changes && $changes[0] =~ m/^[.=+]/) { + local $_ = shift @changes; + if (m/^[.=]/) { + push @refspecs, $_; + } elsif (m/^\+(.*?)\=(.*)\#(.*)$/) { + die $1 if $treespecs{$1}; + $treespecs{$1} = [$2,$3]; + } elsif (m/^-r/) { + push @varspecs, $_; + } else { + die "bad .BUILDJOB / =BUILDJOBRUNVAR / +TREE=URL#REVISION): $_\n"; + } + } + my $testq = db_prepare(<execute($eflight, $ejob); + while (my ($refvar, $bjobref) = $testq->fetchrow_array()) { + my ($bflight, $bjob) = flight_otherjob($eflight,$bjobref); + if (@refspecs) { + my $y = 0; + foreach my $rs (@refspecs) { + if ($rs eq ".$bjob" || $rs eq "=$refvar") { + $y = 1; + last; + } + } + next unless $y; + } + $buildq->execute($bflight,$bjob,@buildq_xvars); + my %got; + while (my ($treevar) = $buildq->fetchrow_array()) { + $treevar =~ m/^tree_/ or die "$treevar ?"; + $got{$'} = 1; + next unless $treespecs{$'}; + $treespecs{$'}[2]++; + } + next unless %got || @refspecs; + + $bjobs{$bjob} //= { Template => $bflight }; + $bjobs{$bjob}{Template} eq $bflight or + die "inconsistent $bjob: $bjobs{$bjob}{Template} != $bflight"; + push @{ $bjobs{$bjob}{Refs} }, $refvar; + $bjobs{$bjob}{Trees}{$_} = 1 foreach keys %got; + } + + foreach my $tree (sort keys %treespecs) { + die "unused tree/revision adjustment $tree" + .(@refspecs + ? " (no tree_$tree var in any of " + .(join ' ', map { "$bjobs{$_}{Template}.$_" } + sort keys %bjobs).")" + : '') + unless $treespecs{$tree}[2]; + } + + my @copy_jobs_qs = copy_jobs_qs(); + foreach my $bjob (sort keys %bjobs) { + copy_jobs_do(\@copy_jobs_qs, $bjobs{$bjob}{Template}, $bjob); + foreach my $tree (sort keys %{ $bjobs{$bjob}{Trees} }) { + runvar_set($bjob, "tree_$tree", $treespecs{$tree}[0]); + runvar_set($bjob, "revision_$tree", $treespecs{$tree}[1]); + } + foreach (@varspecs) { + if (m/^-r!|^-r^/) { + $runvar_rm_q->execute($dstflight, $bjob, $'); + } elsif (m/^-r(.*?)=/) { + runvar_set($bjob, $1, $'); + } else { + die "bad -r $_ ?"; + } + } + foreach my $refvar (@{ $bjobs{$bjob}{Refs} }) { + die "$bjob $refvar $& ?" if + "$bjob.$refvar" =~ m{[^+=_./,:0-9a-z-]}i; + print "runvar-set . $refvar $dstflight.$bjob\n" or die $!; + } + } +} + sub changes () { debug("CHANGES...\n"); diff --git a/mg-repro-setup b/mg-repro-setup index 2e1d3b88..abff2ef7 100755 --- a/mg-repro-setup +++ b/mg-repro-setup @@ -19,7 +19,7 @@ usage () { cat <] [-r...] + [.BUILDJOB | =BUILDJOBRUNVAR...] + +TREE=URL#REVISION... + Also use a different build. Specifically, use URL and REVISION + for certain builds which mention TREE and which are referenced + (directly) in the job JOB in EXAMPLE-FLIGHT. The affected builds + are those referenced by any of the runvars BUILDJOBRUNVAR (in + which case only that job reference is edited) or any whose job + name is any of the BUILDJOB (in which case all references to that + job are adjusted). If no .BUILDJOB and no =BUILDJOBRUNVAR are + specified, then all jobs referred to from the example JOB + which mention TREE are affected. + + For example, + --rebuild =xenbuildjob \ + +xen=git://xenbits.xen.org/people/alice/xen.git#fixes + would look for xenbuild and (say) find that it referred to + build-amd64; it would then copy and use the build-amd64 job + that EXAMPLE-FLIGHT.JOB used, adjusting \`tree_xen' and + \`revision_xen' as specified, and use that for \`xenbuildjob' in + the repro (but not for \`buildjob') + + Whereas + --rebuild .build-amd64 \ + +linux=git://xenbits.xen.org/people/alice/linux.git#fixes + would replace all references to any job named build-amd64 + with a new build-amd64 job. + + And + --rebuild \ + +xen=git://xenbits.xen.org/people/alice/linux.git#fixes + would replace all builds mentioning xen, including for example + build-amd64 (for xenbuildjob) and build-i386 (for buildjob) + in an x86 32-on-64 flight. + + Host allocation for the build is done "normally" (ie, the host + is thrown away after the build is complete), with a default + blessing of \`adhoc'. + + --rebuild is affected by -E or -P but not general -B or -f. + END } @@ -122,10 +165,37 @@ progress "logging to $logfile" savelog "$logfile" exec 3>"$logfile" +rebuilds_flight='' + while [ $# -ne 0 ]; do arg=$1; shift case "$arg" in + + --rebuild) + rebuilds_blessing=adhoc + rebuild_specs=() + while true; do + case "$1" in + [.=+]*|-r) rebuild_specs+=("$1"); shift ;; + -B?*) rebuilds_blessing=${1#-B}; shift ;; + -*) badusage ': bad --rebuild option' ;; + *) break ;; + esac + done + if [ x$rebuilds_flight = x ]; then + rebuilds_flight=$( + ./cs-adjust-flight new:$rebuilds_blessing + ) + fi + + adjusts+=($( + ./cs-adjust-flight $rebuilds_flight \ + repro-buildjobs $example_flight $job \ + "${rebuild_specs[@]}" + )) + ;; + none:) # provided so we can repro a job with no hosts ;; @@ -170,6 +240,18 @@ done flight=$(./cs-adjust-flight new:$blessing) progress "new flight is $flight" +if [ "x$rebuilds_flight" != x ]; then + progress "running build(s) $rebuilds_flight" + ./mg-execute-flight -B$rebuilds_blessing --progress-fd=2 \ + "${mgexecflags[@]}" $rebuilds_flight + + mro=tmp/$rebuilds_flight.mro + if ! egrep '^perfect' >/dev/null $mro; then + echo >&2 "build(s) failed (no 'perfect' in $mro)" + exit 1 + fi +fi + OSSTEST_TASK=$(perl -e ' use Osstest; use Osstest::Executive;