From patchwork Tue Feb 2 14:10:03 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Ian Campbell X-Patchwork-Id: 8190751 Return-Path: X-Original-To: patchwork-xen-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 5F7E7BEEE5 for ; Tue, 2 Feb 2016 14:13:10 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 5291C202EB for ; Tue, 2 Feb 2016 14:13:09 +0000 (UTC) Received: from lists.xen.org (lists.xenproject.org [50.57.142.19]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1D59A202E9 for ; Tue, 2 Feb 2016 14:13:08 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1aQbek-0006eR-Jr; Tue, 02 Feb 2016 14:10:10 +0000 Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1aQbei-0006e3-PU for xen-devel@lists.xensource.com; Tue, 02 Feb 2016 14:10:09 +0000 Received: from [193.109.254.147] by server-13.bemta-14.messagelabs.com id DA/1D-08347-FB8B0B65; Tue, 02 Feb 2016 14:10:07 +0000 X-Env-Sender: prvs=8337811db=Ian.Campbell@citrix.com X-Msg-Ref: server-7.tower-27.messagelabs.com!1454422205!20921115!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n, received_headers: No Received headers X-StarScan-Received: X-StarScan-Version: 7.35.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 46217 invoked from network); 2 Feb 2016 14:10:06 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-7.tower-27.messagelabs.com with RC4-SHA encrypted SMTP; 2 Feb 2016 14:10:06 -0000 X-IronPort-AV: E=Sophos;i="5.22,384,1449532800"; d="scan'208,223";a="335536163" Message-ID: <1454422203.28781.164.camel@citrix.com> From: Ian Campbell To: osstest service owner , , Ian Jackson Date: Tue, 2 Feb 2016 14:10:03 +0000 In-Reply-To: <1453896354.26591.10.camel@citrix.com> References: <1453893497.25257.97.camel@citrix.com> <1453896354.26591.10.camel@citrix.com> X-Mailer: Evolution 3.18.2-1 MIME-Version: 1.0 X-DLP: MIA2 Subject: Re: [Xen-devel] [linux-4.1 test] 79008: regressions - FAIL X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Wed, 2016-01-27 at 12:05 +0000, Ian Campbell wrote: > On Wed, 2016-01-27 at 11:18 +0000, Ian Campbell wrote: > > On Tue, 2016-01-26 at 13:11 +0000, osstest service owner wrote: > > > flight 79008 linux-4.1 real [real] > > > http://logs.test-lab.xenproject.org/osstest/logs/79008/ > > > > > > Regressions :-( > > > > > > Tests which did not succeed and are blocking, > > > including tests which could not be run: > > >  test-armhf-armhf-xl-credit2  15 guest-start/debian.repeat fail REGR. > > > vs. 66399 > > >  test-armhf-armhf-xl-xsm      15 guest-start/debian.repeat fail REGR. > > > vs. 66399 > > > > These were both: > > > > 2016-01-26 01:20:33 Z executing ssh ... root@172.16.147.101 echo guest > > debian.guest.osstest: ok  > > Warning: Permanently added '172.16.147.101' (ECDSA) to the list of > > known hosts. > > key_verify failed for server_host_key So I've narrowed this down a bit, but not yet sufficiently to actually diagnose. The issue only occurs when the userspace is Debian Jessie. Debian Wheezy does not, for some reason, expose this. It seems unlikely (although not impossible) to be a real issue in Jessie vs Wheezy, more likely some different behaviour in Jessie's sshd just exposes some issue somewhere else. When running Jessie userspace the issue only appeared somewhere between Linux v3.18 and v3.19, I'm currently looking at bisecting that range in case the commit which exposed the issue gives a hint (I fear it wont though). The attached ts-fetch-check-file exposes this pretty readily against either dom0 or domU (slightly differing symptoms) on effected versions. I can reproduce on the cubietruck on my desk as well as in the COLO. I cannot reproduce on the same cubietruck on my desk when running native 4.1, it only happens when running under Xen. I cannot reproduce on the arndale on my desk. I've not tried in the COLO since the test results suggest there would be no point -- there is no sign of this class of failure in the colo on arndale nor on any x86 box. Ian. From 337f663c13e46f815ce1f13b070b492f8d248b0c Mon Sep 17 00:00:00 2001 From: Ian Campbell Date: Tue, 2 Feb 2016 10:54:42 +0000 Subject: [PATCH] ts-fetch-check-file: new ts to fetch a file and check for corruption Compares a checksum computed on the target with one computed after cat'ting the file over ssh. Picks up on network corruption errors etc which might be missed with smaller interactions. Works for guests or hosts. To support this add a variant of target_cmd_output which returns the file descriptor instead of the actual data (which could be large), allowing us to pipe it to the local sum. Signed-off-by: Ian Campbell --- Osstest/TestSupport.pm | 10 ++++++-- ts-fetch-check-file | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+), 2 deletions(-) create mode 100755 ts-fetch-check-file diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm index 2141905..3c287b2 100644 --- a/Osstest/TestSupport.pm +++ b/Osstest/TestSupport.pm @@ -51,6 +51,7 @@ BEGIN { target_cmd_root target_cmd target_cmd_build target_cmd_output_root target_cmd_output + target_cmd_stdoutfd target_cmd_stdoutfd_root target_cmd_inputfh_root sshuho target_getfile target_getfile_root target_putfile target_putfile_root @@ -646,9 +647,11 @@ sub target_cmd ($$;$$) { tcmd(undef,undef,'osstest',@_); } sub target_cmd_root ($$;$$) { tcmd(undef,undef,'root',@_); } sub tcmdout { + my $wantfd = shift; my $stdout= IO::File::new_tmpfile(); tcmd(undef,$stdout,@_); $stdout->seek(0,0) or die "$stdout $!"; + return $stdout if $wantfd; my $r; { local ($/) = undef; $r= <$stdout>; } @@ -657,8 +660,11 @@ sub tcmdout { return $r; } -sub target_cmd_output ($$;$) { tcmdout('osstest',@_); } -sub target_cmd_output_root ($$;$) { tcmdout('root',@_); } +sub target_cmd_output ($$;$) { tcmdout(0,'osstest',@_); } +sub target_cmd_output_root ($$;$) { tcmdout(0,'root',@_); } + +sub target_cmd_stdoutfd ($$;$$) { tcmdout(1,'osstest',@_); } +sub target_cmd_stdoutfd_root ($$;$$) { tcmdout(1,'root',@_); } sub target_cmd_inputfh_root ($$$;$$) { my ($tho,$stdinfh,$tcmd,@rest) = @_; diff --git a/ts-fetch-check-file b/ts-fetch-check-file new file mode 100755 index 0000000..bfceb6b --- /dev/null +++ b/ts-fetch-check-file @@ -0,0 +1,68 @@ +#!/usr/bin/perl -w +# This is part of "osstest", an automated testing framework for Xen. +# Copyright (C) 2016 Citrix Inc. +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see . + +use strict qw(vars); +use DBI; +use Osstest; +use Osstest::TestSupport; + +use IO::Pipe; + +tsreadconfig(); + +our ($whhost,$guest) = @ARGV; +$whhost ||= 'host'; + +our ($ho,$gho); +our $fn = "/bin/bash"; # reasonable size, present in most guests and hosts + +$ho= selecthost($whhost); +$gho= selectguest($guest,$ho) if $guest; + +sub fetch_and_check_file ($$) { + my ($t,$fn) = @_; + + target_check_ip($t); + + target_cmd_root($t, "ls -lH $fn"); + + my $expect = target_cmd_output_root($t,"sum $fn"); + logm($expect); + + my $stdout= target_cmd_stdoutfd_root($t,"cat $fn",5,[qw(-v)]); + + my $pipe= IO::Pipe->new(); + my $child= fork; die $! unless defined $child; + if (!$child) { + $pipe->writer(); + open STDIN, "<&", $stdout or die "STDIN $!"; + open STDOUT, ">&", $pipe or die "STDOUT $!"; + exec("sum") or die "pipe writer $!"; + } + + $pipe->reader(); + + my $got = <$pipe>; + chomp($got); + + logm("expected $expect"); + logm("got $got"); + + die unless $expect eq $got; +} + +fetch_and_check_file($gho ? $gho : $ho, $fn); -- 2.6.1