From patchwork Wed Aug 19 16:01:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11724545 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E3F6A1575 for ; Wed, 19 Aug 2020 16:03:34 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BFCA72078D for ; Wed, 19 Aug 2020 16:03:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="XbpfatPY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BFCA72078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k8QXT-000748-R2; Wed, 19 Aug 2020 16:02:11 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k8QXT-0006xV-63 for xen-devel@lists.xenproject.org; Wed, 19 Aug 2020 16:02:11 +0000 X-Inumbo-ID: cfd62f0e-1fcb-4bee-8444-59e20b3db30d Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id cfd62f0e-1fcb-4bee-8444-59e20b3db30d; Wed, 19 Aug 2020 16:02:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597852922; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=zWpQNW1nSTO5KEdMe+zYQIylPdBgmXTMUmQ9itT0K6Y=; b=XbpfatPYZmMYFF48252bcoWxqfkVQd80eaDtF2KIryUNk4gS027IePG7 NTrchS5vu4wtxWUsbxIFVjyiM64ZTmTiu/b9sEUMqvrtKy2hUc3WUp6L+ PRFqUv9gXtiauThE/KB1qZ2rsdZlw7QPWdQjKP2rkh0vyKt0pLiWLZFew Y=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: rmn/7TNe+olCT57QrSkYrJDRdN0+Xdjx2LYCuafupsdii/K/jzGgc7RsLE6o7d/THhkllbPDFt r2tpV7nJHufPRWufvOS8TtWgmgqxF7WFmPmUpZSQg1oKunF0Juwe+6maFU3eqAUC+hV0FM94P+ tv5bFHx4mJr8C1vsp3u6dY1Fhg7IFlooz0FMjLACOi5DRxgd0FhLBRpzxKQ71iREt3KH4JN8ur wfgxPp4R5/wrxI9KmcCjDa0Nmk6quZMql7W2+1xmMmC9WDeqeO7kGcAOtEF1o5biHJncD/EpRx Lq4= X-SBRS: 2.7 X-MesageID: 24885410 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,331,1592884800"; d="scan'208";a="24885410" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 1/7] ts-hosts-allocate-Executive: Fix broken call to $duration_estimator Date: Wed, 19 Aug 2020 17:01:37 +0100 Message-ID: <20200819160143.11178-2-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200819160143.11178-1-ian.jackson@eu.citrix.com> References: <20200819160143.11178-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" The debug subref is passed to the constructor (and indeed we do that). The final argument to the actual estimator is $uptoincl_testid (but we didn't say $will_uptoincl_testid, so it is ignored). The code was wrong, but with no effect. So no functional change. Signed-off-by: Ian Jackson --- ts-hosts-allocate-Executive | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive index 3bc38102..e5a6fbfe 100755 --- a/ts-hosts-allocate-Executive +++ b/ts-hosts-allocate-Executive @@ -516,10 +516,7 @@ sub find_recent_duration ($$) { ($candrow->{Duration}, $candrow->{MostRecentStarted}, $candrow->{MostRecentStatus}) = - $duration_estimator->($job, $hid->{Ident}, $candrow->{resname}, - sub { -# print DEBUG "$dbg DUR-EST @_\n"; - }); + $duration_estimator->($job, $hid->{Ident}, $candrow->{resname}); } From patchwork Wed Aug 19 16:01:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11724547 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E7FB3618 for ; Wed, 19 Aug 2020 16:03:51 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C41882078D for ; Wed, 19 Aug 2020 16:03:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="aVD91ZYk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C41882078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k8QXK-000704-PQ; Wed, 19 Aug 2020 16:02:02 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k8QXJ-0006xV-5Y for xen-devel@lists.xenproject.org; Wed, 19 Aug 2020 16:02:01 +0000 X-Inumbo-ID: 2720d963-099c-4660-9606-f6fff63917c7 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 2720d963-099c-4660-9606-f6fff63917c7; Wed, 19 Aug 2020 16:01:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597852920; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=X+7wgvczDPTfTwDjWG9+pu1n4jK7Ii4wbwsbXI7r9vQ=; b=aVD91ZYkFjxWLb5F9KCEoJPgjYo0V+ZYXdeDhtfwQTQhwFp82u2rDHXN RSKtnSKu+q0GhDI3wYgStz49QscFL66jfpajk4TTBRVV/BxaVJGXdPCxl sSNf6x2QsbD0uT19A6cEc+5Q3u/BSreaeOyQDCNluJm/ZrpaoibIC3JJn 8=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: v0FaXVvy1mbxHR0Z3z5WtampetuyWG/fWbcM3eay2agJQaRO2FP3PfNO3JFwJMPBWZ9gEL+fUy JdU8tWD9xrryVTMIc88jGAFy+2zLj9f5mSD6m5DYo/z8p9CjWxFKZQcRgnv60dwIW7S2qojkc9 Ln+tFDIZUCXr29yo410Mt67Ktk9XrFPAa/U4Nzhzr2ZHtAy5qk+PN+jRuAJ5HkZ3dNNZJ82b5T pMDmD2I0ihoLc22SLFF9gY53vMF4+YfxztnrOgGc2mh2eBdXuYf4ogg+7IesMqDc5fA7Lu/0oV 1LE= X-SBRS: 2.7 X-MesageID: 24885405 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,331,1592884800"; d="scan'208";a="24885405" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 2/7] resource allocation: Provide OSSTEST_ALLOC_FAKE_PLAN test facility Date: Wed, 19 Aug 2020 17:01:38 +0100 Message-ID: <20200819160143.11178-3-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200819160143.11178-1-ian.jackson@eu.citrix.com> References: <20200819160143.11178-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Set this variable (to a data-plan.final.pl, say) and it becomes possible to test host allocation programs without actually allocating anything and without engaging with the queue system. Signed-off-by: Ian Jackson --- Osstest/Executive.pm | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm index 0808202b..d6b2736b 100644 --- a/Osstest/Executive.pm +++ b/Osstest/Executive.pm @@ -767,6 +767,32 @@ sub alloc_resources { ? sub { print $debugfh @_, "\n" or die $!; } : sub { }; + my $fake_plan = $ENV{OSSTEST_ALLOC_FAKE_PLAN}; + if (defined $fake_plan) { + my $fake_data = do { + local $/ = undef; + open FAKEPLAN, "<", $ENV{OSSTEST_ALLOC_FAKE_PLAN} or die $!; + my $r = // die $!; + close FAKEPLAN; + $r; + }; + if ($fake_plan =~ m{\.pl$}) { + $fake_data = eval $fake_data; + } elsif ($fake_plan =~ m{\.json$}) { + $fake_data = from_json($fake_data); + } else { + die; + } + db_retry($flight,'running', $dbh_tests, [], sub { + logm("fake resourcecall.."); + my ($ok, $bookinglist) = $resourcecall->($fake_data, 1); + logm("fake resourcecall ok=$ok"); + $dbh_tests->rollback(); + exit $ok; + }); + die "unexpectedly left db_retry"; + } + my $set_info= sub { return if grep { !defined } @_; my @s; From patchwork Wed Aug 19 16:01:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11724553 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8650F15E4 for ; Wed, 19 Aug 2020 16:04:10 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 61DB82078D for ; Wed, 19 Aug 2020 16:04:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="a5FP4teY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 61DB82078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k8QXQ-000725-4G; Wed, 19 Aug 2020 16:02:08 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k8QXO-0006xV-5q for xen-devel@lists.xenproject.org; Wed, 19 Aug 2020 16:02:06 +0000 X-Inumbo-ID: 7cee65c1-dd86-477d-904f-5c33bd14d7a6 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 7cee65c1-dd86-477d-904f-5c33bd14d7a6; Wed, 19 Aug 2020 16:02:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597852921; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=Y+z6/GUp1gQAbh9BXTGZb0V2zD55Xmss4MsgKfc57wk=; b=a5FP4teYhCjXFmceRGc90WxDQ5Sr+FdH3dfcrAujkXY/c1QiQy+kgzXb f513wOlFhaDTE729ZuqVvFmWwrbd4hYxifjo21aajP/8+z+PLAdYvCFK+ OXkznTiDnKqy3V6+/dp48hLTN6izrqwUs1slz//8zNghHceVLYSYdtYAG g=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: fQuG5WlXHINzhafW331wQcamnzx+XwJ/o4tphoHXcExCRjb9NpUiMcitUV92Ay4jvtqyZIHxl5 rKj22LyXsWHfUzdw24ncyY2wlx/4tQij2Hd+u8EjNJxvSAcJkg1wJtthXjYcWWCdNNDWMkEj7d fgPUniF0Fe7MGhuGgEL29gX17kbCigpHWcGw3cFMVXqv/EVSgq2BBp2nwMW95qnnkym8VAHeiW yeQ4L6j+4MdQa8i7cZBweAmedbH9jJ2zCuDjYhYVDVnzAmTCJKc2ocT8HGrwN/LhabjtX4ACxw zCw= X-SBRS: 2.7 X-MesageID: 24885407 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,331,1592884800"; d="scan'208";a="24885407" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 3/7] ts-hosts-allocate-Executive: Do a pre-check Date: Wed, 19 Aug 2020 17:01:39 +0100 Message-ID: <20200819160143.11178-4-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200819160143.11178-1-ian.jackson@eu.citrix.com> References: <20200819160143.11178-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Call attempt_allocation with an empty plan and $mayalloc=0. In the usual case this will arrange to prime our memoisation caches before we get involved with the queueing system. It will also arrange for various errors to be reported sooner. Signed-off-by: Ian Jackson --- ts-hosts-allocate-Executive | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive index e5a6fbfe..4140b65c 100755 --- a/ts-hosts-allocate-Executive +++ b/ts-hosts-allocate-Executive @@ -692,6 +692,13 @@ sub alloc_hosts () { return; } + { + logm("pre-checking resources..."); + local $Osstest::TestSupport::logm_prefix = $logm_prefix.' (precheck)'; + my ($ok, $bookinglist) = attempt_allocation({}, 0); + die $ok if $ok>1 && $ok != $alloc_starved_r; + } + my $waitstartadjust= $jobinfo->{recipe} =~ m/build/ ? -10000 From patchwork Wed Aug 19 16:01:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11724551 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 34326618 for ; Wed, 19 Aug 2020 16:03:54 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 10C312078D for ; Wed, 19 Aug 2020 16:03:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="PwdbUM05" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 10C312078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k8QXF-0006yQ-Fx; Wed, 19 Aug 2020 16:01:57 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k8QXE-0006xV-5Y for xen-devel@lists.xenproject.org; Wed, 19 Aug 2020 16:01:56 +0000 X-Inumbo-ID: 7fab628c-1110-4506-83fa-2deeb3b05a26 Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 7fab628c-1110-4506-83fa-2deeb3b05a26; Wed, 19 Aug 2020 16:01:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597852911; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=BShF8h9p1rtWxGyCfOusoDWxVy7RyhYqWSNJAm1smZ0=; b=PwdbUM05edbP/8ei5dk58IzwE1DYjYD3GNQKA0MToYjwnfT82jVJkf2k ZGD7v/RaLftOhAfOxxU2KT3sXnYqI2ui43s7eJEGR+91CHK5Hu4e58xSO CyqAMufY6zanHnXsXuff7MU132pt45MTVvAVn0IaqzjJRy8SZIiEdR6k/ s=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: kR4+DsxdUAKyV3QuyttODNk15P5BJICEd4LYVvUg52Lf2JKP2ShdZp7LhGB62+EhZ0YRZS+Zn8 ZpcfYWEOzzDlJ3P28zHBElIK/GTYXAvhP11zLHsRmf8jmbQ7i2O8p9eUTCXFVtyQweGbSLR0Vw xZfW6BThqDAGNY6qCTokdW1qhqY9uZAqtxcHlDyjFFpA19cUzQHanFUfpoCTUGyfFgJjI3y/g0 Phes50Iht7uWLi7Zh1D1C1TKLgnS+AOzn6sqK+ebjJ7DpgBrKLpZL73rLdsuF2qIpyLi5uk1Mh pzk= X-SBRS: 2.7 X-MesageID: 25801621 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,331,1592884800"; d="scan'208";a="25801621" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 4/7] duration estimates: Memoise results Date: Wed, 19 Aug 2020 17:01:40 +0100 Message-ID: <20200819160143.11178-5-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200819160143.11178-1-ian.jackson@eu.citrix.com> References: <20200819160143.11178-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" The caller may provide a memoisation hash. If they don't we embed one in the estimator. The estimator contains a db statement handle so shouldn't be so long-lived that this gives significantly wrong answers. I am aiming this work at ts-hosts-allocate-Executive, but it is possible that this might speed up sg-report-flight. Signed-off-by: Ian Jackson --- Osstest/Executive.pm | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm index d6b2736b..50c84cc3 100644 --- a/Osstest/Executive.pm +++ b/Osstest/Executive.pm @@ -1164,8 +1164,8 @@ sub hostalloc_starvation_calculate_X ($$$) { #---------- duration estimator ---------- -sub duration_estimator ($$;$$) { - my ($branch, $blessing, $debug, $will_uptoincl_testid) = @_; +sub duration_estimator ($$;$$$) { + my ($branch, $blessing, $debug, $will_uptoincl_testid, $our_memo) = @_; # returns a function which you call like this # $durest->($job, $hostidname, $onhost [, $uptoincl_testid]) # and returns one of @@ -1269,9 +1269,15 @@ END my $recentflights_q= $prepare_combi->($recentflights_qtxt); my $duration_anyref_q= $prepare_combi->($duration_anyref_qtxt); + $our_memo //= { }; + return sub { my ($job, $hostidname, $onhost, $uptoincl_testid) = @_; + my $memokey = "$job $hostidname $onhost $uptoincl_testid"; + my $memo = $our_memo->{$memokey}; + return @$memo if $memo; + my @x_params; push @x_params, $uptoincl_testid if $will_uptoincl_testid; @@ -1319,7 +1325,9 @@ END } } - return ($duration_max, $refs->[0]{started}, $refs->[0]{status}); + $memo = [$duration_max, $refs->[0]{started}, $refs->[0]{status}]; + $our_memo->{$memokey} = $memo; + return @$memo; }; } From patchwork Wed Aug 19 16:01:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11724543 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 38C0D618 for ; Wed, 19 Aug 2020 16:02:31 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 122542078D for ; Wed, 19 Aug 2020 16:02:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="TiXvFs1r" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 122542078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k8QXQ-00072S-FV; Wed, 19 Aug 2020 16:02:08 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k8QXP-0006yd-Gs for xen-devel@lists.xenproject.org; Wed, 19 Aug 2020 16:02:07 +0000 X-Inumbo-ID: dc685f10-8370-42b2-8454-55ebff3203f5 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id dc685f10-8370-42b2-8454-55ebff3203f5; Wed, 19 Aug 2020 16:02:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597852922; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=PTQfIV4yQE1BF9Rx3GBSziCx/l3gtDRWZCafZEHerYw=; b=TiXvFs1rOvrPW87eXOdy2IZ58p8e55O7PHOQ4liyFezY7PEu5P/4GyJL iu4mCcYh0cz4yxicUXOpzlIWOCNHq2bejqenPGf0kvqQnLirgzLyXh64I N+xgBm3spTSIVejvPDrWy9Dcd5G8eMTPSpg1DFi4RddkUjXhYVSVRIg7A E=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: TjgKA2w+4zWZHVB7/rNGYO/TzivonIAnv1AH57q/2LClQrAdc2pkpqxumKLkQNCb4swea5A+Ab jqoDR+eyXj2Lc9tq5knQpCLcFPgSg+owIDrUlqXCTH10aQcvM+eoVoL7iHxG1Mk6PQ0RHuFSs/ syNgDq5OKuXr42LXXGrANQxeok91EShJeEckz/5AiIalx0lUHJ5Vq2lHh7OmfOtbyf61XvEqJk r+Hm3yD/3NAMy9NQPJjDFweT7qnwhkQ6MeOb/4zxj6c6md0deninG2mISZOzPnzMR/jixKeDYF dxU= X-SBRS: 2.7 X-MesageID: 24885409 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,331,1592884800"; d="scan'208";a="24885409" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 5/7] host allocation: Memoise duration estimates Date: Wed, 19 Aug 2020 17:01:41 +0100 Message-ID: <20200819160143.11178-6-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200819160143.11178-1-ian.jackson@eu.citrix.com> References: <20200819160143.11178-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" We look at our own branch to estimate durations. If somehow we are one of multiple concurrent flights on this branch with the appropriate blessing, we don't mind not noticing the doing of our peer flights so that if our estimates are a bit out of date. So it is fine to use an estimate no older than our own runtime. Right now we generate a new duration estimator during each queueing round, because it contains a statement handle and we must disconnect from the db while waiting. So the internal memo table gets thrown away each time and is useless. To actually memoise, pass our own hash which lives as long as we do. Signed-off-by: Ian Jackson --- Osstest/Executive.pm | 2 +- ts-hosts-allocate-Executive | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm index 50c84cc3..61a99bc3 100644 --- a/Osstest/Executive.pm +++ b/Osstest/Executive.pm @@ -1274,7 +1274,7 @@ END return sub { my ($job, $hostidname, $onhost, $uptoincl_testid) = @_; - my $memokey = "$job $hostidname $onhost $uptoincl_testid"; + my $memokey = "$job $hostidname $onhost ".($uptoincl_testid//""); my $memo = $our_memo->{$memokey}; return @$memo if $memo; diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive index 4140b65c..39c66346 100755 --- a/ts-hosts-allocate-Executive +++ b/ts-hosts-allocate-Executive @@ -145,8 +145,10 @@ END AND hostflag LIKE 'equiv-%' END + our %duration_memo; $duration_estimator= duration_estimator($fi->{branch}, $blessing, - sub { print DEBUG "@_\n"; }); + sub { print DEBUG "@_\n"; }, + 0, \%duration_memo); $resprop_q= $dbh_tests->prepare(< X-Patchwork-Id: 11724541 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DA838618 for ; Wed, 19 Aug 2020 16:02:21 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B63F42078D for ; Wed, 19 Aug 2020 16:02:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="B3Wgn9rV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B63F42078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k8QXC-0006xl-3I; Wed, 19 Aug 2020 16:01:54 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k8QXA-0006xa-Iy for xen-devel@lists.xenproject.org; Wed, 19 Aug 2020 16:01:52 +0000 X-Inumbo-ID: bd897be7-b67c-49ad-9c2e-445e2509e80f Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id bd897be7-b67c-49ad-9c2e-445e2509e80f; Wed, 19 Aug 2020 16:01:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597852911; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=yp2aFFOJUnSEDY2GZVR4R5A/kjt4Cn3oeWksIf2jOmc=; b=B3Wgn9rVaymYFIwwa5o3Ckbw6e3ifmx7UVdtzf/7YeY1c8uABNwF4ajH ogX4rMQNP2xLxoywb46C0lSawm35wDb2O3ANHUpVY3wYlkYrDIeH5GBVq ngSwRSGnhfhwSH1hnGZQXvxztu/GoLX7+bkpNK5WiiHDDHGvsqA4vngRp A=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: KCXr+Q+fHp8u1JDcNlXp7WyrcfoYrQp6twSLPRGRXZbUC9I/8pQfNcjFO2LNpfY69QZQCk4uD8 iM34qNNEXVtiIqwX1ZX2FzC4hJviphnEkoJRps+S5b55BCOaYZup68ZyoEzpIi99pMGBBQvQJ2 lhgdZmkWpYnBV05m7zNbAyewOMO9hyvvsaVmqEUX+h4nhzvRt8tCOjprylplKwQ4/Q+MnM6FSp s01neUlnJvVjlKiYmr8cmQJiF9KkwQjX1qMAAMgcen1X+L0hVBI9nuqCct7YrwFE5JTN/ywkOf R/4= X-SBRS: 2.7 X-MesageID: 25801622 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,331,1592884800"; d="scan'208";a="25801622" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 6/7] host allocation: Memoise $equivstatus query results Date: Wed, 19 Aug 2020 17:01:42 +0100 Message-ID: <20200819160143.11178-7-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200819160143.11178-1-ian.jackson@eu.citrix.com> References: <20200819160143.11178-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" This provides a very significant speedup. Signed-off-by: Ian Jackson --- ts-hosts-allocate-Executive | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive index 39c66346..a47bc499 100755 --- a/ts-hosts-allocate-Executive +++ b/ts-hosts-allocate-Executive @@ -467,9 +467,13 @@ END find_recent_duration($dbg,$hid,$candrow); if ($candrow->{restype} eq 'host') { - $equivstatusq->execute($job,$blessing,$fi->{branch}, - $hid->{Ident},$candrow->{resname}); - my $esrow = $equivstatusq->fetchrow_hashref(); + our %equivstatus_memo; + my @params = ($job,$blessing,$fi->{branch}, + $hid->{Ident},$candrow->{resname}); + my $esrow = $equivstatus_memo{"@params"} //= do { + $equivstatusq->execute(@params); + $equivstatusq->fetchrow_hashref() // { }; + }; $candrow->{EquivMostRecentStatus} = $esrow->{status}; print DEBUG "$dbg EQUIV-MOST-RECENT "; print DEBUG ("$esrow->{flight}.$esrow->{job}". From patchwork Wed Aug 19 16:01:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11724549 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 75DC1618 for ; Wed, 19 Aug 2020 16:03:53 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 52BFF2078D for ; Wed, 19 Aug 2020 16:03:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="SJ6dw3hE" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 52BFF2078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k8QXZ-00077E-4W; Wed, 19 Aug 2020 16:02:17 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k8QXY-0006xV-64 for xen-devel@lists.xenproject.org; Wed, 19 Aug 2020 16:02:16 +0000 X-Inumbo-ID: ecb68983-4e28-4c7e-bf0a-35f14a563bf6 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id ecb68983-4e28-4c7e-bf0a-35f14a563bf6; Wed, 19 Aug 2020 16:02:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597852923; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=p7mP1vtVn+OU33mr73pbYK282VyBZ/DreNKLS9hTU+0=; b=SJ6dw3hE9BFNgkGRLBmbF6UgjXb/WshyAjk8IVFKdCMrjq2S5nfv9pUs zypsR8G5ACDHfmXyT5tqWy+Qa+JYv/Xrrx0TPhxPyMXgdL0CcSBZ2oRTP vtjof+27AaZm/DOqO1DsfSMR9gwH0BrTLh2MmjWvCW5PDeaknFeRhEyHX E=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: Hi7luPsYY2MTYMITLuDPYOiEvVCM4nubTIq6/jQSBQA+IwwA2WKXOLhfYzAnjxlGI6c3qBm0vB E+HsiBTgSPIanwKl1rI2qnCTAQdXNDJdaRdLi4YJffF9B68oWDBFFck3VPE8aDtuai1EyUSfqR yIMytJ4Zuy/bgzno2A7QPcSZenN/+epoAEFDOHnkyoPsSTf2v4oDjVbSP2o+e91iZeSI6dS8AF LKNmNLzekQSnjv2CFzHJtH2hSI/g0wWg1YjE+bo4Bg5RiIqEEtqy4yd3fbqXqR+fAH7K2Jj5Sm wmk= X-SBRS: 2.7 X-MesageID: 24885411 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,331,1592884800"; d="scan'208";a="24885411" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 7/7] schema: Provide index on flights by start time Date: Wed, 19 Aug 2020 17:01:43 +0100 Message-ID: <20200819160143.11178-8-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200819160143.11178-1-ian.jackson@eu.citrix.com> References: <20200819160143.11178-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" We often use flight number as a proxy for ordering, but this is not always appropriate and not always done (and sometimes it's a bit of a bodge). Provide an index to find flights by start time. This significantly speeds up the host allocation $equivstatusq query, and the duration estimator. (I have tested this by creating a trial index in the production database. That index can be dropped again, preferably after this commit makes it to production.) Signed-off-by: Ian Jackson --- schema/flights-started-index.sql | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 schema/flights-started-index.sql diff --git a/schema/flights-started-index.sql b/schema/flights-started-index.sql new file mode 100644 index 00000000..c230d9d8 --- /dev/null +++ b/schema/flights-started-index.sql @@ -0,0 +1,7 @@ +-- ##OSSTEST## 011 Harmless +-- +-- This index helps ts-hosts-allocate-Executive find recent instances +-- of the same job. It may be useful for other things too. + +CREATE INDEX flights_blessing_started_idx + ON flights (blessing, started);