From patchwork Fri Jul 31 11:37:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11694655 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ED670913 for ; Fri, 31 Jul 2020 11:39:40 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D28CA22CB2 for ; Fri, 31 Jul 2020 11:39:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D28CA22CB2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TNX-00065Q-AR; Fri, 31 Jul 2020 11:39:11 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TNX-0005kZ-0j for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:39:11 +0000 X-Inumbo-ID: 5aa0cf16-d322-11ea-8e26-bc764e2007e4 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 5aa0cf16-d322-11ea-8e26-bc764e2007e4; Fri, 31 Jul 2020 11:38:30 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMr-0001W4-KU; Fri, 31 Jul 2020 12:38:29 +0100 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 09/41] sg-report-flight: Use WITH to use best index use for $flightsq Date: Fri, 31 Jul 2020 12:37:48 +0100 Message-Id: <20200731113820.5765-10-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson , George Dunlap Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" While we're here, convert this EXISTS subquery to a JOIN. Perf: runtime of my test case now ~200-300s. Example query before (from the Perl DBI trace): SELECT * FROM ( SELECT DISTINCT flight, blessing FROM flights JOIN runvars r1 USING (flight) WHERE (branch='xen-unstable') AND ( (TRUE AND flight <= 151903) AND (blessing='real') ) AND EXISTS (SELECT 1 FROM jobs WHERE jobs.flight = flights.flight AND jobs.job = ?) AND r1.name LIKE 'built_revision_%' AND r1.name = ? AND r1.val= ? ORDER BY flight DESC LIMIT 1000 ) AS sub ORDER BY blessing ASC, flight DESC With bind variables: "test-armhf-armhf-libvirt" 'built_revision_xen' '165f3afbfc3db70fcfdccad07085cde0a03c858b' After: WITH sub AS ( SELECT DISTINCT flight, blessing FROM flights JOIN runvars r1 USING (flight) WHERE (branch='xen-unstable') AND ( (TRUE AND flight <= 151903) AND (blessing='real') ) AND r1.name LIKE 'built_revision_%' AND r1.name = ? AND r1.val= ? ORDER BY flight DESC LIMIT 1000 ) SELECT * FROM sub JOIN jobs USING (flight) WHERE (1=1) AND jobs.job = ? ORDER BY blessing ASC, flight DESC With bind variables: 'built_revision_xen' '165f3afbfc3db70fcfdccad07085cde0a03c858b' "test-armhf-armhf-libvirt" Diff to the query: - SELECT * FROM ( + WITH sub AS ( SELECT DISTINCT flight, blessing FROM flights JOIN runvars r1 USING (flight) WHERE (branch='xen-unstable') AND ( (TRUE AND flight <= 151903) AND (blessing='real') ) - AND EXISTS (SELECT 1 - FROM jobs - WHERE jobs.flight = flights.flight - AND jobs.job = ?) - AND r1.name LIKE 'built_revision_%' AND r1.name = ? AND r1.val= ? ORDER BY flight DESC LIMIT 1000 - ) AS sub + ) + SELECT * + FROM sub + JOIN jobs USING (flight) + + WHERE (1=1) + AND jobs.job = ? + ORDER BY blessing ASC, flight DESC Reviewed-by: George Dunlap Signed-off-by: Ian Jackson --- sg-report-flight | 39 ++++++++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 15 deletions(-) diff --git a/sg-report-flight b/sg-report-flight index 10127582..d06be292 100755 --- a/sg-report-flight +++ b/sg-report-flight @@ -180,18 +180,6 @@ END return undef; } - my @flightsq_params; - my $flightsq_jobcond='(1=1)'; - if (defined $job) { - push @flightsq_params, $job; - $flightsq_jobcond = <