From patchwork Fri Aug 14 17:21:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714853 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0F0CD13A4 for ; Fri, 14 Aug 2020 17:23:29 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DFAF820708 for ; Fri, 14 Aug 2020 17:23:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="TIfosQz4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DFAF820708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPD-00024X-Ig; Fri, 14 Aug 2020 17:22:15 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPC-00024Q-JZ for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:22:14 +0000 X-Inumbo-ID: b8347382-924e-4d0d-9e5e-3016f5d321cf Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id b8347382-924e-4d0d-9e5e-3016f5d321cf; Fri, 14 Aug 2020 17:22:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425733; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=RjjORyA+rdseRm2tV6Z7JEe1+gg8kiuTDoKkbr3lFMQ=; b=TIfosQz4s08fgocO6QaE7ES6nuU3DsyC5/8VgJp49lL1gz7dcJ+UkDqF HeGDXznnC//J7UfSmYeSz9dMEWk6QysgZ3aW+RGL97Xuq9RBqUDDMJXbE P0L1paw0mR5Pf+Jerp9zm8jgKxWTfbdFbc2l+IZs9FF236vtXVdtMuEiZ 8=; Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: ONtIFVyXJwkCpDNFv43eVx5sFrmhe57dsGwVmo3cuMWhLTNHKS8SlVvhyewcYHDHV4iZaBObwY vOWuIi5UQbmECw0soiEYJ1dZYFlHVVhGNwYG7BhHuVXjWEZHZ/k9GrTLpK17hTTFkQnsBrmsHi HGwyGw6bxn5M/EXEhdGUyO2QiydwC8e+B/T73qLw0KJe8EHJrrZ9UUtfrPJf/WsaSBFX8nlO77 ptbTv70h5jmjOutwcY53VFfDGfULOTnysQwmeZ3aS0l029RIKr3kZfpLCBCfPw8TmaFPoWogLK iEM= X-SBRS: 2.7 X-MesageID: 24900362 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24900362" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 01/60] history reporting (nfc): Do not key cache on hostname any more Date: Fri, 14 Aug 2020 18:21:06 +0100 Message-ID: <20200814172205.9624-2-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Now we process only one host at a time, so we don't need to distinguish. Replace all references to $tcache with %cache. Signed-off-by: Ian Jackson --- sg-report-host-history | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index 380f8fac..50cc5b58 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -76,7 +76,7 @@ our $restrictflight_cond = restrictflight_cond(); our $flightcond; our $minflight; -our %hcaches; +our %cache; sub read_existing_logs ($) { my ($hostname) = @_; @@ -86,8 +86,7 @@ sub read_existing_logs ($) { return if $!==ENOENT; die "failed to open $html_file: $!"; } - my $tcache = { }; - $hcaches{$hostname} = $tcache; + %cache = (); for (;;) { $_ = // last; next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$}; @@ -106,7 +105,7 @@ sub read_existing_logs ($) { # print DEBUG "GOTCACHE $hostname $k\n"; } # print DEBUG "GOTCACHE $hostname \@ $jr->{flight} $jr->{job} $jr->{status},$jr->{name}\n"; - $tcache->{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}} = $jr; + $cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}} = $jr; } close H; } @@ -253,8 +252,6 @@ END my $inrows = $hosts{$hostname}; print DEBUG "FOUND ", (scalar @$inrows), " ROWS for $hostname\n"; - my $tcache = $hcaches{$hostname}; - # Each entry in @$inrows is a $jr, which is a hash # It has keys for the result columns in mainquery # It also has keys '%' (yes, with a literal '%') @@ -267,7 +264,7 @@ END #print DEBUG "JOB $jr->{flight}.$jr->{job} "; my $cacherow = - $tcache->{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}}; + $cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}}; if ($cacherow) { $jr = $cacherow; $cachehits++; @@ -285,7 +282,7 @@ END } print DEBUG "CACHE $hostname $cachehits / ".(scalar @rows) - ." of ".(scalar %$tcache)."\n"; + ." of ".(scalar %cache)."\n"; my $write_cache_entry = sub { my ($jr) = @_; From patchwork Fri Aug 14 17:21:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714863 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 386CA1392 for ; Fri, 14 Aug 2020 17:23:38 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 14A6620708 for ; Fri, 14 Aug 2020 17:23:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="NzVsSnU+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 14A6620708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPC-00024M-Ae; Fri, 14 Aug 2020 17:22:14 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPB-00024H-4x for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:22:13 +0000 X-Inumbo-ID: 4c243ad5-4a45-4638-aa9b-618e2221c3ca Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 4c243ad5-4a45-4638-aa9b-618e2221c3ca; Fri, 14 Aug 2020 17:22:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425731; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=40Khy0gWh0iQVdg61oKnTFxaeFxIc6NbFWsDCEifoNI=; b=NzVsSnU+jQbW0cnH8oB8pm3JoxuimzwXYrFAqAA5Kum6aAVDLOIQsz8b naITne2R5iz6D3jOC4T5l23Jx7XEDDisVaUY9oXj275enINIByN3NuzTh mtAFXXR2M2XUSdnAjpCsK0vVdLouK6MHqSq9KGzdrkvzFyjsrKh5D8RUp k=; Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: KGE6f1lJ2abRajEOCKn/DzJS37ZkBGsTzNgXxJ1+j42XSJepw13zOvxdV1pXGu5S7w2WK6P8DF Y5pqJzveiVjfrWvPmyWzR6YMzYPN8XSHIRLgFIojNfTgoR58TyEgBGVR3O6nW5Dn3w+ql/w2MR oELiDTNifuYNsgKlXhsVKNBXzr1GAf2jwZTnDTJFg8SpIBkfILxR2w5EVn4plmAtPbkbfFKzGW YrC2M9VuxibYmow5rmG5CL6VWXw0VA+fDi5mW7LTZn0zC1loMIXzN8mZ8g7yie2vJxmNAo9+Pk f9s= X-SBRS: 2.7 X-MesageID: 24879724 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24879724" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 02/60] history reporting (nfc): Add some test runes to the notes Date: Fri, 14 Aug 2020 18:21:07 +0100 Message-ID: <20200814172205.9624-3-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Signed-off-by: Ian Jackson --- runes | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/runes b/runes index 27ece2af..8139f955 100644 --- a/runes +++ b/runes @@ -3,3 +3,11 @@ select * from resource_sharing s full outer join resources r on r.restype = 'sha select * from jobs full outer join steps using (job, flight) where flight=2364 order by job, stepno; + + + +>v; env DBI_TRACE=2=v time ./sg-report-job-history --debug --html-dir=$PWD/d --flight=152223 --max-flight=152223 2>&1 |tee c |ts -s %.s >c.ts + +>v; env DBI_TRACE=2=v time ./sg-report-host-history --debug --html-dir=$PWD/e --max-flight=152223 flight:152223 2>&1 |tee c |ts -s %.s >c.ts +rm -rf d; mkdir d; cp -a e/. d/.; mv c.ts b.ts; mv c b; mv v u +cp -a d/. e/. From patchwork Fri Aug 14 17:21:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714881 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8F058618 for ; Fri, 14 Aug 2020 17:26:57 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6AD6020708 for ; Fri, 14 Aug 2020 17:26:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="hKZIPPKq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6AD6020708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTJ-0003xY-Mc; Fri, 14 Aug 2020 17:26:29 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dR3-00024Q-I8 for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:24:09 +0000 X-Inumbo-ID: 33120668-a05a-4e35-95d1-b92d3baf3302 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 33120668-a05a-4e35-95d1-b92d3baf3302; Fri, 14 Aug 2020 17:22:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425756; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=X6/feefs+EBmTwV9A8+QWNObBLw3cmrBQxuV3E+6g3c=; b=hKZIPPKqx3MovPReQ7KyXonMxOopr+CtQkOduARttGbEPNS2BTVZWMrX c+uhPqexWlwIsvEdyaI80+GscyQs73VHCDO1VmliGtF7OML1wMWMljMU2 XoIdCiqjqNutHOAWn9lWGE5J/zKz4g/b5Z34EZxWQwi87kol71+ZwkTjB M=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: HM1x3iOPWrYXuGhFyidvWlgA3coSfl/c0Ncma2X8a35RmMbAO+wu9fnfzZw7dSnWhmIiacXrNB gWYoLo7OaK1NiOpWRZMprn/iznQXYjamKW9sW+5kqgKcz+JpXObUPFEGjy6lgqKjODOM7jUNjd Hs8s4KwerOGgNLdQrcTYuumVn9jhzQswHuN71q/XerLD4MeOnWXVg20TCQ0PhKBc6oHUNXiiGY rXe6us5Ud2dNrzIJxu+ysDBABxK98kFri41ssHiJu2pzHaJYGdebAPRNldSwUoF57NF9mHvYWO d+8= X-SBRS: 2.7 X-MesageID: 24576909 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24576909" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 03/60] history reporting: Delete two debug prints Date: Fri, 14 Aug 2020 18:21:08 +0100 Message-ID: <20200814172205.9624-4-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" These are going to become invalid and I don't think it's important to save them. Signed-off-by: Ian Jackson --- sg-report-host-history | 2 -- 1 file changed, 2 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index 50cc5b58..4c0022e8 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -102,9 +102,7 @@ sub read_existing_logs ($) { my $k = $1; s{\%([0-9a-f]{2})}{ chr hex $1 }ge; $ch->{$k} = $_; -# print DEBUG "GOTCACHE $hostname $k\n"; } -# print DEBUG "GOTCACHE $hostname \@ $jr->{flight} $jr->{job} $jr->{status},$jr->{name}\n"; $cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}} = $jr; } close H; From patchwork Fri Aug 14 17:21:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714865 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DC4581392 for ; Fri, 14 Aug 2020 17:23:42 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B876D20708 for ; Fri, 14 Aug 2020 17:23:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="copc8rFQ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B876D20708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPH-00024z-Qx; Fri, 14 Aug 2020 17:22:19 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPF-00024H-UD for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:22:17 +0000 X-Inumbo-ID: baab65aa-6d25-44f2-a8a2-8b0d04228735 Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id baab65aa-6d25-44f2-a8a2-8b0d04228735; Fri, 14 Aug 2020 17:22:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425733; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=5gDfq120B1TdLL2gzZ7GTUxIKoMyYxGwxpZ/5dWdhfU=; b=copc8rFQvTBzg46YaIYc5RS35YjX7P+UgncQdf/BpOB7c15FY4W6hAEO /APdeYjvOybXipUHnwWCp+YbEKeMPf3W319vYTMGUIfqzflGyjb8ercWB SJw/AxBRzEZG2Hdy8TYbSb77QPeE1Qax9WW05vBI45tWhcnLnESJM7VmT c=; Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: MlOk7mvYTrWJ2MacOCGhA+a7t/6PeGH6nlGF6cx8cqRH+L7gR+dr7pr46eJz7470Tv2TKggzk3 gvozQQvCa84qDPDO/8OTHtjfhTZoE7xuOpufy9dCaTgqPlk5Ekaqxd5tRTCwMEKKmGihOxMmB3 9g98GFCMh0OegL4qSUxyc7+m37iG0p4R0oofrhQpSIoC1fUSL5Q3uxoNt+JWaVSGfZuyguJQYs KWuxuhS6vUEfxfbghiu+XLxIOQv5nEfZdu8/Le8l71uc0owvcCCCOSs026WnoIDTSS7pmqCEgY We8= X-SBRS: 2.7 X-MesageID: 24879726 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24879726" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 04/60] history reporting (nfc): Refactor to generalise, cache_read_existing Date: Fri, 14 Aug 2020 18:21:09 +0100 Message-ID: <20200814172205.9624-5-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" * Introduce @cache_row_key_cols and cache_row_key * Handle $html_file pathname construction and $read_existing at the call site. * Rename cache_read_existing. Signed-off-by: Ian Jackson --- sg-report-host-history | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index 4c0022e8..83344a40 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -77,11 +77,15 @@ our $flightcond; our $minflight; our %cache; +our @cache_row_key_cols = qw(flight job status name); -sub read_existing_logs ($) { - my ($hostname) = @_; - return unless $read_existing; - my $html_file = "$htmlout/$hostname.html"; +sub cache_row_key ($) { + my ($jr) = @_; + return join $; , map { $jr->{$_} } @cache_row_key_cols; +} + +sub cache_read_existing ($) { + my ($html_file) = @_; if (!open H, $html_file) { return if $!==ENOENT; die "failed to open $html_file: $!"; @@ -103,7 +107,7 @@ sub read_existing_logs ($) { s{\%([0-9a-f]{2})}{ chr hex $1 }ge; $ch->{$k} = $_; } - $cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}} = $jr; + $cache{cache_row_key($jr)} = $jr; } close H; } @@ -261,8 +265,7 @@ END foreach my $jr (@$inrows) { #print DEBUG "JOB $jr->{flight}.$jr->{job} "; - my $cacherow = - $cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}}; + my $cacherow = $cache{cache_row_key($jr)}; if ($cacherow) { $jr = $cacherow; $cachehits++; @@ -492,7 +495,7 @@ foreach my $host (sort keys %hosts) { my $pid = fork // die $!; if (!$pid) { opendb_tests(); - read_existing_logs($host); + cache_read_existing("$htmlout/$host.html") if $read_existing; db_retry($dbh_tests, [], sub { mainquery($host); reporthost $host; From patchwork Fri Aug 14 17:21:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714869 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D6AC51392 for ; Fri, 14 Aug 2020 17:23:44 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B2B2020708 for ; Fri, 14 Aug 2020 17:23:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="cGY9juJr" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B2B2020708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPR-00029z-2r; Fri, 14 Aug 2020 17:22:29 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPP-00024H-Um for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:22:27 +0000 X-Inumbo-ID: 7cfade94-bd8a-4f7c-9479-4c87aaef6b90 Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 7cfade94-bd8a-4f7c-9479-4c87aaef6b90; Fri, 14 Aug 2020 17:22:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425733; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=JGVZgUM9Ou32fNfhr7w/FMv9cNbdn7S2Jl2MIBunzZQ=; b=cGY9juJrgezOfICmiqUbGeVTxrbXt5WqfkYrMtNZcVAvgVUFtgRsJdeN Plr+hsgpEU/vJG0B88etnO+jWF4FHKzbnKbIY8Fnt1cxIqD3WC9wy20Td 0E2J7q/EOYhsFcgpSGA4F9lUeS0GpFXF1qMnug7ATqdkhBC+xmWi8hLST 4=; Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: gGBUpPHDUEVc/bmQu3o7aloOTSEwnX+ETtRovKb0ZTBHJzZzwYL1Qmy7ht8vRXIHRBjFWIvx+c DEGKDM9B8g6IRsFsHl4y3u64j6OUh2xsfj+pJ4uc11DveaHaG6ziaoy6ejSfzHcY1XeYGSVYCY 4fRHCIkn335Rdxk6F6XBJUAbC+MWd9YyT0yg4ykiiA8zDvzRnLpLAi4X3VDILwrPrgxj5ERn2t 6WOzpaGoDMbjDcXF4bDWNdVQi3yf9xdDCSISjL7+C+7luuB/P1Ubo1jWVLrIgk6TSne/i2iGZr wAQ= X-SBRS: 2.7 X-MesageID: 24879728 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24879728" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 05/60] history reporting (nfc): Make cache_write_entry into a top-level sub Date: Fri, 14 Aug 2020 18:21:10 +0100 Message-ID: <20200814172205.9624-6-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Signed-off-by: Ian Jackson --- sg-report-host-history | 46 +++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index 83344a40..fe5b7cc2 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -112,6 +112,27 @@ sub cache_read_existing ($) { close H; } +sub cache_write_entry ($$) { + my ($fh, $jr) = @_; + print $fh "\n"; +} + sub computeflightsrange () { if ($flightlimit) { my $minflightsq = db_prepare(<{$k}; - s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge; - printf H " %s=%s", $k, $_; - } - }; - $whash->($jr); - foreach my $hk (sort keys %$jr) { - next unless $hk =~ m/^\%/; - print H " $'"; - $whash->($jr->{$hk}); - } - print H " -->\n"; - }; - @rows = sort { $b->{finished} <=> $a->{finished} } @rows; print DEBUG "SORTED\n"; @@ -317,7 +317,7 @@ END foreach my $jr (@rows) { if ($wrote++ >= $limit) { - $write_cache_entry->($jr); + cache_write_entry(\*H, $jr); next; } @@ -417,7 +417,7 @@ END print H "" if !$any_power; print H "\n"; - $write_cache_entry->($jr); + cache_write_entry(\*H, $jr); print H "\n\n"; $alternate ^= 1; From patchwork Fri Aug 14 17:21:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714841 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1664A618 for ; Fri, 14 Aug 2020 17:22:52 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E4D3820708 for ; Fri, 14 Aug 2020 17:22:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="A53ykD5L" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E4D3820708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPV-0002DF-Nq; Fri, 14 Aug 2020 17:22:33 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPU-00024H-Up for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:22:32 +0000 X-Inumbo-ID: bbb23550-a72b-4c89-89ff-240451d07c14 Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id bbb23550-a72b-4c89-89ff-240451d07c14; Fri, 14 Aug 2020 17:22:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425734; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=Cz4cBcsHyi+A9DH1w0U0E75BDnFx0uYZsqrpI0k8hFo=; b=A53ykD5Lh80fLfkXZLGye8ElXPZlvlvhbU6+o6k2O8FuTIG9z79EW99B k27MKpWxZ7u0ylv1EKoT3lrh/icmQYjnLWUNnYCFA9bUpeqUFxgu85Nkx uAK6xWaSLn4yiuclHR7fL8Zex3WRbe95LxdiG+4qUL24NN56n5Rwudfvo A=; Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 8iGfY8QFR2avWHYEtqT6dGVdVUz6q/afE0+qO2U3QHO8PKN2InrRhkXaKXDQ4kWgBKX1HvBttX x1qHx8+o859XfnFAf40MpNmG7NhFYUsv3l/PGF+iYeKTUTyQHM54E7bjmM/zF6fVOB4jy5bgBK e9vRn32A5Sw+y7IWB5hYw5fy5/hCPTRdxGMOyEki745V9fC8eTJKPA3eVs0d0erHwRdqO88O2d alFU+cO4LvH+bPKMk2TCy0AspzKl9sPyrlez5Y1qRVcF8DxyM/Q9wpCqlIsSmHrxOyMSB5oU2A 45g= X-SBRS: 2.7 X-MesageID: 24900363 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24900363" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 06/60] history reporting (nfc): Rename jobquery to cacheable_query Date: Fri, 14 Aug 2020 18:21:11 +0100 Message-ID: <20200814172205.9624-7-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Prep work. Signed-off-by: Ian Jackson --- sg-report-host-history | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index fe5b7cc2..a8dae159 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -171,7 +171,7 @@ END our $jqcachemisses = 0; our $jqtotal = 0; -sub jobquery ($$$) { +sub cacheable_query ($$$) { my ($q, $jr, $cachekey) = @_; $jqtotal++; $cachekey = '%'.$cachekey; @@ -292,7 +292,7 @@ END $cachehits++; } - my $endedrow = jobquery($endedq, $jr, 'e'); + my $endedrow = cacheable_query($endedq, $jr, 'e'); if (!$endedrow) { #print DEBUG "no-finished\n"; next; @@ -322,8 +322,8 @@ END } #print DEBUG "JR $jr->{flight}.$jr->{job}\n"; - my $ir = jobquery($infoq, $jr, 'i'); - my $ar = jobquery($allocdq, $jr, 'a'); + my $ir = cacheable_query($infoq, $jr, 'i'); + my $ar = cacheable_query($allocdq, $jr, 'a'); my $ident = $jr->{name}; my $altcolour = report_altcolour($alternate); From patchwork Fri Aug 14 17:21:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714871 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E52071392 for ; Fri, 14 Aug 2020 17:23:45 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C1B9320708 for ; Fri, 14 Aug 2020 17:23:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="NoaWWEXs" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C1B9320708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPg-0002OZ-2n; Fri, 14 Aug 2020 17:22:44 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPe-00024H-V9 for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:22:42 +0000 X-Inumbo-ID: f3d33461-b42b-4aa0-ba62-f5f5675d7a53 Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id f3d33461-b42b-4aa0-ba62-f5f5675d7a53; Fri, 14 Aug 2020 17:22:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425735; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=7526Su95kbsnzuzEUlz4l/oEnSDHWKYNYWxHeDrsvYs=; b=NoaWWEXsuAQyuABfbMgC04W9cahAwBe5mO+2PV/YPbWpxGOCujcBws8O DoNcVz34sb+IQGgA3fAoMVsfVZYVl15LnFLpd8nSjkrj8XYz8IxOxzDXt yJatq6p8EBpMjjQD9zKwOu+tuf2E21/oGjNWgEkBAG+3Zodt1QLvy+ab+ 8=; Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: exZqm6N1k7Xa5mO1hHKBYiL8tU9DKSTz8x5hC0DyddGilu+NrguhueaT/GX2FkK1xOz3nNKyI3 uHEYqLr4HLBoPgsJ5C9XPvzLNElHeDYrx0Mpd3e52kxOFkZ81reo9MQNsFMeqIlU7NNUdhf1l8 tyc+aFls9OQ4u3nrqg+3ApT3VlbWRZMgATFW/nvkTefCqPYN/Gt/VeMlUOgLIsXADbLgYlFFPJ FZQ/aKwSQqAmhVr6amO1ltWG0LDCNQC0iYqbyDmtvYwg1LN+K7DXH7xVG9Xx/VC5Vc+8X2+yPx 078= X-SBRS: 2.7 X-MesageID: 24900365 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24900365" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 07/60] history reporting (nfc): Break out cache_row_lookup_prep Date: Fri, 14 Aug 2020 18:21:12 +0100 Message-ID: <20200814172205.9624-8-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Prep work. Signed-off-by: Ian Jackson --- sg-report-host-history | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index a8dae159..2724ef27 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -112,6 +112,16 @@ sub cache_read_existing ($) { close H; } +sub cache_row_lookup_prep ($) { + my ($jrr) = @_; + + my $cacherow = $cache{cache_row_key($$jrr)}; + if ($cacherow) { + $$jrr = $cacherow; + $cachehits++; + } +} + sub cache_write_entry ($$) { my ($fh, $jr) = @_; print $fh "\n"; } +cache_set_key_cols(qw(flight job status name)); + sub computeflightsrange () { if ($flightlimit) { my $minflightsq = db_prepare(< X-Patchwork-Id: 11714843 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 71E8A1392 for ; Fri, 14 Aug 2020 17:23:09 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4C77420708 for ; Fri, 14 Aug 2020 17:23:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="gN2g7q0J" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4C77420708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPl-0002V7-J4; Fri, 14 Aug 2020 17:22:49 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPj-00024H-VH for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:22:47 +0000 X-Inumbo-ID: a388a4ba-f6fb-4c3f-9179-ae9344d34a0c Received: from esa3.hc3370-68.iphmx.com (unknown [216.71.145.155]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id a388a4ba-f6fb-4c3f-9179-ae9344d34a0c; Fri, 14 Aug 2020 17:22:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425736; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=umud+b+GQgjYN5JC86L2XalvXk3tOUTGHSJ1av/HKnc=; b=gN2g7q0J32HeAtxLfWL86sJyN2dHKm0dnf4E21lExwZq0AsjC0cw/b3v hQefyQ7dvLReKl9T9mTKP0AQ+Qiw+Lhl0aPA8yf0vMpaCGW3stkAKy9bb /efBVmVKGcOkOptZfshdJ4Ae6f0nkvfiQIsEnRx+KrhSDsSXoWRPJ8WLD E=; Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: DfLho/TOm7Qfkx+V8vqWXMYGyKeeFomewX29ayXdO5LBqKUZYxkpYuVP7/NQiMaEvQdDiKk0Qi gr4rrujOCQF5PqUaiqJISlPIFS6scS5B7nbgA9gRO4smqg3ygMgXJPzRuGy/cAOfey+0Id4suD NCtnA0Q2n6k/wBWVVtzO8YN6spdxGVRWy9or4wV/ulMhnoROiqYKx7CTkYcElrgjdDiOe9im90 5tLilPOwmcKDYGRcEw+wiEDmtiZCere0HyzW+toZ8I6MGAEJMLnqZUKOExOdSNHSqIexfPLNox HtE= X-SBRS: 2.7 X-MesageID: 24545453 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24545453" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 16/60] history reporting (nfc): Record more row-specific stats Date: Fri, 14 Aug 2020 18:21:21 +0100 Message-ID: <20200814172205.9624-17-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Prep work. Nothing uses these yet. Signed-off-by: Ian Jackson --- sg-report-host-history | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/sg-report-host-history b/sg-report-host-history index 07e549a0..32cc883d 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -83,6 +83,10 @@ our %cache; our %q_count; our %q_misses; +our $rows_previous = 0; +our $rows_today = 0; +our $rows_hit = 0; + sub cache_set_key_cols { @cache_row_key_cols = @_; } sub cache_row_key ($) { @@ -128,6 +132,7 @@ sub cache_read_previous ($) { } %cache = (); for (;;) { + $rows_previous++; $_ = // last; next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$}; my $jr = {}; @@ -151,6 +156,7 @@ sub cache_read_previous ($) { sub cache_row_lookup_prep ($) { my ($jrr) = @_; + $rows_today++; my $cacherow = $cache{cache_row_key($$jrr)}; if ($cacherow) { $$jrr = $cacherow; @@ -313,7 +319,6 @@ END # The contents of $jr for each job is cached across runs. my @rows; - my $rows_hit = 0; foreach my $jr (@$inrows) { #print DEBUG "JOB $jr->{flight}.$jr->{job} "; From patchwork Fri Aug 14 17:21:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714857 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4EAE31392 for ; Fri, 14 Aug 2020 17:23:32 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2A8D02078D for ; Fri, 14 Aug 2020 17:23:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="V78mDAs8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2A8D02078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPq-0002aY-7O; Fri, 14 Aug 2020 17:22:54 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPo-00024H-Vd for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:22:53 +0000 X-Inumbo-ID: 6f09f715-d41d-4452-ad68-84fe856b129e Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 6f09f715-d41d-4452-ad68-84fe856b129e; Fri, 14 Aug 2020 17:22:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425736; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=47TkWKaaMaSDWsGucq1zdsdrawQzX1oeIufX8OcbbNY=; b=V78mDAs8efXQIUdZJjPyIroEivHDu5bOCuhAc4PwIeS9vGZ4hNBMIXGx JOmK6NYLkEoQDjUfqU4oGAl9LaLk31WAXcoCZ7MqnfWpO4EJ2pwci0YxZ f4XPF3Qu3V8swAOqcwjn1olUMTHK/pXfSzj6Q4Ln1OE9HIEFPqw0R1RC6 k=; Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: zFNVZ/yNaBs/RwWhfhsC3hLuE+5mbJuv5+TSB6jTA/S45Ql/xDkCZYXNQDOD+Ul7Zy5325BDwD 3KQkroufcgpnS9VvdPWkQrPuzWioHFTXLZzN+tnRKTDgF8RW1TOsCKWbDOxK7061p2dAKCGB8E NV6OutNrglmz0SEUtiKRce+PaaGUThdACd7nqujpzX5GkMxAg183rUWvxb82RrHbTjqDuliYat CFsdSVI9qhvr7CL/Sw5YK1sJd7ddLMDaXB9wvMNYcCTDT8WQlF1VeSgxcQalL/opXvIfw3Mwgw JCE= X-SBRS: 2.7 X-MesageID: 24900369 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24900369" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 17/60] history reporting: Cache stats reporting: Centralise and rework Date: Fri, 14 Aug 2020 18:21:22 +0100 Message-ID: <20200814172205.9624-18-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" This uses the new variables instead of the old ones. Signed-off-by: Ian Jackson --- sg-report-host-history | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index 32cc883d..9c70b560 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -185,6 +185,17 @@ sub cache_write_entry ($$) { print $fh " -->\n"; } +sub cache_report_stats ($) { + my ($what) = @_; + print ::DEBUG "CACHE $what read=$rows_previous hits $rows_hit/$rows_today"; + for my $cachekey (sort keys %q_count) { + my $total = $q_count{$cachekey}; + my $hits = $total - ($q_misses{$cachekey} // 0); + print ::DEBUG " $cachekey=$hits/$total"; + } + print ::DEBUG "\n"; +} + cache_set_key_cols(qw(flight job status name)); sub computeflightsrange () { @@ -335,9 +346,6 @@ END push @rows, { %$jr, %$endedrow }; } - print DEBUG "CACHE $hostname $rows_hit / ".(scalar @rows) - ." of ".(scalar %cache)."\n"; - @rows = sort { $b->{finished} <=> $a->{finished} } @rows; print DEBUG "SORTED\n"; @@ -457,8 +465,7 @@ END rename "$html_file.new", "$html_file" or die "$html_file $!" if $doinstall; - print DEBUG "HOST CACHE RQ $runvarq_hits / ". - ($runvarq_hits+$runvarq_misses)."\n"; + cache_report_stats("$hostname"); } foreach my $host (@ARGV) { @@ -528,7 +535,6 @@ foreach my $host (sort keys %hosts) { mainquery($host); reporthost $host; }); - print DEBUG "JQ CACHE ".($jqtotal-$jqcachemisses)." / $jqtotal\n"; exit(0); } print DEBUG "SPAWNED [$pid] $host\n"; From patchwork Fri Aug 14 17:21:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714937 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5960E13A4 for ; Fri, 14 Aug 2020 17:27:43 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 34DE32078D for ; Fri, 14 Aug 2020 17:27:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="EgNsuOHe" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 34DE32078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dU2-0005Ts-Nr; Fri, 14 Aug 2020 17:27:14 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dRS-00024Q-Ir for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:24:34 +0000 X-Inumbo-ID: c3ed870a-8ef5-4ee6-99e9-a71070530036 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id c3ed870a-8ef5-4ee6-99e9-a71070530036; Fri, 14 Aug 2020 17:22:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425759; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=hLH4IlqR4STUHHN+FXp+4qPB6iWG6dzD5c0xFXzYa+s=; b=EgNsuOHeaTlLA3e3m4gjYnk0xcDfD2BB+K6PQ4lpDcVaGeSDZGNqKRz9 0YSOn+sW+kFqnYiXfSz5TfqykYV/+WpQIfFXewMGZtvXDqkmxuPF7/voK mn0CQobm+GcPX/JpgZuvDgSrXI4CUh21zCqhZUhj8RlKSnWwWhQ5wJYVo w=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: rN03X/xWpyNFlP8uKUjQzg7ydkvMigvB9XdODXnIvKmKwA+hYOqXlh3X80p/3EImZMqIx1rHD/ e7+eRc6ok8PCrC84/cgnuGAJWdTZexmGezUF70pcd2lmKfZZtTK3KeKlwve85oylAA2tO9/TPd ykc181jArbFGsi0H/LVHyBzq42BEvZt3YrynPe8RyxeyO0uusxGjE07Mw3WSbwlSmSPtE/QZar Vbzwd4n4D8Wugy/WrK4QFxZbBHJwDM+jN/rp7q3j7qYNpjUUQxpk/EZuLZKoD38m+YYPTd7heV QhM= X-SBRS: 2.7 X-MesageID: 24576916 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24576916" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 18/60] history reporting (nfc): Remove now-obsolete stats variables Date: Fri, 14 Aug 2020 18:21:23 +0100 Message-ID: <20200814172205.9624-19-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Signed-off-by: Ian Jackson --- sg-report-host-history | 5 ----- 1 file changed, 5 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index 9c70b560..1b4566aa 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -94,17 +94,12 @@ sub cache_row_key ($) { return join $; , map { $jr->{$_} } @cache_row_key_cols; } -our $jqcachemisses = 0; -our $jqtotal = 0; - sub cacheable_fn ($$$) { my ($jr, $cachekey, $fn) = @_; - $jqtotal++; $cachekey = '%'.$cachekey; my $cached = $jr->{$cachekey}; $q_count{$cachekey}++; if (!$cached) { - $jqcachemisses++; $q_misses{$cachekey}++; $cached = $fn->(); $jr->{$cachekey} = $cached; From patchwork Fri Aug 14 17:21:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714907 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C1E6A1392 for ; Fri, 14 Aug 2020 17:27:29 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9E35E20866 for ; Fri, 14 Aug 2020 17:27:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="M+QfoL+x" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9E35E20866 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTc-0004P1-FB; Fri, 14 Aug 2020 17:26:48 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dRI-00024Q-Is for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:24:24 +0000 X-Inumbo-ID: 780f4fc4-0a03-42b8-9678-da8eec86efa9 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 780f4fc4-0a03-42b8-9678-da8eec86efa9; Fri, 14 Aug 2020 17:22:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425758; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=I4Nd9Gt6CpMY5s4Khl15W7v0e3Htwl7Hno7gLQnvhCk=; b=M+QfoL+xq9fDh3OgGZ3SlulYtl2slAVuqQ0k5OYnfOMq0eW/Dbc83rza LNdnydxvqPjpjnpTZdCKgbHp613AHrL+QGX3rdMT4KITq/x373TJTB9dc 6duFoFUplUbgyQo7Bg25dqM4kyaFY9FqIZr+dWG8wt0mCxuYgv12mBi3H g=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: qKBuPelv8bhlZJNAUjTqHhwuampfVjCSQAyrcXmT4GiiB4wrcjz/HEs2ZPOX70olOZOh+JCRVo A75TYw1/8GiOGrcSFVVCRBm9cv6Qd1MRC4V83azadyrvvINN4AQLJPeVGvdQmcIxDsIEEZbxGp VGHmn58BmleiYaC9g+rhZndJnF2kyxfnrcInpRW6AQQRKnWw4KRrCzQ4jiZX+vSqxuL9zbN89H eqejur8hjhsiLM0TLvy7yJaneOKjouwi81ljwbM7gR6Nnrd1BQuwKhBlLHpnM6cZx0VU4N00kI 8pY= X-SBRS: 2.7 X-MesageID: 24576915 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24576915" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 20/60] history reporting (nfc): Move cache code into HistoryReport module Date: Fri, 14 Aug 2020 18:21:25 +0100 Message-ID: <20200814172205.9624-21-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Finally this is now reuseable code and we can put it in the HistoryReport module. Pure cut-and-paste. Signed-off-by: Ian Jackson --- Osstest/HistoryReport.pm | 115 +++++++++++++++++++++++++++++++++++++++++++++++ sg-report-host-history | 115 ----------------------------------------------- 2 files changed, 115 insertions(+), 115 deletions(-) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index c5e7e226..a0565b6a 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -40,4 +40,119 @@ BEGIN { use POSIX; +our @cache_row_key_cols; + +our %cache; + +our %q_count; +our %q_misses; + +our $rows_previous = 0; +our $rows_today = 0; +our $rows_hit = 0; + +sub cache_set_key_cols { @cache_row_key_cols = @_; } + +sub cache_row_key ($) { + my ($jr) = @_; + return join $; , map { $jr->{$_} } @cache_row_key_cols; +} + +sub cacheable_fn ($$$) { + my ($jr, $cachekey, $fn) = @_; + $cachekey = '%'.$cachekey; + my $cached = $jr->{$cachekey}; + $q_count{$cachekey}++; + if (!$cached) { + $q_misses{$cachekey}++; + $cached = $fn->(); + $jr->{$cachekey} = $cached; + } + return $cached; +} + +sub cacheable_query ($$$) { + my ($q, $jr, $cachekey) = @_; + cacheable_fn($jr, $cachekey, sub { + foreach my $k (keys %{ $q->{ParamTypes} }) { + $k =~ m/^:/ or die "$k ?"; + $q->bind_param($k, $jr->{$'} // die "$k ?"); + } + $q->execute(); + return $q->fetchrow_hashref(); + }); +} + +sub cache_read_previous ($) { + my ($html_file) = @_; + if (!open H, $html_file) { + return if $!==ENOENT; + die "failed to open $html_file: $!"; + } + %cache = (); + for (;;) { + $rows_previous++; + $_ = // last; + next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$}; + my $jr = {}; + my $ch = $jr; + foreach (split / /, $1) { + if (m{^\w+$}) { + $ch = { }; + $jr->{'%'.$&} = $ch; + next; + } + s{^(\w+)=}{} or die; + my $k = $1; + s{\%([0-9a-f]{2})}{ chr hex $1 }ge; + $ch->{$k} = $_; + } + $cache{cache_row_key($jr)} = $jr; + } + close H; +} + +sub cache_row_lookup_prep ($) { + my ($jrr) = @_; + + $rows_today++; + my $cacherow = $cache{cache_row_key($$jrr)}; + if ($cacherow) { + $$jrr = $cacherow; + $rows_hit++; + } +} + +sub cache_write_entry ($$) { + my ($fh, $jr) = @_; + print $fh "\n"; +} + +sub cache_report_stats ($) { + my ($what) = @_; + print ::DEBUG "CACHE $what read=$rows_previous hits $rows_hit/$rows_today"; + for my $cachekey (sort keys %q_count) { + my $total = $q_count{$cachekey}; + my $hits = $total - ($q_misses{$cachekey} // 0); + print ::DEBUG " $cachekey=$hits/$total"; + } + print ::DEBUG "\n"; +} + 1; diff --git a/sg-report-host-history b/sg-report-host-history index 05a2dfe0..a195bb21 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -77,121 +77,6 @@ our $restrictflight_cond = restrictflight_cond(); our $flightcond; our $minflight; -our @cache_row_key_cols; - -our %cache; - -our %q_count; -our %q_misses; - -our $rows_previous = 0; -our $rows_today = 0; -our $rows_hit = 0; - -sub cache_set_key_cols { @cache_row_key_cols = @_; } - -sub cache_row_key ($) { - my ($jr) = @_; - return join $; , map { $jr->{$_} } @cache_row_key_cols; -} - -sub cacheable_fn ($$$) { - my ($jr, $cachekey, $fn) = @_; - $cachekey = '%'.$cachekey; - my $cached = $jr->{$cachekey}; - $q_count{$cachekey}++; - if (!$cached) { - $q_misses{$cachekey}++; - $cached = $fn->(); - $jr->{$cachekey} = $cached; - } - return $cached; -} - -sub cacheable_query ($$$) { - my ($q, $jr, $cachekey) = @_; - cacheable_fn($jr, $cachekey, sub { - foreach my $k (keys %{ $q->{ParamTypes} }) { - $k =~ m/^:/ or die "$k ?"; - $q->bind_param($k, $jr->{$'} // die "$k ?"); - } - $q->execute(); - return $q->fetchrow_hashref(); - }); -} - -sub cache_read_previous ($) { - my ($html_file) = @_; - if (!open H, $html_file) { - return if $!==ENOENT; - die "failed to open $html_file: $!"; - } - %cache = (); - for (;;) { - $rows_previous++; - $_ = // last; - next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$}; - my $jr = {}; - my $ch = $jr; - foreach (split / /, $1) { - if (m{^\w+$}) { - $ch = { }; - $jr->{'%'.$&} = $ch; - next; - } - s{^(\w+)=}{} or die; - my $k = $1; - s{\%([0-9a-f]{2})}{ chr hex $1 }ge; - $ch->{$k} = $_; - } - $cache{cache_row_key($jr)} = $jr; - } - close H; -} - -sub cache_row_lookup_prep ($) { - my ($jrr) = @_; - - $rows_today++; - my $cacherow = $cache{cache_row_key($$jrr)}; - if ($cacherow) { - $$jrr = $cacherow; - $rows_hit++; - } -} - -sub cache_write_entry ($$) { - my ($fh, $jr) = @_; - print $fh "\n"; -} - -sub cache_report_stats ($) { - my ($what) = @_; - print ::DEBUG "CACHE $what read=$rows_previous hits $rows_hit/$rows_today"; - for my $cachekey (sort keys %q_count) { - my $total = $q_count{$cachekey}; - my $hits = $total - ($q_misses{$cachekey} // 0); - print ::DEBUG " $cachekey=$hits/$total"; - } - print ::DEBUG "\n"; -} - cache_set_key_cols(qw(flight job status name)); sub computeflightsrange () { From patchwork Fri Aug 14 17:21:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714903 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 849E3618 for ; Fri, 14 Aug 2020 17:27:27 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5F4B920708 for ; Fri, 14 Aug 2020 17:27:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="UH5jPwKL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5F4B920708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTo-0004t4-B5; Fri, 14 Aug 2020 17:27:00 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dSP-00024H-55 for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:25:33 +0000 X-Inumbo-ID: cec8a870-aeaf-4337-8028-080daa349966 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id cec8a870-aeaf-4337-8028-080daa349966; Fri, 14 Aug 2020 17:22:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425762; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=NHSD6TGEqM5ArTfICWOWbpPDVCeLImS8L+r0LiJLVRw=; b=UH5jPwKLZGD3oRjXuNJCAxpeBLXQzeuRi7pTDHWJ/uU3qdlBNBOkTB3B zOAgGBBc/oo5kplMkJ0M+Fh6ou9SWU/OgJoF25qXOr+gQnpOjEFQ7DUYK 0zGshZSEXHpc4RUs3RR7eVzRKLb2nMnuQ4eg1geM9dfPF2AyNDa28oTug k=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: Ez1uaDVp3YNnZmB8o5hfowhFbk2KXYlLwJxDccw/0cocOpw8NZX3shD+l6qDoFjxPAEDwqU+21 OkEm8iHYWXcrUFUH4rnB/OVuMPs4Rs1uOwvvVz3P4ygMGUVS/J3rbpxF+jfi8t6sM3QyqCl5QP MYsEpzjII7cXrkHHFoK4hyWFWMzBZk83ag4YEeFS702AY3X//fK7BMRBNiYDFdpyOdF6FFUqvZ xjKYuJLYKaVRIMOWGYNqa2jlR0nHSbodwpe5BmlpNF/YnKLWBwhJLJM8ZEYp0PbiXTvO0faacH Nr8= X-SBRS: 2.7 X-MesageID: 24576917 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24576917" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 21/60] history reporting (nfc): Rename some module variables, remove "cache" Date: Fri, 14 Aug 2020 18:21:26 +0100 Message-ID: <20200814172205.9624-22-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" This makes the code terser and easier to read. No functional change. Signed-off-by: Ian Jackson --- Osstest/HistoryReport.pm | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index a0565b6a..61b1ee8f 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -40,7 +40,7 @@ BEGIN { use POSIX; -our @cache_row_key_cols; +our @key_cols; our %cache; @@ -51,11 +51,11 @@ our $rows_previous = 0; our $rows_today = 0; our $rows_hit = 0; -sub cache_set_key_cols { @cache_row_key_cols = @_; } +sub cache_set_key_cols { @key_cols = @_; } -sub cache_row_key ($) { +sub key ($) { my ($jr) = @_; - return join $; , map { $jr->{$_} } @cache_row_key_cols; + return join $; , map { $jr->{$_} } @key_cols; } sub cacheable_fn ($$$) { @@ -107,7 +107,7 @@ sub cache_read_previous ($) { s{\%([0-9a-f]{2})}{ chr hex $1 }ge; $ch->{$k} = $_; } - $cache{cache_row_key($jr)} = $jr; + $cache{key($jr)} = $jr; } close H; } @@ -116,7 +116,7 @@ sub cache_row_lookup_prep ($) { my ($jrr) = @_; $rows_today++; - my $cacherow = $cache{cache_row_key($$jrr)}; + my $cacherow = $cache{key($$jrr)}; if ($cacherow) { $$jrr = $cacherow; $rows_hit++; From patchwork Fri Aug 14 17:21:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714939 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9875B618 for ; Fri, 14 Aug 2020 17:27:43 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 741FD20791 for ; Fri, 14 Aug 2020 17:27:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="HPJobP6Z" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 741FD20791 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTG-0003vk-Dy; Fri, 14 Aug 2020 17:26:26 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dRX-00024Q-J0 for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:24:39 +0000 X-Inumbo-ID: ae87fef9-d58f-4359-9043-9c9673ed5038 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id ae87fef9-d58f-4359-9043-9c9673ed5038; Fri, 14 Aug 2020 17:22:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425759; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=PY1/CXlgsiMePo4mO4fgMVytfc+lxDhand1L2bBROQs=; b=HPJobP6ZUeEYm94IoQDkSES+8J1Fc/luYOxjnEgMPHHb/LK+YK+RIMoI F6PsnDijKOk3VW4ER7ygCJ5z8Adr6aB8Cq6aus4+O+SQq0SOz2HbITDFU KZ1zKkfMAz2ApN71MnB4/jLyF86Nzv4X8ArWtr3RIEw4SOuN1a/E1FdtX 0=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 7ezcx7x4srd1zeCAkn0zObS+aQXFWeJkpcnYSbEvKrACqJXXyQgttJnGnVtsJ8cn4bvrcQHsJk 6jMVpr67Y+3gu1DrpxqDMFsGWsCmEow3xvhkNvxfYopxRpFUJQTzATY2WoF+CnWyzlyHlsAtSB 6S2JtmytY04WZ0io3dNteeEGLmXYG+Uz51EGsz0KpTSgv0VV5YKI9BYNgsovxMWYa8ev0khE5S ABi5SCz+IULYYyfrMmmmeB67GmknN+9tAlUQCNvf22W3BJtK4qCExs7h9n/GSHNv5X6s0dV+6G fpo= X-SBRS: 2.7 X-MesageID: 24576918 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24576918" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 23/60] history reporting (nfc): Documentation for the new module Date: Fri, 14 Aug 2020 18:21:28 +0100 Message-ID: <20200814172205.9624-24-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Signed-off-by: Ian Jackson --- Osstest/HistoryReport.pm | 27 +++++++++++++++++++++++++++ sg-report-host-history | 6 +----- 2 files changed, 28 insertions(+), 5 deletions(-) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index cb8c6ca1..6e5ed6ec 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -51,6 +51,33 @@ our $rows_previous = 0; our $rows_today = 0; our $rows_hit = 0; + +# History report query and processing cache +# +# Basic principle: +# +# Each HTML output file corresponds to the rows of one +# primary query. That primary query yields rows containing +# (at least) the columns passed to cache_set_key_cols. +# +# The HTML files contain caches of per-primary-row queries and +# computations. Each such subcomputation is identified by a +# $cachekey, which is assigned by the calling code (but should be +# alphanumeric). The memoisation is keyed off the key columns and +# $cachekey, so the subcomputation should not depend on anything else. +# +# The API is based on a %$jr. +# +# %$jr contains one entry for each for each key columns. +# +# It also contains one entry for each cached subcomputation. +# The keys for these entries are "\%$cachekey". + +# These routines all expect the file handle ::DEBUG +# They are not reentrant and can only be used for one HTML +# output file in one program run. + + sub cache_set_key_cols { @key_cols = @_; } sub key ($) { diff --git a/sg-report-host-history b/sg-report-host-history index a195bb21..e281986d 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -204,11 +204,7 @@ END my $inrows = $hosts{$hostname}; print DEBUG "FOUND ", (scalar @$inrows), " ROWS for $hostname\n"; - # Each entry in @$inrows is a $jr, which is a hash - # It has keys for the result columns in mainquery - # It also has keys '%' (yes, with a literal '%') - # which are the results of per-job queries. - # The contents of $jr for each job is cached across runs. + # Each entry in @$inrows is a HistoryReport $jr my @rows; foreach my $jr (@$inrows) { From patchwork Fri Aug 14 17:21:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714877 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6A3A7618 for ; Fri, 14 Aug 2020 17:26:54 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4598820708 for ; Fri, 14 Aug 2020 17:26:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="Y2qyCBW4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4598820708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTG-0003ve-5R; Fri, 14 Aug 2020 17:26:26 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQE-00024H-08 for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:18 +0000 X-Inumbo-ID: e2150a9a-a58e-4e3d-bdbb-40a0a3959b5b Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id e2150a9a-a58e-4e3d-bdbb-40a0a3959b5b; Fri, 14 Aug 2020 17:22:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425737; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=XiPWejwwXpw9o1nBvswJoP0ieCDSqMIwE8v0EYPViaY=; b=Y2qyCBW4529PeSEvr2El4si8IGYS07p1Q4dz7yW+1TU9hIHeTejDkxuc gHrQqdlhmtsPV/FhsTg3ze5xB/zEf6FZll0fL8sqfVqd+4xYoOmtpFVo1 tB4UenCTqfGuQf2yv2i9NxKDk2vI734Ro4XIW6q0XGZtV0ImsvgtxvHF0 E=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: HYCUijgYzXrBpfqBRfw8aIUcGquuKN9q987Qx16Wb2/PEo91PXvCCkNhUFgJgUWqw5g4gDbNMv 5bJNOhVsA6XG5rajEe9HyLxpOjtPj5rgiy5hARCtmwRqCXAO8aPpvxxaV7hFYLPpQzEe+rzyWP yHEc1dPOY5LpLyaVnPj0KwlddE6eLOT4JYa70Cj85jsoPrzS+RoBw90Vm0L3vnRdCtjCqwaiJ6 Q8H45BsWJGDNhDKyp6uCQUhBUGzN+A50AiUqCIla9EiTMYBfMzlWQRE8Q01XPkpONVrYmn8l4f NmU= X-SBRS: 2.7 X-MesageID: 25487820 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="25487820" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 24/60] history reporting: Cache data limit now in History module Date: Fri, 14 Aug 2020 18:21:29 +0100 Message-ID: <20200814172205.9624-25-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Replace the ad-hoc query-specific limit strategy in sg-report-host-history with a new, more principled, arrangement, in HistoryReport. Signed-off-by: Ian Jackson --- Osstest/HistoryReport.pm | 38 ++++++++++++++++++++++++++++++++------ sg-report-host-history | 14 +++++--------- 2 files changed, 37 insertions(+), 15 deletions(-) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index 6e5ed6ec..b576c02a 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -31,7 +31,7 @@ BEGIN { cacheable_query cacheable_fn cache_write_entry - cache_report_stats + cache_finish ); %EXPORT_TAGS = (); @@ -43,6 +43,7 @@ use POSIX; our @key_cols; our %cache; +our @previous; our %q_count; our %q_misses; @@ -50,6 +51,7 @@ our %q_misses; our $rows_previous = 0; our $rows_today = 0; our $rows_hit = 0; +our $rows_extra = 0; # History report query and processing cache @@ -73,6 +75,12 @@ our $rows_hit = 0; # It also contains one entry for each cached subcomputation. # The keys for these entries are "\%$cachekey". +# Limit strategy +# +# It is up to the caller to make a primary query that returns the rows +# that it is interested in. They should be in (perhaps approximate) +# reverse time order. + # These routines all expect the file handle ::DEBUG # They are not reentrant and can only be used for one HTML # output file in one program run. @@ -118,7 +126,6 @@ sub cache_read_previous ($) { } %cache = (); for (;;) { - $rows_previous++; $_ = // last; next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$}; my $jr = {}; @@ -134,6 +141,7 @@ sub cache_read_previous ($) { s{\%([0-9a-f]{2})}{ chr hex $1 }ge; $ch->{$k} = $_; } + push @previous, $jr; $cache{key($jr)} = $jr; } close H; @@ -169,18 +177,36 @@ sub cache_write_entry ($$) { print $fh " $'"; $whash->($jr->{$hk}); } + $jr->{'% written'} = 1; print $fh " -->\n"; } -sub cache_report_stats ($) { - my ($what) = @_; - print ::DEBUG "CACHE $what read=$rows_previous hits $rows_hit/$rows_today"; +sub cache_finish ($$) { + my ($fh, $what) = @_; + + my $nprevious = @previous; + + # Write as many rows again. This saves re-querying in case of + # flapping etc. + for (;;) { + last if + $rows_extra >= $rows_today && + ($rows_today + $rows_extra >= @previous * 0.9); + my $jr = shift @previous; + last unless $jr; + next if $jr->{'% written'}; + cache_write_entry($fh, $jr); + $rows_extra++; + } + + print ::DEBUG + "CACHE $what read=$nprevious hits $rows_hit/$rows_today"; for my $cachekey (sort keys %q_count) { my $total = $q_count{$cachekey}; my $hits = $total - ($q_misses{$cachekey} // 0); print ::DEBUG " $cachekey=$hits/$total"; } - print ::DEBUG "\n"; + print ::DEBUG " extra=$rows_extra\n"; } 1; diff --git a/sg-report-host-history b/sg-report-host-history index e281986d..2ad2134d 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -119,7 +119,7 @@ our %hosts; sub mainquery ($) { my ($host) = @_; - our $runvarq //= db_prepare(<execute($host); @@ -233,11 +234,6 @@ END my $runvarq_misses = 0; foreach my $jr (@rows) { - if ($wrote++ >= $limit) { - cache_write_entry(\*H, $jr); - next; - } - #print DEBUG "JR $jr->{flight}.$jr->{job}\n"; my $ir = cacheable_query($infoq, $jr, 'i'); my $ar = cacheable_query($allocdq, $jr, 'a'); @@ -338,11 +334,11 @@ END print H "\n"; + cache_finish(\*H, "$hostname"); + close H or die $!; rename "$html_file.new", "$html_file" or die "$html_file $!" if $doinstall; - - cache_report_stats("$hostname"); } foreach my $host (@ARGV) { From patchwork Fri Aug 14 17:21:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714867 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D3EA618 for ; Fri, 14 Aug 2020 17:23:43 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 69CBB20708 for ; Fri, 14 Aug 2020 17:23:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="AyIeVP8Q" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 69CBB20708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPu-0002hH-Vh; Fri, 14 Aug 2020 17:22:58 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPt-00024H-Vt for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:22:58 +0000 X-Inumbo-ID: d0438354-2241-4070-8117-2317f027fe47 Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id d0438354-2241-4070-8117-2317f027fe47; Fri, 14 Aug 2020 17:22:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425736; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=+4jPrjqYjTzITy9iRlTBDlMSXCVzvJckoAZyFOWRXhg=; b=AyIeVP8QwxNentNvKB7UO+8bwEdU7Y4Fo8JaMpdJalYVbP3va2Uj7eK9 xKZrkvKbqCTyaZkiYRYnT81Wl0P+MeHJTj52MAJvXoPNSSnkgJh2Q66/2 TB8sTe/qRUERJoXigej2c7HFexui+6dJM/QwA5aRv3XuM8L2jrPIJVCA5 Q=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: FsFWVrJiaHsiJH2DeW2YJWhoZEqur+TUK8B7JrNipjEFplXV+R638crGsqLuE9aNOUXNEAM4OY lCBLIKTgdhkeMogFA27uDOBlv4hL4PdJEUdTDhOYqtvgDHBfv9b4f13HfExo2AvoHbC2qpdHXD pyETK+bvhTwW1sOOlHHeJYzcOrthFic+Q0HlvTp65DugHfLkkjCytVPM5aETZQsplWkyoj2rbZ HvCOwjx2z4DTldswSYNlddugo2+0jq2PuBh5aY4h9jAi6nBB/kGw88DNV0weQLULO/tatq7p3W xXQ= X-SBRS: 2.7 X-MesageID: 25487819 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="25487819" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 25/60] history reporting: Print debug for cache misses Date: Fri, 14 Aug 2020 18:21:30 +0100 Message-ID: <20200814172205.9624-26-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Signed-off-by: Ian Jackson --- Osstest/HistoryReport.pm | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index b576c02a..b593f207 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -155,6 +155,9 @@ sub cache_row_lookup_prep ($) { if ($cacherow) { $$jrr = $cacherow; $rows_hit++; + } else { + print ::DEBUG "CACHE MISS ", + (join " ", map { $$jrr->{$_} } @key_cols), "\n"; } } From patchwork Fri Aug 14 17:21:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714973 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 09E9A1392 for ; Fri, 14 Aug 2020 17:28:03 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D985C2078D for ; Fri, 14 Aug 2020 17:28:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="UdLS7Lvu" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D985C2078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dU5-0005av-9k; Fri, 14 Aug 2020 17:27:17 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dS5-00024H-4Z for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:25:13 +0000 X-Inumbo-ID: 6de50cb7-ecf3-48f5-a5c0-fa71292e5f28 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 6de50cb7-ecf3-48f5-a5c0-fa71292e5f28; Fri, 14 Aug 2020 17:22:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425760; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=7jO5oa9Cvf0sgduE8CWd2v2zY8hx6IqAdysPRIm7Ppc=; b=UdLS7LvuwsPPEoo7qra2GtUEy4x4i/ab91JPmEo91GRA7+w9BDyHlcbP MlbVE+pJ/ED65AJUOxSkX51TodPajmfFQxiYc3h3dS3xsPrRmBs19Aj5G S1TKN+BCQGfSBoYFkeV3uvWKwKt/vGsIreKuYKFXbmYxpPjqkiPaKleIi M=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: ph4SRXbhe4sIQGmZIvgHxjmu9GFETUGky+FvNjxdOr6XZ2LPKMtcZLT93sSQz0cvokwQ8bZHRf njt+SaBnlIADqXi8Lc3wtOhOAvsb2Wo8Qkgf6likKSwlPSXopO8aTPGKX+mBGtszU5YkwQwLX1 kS7Sta9WODbOwSK44PStqtwq0+9WSbHGXTRKCJP6y/VFaNcGKvJPP8ukZoZzLVY6OruwA+jCjj EgG+wWP5d7AS060Nx5ouybvJS2cVRqA3/MEZcvfYoB8TIGISEiCIF2xp0ZBlM5zyMJZ4IbZljN dko= X-SBRS: 2.7 X-MesageID: 24576919 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24576919" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 26/60] history reporting: Improve an error message slightly Date: Fri, 14 Aug 2020 18:21:31 +0100 Message-ID: <20200814172205.9624-27-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Signed-off-by: Ian Jackson --- Osstest/HistoryReport.pm | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index b593f207..6a23cfba 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -128,6 +128,7 @@ sub cache_read_previous ($) { for (;;) { $_ = // last; next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$}; + my $orig = $_; my $jr = {}; my $ch = $jr; foreach (split / /, $1) { @@ -136,7 +137,7 @@ sub cache_read_previous ($) { $jr->{'%'.$&} = $ch; next; } - s{^(\w+)=}{} or die; + s{^(\w+)=}{} or die "$orig -- $_ "; my $k = $1; s{\%([0-9a-f]{2})}{ chr hex $1 }ge; $ch->{$k} = $_; From patchwork Fri Aug 14 17:21:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714885 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E9B54618 for ; Fri, 14 Aug 2020 17:27:18 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C49F320708 for ; Fri, 14 Aug 2020 17:27:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="JJ4fV94L" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C49F320708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTi-0004d9-3k; Fri, 14 Aug 2020 17:26:54 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQ8-00024H-WF for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:13 +0000 X-Inumbo-ID: 1bf497fa-4c22-4c20-9e89-315c3d95e6e1 Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 1bf497fa-4c22-4c20-9e89-315c3d95e6e1; Fri, 14 Aug 2020 17:22:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425737; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=jayi6sGhQIIc8z8XNhCFRTqmNCWkgiucBJ6yZix0Kvw=; b=JJ4fV94L/BV3JCqWJgb3gLa1souVWmWSSG+7j6yTp9JGWu97A6aBst/5 RFytzl0zIfgMMligeZlHt7GZuPa3KlvC+OBSFkUaTOzR3Yvv5V6NSjloK IcZTvszs9Dl82rF8j/t/yThkG0E+JmpIBFuvKPMak6SR7pIYr/51IxB9g g=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: ZM+qnAmamfS8aNtrEEM81CpDEcrPmI36jzG2YueP+r9+lJqBMkYcGDA64gvfd41lerJSmrHH1t +zDi/NW6mYv0DspCJq7gwWk+nNj5jh+TTYk2TY00ZSKcBqqaQghTgKijta7j1QRWvt0w+rJD54 5Q8cB9JMVbcEh0DqcqZJfXk9yXgqdJ39zrT/ZuGVqNp1/z4UP8FSKQDwoimP0DkfgiBe6QRLKp V3hmSTtp+Dg5byve+VY2YnUAlTX4+Pemy0m2kTxGejrfrD9v1OwZWy7VaxtYrYKI4FhB6A7W7/ nBE= X-SBRS: 2.7 X-MesageID: 25487821 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="25487821" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 27/60] sg-report-host-history: Write cache entry for unfinished jobs Date: Fri, 14 Aug 2020 18:21:32 +0100 Message-ID: <20200814172205.9624-28-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" We have to also check ->{finished}, rather than the existence of a row at all, since now unfinished jobs can appear in the cache. Because the cache key includes the job status, when the job becomes finished the cache entry will be invalidated. Signed-off-by: Ian Jackson --- sg-report-host-history | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sg-report-host-history b/sg-report-host-history index 2ad2134d..6bf14aa2 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -214,7 +214,8 @@ END cache_row_lookup_prep(\$jr); my $endedrow = cacheable_query($endedq, $jr, 'e'); - if (!$endedrow) { + if (!defined $endedrow->{finished}) { + cache_write_entry(\*H, $jr); #print DEBUG "no-finished\n"; next; } From patchwork Fri Aug 14 17:21:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714899 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5AD001392 for ; Fri, 14 Aug 2020 17:27:25 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 373E620708 for ; Fri, 14 Aug 2020 17:27:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="dKt2aQ8i" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 373E620708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTh-0004bo-Jy; Fri, 14 Aug 2020 17:26:53 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQJ-00024H-0D for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:23 +0000 X-Inumbo-ID: 759d6e29-9345-4902-94c8-f563ba73d13a Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 759d6e29-9345-4902-94c8-f563ba73d13a; Fri, 14 Aug 2020 17:22:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425738; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=13kzgnNv/DMd96ulOM4YWyYW/kogwexoOjaZdcpN4WA=; b=dKt2aQ8iRZCfWknWLCzg5UHEVwnqiJ4qM7w6Sp8pAawJYdGpQrICPZFo TRYkNN547Pmnl936WUvGqzY1bJ9m2+U7ob0Yf4WDovmUoHHKqc4XeWJWp XD7cm9E1JROBWIU5AIhO57ppwQOSdmn/5n5gycCZv6yZPocQ+4GsUsCu3 w=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: H79NiA21cInK6e0sIQdd3fn5GZjwVzTGAcsKvJYwZIBjmmzdHsCd//5N88AGM2KIvI1jJq3R3Y NCZRWfyzXAg5TTzinxlVcOByTQJJoPbxKesN6t1UegR1C2pm2SMXDkBhPSDRsRQyrc79U06Uu+ MLNoJmPTyY+26W5ImrojK2gR6xl5RFmQW/gXdFlULgf/bptrn7qrBeIqkosmc6nqyER0j1MUHi Fukm/T82xwL4Iqai6OzYih9SmFPEEHK/oShfXP2OsF/Z36rmY8L+0kNaAnmlwzZb9lHewVc9if HpM= X-SBRS: 2.7 X-MesageID: 25487822 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="25487822" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 28/60] parallel by fork: Break out into HistoryReport Date: Fri, 14 Aug 2020 18:21:33 +0100 Message-ID: <20200814172205.9624-29-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Move this code from sg-report-host-history to HistoryReport, so that it can be reused. No significant functional change. Some changes to debug messages. Signed-off-by: Ian Jackson --- Osstest/HistoryReport.pm | 51 ++++++++++++++++++++++++++++++++++++++++++++++++ sg-report-host-history | 49 +++++++++------------------------------------- 2 files changed, 60 insertions(+), 40 deletions(-) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index 6a23cfba..0b206de4 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -19,6 +19,9 @@ package Osstest::HistoryReport; use strict; use warnings; +use Osstest; +use Osstest::Executive; + BEGIN { use Exporter (); our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS); @@ -32,6 +35,7 @@ BEGIN { cacheable_fn cache_write_entry cache_finish + parallel_by_fork ); %EXPORT_TAGS = (); @@ -213,4 +217,51 @@ sub cache_finish ($$) { print ::DEBUG " extra=$rows_extra\n"; } +our %children; +our $worst = 0; +our $whoami; + +sub wait_for_max_children ($) { + my ($lim) = @_; + while (keys(%children) > $lim) { + $!=0; $?=0; my $got = wait; + die "$! $got $?" unless exists $children{$got}; + my $host = $children{$got}; + delete $children{$got}; + $worst = $? if $? > $worst; + if ($?) { + print STDERR $whoami."[$$]: [$got] failed for $host: $?\n"; + } else { + print ::DEBUG "REAPED [$got] $host\n"; + } + } +} + +sub parallel_by_fork ($$$$) { + my ($set_whoami, $maxjobs, $tasks, $fn) = @_; + # does $fn->($_) foreach @$tasks + # but in parallal and then exits. + # entries in @$taskarray should be suitable for print in messages. + # db is reopened in each child. + + $whoami = $set_whoami; + undef $dbh_tests; + + foreach my $task (@$tasks) { + wait_for_max_children($maxjobs); + + my $pid = fork // die $!; + if (!$pid) { + opendb_tests(); + $fn->($task); + exit 0; + } + print ::DEBUG "SPAWNED [$pid] $task\n"; + $children{$pid} = $task; + } + + wait_for_max_children(0); + exit $worst; +} + 1; diff --git a/sg-report-host-history b/sg-report-host-history index 6bf14aa2..0a2e9904 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -377,43 +377,12 @@ db_retry($dbh_tests, [], sub { computeflightsrange(); }); -undef $dbh_tests; - -our %children; -our $worst = 0; - -sub wait_for_max_children ($) { - my ($lim) = @_; - while (keys(%children) > $lim) { - $!=0; $?=0; my $got = wait; - die "$! $got $?" unless exists $children{$got}; - my $host = $children{$got}; - delete $children{$got}; - $worst = $? if $? > $worst; - if ($?) { - print STDERR "sg-report-flight[: [$got] failed for $host: $?\n"; - } else { - print DEBUG "REAPED [$got] $host\n"; - } - } -} - -foreach my $host (sort keys %hosts) { - wait_for_max_children($maxjobs); - - my $pid = fork // die $!; - if (!$pid) { - opendb_tests(); - cache_read_previous("$htmlout/$host.html") if $read_previous; - db_retry($dbh_tests, [], sub { - mainquery($host); - reporthost $host; - }); - exit(0); - } - print DEBUG "SPAWNED [$pid] $host\n"; - $children{$pid} = $host; -} - -wait_for_max_children(0); -exit $worst; +parallel_by_fork('sg-report-flight', $maxjobs, [ sort keys %hosts ], sub { + my ($host) = @_; + cache_read_previous("$htmlout/$host.html") if $read_previous; + db_retry($dbh_tests, [], sub { + mainquery($host); + reporthost $host; + }); + exit(0); +}); From patchwork Fri Aug 14 17:21:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714947 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D7C10618 for ; Fri, 14 Aug 2020 17:27:47 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B001620791 for ; Fri, 14 Aug 2020 17:27:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="W3936b+d" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B001620791 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTQ-00047B-Ha; Fri, 14 Aug 2020 17:26:36 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQT-00024H-0L for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:33 +0000 X-Inumbo-ID: 18e06ca4-8e91-4e14-8870-2ebf0dd220ae Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 18e06ca4-8e91-4e14-8870-2ebf0dd220ae; Fri, 14 Aug 2020 17:22:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425738; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=Q2DHznUSzDCiwJvbTpK+EsflStSwpJzgic3yYjwBj88=; b=W3936b+dKOxBpLppvk2MLg3vgGqGkrBhwO+0A3FffcuuEyOgP9R+ZcaQ FNKO/YM93Rb8pAUT7ayj6pia/7Hv8AsMMjU/7i1WH9NLUPufGScqLFmfD 4BYd6/MmzYdV4kYk/rrNlR2dmNRud/QU0kWzBnljbGA7OfMsgrUztn/HO g=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: mt2cdu9oNkD8xicT9fbKvliQ7A3TX45j3aCLdnPz5GAglhXZhphN5SIeIVjMX23WqG1ZqrWqk0 jyZrJH8rGukQPspN49o847UUVUmOM+/0nh0rQgRtHecmCtvPKdmIq2gV0LHt4W5xI0NO6AvUhy Tus+KnbGK9nIYqzYELKs44iCbUJxtIOJ10PemoNxBVoSdqfgdBoA+IxL2WTE/WhKxwliMYSFNq 6sQRRK9/n20WzhR0BwjaAU9YNMgZvvtK+/Pp/Q5j1d7OgxqeRsKzgMHAo0c1q8XH0ACIIehyph 9LY= X-SBRS: 2.7 X-MesageID: 25487825 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="25487825" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 29/60] parallel by fork: Disconnect $dbh_tests as well as undefing it Date: Fri, 14 Aug 2020 18:21:34 +0100 Message-ID: <20200814172205.9624-30-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" If the caller is buggy and has statement handles still open, they can still "work" even if we have thrown away the db handle. Where, after forking, "work" means "use the same connection in multiple processes simultaneously, without locking". This could result in arbitrary crazy nbehaviour (eg, TLS crypto failures). No functional change with existing callers since they don't have this bug. Acked-by: Ian Jackson --- Osstest/HistoryReport.pm | 1 + 1 file changed, 1 insertion(+) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index 0b206de4..f0571189 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -245,6 +245,7 @@ sub parallel_by_fork ($$$$) { # db is reopened in each child. $whoami = $set_whoami; + $dbh_tests->disconnect(); undef $dbh_tests; foreach my $task (@$tasks) { From patchwork Fri Aug 14 17:21:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714933 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 47FE013A4 for ; Fri, 14 Aug 2020 17:27:42 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 23F6B20791 for ; Fri, 14 Aug 2020 17:27:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="Qiw9NEXO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 23F6B20791 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTa-0004L2-TA; Fri, 14 Aug 2020 17:26:46 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQO-00024H-0H for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:28 +0000 X-Inumbo-ID: daa8a508-c75e-4f20-aff3-e79041e9cc43 Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id daa8a508-c75e-4f20-aff3-e79041e9cc43; Fri, 14 Aug 2020 17:22:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425738; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=Cb4uZBmyPAuwXWIZ5X4FCPlExKibgY6AbAI+s7FEbxM=; b=Qiw9NEXOVWOoiiq26utDZV1x396xP6pLIk8y/MWC+kvB2rlGoupfjdJr 4WX7+aH5Vv0QmHXW7M1sDY8JfIRGH4fgOiPJ0lGzyzvG8kqv+IoMvyOuT vq45plBzz/K4hSlrsnlGQqU19R5ZYoUk+RKWG7Utp7MgPtq8MY/xoO5s6 I=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: bAJhGQySFnlcl+mQrLKaBRSO9NFhcConVYT6M9+EgceM8Wg+EOnZ1bETigIBj70kHwyKhqRkVA 5PmKrbd0iNhKJUPGjMRPDHeeMbVk+koXDgG6Tb6YiBdRbb5/WUUDplxpq17Eo8OWgdcQ8i9IXh WzbkXgA1KW/zXrENHfa3BZCWXe3gy1iNOSIizrESui2vUVN7eHX4xJUQ1CejK1AvKRs6wyRgnb rsrY4TOmu1iKP1Hji/YIk+0HahDQbyPp913ZrEdxPG6zRElI2BLZGCP0qWnCMW6Bdbk4blK0ZI Pes= X-SBRS: 2.7 X-MesageID: 25487824 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="25487824" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 30/60] parallel by fork: Fix a variable name to $task Date: Fri, 14 Aug 2020 18:21:35 +0100 Message-ID: <20200814172205.9624-31-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" This code came from sg-report-host-history where tasks were hosts. But in the more general context, the names are wrong. Signed-off-by: Ian Jackson --- Osstest/HistoryReport.pm | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index f0571189..695d744a 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -226,13 +226,13 @@ sub wait_for_max_children ($) { while (keys(%children) > $lim) { $!=0; $?=0; my $got = wait; die "$! $got $?" unless exists $children{$got}; - my $host = $children{$got}; + my $task = $children{$got}; delete $children{$got}; $worst = $? if $? > $worst; if ($?) { - print STDERR $whoami."[$$]: [$got] failed for $host: $?\n"; + print STDERR $whoami."[$$]: [$got] failed for $task: $?\n"; } else { - print ::DEBUG "REAPED [$got] $host\n"; + print ::DEBUG "REAPED [$got] $task\n"; } } } From patchwork Fri Aug 14 17:21:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714913 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DEA5813A4 for ; Fri, 14 Aug 2020 17:27:32 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B875A20708 for ; Fri, 14 Aug 2020 17:27:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="HzWNdqZk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B875A20708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTx-0005FO-27; Fri, 14 Aug 2020 17:27:09 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQY-00024H-0Y for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:38 +0000 X-Inumbo-ID: fe115fab-4164-4cf4-a2e1-fd75a7cc3094 Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id fe115fab-4164-4cf4-a2e1-fd75a7cc3094; Fri, 14 Aug 2020 17:22:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425739; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=0DVXE2AqFVp60oYFsg0RponXDX/ELl0fuRQ+BVVXl9o=; b=HzWNdqZkyV0t5uHJQkeXWZWg78UeprqYbDqqwTtXsYBUmZmxkiCZj82s tHtwuunSxWX0d9iVQR80FYx26KKW/OnpilEhN04vO4AvJUiyJgfRZo6DX I3b6qvtaBIdtfwnaU1O9rqtth+6Vdfq/JIG0oKTlFDqv6EsJim4uyr6lP Q=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 9BULI4WaFnmOP0IWATDe50lEkRiAS7v+k23D/fvybUNsECBZfnrUx7zmA/yS9oUAUzi3cngFFh KLi9kzPWUexJx9iAIdYN7KSa28buvQUut7rC6cYqI06sMB8Aas5tKTrTJnXGXPvcUV9A0TgP2W Fx4k/thKN3yAZH44l5lCwQmHyFKVr4GoRj/6ekHxYvoa5BMnjvaAIOrPMLk1mpyHjRvfjNFhE8 oukIIb3S+yIzoS9d2iT0NywaUd7ls4UkXBujJMv2CNAMJZSDLd2oClxdrBr8Fpl/il5tYZUHhm ehY= X-SBRS: 2.7 X-MesageID: 25487826 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="25487826" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 31/60] sg-report-job-history: Prep for fork: Move $buildsq query Date: Fri, 14 Aug 2020 18:21:36 +0100 Message-ID: <20200814172205.9624-32-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" We will need to prepare this once per (job,branch) so that it works when we do each of those in a different process. Signed-off-by: Ian Jackson --- sg-report-job-history | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sg-report-job-history b/sg-report-job-history index 6008ca72..47fc7a62 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -106,15 +106,15 @@ sub add_revisions ($$$$) { } } -our $buildsq= db_prepare(< X-Patchwork-Id: 11714959 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2C88B618 for ; Fri, 14 Aug 2020 17:27:51 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 847CD20791 for ; Fri, 14 Aug 2020 17:27:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="bx2NKp19" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 847CD20791 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dUC-0005tg-6r; Fri, 14 Aug 2020 17:27:24 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQd-00024H-0k for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:43 +0000 X-Inumbo-ID: 760e6777-aa2f-4a1a-a86d-ef4f670b4791 Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 760e6777-aa2f-4a1a-a86d-ef4f670b4791; Fri, 14 Aug 2020 17:22:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425738; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=zLcHhKo1cmGICQ2dzNfIzPI8l1nwyFvOoh6iuMg7/JM=; b=bx2NKp19tbBnW4q0/AagMciZEdpWgUiet9wg5LeAZrUMhA9quBE+WMLk KzSrHr76JyeDhlfDLHVWj5XcTiByyOA8D2bEmm1v8NZxQ4iOXRxwCU+6e 14avMiGeLqgnQVKntdc7AdjU+dHoy0wOymZxVY7oA5e46fyL2zr5gIvs2 4=; Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: VZVhd9rwqwtWKEI/K/ZXRLl7EVsrakjNSOWIcBpLpsx/jCDOZkn0BqdG8WFs+zCFYIBTlTukA0 h7juCpDFUL/KX80J/DYcnhygLz85c0/ADnz8CNAp8ChooR4IyJhyPmgMPJqOiTSZeKlFGSxJoF oXoFf7hvZCFGbhzB9JwWvN4H/e2+8KJA9HOa9emrMyZqkcoNxmvH25IUeNJrJiyv22nIZCGX35 vQVmyQT2Vis+KM7uYz8clDy31YmfNHWZrKS9QQWQcFoEDnJkUxZ/d8/Kyx8FhoRGLzOhTCxuj6 ARY= X-SBRS: 2.7 X-MesageID: 24725183 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24725183" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 32/60] sg-report-job-history: Prep for fork: Move $revisionsq query Date: Fri, 14 Aug 2020 18:21:37 +0100 Message-ID: <20200814172205.9624-33-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" We will need to prepare this in add_revisions so that it works when we do each (job,branch) in a different process. It is OK that it is still global, becauswe we only call add_revisions in the children. Signed-off-by: Ian Jackson --- sg-report-job-history | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/sg-report-job-history b/sg-report-job-history index 47fc7a62..424053f1 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -89,15 +89,16 @@ if (defined($flight)) { push @jobs, $job; } -our $revisionsq= db_prepare(<execute($flightnum, $j); while (my $row= $revisionsq->fetchrow_hashref()) { my $n= $row->{name}; From patchwork Fri Aug 14 17:21:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714873 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2C58C618 for ; Fri, 14 Aug 2020 17:23:55 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 06CA120708 for ; Fri, 14 Aug 2020 17:23:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="NT7a8dJ6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 06CA120708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPc-0002KT-Nk; Fri, 14 Aug 2020 17:22:40 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPb-00024Q-FV for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:22:39 +0000 X-Inumbo-ID: 9ad93cbc-9ce4-49b5-8aee-e6cf3bc17b08 Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 9ad93cbc-9ce4-49b5-8aee-e6cf3bc17b08; Fri, 14 Aug 2020 17:22:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425739; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=ss5D/qaRmFTGSDiYRg9YFPjsfIqGXfS+GnKQIZiycbg=; b=NT7a8dJ62rE3kVYNyhByNSckCSwfoQNMxwZqyc0I6BrJ4C2n9i0dGRm6 6dbMl/AYiDnii4dYt7LAwlv+evGFqQ3ZV684s6teR/9s/JE1nGU7WlfUl GFpz7FzK0tIEh6nk74Fupci2aWfe78not4hYj/3Ikm6GFDe7w8WenbZY/ Y=; Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: YROxpqmpJnw+hqp3Viw9ketkdbRBZUlJk2QHsBCtgt++Wq0+86m8LUPrRx6JoHJoDcXgaVZcV3 yRR24GZIeL6spqBiCIh86SY80w+orkcRLa2RsmPYNNAjaVoLJc50qx5VSBrEHMeOKhyU1yWA+I lNJVQUeCcdX4YvIK43+iTLu1oQkH32yrElpH+ZC4JPI0zr+vKp3enUwrMbi3xanjd6Z54HcJG+ UUmqtlT8DoyNysOTvgsk1j9wzMx5MtznEIr4lw1JqwDg9mZ+9KsZ3HtDdLZlf4oUVl0lEXW7Qi Hyc= X-SBRS: 2.7 X-MesageID: 24879737 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24879737" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 33/60] sg-report-job-history: Use fork-based parallelism Date: Fri, 14 Aug 2020 18:21:38 +0100 Message-ID: <20200814172205.9624-34-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" For now, one child per job (for all branches). This is already a speedup. Signed-off-by: Ian Jackson --- sg-report-job-history | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/sg-report-job-history b/sg-report-job-history index 424053f1..384a7670 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -26,11 +26,13 @@ use IO::Handle; use HTML::Entities; use Osstest::Executive; +use Osstest::HistoryReport; our (@blessings,@branches); our $limit= 100; our $htmlout; our $flight; +our $maxjobs=10; our $job; open DEBUG, ">/dev/null"; @@ -38,7 +40,7 @@ open DEBUG, ">/dev/null"; while (@ARGV && $ARGV[0] =~ m/^-/) { $_= shift @ARGV; last if m/^--?$/; - if (m/^--(job|flight)\=(.*)$/) { + if (m/^--(job|flight|maxjobs)\=(.*)$/) { $$1= $2; } elsif (m/^--(limit)\=([1-9]\d*)$/) { $$1= $2; @@ -300,6 +302,9 @@ sub processjob ($) { processjobbranch($j,$_) foreach @branches; } -db_begin_work($dbh_tests, []); -db_readonly_report(); -foreach my $j (@jobs) { processjob($j); } +parallel_by_fork('sg-report-job-history', $maxjobs, \@jobs, sub { + my ($j) = @_; + db_retry($dbh_tests, [], sub { + processjob($j); + }); +}); From patchwork Fri Aug 14 17:21:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714851 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6303A1392 for ; Fri, 14 Aug 2020 17:23:28 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3EAFF20708 for ; Fri, 14 Aug 2020 17:23:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="W+uVArh+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3EAFF20708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPm-0002Wo-TW; Fri, 14 Aug 2020 17:22:50 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPl-00024Q-Fp for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:22:49 +0000 X-Inumbo-ID: a392a876-06c2-417b-846d-04f0f9c402c4 Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id a392a876-06c2-417b-846d-04f0f9c402c4; Fri, 14 Aug 2020 17:22:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425739; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=7tEJzOFe1rNL6H1YNP7my17N/c5tTg3jF4HNaa1FTqs=; b=W+uVArh+J46cKo0AIaLCSPdrcO8P/2lwo2A4hGNxDuktWkAUi44VtjXn 9Mr74pWQKEqY9zmiPJdVANSeQ006YIWsBNmI+mkc0Ng2W2euhiXTawtuR dIC5zipGDr6Gpk/Tba8H08Dba+s4EkmAiBJlv4X8wS5+U8BVmJDY/lJ/R w=; Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 6ARKZM0gZtes4QSkjJYnK72EiWwb9mWVaqcYBhEIBbYXHm0QFvsTcppTGOls0TmEKKm2deWvsY ZgX6WQW0r2Hl77IPdYgblLsKlMfT/vZExgBVdEiEpWme5NskbP2zT0e0at+7y79lLLKavJmuSD 1D7Qqc7uG2I35bv5r72TKpNDq+7JezrRpeBNylQGm+MYuqGGQZxjW7fES0TYjlVW4ClYglW3AB T8abbixo/T81ZtDTd1i5sF13vAUiNwku12qePAXTIiH2L+qiH3twMe9N4WZ9KMHx18TO14stP3 +ZQ= X-SBRS: 2.7 X-MesageID: 24879738 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24879738" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 34/60] sg-report-job-history: Use one child per report Date: Fri, 14 Aug 2020 18:21:39 +0100 Message-ID: <20200814172205.9624-35-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Rather than one child per job, which then did one report per branch. This will mean we can use the cache machinery, which is rather global so wouldn't cope well with processing multiple job history reports within a process. Signed-off-by: Ian Jackson --- sg-report-job-history | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/sg-report-job-history b/sg-report-job-history index 384a7670..3307b281 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -296,15 +296,16 @@ END } } -sub processjob ($) { - my ($j) = @_; - processjobbranch($j,undef); - processjobbranch($j,$_) foreach @branches; +my @tasks; +foreach my $j (@jobs) { + push @tasks, $j; + push @tasks, "$j $_" foreach @branches; } -parallel_by_fork('sg-report-job-history', $maxjobs, \@jobs, sub { - my ($j) = @_; +parallel_by_fork('sg-report-job-history', $maxjobs, \@tasks, sub { + my ($task) = @_; + my ($job, $branch) = split / /, $task; # $branch might be undef db_retry($dbh_tests, [], sub { - processjob($j); + processjobbranch($job, $branch); }); }); From patchwork Fri Aug 14 17:21:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714917 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EF191618 for ; Fri, 14 Aug 2020 17:27:33 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CA4B420708 for ; Fri, 14 Aug 2020 17:27:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="S5h7qEMS" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CA4B420708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTN-00040C-9R; Fri, 14 Aug 2020 17:26:33 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQi-00024H-0y for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:48 +0000 X-Inumbo-ID: e8fe2ef3-73ed-437b-9f28-16e53105f8cb Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id e8fe2ef3-73ed-437b-9f28-16e53105f8cb; Fri, 14 Aug 2020 17:22:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425739; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=TQcIqqhFJdYsJk5BMaUT3ykNr/SrI/Z/YpnxsNoFOD0=; b=S5h7qEMS3QNYtw2m1bVexcC1+gi5OVUziYT27jMXipVgVOUnO4d2wsSx fhbMzrTegSizWhGefOi5F7/QY70FVpJyxmdNR978KksIJRSaKo1WUfZnc KNw14moBN0yQTy79WeXy6EHKwYtGdOsagUIuWqavtP4NdQDiiCQp2iqm0 E=; Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 13x5v+jN+N7cZeeL23w1/uVLruEaZDX2PvS1Bv/h4AYAyeBpFf/B+gq+xqzeB+qDtYrCXfVFnu gZfj7GBZIxapXcfPpRr0ZGlc/ceqdBEpRfm7+IocuNWp2+oKr8vQBhjqlTzTEA9+yR7YxyQBQb hFXTy4rrQ9503CK45bPzBhXP6Wka4F5ACTVtX9AD908rcXyd76lo6yGLC2oseULiDew0OAqZiK PO/QFex9tA7sQ9yc0CSD69keBbH+ZVMJQO6AgNrq3bvIcOZw+EIEpHEjcoWLo043AN2AHTc5yy 3j0= X-SBRS: 2.7 X-MesageID: 24725187 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24725187" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 35/60] sg-report-job-history (nfc): Have main program decide HTML filename Date: Fri, 14 Aug 2020 18:21:40 +0100 Message-ID: <20200814172205.9624-36-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Signed-off-by: Ian Jackson --- sg-report-job-history | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/sg-report-job-history b/sg-report-job-history index 3307b281..3f10a74c 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -109,8 +109,8 @@ END } } -sub processjobbranch ($$) { - my ($j,$bra) = @_; +sub processjobbranch ($$$) { + my ($j,$bra,$html_file) = @_; my $buildsq= db_prepare(< $html_file.new" or die "$html_file $!"; @@ -305,7 +303,8 @@ foreach my $j (@jobs) { parallel_by_fork('sg-report-job-history', $maxjobs, \@tasks, sub { my ($task) = @_; my ($job, $branch) = split / /, $task; # $branch might be undef + my $html_file = sprintf "history/%s/%s.html", $job, ($branch // 'ALL'); db_retry($dbh_tests, [], sub { - processjobbranch($job, $branch); + processjobbranch($job, $branch, $html_file); }); }); From patchwork Fri Aug 14 17:21:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714977 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6599B618 for ; Fri, 14 Aug 2020 17:28:05 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 41B8B20855 for ; Fri, 14 Aug 2020 17:28:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="cSx3K5O2" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 41B8B20855 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTJ-0003xH-Cl; Fri, 14 Aug 2020 17:26:29 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQn-00024H-1P for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:53 +0000 X-Inumbo-ID: 896d20ac-b3ff-4c88-86d6-f9658b05afd8 Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 896d20ac-b3ff-4c88-86d6-f9658b05afd8; Fri, 14 Aug 2020 17:22:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425740; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=SF6JrZam2+wPokpUiTs03Zat4tqLyBtCgJsqjxxZn6A=; b=cSx3K5O2yGIqk+n5pldrQ6ksDcSGif3/S9Iu0gydXzeECVyHh6OqvT9S b3f6Rxve1/82v6acfw7Olk0hiRuCKPo55TXLSJJFQvq97ccpJdyxo16L3 PdA0WGx90yoqVQYq0VPU5AGjUelnYFhTI52aoJ6L1qAys8Ys9x14xFvN3 c=; Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: yQR9GLMmyubOyPDCTK8gsCAWRoXoHZc8EdYCy67UCCHFoPuav9VuDh53vdLYGzafy4ico5lbEj k7NCKrjP6mEGV8dmisz00J3VZAfFOFae5Cb/MvQXdyJZZCqFai8uQHsXwCqVWo6YWSM7iaoa4v uifl8V955WKHTf9MJMiYGFBJo+MO8Y8yQ61qzmhgWNuoMnsxPDAFJsjqUWJgWFzsEjPDyB10GQ /W3QmIzLuzYW57rMag2nDahBaR//oVTTB5iOUw2NLT/xn05XaRO3Icf35S7KhkO1hfkHopzktp BzQ= X-SBRS: 2.7 X-MesageID: 24725188 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24725188" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 36/60] sg-report-job-history: Always write HTML output Date: Fri, 14 Aug 2020 18:21:41 +0100 Message-ID: <20200814172205.9624-37-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Previously, unlike sg-report-host-history, if you didn't specify --html-dir, it would would do a lot work to no effect. This is not useful and nothing calls it this way. So abolish this notion. Signed-off-by: Ian Jackson --- sg-report-job-history | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sg-report-job-history b/sg-report-job-history index 3f10a74c..7197a5f6 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -30,7 +30,7 @@ use Osstest::HistoryReport; our (@blessings,@branches); our $limit= 100; -our $htmlout; +our $htmlout = '.'; our $flight; our $maxjobs=10; our $job; From patchwork Fri Aug 14 17:21:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714951 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1FA071744 for ; Fri, 14 Aug 2020 17:27:48 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EBA1720791 for ; Fri, 14 Aug 2020 17:27:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="gpkxoW1B" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EBA1720791 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dU3-0005X4-Qt; Fri, 14 Aug 2020 17:27:15 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQ0-00024Q-GL for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:04 +0000 X-Inumbo-ID: 30a1e61e-ca5f-412d-880f-2a75d50afc69 Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 30a1e61e-ca5f-412d-880f-2a75d50afc69; Fri, 14 Aug 2020 17:22:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425740; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=j/mEAsKDtlwUNCaXH8/bW4L0IGIqwSTG9xQXAGBwHu4=; b=gpkxoW1Ba0l5GKytPhyRLngNacmdykcpa/ywnjKckZtGtVPp/xbGlane YpDNZS+dq2jTz00fpG4SyiNIuoGeRyU/28lcELRIfZ+m/UU/3iG61HB+1 JNrlDTQNi6+eONJDOz1tBQdYUr5ZClKRkHKHgu6OpjHP79Cw85HtP4pmb E=; Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: rjct46FF8MaRfBkdDt1lC6z5JK2bNyCYj5PShzuoSNVBNSa1/R/zVC2ecsK12vi4OyaOuj2wj5 P56WMgfKI3q2WAVTf4721n+ANPtQZ8VoYBN4vAPYpKfkQktmR+7Kp6IruRgj5b9x+4K1CnOyq/ DADEEJw9n+ePjivIyAzdeDMiomFK0f9z1W+oqG6RVYtCWcfSOkpOUH3C2xno1Ox6aIsgQLmXCU gbYdbSjwpi1eT1owlzEMDhJAajsxyOLns1dotpA+VZ1MykTb5J8jvkjCTZeqb9YRCkq6soYfHV 38o= X-SBRS: 2.7 X-MesageID: 24879743 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24879743" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 38/60] history reporting (nfc): Add another test rune to the notes Date: Fri, 14 Aug 2020 18:21:43 +0100 Message-ID: <20200814172205.9624-39-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Signed-off-by: Ian Jackson --- runes | 2 ++ 1 file changed, 2 insertions(+) diff --git a/runes b/runes index 8139f955..829e6106 100644 --- a/runes +++ b/runes @@ -11,3 +11,5 @@ select * from jobs full outer join steps using (job, flight) where flight=2364 o >v; env DBI_TRACE=2=v time ./sg-report-host-history --debug --html-dir=$PWD/e --max-flight=152223 flight:152223 2>&1 |tee c |ts -s %.s >c.ts rm -rf d; mkdir d; cp -a e/. d/.; mv c.ts b.ts; mv c b; mv v u cp -a d/. e/. + +>v; env DBI_TRACE=2=v time ./sg-report-job-history --debug --html-dir=$PWD/e --max-flight=152223 --flight=152223 2>&1 |tee c |ts -s %.s >c.ts From patchwork Fri Aug 14 17:21:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714897 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ADDD61392 for ; Fri, 14 Aug 2020 17:27:23 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8A0A020774 for ; Fri, 14 Aug 2020 17:27:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="cewrDWv6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8A0A020774 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTZ-0004JA-Vg; Fri, 14 Aug 2020 17:26:45 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQ5-00024Q-GL for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:09 +0000 X-Inumbo-ID: a7390e41-f3d2-4171-a822-ee700fc01ca5 Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id a7390e41-f3d2-4171-a822-ee700fc01ca5; Fri, 14 Aug 2020 17:22:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425741; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=uHxURdk47z4WA/HkYNY5Reoic0Udo2X39Jbx4cyBODE=; b=cewrDWv6rRHk1rkLtI94dpZPYOZSixQFqBj7TdSe/yfEk0sAO+ovzpBT FZNE0BZALd3w1nILbnURDUaUrieWj4D9Ie3gjZylPrNFyWiMZV3i4GHfO yGLBQp/PqKLLE9lqgdaXXHLhXrNlZulWIXOTiSIZnRbp+WzCvn/WgTYzJ M=; Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: XLsqIzKr+LbHvXDHYGeXcxiwqJmXkCGLhWZc6reQ6QXPuqklIpvyACl3ovNfEqdFwoW+SRnBkN bFwoJYEmB+cFYC4gX3Jpu9p5RSbA4UVnpAvmGbfqf6bfScRb8HbTlPQosR2n6RH3AcEvbpsCtG Ec9GdzWVQ2zoEKMrB7QOXNMb1CV2q5WhLElXBQhbaFjHsgBNAatTvE9fS23uXQ8WUN8rSkS3sU dYSjcztV/qKTW/XNImmVgECkcHLkUOUMaf5MpQYsQW9qHGwpwIAacsHf2y73t2xa68glEoUqYi BCk= X-SBRS: 2.7 X-MesageID: 24879744 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24879744" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 39/60] history reporting (nfc): Make cacheable_fn work without cache Date: Fri, 14 Aug 2020 18:21:44 +0100 Message-ID: <20200814172205.9624-40-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" This would allow us to use this in call sites without a cache. I changed my mind about the code that prompted this, but it still seems plausibly useful, so I'm keeping this commit. Signed-off-by: Ian Jackson --- Osstest/HistoryReport.pm | 1 + 1 file changed, 1 insertion(+) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index 695d744a..5d2c83b7 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -99,6 +99,7 @@ sub key ($) { sub cacheable_fn ($$$) { my ($jr, $cachekey, $fn) = @_; + return $fn->() if !defined $cachekey; $cachekey = '%'.$cachekey; my $cached = $jr->{$cachekey}; $q_count{$cachekey}++; From patchwork Fri Aug 14 17:21:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714861 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F36B2618 for ; Fri, 14 Aug 2020 17:23:32 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D025520774 for ; Fri, 14 Aug 2020 17:23:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="gOZomQpB" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D025520774 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPw-0002jW-9e; Fri, 14 Aug 2020 17:23:00 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPv-00024Q-GG for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:22:59 +0000 X-Inumbo-ID: 252a8173-d90c-4431-960b-bc72e22bcc23 Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 252a8173-d90c-4431-960b-bc72e22bcc23; Fri, 14 Aug 2020 17:22:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425740; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=rk4K15u28Nj+M4CrVc/t4g31Ehc9MfDdyP/m7plabjo=; b=gOZomQpBfGEYLR6lsBvIHwsVTKup4AH5osxza0ZDgV6AWqKqu/3J3IX1 Dn8PEn+gYrzDjSMy83US7WrqtcTK4CUrSddKpS4Nt5PjLezJ2zSYLocJQ EwsV3a6xq0ZSZ1s7iqz14P+r4HM8OrR7OdfSU9/rNQYktREBqvnfnfKFI 4=; Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 57fGnd9QyUDalP3QxlumkonaepdRwz5uQcg/r75btipjpiouLVGxOdsUJRVO7f75Ub7Se1bMlf 2pY4pQRgnBIn8X8DMydzXZYxaNF826KG2gVX7nJiGvO5K4oN1d/phLmrG5VCgF7sNjjUeoD+4i OhtYmFOuvawtIO+PU09jl4Itj1YuSVQlbNH/rfZ8lHOD2qsQhiEAxLO0dFJCty3Larnog28fYY xXw/DY8yDJmQo4xAFsKGspP89Zke4SESow4M9lYz5njnauhZB2l8eyRuy27udDrfa4HaVySD9z Prg= X-SBRS: 2.7 X-MesageID: 24879740 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24879740" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 40/60] sg-report-job-history: Refactor "ALL" handling Date: Fri, 14 Aug 2020 18:21:45 +0100 Message-ID: <20200814172205.9624-41-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" * Make an explicit entry ALL in @branches, rather than implicitly processing ALL as well. * Consequently, put explicit ALL entries in @tasks too, rather than putting in entries without a branch name. * Pass ALL to processjobbranch rather than undef, and turn it into the internally-used undef at the start. When used with --flight (findflight), this has no functional change. When used with --job, ALL must now be included in the branch list passed to --branches. The only in-tree call is with --flight. Signed-off-by: Ian Jackson --- sg-report-job-history | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/sg-report-job-history b/sg-report-job-history index 3b45992f..8932458e 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -72,7 +72,7 @@ sub findflight () { SELECT branch FROM flights WHERE flight=? END die "$flight ? @$branches ?" if @$branches!=1; - @branches= @$branches; + @branches= ('ALL', @$branches); my $selectq= db_prepare(< X-Patchwork-Id: 11714855 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5BDB8618 for ; Fri, 14 Aug 2020 17:23:30 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3859920708 for ; Fri, 14 Aug 2020 17:23:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="OaHP0596" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3859920708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPr-0002ce-JS; Fri, 14 Aug 2020 17:22:55 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dPq-00024Q-Fy for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:22:54 +0000 X-Inumbo-ID: 5d65d4e3-876c-4136-98ca-f3e8e235ac0a Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 5d65d4e3-876c-4136-98ca-f3e8e235ac0a; Fri, 14 Aug 2020 17:22:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425740; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=cgymAWCuGG6G0PHhWvU8N3IYB47K2vDm+DZU0S9yr2A=; b=OaHP0596L+MgxsEDrAQe1piklO+CgZaEuoo9nTqOwvA4IVd7Be/0cyaf eoJwSf60q8leQs6+XFeZthrU3J50VCoX0KuZNttKT54nBqntbhULcNqo8 Aubr/s1u3TsP3B+vSdZ4oSw0QZxmBZoxJgwzJwtSty59/Tav56/rqf4zL E=; Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: XLTBOLShMunTVXpzUPk0nL30OigiYSi84mZ8SU1DYgCm8dckGpD+ff64M3PkY9hrJUNTxBlzKs dxjwmp/nV5gCvaCGIXPNgoQkojgEJDz7k3+/Yajo79Hsp/OFfvNQAfa1l2U9BVuk3m8vBtqayB 1yoJu4XWTtS0qKEsnSZi9v8icnD2uUbbFbIsxZbNz+sxCVhzgwbLSbtMZysSIHCiPhQIxvCILn 2Zr9JovgA0ZOfe9XxcAyvCNjX3G0PfZf9Mmv6KiSEwhryICoE9Uz87t2rYH2x9Pl+teg4jtIYc 60Q= X-SBRS: 2.7 X-MesageID: 24900377 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24900377" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 41/60] sg-report-job-history (nfc): Make $ri->{Hosts} a hash Date: Fri, 14 Aug 2020 18:21:46 +0100 Message-ID: <20200814172205.9624-42-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" This will make it easier to cache this. Signed-off-by: Ian Jackson --- sg-report-job-history | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sg-report-job-history b/sg-report-job-history index 8932458e..118926c6 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -173,11 +173,11 @@ END while (my $f= $flightsq->fetchrow_hashref()) { my $ri= report_run_getinfo($f); - $ri->{Hosts} = [ ]; + $ri->{Hosts} = { }; foreach my $hostvar (@hostvarcols) { $hostq->execute($f->{flight}, $f->{job}, $hostvar); my ($host) = $hostq->fetchrow_array(); - push @{ $ri->{Hosts} }, ($host // "-"); + $ri->{Hosts}{$hostvar} = $host; } my %revisions; @@ -255,7 +255,7 @@ END print H "{ColourAttr}>". $r->{Content}."\n"; my $lastrev; - my $hosts = join ", ", map { $_ // "-" } @{ $r->{Hosts} }; + my $hosts = join ", ", map { $r->{Hosts}{$_} // "-" } @hostvarcols; my $hosts_colour = report_altchangecolour(\$alt_hosts, $hosts); print H "".encode_entities($hosts)."\n"; $osstestverq->execute($r->{Flight}{flight}); From patchwork Fri Aug 14 17:21:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714927 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BD7C713A4 for ; Fri, 14 Aug 2020 17:27:38 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 998BA2078D for ; Fri, 14 Aug 2020 17:27:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="L4KHtTcG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 998BA2078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTf-0004VR-3D; Fri, 14 Aug 2020 17:26:51 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQA-00024Q-GT for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:14 +0000 X-Inumbo-ID: e7f34f1b-859e-4d7e-8e46-9bfbb0823a41 Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id e7f34f1b-859e-4d7e-8e46-9bfbb0823a41; Fri, 14 Aug 2020 17:22:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425741; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=SS9u5RjMdUWJ2MRfTSjZvF1Vqzd0OuPHfvTpp/2jiqY=; b=L4KHtTcGqhb9jvh49Aqvmet1RysFlLr1pwlZLO5HT7rmuQVTsiFCxjnT GaUlcBww0NLs22FpFYE8N40kLpbYYiZyvanUnlEc66jgc0N+bVM/KeTNs KzrttvG2QI6NjuzzUsN7nT+2w+SBOUzl2wlSHHVorS42MhOQhlFu/S1Rr 8=; Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: xb4J0d22X3Zh3ZJ3X8aJtKQ9o14b9OJE/B1nJdYOFR4A/tgIUjPRFj3MZTx8GDMinayn7e3Fti ewWMlXXmlhmAz5xTJ3u1WpMjPK5wHua7z1en6QY+e4AKZvAfOjmg03AWQ0gfEAdZJXdgQvOlMM /Ymy11hXV6UJJKVulCy/RUXWtt0FzfG/mefN1acde3uL8igJo9HwwNTwQLc4jUmhrti89sgITg ckQI1MihPzBrYxP6kvPpnPD6yIHOzjfZMzKyBQtDsRNaV4bvAYEm3a32WUZ92LBB/85Y02zXan vnI= X-SBRS: 2.7 X-MesageID: 24900378 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24900378" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 42/60] sg-report-job-history (nfc): Add new hostvarcols calculation Date: Fri, 14 Aug 2020 18:21:47 +0100 Message-ID: <20200814172205.9624-43-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" We are going to want to replace the existing @hostvarcols calculation. Provide a new one based on $ri->{Hosts}. Right now, die if they don't produce the same answers. This still works, which shows that the calculation is right. Tested-by: Ian Jackson Signed-off-by: Ian Jackson --- sg-report-job-history | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/sg-report-job-history b/sg-report-job-history index 118926c6..fb2930a9 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -206,6 +206,13 @@ END push @test_rows, $ri; } + my %hostvarnames; + foreach my $ri (@test_rows) { + $hostvarnames{$_}=1 foreach keys %{ $ri->{Hosts} } + } + my @hostvarcols2 = sort keys %hostvarnames; + die unless "@hostvarcols" eq "@hostvarcols2"; + my $osstestverq= db_prepare(< X-Patchwork-Id: 11714887 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 54F8713A4 for ; Fri, 14 Aug 2020 17:27:19 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3129920708 for ; Fri, 14 Aug 2020 17:27:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="d5GTOiKI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3129920708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTd-0004RE-DR; Fri, 14 Aug 2020 17:26:49 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQF-00024Q-Gb for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:19 +0000 X-Inumbo-ID: 43930bea-aa8c-4fa9-83a8-e5fa9563c9f2 Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 43930bea-aa8c-4fa9-83a8-e5fa9563c9f2; Fri, 14 Aug 2020 17:22:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425741; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=MglmACUtUxOLGTuI7w9qwbU/ZWU0cGEyX3Hyd2Jcx3I=; b=d5GTOiKIfRsY3TRj5A7DmrR9DxiRB+AWySNU1enYld9sLyWSgfxcSl9A EWj7MEMQgQJ9xbt0tDgSlKCE+/ynQE3dDmXW0D1k7MgxmVstgppdpwCWO Hc12Tz42bbimIjKPPX8PS58QVAGID0sbyAZXVDImfsI8Nqa2oGLnq2yRG c=; Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: c+HtNVfBRqiEuS7oPCcV+A5GgAx4Melnt4EpEKEP6BqPLAPx09xcs3kNdbSCGHAVPVf+IdX4UO fgVU/EUhnHq7HDQCkrp8i6GcLPdbTYj+GS9G9XC5CHVY9JW6Q/ljlbp0tOVnyu7bzED4ZoQ6t1 pEABwFd1hl0rpP4mQDtjLClo/XQXOhHOD62xVYTU+EAI47veStj2bf3VI6yp2c8fOtmt7fXvBL luJ/yRg+foH1J6njqR6f5tRnQXipxTFOT0uEMWl3LUKQumQDCYA/jKuawmDkcXvoz/NLoWqCMg BwQ= X-SBRS: 2.7 X-MesageID: 24900379 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24900379" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 43/60] sg-report-job-history (nfc): Query hosts runvars in one go Date: Fri, 14 Aug 2020 18:21:48 +0100 Message-ID: <20200814172205.9624-44-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Rather than doing one query for each entry in @hostvarcols, do one query for all the relevant runvars. This is quite a bit faster and will enable us to use the cache. This is correct because @hostvarcols was the union of all the host runvars, so this produces the same answers as the individual queries. Signed-off-by: Ian Jackson --- sg-report-job-history | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/sg-report-job-history b/sg-report-job-history index fb2930a9..5d913b96 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -167,16 +167,19 @@ END } my $hostq= db_prepare(<fetchrow_hashref()) { my $ri= report_run_getinfo($f); $ri->{Hosts} = { }; - foreach my $hostvar (@hostvarcols) { - $hostq->execute($f->{flight}, $f->{job}, $hostvar); - my ($host) = $hostq->fetchrow_array(); + $hostq->execute($f->{flight}, $f->{job}); + while (my ($hostvar,$host) = $hostq->fetchrow_array()) { $ri->{Hosts}{$hostvar} = $host; } From patchwork Fri Aug 14 17:21:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714941 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AC19217C7 for ; Fri, 14 Aug 2020 17:27:43 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8525D20866 for ; Fri, 14 Aug 2020 17:27:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="ZLNkcwUj" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8525D20866 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dUB-0005qZ-4p; Fri, 14 Aug 2020 17:27:23 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQK-00024Q-Gg for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:24 +0000 X-Inumbo-ID: dbd1591f-7a00-40b0-a763-8771cac475ea Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id dbd1591f-7a00-40b0-a763-8771cac475ea; Fri, 14 Aug 2020 17:22:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425741; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=mO+R3Wi2onO1YjU9v4vCILxbaexYwro9M3qETJvjXTI=; b=ZLNkcwUji82rzZS1vLKsNzPoQVPFmrJoNmhW/WSpOohTC0SPK+r3mIPW u+ahElsvLV3zGeO4FZTdbE2Ea//gw9cIhzk3Hmx+4lprQ4R/l1hY90LtX yZzgSIJkYtKYN3AP/dMdeWoQsy/NC26BgV/Z9Sh56JJlCdaSCrkIpilcB 4=; Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: CZn+kKqSsQyEdTyVBEs3tVph5l12B1LN6jB3G80PxBt8N3bnVtvqWSvbDQol7OEFVL4j55WbaX ayn1TYKpmouIWRoNY4992zWPPc5hBrjgIJl8xDswnDS+qpFxNZwr440YJDZouh4w2mXfDeeFlo 4EVhgq4GnmZDwIfHBUzOqq3QCrkZ/ukXtKdqXex/Ehg3gtFzE05qhtxl4e+BN3ofCjJE3neYup KorGP3npk7s701bFqTwx80cBCrO32A9KztZlRHE1epJAYYWk8VJU3IbIEWTQ39RmcaITZxBALA QhI= X-SBRS: 2.7 X-MesageID: 24879745 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24879745" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 44/60] sg-report-job-history (nfc): Drop $hostsq query Date: Fri, 14 Aug 2020 18:21:49 +0100 Message-ID: <20200814172205.9624-45-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" We have eliminated all the users of @hostvarcols before @hostvarcols2 is calculated from the row data. The query which produces this is very slow and can't be cached. We can abolish it now and just use the @hostvarcols2 calculation. Signed-off-by: Ian Jackson --- sg-report-job-history | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-) diff --git a/sg-report-job-history b/sg-report-job-history index 5d913b96..5684ac02 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -147,25 +147,6 @@ END END $flightsq->execute(@params); - my $hostsq= db_prepare(<= COALESCE( - ( - SELECT flight $fromstuff - LIMIT 1 OFFSET $offset - ), 0) - ORDER BY name; -END - $hostsq->execute(@params, @params); # sql text contains $cond twice - my @hostvarcols; - while (my ($hostvar) = $hostsq->fetchrow_array()) { - next unless $hostvar =~ m/(^|_)host$/; - push @hostvarcols, $hostvar; - } - my $hostq= db_prepare(<{Hosts} } } - my @hostvarcols2 = sort keys %hostvarnames; - die unless "@hostvarcols" eq "@hostvarcols2"; + my @hostvarcols = sort keys %hostvarnames; my $osstestverq= db_prepare(< X-Patchwork-Id: 11714935 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 436491392 for ; Fri, 14 Aug 2020 17:27:43 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1EF5E214F1 for ; Fri, 14 Aug 2020 17:27:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="b1aac2nP" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1EF5E214F1 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTk-0004iA-3P; Fri, 14 Aug 2020 17:26:56 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dR7-00024H-2N for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:24:13 +0000 X-Inumbo-ID: 0cc4a31a-a8ff-42df-9468-3999946ea39f Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 0cc4a31a-a8ff-42df-9468-3999946ea39f; Fri, 14 Aug 2020 17:22:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425741; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=K4MIaKessBz20HyGZXTzw+K8QuBXZ4UICpsMkuTqCYc=; b=b1aac2nP9l8qvgqxMWMXp7BNgrx5Qr1qsmlYeXTspp0ml44haC9t7aJa 2D2oMglJjlUxeFZvbJH3hkuZreClYTg9m8KecBGxtY1hhtcWw2jp561tZ NU2VNqhmxOHEkwiCilqhQMbK6slOTuDiSndn0XyDJtJeIMEJ73EL1+7rX w=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: tOXW0E1ASfq6vWuXstSZBpN89HTycJbZPlMC4UcJIvO32tDN6HB4Vn4dSHJSCzArMaHVFRl8Ff ii/oz83F8N2JgFYgmDZhls9FzVfl51Ez3mS7SGcmwLSK5F9Kbl8D+bqFCDeZBvT0mqDERp5JXS RTjWNAHmrB0uGAZO/gnQzgToY0XOf2fYSLQWGtHF3qXj1vGtFY6xfoM1Sk+fjzk0eTgOGqoFl+ YId1riTVqEJhVYC+o+L0GeZfRPLOJNjmpAj+62VViNl3q5qhMdevKA17THPZS9BHNR/NPAkFE6 N4M= X-SBRS: 2.7 X-MesageID: 25487831 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="25487831" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 46/60] sg-report-job-history: Introduce use of cache, for hosts query Date: Fri, 14 Aug 2020 18:21:51 +0100 Message-ID: <20200814172205.9624-47-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" * Set up the cache. * Call the per-row setup hook. * Cache the computation of $ri->{Hosts}. * Call the per-row cache write hook. * Finalise the cache. Output is the same, but with cache information in the output html, and faster. Signed-off-by: Ian Jackson --- sg-report-job-history | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/sg-report-job-history b/sg-report-job-history index 5684ac02..9f1216a6 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -121,6 +121,10 @@ sub processjobbranch ($$$) { AND name LIKE '%buildjob' END + cache_set_task_print("$j $branch_or_all"); + cache_set_key_cols(qw(flight status)); # we process one job per process + cache_read_previous("$htmlout/$html_file"); + print DEBUG "processjobbranch('$j',", ($bra ? "'$bra'" : 'undef'), ")\n"; my %rev_grid_col; @@ -156,13 +160,17 @@ END END while (my $f= $flightsq->fetchrow_hashref()) { + cache_row_lookup_prep(\$f); my $ri= report_run_getinfo($f); - $ri->{Hosts} = { }; - $hostq->execute($f->{flight}, $f->{job}); - while (my ($hostvar,$host) = $hostq->fetchrow_array()) { - $ri->{Hosts}{$hostvar} = $host; - } + $ri->{Hosts} = cacheable_fn($f, 'h', sub { + my %hosts; + $hostq->execute($f->{flight}, $f->{job}); + while (my ($hostvar,$host) = $hostq->fetchrow_array()) { + $hosts{$hostvar} = $host; + } + \%hosts; + }); my %revisions; add_revisions(\%revisions, $f->{flight}, $f->{job}, ''); @@ -276,8 +284,10 @@ END print H ""; } print H "\n"; + cache_write_entry(\*H, $r->{Flight}); $alternate ^= 1; } + cache_finish(\*H, ""); print H "\n"; H->error and die $!; close H or die $!; From patchwork Fri Aug 14 17:21:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714953 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 615A51392 for ; Fri, 14 Aug 2020 17:27:48 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3BDF42078D for ; Fri, 14 Aug 2020 17:27:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="FIze4JOf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3BDF42078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dU1-0005Qj-DQ; Fri, 14 Aug 2020 17:27:13 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQP-00024Q-Gh for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:29 +0000 X-Inumbo-ID: a375254f-663b-4e62-9d4e-3d9ccfa6426a Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id a375254f-663b-4e62-9d4e-3d9ccfa6426a; Fri, 14 Aug 2020 17:22:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425742; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=CxW4jAZ/vIcg8LNWdKZ3/W4BiUjaoX6Yw8o5gSt9P7U=; b=FIze4JOfdeWO1QKm59nyYj9vlMnz/7KyV6qEF8ZlY6u5vtdgo5eBGtZf 9B3R6zvDPDf8pJiXXGRwVu/tHZ9ZRi/uzr7GB9JpUi8LnIRE8NUr3acKW TBUVFxW1xuc8X+nZFkSmvOUQNsuJ8EnTw62tzxnW1+qdhIHoAdzo78JO5 U=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: W0PWBOGcTaOh5PxF3EZHPnw60gTjk5P744aQzgHSdUZ9MmyDCS2OFSUwCRJdLgyDSNRire5bPW SpBE8KyJ5swUsmKC5n5EA79AnNS7BWpodpwJZ+mvQo9WUL3zn5jyO0w7299UEZI8UBq/Be0RQf vc5UQf4qSkjf/s1UDEkpEi0eYJYFkITn55SYQ5e5txpYtPNKkux9/MxwqaNNzGSpCfhQSa3OOh odA77Gl3CuVaOtmq6rlk/R7HGmkG/x/eRwJBcVEvUsCE6rRDqym7sIM6QmzuWhxpe9iKbCB5xP LZM= X-SBRS: 2.7 X-MesageID: 25487833 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="25487833" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 47/60] history reporting (nfc): Break out url_quote Date: Fri, 14 Aug 2020 18:21:52 +0100 Message-ID: <20200814172205.9624-48-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Signed-off-by: Ian Jackson --- Osstest/HistoryReport.pm | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index bfbee28c..f30ded5b 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -156,6 +156,12 @@ sub cache_read_previous ($) { close H; } +sub url_quote ($) { + local ($_) = "$_[0]"; + s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge; + $_; +} + sub cache_row_lookup_prep ($) { my ($jrr) = @_; @@ -179,8 +185,7 @@ sub cache_write_entry ($$) { next if $k =~ m/^\%/; $_ = $h->{$k}; next unless defined; - s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge; - printf $fh " %s=%s", $k, $_; + printf $fh " %s=%s", $k, url_quote($_); } }; $whash->($jr); From patchwork Fri Aug 14 17:21:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714923 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9AEB41744 for ; Fri, 14 Aug 2020 17:27:36 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 76F752078D for ; Fri, 14 Aug 2020 17:27:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="VMhQYvIA" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 76F752078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTt-00055q-PD; Fri, 14 Aug 2020 17:27:05 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dRv-00024H-4A for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:25:03 +0000 X-Inumbo-ID: fec89442-f640-4447-b2f8-be604317b767 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id fec89442-f640-4447-b2f8-be604317b767; Fri, 14 Aug 2020 17:22:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425760; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=xNFNARb2DH3nytu+LwtUfgJ/aOgJiJDqOJzo2YjtfkA=; b=VMhQYvIAtWlrYNIOM6K4u4aIU0TkIb93fsERkY2hfUCARoHBPW37R7MJ yKHbCfohHZcQpiCRpCh0X5vyrNtHgRzcB5NaE2j76QKoI7O2GKHW84YeZ bMOmiIynXF+/MeqTlK+qmDMfNyW8IoBaGIcr3jmQyJxHSDd5tK/J21Bo7 E=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: wuducQNRVAq8B5xzQ7UBCn0VqtsE5FEzld/9mfYnWqs9kS+ZM3RiooT5SVwl4qR/LReD/d/IVD mZ+f/3qkdGLpngT10HDEzjOq0KsovPBBG7FsR05OLOVPIJSaVGmKjqGQKSlacEYNvL/EyVDOTa YKRXO2FI4JP7Sc/fRoWp2HyUcafxY10wm5MlpwD5gHsZ+cA3UXo0HeNuE2Gr/sXBjqNCNs3oIh SP087T4ZFTpA672YoeLpUzoMSW32+S8RP9IUuEARqIFWqkE9i2+ioXjW+RValGwfsgfrmUTkxr lzA= X-SBRS: 2.7 X-MesageID: 24576920 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24576920" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 48/60] history reporting (nfc): Break out url_unquote Date: Fri, 14 Aug 2020 18:21:53 +0100 Message-ID: <20200814172205.9624-49-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Signed-off-by: Ian Jackson --- Osstest/HistoryReport.pm | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index f30ded5b..f8a49955 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -147,8 +147,7 @@ sub cache_read_previous ($) { } s{^(\w+)=}{} or die "$orig -- $_ "; my $k = $1; - s{\%([0-9a-f]{2})}{ chr hex $1 }ge; - $ch->{$k} = $_; + $ch->{$k} = url_unquote($_); } push @previous, $jr; $cache{key($jr)} = $jr; @@ -156,6 +155,12 @@ sub cache_read_previous ($) { close H; } +sub url_unquote ($) { + local ($_) = "$_[0]"; + s{\%([0-9a-f]{2})}{ chr hex $1 }ge; + $_; +} + sub url_quote ($) { local ($_) = "$_[0]"; s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge; From patchwork Fri Aug 14 17:21:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714895 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8BA1313A4 for ; Fri, 14 Aug 2020 17:27:21 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 685162078D for ; Fri, 14 Aug 2020 17:27:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="cXUzS8fO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 685162078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTe-0004U4-I9; Fri, 14 Aug 2020 17:26:50 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dRC-00024H-2M for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:24:18 +0000 X-Inumbo-ID: df00126f-7b1c-477e-b7cd-65eaef15690b Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id df00126f-7b1c-477e-b7cd-65eaef15690b; Fri, 14 Aug 2020 17:22:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425742; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=9AKhCfm/1BqFPkQOzF0eCZveb5uEDpP3/r6hBuTaDWU=; b=cXUzS8fOItRDucrGknS6yM3y3J5cZXXs0mvFdSQC4TzAuZMhWx8dLfld AihCEKIXkanttD6xe/HvaTeV7nmqnG73dMDmGnoL7dOZRgq7PCP1A4p9b h+hAn11Z1I29NQU8gkuAJR101VNX2eko5PCECFlwoWMgpGZlQrVqjyibp A=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: SENo5FDhy3QIJqjjqhN0YgzzQvC6XhG/zVSlqWh9KzytsK1lMr3iGQse5KFvQ38y3o5yDgD1Y8 v4mL7uU48DjZYMFsYqYv6r20us/3Bwnn131HDb+9p+S/0ExMQS6VYlsJ7CIDyNQggbBh/IH2aq mi8XfIulczwkqL6nMPI2rrKhRBYkz9ck1doc/RUKzRN1U/L3+rwjTryu6dH9aLubISEYfosF7X Gk2ax/mou8GDuH4joLdcUNwPsPMldlIqUEoQLTR28JNbRdAOEUcNbLUbtygKtdn7HukVfYT5Vh xFo= X-SBRS: 2.7 X-MesageID: 25487836 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="25487836" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 49/60] history reporting (nfc): Break out $url_ok_chars Date: Fri, 14 Aug 2020 18:21:54 +0100 Message-ID: <20200814172205.9624-50-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" We will want this in a moment. Signed-off-by: Ian Jackson --- Osstest/HistoryReport.pm | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index f8a49955..75ec75f3 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -45,6 +45,8 @@ BEGIN { use POSIX; +our $url_ok_chars = '-+=/~:;_.,\w'; + our @key_cols; our $taskprint=''; @@ -163,7 +165,7 @@ sub url_unquote ($) { sub url_quote ($) { local ($_) = "$_[0]"; - s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge; + s{[^$url_ok_chars]}{ sprintf "%%%02x", ord $& }ge; $_; } From patchwork Fri Aug 14 17:21:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714893 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 871D9618 for ; Fri, 14 Aug 2020 17:27:21 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6255A20774 for ; Fri, 14 Aug 2020 17:27:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="DJV/gOVI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6255A20774 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTa-0004Ju-BP; Fri, 14 Aug 2020 17:26:46 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dRh-00024Q-JR for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:24:49 +0000 X-Inumbo-ID: ca6137f4-06f0-4a7e-b61b-7d8f3aeb47da Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id ca6137f4-06f0-4a7e-b61b-7d8f3aeb47da; Fri, 14 Aug 2020 17:22:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425760; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=0olnZbLoPJuh9BMaljhOFagtvg5I4UYKHKHAXNBW7Sw=; b=DJV/gOVIfB9h5nhHosmcRF1ZisrHFNmgxS/eNrEVsgeNXe22zqVS/BEU yd6xSWyeSh2nch9ypeSARsbiW/Eivdsk6yRGst5kckP0ZVi2Mmmd2EBtf O/dtdnLpQ/S+W8bNRRhyqqU5KElpgYIk2FZw83QC/9wsY0D9rFZTy0MCp E=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: XoG6gYs8oHYMIMiQUF2RL0BFjmGVARyQUFF3LUF53oE12LXrKBkrB29+qKkWDNMPeqYRUAzB+X 26lzMu667c4cNPHXWm6uftejAtZIK34FcWb4b28MKb/U5cFQNAqGGdXFzKJZrBLzrdCf3Gg92K KpXDVqQ9YO8hQALwzDfx7VGO55Q4b5izAiBhkbeJHJirXGNPCKKzVFwuFTVwhXqt9ZrlBVwYUL 5cWb+TzEHshmQMHzS6P2j7sARBw7ByMJDN92bL4vnBw6AxFVODndz4nBcERyPZkgR1kAUE2Z0v MIA= X-SBRS: 2.7 X-MesageID: 24576922 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24576922" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 51/60] history reporting (nfc): Quote keys too Date: Fri, 14 Aug 2020 18:21:56 +0100 Message-ID: <20200814172205.9624-52-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Right now all the callers have keys which don't need quoting. Signed-off-by: Ian Jackson --- Osstest/HistoryReport.pm | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index 20913d27..3e2ad0d3 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -147,8 +147,9 @@ sub cache_read_previous ($) { $jr->{'%'.$&} = $ch; next; } - s{^(\w+)=}{} or die "$orig -- $_ "; - my $k = $1; + # Tolerate %, which will be de-quoted by url_unquote + s{^([\%$url_ok_chars]+)=}{}i or die "$orig -- $_ "; + my $k = url_unquote($1); $ch->{$k} = url_unquote($_); } push @previous, $jr; @@ -192,7 +193,7 @@ sub cache_write_entry ($$) { next if $k =~ m/^\%/; $_ = $h->{$k}; next unless defined; - printf $fh " %s=%s", $k, url_quote($_); + printf $fh " %s=%s", url_quote($k), url_quote($_); } }; $whash->($jr); From patchwork Fri Aug 14 17:21:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714889 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6C9E31392 for ; Fri, 14 Aug 2020 17:27:19 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 490AB20774 for ; Fri, 14 Aug 2020 17:27:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="g02raETM" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 490AB20774 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTQ-000463-1Q; Fri, 14 Aug 2020 17:26:36 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dSF-00024H-4s for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:25:23 +0000 X-Inumbo-ID: d27ef01e-6d63-4b80-af4d-bcf3d76fcf90 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id d27ef01e-6d63-4b80-af4d-bcf3d76fcf90; Fri, 14 Aug 2020 17:22:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425761; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=KK2C5LTSWHXRXWbqc/OKGwTBW3Gmuf54SK1hkOQC7AY=; b=g02raETMvwmVIvNu6KnE3XwfEM5ByaxhqKLkD8qB6XNXG5poitQVkhpv rLDV6pwGXMEr+BpEBvP+oLtEFkTHQlTb8NDArqwmKZirilmxmxh9pL/4R q8OTeHnlU3z7+c7g8G7/sMlrN6i5ByKoiBQQQqldBcvFmjaccx7HMsLuD A=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: yMkTEtTkwa6Z7Zl0aTNzX97qii72UIAqi31sn8bj30x1CXvyvZJyQnGo3ygB0kQm23ueBsl2Ao X/HU9lATQLMrkbqxH2bQI1fGbKW8cZIujYo5+J7eFXbFXOU7YiF7cBpCtj7rjJJGdWE5gKJ1Yc SlU47uDFYStfbpZKjx/EQIqbAiSOzlqPstrxQ5b9Nb4d6cK5VLu4rjr9/l7hftTAmHST1piljH L9o/VlEtXUQFNWzNjl0VFXg2ayMTdEDO6vuTyfikLahoYg1R7mPmqpnWsd8jEBpRdaSffPiAk6 J3Y= X-SBRS: 2.7 X-MesageID: 24576924 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24576924" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 52/60] sg-report-job-history: Cache the per-flight revisions Date: Fri, 14 Aug 2020 18:21:57 +0100 Message-ID: <20200814172205.9624-53-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" This involves changing %revisions to %$revisions in the code which uses the value. No logical change. Signed-off-by: Ian Jackson --- sg-report-job-history | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/sg-report-job-history b/sg-report-job-history index 9f1216a6..39ebe024 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -172,25 +172,29 @@ END \%hosts; }); - my %revisions; - add_revisions(\%revisions, $f->{flight}, $f->{job}, ''); + my $revisions = cacheable_fn($f, 'r', sub { + my %revisions; + add_revisions(\%revisions, $f->{flight}, $f->{job}, ''); - $buildsq->execute($f->{flight}, $f->{job}); - while (my $bld= $buildsq->fetchrow_hashref()) { - my $bn= $bld->{name}; - $bn =~ s/buildjob$//; - my ($oflight,$ojob) = flight_otherjob($f->{flight}, $bld->{val}); - add_revisions(\%revisions, $oflight, $ojob, " ($bn)"); - } + $buildsq->execute($f->{flight}, $f->{job}); + while (my $bld= $buildsq->fetchrow_hashref()) { + my $bn= $bld->{name}; + $bn =~ s/buildjob$//; + my ($oflight,$ojob) = + flight_otherjob($f->{flight}, $bld->{val}); + add_revisions(\%revisions, $oflight, $ojob, " ($bn)"); + } + \%revisions; + }); my @rev_cols; - foreach my $revk (sort keys %revisions) { + foreach my $revk (sort keys %$revisions) { my $col= $rev_grid_col{$revk}; if (!defined $col) { $col= @rev_grid_cols; push @rev_grid_cols, $revk; $rev_grid_col{$revk}= $col; } - $rev_cols[$col]= $revisions{$revk}; + $rev_cols[$col]= $$revisions{$revk}; } $ri->{Revisions}= \@rev_cols; $ri->{Flight}= $f; From patchwork Fri Aug 14 17:21:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714957 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8CAC913A4 for ; Fri, 14 Aug 2020 17:27:50 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 66F1E2078D for ; Fri, 14 Aug 2020 17:27:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="fdvJNSaf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 66F1E2078D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTF-0003vW-Th; Fri, 14 Aug 2020 17:26:25 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dRr-00024Q-Jq for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:24:59 +0000 X-Inumbo-ID: 4c919405-4ff3-407c-9937-c3680a335396 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 4c919405-4ff3-407c-9937-c3680a335396; Fri, 14 Aug 2020 17:22:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425761; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=qblTmgTO05UZJXst81Uqf3+0V5Ri789DAqTYXLAzAGk=; b=fdvJNSafRx9VinhJTSeaZTFD4zNxd/aVTcLtVD5oN699POVirfEhXqtQ V0Iwm63itomA72PnQBCknXo31GFdHZYw0hDRE89kJTNfnht1DRVGn2wZ6 pxYa3ONfrwlRlhmgRZHUglYgK54mQEmaLDl5BnPZeFmQqkg7QcXOEeF2G 0=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 9eVFVQUT5Pv2GmxWznXw+ifds/t8RZyy80dwV4K1o/3cana6JGZQ6ZDmLhzc5EHuNORqjCyLUq lrY5Uv11696lDlw0+R5cZPnVLcQAcihCIWYpjcz3A1EUrmWEAnLfaJMRBDbm6+2aeD3mJLDhOU 0bxwCDAHdYHCGs9Iu78tDG0a8MKU1WPQhLgVC7Zx5sfw+1X3iiu3iDn1Iv9d27/g7f3xWRFUDd FNUAEDNbDE9ZjoGewFTAVIJDh4N4LGNdsu3v5gpBOYovvgfapq7jhDxP0nrdi5PMffduAYfVME NlQ= X-SBRS: 2.7 X-MesageID: 24576926 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24576926" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 53/60] sg-report-job-history (nfc): Refactor osstestrevs code Date: Fri, 14 Aug 2020 18:21:58 +0100 Message-ID: <20200814172205.9624-54-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Split this into (1) get the data from the db (2) process it into the form we want. This will make it easy to cache (1). Signed-off-by: Ian Jackson --- sg-report-job-history | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/sg-report-job-history b/sg-report-job-history index 39ebe024..d5e37bcd 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -260,14 +260,17 @@ END my $hosts = join ", ", map { $r->{Hosts}{$_} // "-" } @hostvarcols; my $hosts_colour = report_altchangecolour(\$alt_hosts, $hosts); print H "".encode_entities($hosts)."\n"; + my %osstestrevs; $osstestverq->execute($r->{Flight}{flight}); + while (my ($harness) = $osstestverq->fetchrow_array()) { + $osstestrevs{$harness}++; + } my $osstestrevs = join ' ', map { - $_ = $_->{harness}; s/^([0-9a-f]{12})[0-9a-f]+\b/$1/; "".encode_entities($_).""; } - @{ $osstestverq->fetchall_arrayref({}) }; + sort keys %osstestrevs; my $osstest_colour = report_altchangecolour(\$alt_osstest, $osstestrevs); print H "$osstestrevs\n"; foreach my $i (0..$#rev_grid_cols) { From patchwork Fri Aug 14 17:21:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714905 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6C2F7618 for ; Fri, 14 Aug 2020 17:27:29 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4877320855 for ; Fri, 14 Aug 2020 17:27:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="hroHY/Pu" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4877320855 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTd-0004Sk-VG; Fri, 14 Aug 2020 17:26:49 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQU-00024Q-Go for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:34 +0000 X-Inumbo-ID: 90a2a242-43f8-4d80-b009-045a4ce57d5b Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 90a2a242-43f8-4d80-b009-045a4ce57d5b; Fri, 14 Aug 2020 17:22:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425743; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=6jsNPgkrbQxgzqdAWU+v8WyFAwyN1dTTfEckfC/Xnrw=; b=hroHY/PuRAP1+qpuwFyUQWI9caDlh22mnwtD1RKlIe/xPGO2IruOlnTs CpqlrMCbIVGoDaw4tSrHwyUhg6grcm6RN2gU407uJUq0cxCi9BaRCdTYz eYWOvji7O8+OAdbu6ctwqv+JIY7gv1qgDz8TTMDCcUtTPtQIc4Z++7QeB U=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: IichEY6ZbjBfBltydfbxGNc45YhISME2fOnywk2FxnlR5Yi55yUdPVBUQdCZ+R3/xfjTVUvoVr D2eDWOwNYmeTiSRdecYPIFFAV87WYBsltBYIicUTlEYEdVXtcMYRQ1NEu60qM8Dlc3Cizss2+7 smCeAhgKQJmvF8R6x6aNgjG0xvJs70rwhVUsVnmZDrLAyf7x3zKVo1GBc3UCWndiCtYzt3KnBM 8LblB9vH6EohahVqiPCBGEBP/S0XcllJiU83/fU4Zi+BseQ/EgJZqeVjCXbyKMzcdj8gWzcgj3 IyU= X-SBRS: 2.7 X-MesageID: 25487837 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="25487837" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 54/60] sg-report-job-history: Cache osstestrevs Date: Fri, 14 Aug 2020 18:21:59 +0100 Message-ID: <20200814172205.9624-55-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" No logical change. Signed-off-by: Ian Jackson --- sg-report-job-history | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/sg-report-job-history b/sg-report-job-history index d5e37bcd..48dc2f57 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -260,17 +260,20 @@ END my $hosts = join ", ", map { $r->{Hosts}{$_} // "-" } @hostvarcols; my $hosts_colour = report_altchangecolour(\$alt_hosts, $hosts); print H "".encode_entities($hosts)."\n"; - my %osstestrevs; - $osstestverq->execute($r->{Flight}{flight}); - while (my ($harness) = $osstestverq->fetchrow_array()) { - $osstestrevs{$harness}++; - } + my $osstestrevmap = cacheable_fn($r->{Flight}, 'o', sub { + my %osstestrevs; + $osstestverq->execute($r->{Flight}{flight}); + while (my ($harness) = $osstestverq->fetchrow_array()) { + $osstestrevs{$harness}++; + } + \%osstestrevs; + }); my $osstestrevs = join ' ', map { s/^([0-9a-f]{12})[0-9a-f]+\b/$1/; "".encode_entities($_).""; } - sort keys %osstestrevs; + sort keys %$osstestrevmap; my $osstest_colour = report_altchangecolour(\$alt_osstest, $osstestrevs); print H "$osstestrevs\n"; foreach my $i (0..$#rev_grid_cols) { From patchwork Fri Aug 14 17:22:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714879 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9412D618 for ; Fri, 14 Aug 2020 17:26:55 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6FC7320708 for ; Fri, 14 Aug 2020 17:26:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="SDvWMNtT" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6FC7320708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTP-00045E-JN; Fri, 14 Aug 2020 17:26:35 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dSB-00024Q-Kt for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:25:19 +0000 X-Inumbo-ID: 8932bc25-9404-4cb3-a7c7-6cee3dadcbd5 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 8932bc25-9404-4cb3-a7c7-6cee3dadcbd5; Fri, 14 Aug 2020 17:22:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425761; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=Ph/KZZdUJIILbzz5zhADuCUKbWqNr0e5tyD4DfmxFCg=; b=SDvWMNtTApNoWyhZoIe8gOdkpGqUeQ6jGr00R+K9Ys1K2/xb4Tu0GO24 bJmWe6c2E6tdAF4mSYY11DplgRWpdx616OIcmkKxu+1ZOTSWmVbrcqGAc sDz5ygafN+eHGRCSQWguC/zZespGLatY4dXEabElg5mELrOU7QMERaKT0 g=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: ug+qUfbi1jD6fWmU3D7GzJNjm2AMiFCe/B6YfTaJvtFfHm82p5CGNHstCHu598gBeKHCCC0ts2 GA18nUliP1sg2jraTK6VXILohL+lA9d/wE8Vi+/Pa/Am7vv3Gi/OrmxY76ec6Kw2lFFlfnr/PT y5WCeHMaflgJeZj3E8f8npofG3IJeTFtfppj6OfkSrzj6CWKnntsQ9rbgRJVUkGZtja3dSKBkC hPyCfhIFMORIePBVo7YaKILsiflxt9I2xFuwY7Pvfar/DYES+wiDv8jETTNDDKPnENNnCTIV7s Rl0= X-SBRS: 2.7 X-MesageID: 24576923 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24576923" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 55/60] history reporting: Break out minflight_by_time Date: Fri, 14 Aug 2020 18:22:00 +0100 Message-ID: <20200814172205.9624-56-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Move this from sg-report-host-history so we can reuse it. Signed-off-by: Ian Jackson --- Osstest/HistoryReport.pm | 16 ++++++++++++++++ sg-report-host-history | 13 ++----------- 2 files changed, 18 insertions(+), 11 deletions(-) diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm index 3e2ad0d3..b851ae45 100644 --- a/Osstest/HistoryReport.pm +++ b/Osstest/HistoryReport.pm @@ -37,6 +37,7 @@ BEGIN { cache_write_entry cache_finish parallel_by_fork + minflight_by_time ); %EXPORT_TAGS = (); @@ -282,4 +283,19 @@ sub parallel_by_fork ($$$$) { exit $worst; } +sub minflight_by_time ($) { + my ($timelimit) = @_; + my $minflightsq = db_prepare(<= ? + ORDER BY flight ASC + LIMIT 1 +END + my $now = time // die $!; + $minflightsq->execute($now - $timelimit); + my ($minflight,) = $minflightsq->fetchrow_array(); + $minflight //= 0; +} + 1; diff --git a/sg-report-host-history b/sg-report-host-history index 0a2e9904..efe652bb 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -95,19 +95,10 @@ sub computeflightsrange () { END $minflightsq->execute(); ($minflight,) = $minflightsq->fetchrow_array(); + $minflight //= 0; } else { - my $minflightsq = db_prepare(<= ? - ORDER BY flight ASC - LIMIT 1 -END - my $now = time // die $!; - $minflightsq->execute($now - $timelimit); - ($minflight,) = $minflightsq->fetchrow_array(); + $minflight = minflight_by_time($timelimit); } - $minflight //= 0; $flightcond = "(flight > $minflight)"; From patchwork Fri Aug 14 17:22:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714945 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D6CC013A4 for ; Fri, 14 Aug 2020 17:27:44 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B040E20708 for ; Fri, 14 Aug 2020 17:27:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="d2D4IP0K" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B040E20708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTO-00043b-QT; Fri, 14 Aug 2020 17:26:34 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dRM-00024H-2y for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:24:28 +0000 X-Inumbo-ID: 8045ba7c-ba19-4aad-af45-0587ec95cdba Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 8045ba7c-ba19-4aad-af45-0587ec95cdba; Fri, 14 Aug 2020 17:22:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425744; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=35wvHPbtsS4tCSQHRh9nXy4l+Oi4ard8rAoLOTjQwQ0=; b=d2D4IP0Khr7ewUNEvkJydKo04R6qeBv8RoSiWDuhidJDA0caJKGJpw+W 9qTc0T7A70hwdikm5SvXaPFoveUz4aBCAuSytqxmO7NxsPacnruqAOxAo AlVij8XVCW49w5dxSaYqNsEJ8B/KC8ImV3sLq705TNbh4jqbgq//uu6yK o=; Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: AJGGAk75SevsXl08laJ2+S4mDXZNhtia1LZ25Qh+RUY6nFuht/BPpRYY83D2jNqeBDKEXO4qfi JccJhqA2d5G0orCqEevCxL/69MOqJMueCOSLdWLkSRuLVwFsZyqlLZTnnmQitqZWNnH34G8Na6 6sfpV5IScV9I+j93ArXfXU7vbTtin0LBDO6f0H2s4v89ncld6yoq8tkVv/NK6flnL7k7ywu1jj s0hTZhSSkKia8zvJPE8gUZjM6QlQ0kNYVHy6CdahOoG5SgWhYnWipXNYcFxV1e+PtkJOvy31nJ nrI= X-SBRS: 2.7 X-MesageID: 24725193 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24725193" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 56/60] sg-report-job-history (nfc): Abolish $fromstuff Date: Fri, 14 Aug 2020 18:22:01 +0100 Message-ID: <20200814172205.9624-57-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" This used to be reused, but that is no longer the case. Do away with it, for clarity and simplicity. Signed-off-by: Ian Jackson --- sg-report-job-history | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/sg-report-job-history b/sg-report-job-history index 48dc2f57..2cac123b 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -140,13 +140,11 @@ END } my $offset= $limit-1; - my $fromstuff= <execute(@params); From patchwork Fri Aug 14 17:22:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714891 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EE53C618 for ; Fri, 14 Aug 2020 17:27:19 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CA64B20708 for ; Fri, 14 Aug 2020 17:27:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="FkWFF/r6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CA64B20708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTm-0004p7-P5; Fri, 14 Aug 2020 17:26:58 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dRm-00024Q-Jf for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:24:54 +0000 X-Inumbo-ID: 8ece59ae-6f11-4c58-bde3-308e5aa54450 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 8ece59ae-6f11-4c58-bde3-308e5aa54450; Fri, 14 Aug 2020 17:22:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425761; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=2hKHQMm9lLk+HPPyAXgeOLKFpKDe8arF7r1l0PzRvic=; b=FkWFF/r6MMbEmBQr07kuhErzeIx/aSLCMNO5M5kTZemDT3atGO10ZX+0 gern9u4hwLD62T53XRGjovHLA4zhi54VmQTtWWVZ7G68OsHQ0S6F3X/Ty nNsu/yivIsDQqNoV8HdpfV3TBjFT9A6CzP4F4CeTX+Cjf3Drizl8h+eXr c=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: Si1LiZ6+uGvgQeCovR0vFboL6vY7HEzbv+EJ2TxQnGItq26WNjNXEWlI+sutz2urnTRhOq2ZAN sgdVA+LQsKSiTHTpCrxC1WxXZCOmB+g18H7dRLhExvkU6wD2Of+W9E2ArUxDs0SNpDdUMUmbWg 4LkqsVBL81V+2+eBQneabxM59ogtcqrB5AcPPIne02CCuRPWVdzOTOQ5fW40AJFIvrG3ytIPH/ CrYpmIvxUn4+sEsL4804TJao+DDKGjLmvsnTDylckLwYCtJ2oSCpsQ9R2ec06YzULZ/GtetWok sLc= X-SBRS: 2.7 X-MesageID: 24576925 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24576925" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 57/60] sg-report-job-history: Cache report_run_getinfo Date: Fri, 14 Aug 2020 18:22:02 +0100 Message-ID: <20200814172205.9624-58-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" No logical change. Signed-off-by: Ian Jackson --- sg-report-job-history | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sg-report-job-history b/sg-report-job-history index 2cac123b..5d2a7e15 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -159,7 +159,7 @@ END while (my $f= $flightsq->fetchrow_hashref()) { cache_row_lookup_prep(\$f); - my $ri= report_run_getinfo($f); + my $ri= cacheable_fn($f, 'rri', sub { report_run_getinfo($f) }); $ri->{Hosts} = cacheable_fn($f, 'h', sub { my %hosts; From patchwork Fri Aug 14 17:22:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714985 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1CD0213A4 for ; Fri, 14 Aug 2020 17:28:13 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id ED02F20774 for ; Fri, 14 Aug 2020 17:28:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="dYnmCBfC" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ED02F20774 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTP-00044V-7i; Fri, 14 Aug 2020 17:26:35 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dRq-00024H-49 for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:24:58 +0000 X-Inumbo-ID: d5ab76a6-94e0-4c6a-93b5-237daa29d762 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id d5ab76a6-94e0-4c6a-93b5-237daa29d762; Fri, 14 Aug 2020 17:22:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425759; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=EbbvYXVycJDJO2skBAFjJlWzRMxKmAJD904vZ3lFpjI=; b=dYnmCBfC/jKMOz7TBA+ONSrddnvNfk5xnb9SLiPUWztJRth4tvVooqfd OJuHtcHSCZgHVWJKve4NMCCdkMuh20YZ3Z2o0PMBRyrXVSodL4EBFTBne SxgRZavWuiSJF6h9fQfkjGdbYyVyRSFV5ANovv+YvjIIA1LA5P3z+57A0 0=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: Ttne0r5bkUnbUn8tKVnj2W5yAnkY9A665WG9yujMtyhHyu49ixe82UGhah0IkSfUYWyNivVWPR 5H1DVurFh7Y0Tbpy5My9AZ4N3gnRcuaRpqMeCdnHnWpxMAPhW2EsZUP2Gulv+QoH9k6qEecZR0 OmiKb6fAKuX9f+LVlvQc1ItcYMT8EScEmAwasSbC6ehupnFvNehm9p3+965bfIcYZmYNBXK4YO T6p00m/1MCrAGa77M9Zg8yFww0kUTy2lzE78MOAIthLe3zx5PWyNIv/53IgFNb0J8E930QrEeX J74= X-SBRS: 2.7 X-MesageID: 24576928 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24576928" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 58/60] sg-report-host-history: Cache report_run_getinfo Date: Fri, 14 Aug 2020 18:22:03 +0100 Message-ID: <20200814172205.9624-59-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" No logical change. Signed-off-by: Ian Jackson --- sg-report-host-history | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sg-report-host-history b/sg-report-host-history index efe652bb..a814ef37 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -266,7 +266,9 @@ END print H "$jr->{job}\n"; - my $ri = report_run_getinfo({ %$jr, %$ir }); + my $ri = cacheable_fn($jr, 'rri', sub { + report_run_getinfo({ %$jr, %$ir }) + }); print H "{ColourAttr}>$ri->{Content}\n"; my $powers = cacheable_fn($jr, 'b', sub { From patchwork Fri Aug 14 17:22:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714921 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0AF2C618 for ; Fri, 14 Aug 2020 17:27:36 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DA28520791 for ; Fri, 14 Aug 2020 17:27:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="VYkpXh6q" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DA28520791 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTs-00053q-TO; Fri, 14 Aug 2020 17:27:04 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dRR-00024H-36 for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:24:33 +0000 X-Inumbo-ID: 1a689d73-2d84-4814-9b5c-03cf1c59b582 Received: from esa3.hc3370-68.iphmx.com (unknown [216.71.145.155]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 1a689d73-2d84-4814-9b5c-03cf1c59b582; Fri, 14 Aug 2020 17:22:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425746; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=vk1erg2EhIozZ671QuB2WFTSrxHDQXDO2TUsLVUzIrA=; b=VYkpXh6qyq/3nOnPnhqHd5XrTOxc09fxGuTuAbwZYbp05i40Vzu765Cf yv0MeqW0O1UZqw4oE80JcCsFwG97mkrNKAUjzmyFMnK63zDj2u2AHSP/E 1QWF4ukj42v3HLQmiR/ZSV5qVoYs33TkDbIAgtS54ocL8p5Bew923soeU o=; Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 2GK7l3G/uOcKXbY4SjRQWQUlQEzoo5dZmdQlsaPFIRd6/Bs+9n55B0Vghq5A0xGQtUfKni4ICU +5qZt2lVKqGkOPDCXSNi1JSmSP74g64HGAmdZBJsyBSPgHzfpv+97ea9FKR6uNd3KPsY7qbpah YxR9GumtmQbVyhm4RVuTe1YGNXeU2YCtSQ5dR6efxtAteefE5L8ygiEcm8YAYPjijw9RGgi/s6 38kto9Jv2O2LiOnVD2QR7G5cvLGQSd3pa5LvemQ4jabUThiukgOzTlOvbDAru1PCzlvnrecGKb Ma0= X-SBRS: 2.7 X-MesageID: 24545479 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24545479" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 59/60] sg-report-job-history: Provide --time-limit Date: Fri, 14 Aug 2020 18:22:04 +0100 Message-ID: <20200814172205.9624-60-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Calculate a minflight based on the time limit, and set the time limit to a year ago by default. Signed-off-by: Ian Jackson --- sg-report-job-history | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/sg-report-job-history b/sg-report-job-history index 5d2a7e15..e09c694f 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -30,6 +30,7 @@ use Osstest::HistoryReport; our (@blessings,@branches); our $limit= 100; +our $timelimit= 86400 * (366 + 14); our $htmlout = '.'; our $flight; our $maxjobs=10; @@ -44,6 +45,8 @@ while (@ARGV && $ARGV[0] =~ m/^-/) { $$1= $2; } elsif (m/^--(limit)\=([1-9]\d*)$/) { $$1= $2; + } elsif (m/^--time-limit\=([1-9]\d*)$/) { + $timelimit= $1; } elsif (restrictflight_arg($_)) { # Handled by Executive } elsif (m/^--html-dir=(.*)$/) { @@ -67,6 +70,9 @@ csreadconfig(); our @jobs; +our $minflight = minflight_by_time($timelimit); +print DEBUG "MINFLIGHT $minflight\n"; + sub findflight () { my $branches= $dbh_tests->selectcol_arrayref(< $minflight ORDER BY flight DESC LIMIT $limit END From patchwork Fri Aug 14 17:22:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11714883 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A1B7D618 for ; Fri, 14 Aug 2020 17:27:10 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7DD0D20708 for ; Fri, 14 Aug 2020 17:27:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="UGWE1S45" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7DD0D20708 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dTf-0004Xa-Ri; Fri, 14 Aug 2020 17:26:51 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k6dQZ-00024Q-Gy for xen-devel@lists.xenproject.org; Fri, 14 Aug 2020 17:23:39 +0000 X-Inumbo-ID: 1d35dbac-2730-4307-8984-dfc8985cc8d2 Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 1d35dbac-2730-4307-8984-dfc8985cc8d2; Fri, 14 Aug 2020 17:22:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1597425745; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=HuGiUl9r9GZbvDQ8LofQzM7tXyOp/3WzRBA9aRRSObY=; b=UGWE1S45GUao3fFbrNyfRPnFwXgou8DQJQBzvIJ6LjDG2uBSWV6xMils USPuLkZfV1i3/RmWrJECkap5x+Eb0HsodW5/LMNt1lk1QAJ24Z2eIbSXV b6qbn37kwi+onxHlRN8o+/AByyt4e+E4wH+zt6oFgHxEqgN20A+LcAwEc w=; Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: V6otqD9WK/PLcAD9hC3YWrdxwFbF1aLBK+Y0vWEvhBSBJiGErkW+Q8MjVFZUyWBx1MkWGwguTG vRRFn5hybXX+hFJbOwaljfjIW2J3qyRlrukSZdl37xavPM9IaiNraWqvRHLeJm0aAPUXyfUpj+ Vo2e9YUviAX5nh59y6KmX87skxjY08lt9Tcq9iqUNiE90PTzI8NNOisoniFjKHhCYbEaeS+YLr HBkRzD/fjY/8IQChshxOPLSmyWQ358KN9AEKCtwiewf8OexZfHhUFno2xp2khx4RCkumloIuEb rFw= X-SBRS: 2.7 X-MesageID: 24879752 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.76,313,1592884800"; d="scan'208";a="24879752" From: Ian Jackson To: CC: Ian Jackson Subject: [OSSTEST PATCH 60/60] sg-report-job-history: Increase default limit Date: Fri, 14 Aug 2020 18:22:05 +0100 Message-ID: <20200814172205.9624-61-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com> References: <20200814172205.9624-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Now this is a *lot* faster, we can print a lot more history. Signed-off-by: Ian Jackson --- sg-report-job-history | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sg-report-job-history b/sg-report-job-history index e09c694f..843dd776 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -29,7 +29,7 @@ use Osstest::Executive; use Osstest::HistoryReport; our (@blessings,@branches); -our $limit= 100; +our $limit= 2000; our $timelimit= 86400 * (366 + 14); our $htmlout = '.'; our $flight;