[OSSTEST,5/5,v2] mg-show-flight-runvars: recurse on buildjobs upon request

Commit Message

Ian Campbell Feb. 3, 2016, 9:48 a.m. UTC

By looping over @rows looking for buildjobs runvars and adding those
jobs to the output until nothing changes.

The output is resorted by runvar name which is the desired default
behaviour. As usual can be piped to sort(1) to sort by flight+job.

Signed-off-by: Ian Campbell <[e-mail]>
---
v2:
 - Use $jobcond,@jobconfparams to avoid SQL injection
 - Only recurse if the option was given
 - Drop synth from ORDER BY
 - Use a Schwatzian transform for the sort, at the same time allowing
   retention of the sorting of synth runvars last.
---
 mg-show-flight-runvars | 44 +++++++++++++++++++++++++++++++++++++-------
 1 file changed, 37 insertions(+), 7 deletions(-)

Comments

Ian Jackson Feb. 3, 2016, 12:01 p.m. UTC | #1

Ian Campbell writes ("[PATCH OSSTEST 5/5 v2] mg-show-flight-runvars: recurse on buildjobs upon request"):
> By looping over @rows looking for buildjobs runvars and adding those
> jobs to the output until nothing changes.
> 
> The output is resorted by runvar name which is the desired default
> behaviour. As usual can be piped to sort(1) to sort by flight+job.
...
> +if ($recurse) {
> +    foreach my $row (@rows) {
> +	next unless $row->[1] =~ m/^(?:.*_)?([^_]*)buildjob$/;
> +	next if $jobs{$row->[2]};

This does the deduping based on $row->[2] which is the runvar value.
But the same job might be referred to by different runvar values.  I
think this means that you might report the same job twice.

If
   1234.foo  buildjob   wombat
   1234.bar  buildjob   1234.wombat
then you dump 1234.wombat twice.

> +	my ($oflight, $ojob) = flight_otherjob($tflight, $row->[2]);

Also the place you set $jobs{} is somewhat wrong.  If there is a job
with no rows you can process it twice (harmlessly, I think, as it
happens).


So I think you want
        next if $jobs{$oflight,$ojob}++;

> +	collect($oflight, "job = ?", $ojob);
> +
> +	# collect() appends to @rows, so we don't need any special
> +	# handling to pickup anything which was newly added.

I don't think this is true.  Or at least, it happens to be true, but
my copy of the FM says:

       If any part of LIST is an array, "foreach" will get very
       confused if you add or remove elements within the loop body,
       for example with "splice".  So don't do that.

AIUI you're avoiding actual recursion, inside collect()'s row loop, to
avoid trying to have one SQL query for each nesting level.  It might
be worth mentioning that somewhere.

Maybe you should have the foreach use an explicit row index.

Ian.

Patch

diff --git a/mg-show-flight-runvars b/mg-show-flight-runvars
index f96539f..faa0ea1 100755
--- a/mg-show-flight-runvars
+++ b/mg-show-flight-runvars
@@ -46,30 +46,56 @@  for (;;) {
 
 die unless @ARGV==1 && $ARGV[0] =~ m/^\w+$/;
 
-
 our @cols = qw(job name val);
 our @rows;
+our %jobs;
+
+sub collect ($;$@) {
+    my ($flight,$jobcond,@jobcondparams) = @_;
 
-sub collect ($) {
-    my ($flight) = @_;
+    $jobcond //= "TRUE";
 
     $flight =~ m/^\d+/ or $flight = "'$flight'";
-    my $qfrom = "FROM runvars WHERE flight=$flight AND $synthcond";
+    my $qfrom = "FROM runvars WHERE flight=$flight AND $synthcond AND $jobcond";
 
     my $q = $dbh_tests->prepare
-	("SELECT synth, ".(join ',', @cols)." $qfrom ORDER BY synth, name, job");
-    $q->execute();
+	("SELECT synth, ".(join ',', @cols)." $qfrom ORDER BY name, job");
+    $q->execute(@jobcondparams);
 
     while (my (@row) = $q->fetchrow_array()) {
 	my $synth = shift @row;
 	$row[0] = "$flight.$row[0]" if $recurse;
 	$row[1] .= $synthsufx if $synth && $synth ne 'f'; # sqlite3 is typeless
 	push @rows, \@row;
+	$jobs{$row[0]} = 1;
     }
 }
 
 collect($ARGV[0]);
 
+if ($recurse) {
+    foreach my $row (@rows) {
+	next unless $row->[1] =~ m/^(?:.*_)?([^_]*)buildjob$/;
+	next if $jobs{$row->[2]};
+
+	# parse this flight and job, which must be in $flight.$job
+	# form if $recurse is true (see collect())
+	my ($tflight, $tjob) = flight_otherjob(undef, $row->[0]);
+	die "$row->[1]" unless $tflight;
+
+	# parse the buildjob reference and recurse. might be a job in
+	# this flight, in which case we still recurse since it might
+	# be a chain from a non-top-level job which hasn't been
+	# included yet. %jobs will prevent us from duplicating or
+	# infinite loops.
+	my ($oflight, $ojob) = flight_otherjob($tflight, $row->[2]);
+	collect($oflight, "job = ?", $ojob);
+
+	# collect() appends to @rows, so we don't need any special
+	# handling to pickup anything which was newly added.
+    }
+}
+
 our @colws;
 sub max ($$) { $_[$_[0] < $_[1]] }
 foreach my $row (@rows) {
@@ -77,7 +103,11 @@  foreach my $row (@rows) {
 }
 $colws[1] += length $synthsufx;
 
-foreach my $row (@rows) {
+# Sort by runvar name, then (flight+)job, synth runvars come last.
+foreach my $row (map { $_->[0] }
+		 sort { $a->[1] cmp $b->[1] }
+		 map { [ $_, ($_->[1] =~ m/~$/)." $_->[1] $_->[0]" ] }
+		 @rows) {
     printf "%-*s %-*s %-*s\n", map { $colws[$_], $row->[$_] } qw(0 1 2)
         or die $!;
 }