From patchwork Mon Jul 24 14:41:44 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 9859613 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 45EB0600F5 for ; Mon, 24 Jul 2017 14:44:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 370BF2808F for ; Mon, 24 Jul 2017 14:44:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2C00C283D9; Mon, 24 Jul 2017 14:44:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 37AC528419 for ; Mon, 24 Jul 2017 14:44:41 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dZeYj-00082m-1L; Mon, 24 Jul 2017 14:42:09 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dZeYh-00081N-Jq for xen-devel@lists.xenproject.org; Mon, 24 Jul 2017 14:42:07 +0000 Received: from [85.158.143.35] by server-2.bemta-6.messagelabs.com id 60/3A-27137-F3706795; Mon, 24 Jul 2017 14:42:07 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprMIsWRWlGSWpSXmKPExsXitHSDva4de1m kwf+fWhbft0xmcmD0OPzhCksAYxRrZl5SfkUCa8bWp8kF18Irpr3rYG9g3OXcxcjJISHgL9F6 uZkFxGYT0JVo2vKXrYuRg0NEQEXi9l4DkDCzQK3Eqj0z2EFsYYFoieMvzzCC2CwCqhKtF38zg 9i8Ap4Sc5umMUGMlJM4f/wnWJxTwEvi9PfnrCAjhQTqJbY2yEOYahJz18dDdApKnJz5hAVik4 TEwRcvmCcw8s5CkpqFJLWAkWkVo0ZxalFZapGuobFeUlFmekZJbmJmjq6hgZlebmpxcWJ6ak5 iUrFecn7uJkZg0DAAwQ7GL8sCDjFKcjApifIKMZVFCvEl5adUZiQWZ8QXleakFh9ilOHgUJLg /cEKlBMsSk1PrUjLzAGGL0xagoNHSYS3iA0ozVtckJhbnJkOkTrFaMyxoGfDFyaOVxP+f2MSY snLz0uVEudVBikVACnNKM2DGwSLq0uMslLCvIxApwnxFKQW5WaWoMq/YhTnYFQS5o0AmcKTmV cCt+8V0ClMQKfMmVEKckpJIkJKqoFx8S7TmsDnutyFFleChDzOTONaG7d8n85X+T0hz7rE23O 3PL77Mn7dxhXswXN0/4pP6rl1fGfxuW2P4vnS26YYnnKeVv5TfsW/ZoFFob9NNf+9eMyrvH3d V75FTzXf/27UPBDVr7huxnM5Xgvjo1O2639uW/6N/2ilfpHsi1NcLJe6sh5sT2xjUmIpzkg01 GIuKk4EAKPHk/mmAgAA X-Env-Sender: prvs=371027d51=Ian.Jackson@citrix.com X-Msg-Ref: server-2.tower-21.messagelabs.com!1500907322!60769322!2 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n, received_headers: No Received headers X-StarScan-Received: X-StarScan-Version: 9.4.25; banners=-,-,- X-VirusChecked: Checked Received: (qmail 33023 invoked from network); 24 Jul 2017 14:42:06 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-2.tower-21.messagelabs.com with RC4-SHA encrypted SMTP; 24 Jul 2017 14:42:06 -0000 X-IronPort-AV: E=Sophos;i="5.40,407,1496102400"; d="scan'208";a="441062538" From: Ian Jackson To: Date: Mon, 24 Jul 2017 15:41:44 +0100 Message-ID: <1500907304-12791-4-git-send-email-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1500907304-12791-1-git-send-email-ian.jackson@eu.citrix.com> References: <22896.54014.418454.891770@mariner.uk.xensource.com> <1500907304-12791-1-git-send-email-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Cc: Ian Jackson , Roger Pau Monne Subject: [Xen-devel] [OSSTEST PATCH 3/3] anointments: Introduce anointment scheme, and mg-anoint script X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP CC: Roger Pau Monne Signed-off-by: Ian Jackson --- README.planner | 4 + mg-anoint | 347 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 351 insertions(+) create mode 100755 mg-anoint diff --git a/README.planner b/README.planner index b3b41a9..255460b 100644 --- a/README.planner +++ b/README.planner @@ -133,6 +133,10 @@ Types of task mg-execute-flight). They are automatically created and destroyed - see above. + * `anoint' tasks. These are not really ongoing tasks; but they + represent something that can `own' a flight. Each anointment kind + is represented by an `anoint' task. + * `xdbref' tasks. These are used to own resources whose allocation authority has been transferred to a separate database, eg a test database. The refkey is an indication of the other database. diff --git a/mg-anoint b/mg-anoint new file mode 100755 index 0000000..4bb8ab0 --- /dev/null +++ b/mg-anoint @@ -0,0 +1,347 @@ +#!/usr/bin/perl -w +# +# usage: +# +# ./mg-anoint prepare [PARAM...] REFKEY DESCRIPTION +# REFKEY should be 'SCOPE DETAILS...' +# PARAMs are: +# --keep= default is 3 +# --blessings=,... default is just "real" +# +# ./mg-anoint destroy REFKEY +# +# ./mg-anoint anoint [ANOINT-OPTION...] REFKEY FLIGHT JOB +# ANOINT-OPTIONs are: +# --allow-blessed=BLESSING,... default is from `prepare' +# --allow-job-status=STATUS,... default is only `pass' +# +# ./mg-anoint retrieve REFKEY +# => FLIGHT JOB +# if nothing anointed yet, prints nothing and exits 0 +# if anointment not prepared, fails +# +# ./mg-anoint list +# => human-readable output + +# This is part of "osstest", an automated testing framework for Xen. +# Copyright (C) 2009-2013,2017 Citrix Inc. +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see . + +# Each flavour of anointment is represented by a row in tasks: +# +# type anoint +# refkey +# the flavour, eg freebsd build +# username NULL +# comment +# live t +# refinfo keep= blessings=,... +# +# Each anointment is represented by a row in resources: +# +# restype 'share-flight' +# resname +# owntaskid +# shareix +# subtask + + +use strict qw(vars refs); +use DBI; +unshift @INC, qw(.); +use Osstest; +use Osstest::TestSupport; +use Osstest::Executive; +use IO::Handle; + +csreadconfig(); + +our @o; +sub pr_o () { print $_, "\n" or die $! foreach @o; } + +our $anointments_qtxt = <prepare(<prepare(< 3, + blessings => 'real'); + while (@ARGV && $ARGV[0] =~ m/^-/) { + $_= shift @ARGV; + last if m/^\-\-?$/; + if (m/^--(keep)=(\d+)$/) { + $params{$1} = $2 + 0; + } elsif (m/^--(blessings)=([-+0-9a-z,]+)$/) { + $params{$1} = $2; + } else { + die "unknown option $_ ?"; + } + } + die "usage: mg-anoint prepare [OPTIONS] 'SCOPE DETAILS...' DESCRIPTION" + unless @ARGV==2 || $ARGV[0] =~ m/ /; + my ($refkey, $description) = @ARGV; + my $dq = $dbh_tests->prepare(<prepare(<execute($refkey); + $iq->execute($refkey, $description, $refinfo); + }); + pr_o(); +} + +sub cmd_destroy { + die unless @ARGV==1; + die if $ARGV[0] =~ m/^-/; + my ($refkey) = @ARGV; + my $rdq = $dbh_tests->prepare(<prepare(<execute($refkey); + $tdq->execute($refkey); + }); + pr_o(); +} + +sub cmd_anoint { + my $allow_blessed = ''; + my $allow_jobstatus = ''; + while (@ARGV && $ARGV[0] =~ m/^-/) { + $_= shift @ARGV; + last if m/^\-\-?$/; + if (m/^--allow-blessed=/) { + $allow_blessed = $'; + } elsif (m/^--allow-job-status=/) { + $allow_jobstatus = $'; + } else { + die "unknown option $_ ?"; + } + } + die unless @ARGV==3; + my ($refkey, $flight, $job) = @ARGV; + + prep_queries(); + + my $newflight_q = $dbh_tests->prepare(<prepare(<prepare(<prepare(<prepare(<prepare(<execute($refkey); + + # find the task row (ie, the anointment kind) + my ($task, $refinfo) = $task_q->fetchrow_array(); + die "no such anointment kind \`$refkey' (no prepare?)\n" + unless defined $task; + my %params; + foreach (split /\s+/, $refinfo) { + die unless m/=/; + $params{$`} = $'; + } + my %blessings; + $blessings{$_}++ foreach + grep /./, + (split /,/, $params{blessings}), + (split /,/, $allow_blessed); + + my %jobstatus; + $jobstatus{pass}++; + $jobstatus{$_}++ foreach grep /./, split /,/, $allow_jobstatus; + + # check the to-be-anointed flight's blessing + $newflight_q->execute($flight); + my $frow = $newflight_q->fetchrow_hashref(); + die "flight $flight missing" unless $frow; + die "flight $flight not started" unless defined $frow->{started}; + + # check the job status + $newjob_q->execute($flight, $job); + my ($jstatus) = $newjob_q->fetchrow_array(); + die "job $flight.$job missing" unless defined $jstatus; + die "job $flight.$job status $jstatus" unless $jobstatus{$jstatus}; + + push @o, "flight $flight blessed $frow->{blessing}". + " started ".show_abs_time($frow->{started}); + + die "flight $flight blessing $frow->{blessing}". + " (not $params{blessings} / $allow_blessed)" + unless $blessings{ $frow->{blessing} }; + + # check to-be-annointed flight is most recent + $mostrecent_q->execute($task); + my $mostrecent = $mostrecent_q->fetchrow_hashref(); + die "flight $flight not newer than $mostrecent->{flight}" + unless $frow->{started} > ($mostrecent->{started} // 0); + + # expire old anointments + $count_q->execute($task); + my ($current) = $count_q->fetchrow_array(); + my $want_delete = ($current+1) - $params{keep}; + push @o, "anointment $refkey: currently $current anointed"; + if ($want_delete > 0) { + $todelete_q->execute($task, $want_delete); + while (my $d = $todelete_q->fetchrow_hashref()) { + push @o, " expiring $d->{flight}.$d->{job} [/$d->{shareix}]". + " started ".show_abs_time($d->{started}); + $delete_res_q->execute($task, $d->{flight}, $d->{shareix}); + } + } + + # at last! + $insert_q->execute($flight,$task,$task,$job); + push @o, "anointed $flight.$job"; + }); + pr_o(); +} + +sub cmd_retrieve { + die unless @ARGV==1; + die if $ARGV[0] =~ m/^-/; + my ($refkey) = @ARGV; + + prep_queries(); + + db_retry($dbh_tests, [], sub { + @o = (); + $task_q->execute($refkey); + my ($task) = $task_q->fetchrow_array(); + die "no such anointment kind \`$refkey'" unless defined $task; + + $mostrecent_q->execute($task); + my $row = $mostrecent_q->fetchrow_hashref(); + if ($row) { + push @o, "$row->{flight} $row->{job}"; + } else { + print STDERR "warning: nothing anointed $refkey\n"; + } + }); + pr_o(); +}; + +sub cmd_list { + die "no options to list" if @ARGV; + + my $tq = $dbh_tests->prepare(<prepare(<execute(); + while (my $t = $tq->fetchrow_hashref()) { + push @o, sprintf "%s [%s] (%s)", + $t->{refkey}, $t->{refinfo}, $t->{comment}; + push @o, " ! NOT LIVE" unless $t->{live}; + push @o, " ! username=$t->{username}" if defined $t->{username}; + $rq->execute($t->{taskid}); + while (my $r = $rq->fetchrow_hashref()) { + if ($r->{restype} eq 'share-flight') { + push @o, " $r->{resname}.$r->{subtask}"; + } else { + push @o, " ! $r->{restype}/$r->{resname}/$r->{subtask}"; + } + } + } + }); + pr_o(); +} + +die 'need operation' unless @ARGV; +die 'no global options understood' if $ARGV[0] =~ m/^-/; + +my $subcmd= shift @ARGV; +$subcmd =~ s/-/_/g; +my $subcmdproc = ${*::}{"cmd_$subcmd"}; +die "unknown subcommand" unless $subcmdproc; +$subcmdproc->();