| Message ID | 1504878756-28401-1-git-send-email-ian.jackson@eu.citrix.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Fri, Sep 08, 2017 at 02:52:36PM +0100, Ian Jackson wrote: > CC: George Dunlap <george.dunlap@citrix.com> > Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com> Just one comment, which is probably pointless... > --- > README | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------------- > 1 file changed, 60 insertions(+), 15 deletions(-) > > diff --git a/README b/README > index ffe0018..93129e3 100644 > --- a/README > +++ b/README > @@ -8,32 +8,55 @@ Terminology > > "flight": > > - Each run of osstest is referred to as a "flight". Each flight is > - given a unique ID (a number or name). > + Each run of osstest is referred to as a "flight". Each flight is > + given a unique ID. > + > + Standalone mode generally uses a flight named "standalone", which > + is frequently erased and reused. In "Executive" mode (used for > + production osstest instances) flights are numbered and the > + metadata is permanently recorded. > > "job": > > - Each flight consists of one or more "jobs". These are a sequence > + Each flight consists of one or more "jobs". These are a sequence > of test steps run in order and correspond to a column in the test > - report grid. They have names like "build-amd64" or > - "test-amd64-amd64-pv". A job can depend on the output of another > + report grid. They have names like "build-amd64" or > + "test-amd64-amd64-pv". A job can depend on the output of another > job in the flight -- e.g. most test-* jobs depend on one or more > build-* jobs. > > + A job has a named "recipe", which indicates what things need to be ^ steps maybe? Thanks, Roger.
On 09/08/2017 02:52 PM, Ian Jackson wrote: > CC: George Dunlap <george.dunlap@citrix.com> > Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> This is a significant improvement, thanks: Reviewed-by: George Dunlap <george.dunlap@citrix.com> > --- > README | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------------- > 1 file changed, 60 insertions(+), 15 deletions(-) > > diff --git a/README b/README > index ffe0018..93129e3 100644 > --- a/README > +++ b/README > @@ -8,32 +8,55 @@ Terminology > > "flight": > > - Each run of osstest is referred to as a "flight". Each flight is > - given a unique ID (a number or name). > + Each run of osstest is referred to as a "flight". Each flight is > + given a unique ID. > + > + Standalone mode generally uses a flight named "standalone", which > + is frequently erased and reused. In "Executive" mode (used for > + production osstest instances) flights are numbered and the > + metadata is permanently recorded. > > "job": > > - Each flight consists of one or more "jobs". These are a sequence > + Each flight consists of one or more "jobs". These are a sequence > of test steps run in order and correspond to a column in the test > - report grid. They have names like "build-amd64" or > - "test-amd64-amd64-pv". A job can depend on the output of another > + report grid. They have names like "build-amd64" or > + "test-amd64-amd64-pv". A job can depend on the output of another > job in the flight -- e.g. most test-* jobs depend on one or more > build-* jobs. > > + A job has a named "recipe", which indicates what things need to be > + done in what order: generally, a sequence of ts-* scripts. The > + recipes are defined by their implementations in sg-run-job. > + > + The set of jobs for any particular kind of flight is defined in > + make-flight, make-*-flight, and mfi-*. > + > "step": > > - Each job consists of multiple "steps" which is an individual test > - operation, such as "build the hypervisor", "install a guest", > - "start a guest", "migrate a guest", etc. A step corresponds to a > - cell in the results grid. A given step can be reused in multiple > - different jobs, e.g. the "xen build" step is used in several > - different build-* jobs. This reuse can be seen in the rows of the > - results grid. > + Running a job consists of running its individual "steps". Each > + step is an individual test operation, such as "build the > + hypervisor", "install a guest", "start a guest", "migrate a > + guest", etc. > + > + Generally a step corresponds to one execution of some ts-* script. > + > + steps have a "testid" which is used to uniquely identify the same > + operation in different test runs. This is used for identifying > + regressions, bisection, and so on. > + > + The steps for a job, including the testids, are defined by the > + recipe in sg-run-job (see "job", above). > + > + Each step run in a job becomes a cell in the HTML results grid. A > + given step might appear in multiple different jobs, e.g. the "xen > + build" step is used in several different build-* jobs. This can > + be seen in the rows of the results grid. > > "runvar": > > - A runvar is a named textual variable associated with each job in a > - given flight. They serve as both the inputs and outputs to the > + A runvar is a named textual variable associated with a given job > + and flight. They serve as both the inputs and outputs to the > job. > > For example a Xen build job may have input runvars "tree_xen" (the > @@ -55,6 +78,15 @@ Terminology > * the parameters of the guest to test (e.g. distro, PV vs HVM > etc). > > + Runvar names often have structure, being assembled out of various > + pieces; and the names are sometimes parsed, too. > + > +flights, jobs and runvars are kept in the database (in standalone > +mode, "standalone.db"). As for steps, only steps which have been > +started are recorded in the db; steps which have yet to be attempted > +are not in the database. Also, if a ts-* script is run by hand, this > +is not recorded as a step. > + > Operation > ========= > > @@ -73,12 +105,25 @@ referenced by each job's configuration. It then runs each of these in > turn, taking into account the prerequisites etc, by calling the > relevant "ts-*" scripts. > > +Each ts-* is a convenient unit of test execution and reporting. Most > +ts-* scripts are written in perl, although simple shell wrappers are > +sometimes used. ts-* scripts expect OSSTEST_FLIGHT and OSSTEST_JOB to > +be set in the environment. Most of them also expect to be told extra > +information on the command line - notably, which host(s) to operate > +on. > + > +In automatic operation, the command line arguments for each test > +step's ts-* script invocation are fixed by the recipe defined in > +sg-run-job. > + > When running in standalone mode it is possible to run any of these > steps by hand, ("mg-execute-flight", "sg-run-job", "ts-*") although > you will need to find the correct inputs (some of which are documented > below) and perhaps take care of prerequisites yourself (e.g. running > "./sg-run-job test-armhf-armhf-xl" means you must have done > -"./sg-runjob build-armhf" and "build-armhf-pvops" first. > +"./sg-runjob build-armhf" and "build-armhf-pvops" first. When running > +a ts-* script manually one often wants to specify "host=<hostname>". > +(See the head comment for "selecthost" in Osstest/TestSupport.pm.) > > Results > ======= >
diff --git a/README b/README index ffe0018..93129e3 100644 --- a/README +++ b/README @@ -8,32 +8,55 @@ Terminology "flight": - Each run of osstest is referred to as a "flight". Each flight is - given a unique ID (a number or name). + Each run of osstest is referred to as a "flight". Each flight is + given a unique ID. + + Standalone mode generally uses a flight named "standalone", which + is frequently erased and reused. In "Executive" mode (used for + production osstest instances) flights are numbered and the + metadata is permanently recorded. "job": - Each flight consists of one or more "jobs". These are a sequence + Each flight consists of one or more "jobs". These are a sequence of test steps run in order and correspond to a column in the test - report grid. They have names like "build-amd64" or - "test-amd64-amd64-pv". A job can depend on the output of another + report grid. They have names like "build-amd64" or + "test-amd64-amd64-pv". A job can depend on the output of another job in the flight -- e.g. most test-* jobs depend on one or more build-* jobs. + A job has a named "recipe", which indicates what things need to be + done in what order: generally, a sequence of ts-* scripts. The + recipes are defined by their implementations in sg-run-job. + + The set of jobs for any particular kind of flight is defined in + make-flight, make-*-flight, and mfi-*. + "step": - Each job consists of multiple "steps" which is an individual test - operation, such as "build the hypervisor", "install a guest", - "start a guest", "migrate a guest", etc. A step corresponds to a - cell in the results grid. A given step can be reused in multiple - different jobs, e.g. the "xen build" step is used in several - different build-* jobs. This reuse can be seen in the rows of the - results grid. + Running a job consists of running its individual "steps". Each + step is an individual test operation, such as "build the + hypervisor", "install a guest", "start a guest", "migrate a + guest", etc. + + Generally a step corresponds to one execution of some ts-* script. + + steps have a "testid" which is used to uniquely identify the same + operation in different test runs. This is used for identifying + regressions, bisection, and so on. + + The steps for a job, including the testids, are defined by the + recipe in sg-run-job (see "job", above). + + Each step run in a job becomes a cell in the HTML results grid. A + given step might appear in multiple different jobs, e.g. the "xen + build" step is used in several different build-* jobs. This can + be seen in the rows of the results grid. "runvar": - A runvar is a named textual variable associated with each job in a - given flight. They serve as both the inputs and outputs to the + A runvar is a named textual variable associated with a given job + and flight. They serve as both the inputs and outputs to the job. For example a Xen build job may have input runvars "tree_xen" (the @@ -55,6 +78,15 @@ Terminology * the parameters of the guest to test (e.g. distro, PV vs HVM etc). + Runvar names often have structure, being assembled out of various + pieces; and the names are sometimes parsed, too. + +flights, jobs and runvars are kept in the database (in standalone +mode, "standalone.db"). As for steps, only steps which have been +started are recorded in the db; steps which have yet to be attempted +are not in the database. Also, if a ts-* script is run by hand, this +is not recorded as a step. + Operation ========= @@ -73,12 +105,25 @@ referenced by each job's configuration. It then runs each of these in turn, taking into account the prerequisites etc, by calling the relevant "ts-*" scripts. +Each ts-* is a convenient unit of test execution and reporting. Most +ts-* scripts are written in perl, although simple shell wrappers are +sometimes used. ts-* scripts expect OSSTEST_FLIGHT and OSSTEST_JOB to +be set in the environment. Most of them also expect to be told extra +information on the command line - notably, which host(s) to operate +on. + +In automatic operation, the command line arguments for each test +step's ts-* script invocation are fixed by the recipe defined in +sg-run-job. + When running in standalone mode it is possible to run any of these steps by hand, ("mg-execute-flight", "sg-run-job", "ts-*") although you will need to find the correct inputs (some of which are documented below) and perhaps take care of prerequisites yourself (e.g. running "./sg-run-job test-armhf-armhf-xl" means you must have done -"./sg-runjob build-armhf" and "build-armhf-pvops" first. +"./sg-runjob build-armhf" and "build-armhf-pvops" first. When running +a ts-* script manually one often wants to specify "host=<hostname>". +(See the head comment for "selecthost" in Osstest/TestSupport.pm.) Results =======
CC: George Dunlap <george.dunlap@citrix.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> --- README | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 60 insertions(+), 15 deletions(-)