| Message ID | 20161114123334.5585-3-anthony.perard@citrix.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Mon, Nov 14, 2016 at 12:33:33PM +0000, Anthony PERARD wrote: > This script runs the OpenStack integration test suite, Tempest. > > Signed-off-by: Anthony PERARD <anthony.perard@citrix.com> > Acked-by: Ian Campbell <ian.campbell@citrix.com> > Acked-by: Ian Jackson <ian.jackson@eu.citrix.com> > > --- > Changes in V7: > - reindent > - acked > > No change in V5 > > Change in V4: > - use \Q\E for tests names > - write the full name of the tests to skip > - use push @ignored_test then join() > - use variables to store common prefix > - rewrite comments > > Change in V3: > - Use host as argument of the script. > - Use selecthost() and get rid of $gho. > - Use target_jobdir() instead of builddirsprops(). > - Put the ignored Tempest tests into a var and try to explain why there are > skip. > > --- > --- > sg-run-job | 1 + > ts-openstack-tempest | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 65 insertions(+) > create mode 100755 ts-openstack-tempest > > diff --git a/sg-run-job b/sg-run-job > index 7b9296c..f086e44 100755 > --- a/sg-run-job > +++ b/sg-run-job > @@ -477,6 +477,7 @@ proc run-job/test-rumprun {} { > proc need-hosts/test-devstack {} { return host } > proc run-job/test-devstack {} { > run-ts . = ts-openstack-deploy host > + run-ts . = ts-openstack-tempest host > } > > if {[file exists sg-run-job-adhoc]} { > diff --git a/ts-openstack-tempest b/ts-openstack-tempest > new file mode 100755 > index 0000000..8cc993a > --- /dev/null > +++ b/ts-openstack-tempest > @@ -0,0 +1,64 @@ > +#!/usr/bin/perl > +# This is part of "osstest", an automated testing framework for Xen. > +# Copyright (C) 2015 Citrix Inc. 2016. > +# > +# This program is free software: you can redistribute it and/or modify > +# it under the terms of the GNU Affero General Public License as published by > +# the Free Software Foundation, either version 3 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU Affero General Public License for more details. > +# > +# You should have received a copy of the GNU Affero General Public License > +# along with this program. If not, see <http://www.gnu.org/licenses/>. > + > +use strict qw(vars); > +use Osstest; > +use Osstest::TestSupport; > +use Osstest::BuildSupport; > + > +tsreadconfig(); > +our ($whhost) = @ARGV; > +$whhost ||= 'host'; > +our $ho = selecthost($whhost); > +our $builddir = target_jobdir($ho); > + > +sub tempest() { > + my @ignored_tests; > + my $scenario = 'tempest.scenario'; > + my $volume_boot_pattern = > + "$scenario.test_volume_boot_pattern.TestVolumeBootPattern"; > + my $shelve_instance = "$scenario.test_shelve_instance.TestShelveInstance"; > + > + # Ignore tests which try to boot a guest with /dev/vda as boot device name. > + push @ignored_tests, > + "^\Q$volume_boot_pattern.test_volume_boot_pattern\E"; > + push @ignored_tests, > + "^\Q$volume_boot_pattern.test_create_ebs_image_and_check_boot\E"; > + push @ignored_tests, > + "^\Q$shelve_instance.test_shelve_volume_backed_instance\E"; > + > + # Those tests access a volume through iSCSI. This does not work when both > + # the server and client of iSCSI are on the same Xen host, Linux 4.0 is the > + # first Linux to have a fix. What fix? Would it make sense to point this out? Also I am going to assume the 'server' and 'client' are guests, and 'server' is not dom0? > + push @ignored_tests, > + "^\Q${volume_boot_pattern}V2.test_volume_boot_pattern\E"; > + push @ignored_tests, > + "^\Q${volume_boot_pattern}V2.test_create_ebs_image_and_check_boot\E"; > + > + # This regex below select the tests to run and exclude the ones marked as > + # slow as well as the explicit tests listed above. It is based on the one > + # that can be found in tempest.git/tox.ini in the section [testenv:full]. > + my $ignored_tests = join("|", @ignored_tests); > + my $regex = > + "(?!.*\\[.*\\bslow\\b.*\\]|$ignored_tests)(^tempest\\.(api|scenario|thirdparty))"; > + > + target_cmd($ho, <<END, 7200); > + $builddir/tempest/run_tempest.sh --virtual-env -- --concurrency=2 '$regex' How come you are hardcoding 2? > +END > +} > + > +tempest(); > -- > Anthony PERARD > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > https://lists.xen.org/xen-devel
On Tue, Nov 15, 2016 at 04:42:50PM -0500, Konrad Rzeszutek Wilk wrote: > On Mon, Nov 14, 2016 at 12:33:33PM +0000, Anthony PERARD wrote: > > + # Those tests access a volume through iSCSI. This does not work when both > > + # the server and client of iSCSI are on the same Xen host, Linux 4.0 is the > > + # first Linux to have a fix. > > What fix? Would it make sense to point this out? Also I am going to > assume the 'server' and 'client' are guests, and 'server' is not dom0? It was a patch series, I don't know if I can find it again. I thought pointing to a release was enough. I think iSCSI have other terms for 'server' and 'client', maybe 'target' and something else. But those two sides are running in dom0, in this test. > > + push @ignored_tests, > > + "^\Q${volume_boot_pattern}V2.test_volume_boot_pattern\E"; > > + push @ignored_tests, > > + "^\Q${volume_boot_pattern}V2.test_create_ebs_image_and_check_boot\E"; > > + > > + # This regex below select the tests to run and exclude the ones marked as > > + # slow as well as the explicit tests listed above. It is based on the one > > + # that can be found in tempest.git/tox.ini in the section [testenv:full]. > > + my $ignored_tests = join("|", @ignored_tests); > > + my $regex = > > + "(?!.*\\[.*\\bslow\\b.*\\]|$ignored_tests)(^tempest\\.(api|scenario|thirdparty))"; > > + > > + target_cmd($ho, <<END, 7200); > > + $builddir/tempest/run_tempest.sh --virtual-env -- --concurrency=2 '$regex' > > > How come you are hardcoding 2? By default, this number is equal to the number of cpus. I think it would be too much, and increase the failure rate of openstack it self, which I don't think it would be usefull to test. For their CI loop, I think this is equal to half the number of cpus. We could do that but I don't know if it's going to be usefull. We do use 2 on the XenProject OpenStack CI loop, because I don't think it can do more than that.
diff --git a/sg-run-job b/sg-run-job index 7b9296c..f086e44 100755 --- a/sg-run-job +++ b/sg-run-job @@ -477,6 +477,7 @@ proc run-job/test-rumprun {} { proc need-hosts/test-devstack {} { return host } proc run-job/test-devstack {} { run-ts . = ts-openstack-deploy host + run-ts . = ts-openstack-tempest host } if {[file exists sg-run-job-adhoc]} { diff --git a/ts-openstack-tempest b/ts-openstack-tempest new file mode 100755 index 0000000..8cc993a --- /dev/null +++ b/ts-openstack-tempest @@ -0,0 +1,64 @@ +#!/usr/bin/perl +# This is part of "osstest", an automated testing framework for Xen. +# Copyright (C) 2015 Citrix Inc. +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +use strict qw(vars); +use Osstest; +use Osstest::TestSupport; +use Osstest::BuildSupport; + +tsreadconfig(); +our ($whhost) = @ARGV; +$whhost ||= 'host'; +our $ho = selecthost($whhost); +our $builddir = target_jobdir($ho); + +sub tempest() { + my @ignored_tests; + my $scenario = 'tempest.scenario'; + my $volume_boot_pattern = + "$scenario.test_volume_boot_pattern.TestVolumeBootPattern"; + my $shelve_instance = "$scenario.test_shelve_instance.TestShelveInstance"; + + # Ignore tests which try to boot a guest with /dev/vda as boot device name. + push @ignored_tests, + "^\Q$volume_boot_pattern.test_volume_boot_pattern\E"; + push @ignored_tests, + "^\Q$volume_boot_pattern.test_create_ebs_image_and_check_boot\E"; + push @ignored_tests, + "^\Q$shelve_instance.test_shelve_volume_backed_instance\E"; + + # Those tests access a volume through iSCSI. This does not work when both + # the server and client of iSCSI are on the same Xen host, Linux 4.0 is the + # first Linux to have a fix. + push @ignored_tests, + "^\Q${volume_boot_pattern}V2.test_volume_boot_pattern\E"; + push @ignored_tests, + "^\Q${volume_boot_pattern}V2.test_create_ebs_image_and_check_boot\E"; + + # This regex below select the tests to run and exclude the ones marked as + # slow as well as the explicit tests listed above. It is based on the one + # that can be found in tempest.git/tox.ini in the section [testenv:full]. + my $ignored_tests = join("|", @ignored_tests); + my $regex = + "(?!.*\\[.*\\bslow\\b.*\\]|$ignored_tests)(^tempest\\.(api|scenario|thirdparty))"; + + target_cmd($ho, <<END, 7200); + $builddir/tempest/run_tempest.sh --virtual-env -- --concurrency=2 '$regex' +END +} + +tempest();