diff mbox series

[v3,09/20] Boot Linux Console Test: update the x86_64 kernel

Message ID 20190221005753.27955-10-crosa@redhat.com (mailing list archive)
State New, archived
Headers show
Series Acceptance Tests: target architecture support | expand

Commit Message

Cleber Rosa Feb. 21, 2019, 12:57 a.m. UTC
To the stock Fedora 29 kernel, from the Fedora 28.  New tests will be
added using the 29 kernel, so for consistency, let's also update it
here.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comments

Cornelia Huck March 1, 2019, 10:41 a.m. UTC | #1
On Wed, 20 Feb 2019 19:57:42 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> To the stock Fedora 29 kernel, from the Fedora 28.  New tests will be
> added using the 29 kernel, so for consistency, let's also update it
> here.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> ---
>  tests/acceptance/boot_linux_console.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 89df7f6e4f..35b31162d4 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -28,9 +28,9 @@ class BootLinuxConsole(Test):
>          :avocado: tags=arch:x86_64
>          :avocado: tags=machine:pc
>          """
> -        kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
> +        kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/'
>                        'Everything/x86_64/os/images/pxeboot/vmlinuz')
> -        kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
> +        kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
>          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>  
>          self.vm.set_machine('pc')

This looks reasonable; but an obvious follow-up question is how we
handle updates to new distro levels in general.

I don't think we'll need to run acceptance tests on old qemus, though.
Philippe Mathieu-Daudé March 2, 2019, 9:42 p.m. UTC | #2
On 2/21/19 1:57 AM, Cleber Rosa wrote:
> To the stock Fedora 29 kernel, from the Fedora 28.  New tests will be

[Update] to ... ?

> added using the 29 kernel, so for consistency, let's also update it
> here.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> ---
>  tests/acceptance/boot_linux_console.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 89df7f6e4f..35b31162d4 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -28,9 +28,9 @@ class BootLinuxConsole(Test):
>          :avocado: tags=arch:x86_64
>          :avocado: tags=machine:pc
>          """
> -        kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
> +        kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/'
>                        'Everything/x86_64/os/images/pxeboot/vmlinuz')
> -        kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
> +        kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
>          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>  
>          self.vm.set_machine('pc')
>
Cleber Rosa March 11, 2019, 11:58 p.m. UTC | #3
On Fri, Mar 01, 2019 at 11:41:44AM +0100, Cornelia Huck wrote:
> On Wed, 20 Feb 2019 19:57:42 -0500
> Cleber Rosa <crosa@redhat.com> wrote:
> 
> > To the stock Fedora 29 kernel, from the Fedora 28.  New tests will be
> > added using the 29 kernel, so for consistency, let's also update it
> > here.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> > ---
> >  tests/acceptance/boot_linux_console.py | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> > index 89df7f6e4f..35b31162d4 100644
> > --- a/tests/acceptance/boot_linux_console.py
> > +++ b/tests/acceptance/boot_linux_console.py
> > @@ -28,9 +28,9 @@ class BootLinuxConsole(Test):
> >          :avocado: tags=arch:x86_64
> >          :avocado: tags=machine:pc
> >          """
> > -        kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
> > +        kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/'
> >                        'Everything/x86_64/os/images/pxeboot/vmlinuz')
> > -        kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
> > +        kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
> >          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >  
> >          self.vm.set_machine('pc')
> 
> This looks reasonable; but an obvious follow-up question is how we
> handle updates to new distro levels in general.
> 
> I don't think we'll need to run acceptance tests on old qemus, though.

No, I don't think we have to bother with that.  Backport(er)s would be
expected to handle this (if they ever come to existence).

You do have a point about consistenly updating those images, though.
I won't pretend to have an answer for that just yet... but it's
possible that these "kernel_urls" can be turned into parameters
that will eventually be expand based on the user's selected distro
and version (similar to the Avocado vmimage library[1]).

- Cleber.

[1] https://avocado-framework.readthedocs.io/en/69.0/api/utils/avocado.utils.html#avocado.utils.vmimage.get
Cleber Rosa March 12, 2019, midnight UTC | #4
On Sat, Mar 02, 2019 at 10:42:13PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/21/19 1:57 AM, Cleber Rosa wrote:
> > To the stock Fedora 29 kernel, from the Fedora 28.  New tests will be
> 
> [Update] to ... ?
>

Right... I used this line as a continuation of the commit title,
but probably shouldn't have done that.

> > added using the 29 kernel, so for consistency, let's also update it
> > here.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> 
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>

Thanks!
- Cleber.

> > ---
> >  tests/acceptance/boot_linux_console.py | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> > index 89df7f6e4f..35b31162d4 100644
> > --- a/tests/acceptance/boot_linux_console.py
> > +++ b/tests/acceptance/boot_linux_console.py
> > @@ -28,9 +28,9 @@ class BootLinuxConsole(Test):
> >          :avocado: tags=arch:x86_64
> >          :avocado: tags=machine:pc
> >          """
> > -        kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
> > +        kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/'
> >                        'Everything/x86_64/os/images/pxeboot/vmlinuz')
> > -        kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
> > +        kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
> >          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >  
> >          self.vm.set_machine('pc')
> >
Gerd Hoffmann March 12, 2019, 9:35 a.m. UTC | #5
Hi,

> possible that these "kernel_urls" can be turned into parameters
> that will eventually be expand based on the user's selected distro
> and version (similar to the Avocado vmimage library[1]).

Speaking of distros ...

I'd like automate some of the graphics testing.  For starters boot
guest, run test utility to display something, take a screendump, check
whenever it actually shows what I expect it to show.  Maybe later run
test suites like https://cgit.freedesktop.org/drm/igt-gpu-tools/.

Ideally it would allow to not only test qemu, but to also test the
kernel drivers (i.e. allows to boot and test with some custom
development kernel).

For that I need a bit more than just a kernel.  Suggestions how to do
that best with autotest?

Fetch distro image, install/compile what I need, cache the image
somehow, then use it to run the tests?  Would the vmimage library allow
to do that?  From a brief look it seems to only support fetching the
images, not customizing them ...

Create an appliance using supermin?

Create an special initrd using dracut?

Something else?

thanks,
  Gerd
Cleber Rosa March 12, 2019, 6:29 p.m. UTC | #6
On Tue, Mar 12, 2019 at 10:35:15AM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > possible that these "kernel_urls" can be turned into parameters
> > that will eventually be expand based on the user's selected distro
> > and version (similar to the Avocado vmimage library[1]).
> 
> Speaking of distros ...
> 
> I'd like automate some of the graphics testing.  For starters boot
> guest, run test utility to display something, take a screendump, check
> whenever it actually shows what I expect it to show.  Maybe later run
> test suites like https://cgit.freedesktop.org/drm/igt-gpu-tools/.
> 
> Ideally it would allow to not only test qemu, but to also test the
> kernel drivers (i.e. allows to boot and test with some custom
> development kernel).
> 
> For that I need a bit more than just a kernel.  Suggestions how to do
> that best with autotest?
> 
> Fetch distro image, install/compile what I need, cache the image
> somehow, then use it to run the tests?  Would the vmimage library allow
> to do that?  From a brief look it seems to only support fetching the
> images, not customizing them ...
>

Do you feel like most of the customization would be done constantly?
I'm trying to grasp if it would make sense to have an "almost ready"
image, and adding a final level of customization (adding freshly built
drivers?) on top of that.

If that sounds like a possibility, than it should be possible to host
your images somewhere[1] and let the vmimage library get and cache them
(you can expand the list of providers).

Then, when it comes to the customization part, do you expect your
running environments to have a "reasonable" level of high level tools
such as SSH and Python?  The reason I ask is, as much as it sounds
like overkill, tools like ansible (that depend on both of these in the
managed system) can take a considerable amount of complexity out of
test.  In my experience it's only when tools like ansible are used as
programming languages that everything starts to crumble and fall apart.

In the context of "acceptance" tests, an "ansible" dependency can be
expressed as a requirement on "tests/requirements.txt" so the
bootstrap should not be a problem.  Then, I'd really like to add some
level of integration between the:

 * the avocado.utils.vmimage library
 * the avocado.utils.cloudinit library
 * the avocado.utils.ssh library
 * the QEMUMachine instances we spawn
 * running an ansible playbook on the previous

Brainstorming about it:

   def setUp(self):
       with self.get_vm() as vm:
          vm.add_drive(utils.vmimage.get("mydistro"))
          vm.add_drive(utils.cloudinit.iso())
          vm.start()
          utils.ansible.run(vm, "graphic_drivers.yml")
          # failure to get here can CANCEL the test

   def test(self):
       with utils.ssh.SSHSession(self.get_vm("default_vm")) as ssh:
           ...

> Create an appliance using supermin?
> 
> Create an special initrd using dracut?
>

If your systems are not expected to contain higher level tools, than,
I'm afraid those are good options indeed.

> Something else?
> 
> thanks,
>   Gerd
>

PS: I have strong feelings about what a test should do, and that should not
include any kind of setup (including downloading files it'll use).  This
is somehow loosely expressed in work I intend to do at a given point[2].

[1] - There's lot of unused disk space and bandwidth available in the
      "avocado-project.org" server.

[2] - https://trello.com/c/WPd4FrIy/1479-add-support-to-specify-assets-in-test-docstring
Gerd Hoffmann March 13, 2019, 7:25 a.m. UTC | #7
Hi,

> > I'd like automate some of the graphics testing.  For starters boot
> > guest, run test utility to display something, take a screendump, check
> > whenever it actually shows what I expect it to show.  Maybe later run
> > test suites like https://cgit.freedesktop.org/drm/igt-gpu-tools/.
> > 
> > Ideally it would allow to not only test qemu, but to also test the
> > kernel drivers (i.e. allows to boot and test with some custom
> > development kernel).
> > 
> > For that I need a bit more than just a kernel.  Suggestions how to do
> > that best with autotest?
> > 
> > Fetch distro image, install/compile what I need, cache the image
> > somehow, then use it to run the tests?  Would the vmimage library allow
> > to do that?  From a brief look it seems to only support fetching the
> > images, not customizing them ...
> 
> Do you feel like most of the customization would be done constantly?

For qemu testing no.
For linux kernel testing yes.

> I'm trying to grasp if it would make sense to have an "almost ready"
> image, and adding a final level of customization (adding freshly built
> drivers?) on top of that.
> 
> If that sounds like a possibility, than it should be possible to host
> your images somewhere[1] and let the vmimage library get and cache them
> (you can expand the list of providers).

Hmm.

Put kernel testing aside for now.  I don't think it is that an good idea
to host a bunch of custom images just because some test tools are needed
for some test.  The cloud images are typically pretty stripped down, so
when using them for testing the need to install additional is probably
not rare.  Be it my graphics test tools, or usbutils for usb testing, or
something else network/block testing.

I think it would make sense to support creating and caching such custom
images, i.e. download standard fedora image, install test tools (fetch
rpms or compile stuff), then put that image into a local cache and use
it for test runs.

> Then, when it comes to the customization part, do you expect your
> running environments to have a "reasonable" level of high level tools
> such as SSH and Python?

Wouldn't hurt, but isn't required.  At least initially.

When maybe running a whole gfx test suite inside the guest some day
it'll be a different story.  That'll probably out of scope for
acceptance tests, but using avocado might still be useful for that.

> PS: I have strong feelings about what a test should do, and that should not
> include any kind of setup (including downloading files it'll use).  This
> is somehow loosely expressed in work I intend to do at a given point[2].

Yep.  Setup and test run should be separate, so you setup once and don't
fetch stuff from the Internet for each test run.

cheers,
  Gerd
diff mbox series

Patch

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 89df7f6e4f..35b31162d4 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -28,9 +28,9 @@  class BootLinuxConsole(Test):
         :avocado: tags=arch:x86_64
         :avocado: tags=machine:pc
         """
-        kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
+        kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/'
                       'Everything/x86_64/os/images/pxeboot/vmlinuz')
-        kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
+        kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
 
         self.vm.set_machine('pc')