| Message ID | 20190520124716.30472-1-kraxel@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | tests/vm: serial console autoinstall, misc fixes. | expand |
On Mon, May 20, 2019 at 02:47:02PM +0200, Gerd Hoffmann wrote: > This patch series changes the way virtual machines for test builds are > managed. They are created locally on the developer machine now. The > installer is booted on the serial console and the scripts walks through > the dialogs to install and configure the guest. > > That takes the download.patchew.org server out of the loop and makes it > alot easier to tweak the guest images (adding build dependencies for > example). > > The install scripts take care to apply host proxy settings (from *_proxy > environment variables) to the guest, so any package downloads will be > routed through the proxy and can be cached that way. This also makes > them work behind strict firewalls. > > There are also a bunch of smaller tweaks for tests/vm to fix issues I > was struggling with. See commit messages of individual patches for > details. Ping. Alex? Fam? Are you going to pick this up? Or should I send a pull request? cheers, Gerd
Gerd Hoffmann <kraxel@redhat.com> writes: > On Mon, May 20, 2019 at 02:47:02PM +0200, Gerd Hoffmann wrote: >> This patch series changes the way virtual machines for test builds are >> managed. They are created locally on the developer machine now. The >> installer is booted on the serial console and the scripts walks through >> the dialogs to install and configure the guest. >> >> That takes the download.patchew.org server out of the loop and makes it >> alot easier to tweak the guest images (adding build dependencies for >> example). >> >> The install scripts take care to apply host proxy settings (from *_proxy >> environment variables) to the guest, so any package downloads will be >> routed through the proxy and can be cached that way. This also makes >> them work behind strict firewalls. >> >> There are also a bunch of smaller tweaks for tests/vm to fix issues I >> was struggling with. See commit messages of individual patches for >> details. > > Ping. Alex? Fam? Are you going to pick this up? Or should I send a > pull request? I'm just looking at it now. It should be in the next testing/next. I'm just checking to see if there is any crossover with: Subject: [PATCH 0/5] tests/vm: Python 3, improve image caching, and misc Date: Fri, 29 Mar 2019 17:07:59 -0400 Message-Id: <20190329210804.22121-1-wainersm@redhat.com> -- Alex Bennée
Gerd Hoffmann <kraxel@redhat.com> writes: > This patch series changes the way virtual machines for test builds are > managed. They are created locally on the developer machine now. The > installer is booted on the serial console and the scripts walks through > the dialogs to install and configure the guest. > > That takes the download.patchew.org server out of the loop and makes it > alot easier to tweak the guest images (adding build dependencies for > example). > > The install scripts take care to apply host proxy settings (from *_proxy > environment variables) to the guest, so any package downloads will be > routed through the proxy and can be cached that way. This also makes > them work behind strict firewalls. > > There are also a bunch of smaller tweaks for tests/vm to fix issues I > was struggling with. See commit messages of individual patches for > details. Queued to testing/next, thanks. One of the machines I'm testing on seems to have problems with getting the installer working over the serial link but it works on my main dev box and others have it working as well so I suspect it might be a local problem. > > v3: > - python3 fixes. > - openbsd: configure memory limits. > - freebsd: configure autoboot delay. > > Gerd Hoffmann (14): > scripts: use git archive in archive-source > tests/vm: python3 fixes > tests/vm: send proxy environment variables over ssh > tests/vm: use ssh with pty unconditionally > tests/vm: run test builds on snapshot > tests/vm: proper guest shutdown > tests/vm: add vm-boot-{ssh,serial}-<guest> targets > tests/vm: add DEBUG=1 to help text > tests/vm: serial console support helpers > tests/vm: openbsd autoinstall, using serial console > tests/vm: freebsd autoinstall, using serial console > tests/vm: netbsd autoinstall, using serial console > tests/vm: fedora autoinstall, using serial console > tests/vm: ubuntu.i386: apt proxy setup > > tests/vm/basevm.py | 144 ++++++++++++++++++++++++----- > scripts/archive-source.sh | 72 +++++++-------- > tests/vm/Makefile.include | 25 ++++- > tests/vm/fedora | 187 ++++++++++++++++++++++++++++++++++++++ > tests/vm/freebsd | 179 ++++++++++++++++++++++++++++++++++-- > tests/vm/netbsd | 187 ++++++++++++++++++++++++++++++++++++-- > tests/vm/openbsd | 158 +++++++++++++++++++++++++++++--- > tests/vm/ubuntu.i386 | 4 + > 8 files changed, 858 insertions(+), 98 deletions(-) > create mode 100755 tests/vm/fedora -- Alex Bennée
no-reply@patchew.org writes: > Patchew URL: https://patchew.org/QEMU/20190520124716.30472-1-kraxel@redhat.com/ > > > > Hi, > > This series seems to have some coding style problems. See output below for > more information: > > Subject: [Qemu-devel] [PATCH v3 00/14] tests/vm: serial console autoinstall, misc fixes. > Message-id: 20190520124716.30472-1-kraxel@redhat.com > Type: series > > === TEST SCRIPT BEGIN === > #!/bin/bash > git rev-parse base > /dev/null || exit 0 > git config --local diff.renamelimit 0 > git config --local diff.renames True > git config --local diff.algorithm histogram > ./scripts/checkpatch.pl --mailback base.. > === TEST SCRIPT END === > > From https://github.com/patchew-project/qemu > 1b46b4daa6..6d8e75d41c master -> master > From https://github.com/patchew-project/qemu > * [new tag] patchew/20190520124716.30472-1-kraxel@redhat.com -> patchew/20190520124716.30472-1-kraxel@redhat.com > Switched to a new branch 'test' > 6f7f5d5a99 tests/vm: ubuntu.i386: apt proxy setup > ef70a753bb tests/vm: fedora autoinstall, using serial console > 0debc608fb tests/vm: netbsd autoinstall, using serial console > 5e005a2c07 tests/vm: freebsd autoinstall, using serial console > bff0400f4e tests/vm: openbsd autoinstall, using serial console > 59085882aa tests/vm: serial console support helpers > 27ef52c336 tests/vm: add DEBUG=1 to help text > 128b742fc5 tests/vm: add vm-boot-{ssh, serial}-<guest> targets > 2eb095b079 tests/vm: proper guest shutdown > 4f3a1fed99 tests/vm: run test builds on snapshot > d3850e7e8e tests/vm: use ssh with pty unconditionally > 9d18c76436 tests/vm: send proxy environment variables over ssh > 9bb50d9be2 tests/vm: python3 fixes > 76e6f4270e scripts: use git archive in archive-source > > === OUTPUT BEGIN === > 1/14 Checking commit 76e6f4270ef3 (scripts: use git archive in archive-source) > ERROR: code indent should never use tabs > #41: FILE: scripts/archive-source.sh:36: > +^Igit submodule deinit $sub_deinit$ > > ERROR: code indent should never use tabs > #68: FILE: scripts/archive-source.sh:56: > +^I-*)$ > > ERROR: code indent should never use tabs > #69: FILE: scripts/archive-source.sh:57: > +^I sub_deinit="$sub_deinit $sm"$ > > ERROR: code indent should never use tabs > #70: FILE: scripts/archive-source.sh:58: > +^I git submodule update --init "$sm"$ > > ERROR: code indent should never use tabs > #71: FILE: scripts/archive-source.sh:59: > +^I test $? -ne 0 && error "failed to update submodule $sm"$ > > ERROR: code indent should never use tabs > #72: FILE: scripts/archive-source.sh:60: > +^I ;;$ > > ERROR: code indent should never use tabs > #73: FILE: scripts/archive-source.sh:61: > +^I+*)$ > > ERROR: code indent should never use tabs > #74: FILE: scripts/archive-source.sh:62: > +^I echo "WARNING: submodule $sm is out of sync"$ > > ERROR: code indent should never use tabs > #75: FILE: scripts/archive-source.sh:63: > +^I ;;$ > > total: 9 errors, 0 warnings, 93 lines checked I'll fix these up but it might be worth seeing how that crept into your editor? I'll include an update to editorconfig while I'm at it. -- Alex Bennée
On 5/30/19 11:33 AM, Alex Bennée wrote: > > Gerd Hoffmann <kraxel@redhat.com> writes: > >> This patch series changes the way virtual machines for test builds are >> managed. They are created locally on the developer machine now. The >> installer is booted on the serial console and the scripts walks through >> the dialogs to install and configure the guest. >> >> That takes the download.patchew.org server out of the loop and makes it >> alot easier to tweak the guest images (adding build dependencies for >> example). >> >> The install scripts take care to apply host proxy settings (from *_proxy >> environment variables) to the guest, so any package downloads will be >> routed through the proxy and can be cached that way. This also makes >> them work behind strict firewalls. >> >> There are also a bunch of smaller tweaks for tests/vm to fix issues I >> was struggling with. See commit messages of individual patches for >> details. > > Queued to testing/next, thanks. > > One of the machines I'm testing on seems to have problems with getting > the installer working over the serial link but it works on my main dev > box and others have it working as well so I suspect it might be a local > problem. Is this the same issue I described there? https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg06784.html
Philippe Mathieu-Daudé <philmd@redhat.com> writes: > On 5/30/19 11:33 AM, Alex Bennée wrote: >> >> Gerd Hoffmann <kraxel@redhat.com> writes: >> >>> This patch series changes the way virtual machines for test builds are >>> managed. They are created locally on the developer machine now. The >>> installer is booted on the serial console and the scripts walks through >>> the dialogs to install and configure the guest. >>> >>> That takes the download.patchew.org server out of the loop and makes it >>> alot easier to tweak the guest images (adding build dependencies for >>> example). >>> >>> The install scripts take care to apply host proxy settings (from *_proxy >>> environment variables) to the guest, so any package downloads will be >>> routed through the proxy and can be cached that way. This also makes >>> them work behind strict firewalls. >>> >>> There are also a bunch of smaller tweaks for tests/vm to fix issues I >>> was struggling with. See commit messages of individual patches for >>> details. >> >> Queued to testing/next, thanks. >> >> One of the machines I'm testing on seems to have problems with getting >> the installer working over the serial link but it works on my main dev >> box and others have it working as well so I suspect it might be a local >> problem. > > Is this the same issue I described there? > https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg06784.html Not quite.. on the failing machine I see it hang at: make[1]: Leaving directory '/home/alex.bennee/lsrc/qemu.git/dtc' python3 -B /home/alex.bennee/lsrc/qemu.git/tests/vm/openbsd --debug --image "/home/alex.bennee/.cache/qemu-vm/images/openbsd.img" --force --build-image /home/alex.bennee/.cache/qemu-vm/images/openbsd.img ### Downloading install iso ... ### Preparing iso and disk image ... Formatting '/home/alex.bennee/.cache/qemu-vm/images/openbsd.img.tmp', fmt=qcow2 size=21474836480 cluster_size=65536 lazy_refcounts=off refcount_bits=16 ### Booting installer ... DEBUG:root:QEMU args: -nodefaults -m 4G -cpu max -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 18 -enable-kvm -device VGA -drive file=/home/alex.bennee/.cache/qemu-vm/images/openbsd.img.tmp,if=none,id=drive0,cache=writeback -device virtio-blk,drive=drive0,bootindex=0 -machine graphics=off -cdrom /home/alex.bennee/.cache/qemu-vm/images/openbsd.img.install.iso DEBUG:qemu:VM launch command: 'qemu-system-x86_64 -chardev socket,id=mon,path=/var/tmp/tmp0uvsee9z/qemu-18506-monitor.sock -mon chardev=mon,mode=control -display none -vga none -machine pc -chardev socket,id=console,path=/var/tmp/tmp0uvsee9z/qemu-18506-console.sock,server,nowait -serial chardev:console -nodefaults -m 4G -cpu max -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 18 -enable-kvm -device VGA -drive file=/home/alex.bennee/.cache/qemu-vm/images/openbsd.img.tmp,if=none,id=drive0,cache=writeback -device virtio-blk,drive=drive0,bootindex=0 -machine graphics=off -cdrom /home/alex.bennee/.cache/qemu-vm/images/openbsd.img.install.iso' DEBUG:QMP:>>> {'execute': 'qmp_capabilities'} DEBUG:QMP:<<< {'return': {}} DEBUG:QMP:>>> {'execute': 'human-monitor-command', 'arguments': {'command-line': 'info usernet'}} DEBUG:QMP:<<< {'return': 'VLAN -1 (vnet):\r\n Protocol[State] FD Source Address Port Dest. Address Port RecvQ SendQ\r\n TCP[HOST_FORWARD] 13 127.0.0.1 33465 10.0.2.15 22 0 0\r\n'} console: *** read timeout *** console: waiting for: 'boot>' console: line buffer: Failed to prepare guest environment Traceback (most recent call last): File "/home/alex.bennee/lsrc/qemu.git/tests/vm/basevm.py", line 350, in main return vm.build_image(args.image) File "/home/alex.bennee/lsrc/qemu.git/tests/vm/openbsd", line 85, in build_image self.console_wait_send("boot>", "set tty com0\n") File "/home/alex.bennee/lsrc/qemu.git/tests/vm/basevm.py", line 250, in console_wait_send self.console_wait(wait) File "/home/alex.bennee/lsrc/qemu.git/tests/vm/basevm.py", line 212, in console_wait chars = vm.console_socket.recv(1024) socket.timeout: timed out DEBUG:QMP:>>> {'execute': 'quit'} DEBUG:QMP:<<< {'return': {}} /home/alex.bennee/lsrc/qemu.git/tests/vm/Makefile.include:47: recipe for target '/home/alex.bennee/.cache/qemu-vm/images/openbsd.img' failed make: *** [/home/alex.bennee/.cache/qemu-vm/images/openbsd.img] Error 2 I can see the machine has actually booted as VNC can connect but for some reason the serial isn't working. -- Alex Bennée
Cc'ing the Python team. On 5/30/19 12:18 PM, Alex Bennée wrote: > > Philippe Mathieu-Daudé <philmd@redhat.com> writes: > >> On 5/30/19 11:33 AM, Alex Bennée wrote: >>> >>> Gerd Hoffmann <kraxel@redhat.com> writes: >>> >>>> This patch series changes the way virtual machines for test builds are >>>> managed. They are created locally on the developer machine now. The >>>> installer is booted on the serial console and the scripts walks through >>>> the dialogs to install and configure the guest. >>>> >>>> That takes the download.patchew.org server out of the loop and makes it >>>> alot easier to tweak the guest images (adding build dependencies for >>>> example). >>>> >>>> The install scripts take care to apply host proxy settings (from *_proxy >>>> environment variables) to the guest, so any package downloads will be >>>> routed through the proxy and can be cached that way. This also makes >>>> them work behind strict firewalls. >>>> >>>> There are also a bunch of smaller tweaks for tests/vm to fix issues I >>>> was struggling with. See commit messages of individual patches for >>>> details. >>> >>> Queued to testing/next, thanks. >>> >>> One of the machines I'm testing on seems to have problems with getting >>> the installer working over the serial link but it works on my main dev >>> box and others have it working as well so I suspect it might be a local >>> problem. >> >> Is this the same issue I described there? >> https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg06784.html > > Not quite.. on the failing machine I see it hang at: > > make[1]: Leaving directory '/home/alex.bennee/lsrc/qemu.git/dtc' > python3 -B /home/alex.bennee/lsrc/qemu.git/tests/vm/openbsd --debug --image "/home/alex.bennee/.cache/qemu-vm/images/openbsd.img" --force --build-image /home/alex.bennee/.cache/qemu-vm/images/openbsd.img > ### Downloading install iso ... > ### Preparing iso and disk image ... > Formatting '/home/alex.bennee/.cache/qemu-vm/images/openbsd.img.tmp', fmt=qcow2 size=21474836480 cluster_size=65536 lazy_refcounts=off refcount_bits=16 > ### Booting installer ... > DEBUG:root:QEMU args: -nodefaults -m 4G -cpu max -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 18 -enable-kvm > -device VGA -drive file=/home/alex.bennee/.cache/qemu-vm/images/openbsd.img.tmp,if=none,id=drive0,cache=writeback -device virtio-blk,drive=drive0,bootindex=0 -machine graphics=off -cdrom /home/alex.bennee/.cache/qemu-vm/images/openbsd.img.install.iso > DEBUG:qemu:VM launch command: 'qemu-system-x86_64 -chardev socket,id=mon,path=/var/tmp/tmp0uvsee9z/qemu-18506-monitor.sock -mon chardev=mon,mode=control -display none -vga none -machine pc -chardev socket,id=console,path=/var/tmp/tmp0uvsee9z/qemu-18506-console.sock,server,nowait -serial chardev:console -nodefaults -m 4G -cpu max -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 18 -enable-kvm -device VGA -drive file=/home/alex.bennee/.cache/qemu-vm/images/openbsd.img.tmp,if=none,id=drive0,cache=writeback -device virtio-blk,drive=drive0,bootindex=0 -machine graphics=off -cdrom /home/alex.bennee/.cache/qemu-vm/images/openbsd.img.install.iso' > DEBUG:QMP:>>> {'execute': 'qmp_capabilities'} > DEBUG:QMP:<<< {'return': {}} > DEBUG:QMP:>>> {'execute': 'human-monitor-command', 'arguments': {'command-line': 'info usernet'}} > DEBUG:QMP:<<< {'return': 'VLAN -1 (vnet):\r\n Protocol[State] FD Source Address Port Dest. Address Port RecvQ SendQ\r\n TCP[HOST_FORWARD] 13 127.0.0.1 33465 10.0.2.15 22 0 0\r\n'} > console: *** read timeout *** > console: waiting for: 'boot>' > console: line buffer: > > > Failed to prepare guest environment > Traceback (most recent call last): > File "/home/alex.bennee/lsrc/qemu.git/tests/vm/basevm.py", line 350, in main > return vm.build_image(args.image) > File "/home/alex.bennee/lsrc/qemu.git/tests/vm/openbsd", line 85, in build_image > self.console_wait_send("boot>", "set tty com0\n") > File "/home/alex.bennee/lsrc/qemu.git/tests/vm/basevm.py", line 250, in console_wait_send > self.console_wait(wait) > File "/home/alex.bennee/lsrc/qemu.git/tests/vm/basevm.py", line 212, in console_wait > chars = vm.console_socket.recv(1024) > socket.timeout: timed out > DEBUG:QMP:>>> {'execute': 'quit'} > DEBUG:QMP:<<< {'return': {}} > /home/alex.bennee/lsrc/qemu.git/tests/vm/Makefile.include:47: recipe for target '/home/alex.bennee/.cache/qemu-vm/images/openbsd.img' failed > make: *** [/home/alex.bennee/.cache/qemu-vm/images/openbsd.img] Error 2 > > I can see the machine has actually booted as VNC can connect but for > some reason the serial isn't working. Looks related IMO :) Some desync with the serial. Maybe some Python detail we are missing (like an Exception we are silently ignoring). BTW I can reproduce my case, and it looks you can reproduce yours too.
Cc'ing Paolo & Marc-André (and removing the BSD folks). On 5/30/19 12:30 PM, Philippe Mathieu-Daudé wrote: > Cc'ing the Python team. > > On 5/30/19 12:18 PM, Alex Bennée wrote: >> >> Philippe Mathieu-Daudé <philmd@redhat.com> writes: >> >>> On 5/30/19 11:33 AM, Alex Bennée wrote: >>>> >>>> Gerd Hoffmann <kraxel@redhat.com> writes: >>>> >>>>> This patch series changes the way virtual machines for test builds are >>>>> managed. They are created locally on the developer machine now. The >>>>> installer is booted on the serial console and the scripts walks through >>>>> the dialogs to install and configure the guest. >>>>> >>>>> That takes the download.patchew.org server out of the loop and makes it >>>>> alot easier to tweak the guest images (adding build dependencies for >>>>> example). >>>>> >>>>> The install scripts take care to apply host proxy settings (from *_proxy >>>>> environment variables) to the guest, so any package downloads will be >>>>> routed through the proxy and can be cached that way. This also makes >>>>> them work behind strict firewalls. >>>>> >>>>> There are also a bunch of smaller tweaks for tests/vm to fix issues I >>>>> was struggling with. See commit messages of individual patches for >>>>> details. >>>> >>>> Queued to testing/next, thanks. >>>> >>>> One of the machines I'm testing on seems to have problems with getting >>>> the installer working over the serial link but it works on my main dev >>>> box and others have it working as well so I suspect it might be a local >>>> problem. >>> >>> Is this the same issue I described there? >>> https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg06784.html >> >> Not quite.. on the failing machine I see it hang at: >> >> make[1]: Leaving directory '/home/alex.bennee/lsrc/qemu.git/dtc' >> python3 -B /home/alex.bennee/lsrc/qemu.git/tests/vm/openbsd --debug --image "/home/alex.bennee/.cache/qemu-vm/images/openbsd.img" --force --build-image /home/alex.bennee/.cache/qemu-vm/images/openbsd.img >> ### Downloading install iso ... >> ### Preparing iso and disk image ... >> Formatting '/home/alex.bennee/.cache/qemu-vm/images/openbsd.img.tmp', fmt=qcow2 size=21474836480 cluster_size=65536 lazy_refcounts=off refcount_bits=16 >> ### Booting installer ... >> DEBUG:root:QEMU args: -nodefaults -m 4G -cpu max -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 18 -enable-kvm >> -device VGA -drive file=/home/alex.bennee/.cache/qemu-vm/images/openbsd.img.tmp,if=none,id=drive0,cache=writeback -device virtio-blk,drive=drive0,bootindex=0 -machine graphics=off -cdrom /home/alex.bennee/.cache/qemu-vm/images/openbsd.img.install.iso >> DEBUG:qemu:VM launch command: 'qemu-system-x86_64 -chardev socket,id=mon,path=/var/tmp/tmp0uvsee9z/qemu-18506-monitor.sock -mon chardev=mon,mode=control -display none -vga none -machine pc -chardev socket,id=console,path=/var/tmp/tmp0uvsee9z/qemu-18506-console.sock,server,nowait -serial chardev:console -nodefaults -m 4G -cpu max -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 18 -enable-kvm -device VGA -drive file=/home/alex.bennee/.cache/qemu-vm/images/openbsd.img.tmp,if=none,id=drive0,cache=writeback -device virtio-blk,drive=drive0,bootindex=0 -machine graphics=off -cdrom /home/alex.bennee/.cache/qemu-vm/images/openbsd.img.install.iso' >> DEBUG:QMP:>>> {'execute': 'qmp_capabilities'} >> DEBUG:QMP:<<< {'return': {}} >> DEBUG:QMP:>>> {'execute': 'human-monitor-command', 'arguments': {'command-line': 'info usernet'}} >> DEBUG:QMP:<<< {'return': 'VLAN -1 (vnet):\r\n Protocol[State] FD Source Address Port Dest. Address Port RecvQ SendQ\r\n TCP[HOST_FORWARD] 13 127.0.0.1 33465 10.0.2.15 22 0 0\r\n'} >> console: *** read timeout *** >> console: waiting for: 'boot>' >> console: line buffer: >> >> >> Failed to prepare guest environment >> Traceback (most recent call last): >> File "/home/alex.bennee/lsrc/qemu.git/tests/vm/basevm.py", line 350, in main >> return vm.build_image(args.image) >> File "/home/alex.bennee/lsrc/qemu.git/tests/vm/openbsd", line 85, in build_image >> self.console_wait_send("boot>", "set tty com0\n") >> File "/home/alex.bennee/lsrc/qemu.git/tests/vm/basevm.py", line 250, in console_wait_send >> self.console_wait(wait) >> File "/home/alex.bennee/lsrc/qemu.git/tests/vm/basevm.py", line 212, in console_wait >> chars = vm.console_socket.recv(1024) >> socket.timeout: timed out >> DEBUG:QMP:>>> {'execute': 'quit'} >> DEBUG:QMP:<<< {'return': {}} >> /home/alex.bennee/lsrc/qemu.git/tests/vm/Makefile.include:47: recipe for target '/home/alex.bennee/.cache/qemu-vm/images/openbsd.img' failed >> make: *** [/home/alex.bennee/.cache/qemu-vm/images/openbsd.img] Error 2 >> >> I can see the machine has actually booted as VNC can connect but for >> some reason the serial isn't working. > > Looks related IMO :) Some desync with the serial. Maybe some Python > detail we are missing (like an Exception we are silently ignoring). > > BTW I can reproduce my case, and it looks you can reproduce yours too. I now see a similar issue with Avocado. Using this patch https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg05747.html I get: $ make check-venv $ ./tests/venv/bin/python -m avocado --show=console run -t arch:rx tests/acceptance/boot_linux_console.py console: Linux version 4.19.0+ (yo-satoh@yo-satoh-debian) (gcc version 9.0.0 20181105 (experimental) (GCC)) #137 Wed Feb 20 23:20:02 JST 2019 console: Built 1 zonelists, mobility grouping on. Total pages: 8128 console: Kernel command line: console: Dentry cache hash table entries: 4096 (order: 2, 16384 bytes) console: Inode-cache hash table entries: 2048 (order: 1, 8192 bytes) console: Memory: 14648K/32768K available (871K kernel code, 95K rwdata, 140K rodata, 96K init, 175K bss, 18120K reserved, 0K cma-reserved) console: NR_IRQS: 256 console: rx-cmt: used for periodic clock events console: clocksource: rx-tpu: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1274173631191 ns console: 96.00 BogoMIPS (lpj=480000) console: pid_max: default: 4096 minimum: 301 console: Mount-cache hash table entries: 1024 (order: 0, 4096 bytes) console: Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes) console: clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns console: clocksource: Switched to clocksource rx-tpu console: workingset: timestamp_bits=30 max_order=12 bucket_order=0 console: SuperH (H)SCI(F) driver initialized console: 88240.serial: ttySC0 at MMIO 0x88240 (irq = 215, base_baud = 0) is a sci console: console [ttySC0] enabled console: 88248.serial: ttySC1 at MMIO 0x88248 (irq = 219, base_baud = 0) is a sci console: random: get_random_bytes called from 0x01002e48 with crng_init=0 console: Freeing unused kernel memory: 96K console: This architecture does not have kernel memory protection. console: Run /sbin/init as init process console: Run /etc/init as init process console: Run /bin/init as init process console: Run /bin/sh as init process console: console: Sash command shell (version 1.1.1) console: />sh-sci 88240.serial: overrun error console: sh-sci 88240.serial: frame error console: sh-sci 88240.serial: parity error console: prrandom: fast init done [HANG] Most of the time this test succeed. console: This architecture does not have kernel memory protection. console: Run /sbin/init as init process console: Run /etc/init as init process console: Run /bin/init as init process console: Run /bin/sh as init process console: console: Sash command shell (version 1.1.1) console: /> printenv console: HOME=/ console: TERM=linux $ echo $? 0
Alex Bennée <alex.bennee@linaro.org> writes: > Gerd Hoffmann <kraxel@redhat.com> writes: > >> This patch series changes the way virtual machines for test builds are >> managed. They are created locally on the developer machine now. The >> installer is booted on the serial console and the scripts walks through >> the dialogs to install and configure the guest. >> >> That takes the download.patchew.org server out of the loop and makes it >> alot easier to tweak the guest images (adding build dependencies for >> example). >> >> The install scripts take care to apply host proxy settings (from *_proxy >> environment variables) to the guest, so any package downloads will be >> routed through the proxy and can be cached that way. This also makes >> them work behind strict firewalls. >> >> There are also a bunch of smaller tweaks for tests/vm to fix issues I >> was struggling with. See commit messages of individual patches for >> details. > > Queued to testing/next, thanks. > > One of the machines I'm testing on seems to have problems with getting > the installer working over the serial link but it works on my main dev > box and others have it working as well so I suspect it might be a local > problem. <snip> OK I had to drop this series due to persistent problems with the serial link on one of the boxes Peter uses to do his merge testing. Some of it seems to be QEMU specific (Ubuntu 18.04 system qemu just hangs) but I suspect there is also an issue with BSDs and our serial emulation generally. -- Alex Bennée
On 12.06.2019 19:35, Alex Bennée wrote: > > Alex Bennée <alex.bennee@linaro.org> writes: > >> Gerd Hoffmann <kraxel@redhat.com> writes: >> >>> This patch series changes the way virtual machines for test builds are >>> managed. They are created locally on the developer machine now. The >>> installer is booted on the serial console and the scripts walks through >>> the dialogs to install and configure the guest. >>> >>> That takes the download.patchew.org server out of the loop and makes it >>> alot easier to tweak the guest images (adding build dependencies for >>> example). >>> >>> The install scripts take care to apply host proxy settings (from *_proxy >>> environment variables) to the guest, so any package downloads will be >>> routed through the proxy and can be cached that way. This also makes >>> them work behind strict firewalls. >>> >>> There are also a bunch of smaller tweaks for tests/vm to fix issues I >>> was struggling with. See commit messages of individual patches for >>> details. >> >> Queued to testing/next, thanks. >> >> One of the machines I'm testing on seems to have problems with getting >> the installer working over the serial link but it works on my main dev >> box and others have it working as well so I suspect it might be a local >> problem. > <snip> > > OK I had to drop this series due to persistent problems with the serial > link on one of the boxes Peter uses to do his merge testing. Some of it > seems to be QEMU specific (Ubuntu 18.04 system qemu just hangs) but I > suspect there is also an issue with BSDs and our serial emulation > generally. > > -- > Alex Bennée > What version of NetBSD did you test? There was recently a short lived regression in curses(3) that affected sysinst(8) (installer) in HEAD, but it was fixed.
Kamil Rytarowski <n54@gmx.com> writes: > On 12.06.2019 19:35, Alex Bennée wrote: >> >> Alex Bennée <alex.bennee@linaro.org> writes: >> >>> Gerd Hoffmann <kraxel@redhat.com> writes: >>> >>>> This patch series changes the way virtual machines for test builds are >>>> managed. They are created locally on the developer machine now. The >>>> installer is booted on the serial console and the scripts walks through >>>> the dialogs to install and configure the guest. >>>> >>>> That takes the download.patchew.org server out of the loop and makes it >>>> alot easier to tweak the guest images (adding build dependencies for >>>> example). >>>> >>>> The install scripts take care to apply host proxy settings (from *_proxy >>>> environment variables) to the guest, so any package downloads will be >>>> routed through the proxy and can be cached that way. This also makes >>>> them work behind strict firewalls. >>>> >>>> There are also a bunch of smaller tweaks for tests/vm to fix issues I >>>> was struggling with. See commit messages of individual patches for >>>> details. >>> >>> Queued to testing/next, thanks. >>> >>> One of the machines I'm testing on seems to have problems with getting >>> the installer working over the serial link but it works on my main dev >>> box and others have it working as well so I suspect it might be a local >>> problem. >> <snip> >> >> OK I had to drop this series due to persistent problems with the serial >> link on one of the boxes Peter uses to do his merge testing. Some of it >> seems to be QEMU specific (Ubuntu 18.04 system qemu just hangs) but I >> suspect there is also an issue with BSDs and our serial emulation >> generally. >> >> -- >> Alex Bennée >> > > What version of NetBSD did you test? There was recently a short lived > regression in curses(3) that affected sysinst(8) (installer) in HEAD, > but it was fixed. The error for NetBSD was navigating the installer which was dumping a lot of q's in the stream (unicode bar characters?). We were starting with: link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.0/images/NetBSD-8.0-amd64.iso" -- Alex Bennée
> > What version of NetBSD did you test? There was recently a short lived > > regression in curses(3) that affected sysinst(8) (installer) in HEAD, > > but it was fixed. > > The error for NetBSD was navigating the installer which was dumping a > lot of q's in the stream (unicode bar characters?). Not unicode, but yes these are bars for boxes. The console trace you get printed to stdout with V=1 or DEBUG=1 has terminal sequences and non-printable characters stripped off, so the switch to the box drawing charset doesn't happen and you get those q's instead. cheers, Gerd PS: /me is busy installing ubuntu 18.04 guest to have a look ...
Gerd Hoffmann <kraxel@redhat.com> writes: >> > What version of NetBSD did you test? There was recently a short lived >> > regression in curses(3) that affected sysinst(8) (installer) in HEAD, >> > but it was fixed. >> >> The error for NetBSD was navigating the installer which was dumping a >> lot of q's in the stream (unicode bar characters?). > > Not unicode, but yes these are bars for boxes. > > The console trace you get printed to stdout with V=1 or DEBUG=1 has > terminal sequences and non-printable characters stripped off, so the > switch to the box drawing charset doesn't happen and you get those > q's instead. > > cheers, > Gerd > > PS: /me is busy installing ubuntu 18.04 guest to have a look ... Thanks. If you do a re-spin then could you consider including: Subject: [PATCH] tests/vm: favour the locally built QEMU for bootstrapping Date: Fri, 7 Jun 2019 19:53:37 +0100 Message-Id: <20190607185337.14524-1-alex.bennee@linaro.org> Also your scripts and python3 fixes are already merged. -- Alex Bennée
Hi, > If you do a re-spin then could you consider including: > > Subject: [PATCH] tests/vm: favour the locally built QEMU for bootstrapping > Date: Fri, 7 Jun 2019 19:53:37 +0100 > Message-Id: <20190607185337.14524-1-alex.bennee@linaro.org> Local seabios is enough, and that'll more reliable work better because we don't have to compile something for that. ubuntu 18.04 seabios lacks serial console support, which is one of the problems found so far. > Also your scripts and python3 fixes are already merged. Yep, saw that. cheers, Gerd
Gerd Hoffmann <kraxel@redhat.com> writes: > Hi, > >> If you do a re-spin then could you consider including: >> >> Subject: [PATCH] tests/vm: favour the locally built QEMU for bootstrapping >> Date: Fri, 7 Jun 2019 19:53:37 +0100 >> Message-Id: <20190607185337.14524-1-alex.bennee@linaro.org> > > Local seabios is enough, and that'll more reliable work better because > we don't have to compile something for that. ubuntu 18.04 seabios lacks > serial console support, which is one of the problems found so far. Ahh that makes sense. I'm never quite sure where qemu looks for these blobs - they seem to me automagically chosen. > >> Also your scripts and python3 fixes are already merged. > > Yep, saw that. > > cheers, > Gerd -- Alex Bennée
This patch series changes the way virtual machines for test builds are managed. They are created locally on the developer machine now. The installer is booted on the serial console and the scripts walks through the dialogs to install and configure the guest. That takes the download.patchew.org server out of the loop and makes it alot easier to tweak the guest images (adding build dependencies for example). The install scripts take care to apply host proxy settings (from *_proxy environment variables) to the guest, so any package downloads will be routed through the proxy and can be cached that way. This also makes them work behind strict firewalls. There are also a bunch of smaller tweaks for tests/vm to fix issues I was struggling with. See commit messages of individual patches for details. v3: - python3 fixes. - openbsd: configure memory limits. - freebsd: configure autoboot delay. Gerd Hoffmann (14): scripts: use git archive in archive-source tests/vm: python3 fixes tests/vm: send proxy environment variables over ssh tests/vm: use ssh with pty unconditionally tests/vm: run test builds on snapshot tests/vm: proper guest shutdown tests/vm: add vm-boot-{ssh,serial}-<guest> targets tests/vm: add DEBUG=1 to help text tests/vm: serial console support helpers tests/vm: openbsd autoinstall, using serial console tests/vm: freebsd autoinstall, using serial console tests/vm: netbsd autoinstall, using serial console tests/vm: fedora autoinstall, using serial console tests/vm: ubuntu.i386: apt proxy setup tests/vm/basevm.py | 144 ++++++++++++++++++++++++----- scripts/archive-source.sh | 72 +++++++-------- tests/vm/Makefile.include | 25 ++++- tests/vm/fedora | 187 ++++++++++++++++++++++++++++++++++++++ tests/vm/freebsd | 179 ++++++++++++++++++++++++++++++++++-- tests/vm/netbsd | 187 ++++++++++++++++++++++++++++++++++++-- tests/vm/openbsd | 158 +++++++++++++++++++++++++++++--- tests/vm/ubuntu.i386 | 4 + 8 files changed, 858 insertions(+), 98 deletions(-) create mode 100755 tests/vm/fedora