| Message ID | 20190508085645.11595-1-kraxel@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | tests/vm: serial console autoinstall, misc fixes. | expand |
On 08/05/2019 10.56, Gerd Hoffmann wrote: > This patch series changes the way virtual machines for test builds are > managed. They are created locally on the developer machine now. The > installer is booted on the serial console and the scripts walks through > the dialogs to install and configure the guest. > > That takes the download.patchew.org server out of the loop and makes it > alot easier to tweak the guest images (adding build dependencies for > example). > > The install scripts take care to apply host proxy settings (from *_proxy > environment variables) to the guest, so any package downloads will be > routed through the proxy and can be cached that way. This also makes > them work behind strict firewalls. > > There are also a bunch of smaller tweaks for tests/vm to fix issues I > was struggling with. See commit messages of individual patches for > details. > > Known issue: NetBSD package install is not working for me right now. > It did work a while ago. Not sure what is going on here. I now gave your series another try and replaced patch 3 with the python3 fix from Eduardo locally here. FreeBSD works great. OpenBSD is fine too, except for the known issue that the "gmake check" does not work - but this issue has been there before already. NetBSD also does not work for me, so I guess you should hold off that patch for now? So for patches 1, 2 and 4 - 10 (I did not check the Linux images yet): Tested-by: Thomas Huth <thuth@redhat.com> > Do we have accelerator support for the BSDs? A "make check" for a full > build takes ages, and I suspect tcg being used is part of the problem. > I did my tests using "TARGET_LIST=x86_64-softmmu" because of that. I think they should be running with "--enable-kvm". Did you make sure that you've enabled multiple CPUs with J=8 for example? ... but for me, the compilation is also quite a bit slower, indeed. I think part of the problem might be clang which is compiling a little bit slower than GCC as far as I know...? Thomas
Hi Thomas, On 5/9/19 1:53 PM, Thomas Huth wrote: > On 08/05/2019 10.56, Gerd Hoffmann wrote: >> This patch series changes the way virtual machines for test builds are >> managed. They are created locally on the developer machine now. The >> installer is booted on the serial console and the scripts walks through >> the dialogs to install and configure the guest. >> >> That takes the download.patchew.org server out of the loop and makes it >> alot easier to tweak the guest images (adding build dependencies for >> example). >> >> The install scripts take care to apply host proxy settings (from *_proxy >> environment variables) to the guest, so any package downloads will be >> routed through the proxy and can be cached that way. This also makes >> them work behind strict firewalls. >> >> There are also a bunch of smaller tweaks for tests/vm to fix issues I >> was struggling with. See commit messages of individual patches for >> details. >> >> Known issue: NetBSD package install is not working for me right now. >> It did work a while ago. Not sure what is going on here. > > I now gave your series another try and replaced patch 3 with the python3 > fix from Eduardo locally here. FreeBSD works great. OpenBSD is fine too, > except for the known issue that the "gmake check" does not work - but > this issue has been there before already. [...] "gmake check" was working on OpenBSD with this series: https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg07513.html I think most of the patch proposed there have been merged, so are you talking about a new issue?
On 09/05/2019 14.04, Philippe Mathieu-Daudé wrote: [...] >> I now gave your series another try and replaced patch 3 with the python3 >> fix from Eduardo locally here. FreeBSD works great. OpenBSD is fine too, >> except for the known issue that the "gmake check" does not work - but >> this issue has been there before already. [...] > > "gmake check" was working on OpenBSD with this series: > https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg07513.html > I think most of the patch proposed there have been merged, so are you > talking about a new issue? Oh, true, I remembered the patches, but was not aware that they've been merged. The issue that I've seen is this one: [...] TEST check-qtest-arm: tests/pca9552-test TEST check-qtest-arm: tests/ds1338-test TEST check-qtest-arm: tests/microbit-test TEST check-qtest-arm: tests/m25p80-test TEST check-qtest-arm: tests/test-arm-mptimer TEST check-qtest-arm: tests/boot-serial-test qemu-system-arm: cannot set up guest memory 'ram': Cannot allocate memory Broken pipe /home/qemu/qemu-test.znJ6fy/src/tests/libqtest.c:135: kill_qemu() tried to terminate QEMU process but encountered exit status 1 ERROR - too few tests run (expected 2, got 0) Abort trap (core dumped) gmake: *** [/home/qemu/qemu-test.znJ6fy/src/tests/Makefile.include:903: check-qtest-arm] Error 1 Thomas
Hi, > > Do we have accelerator support for the BSDs? A "make check" for a full > > build takes ages, and I suspect tcg being used is part of the problem. > > I did my tests using "TARGET_LIST=x86_64-softmmu" because of that. > > I think they should be running with "--enable-kvm". The images themself yes, but the tests running *inside* (on make check) don't. cheers, Gerd
On 09/05/2019 15.50, Gerd Hoffmann wrote: > Hi, > >>> Do we have accelerator support for the BSDs? A "make check" for a full >>> build takes ages, and I suspect tcg being used is part of the problem. >>> I did my tests using "TARGET_LIST=x86_64-softmmu" because of that. >> >> I think they should be running with "--enable-kvm". > > The images themself yes, but the tests running *inside* (on make check) don't. No, we don't have accelerator support for *BSD, as far as I know. But we also do not run that much TCG tests during "make check" that you should see such a big difference here. And for me, the compilation step is already way slower than on the host, so I think the problem is likely something else... Thomas
On 08.05.2019 10:56, Gerd Hoffmann wrote: > This patch series changes the way virtual machines for test builds are > managed. They are created locally on the developer machine now. The > installer is booted on the serial console and the scripts walks through > the dialogs to install and configure the guest. > > That takes the download.patchew.org server out of the loop and makes it > alot easier to tweak the guest images (adding build dependencies for > example). > > The install scripts take care to apply host proxy settings (from *_proxy > environment variables) to the guest, so any package downloads will be > routed through the proxy and can be cached that way. This also makes > them work behind strict firewalls. > > There are also a bunch of smaller tweaks for tests/vm to fix issues I > was struggling with. See commit messages of individual patches for > details. > > Known issue: NetBSD package install is not working for me right now. > It did work a while ago. Not sure what is going on here. > Error log? What is the command? pkgin install? > Do we have accelerator support for the BSDs? KVM-style? NetBSD does support HAXM (--accel hax) and in a downstream copy NVMM (-accel nvmm). http://blog.netbsd.org/tnf/entry/the_hardware_assisted_virtualization_challenge http://blog.netbsd.org/tnf/entry/from_zero_to_nvmm Once NVMM will stabilize we intend to submit it upstream. There is no support for hardware assisted acceleration in qemu for any other BSD. > A "make check" for a full > build takes ages, and I suspect tcg being used is part of the problem. > I did my tests using "TARGET_LIST=x86_64-softmmu" because of that. > > Gerd Hoffmann (13): > scripts: use git archive in archive-source > tests/vm: send proxy environment variables over ssh > tests/vm: send locale environment variables over ssh > tests/vm: use ssh with pty unconditionally > tests/vm: run test builds on snapshot > tests/vm: add vm-boot-{ssh,serial}-<guest> targets > tests/vm: add DEBUG=1 to help text > tests/vm: serial console support helpers > tests/vm: openbsd autoinstall, using serial console > tests/vm: freebsd autoinstall, using serial console > tests/vm: netbsd autoinstall, using serial console > tests/vm: fedora autoinstall, using serial console > tests/vm: ubuntu.i386: apt proxy setup > > tests/vm/basevm.py | 125 ++++++++++++++++++++++--- > scripts/archive-source.sh | 72 +++++++-------- > tests/vm/Makefile.include | 25 ++++- > tests/vm/fedora | 187 ++++++++++++++++++++++++++++++++++++++ > tests/vm/freebsd | 172 +++++++++++++++++++++++++++++++++-- > tests/vm/netbsd | 178 ++++++++++++++++++++++++++++++++++-- > tests/vm/openbsd | 150 +++++++++++++++++++++++++++--- > tests/vm/ubuntu.i386 | 4 + > 8 files changed, 830 insertions(+), 83 deletions(-) > create mode 100755 tests/vm/fedora >
On 09.05.2019 15:57, Thomas Huth wrote: > On 09/05/2019 15.50, Gerd Hoffmann wrote: >> Hi, >> >>>> Do we have accelerator support for the BSDs? A "make check" for a full >>>> build takes ages, and I suspect tcg being used is part of the problem. >>>> I did my tests using "TARGET_LIST=x86_64-softmmu" because of that. >>> >>> I think they should be running with "--enable-kvm". >> >> The images themself yes, but the tests running *inside* (on make check) don't. > > No, we don't have accelerator support for *BSD, as far as I know. As mentioned in the other mail, KVM-style? NetBSD does support HAXM (--accel hax) and in a downstream copy NVMM (-accel nvmm). http://blog.netbsd.org/tnf/entry/the_hardware_assisted_virtualization_challenge http://blog.netbsd.org/tnf/entry/from_zero_to_nvmm Once NVMM will stabilize we intend to submit it upstream. There is no support for hardware assisted acceleration in qemu for any other BSD. > But we > also do not run that much TCG tests during "make check" that you should > see such a big difference here. And for me, the compilation step is > already way slower than on the host, so I think the problem is likely > something else... > > Thomas >
On Thu, May 09, 2019 at 08:52:23PM +0200, Kamil Rytarowski wrote: > On 08.05.2019 10:56, Gerd Hoffmann wrote: > > This patch series changes the way virtual machines for test builds are > > managed. They are created locally on the developer machine now. The > > installer is booted on the serial console and the scripts walks through > > the dialogs to install and configure the guest. > > > > That takes the download.patchew.org server out of the loop and makes it > > alot easier to tweak the guest images (adding build dependencies for > > example). > > > > The install scripts take care to apply host proxy settings (from *_proxy > > environment variables) to the guest, so any package downloads will be > > routed through the proxy and can be cached that way. This also makes > > them work behind strict firewalls. > > > > There are also a bunch of smaller tweaks for tests/vm to fix issues I > > was struggling with. See commit messages of individual patches for > > details. > > > > Known issue: NetBSD package install is not working for me right now. > > It did work a while ago. Not sure what is going on here. > > > > Error log? What is the command? pkgin install? Looked like a dependency problem, the error log complained that it couldn't find a new enough tcl version for tk. "fixed" that by installing git-base instead of git, which drop the tk dependency of git. cheers, Gerd
This patch series changes the way virtual machines for test builds are managed. They are created locally on the developer machine now. The installer is booted on the serial console and the scripts walks through the dialogs to install and configure the guest. That takes the download.patchew.org server out of the loop and makes it alot easier to tweak the guest images (adding build dependencies for example). The install scripts take care to apply host proxy settings (from *_proxy environment variables) to the guest, so any package downloads will be routed through the proxy and can be cached that way. This also makes them work behind strict firewalls. There are also a bunch of smaller tweaks for tests/vm to fix issues I was struggling with. See commit messages of individual patches for details. Known issue: NetBSD package install is not working for me right now. It did work a while ago. Not sure what is going on here. Do we have accelerator support for the BSDs? A "make check" for a full build takes ages, and I suspect tcg being used is part of the problem. I did my tests using "TARGET_LIST=x86_64-softmmu" because of that. Gerd Hoffmann (13): scripts: use git archive in archive-source tests/vm: send proxy environment variables over ssh tests/vm: send locale environment variables over ssh tests/vm: use ssh with pty unconditionally tests/vm: run test builds on snapshot tests/vm: add vm-boot-{ssh,serial}-<guest> targets tests/vm: add DEBUG=1 to help text tests/vm: serial console support helpers tests/vm: openbsd autoinstall, using serial console tests/vm: freebsd autoinstall, using serial console tests/vm: netbsd autoinstall, using serial console tests/vm: fedora autoinstall, using serial console tests/vm: ubuntu.i386: apt proxy setup tests/vm/basevm.py | 125 ++++++++++++++++++++++--- scripts/archive-source.sh | 72 +++++++-------- tests/vm/Makefile.include | 25 ++++- tests/vm/fedora | 187 ++++++++++++++++++++++++++++++++++++++ tests/vm/freebsd | 172 +++++++++++++++++++++++++++++++++-- tests/vm/netbsd | 178 ++++++++++++++++++++++++++++++++++-- tests/vm/openbsd | 150 +++++++++++++++++++++++++++--- tests/vm/ubuntu.i386 | 4 + 8 files changed, 830 insertions(+), 83 deletions(-) create mode 100755 tests/vm/fedora