From patchwork Tue Oct 16 17:50:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Cleber Rosa X-Patchwork-Id: 10644095 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 332A115E2 for ; Tue, 16 Oct 2018 18:28:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 244F42A5F9 for ; Tue, 16 Oct 2018 18:28:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 18BB02A605; Tue, 16 Oct 2018 18:28:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7C2CB2A5F9 for ; Tue, 16 Oct 2018 18:28:11 +0000 (UTC) Received: from localhost ([::1]:59610 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gCU4g-0006MU-PA for patchwork-qemu-devel@patchwork.kernel.org; Tue, 16 Oct 2018 14:28:10 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:60489) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gCTUw-0005Q5-NO for qemu-devel@nongnu.org; Tue, 16 Oct 2018 13:51:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gCTUs-0003xX-2n for qemu-devel@nongnu.org; Tue, 16 Oct 2018 13:51:14 -0400 Received: from mx1.redhat.com ([209.132.183.28]:36042) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gCTUr-0003vr-M2 for qemu-devel@nongnu.org; Tue, 16 Oct 2018 13:51:10 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id CF2A73082A42; Tue, 16 Oct 2018 17:51:08 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-123-183.rdu2.redhat.com [10.10.123.183]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8792C60150; Tue, 16 Oct 2018 17:50:44 +0000 (UTC) From: Cleber Rosa To: qemu-devel@nongnu.org Date: Tue, 16 Oct 2018 13:50:39 -0400 Message-Id: <20181016175042.578-1-crosa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Tue, 16 Oct 2018 17:51:08 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v5 0/3] Bootstrap Python venv and acceptance/functional tests X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Eduardo Habkost , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Cleber Rosa , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Laszlo Ersek , Stefan Hajnoczi , Caio Carrara , =?utf-8?q?Alex_Benn=C3=A9e?= Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP TL;DR ===== Allow acceptance tests to be run with `make check-acceptance`. Details ======= This introduces a Python virtual environment that will be setup within the QEMU build directory, that will contain the exact environment that tests may require. There's one current caveat: it requires Python 3, as it's based on the venv module. This was based on some discussions and perception about standardizing on Python 3, but can easily be made to accommodate Python 2 as well. Example of bootstrap and test execution on Travis-CI: https://travis-ci.org/qemu/qemu/jobs/439331028#L2508 ... VENV /home/travis/build/qemu/qemu/tests/venv MKDIR /home/travis/build/qemu/qemu/tests/results PIP /home/travis/build/qemu/qemu/tests/venv-requirements.txt AVOCADO tests/acceptance JOB ID : 920e4fcf55a1782f1ae77bee64b20ccdc2e1111d JOB LOG : /home/travis/build/qemu/qemu/tests/results/job-2018-10-09T21.42-920e4fc/job.log (1/6) /home/travis/build/qemu/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: PASS (3.57 s) (2/6) /home/travis/build/qemu/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version: PASS (0.04 s) (3/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc: PASS (0.04 s) (4/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: PASS (0.04 s) (5/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: PASS (0.04 s) (6/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password: PASS (0.04 s) RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 3.90 s ... Changes from v4: ================ * Added Python 3 version check and error message when running under Python 2. (Philippe) * Added notes about using Avocado from the virtual environment without previous activation. (Phillipe) Ideas discussed, but not implemented: * Instructions to activate/deactivate the venv: the idea is to be use the venv as transparently as possible. * Move/rename tests/venv-requirements: the ideal, as per the first commit is to provide one venv for all Python based tests to run. * Changed the Travis package requirements (python3-pip, python3.4-venv) from the global setting to the job specific one: waiting on further feedback from'Alex Bennée. Changes from v3: ================ * Fixed typo in commit message (s/requiment/requirement/). (Eric) Changes from v2: ================ * Make the $(TESTS_VENV_DIR) target depend on the venv-requirements.txt file, and touch $(TESTS_VENV_DIR) after venv runs. With this, updates on the file are reflected on the venv. (Philippe) * Run pip with "python -m pip". It may have been installed reusing the system wide packages, and then the script may not be available on the venv. (Philippe) * Dropped Python version on Travis, and using the version supplied by the distro (3.4). (Philippe) * Added "python3.4-venv" package requirement on Travis. (Philippe) * Added variable (AVOCADO_SHOW) with logging streams to be shown while running the acceptance tests. By default it's set to none, the equivalent of the quiet mode used on previous versions. (Philippe) * On Travis, set the AVOCADO_SHOW variable to "app", so that the individual test results can be easily seen. (Philippe) Ideas discussed, but not implemented: * Run pip with "$(PYTHON) -m pip -q install ..." because it points to the system wide Python installation. (Philippe) * Drop the "--system-site-packages" flag. Waiting on another round of tests to determine if they are really the cause of some package installation problems. Changes from v1: ================ * TESTS_VENV_REQ (the path of "venv-requirements.txt") now points to the source path ($SRC_PATH instead of $BUILD_DIR) * Create the venv with "--system-site-packages", which allows the reuse of packages (and no additional downloads) in case there's a package installed system wide providing the same package and version. * Run Avocado with "python -m avocado". It may have been installed reusing the system wide packages, and then the script may not be available on the venv. * Improved documentation describing the Python 3, venv and pip requirements. * Updated avocado-framework requirement to latest released version (65.0) * (New commit) Added support for running the acceptance tests on Travis. Ideas discussed, but not implemented: * Install external packages such as python3-pip on Debian based systems, deemed too invasive on developer's systems. * Allow the use of Python 2, and consequently the "virtualenv" module. Cleber Rosa (3): Bootstrap Python venv for tests Acceptance tests: add make rule for running them Travis support for the acceptance tests .travis.yml | 5 +++++ docs/devel/testing.rst | 43 ++++++++++++++++++++++++++++++++----- tests/Makefile.include | 43 +++++++++++++++++++++++++++++++++++++ tests/venv-requirements.txt | 4 ++++ 4 files changed, 90 insertions(+), 5 deletions(-) create mode 100644 tests/venv-requirements.txt Acked-by: Stefan Hajnoczi