From patchwork Fri Apr 14 05:54:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13210966 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2C580C77B79 for ; Fri, 14 Apr 2023 05:55:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pnCOo-0007Ku-Sh; Fri, 14 Apr 2023 01:55:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOm-0007KY-Ak for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:04 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOi-0000Ih-BA for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681451694; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8fRsZFhLyR6e1nO5VCN8SkoCsQI4OWXbpIyhKdc+2nA=; b=jBWSaoQFOdE2uyufYKhzhguKmAjdzK++ure+xAzEjjbeC+tGVDEGCo2InNjyuquBXdLlpt o2ff+JMF61e67myO4gVnkyrAc2VbBp0cONAnjJgEes0FCnCsAn4Scqz1DGhHkXoJmHsKh0 FIrXgC+7f+8Qt5f1MdmF/Ul+HAlvUwU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-611-gpW3d7rENFWI5U1jX_KtCg-1; Fri, 14 Apr 2023 01:54:51 -0400 X-MC-Unique: gpW3d7rENFWI5U1jX_KtCg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 15CDE3C0F388; Fri, 14 Apr 2023 05:54:51 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.197]) by smtp.corp.redhat.com (Postfix) with ESMTP id 80532C16031; Fri, 14 Apr 2023 05:54:50 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Cleber Rosa , Ani Sinha , Warner Losh , =?utf-8?q?Alex_Benn=C3=A9e?= , Reinoud Zandijk , Thomas Huth , Paolo Bonzini , John Snow , Ryo ONODERA , Kyle Evans , Beraldo Leal , "Michael S. Tsirkin" , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= Subject: [RFC PATCH v2 01/10] python: add mkvenv.py Date: Fri, 14 Apr 2023 01:54:40 -0400 Message-Id: <20230414055449.4028284-2-jsnow@redhat.com> In-Reply-To: <20230414055449.4028284-1-jsnow@redhat.com> References: <20230414055449.4028284-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This script will be responsible for building a lightweight Python virtual environment at configure time. It works with Python 3.7 or newer. It has been designed to: - work *offline*, no PyPI required. - work *quickly*, The fast path is only ~65ms on my machine. - work *robustly*, with multiple fallbacks to keep things working. - work *cooperatively*, using system packages where possible. (You can use your distro's meson, no problem.) Due to its unique position in the build chain, it exists outside of the installable python packages in-tree and *must* be runnable without any third party dependencies. Under normal circumstances, the only dependency required to execute this script is Python 3.7+ itself. The script is *faster* by several seconds when setuptools and pip are installed in the host environment, which is probably the case for a typical multi-purpose developer workstation. In the event that pip/setuptools are missing or not usable, additional dependencies may be required on some distributions which remove certain Python stdlib modules to package them separately: - Debian may require python3-venv to provide "ensurepip" - NetBSD may require py310-expat to provide "pyexpat" * (* Or whichever version is current for NetBSD.) Signed-off-by: John Snow --- python/scripts/mkvenv.py | 379 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 379 insertions(+) create mode 100644 python/scripts/mkvenv.py diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py new file mode 100644 index 0000000000..6df95382f3 --- /dev/null +++ b/python/scripts/mkvenv.py @@ -0,0 +1,379 @@ +""" +mkvenv - QEMU venv bootstrapping utility + +usage: TODO/FIXME +""" + +# Copyright (C) 2022-2023 Red Hat, Inc. +# +# Authors: +# John Snow +# Paolo Bonzini +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import argparse +from importlib.util import find_spec +import logging +import os +from os import PathLike +from pathlib import Path +import re +import stat +import subprocess +import sys +import traceback +from types import SimpleNamespace +from typing import Dict, Sequence, Optional, Union, Iterator +import venv + + +# Do not add any mandatory dependencies from outside the stdlib: +# This script *must* be usable standalone! + +DirType = Union[str, bytes, 'PathLike[str]', 'PathLike[bytes]'] +logger = logging.getLogger('mkvenv') + + +class Ouch(RuntimeError): + """An Exception class we can't confuse with a builtin.""" + + +class QemuEnvBuilder(venv.EnvBuilder): + """ + An extension of venv.EnvBuilder for building QEMU's configure-time venv. + + The only functional change is that it adds the ability to regenerate + console_script shims for packages available via system_site + packages. + + Parameters for base class init: + - system_site_packages: bool = False + - clear: bool = False + - symlinks: bool = False + - upgrade: bool = False + - with_pip: bool = False + - prompt: Optional[str] = None + - upgrade_deps: bool = False (Since 3.9) + """ + def __init__(self, *args, **kwargs) -> None: + logger.debug("QemuEnvBuilder.__init__(...)") + self.script_packages = kwargs.pop('script_packages', ()) + super().__init__(*args, **kwargs) + + # The EnvBuilder class is cute and toggles this setting off + # before post_setup, but we need it to decide if we want to + # generate shims or not. + self._system_site_packages = self.system_site_packages + + def post_setup(self, context: SimpleNamespace) -> None: + logger.debug("post_setup(...)") + + # Generate console_script entry points for system packages: + if self._system_site_packages: + generate_console_scripts( + context.env_exe, context.bin_path, self.script_packages) + + # print the python executable to stdout for configure. + print(context.env_exe) + + +def need_ensurepip() -> bool: + """ + Tests for the presence of setuptools and pip. + + :return: `True` if we do not detect both packages. + """ + # Don't try to actually import them, it's fraught with danger: + # https://github.com/pypa/setuptools/issues/2993 + if find_spec("setuptools") and find_spec("pip"): + return False + return True + + +def check_ensurepip(with_pip: bool) -> None: + """ + Check that we have ensurepip. + + Raise a fatal exception with a helpful hint if it isn't available. + """ + if with_pip and not find_spec("ensurepip"): + msg = ("Python's ensurepip module is not found.\n" + + "It's normally part of the Python standard library, " + "maybe your distribution packages it separately?\n" + + "Either install ensurepip, or alleviate the need for it in the " + "first place by installing pip and setuptools for " + f"'{sys.executable}'.\n" + + "(Hint: Debian puts ensurepip in its python3-venv package.)") + raise Ouch(msg) + + +def make_venv( # pylint: disable=too-many-arguments + venv_path: Union[str, Path], + system_site_packages: bool = False, + clear: bool = True, + symlinks: Optional[bool] = None, + upgrade: bool = False, + with_pip: Optional[bool] = None, + script_packages: Sequence[str]=(), +) -> None: + """ + Create a venv using the QemuEnvBuilder class. + + TODO/FIXME: write docs O:-) + """ + logging.debug("%s: make_venv(venv_path=%s, system_site_packages=%s, " + "clear=%s, symlinks=%s, upgrade=%s, with_pip=%s, " + "script_packages=%s)", + __file__, str(venv_path), system_site_packages, + clear, symlinks, upgrade, with_pip, script_packages) + + print(f"MKVENV {str(venv_path)}", file=sys.stderr) + + # ensurepip is slow: venv creation can be very fast for cases where + # we allow the use of system_site_packages. Toggle ensure_pip on only + # in the cases where we really need it. + if with_pip is None: + with_pip = True if not system_site_packages else need_ensurepip() + logger.debug("with_pip unset, choosing %s", with_pip) + + check_ensurepip(with_pip) + + if symlinks is None: + # Default behavior of standard venv CLI + symlinks = os.name != "nt" + + builder = QemuEnvBuilder( + system_site_packages=system_site_packages, + clear=clear, + symlinks=symlinks, + upgrade=upgrade, + with_pip=with_pip, + script_packages=script_packages, + ) + try: + logger.debug("Invoking builder.create()") + try: + builder.create(str(venv_path)) + except SystemExit as exc: + # Some versions of the venv module raise SystemExit; *nasty*! + # We want the exception that prompted it. It might be a subprocess + # error that has output we *really* want to see. + logger.debug("Intercepted SystemExit from EnvBuilder.create()") + raise exc.__cause__ or exc.__context__ or exc + logger.debug("builder.create() finished") + except subprocess.CalledProcessError as exc: + print(f"cmd: {exc.cmd}", file=sys.stderr) + print(f"returncode: {exc.returncode}", file=sys.stderr) + + def _stringify(data: Union[str, bytes]) -> str: + if data and isinstance(data, bytes): + return data.decode() + return data + + if exc.stdout: + print("========== stdout ==========", + _stringify(exc.stdout), + "============================", + sep='\n', file=sys.stderr) + if exc.stderr: + print("========== stderr ==========", + _stringify(exc.stderr), + "============================", + sep='\n', file=sys.stderr) + raise Ouch("VENV creation subprocess failed.") from exc + + +def _gen_importlib(packages: Sequence[str]) -> Iterator[Dict[str, str]]: + # pylint: disable=import-outside-toplevel + try: + # First preference: Python 3.8+ stdlib + from importlib.metadata import ( + PackageNotFoundError, + distribution, + ) + except ImportError as exc: + logger.debug("%s", str(exc)) + # Second preference: Commonly available PyPI backport + from importlib_metadata import ( + PackageNotFoundError, + distribution, + ) + + # Borrowed from CPython (Lib/importlib/metadata/__init__.py) + pattern = re.compile( + r'(?P[\w.]+)\s*' + r'(:\s*(?P[\w.]+)\s*)?' + r'((?P\[.*\])\s*)?$' + ) + + def _generator() -> Iterator[Dict[str, str]]: + for package in packages: + try: + entry_points = distribution(package).entry_points + except PackageNotFoundError: + continue + + # The EntryPoints type is only available in 3.10+, + # treat this as a vanilla list and filter it ourselves. + entry_points = filter( + lambda ep: ep.group == 'console_scripts', entry_points) + + for entry_point in entry_points: + # Python 3.8 doesn't have 'module' or 'attr' attributes + if not (hasattr(entry_point, 'module') and + hasattr(entry_point, 'attr')): + match = pattern.match(entry_point.value) + assert match is not None + module = match.group('module') + attr = match.group('attr') + else: + module = entry_point.module + attr = entry_point.attr + yield { + 'name': entry_point.name, + 'module': module, + 'import_name': attr, + 'func': attr, + } + + return _generator() + + +def _gen_pkg_resources(packages: Sequence[str]) -> Iterator[Dict[str, str]]: + # pylint: disable=import-outside-toplevel + # Bundled with setuptools; has a good chance of being available. + import pkg_resources + + def _generator() -> Iterator[Dict[str, str]]: + for package in packages: + try: + eps = pkg_resources.get_entry_map(package, 'console_scripts') + except pkg_resources.DistributionNotFound: + continue + + for entry_point in eps.values(): + yield { + 'name': entry_point.name, + 'module': entry_point.module_name, + 'import_name': ".".join(entry_point.attrs), + 'func': ".".join(entry_point.attrs), + } + + return _generator() + + +# Borrowed/adapted from pip's vendored version of distutils: +SCRIPT_TEMPLATE = r'''#!{python_path:s} +# -*- coding: utf-8 -*- +import re +import sys +from {module:s} import {import_name:s} +if __name__ == '__main__': + sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) + sys.exit({func:s}()) +''' + + +def generate_console_scripts(python_path: str, bin_path: str, + packages: Sequence[str]) -> None: + """ + Generate script shims for console_script entry points in @packages. + """ + if not packages: + return + + def _get_entry_points() -> Iterator[Dict[str, str]]: + """Python 3.7 compatibility shim for iterating entry points.""" + # Python 3.8+, or Python 3.7 with importlib_metadata installed. + try: + return _gen_importlib(packages) + except ImportError as exc: + logger.debug("%s", str(exc)) + + # Python 3.7 with setuptools installed. + try: + return _gen_pkg_resources(packages) + except ImportError as exc: + logger.debug("%s", str(exc)) + raise Ouch( + "Neither importlib.metadata nor pkg_resources found, " + "can't generate console script shims.\n" + "Use Python 3.8+, or install importlib-metadata, or setuptools." + ) from exc + + for entry_point in _get_entry_points(): + script_path = os.path.join(bin_path, entry_point['name']) + script = SCRIPT_TEMPLATE.format(python_path=python_path, **entry_point) + with open(script_path, "w", encoding='UTF-8') as file: + file.write(script) + mode = os.stat(script_path).st_mode | stat.S_IEXEC + os.chmod(script_path, mode) + + logger.debug("wrote '%s'", script_path) + + +def main() -> int: + """CLI interface to make_qemu_venv. See module docstring.""" + if os.environ.get('DEBUG') or os.environ.get('GITLAB_CI'): + # You're welcome. + logging.basicConfig(level=logging.DEBUG) + elif os.environ.get('V'): + logging.basicConfig(level=logging.INFO) + + parser = argparse.ArgumentParser(description="Bootstrap QEMU venv.") + subparsers = parser.add_subparsers( + title="Commands", + description="Various actions this utility can perform", + prog="prog", + dest="command", + required=True, + metavar="command", + help='Description') + + subparser = subparsers.add_parser('create', help='create a venv') + subparser.add_argument( + '--gen', + type=str, + action='append', + help="Regenerate console_scripts for given packages, if found.", + ) + subparser.add_argument( + 'target', + type=str, + action='store', + help="Target directory to install virtual environment into.", + ) + + args = parser.parse_args() + try: + if args.command == 'create': + script_packages = [] + for element in args.gen or (): + script_packages.extend(element.split(",")) + make_venv( + args.target, + system_site_packages=True, + clear=True, + upgrade=False, + with_pip=None, # Autodetermine + script_packages=script_packages, + ) + logger.debug("mkvenv.py create - exiting") + except Ouch as exc: + print("\n*** Ouch! ***\n", file=sys.stderr) + print(str(exc), "\n\n", file=sys.stderr) + return 1 + except: # pylint: disable=bare-except + print("mkvenv did not complete successfully:", file=sys.stderr) + traceback.print_exc() + return 2 + return 0 + + +if __name__ == '__main__': + sys.exit(main()) From patchwork Fri Apr 14 05:54:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13210972 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 82C82C77B70 for ; Fri, 14 Apr 2023 05:57:13 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pnCOt-0007Lh-AI; Fri, 14 Apr 2023 01:55:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOr-0007LY-Eu for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:09 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOi-0000IU-Jm for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681451693; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iXu1iBWmPnnbcqJanHFlz0WgmsKQEotu0q7wUfsyNgQ=; b=gLR5+RbMUe1+65pWK4uRr46HH5x9Z8aRbY9Wjadc/9jmeUwuxL+ouINxAYLQdDWkI+/Twb CI5mtf0hC6BW7w9KSOk7X/WOGk2hWtVuYF4zE1LlnX3/1SKslt+9qM/dDrvlSTaO+XZzAP WZjUSSEXRh0Rv8lMmAeDVIRTYM3Zi7w= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-260-dzCrsnhrOB2WuXAaLA-tKQ-1; Fri, 14 Apr 2023 01:54:52 -0400 X-MC-Unique: dzCrsnhrOB2WuXAaLA-tKQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A9C5A855304; Fri, 14 Apr 2023 05:54:51 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.197]) by smtp.corp.redhat.com (Postfix) with ESMTP id 25A2EC1602A; Fri, 14 Apr 2023 05:54:51 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Cleber Rosa , Ani Sinha , Warner Losh , =?utf-8?q?Alex_Benn=C3=A9e?= , Reinoud Zandijk , Thomas Huth , Paolo Bonzini , John Snow , Ryo ONODERA , Kyle Evans , Beraldo Leal , "Michael S. Tsirkin" , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= Subject: [RFC PATCH v2 02/10] tests: add python3-venv dependency Date: Fri, 14 Apr 2023 01:54:41 -0400 Message-Id: <20230414055449.4028284-3-jsnow@redhat.com> In-Reply-To: <20230414055449.4028284-1-jsnow@redhat.com> References: <20230414055449.4028284-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Several debian-based tests need the python3-venv dependency as a consequence of Debian debundling the "ensurepip" module normally included with Python. As mkvenv.py stands as of this commit, Debian requires EITHER: (A) setuptools and pip, or (B) ensurepip mkvenv is a few seconds faster if you have setuptools and pip, so developers should prefer the first requirement. For the purposes of CI, the time-save is a wash; it's only a matter of who is responsible for installing pip and when; the timing is about the same. Arbitrarily, I chose adding ensurepip to the test configuration because it is normally part of the Python stdlib, and always having it allows us a more consistent cross-platform environment. Signed-off-by: John Snow Reviewed-by: Philippe Mathieu-Daudé --- tests/docker/dockerfiles/debian-all-test-cross.docker | 3 ++- tests/docker/dockerfiles/debian-hexagon-cross.docker | 3 ++- tests/docker/dockerfiles/debian-riscv64-cross.docker | 3 ++- tests/docker/dockerfiles/debian-tricore-cross.docker | 3 ++- 4 files changed, 8 insertions(+), 4 deletions(-) diff --git a/tests/docker/dockerfiles/debian-all-test-cross.docker b/tests/docker/dockerfiles/debian-all-test-cross.docker index 981e9bdc7b..f9f401544a 100644 --- a/tests/docker/dockerfiles/debian-all-test-cross.docker +++ b/tests/docker/dockerfiles/debian-all-test-cross.docker @@ -57,7 +57,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \ gcc-sh4-linux-gnu \ libc6-dev-sh4-cross \ gcc-sparc64-linux-gnu \ - libc6-dev-sparc64-cross + libc6-dev-sparc64-cross \ + python3-venv ENV QEMU_CONFIGURE_OPTS --disable-system --disable-docs --disable-tools ENV DEF_TARGET_LIST aarch64-linux-user,alpha-linux-user,arm-linux-user,hppa-linux-user,i386-linux-user,m68k-linux-user,mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,ppc-linux-user,ppc64-linux-user,ppc64le-linux-user,riscv64-linux-user,s390x-linux-user,sh4-linux-user,sparc64-linux-user diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker b/tests/docker/dockerfiles/debian-hexagon-cross.docker index b99d99f943..c2cfb6a5d0 100644 --- a/tests/docker/dockerfiles/debian-hexagon-cross.docker +++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker @@ -20,7 +20,8 @@ RUN apt-get update && \ bison \ flex \ git \ - ninja-build && \ + ninja-build \ + python3-venv && \ # Install QEMU build deps for use in CI DEBIAN_FRONTEND=noninteractive eatmydata \ apt build-dep -yy --arch-only qemu diff --git a/tests/docker/dockerfiles/debian-riscv64-cross.docker b/tests/docker/dockerfiles/debian-riscv64-cross.docker index 803afb9573..081404e014 100644 --- a/tests/docker/dockerfiles/debian-riscv64-cross.docker +++ b/tests/docker/dockerfiles/debian-riscv64-cross.docker @@ -28,7 +28,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata apt install -yy \ libglib2.0-dev \ ninja-build \ pkg-config \ - python3 + python3 \ + python3-venv # Add ports and riscv64 architecture RUN echo "deb http://ftp.ports.debian.org/debian-ports/ sid main" >> /etc/apt/sources.list diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker index cfd2faf9a8..269bfa8d42 100644 --- a/tests/docker/dockerfiles/debian-tricore-cross.docker +++ b/tests/docker/dockerfiles/debian-tricore-cross.docker @@ -33,7 +33,8 @@ RUN apt update && \ pkgconf \ python3-pip \ python3-setuptools \ - python3-wheel + python3-wheel \ + python3-venv RUN curl -#SL https://github.com/bkoppelmann/package_940/releases/download/tricore-toolchain-9.40/tricore-toolchain-9.4.0.tar.gz \ | tar -xzC /usr/local/ From patchwork Fri Apr 14 05:54:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13210971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 02EEDC77B6E for ; Fri, 14 Apr 2023 05:57:05 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pnCOx-0007MY-KF; Fri, 14 Apr 2023 01:55:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOw-0007MA-1v for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:14 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOk-0000J4-6o for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681451697; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eQ2erG1MU1ZY39NyEs1hlfOGYD0KejDWROaGFqr2jJs=; b=WkYX3s4cHDCnGeoJD9babQ6oA6TQ8uICOPQ8YN9LCqpwvT4lrQp6lx3hJLPiw/2MGjoKZs 2YkJTBjtntHJ4Wb6Clh+H/1U1u21/ImBwyth5MThi0/iBLOrXGppHiJpF6XKzIcNoGUURc W943eTvUtBv8VZd3J+fkiD0ftpsmGf8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-281-8KBWm51zM4S2x8F5grVx4w-1; Fri, 14 Apr 2023 01:54:52 -0400 X-MC-Unique: 8KBWm51zM4S2x8F5grVx4w-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 458C685A588; Fri, 14 Apr 2023 05:54:52 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.197]) by smtp.corp.redhat.com (Postfix) with ESMTP id B712FC1602A; Fri, 14 Apr 2023 05:54:51 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Cleber Rosa , Ani Sinha , Warner Losh , =?utf-8?q?Alex_Benn=C3=A9e?= , Reinoud Zandijk , Thomas Huth , Paolo Bonzini , John Snow , Ryo ONODERA , Kyle Evans , Beraldo Leal , "Michael S. Tsirkin" , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= Subject: [RFC PATCH v2 03/10] mkvenv: Add better error message for missing pyexapt module Date: Fri, 14 Apr 2023 01:54:42 -0400 Message-Id: <20230414055449.4028284-4-jsnow@redhat.com> In-Reply-To: <20230414055449.4028284-1-jsnow@redhat.com> References: <20230414055449.4028284-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org NetBSD debundles pyexpat from python, but ensurepip needs pyexpat. Try our best to offer a helpful error message instead of just failing catastrophically. Signed-off-by: John Snow --- python/scripts/mkvenv.py | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py index 6df95382f3..a3284e9ef1 100644 --- a/python/scripts/mkvenv.py +++ b/python/scripts/mkvenv.py @@ -98,7 +98,10 @@ def check_ensurepip(with_pip: bool) -> None: Raise a fatal exception with a helpful hint if it isn't available. """ - if with_pip and not find_spec("ensurepip"): + if not with_pip: + return + + if not find_spec("ensurepip"): msg = ("Python's ensurepip module is not found.\n" "It's normally part of the Python standard library, " @@ -111,6 +114,20 @@ def check_ensurepip(with_pip: bool) -> None: "(Hint: Debian puts ensurepip in its python3-venv package.)") raise Ouch(msg) + # ensurepip uses pyexpat, which can also go missing on us: + if not find_spec("pyexpat"): + msg = ("Python's pyexpat module is not found.\n" + + "It's normally part of the Python standard library, " + "maybe your distribution packages it separately?\n" + + "Either install pyexpat, or alleviate the need for it in the " + "first place by installing pip and setuptools for " + f"'{sys.executable}'.\n\n" + + "(Hint: NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)") + raise Ouch(msg) + def make_venv( # pylint: disable=too-many-arguments venv_path: Union[str, Path], From patchwork Fri Apr 14 05:54:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13210970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 53A38C77B70 for ; Fri, 14 Apr 2023 05:57:03 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pnCP7-0007Qa-85; Fri, 14 Apr 2023 01:55:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCP0-0007Og-C9 for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:18 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOn-0000LP-Hu for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681451704; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ghbh4qRNVDBPRglbq18ow59ZrTEcqWklHbrforGx4nU=; b=K74J+SxhZrMbxNDC1bh25KE/GRt836ZoP1k9pDg3hYkr8CrXDP4ZNhcGwARWsb9HcK1rn0 X/yziQosR/85b2sBUUPAkHCpI7M0LYDcHn2rqp0F4SQYgfDkOLatnGG7QFfXbxzwHi4nHa CD9naGdwg9fYe34vZxEiSY9FpzzHEiU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-315-9Yj4XkhvNM-fdzw72wrKrg-1; Fri, 14 Apr 2023 01:54:53 -0400 X-MC-Unique: 9Yj4XkhvNM-fdzw72wrKrg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D73173C0F38C; Fri, 14 Apr 2023 05:54:52 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.197]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5280DC1602A; Fri, 14 Apr 2023 05:54:52 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Cleber Rosa , Ani Sinha , Warner Losh , =?utf-8?q?Alex_Benn=C3=A9e?= , Reinoud Zandijk , Thomas Huth , Paolo Bonzini , John Snow , Ryo ONODERA , Kyle Evans , Beraldo Leal , "Michael S. Tsirkin" , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= Subject: [RFC PATCH v2 04/10] tests/vm: Configure netbsd to use Python 3.10 Date: Fri, 14 Apr 2023 01:54:43 -0400 Message-Id: <20230414055449.4028284-5-jsnow@redhat.com> In-Reply-To: <20230414055449.4028284-1-jsnow@redhat.com> References: <20230414055449.4028284-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org NetBSD removes some packages from the Python stdlib, but only re-packages them for Python 3.10. Switch to using Python 3.10. Signed-off-by: John Snow --- tests/vm/netbsd | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/vm/netbsd b/tests/vm/netbsd index 0b9536ca17..13eae109c0 100755 --- a/tests/vm/netbsd +++ b/tests/vm/netbsd @@ -30,6 +30,7 @@ class NetBSDVM(basevm.BaseVM): "git-base", "pkgconf", "xz", + "python310", "ninja-build", # gnu tools From patchwork Fri Apr 14 05:54:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13210973 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E86B2C77B70 for ; Fri, 14 Apr 2023 05:57:28 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pnCOr-0007LP-9R; Fri, 14 Apr 2023 01:55:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOm-0007Kh-JT for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:04 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOi-0000Ix-B2 for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681451695; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qCN5N1sd5u3Iw2MtpMYHLDxOFq022o0H7fXfF2oJZDM=; b=Du1EvGoIs5BBOPwr5JiMAMLLu7o2eqet+zW5UQ1JwnVZVCJXyM5bKJ9MYWgv/2C2GMPqf7 95pYFcU7HO53GA77IcFjU1KMHWiTTy4WEfEsrwpG5m1gASPuCoiGpebY5fybkQO5jV9FA2 upQuGi+Rir1VtPNmTM4Rs+yKAkL0Zv0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-118-F0CpcGbyOna0z2yVRIn3wQ-1; Fri, 14 Apr 2023 01:54:54 -0400 X-MC-Unique: F0CpcGbyOna0z2yVRIn3wQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 73BAD85C6E1; Fri, 14 Apr 2023 05:54:53 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.197]) by smtp.corp.redhat.com (Postfix) with ESMTP id E5069C1602A; Fri, 14 Apr 2023 05:54:52 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Cleber Rosa , Ani Sinha , Warner Losh , =?utf-8?q?Alex_Benn=C3=A9e?= , Reinoud Zandijk , Thomas Huth , Paolo Bonzini , John Snow , Ryo ONODERA , Kyle Evans , Beraldo Leal , "Michael S. Tsirkin" , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= Subject: [RFC PATCH v2 05/10] tests/vm: add py310-expat to NetBSD Date: Fri, 14 Apr 2023 01:54:44 -0400 Message-Id: <20230414055449.4028284-6-jsnow@redhat.com> In-Reply-To: <20230414055449.4028284-1-jsnow@redhat.com> References: <20230414055449.4028284-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org NetBSD cannot successfully run "ensurepip" without access to the pyexpat module, which NetBSD debundles. Like the Debian patch, it would be strictly faster long term to install pip/setuptools, and I recommend developers at their workstations take that approach instead. For the purposes of a throwaway VM, there's not really a speed difference for who is responsible for installing pip; us (needs py310-pip) or Python (needs py310-expat). Signed-off-by: John Snow --- tests/vm/netbsd | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/vm/netbsd b/tests/vm/netbsd index 13eae109c0..c7e3f1e735 100755 --- a/tests/vm/netbsd +++ b/tests/vm/netbsd @@ -31,6 +31,7 @@ class NetBSDVM(basevm.BaseVM): "pkgconf", "xz", "python310", + "py310-expat", "ninja-build", # gnu tools From patchwork Fri Apr 14 05:54:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13210964 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1A77FC77B78 for ; Fri, 14 Apr 2023 05:55:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pnCP0-0007Oe-Cp; Fri, 14 Apr 2023 01:55:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOx-0007Mb-OG for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:15 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOi-0000JF-Ff for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681451699; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YWbXIR6WGdAP0EgxpzDb1P+wi33ty82hF8pv1+xBob0=; b=hpWvE5NK0eOrAqJD6lKpVuwRSbc25epexAZcMygdfN/bXJuCDceq757G1s7Moo/rklWqZo 5qQIDTBEx2bGVgL9l3zk4X4eVK1NxIjfBrEW1XSKCLngKFIf6NECUh8NQmuWQGLjzVUc8c Y0VvsRMTMANWk8eYVSveA9w3OTdmMp8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-384-w50JMwsIPKa7T080weyDtA-1; Fri, 14 Apr 2023 01:54:54 -0400 X-MC-Unique: w50JMwsIPKa7T080weyDtA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 10D93185A78F; Fri, 14 Apr 2023 05:54:54 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.197]) by smtp.corp.redhat.com (Postfix) with ESMTP id 80587C1602A; Fri, 14 Apr 2023 05:54:53 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Cleber Rosa , Ani Sinha , Warner Losh , =?utf-8?q?Alex_Benn=C3=A9e?= , Reinoud Zandijk , Thomas Huth , Paolo Bonzini , John Snow , Ryo ONODERA , Kyle Evans , Beraldo Leal , "Michael S. Tsirkin" , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= Subject: [RFC PATCH v2 06/10] mkvenv: generate console entry shims from inside the venv Date: Fri, 14 Apr 2023 01:54:45 -0400 Message-Id: <20230414055449.4028284-7-jsnow@redhat.com> In-Reply-To: <20230414055449.4028284-1-jsnow@redhat.com> References: <20230414055449.4028284-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, T_SPF_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch is meant to ensure that console entry point scripts will always generate on Python 3.7 installations where we may not have access to importlib.metadata. By running it from a separate process *inside* the venv, we can be assured to have access to setuptools and by extension pkg_resources as a fallback. As a coincidence, it also gives us a pretty good place to do all kinds of other "in-venv" setup that we might want to do later; for instance meson and sphinx bootstrapping might be good candidates. Signed-off-by: John Snow --- python/scripts/mkvenv.py | 71 ++++++++++++++++++++++++++++++++-------- 1 file changed, 57 insertions(+), 14 deletions(-) diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py index a3284e9ef1..317697a953 100644 --- a/python/scripts/mkvenv.py +++ b/python/scripts/mkvenv.py @@ -62,22 +62,33 @@ def __init__(self, *args, **kwargs) -> None: self.script_packages = kwargs.pop('script_packages', ()) super().__init__(*args, **kwargs) - # The EnvBuilder class is cute and toggles this setting off - # before post_setup, but we need it to decide if we want to - # generate shims or not. - self._system_site_packages = self.system_site_packages + # Make the context available post-creation: + self._context: Optional[SimpleNamespace] = None + + def ensure_directories(self, env_dir: DirType) -> SimpleNamespace: + logger.debug("ensure_directories(env_dir=%s)", env_dir) + self._context = super().ensure_directories(env_dir) + return self._context + + def create(self, env_dir: DirType) -> None: + logger.debug("create(env_dir=%s)", env_dir) + super().create(env_dir) + self.post_post_setup(self._context) def post_setup(self, context: SimpleNamespace) -> None: logger.debug("post_setup(...)") - - # Generate console_script entry points for system packages: - if self._system_site_packages: - generate_console_scripts( - context.env_exe, context.bin_path, self.script_packages) - # print the python executable to stdout for configure. print(context.env_exe) + def post_post_setup(self, context: SimpleNamespace) -> None: + # The final, final hook; enter the venv and run any + # last steps we want to occur *inside* the venv. + args = [context.env_exe, __file__, 'post_init', + '--binpath', context.bin_path] + if self.system_site_packages: + args += ['--gen', ",".join(self.script_packages)] + subprocess.run(args, check=True) + def need_ensurepip() -> bool: """ @@ -301,6 +312,10 @@ def generate_console_scripts(python_path: str, bin_path: str, """ Generate script shims for console_script entry points in @packages. """ + logger.debug( + "generate_console_scripts(python_path=%s, bin_path=%s, packages=%s)", + python_path, bin_path, packages) + if not packages: return @@ -334,6 +349,16 @@ def _get_entry_points() -> Iterator[Dict[str, str]]: logger.debug("wrote '%s'", script_path) +def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None: + """ + This is intended to be run *inside the venv* after it is created. + """ + python_path = sys.executable + logger.debug("post_venv_setup(bin_path=%s, packages=%s)", + bin_path, packages) + generate_console_scripts(python_path, bin_path, packages) + + def main() -> int: """CLI interface to make_qemu_venv. See module docstring.""" if os.environ.get('DEBUG') or os.environ.get('GITLAB_CI'): @@ -366,12 +391,28 @@ def main() -> int: help="Target directory to install virtual environment into.", ) + subparser = subparsers.add_parser( + 'post_init', help='post-venv initialization') + subparser.add_argument( + '--gen', + type=str, + action='append', + help="Regenerate console_scripts for given packages, if found.", + ) + subparser.add_argument( + '--binpath', + type=str, + action='store', + help="Path where console script shims should be generated", + ) + args = parser.parse_args() + script_packages = [] + for element in args.gen or (): + script_packages.extend(element.split(",")) + try: if args.command == 'create': - script_packages = [] - for element in args.gen or (): - script_packages.extend(element.split(",")) make_venv( args.target, system_site_packages=True, @@ -380,7 +421,9 @@ def main() -> int: with_pip=None, # Autodetermine script_packages=script_packages, ) - logger.debug("mkvenv.py create - exiting") + if args.command == 'post_init': + post_venv_setup(args.binpath, script_packages) + logger.debug("mkvenv.py %s: exiting", args.command) except Ouch as exc: print("\n*** Ouch! ***\n", file=sys.stderr) print(str(exc), "\n\n", file=sys.stderr) From patchwork Fri Apr 14 05:54:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13210969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A30E6C77B70 for ; Fri, 14 Apr 2023 05:56:31 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pnCP4-0007Ph-C2; Fri, 14 Apr 2023 01:55:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOy-0007OZ-Ul for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:17 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOk-0000JB-7F for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681451698; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U5H8/uHXs7YBOGUYE/CWcIe1725RoVciWQKk9ZWDMi8=; b=LaVxqvZx1MBEa2WREYavc2LGUV5mnLF9YdIvCzZ0TI4MiB3zS3pMpFJPum4N7vlWSFTgjS UAX635b2Lvr6IMSXVWbOFKFJGgPL+uFueBPabuwZXc9/iw1yToO8iXYJT/YZ9Uphbd86QQ +94Ue+5/HK4PG1aZz4pp74OnOCUypls= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-449-KzR5LbaXMPC3ND56hAyc1g-1; Fri, 14 Apr 2023 01:54:55 -0400 X-MC-Unique: KzR5LbaXMPC3ND56hAyc1g-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A1842101A552; Fri, 14 Apr 2023 05:54:54 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.197]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1FAC0C1602A; Fri, 14 Apr 2023 05:54:54 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Cleber Rosa , Ani Sinha , Warner Losh , =?utf-8?q?Alex_Benn=C3=A9e?= , Reinoud Zandijk , Thomas Huth , Paolo Bonzini , John Snow , Ryo ONODERA , Kyle Evans , Beraldo Leal , "Michael S. Tsirkin" , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= Subject: [RFC PATCH v2 07/10] mkvenv: work around broken pip installations on Debian 10 Date: Fri, 14 Apr 2023 01:54:46 -0400 Message-Id: <20230414055449.4028284-8-jsnow@redhat.com> In-Reply-To: <20230414055449.4028284-1-jsnow@redhat.com> References: <20230414055449.4028284-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This is a workaround intended for Debian 10, where the debian-patched pip does not function correctly if accessed from within a virtual environment. We don't support Debian 10 any longer, but it's possible that this bug might appear on other derivative platforms and this workaround may prove useful. RFC, a note from Paolo: "BTW, another way to repair Debian 10's pip is to create a symbolic link to sys.base_prefix + '/share/python-wheels' in sys.prefix + '/share/python-wheels'. Since this is much faster, perhaps it can be done unconditionally [...] ?" I was slightly apprehensive about this as it felt "hackier", but it is indeed a lot less code and much faster. It's probably low-risk. Should we do that instead, or should we just scrap any fix at all under the premise that Debian 10 support is dropped anyway? Signed-off-by: John Snow --- python/scripts/mkvenv.py | 61 +++++++++++++++++++++++++++++----------- 1 file changed, 44 insertions(+), 17 deletions(-) diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py index 317697a953..6e1a979355 100644 --- a/python/scripts/mkvenv.py +++ b/python/scripts/mkvenv.py @@ -103,27 +103,25 @@ def need_ensurepip() -> bool: return True -def check_ensurepip(with_pip: bool) -> None: +def check_ensurepip(prefix: str = '', suggest_remedy: bool = False) -> None: """ Check that we have ensurepip. Raise a fatal exception with a helpful hint if it isn't available. """ - if not with_pip: - return - if not find_spec("ensurepip"): msg = ("Python's ensurepip module is not found.\n" "It's normally part of the Python standard library, " "maybe your distribution packages it separately?\n" - "Either install ensurepip, or alleviate the need for it in the " - "first place by installing pip and setuptools for " - f"'{sys.executable}'.\n" - - "(Hint: Debian puts ensurepip in its python3-venv package.)") - raise Ouch(msg) + "(Debian puts ensurepip in its python3-venv package.)\n") + if suggest_remedy: + msg += ( + "Either install ensurepip, or alleviate the need for it in the " + "first place by installing pip and setuptools for " + f"'{sys.executable}'.\n") + raise Ouch(prefix + msg) # ensurepip uses pyexpat, which can also go missing on us: if not find_spec("pyexpat"): @@ -132,12 +130,13 @@ def check_ensurepip(with_pip: bool) -> None: "It's normally part of the Python standard library, " "maybe your distribution packages it separately?\n" - "Either install pyexpat, or alleviate the need for it in the " - "first place by installing pip and setuptools for " - f"'{sys.executable}'.\n\n" - - "(Hint: NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)") - raise Ouch(msg) + "(NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)\n") + if suggest_remedy: + msg += ( + "Either install pyexpat, or alleviate the need for it in the " + "first place by installing pip and setuptools for " + f"'{sys.executable}'.\n") + raise Ouch(prefix + msg) def make_venv( # pylint: disable=too-many-arguments @@ -169,7 +168,8 @@ def make_venv( # pylint: disable=too-many-arguments with_pip = True if not system_site_packages else need_ensurepip() logger.debug("with_pip unset, choosing %s", with_pip) - check_ensurepip(with_pip) + if with_pip: + check_ensurepip(suggest_remedy=True) if symlinks is None: # Default behavior of standard venv CLI @@ -349,6 +349,31 @@ def _get_entry_points() -> Iterator[Dict[str, str]]: logger.debug("wrote '%s'", script_path) +def checkpip(): + """ + Debian10 has a pip that's broken when used inside of a virtual environment. + + We try to detect and correct that case here. + """ + try: + import pip._internal + logger.debug("pip appears to be working correctly.") + return + except ModuleNotFoundError as exc: + if exc.name == 'pip._internal': + # Uh, fair enough. They did say "internal". + # Let's just assume it's fine. + return + logger.warning("pip appears to be malfunctioning: %s", str(exc)) + + check_ensurepip("pip appears to be non-functional, and ") + + logging.debug("Attempting to repair pip ...") + subprocess.run((sys.executable, '-m', 'ensurepip'), + stdout=subprocess.DEVNULL, check=True) + logging.debug("Pip is now (hopefully) repaired!") + + def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None: """ This is intended to be run *inside the venv* after it is created. @@ -358,6 +383,8 @@ def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None: bin_path, packages) generate_console_scripts(python_path, bin_path, packages) + # Test for a broken pip (Debian 10 or derivative?) and fix it if needed + checkpip() def main() -> int: """CLI interface to make_qemu_venv. See module docstring.""" From patchwork Fri Apr 14 05:54:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13210963 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 06C70C77B77 for ; Fri, 14 Apr 2023 05:55:39 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pnCOw-0007MX-T6; Fri, 14 Apr 2023 01:55:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOu-0007Lx-MZ for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:12 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOi-0000JS-Sb for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681451700; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+obdMsDY+2vz54b4K1bwW0I3Mj/CWs+VDCV48iPziCI=; b=QwuXEFv1C2wl+8Doqzrova+P8b1g8Ef3sT87yjk323umlJFozWUeW8OfYmMwMvsylUpB9X sg0pWt/2RjnGyvlxN7iy2NPAKY/oZrC5khwvoXgazfO5+7MCCgvNG67I5Xh/ngaVX2oAYs k/4ElkAQAWEtpznbROEY5X4rjCM/OEY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-418-w6WVnIK-PuOfrEJgVMpCBw-1; Fri, 14 Apr 2023 01:54:55 -0400 X-MC-Unique: w6WVnIK-PuOfrEJgVMpCBw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3DAA08996F1; Fri, 14 Apr 2023 05:54:55 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.197]) by smtp.corp.redhat.com (Postfix) with ESMTP id AF413C1602A; Fri, 14 Apr 2023 05:54:54 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Cleber Rosa , Ani Sinha , Warner Losh , =?utf-8?q?Alex_Benn=C3=A9e?= , Reinoud Zandijk , Thomas Huth , Paolo Bonzini , John Snow , Ryo ONODERA , Kyle Evans , Beraldo Leal , "Michael S. Tsirkin" , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= Subject: [RFC PATCH v2 08/10] configure: create a python venv unconditionally Date: Fri, 14 Apr 2023 01:54:47 -0400 Message-Id: <20230414055449.4028284-9-jsnow@redhat.com> In-Reply-To: <20230414055449.4028284-1-jsnow@redhat.com> References: <20230414055449.4028284-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, T_SPF_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch changes the configure script so that it always creates a python virtual environment unconditionally. Meson bootstrapping is temporarily altered to force the use of meson from git or vendored source. (A patch later in this series restores the use of a distribution-provided Meson.) Signed-off-by: John Snow --- configure | 34 +++++++++++++++++++++++++++++----- 1 file changed, 29 insertions(+), 5 deletions(-) diff --git a/configure b/configure index 800b5850f4..03278fd891 100755 --- a/configure +++ b/configure @@ -608,7 +608,6 @@ check_py_version() { python= first_python= if test -z "${PYTHON}"; then - explicit_python=no # A bare 'python' is traditionally python 2.x, but some distros # have it as python 3.x, so check in both places. for binary in python3 python python3.11 python3.10 python3.9 python3.8 python3.7 python3.6; do @@ -627,7 +626,6 @@ else # Same as above, but only check the environment variable. has "${PYTHON}" || error_exit "The PYTHON environment variable does not point to an executable" python=$(command -v "$PYTHON") - explicit_python=yes if check_py_version "$python"; then # This one is good. first_python= @@ -719,7 +717,7 @@ for opt do ;; --install=*) ;; - --python=*) python="$optarg" ; explicit_python=yes + --python=*) python="$optarg" ;; --skip-meson) skip_meson=yes ;; @@ -1079,8 +1077,34 @@ if ! check_py_version "$python"; then "Use --python=/path/to/python to specify a supported Python." fi -# Resolve PATH + suppress writing compiled files -python="$(command -v "$python") -B" +# Resolve PATH +python="$(command -v "$python")" +explicit_python=yes + +# Create a Python virtual environment using our configured python. +# The stdout of this script will be the location of a symlink that +# points to the configured Python. +# Entry point scripts for pip, meson, and sphinx are generated if those +# packages are present. + +# Defaults assumed for now: +# - venv is cleared if it exists already; +# - venv is allowed to use system packages; +# - all setup is performed **offline**; +# - No packages are installed by default; +# - pip is not installed into the venv when possible, +# but ensurepip is called as a fallback when necessary. + +echo "python determined to be '$python'" +echo "python version: $($python --version)" + +python="$($python -B "${source_path}/python/scripts/mkvenv.py" create --gen pip,meson,sphinx pyvenv)" +if test "$?" -ne 0 ; then + error_exit "python venv creation failed" +fi + +# Suppress writing compiled files +python="$python -B" has_meson() { local python_dir=$(dirname "$python") From patchwork Fri Apr 14 05:54:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13210968 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4A909C77B6E for ; Fri, 14 Apr 2023 05:56:25 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pnCP6-0007Qf-OP; Fri, 14 Apr 2023 01:55:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOz-0007Od-Li for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:18 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOk-0000Ju-94 for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681451700; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Hoi18Z99d/5Y09VoN24QIWuhDsmKwXiy9nDSUT3ER8U=; b=h3y9dD+WxsWdVnkZ8vkgdCZQ98MFqn2L/uEBFf6zHB5krYj5YNbknoyAF4xXlxCIxDH0Cg aG+zo5KaQmm634khJO3b74GFVpu9UXmfuPK0zej/ceZSfek2T/xQFYRBh0NMixPfqwSt3L /2UFUW2hX6TH0YOXSndqu7Z0tKTL/NE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-434-iLZK1QPdNiWhTof4_J0xag-1; Fri, 14 Apr 2023 01:54:56 -0400 X-MC-Unique: iLZK1QPdNiWhTof4_J0xag-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D2C58101A54F; Fri, 14 Apr 2023 05:54:55 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.197]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4DA75C1602A; Fri, 14 Apr 2023 05:54:55 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Cleber Rosa , Ani Sinha , Warner Losh , =?utf-8?q?Alex_Benn=C3=A9e?= , Reinoud Zandijk , Thomas Huth , Paolo Bonzini , John Snow , Ryo ONODERA , Kyle Evans , Beraldo Leal , "Michael S. Tsirkin" , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= Subject: [RFC PATCH v2 09/10] configure: remove --meson=; install meson to the pyvenv Date: Fri, 14 Apr 2023 01:54:48 -0400 Message-Id: <20230414055449.4028284-10-jsnow@redhat.com> In-Reply-To: <20230414055449.4028284-1-jsnow@redhat.com> References: <20230414055449.4028284-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch changes how we detect and install meson. The previous patch creates a lightweight Python virtual environment unconditionally using the user's configured $python that inherits system packages. If Meson is installed there and meets our minimum version requirements, we will use that Meson. In the event that Meson is installed but *not for the chosen Python interpreter*, not found, or of insufficient version, we will attempt to install Meson from source into the newly created Python virtual environment. At present, the source is obtained in the same manner as it has been; preferring git submodules first and relying on vendored source as a backup. This patch (as of now) does *not* connect to PyPI and will work offline for all supported build platforms. As a result of this patch, the Python interpreter we use for both our own build scripts *and* Meson extensions are always known to be the exact same Python. As a further benefit, there will also be a symlink available in the build directory that points to the correct, configured python and can be used by e.g. manual tests to invoke the correct, configured Python unambiguously. Notes: - The meson git submodule can be removed in favor of just loading meson from PyPI; but we probably want a configure flag to toggle online/offline behavior. (What do we want the default to be? Online is my guess.) - Installing meson from the source tree for vendored cases (tarball releases) can be replaced by vendoring a .whl file instead. This will remove some of the `--no-use-pep517` hackiness and alleviates the requirement that users install the python3 'wheel' dependency. - Most of this logic can be moved into mkvenv.py, *especially* if the meson git submodule is removed. Signed-off-by: John Snow --- configure | 104 +++++++++++++++------------- .gitlab-ci.d/buildtest-template.yml | 4 +- 2 files changed, 59 insertions(+), 49 deletions(-) diff --git a/configure b/configure index 03278fd891..c8f73d4ff7 100755 --- a/configure +++ b/configure @@ -721,8 +721,6 @@ for opt do ;; --skip-meson) skip_meson=yes ;; - --meson=*) meson="$optarg" - ;; --ninja=*) ninja="$optarg" ;; --smbd=*) smbd="$optarg" @@ -1006,7 +1004,6 @@ Advanced options (experts only): --cross-prefix-ARCH=PREFIX cross compiler prefix when building ARCH guest test cases --make=MAKE use specified make [$make] --python=PYTHON use specified python [$python] - --meson=MESON use specified meson [$meson] --ninja=NINJA use specified ninja [$ninja] --smbd=SMBD use specified smbd [$smbd] --with-git=GIT use specified git [$git] @@ -1079,7 +1076,8 @@ fi # Resolve PATH python="$(command -v "$python")" -explicit_python=yes +# This variable is intended to be used only for error messages: +target_python=$python # Create a Python virtual environment using our configured python. # The stdout of this script will be the location of a symlink that @@ -1106,58 +1104,70 @@ fi # Suppress writing compiled files python="$python -B" -has_meson() { - local python_dir=$(dirname "$python") - # PEP405: pyvenv.cfg is either adjacent to the Python executable - # or one directory above - if test -f $python_dir/pyvenv.cfg || test -f $python_dir/../pyvenv.cfg; then - # Ensure that Meson and Python come from the same virtual environment - test -x "$python_dir/meson" && - test "$(command -v meson)" -ef "$python_dir/meson" - else - has meson - fi +pip_install() { + $python -m pip install -v \ + --disable-pip-version-check \ + --no-index \ + "$@" } -if test -z "$meson"; then - if test "$explicit_python" = no && has_meson && version_ge "$(meson --version)" 0.61.5; then - meson=meson - elif test "$git_submodules_action" != 'ignore' ; then - meson=git - elif test -e "${source_path}/meson/meson.py" ; then - meson=internal +# OK, let's have some fun! + +# This install command is intended to either fail or be a NOP; +# because we're offline, it's just a convenient version check. +if ! pip_install 'meson>=0.61.5'; then + # Either we don't have Meson, or our Meson is too old. + # (Future revisions of this patchset can be less chatty.) + if test -e pyvenv/bin/meson; then + echo "Meson in pyvenv is too old: $(pyvenv/bin/meson --version)" + elif has meson ; then + echo "Meson was found installed on your system," \ + "but not for the configured Python interpreter ($target_python)." + echo "(Hint: check '$(which meson)' to see which interpreter its shebang uses.)" else - if test "$explicit_python" = yes; then - error_exit "--python requires using QEMU's embedded Meson distribution, but it was not found." - else - error_exit "Meson not found. Use --meson=/path/to/meson" + echo "Meson was not found." + fi + + # OK, but can we fix it, though? :~) + if test "$git_submodules_action" != 'ignore' ; then + git_submodules="${git_submodules} meson" + echo "Attempting to install meson from git submodule ..." + # Stolen from later in the configure file. + # Is it a problem if we front-load this now and run it again later? + if ! (GIT="$git" "$source_path/scripts/git-submodule.sh" "$git_submodules_action" "$git_submodules"); then + exit 1 fi + elif test -e "${source_path}/meson/setup.cfg" ; then + echo "Attempting to install meson from vendored source ..." + else + # In the future, we could use PyPI as a source if the user allows it. + # For now, you're outta luck! + error_exit "A suitable version of Meson was not found." fi -else - # Meson uses its own Python interpreter to invoke other Python scripts, - # but the user wants to use the one they specified with --python. + + # If we're here, we have the meson source and we can attempt to + # install it into our venv. + + # We want to install meson with --no-use-pep517 if possible, + # because it avoids needing a 'wheel' dependency. Old versions + # of pip do this by default, so test for the behavior. # - # We do not want to override the distro Python interpreter (and sometimes - # cannot: for example in Homebrew /usr/bin/meson is a bash script), so - # just require --meson=git|internal together with --python. - if test "$explicit_python" = yes; then - case "$meson" in - git | internal) ;; - *) error_exit "--python requires using QEMU's embedded Meson distribution." ;; - esac + # --no-build-isolation was added to pip 10.0. + # --no-use-pep517 was added ... sometime after 18.1? + pip_flags='--no-build-isolation' + if $python -m pip install --help | grep 'no-use-pep517' > /dev/null 2>&1 ; then + pip_flags="${pip_flags} --no-use-pep517" + fi + if ! pip_install $pip_flags "${source_path}/meson" ; then + exit 1 fi fi -if test "$meson" = git; then - git_submodules="${git_submodules} meson" -fi - -case "$meson" in - git | internal) - meson="$python ${source_path}/meson/meson.py" - ;; - *) meson=$(command -v "$meson") ;; -esac +# At this point, we expect Meson to be installed and available. +# We expect mkvenv or pip to have created pyvenv/bin/meson for us. +# We ignore PATH completely here: we want to use the venv's Meson +# *exclusively*. +meson="$(cd pyvenv/bin; pwd)/meson" # Probe for ninja diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml index a6cfe9be97..7edb50b760 100644 --- a/.gitlab-ci.d/buildtest-template.yml +++ b/.gitlab-ci.d/buildtest-template.yml @@ -12,12 +12,12 @@ - mkdir build - cd build - ../configure --enable-werror --disable-docs --enable-fdt=system - ${LD_JOBS:+--meson=git} ${TARGETS:+--target-list="$TARGETS"} + ${TARGETS:+--target-list="$TARGETS"} $CONFIGURE_ARGS || { cat config.log meson-logs/meson-log.txt && exit 1; } - if test -n "$LD_JOBS"; then - ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ; + pyvenv/bin/meson configure . -Dbackend_max_links="$LD_JOBS" ; fi || exit 1; - make -j"$JOBS" - if test -n "$MAKE_CHECK_ARGS"; From patchwork Fri Apr 14 05:54:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13210965 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 59FDDC77B7A for ; Fri, 14 Apr 2023 05:55:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pnCP9-0007RF-Ed; Fri, 14 Apr 2023 01:55:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCP3-0007Px-Pk for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:22 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnCOj-0000Jl-4q for qemu-devel@nongnu.org; Fri, 14 Apr 2023 01:55:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681451700; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B+10s9fHZ2Rv7aCFmhbMY0RNnrVjbvpFyStf3lJffaU=; b=g/vRgrS41jaiEdiGVA0liA/G0cAMJ72MkYbTJskFq2YD7s9ZusJjnpij+snbyGz6WKTjwW 2zqXLmkT3zfx92pcYFO7MYtMkR3DuJGqZ56BIc+AnWMuSJWs7H+yYLHnvvsvZQiYRUZtDu 6x6QOr+RZzDGN1QHuXncuxM2XQAmgvc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-639-xYrR37V9PW6_S5Vt1Q2EVg-1; Fri, 14 Apr 2023 01:54:56 -0400 X-MC-Unique: xYrR37V9PW6_S5Vt1Q2EVg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6D2A285A5A3; Fri, 14 Apr 2023 05:54:56 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.197]) by smtp.corp.redhat.com (Postfix) with ESMTP id E0132C1602A; Fri, 14 Apr 2023 05:54:55 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Cleber Rosa , Ani Sinha , Warner Losh , =?utf-8?q?Alex_Benn=C3=A9e?= , Reinoud Zandijk , Thomas Huth , Paolo Bonzini , John Snow , Ryo ONODERA , Kyle Evans , Beraldo Leal , "Michael S. Tsirkin" , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= Subject: [RFC PATCH v2 10/10] tests: Use configure-provided pyvenv for tests Date: Fri, 14 Apr 2023 01:54:49 -0400 Message-Id: <20230414055449.4028284-11-jsnow@redhat.com> In-Reply-To: <20230414055449.4028284-1-jsnow@redhat.com> References: <20230414055449.4028284-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch changes how the avocado tests are provided, ever so slightly. Instead of creating a new testing venv, use instead the configure-provided 'pyvenv' and install optional packages into that. Note: At the time of writing, avocado tests require avocado-framework < 90 whereas the qemu.qmp self-tests rely on avocado-framework >= 90. This collision is avoided for now because the qemu.git/python/qemu/ code does not need avocado at *runtime*; it does not install avocado-framework as a necessary dependency and is skipped in this circumstance. Nevertheless, we do want to address that discrepancy in the future so that it will be possible to re-use the same venv for qemu.git/python/qemu self-tests to introduce them to make check as "make check-python". Signed-off-by: John Snow --- docs/devel/acpi-bits.rst | 6 +++--- docs/devel/testing.rst | 14 +++++++------- .gitlab-ci.d/buildtest.yml | 6 +++--- scripts/ci/org.centos/stream/8/x86_64/test-avocado | 4 ++-- scripts/device-crash-test | 2 +- tests/Makefile.include | 10 +++++----- tests/requirements.txt | 7 +++++-- 7 files changed, 26 insertions(+), 23 deletions(-) diff --git a/docs/devel/acpi-bits.rst b/docs/devel/acpi-bits.rst index 9eb4b9e666..0c40359109 100644 --- a/docs/devel/acpi-bits.rst +++ b/docs/devel/acpi-bits.rst @@ -61,19 +61,19 @@ Under ``tests/avocado/`` as the root we have: :: $ make check-venv (needed only the first time to create the venv) - $ ./tests/venv/bin/avocado run -t acpi tests/avocado + $ ./pyvenv/bin/avocado run -t acpi tests/avocado The above will run all acpi avocado tests including this one. In order to run the individual tests, perform the following: :: - $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py --tap - + $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py --tap - The above will produce output in tap format. You can omit "--tap -" in the end and it will produce output like the following: :: - $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py + $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits JOB ID : eab225724da7b64c012c65705dc2fa14ab1defef JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-10T17.58-eab2257/job.log diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 4071e72710..50664d9eb9 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -882,9 +882,9 @@ You can run the avocado tests simply by executing: make check-avocado -This involves the automatic creation of Python virtual environment -within the build tree (at ``tests/venv``) which will have all the -right dependencies, and will save tests results also within the +This involves the automatic installation, from PyPI, of all the +necessary avocado-framework dependencies into the QEMU venv within the +build tree (at ``./pyvenv``). Test results are also saved within the build tree (at ``tests/results``). Note: the build environment must be using a Python 3 stack, and have @@ -941,7 +941,7 @@ may be invoked by running: .. code:: - tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/ + pyvenv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/ Note that if ``make check-avocado`` was not executed before, it is possible to create the Python virtual environment with the dependencies @@ -956,20 +956,20 @@ a test file. To run tests from a single file within the build tree, use: .. code:: - tests/venv/bin/avocado run tests/avocado/$TESTFILE + pyvenv/bin/avocado run tests/avocado/$TESTFILE To run a single test within a test file, use: .. code:: - tests/venv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME + pyvenv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME Valid test names are visible in the output from any previous execution of Avocado or ``make check-avocado``, and can also be queried using: .. code:: - tests/venv/bin/avocado list tests/avocado + pyvenv/bin/avocado list tests/avocado Manual Installation ~~~~~~~~~~~~~~~~~~~ diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml index ba6f551752..53de9f23c4 100644 --- a/.gitlab-ci.d/buildtest.yml +++ b/.gitlab-ci.d/buildtest.yml @@ -103,7 +103,7 @@ crash-test-debian: script: - cd build - make check-venv - - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-i386 + - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-i386 build-system-fedora: extends: @@ -146,8 +146,8 @@ crash-test-fedora: script: - cd build - make check-venv - - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc - - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32 + - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc + - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32 build-system-centos: extends: diff --git a/scripts/ci/org.centos/stream/8/x86_64/test-avocado b/scripts/ci/org.centos/stream/8/x86_64/test-avocado index d2c0e5fb4c..7bb5b317b6 100755 --- a/scripts/ci/org.centos/stream/8/x86_64/test-avocado +++ b/scripts/ci/org.centos/stream/8/x86_64/test-avocado @@ -4,7 +4,7 @@ # KVM and x86_64, or tests that are generic enough to be valid for all # targets. Such a test list can be generated with: # -# ./tests/venv/bin/avocado list --filter-by-tags-include-empty \ +# ./pyvenv/bin/avocado list --filter-by-tags-include-empty \ # --filter-by-tags-include-empty-key -t accel:kvm,arch:x86_64 \ # tests/avocado/ # @@ -22,7 +22,7 @@ # - tests/avocado/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types # make get-vm-images -./tests/venv/bin/avocado run \ +./pyvenv/bin/avocado run \ --job-results-dir=tests/results/ \ tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_i440fx_kvm \ tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_kvm \ diff --git a/scripts/device-crash-test b/scripts/device-crash-test index 73bcb98693..ef6ac262fa 100755 --- a/scripts/device-crash-test +++ b/scripts/device-crash-test @@ -43,7 +43,7 @@ except ModuleNotFoundError as exc: print(f"Module '{exc.name}' not found.") print(" Try 'make check-venv' from your build directory,") print(" and then one way to run this script is like so:") - print(f' > $builddir/tests/venv/bin/python3 "{path}"') + print(f' > $builddir/pyvenv/bin/python3 "{path}"') sys.exit(1) logger = logging.getLogger('device-crash-test') diff --git a/tests/Makefile.include b/tests/Makefile.include index 9422ddaece..3f39c607cc 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -89,7 +89,8 @@ distclean-tcg: $(DISTCLEAN_TCG_TARGET_RULES) # Build up our target list from the filtered list of ninja targets TARGETS=$(patsubst libqemu-%.fa, %, $(filter libqemu-%.fa, $(ninja-targets))) -TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv +TESTS_VENV_DIR=$(BUILD_DIR)/pyvenv +TESTS_VENV_TOKEN=$(BUILD_DIR)/pyvenv/tests.group TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results TESTS_PYTHON=$(TESTS_VENV_DIR)/bin/python3 @@ -111,8 +112,7 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \ $(TESTS_PYTHON) -m pip -q --disable-pip-version-check $1, \ "VENVPIP","$1") -$(TESTS_VENV_DIR): $(TESTS_VENV_REQ) - $(call quiet-command, $(PYTHON) -m venv $@, VENV, $@) +$(TESTS_VENV_TOKEN): $(TESTS_VENV_REQ) $(call quiet-venv-pip,install -e "$(SRC_PATH)/python/") $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) $(call quiet-command, touch $@) @@ -121,7 +121,7 @@ $(TESTS_RESULTS_DIR): $(call quiet-command, mkdir -p $@, \ MKDIR, $@) -check-venv: $(TESTS_VENV_DIR) +check-venv: $(TESTS_VENV_TOKEN) FEDORA_31_ARCHES_TARGETS=$(patsubst %-softmmu,%, $(filter %-softmmu,$(TARGETS))) FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(FEDORA_31_ARCHES_TARGETS)) @@ -163,7 +163,7 @@ check: check-build: run-ninja check-clean: - rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR) + rm -rf $(TESTS_RESULTS_DIR) clean: check-clean clean-tcg distclean: distclean-tcg diff --git a/tests/requirements.txt b/tests/requirements.txt index 0ba561b6bd..07e713ef5a 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -1,6 +1,9 @@ # Add Python module requirements, one per line, to be installed -# in the tests/venv Python virtual environment. For more info, +# in the qemu build_dir/pyvenv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 -# Note that qemu.git/python/ is always implicitly installed. +# +# Note that qemu.git/python/ is implicitly installed to this venv when +# 'make check-venv' is run, and will persist until configure is run +# again. avocado-framework==88.1 pycdlib==1.11.0