From patchwork Mon Apr 24 20:02:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222558 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7AE0EC7618E for ; Mon, 24 Apr 2023 20:07:34 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2P0-0007tt-8o; Mon, 24 Apr 2023 16:03:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Oy-0007rb-N9 for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:08 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ok-0000wD-Pc for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366572; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lfFnDxcdLCgtQJTniq4KqKqcC9fRHBIEUY7g2eTE2yw=; b=RJ8t6LJ9Y6t9e+8gDHHQkPFsLmAOskrO289UrTHwnhfhtb0CMQcl9pLQ0dlAbEpvv8jlPI iUmmh45pGH5TfqXXjTuGWJqBMLcWvZvznge17jC/u2c1fuYsK00j4zbm3vAOX8tI2OBv0/ EomHSwQWLlE7fqqUdozhzLf6nP5Y9q4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-454-36ZQnfN5NO2-5gs7DjhJlQ-1; Mon, 24 Apr 2023 16:02:50 -0400 X-MC-Unique: 36ZQnfN5NO2-5gs7DjhJlQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5808688B7A6; Mon, 24 Apr 2023 20:02:50 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id A30B01121318; Mon, 24 Apr 2023 20:02:49 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 01/20] python: update pylint configuration Date: Mon, 24 Apr 2023 16:02:29 -0400 Message-Id: <20230424200248.1183394-2-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Pylint 2.17.x decided that SocketAddrT was a bad name for a Type Alias for some reason. Sure, fine, whatever. Signed-off-by: John Snow Reviewed-by: Daniel P. Berrangé --- python/setup.cfg | 1 + 1 file changed, 1 insertion(+) diff --git a/python/setup.cfg b/python/setup.cfg index 9e923d9762..3f36502954 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -132,6 +132,7 @@ good-names=i, fd, # fd = os.open(...) c, # for c in string: ... T, # for TypeVars. See pylint#3401 + SocketAddrT, # Not sure why this is invalid. [pylint.similarities] # Ignore imports when computing similarities. From patchwork Mon Apr 24 20:02:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222554 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 63DF9C77B61 for ; Mon, 24 Apr 2023 20:05:28 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Ou-0007m0-I1; Mon, 24 Apr 2023 16:03:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Oq-0007eG-Vx for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:01 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Om-0000wS-RK for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:02:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366576; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ogLTBXOFFGQ3qnUk+eH1NMMDnmmeCsYawgY2haKip6Y=; b=abqjCigvVQq0GN/FtnTCOFg5TDU75s2i9mW4W4g+Ig5pbhXCDbcKcvcnCpeJNJ8mbBRgI5 LxCYvj8u+tFxaPCiJ7ujml/p5qZ0loZuqvLHF1pNSuKdh2ka7GnAZGbFGpGgzwY/DffgkF h4RlDyuYl6JL44uBAm8uy8mgc+GYhkA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-3-xwIBJvlLOsi-CsQDVStu2Q-1; Mon, 24 Apr 2023 16:02:51 -0400 X-MC-Unique: xwIBJvlLOsi-CsQDVStu2Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 18E8BA0F382; Mon, 24 Apr 2023 20:02:51 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 659221121318; Mon, 24 Apr 2023 20:02:50 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 02/20] python: add mkvenv.py Date: Mon, 24 Apr 2023 16:02:30 -0400 Message-Id: <20230424200248.1183394-3-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This script will be responsible for building a lightweight Python virtual environment at configure time. It works with Python 3.7 or newer. It has been designed to: - work *offline*, no PyPI required. - work *quickly*, The fast path is only ~65ms on my machine. - work *robustly*, with multiple fallbacks to keep things working. - work *cooperatively*, using system packages where possible. (You can use your distro's meson, no problem.) Due to its unique position in the build chain, it exists outside of the installable python packages in-tree and *must* be runnable without any third party dependencies. Under normal circumstances, the only dependency required to execute this script is Python 3.7+ itself. The script is *faster* by several seconds when setuptools and pip are installed in the host environment, which is probably the case for a typical multi-purpose developer workstation. In the event that pip/setuptools are missing or not usable, additional dependencies may be required on some distributions which remove certain Python stdlib modules to package them separately: - Debian may require python3-venv to provide "ensurepip" - NetBSD may require py310-expat to provide "pyexpat" * (* Or whichever version is current for NetBSD.) Signed-off-by: John Snow --- python/scripts/mkvenv.py | 277 +++++++++++++++++++++++++++++++++++++++ python/setup.cfg | 9 ++ python/tests/flake8.sh | 1 + python/tests/isort.sh | 1 + python/tests/mypy.sh | 1 + python/tests/pylint.sh | 1 + 6 files changed, 290 insertions(+) create mode 100644 python/scripts/mkvenv.py diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py new file mode 100644 index 0000000000..1dfcc0198a --- /dev/null +++ b/python/scripts/mkvenv.py @@ -0,0 +1,277 @@ +""" +mkvenv - QEMU pyvenv bootstrapping utility + +usage: mkvenv [-h] command ... + +QEMU pyvenv bootstrapping utility + +options: + -h, --help show this help message and exit + +Commands: + command Description + create create a venv + +-------------------------------------------------- + +usage: mkvenv create [-h] target + +positional arguments: + target Target directory to install virtual environment into. + +options: + -h, --help show this help message and exit + +""" + +# Copyright (C) 2022-2023 Red Hat, Inc. +# +# Authors: +# John Snow +# Paolo Bonzini +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import argparse +from importlib.util import find_spec +import logging +import os +from pathlib import Path +import subprocess +import sys +import traceback +from types import SimpleNamespace +from typing import Any, Optional, Union +import venv + + +# Do not add any mandatory dependencies from outside the stdlib: +# This script *must* be usable standalone! + +logger = logging.getLogger("mkvenv") + + +class Ouch(RuntimeError): + """An Exception class we can't confuse with a builtin.""" + + +class QemuEnvBuilder(venv.EnvBuilder): + """ + An extension of venv.EnvBuilder for building QEMU's configure-time venv. + + As of this commit, it does not yet do anything particularly + different than the standard venv-creation utility. The next several + commits will gradually change that in small commits that highlight + each feature individually. + + Parameters for base class init: + - system_site_packages: bool = False + - clear: bool = False + - symlinks: bool = False + - upgrade: bool = False + - with_pip: bool = False + - prompt: Optional[str] = None + - upgrade_deps: bool = False (Since 3.9) + """ + + def __init__(self, *args: Any, **kwargs: Any) -> None: + logger.debug("QemuEnvBuilder.__init__(...)") + super().__init__(*args, **kwargs) + + # The EnvBuilder class is cute and toggles this setting off + # before post_setup, but we need it to decide if we want to + # generate shims or not. + self._system_site_packages = self.system_site_packages + + def post_setup(self, context: SimpleNamespace) -> None: + logger.debug("post_setup(...)") + + # print the python executable to stdout for configure. + print(context.env_exe) + + +def need_ensurepip() -> bool: + """ + Tests for the presence of setuptools and pip. + + :return: `True` if we do not detect both packages. + """ + # Don't try to actually import them, it's fraught with danger: + # https://github.com/pypa/setuptools/issues/2993 + if find_spec("setuptools") and find_spec("pip"): + return False + return True + + +def check_ensurepip(with_pip: bool) -> None: + """ + Check that we have ensurepip. + + Raise a fatal exception with a helpful hint if it isn't available. + """ + if with_pip and not find_spec("ensurepip"): + msg = ( + "Python's ensurepip module is not found.\n" + "It's normally part of the Python standard library, " + "maybe your distribution packages it separately?\n" + "Either install ensurepip, or alleviate the need for it in the " + "first place by installing pip and setuptools for " + f"'{sys.executable}'.\n" + "(Hint: Debian puts ensurepip in its python3-venv package.)" + ) + raise Ouch(msg) + + +def make_venv( # pylint: disable=too-many-arguments + env_dir: Union[str, Path], + system_site_packages: bool = False, + clear: bool = True, + symlinks: Optional[bool] = None, + with_pip: Optional[bool] = None, +) -> None: + """ + Create a venv using `QemuEnvBuilder`. + + This is analogous to the `venv.create` module-level convenience + function that is part of the Python stdblib, except it uses + `QemuEnvBuilder` instead. + + :param env_dir: The directory to create/install to. + :param system_site_packages: + Allow inheriting packages from the system installation. + :param clear: When True, fully remove any prior venv and files. + :param symlinks: + Whether to use symlinks to the target interpreter or not. If + left unspecified, it will use symlinks except on Windows to + match behavior with the "venv" CLI tool. + :param with_pip: + Whether to run "ensurepip" or not. If unspecified, this will + default to False if system_site_packages is True and a usable + version of pip is found. + """ + logging.debug( + "%s: make_venv(env_dir=%s, system_site_packages=%s, " + "clear=%s, symlinks=%s, with_pip=%s)", + __file__, + str(env_dir), + system_site_packages, + clear, + symlinks, + with_pip, + ) + + print(f"MKVENV {str(env_dir)}", file=sys.stderr) + + # ensurepip is slow: venv creation can be very fast for cases where + # we allow the use of system_site_packages. Toggle ensure_pip on only + # in the cases where we really need it. + if with_pip is None: + with_pip = True if not system_site_packages else need_ensurepip() + logger.debug("with_pip unset, choosing %s", with_pip) + + check_ensurepip(with_pip) + + if symlinks is None: + # Default behavior of standard venv CLI + symlinks = os.name != "nt" + + builder = QemuEnvBuilder( + system_site_packages=system_site_packages, + clear=clear, + symlinks=symlinks, + with_pip=with_pip, + ) + try: + logger.debug("Invoking builder.create()") + try: + builder.create(str(env_dir)) + except SystemExit as exc: + # Some versions of the venv module raise SystemExit; *nasty*! + # We want the exception that prompted it. It might be a subprocess + # error that has output we *really* want to see. + logger.debug("Intercepted SystemExit from EnvBuilder.create()") + raise exc.__cause__ or exc.__context__ or exc + logger.debug("builder.create() finished") + except subprocess.CalledProcessError as exc: + print(f"cmd: {exc.cmd}", file=sys.stderr) + print(f"returncode: {exc.returncode}", file=sys.stderr) + + def _stringify(data: Optional[Union[str, bytes]]) -> Optional[str]: + if isinstance(data, bytes): + return data.decode() + return data + + if exc.stdout: + print( + "========== stdout ==========", + _stringify(exc.stdout), + "============================", + sep="\n", + file=sys.stderr, + ) + if exc.stderr: + print( + "========== stderr ==========", + _stringify(exc.stderr), + "============================", + sep="\n", + file=sys.stderr, + ) + raise Ouch("VENV creation subprocess failed.") from exc + + +def _add_create_subcommand(subparsers: Any) -> None: + subparser = subparsers.add_parser("create", help="create a venv") + subparser.add_argument( + "target", + type=str, + action="store", + help="Target directory to install virtual environment into.", + ) + + +def main() -> int: + """CLI interface to make_qemu_venv. See module docstring.""" + if os.environ.get("DEBUG") or os.environ.get("GITLAB_CI"): + # You're welcome. + logging.basicConfig(level=logging.DEBUG) + elif os.environ.get("V"): + logging.basicConfig(level=logging.INFO) + + parser = argparse.ArgumentParser( + prog="mkvenv", + description="QEMU pyvenv bootstrapping utility", + ) + subparsers = parser.add_subparsers( + title="Commands", + dest="command", + required=True, + metavar="command", + help="Description", + ) + + _add_create_subcommand(subparsers) + + args = parser.parse_args() + try: + if args.command == "create": + make_venv( + args.target, + system_site_packages=True, + clear=True, + ) + logger.debug("mkvenv.py %s: exiting", args.command) + except Ouch as exc: + print("\n*** Ouch! ***\n", file=sys.stderr) + print(str(exc), "\n\n", file=sys.stderr) + return 1 + except: # pylint: disable=bare-except + print("mkvenv did not complete successfully:", file=sys.stderr) + traceback.print_exc() + return 2 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/python/setup.cfg b/python/setup.cfg index 3f36502954..5b25f810fa 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -103,6 +103,15 @@ ignore_missing_imports = True [mypy-pygments] ignore_missing_imports = True +[mypy-importlib.metadata] +ignore_missing_imports = True + +[mypy-importlib_metadata] +ignore_missing_imports = True + +[mypy-pkg_resources] +ignore_missing_imports = True + [pylint.messages control] # Disable the message, report, category or checker with the given id(s). You # can either give multiple identifiers separated by comma (,) or put this diff --git a/python/tests/flake8.sh b/python/tests/flake8.sh index 1cd7d40fad..e013699645 100755 --- a/python/tests/flake8.sh +++ b/python/tests/flake8.sh @@ -1,2 +1,3 @@ #!/bin/sh -e python3 -m flake8 qemu/ +python3 -m flake8 scripts/ diff --git a/python/tests/isort.sh b/python/tests/isort.sh index 4480405bfb..66c2f7df0f 100755 --- a/python/tests/isort.sh +++ b/python/tests/isort.sh @@ -1,2 +1,3 @@ #!/bin/sh -e python3 -m isort -c qemu/ +python3 -m isort -c scripts/ diff --git a/python/tests/mypy.sh b/python/tests/mypy.sh index 5f980f563b..a33a3f58ab 100755 --- a/python/tests/mypy.sh +++ b/python/tests/mypy.sh @@ -1,2 +1,3 @@ #!/bin/sh -e python3 -m mypy -p qemu +python3 -m mypy scripts/ diff --git a/python/tests/pylint.sh b/python/tests/pylint.sh index 03d64705a1..2b68da90df 100755 --- a/python/tests/pylint.sh +++ b/python/tests/pylint.sh @@ -1,3 +1,4 @@ #!/bin/sh -e # See commit message for environment variable explainer. SETUPTOOLS_USE_DISTUTILS=stdlib python3 -m pylint qemu/ +SETUPTOOLS_USE_DISTUTILS=stdlib python3 -m pylint scripts/ From patchwork Mon Apr 24 20:02:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1D22DC7618E for ; Mon, 24 Apr 2023 20:09:31 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Os-0007ij-BN; Mon, 24 Apr 2023 16:03:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Op-0007dg-IY for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:00 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Om-0000wO-M6 for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:02:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366575; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AYaJSVHUp8hRHnw2+FO+zlIVYCp3jgnfsdKnX8ybISM=; b=S9WPgTKtyzHTxGazmpEETnxeQbLmoFv5Mr54/9t+sq1mU8tIZUCIOLGbRnDpv3eubJ97OJ w7xzD45EkjOfLcZ0smZ+34nTy1c2Q9KO8TgAsd9xNtMgeCoawWTUuQyjncRcNxAYODLsG0 oFk+UB6DQpHxUZxXK7QpxovBZtP5R2M= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-567-kd1mArEMNaS9vHUqLnexcQ-1; Mon, 24 Apr 2023 16:02:52 -0400 X-MC-Unique: kd1mArEMNaS9vHUqLnexcQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CB48687B2A2; Mon, 24 Apr 2023 20:02:51 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 251FA1121318; Mon, 24 Apr 2023 20:02:51 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 03/20] mkvenv: add console script entry point generation Date: Mon, 24 Apr 2023 16:02:31 -0400 Message-Id: <20230424200248.1183394-4-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org When creating a virtual environment that inherits system packages, script entry points (like "meson", "sphinx-build", etc) are not re-generated with the correct shebang. When you are *inside* of the venv, this is not a problem, but if you are *outside* of it, you will not have a script that engages the virtual environment appropriately. Add a mechanism that generates new entry points for pre-existing packages so that we can use these scripts to run "meson", "sphinx-build", "pip", unambiguously inside the venv. Signed-off-by: John Snow --- python/scripts/mkvenv.py | 179 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 172 insertions(+), 7 deletions(-) diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py index 1dfcc0198a..f355cb54fb 100644 --- a/python/scripts/mkvenv.py +++ b/python/scripts/mkvenv.py @@ -14,13 +14,14 @@ -------------------------------------------------- -usage: mkvenv create [-h] target +usage: mkvenv create [-h] [--gen GEN] target positional arguments: target Target directory to install virtual environment into. options: -h, --help show this help message and exit + --gen GEN Regenerate console_scripts for given packages, if found. """ @@ -38,11 +39,20 @@ import logging import os from pathlib import Path +import re +import stat import subprocess import sys import traceback from types import SimpleNamespace -from typing import Any, Optional, Union +from typing import ( + Any, + Dict, + Iterator, + Optional, + Sequence, + Union, +) import venv @@ -60,10 +70,9 @@ class QemuEnvBuilder(venv.EnvBuilder): """ An extension of venv.EnvBuilder for building QEMU's configure-time venv. - As of this commit, it does not yet do anything particularly - different than the standard venv-creation utility. The next several - commits will gradually change that in small commits that highlight - each feature individually. + The only functional change is that it adds the ability to regenerate + console_script shims for packages available via system_site + packages. Parameters for base class init: - system_site_packages: bool = False @@ -77,6 +86,7 @@ class QemuEnvBuilder(venv.EnvBuilder): def __init__(self, *args: Any, **kwargs: Any) -> None: logger.debug("QemuEnvBuilder.__init__(...)") + self.script_packages = kwargs.pop("script_packages", ()) super().__init__(*args, **kwargs) # The EnvBuilder class is cute and toggles this setting off @@ -87,6 +97,12 @@ def __init__(self, *args: Any, **kwargs: Any) -> None: def post_setup(self, context: SimpleNamespace) -> None: logger.debug("post_setup(...)") + # Generate console_script entry points for system packages: + if self._system_site_packages: + generate_console_scripts( + context.env_exe, context.bin_path, self.script_packages + ) + # print the python executable to stdout for configure. print(context.env_exe) @@ -129,6 +145,7 @@ def make_venv( # pylint: disable=too-many-arguments clear: bool = True, symlinks: Optional[bool] = None, with_pip: Optional[bool] = None, + script_packages: Sequence[str] = (), ) -> None: """ Create a venv using `QemuEnvBuilder`. @@ -149,16 +166,20 @@ def make_venv( # pylint: disable=too-many-arguments Whether to run "ensurepip" or not. If unspecified, this will default to False if system_site_packages is True and a usable version of pip is found. + :param script_packages: + A sequence of package names to generate console entry point + shims for, when system_site_packages is True. """ logging.debug( "%s: make_venv(env_dir=%s, system_site_packages=%s, " - "clear=%s, symlinks=%s, with_pip=%s)", + "clear=%s, symlinks=%s, with_pip=%s, script_packages=%s)", __file__, str(env_dir), system_site_packages, clear, symlinks, with_pip, + script_packages, ) print(f"MKVENV {str(env_dir)}", file=sys.stderr) @@ -181,6 +202,7 @@ def make_venv( # pylint: disable=too-many-arguments clear=clear, symlinks=symlinks, with_pip=with_pip, + script_packages=script_packages, ) try: logger.debug("Invoking builder.create()") @@ -221,8 +243,147 @@ def _stringify(data: Optional[Union[str, bytes]]) -> Optional[str]: raise Ouch("VENV creation subprocess failed.") from exc +def _gen_importlib(packages: Sequence[str]) -> Iterator[Dict[str, str]]: + # pylint: disable=import-outside-toplevel + try: + # First preference: Python 3.8+ stdlib + from importlib.metadata import ( + PackageNotFoundError, + distribution, + ) + except ImportError as exc: + logger.debug("%s", str(exc)) + # Second preference: Commonly available PyPI backport + from importlib_metadata import ( + PackageNotFoundError, + distribution, + ) + + # Borrowed from CPython (Lib/importlib/metadata/__init__.py) + pattern = re.compile( + r"(?P[\w.]+)\s*" + r"(:\s*(?P[\w.]+)\s*)?" + r"((?P\[.*\])\s*)?$" + ) + + def _generator() -> Iterator[Dict[str, str]]: + for package in packages: + try: + entry_points = distribution(package).entry_points + except PackageNotFoundError: + continue + + # The EntryPoints type is only available in 3.10+, + # treat this as a vanilla list and filter it ourselves. + entry_points = filter( + lambda ep: ep.group == "console_scripts", entry_points + ) + + for entry_point in entry_points: + # Python 3.8 doesn't have 'module' or 'attr' attributes + if not ( + hasattr(entry_point, "module") + and hasattr(entry_point, "attr") + ): + match = pattern.match(entry_point.value) + assert match is not None + module = match.group("module") + attr = match.group("attr") + else: + module = entry_point.module + attr = entry_point.attr + yield { + "name": entry_point.name, + "module": module, + "import_name": attr, + "func": attr, + } + + return _generator() + + +def _gen_pkg_resources(packages: Sequence[str]) -> Iterator[Dict[str, str]]: + # pylint: disable=import-outside-toplevel + # Bundled with setuptools; has a good chance of being available. + import pkg_resources + + def _generator() -> Iterator[Dict[str, str]]: + for package in packages: + try: + eps = pkg_resources.get_entry_map(package, "console_scripts") + except pkg_resources.DistributionNotFound: + continue + + for entry_point in eps.values(): + yield { + "name": entry_point.name, + "module": entry_point.module_name, + "import_name": ".".join(entry_point.attrs), + "func": ".".join(entry_point.attrs), + } + + return _generator() + + +# Borrowed/adapted from pip's vendored version of distutils: +SCRIPT_TEMPLATE = r"""#!{python_path:s} +# -*- coding: utf-8 -*- +import re +import sys +from {module:s} import {import_name:s} +if __name__ == '__main__': + sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) + sys.exit({func:s}()) +""" + + +def generate_console_scripts( + python_path: str, bin_path: str, packages: Sequence[str] +) -> None: + """ + Generate script shims for console_script entry points in @packages. + """ + if not packages: + return + + def _get_entry_points() -> Iterator[Dict[str, str]]: + """Python 3.7 compatibility shim for iterating entry points.""" + # Python 3.8+, or Python 3.7 with importlib_metadata installed. + try: + return _gen_importlib(packages) + except ImportError as exc: + logger.debug("%s", str(exc)) + + # Python 3.7 with setuptools installed. + try: + return _gen_pkg_resources(packages) + except ImportError as exc: + logger.debug("%s", str(exc)) + raise Ouch( + "Neither importlib.metadata nor pkg_resources found, " + "can't generate console script shims.\n" + "Use Python 3.8+, or install importlib-metadata or setuptools." + ) from exc + + for entry_point in _get_entry_points(): + script_path = os.path.join(bin_path, entry_point["name"]) + script = SCRIPT_TEMPLATE.format(python_path=python_path, **entry_point) + with open(script_path, "w", encoding="UTF-8") as file: + file.write(script) + mode = os.stat(script_path).st_mode | stat.S_IEXEC + os.chmod(script_path, mode) + + logger.debug("wrote '%s'", script_path) + + def _add_create_subcommand(subparsers: Any) -> None: subparser = subparsers.add_parser("create", help="create a venv") + subparser.add_argument( + "--gen", + type=str, + action="append", + help="Regenerate console_scripts for given packages, if found.", + ) subparser.add_argument( "target", type=str, @@ -256,10 +417,14 @@ def main() -> int: args = parser.parse_args() try: if args.command == "create": + script_packages = [] + for element in args.gen or (): + script_packages.extend(element.split(",")) make_venv( args.target, system_site_packages=True, clear=True, + script_packages=script_packages, ) logger.debug("mkvenv.py %s: exiting", args.command) except Ouch as exc: From patchwork Mon Apr 24 20:02:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BB227C77B76 for ; Mon, 24 Apr 2023 20:04:13 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Os-0007ii-0P; Mon, 24 Apr 2023 16:03:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Op-0007cw-7r for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:02:59 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2On-0000wa-CR for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:02:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366576; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ybu6c13xqrBGEvV5edArV3Tyo0lwbd+W+IFGF/gprEM=; b=YjvSoGdcugPtKqUPB3z3nwzSwAVPY2DUTZ9pxzB+N1g+I8VlvQ9JCB9JDinTNfdxzrrQHH QhwUGMenPp2a8KXSj1/kf6bUtLdZX24BNj6HnPWW8iR5H+wDwJ0+vvMMuspMokF7rYQFk5 A4gbAJIpj5sGdfgElfkEc07UVnOoCCQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-119-5lKJAV2aMV-k9dHJEJHHpQ-1; Mon, 24 Apr 2023 16:02:53 -0400 X-MC-Unique: 5lKJAV2aMV-k9dHJEJHHpQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8B9DC857FB3; Mon, 24 Apr 2023 20:02:52 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id D6E7A1121318; Mon, 24 Apr 2023 20:02:51 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 04/20] mkvenv: Add better error message for missing pyexpat module Date: Mon, 24 Apr 2023 16:02:32 -0400 Message-Id: <20230424200248.1183394-5-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org NetBSD debundles pyexpat from python, but ensurepip needs pyexpat. Try our best to offer a helpful error message instead of just failing catastrophically. Signed-off-by: John Snow --- python/scripts/mkvenv.py | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py index f355cb54fb..2172774403 100644 --- a/python/scripts/mkvenv.py +++ b/python/scripts/mkvenv.py @@ -126,7 +126,10 @@ def check_ensurepip(with_pip: bool) -> None: Raise a fatal exception with a helpful hint if it isn't available. """ - if with_pip and not find_spec("ensurepip"): + if not with_pip: + return + + if not find_spec("ensurepip"): msg = ( "Python's ensurepip module is not found.\n" "It's normally part of the Python standard library, " @@ -138,6 +141,19 @@ def check_ensurepip(with_pip: bool) -> None: ) raise Ouch(msg) + # ensurepip uses pyexpat, which can also go missing on us: + if not find_spec("pyexpat"): + msg = ( + "Python's pyexpat module is not found.\n" + "It's normally part of the Python standard library, " + "maybe your distribution packages it separately?\n" + "Either install pyexpat, or alleviate the need for it in the " + "first place by installing pip and setuptools for " + f"'{sys.executable}'.\n\n" + "(Hint: NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)" + ) + raise Ouch(msg) + def make_venv( # pylint: disable=too-many-arguments env_dir: Union[str, Path], From patchwork Mon Apr 24 20:02:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B5E3DC77B61 for ; Mon, 24 Apr 2023 20:04:13 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Op-0007ci-7J; Mon, 24 Apr 2023 16:02:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2On-0007Zk-PW for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:02:57 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Om-0000wJ-4B for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:02:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366575; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zrQG5EG4WH92khExZ1+Awaq414GPQVReIfZciIMh9fo=; b=ZWWY2fLow4jC6IDg/ob/SHD0y3Bkw7Mo/ZB/wMJHMNWr264Z3erIGC6GUqLlITEcRlxYj0 3inahq1PdzcOeFAs0xKg1dZg+EFzU5+9JRGSRAGTqXpjm+B7g4Ln/ea7kv5x3+8xXPjBQz 25vwKAz9nkN4Srxa4NF0qYvEeS7hemQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-510-7RqXZyvYPWeFs6BWN6UXXA-1; Mon, 24 Apr 2023 16:02:54 -0400 X-MC-Unique: 7RqXZyvYPWeFs6BWN6UXXA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4918AA0F392; Mon, 24 Apr 2023 20:02:53 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 97B591121319; Mon, 24 Apr 2023 20:02:52 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 05/20] mkvenv: generate console entry shims from inside the venv Date: Mon, 24 Apr 2023 16:02:33 -0400 Message-Id: <20230424200248.1183394-6-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch is meant to ensure that console entry point scripts will always generate on Python 3.7 installations where we may not have access to importlib.metadata. By running it from a separate process *inside* the venv, we can be assured to have access to setuptools and by extension pkg_resources as a fallback. It isn't strictly necessary to do this for Python 3.8+, which will always have importlib.metadata available. Signed-off-by: John Snow --- python/scripts/mkvenv.py | 99 ++++++++++++++++++++++++++++++++++------ 1 file changed, 85 insertions(+), 14 deletions(-) diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py index 2172774403..4daa652f12 100644 --- a/python/scripts/mkvenv.py +++ b/python/scripts/mkvenv.py @@ -11,6 +11,8 @@ Commands: command Description create create a venv + post_init + post-venv initialization -------------------------------------------------- @@ -23,6 +25,15 @@ -h, --help show this help message and exit --gen GEN Regenerate console_scripts for given packages, if found. +-------------------------------------------------- + +usage: mkvenv post_init [-h] [--gen GEN] [--binpath BINPATH] + +options: + -h, --help show this help message and exit + --gen GEN Regenerate console_scripts for given packages, if found. + --binpath BINPATH Path where console script shims should be generated + """ # Copyright (C) 2022-2023 Red Hat, Inc. @@ -59,6 +70,7 @@ # Do not add any mandatory dependencies from outside the stdlib: # This script *must* be usable standalone! +DirType = Union[str, bytes, "os.PathLike[str]", "os.PathLike[bytes]"] logger = logging.getLogger("mkvenv") @@ -89,23 +101,42 @@ def __init__(self, *args: Any, **kwargs: Any) -> None: self.script_packages = kwargs.pop("script_packages", ()) super().__init__(*args, **kwargs) - # The EnvBuilder class is cute and toggles this setting off - # before post_setup, but we need it to decide if we want to - # generate shims or not. - self._system_site_packages = self.system_site_packages + # Make the context available post-creation: + self._context: Optional[SimpleNamespace] = None + + def ensure_directories(self, env_dir: DirType) -> SimpleNamespace: + logger.debug("ensure_directories(env_dir=%s)", env_dir) + self._context = super().ensure_directories(env_dir) + return self._context + + def create(self, env_dir: DirType) -> None: + logger.debug("create(env_dir=%s)", env_dir) + super().create(env_dir) + assert self._context is not None + self.post_post_setup(self._context) def post_setup(self, context: SimpleNamespace) -> None: logger.debug("post_setup(...)") - - # Generate console_script entry points for system packages: - if self._system_site_packages: - generate_console_scripts( - context.env_exe, context.bin_path, self.script_packages - ) - # print the python executable to stdout for configure. print(context.env_exe) + def post_post_setup(self, context: SimpleNamespace) -> None: + """ + The final, final hook. Enter the venv and run commands inside of it. + """ + args = [ + context.env_exe, + __file__, + "post_init", + "--binpath", + context.bin_path, + ] + if self.system_site_packages: + scripts = ",".join(self.script_packages) + if scripts: + args += ["--gen", scripts] + subprocess.run(args, check=True) + def need_ensurepip() -> bool: """ @@ -359,6 +390,13 @@ def generate_console_scripts( """ Generate script shims for console_script entry points in @packages. """ + logger.debug( + "generate_console_scripts(python_path=%s, bin_path=%s, packages=%s)", + python_path, + bin_path, + packages, + ) + if not packages: return @@ -392,6 +430,17 @@ def _get_entry_points() -> Iterator[Dict[str, str]]: logger.debug("wrote '%s'", script_path) +def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None: + """ + This is intended to be run *inside the venv* after it is created. + """ + python_path = sys.executable + logger.debug( + "post_venv_setup(bin_path=%s, packages=%s)", bin_path, packages + ) + generate_console_scripts(python_path, bin_path, packages) + + def _add_create_subcommand(subparsers: Any) -> None: subparser = subparsers.add_parser("create", help="create a venv") subparser.add_argument( @@ -408,6 +457,24 @@ def _add_create_subcommand(subparsers: Any) -> None: ) +def _add_post_init_subcommand(subparsers: Any) -> None: + subparser = subparsers.add_parser( + "post_init", help="post-venv initialization" + ) + subparser.add_argument( + "--gen", + type=str, + action="append", + help="Regenerate console_scripts for given packages, if found.", + ) + subparser.add_argument( + "--binpath", + type=str, + action="store", + help="Path where console script shims should be generated", + ) + + def main() -> int: """CLI interface to make_qemu_venv. See module docstring.""" if os.environ.get("DEBUG") or os.environ.get("GITLAB_CI"): @@ -429,19 +496,23 @@ def main() -> int: ) _add_create_subcommand(subparsers) + _add_post_init_subcommand(subparsers) args = parser.parse_args() + script_packages = [] + for element in args.gen or (): + script_packages.extend(element.split(",")) + try: if args.command == "create": - script_packages = [] - for element in args.gen or (): - script_packages.extend(element.split(",")) make_venv( args.target, system_site_packages=True, clear=True, script_packages=script_packages, ) + if args.command == "post_init": + post_venv_setup(args.binpath, script_packages) logger.debug("mkvenv.py %s: exiting", args.command) except Ouch as exc: print("\n*** Ouch! ***\n", file=sys.stderr) From patchwork Mon Apr 24 20:02:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222556 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C724BC7618E for ; Mon, 24 Apr 2023 20:05:59 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Ot-0007lJ-QJ; Mon, 24 Apr 2023 16:03:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Oq-0007eQ-RC for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:00 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Oo-0000wo-Lr for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366578; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vMhulmk5OML72JeUi2RGWnSylGPWl5bnlwsN1SoB3nM=; b=XadXVMWJntvB12QbvDzjehsFWrEdj0n+/N/koNCxNk4QlCrvCawAsWa8/FX+QjtI1PP1E9 p5peq7ZfVOZ+mjP0cQKwUSJD2fXmd5/4/JB81VXsJG5Qan6fGkZn+kO822kpfAZ+jotN/G 3/XjGo8udN0zgG8UxbW7LOhyBIXbIkk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-86-uYsexOANN2mSB2g90lY01g-1; Mon, 24 Apr 2023 16:02:54 -0400 X-MC-Unique: uYsexOANN2mSB2g90lY01g-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 145F8185A78F; Mon, 24 Apr 2023 20:02:54 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 58CB91121319; Mon, 24 Apr 2023 20:02:53 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 06/20] mkvenv: work around broken pip installations on Debian 10 Date: Mon, 24 Apr 2023 16:02:34 -0400 Message-Id: <20230424200248.1183394-7-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This is a workaround intended for Debian 10, where the debian-patched pip does not function correctly if accessed from within a virtual environment. We don't support Debian 10 any longer, but it's possible that this bug might appear on other derivative platforms and this workaround may prove useful. RFC, a note from Paolo: "BTW, another way to repair Debian 10's pip is to create a symbolic link to sys.base_prefix + '/share/python-wheels' in sys.prefix + '/share/python-wheels'. Since this is much faster, perhaps it can be done unconditionally [...] ?" I was slightly apprehensive about this as it felt "hackier", but it is indeed a lot less code and much faster. It's probably low-risk. Should we do that instead, or should we just scrap any fix at all under the premise that Debian 10 support is dropped anyway? Signed-off-by: John Snow --- python/scripts/mkvenv.py | 67 +++++++++++++++++++++++++++++++--------- 1 file changed, 52 insertions(+), 15 deletions(-) diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py index 4daa652f12..445f4eb092 100644 --- a/python/scripts/mkvenv.py +++ b/python/scripts/mkvenv.py @@ -151,26 +151,26 @@ def need_ensurepip() -> bool: return True -def check_ensurepip(with_pip: bool) -> None: +def check_ensurepip(prefix: str = "", suggest_remedy: bool = False) -> None: """ Check that we have ensurepip. Raise a fatal exception with a helpful hint if it isn't available. """ - if not with_pip: - return - if not find_spec("ensurepip"): msg = ( "Python's ensurepip module is not found.\n" "It's normally part of the Python standard library, " "maybe your distribution packages it separately?\n" - "Either install ensurepip, or alleviate the need for it in the " - "first place by installing pip and setuptools for " - f"'{sys.executable}'.\n" - "(Hint: Debian puts ensurepip in its python3-venv package.)" + "(Debian puts ensurepip in its python3-venv package.)\n" ) - raise Ouch(msg) + if suggest_remedy: + msg += ( + "Either install ensurepip, or alleviate the need for it in the" + " first place by installing pip and setuptools for " + f"'{sys.executable}'.\n" + ) + raise Ouch(prefix + msg) # ensurepip uses pyexpat, which can also go missing on us: if not find_spec("pyexpat"): @@ -178,12 +178,15 @@ def check_ensurepip(with_pip: bool) -> None: "Python's pyexpat module is not found.\n" "It's normally part of the Python standard library, " "maybe your distribution packages it separately?\n" - "Either install pyexpat, or alleviate the need for it in the " - "first place by installing pip and setuptools for " - f"'{sys.executable}'.\n\n" - "(Hint: NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)" + "(NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)\n" ) - raise Ouch(msg) + if suggest_remedy: + msg += ( + "Either install pyexpat, or alleviate the need for it in the " + "first place by installing pip and setuptools for " + f"'{sys.executable}'.\n" + ) + raise Ouch(prefix + msg) def make_venv( # pylint: disable=too-many-arguments @@ -238,7 +241,8 @@ def make_venv( # pylint: disable=too-many-arguments with_pip = True if not system_site_packages else need_ensurepip() logger.debug("with_pip unset, choosing %s", with_pip) - check_ensurepip(with_pip) + if with_pip: + check_ensurepip(suggest_remedy=True) if symlinks is None: # Default behavior of standard venv CLI @@ -430,6 +434,36 @@ def _get_entry_points() -> Iterator[Dict[str, str]]: logger.debug("wrote '%s'", script_path) +def checkpip() -> None: + """ + Debian10 has a pip that's broken when used inside of a virtual environment. + + We try to detect and correct that case here. + """ + try: + # pylint: disable=import-outside-toplevel, unused-import + import pip._internal # noqa: F401 + + logger.debug("pip appears to be working correctly.") + return + except ModuleNotFoundError as exc: + if exc.name == "pip._internal": + # Uh, fair enough. They did say "internal". + # Let's just assume it's fine. + return + logger.warning("pip appears to be malfunctioning: %s", str(exc)) + + check_ensurepip("pip appears to be non-functional, and ") + + logging.debug("Attempting to repair pip ...") + subprocess.run( + (sys.executable, "-m", "ensurepip"), + stdout=subprocess.DEVNULL, + check=True, + ) + logging.debug("Pip is now (hopefully) repaired!") + + def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None: """ This is intended to be run *inside the venv* after it is created. @@ -440,6 +474,9 @@ def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None: ) generate_console_scripts(python_path, bin_path, packages) + # Test for a broken pip (Debian 10 or derivative?) and fix it if needed + checkpip() + def _add_create_subcommand(subparsers: Any) -> None: subparser = subparsers.add_parser("create", help="create a venv") From patchwork Mon Apr 24 20:02:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222562 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0BFD9C7618E for ; Mon, 24 Apr 2023 20:09:17 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Ov-0007nV-AC; Mon, 24 Apr 2023 16:03:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Os-0007jX-EY for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:03 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Oq-0000wv-As for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366578; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kOo7dSyIG8DYno+AxpA5II1Cc5cRU+Bp4ZgeSQSSi6c=; b=bXN0+YXESOmW2d/xC3hTa/4d6OaXfdgno/aFC9cTCqDWg9KzqKaODpvKfvsGALYW9Ooztz 4A2/33KFlMeaSrKQP4hiX546mQdP3M+2/xkCguRw2ajhmtmvrF3EmfxDpdIAdTSybTG3q2 1CWq30P/1HSGAAM0Iy0HtMkkH0ZLLg8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-663-3IVgkhjoN9OBRVC9GlbUdA-1; Mon, 24 Apr 2023 16:02:55 -0400 X-MC-Unique: 3IVgkhjoN9OBRVC9GlbUdA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CB329101A54F; Mon, 24 Apr 2023 20:02:54 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 23E891121318; Mon, 24 Apr 2023 20:02:54 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 07/20] mkvenv: add nested venv workaround Date: Mon, 24 Apr 2023 16:02:35 -0400 Message-Id: <20230424200248.1183394-8-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Python virtual environments do not typically nest; they may inherit from the top-level system packages or not at all. For our purposes, it would be convenient to emulate "nested" virtual environments to allow callers of the configure script to install specific versions of python utilities in order to test build system features, utility version compatibility, etc. While it is possible to install packages into the system environment (say, by using the --user flag), it's nicer to install test packages into a totally isolated environment instead. As detailed in https://www.qemu.org/2023/03/24/python/, Emulate a nested venv environment by using .pth files installed into the site-packages folder that points to the parent environment when appropriate. Signed-off-by: Paolo Bonzini Signed-off-by: John Snow --- python/scripts/mkvenv.py | 72 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 69 insertions(+), 3 deletions(-) diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py index 445f4eb092..45d1b772e5 100644 --- a/python/scripts/mkvenv.py +++ b/python/scripts/mkvenv.py @@ -51,9 +51,11 @@ import os from pathlib import Path import re +import site import stat import subprocess import sys +import sysconfig import traceback from types import SimpleNamespace from typing import ( @@ -74,6 +76,11 @@ logger = logging.getLogger("mkvenv") +def inside_a_venv() -> bool: + """Returns True if it is executed inside of a virtual environment.""" + return sys.prefix != sys.base_prefix + + class Ouch(RuntimeError): """An Exception class we can't confuse with a builtin.""" @@ -82,9 +89,15 @@ class QemuEnvBuilder(venv.EnvBuilder): """ An extension of venv.EnvBuilder for building QEMU's configure-time venv. - The only functional change is that it adds the ability to regenerate - console_script shims for packages available via system_site - packages. + The primary differences are: + + (1) It adds the ability to regenerate console_script shims for + packages available via system_site_packages for any packages + specified by the 'script_packages' argument + + (2) It emulates a "nested" virtual environment when invoked from + inside of an existing virtual environment by including packages from + the parent. Parameters for base class init: - system_site_packages: bool = False @@ -99,11 +112,51 @@ class QemuEnvBuilder(venv.EnvBuilder): def __init__(self, *args: Any, **kwargs: Any) -> None: logger.debug("QemuEnvBuilder.__init__(...)") self.script_packages = kwargs.pop("script_packages", ()) + + # For nested venv emulation: + self.use_parent_packages = False + if inside_a_venv(): + # Include parent packages only if we're in a venv and + # system_site_packages was True. + self.use_parent_packages = kwargs.pop( + "system_site_packages", False + ) + # Include system_site_packages only when the parent, + # The venv we are currently in, also does so. + kwargs["system_site_packages"] = sys.base_prefix in site.PREFIXES + super().__init__(*args, **kwargs) # Make the context available post-creation: self._context: Optional[SimpleNamespace] = None + def compute_venv_libpath(self, context: SimpleNamespace) -> str: + """ + Compatibility wrapper for context.lib_path for Python < 3.12 + """ + # Python 3.12+, not strictly necessary because it's documented + # to be the same as 3.10 code below: + if sys.version_info >= (3, 12): + return context.lib_path + + # Python 3.10+ + if "venv" in sysconfig.get_scheme_names(): + return sysconfig.get_path( + "purelib", scheme="venv", vars={"base": context.env_dir} + ) + + # For Python <= 3.9 we need to hardcode this. Fortunately the + # code below was the same in Python 3.6-3.10, so there is only + # one case. + if sys.platform == "win32": + return os.path.join(context.env_dir, "Lib", "site-packages") + return os.path.join( + context.env_dir, + "lib", + "python%d.%d" % sys.version_info[:2], + "site-packages", + ) + def ensure_directories(self, env_dir: DirType) -> SimpleNamespace: logger.debug("ensure_directories(env_dir=%s)", env_dir) self._context = super().ensure_directories(env_dir) @@ -124,6 +177,19 @@ def post_post_setup(self, context: SimpleNamespace) -> None: """ The final, final hook. Enter the venv and run commands inside of it. """ + if self.use_parent_packages: + # We're inside of a venv and we want to include the parent + # venv's packages. + parent_libpath = sysconfig.get_path("purelib") + logger.debug("parent_libpath: %s", parent_libpath) + + our_libpath = self.compute_venv_libpath(context) + logger.debug("our_libpath: %s", our_libpath) + + pth_file = os.path.join(our_libpath, "nested.pth") + with open(pth_file, "w", encoding="UTF-8") as file: + file.write(parent_libpath + os.linesep) + args = [ context.env_exe, __file__, From patchwork Mon Apr 24 20:02:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222552 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7A639C77B61 for ; Mon, 24 Apr 2023 20:05:16 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Ou-0007lX-0k; Mon, 24 Apr 2023 16:03:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Os-0007jW-Cy for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:02 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Oq-0000xF-Ah for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366579; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q0FGmk7ZrXhx4D40Jtrg77ftnsvWCd0BcH5TudDIxTk=; b=Lb1Gdi4qng+T6mPfHh43OkBCk+HkArNOOiJz7X/2X591x2UWjhBk3iB5zcnW97QkMDV+II IRSdu5FCnLnUtPh1nfMT7Lb9vDrR15DYiSSGTG15lUOLk+Qc4teO5j55KWq28F8L8JK+A2 2ebUQObb1N9pem7fKXevECPi6a0pYNg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-278-6Y6aY-XKP-qSvn_Uy-w0-w-1; Mon, 24 Apr 2023 16:02:56 -0400 X-MC-Unique: 6Y6aY-XKP-qSvn_Uy-w0-w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8DC3185A5A3; Mon, 24 Apr 2023 20:02:55 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id D85131121318; Mon, 24 Apr 2023 20:02:54 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 08/20] mkvenv: add ensure subcommand Date: Mon, 24 Apr 2023 16:02:36 -0400 Message-Id: <20230424200248.1183394-9-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This command is to be used to add various packages (or ensure they're already present) into the configure-provided venv in a modular fashion. Examples: mkvenv ensure --online --dir "${source_dir}/python/wheels/" "meson>=0.61.5" mkvenv ensure --online "sphinx>=1.6.0" mkvenv ensure "qemu.qmp==0.0.2" It's designed to look for packages in three places, in order: (1) In system packages, if the version installed is already good enough. This way your distribution-provided meson, sphinx, etc are always used as first preference. (2) In a vendored packages directory. Here I am suggesting qemu.git/python/wheels/ as that directory. This is intended to serve as a replacement for vendoring the meson source for QEMU tarballs. It is also highly likely to be extremely useful for packaging the "qemu.qmp" package in source distributions for platforms that do not yet package qemu.qmp separately. (3) Online, via PyPI, ***only when "--online" is passed***. This is only ever used as a fallback if the first two sources do not have an appropriate package that meets the requirement. The ability to build QEMU and run tests *completely offline* is not impinged. Signed-off-by: John Snow --- python/scripts/mkvenv.py | 116 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 114 insertions(+), 2 deletions(-) diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py index 45d1b772e5..937664ea9c 100644 --- a/python/scripts/mkvenv.py +++ b/python/scripts/mkvenv.py @@ -13,6 +13,7 @@ create create a venv post_init post-venv initialization + ensure Ensure that the specified package is installed. -------------------------------------------------- @@ -34,6 +35,18 @@ --gen GEN Regenerate console_scripts for given packages, if found. --binpath BINPATH Path where console script shims should be generated +-------------------------------------------------- + +usage: mkvenv ensure [-h] [--online] [--dir DIR] dep_spec + +positional arguments: + dep_spec PEP 508 Dependency specification, e.g. 'meson>=0.61.5' + +options: + -h, --help show this help message and exit + --online Install packages from PyPI, if necessary. + --dir DIR Path to vendored packages where we may install from. + """ # Copyright (C) 2022-2023 Red Hat, Inc. @@ -530,6 +543,71 @@ def checkpip() -> None: logging.debug("Pip is now (hopefully) repaired!") +def pip_install( + args: Sequence[str], + online: bool = False, + wheels_dir: Optional[Union[str, Path]] = None, +) -> None: + """ + Use pip to install a package or package(s) as specified in @args. + """ + loud = bool( + os.environ.get("DEBUG") + or os.environ.get("GITLAB_CI") + or os.environ.get("V") + ) + + full_args = [ + sys.executable, + "-m", + "pip", + "install", + "--disable-pip-version-check", + "-v" if loud else "-q", + ] + if not online: + full_args += ["--no-index"] + if wheels_dir: + full_args += ["--find-links", f"file://{str(wheels_dir)}"] + full_args += list(args) + subprocess.run(full_args, check=True) + + +def ensure( + dep_spec: str, + online: bool = False, + wheels_dir: Optional[Union[str, Path]] = None, +) -> None: + """ + Use pip to ensure we have the package specified by @dep_spec. + + If the package is already installed, do nothing. If online and + wheels_dir are both provided, prefer packages found in wheels_dir + first before connecting to PyPI. + + :param dep_spec: + A PEP 508 dependency specification. e.g. 'meson>=0.61.5'. + :param online: If True, fall back to PyPI. + :param wheels_dir: If specified, search this path for packages. + """ + # This first install command will: + # (A) Do nothing, if we already have a suitable package. + # (B) Install the package from vendored source, if possible. + # (C) Fail if neither A nor B. + try: + pip_install([dep_spec], online=False, wheels_dir=wheels_dir) + # (A) or (B) happened. Success. + return + except subprocess.CalledProcessError: + # (C) Happened. + # The package is missing or isn't a suitable version, + # and we weren't able to install a suitable vendored package. + if online: + pip_install([dep_spec], online=True) + else: + raise + + def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None: """ This is intended to be run *inside the venv* after it is created. @@ -578,6 +656,29 @@ def _add_post_init_subcommand(subparsers: Any) -> None: ) +def _add_ensure_subcommand(subparsers: Any) -> None: + subparser = subparsers.add_parser( + "ensure", help="Ensure that the specified package is installed." + ) + subparser.add_argument( + "--online", + action="store_true", + help="Install packages from PyPI, if necessary.", + ) + subparser.add_argument( + "--dir", + type=str, + action="store", + help="Path to vendored packages where we may install from.", + ) + subparser.add_argument( + "dep_spec", + type=str, + action="store", + help="PEP 508 Dependency specification, e.g. 'meson>=0.61.5'", + ) + + def main() -> int: """CLI interface to make_qemu_venv. See module docstring.""" if os.environ.get("DEBUG") or os.environ.get("GITLAB_CI"): @@ -600,14 +701,18 @@ def main() -> int: _add_create_subcommand(subparsers) _add_post_init_subcommand(subparsers) + _add_ensure_subcommand(subparsers) args = parser.parse_args() script_packages = [] - for element in args.gen or (): - script_packages.extend(element.split(",")) + + def _normalize_gen() -> None: + for element in args.gen or (): + script_packages.extend(element.split(",")) try: if args.command == "create": + _normalize_gen() make_venv( args.target, system_site_packages=True, @@ -615,7 +720,14 @@ def main() -> int: script_packages=script_packages, ) if args.command == "post_init": + _normalize_gen() post_venv_setup(args.binpath, script_packages) + if args.command == "ensure": + ensure( + dep_spec=args.dep_spec, + online=args.online, + wheels_dir=args.dir, + ) logger.debug("mkvenv.py %s: exiting", args.command) except Ouch as exc: print("\n*** Ouch! ***\n", file=sys.stderr) From patchwork Mon Apr 24 20:02:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 65709C77B61 for ; Mon, 24 Apr 2023 20:04:36 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Ow-0007p3-Ig; Mon, 24 Apr 2023 16:03:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ou-0007la-1d for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:04 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Or-0000xU-3g for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366580; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iXu1iBWmPnnbcqJanHFlz0WgmsKQEotu0q7wUfsyNgQ=; b=cu4JsdIaIIp18m/3x3ecNTwMNqPPsx3QEnDUlm+R4bUmJh2YXrfhZjNxHHLRxLp2oNj3Q7 vls6I1dohZkievzb/ts8GDlom3LgRsMwMqZSlmmS1srm9DfI+bDLH5bBfqd5pXVsl0830j HKmG/qUGc5tFx1SJTlnEy2ILu3pX7Lk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-607-0qDOmkryODypTVXjqufBwA-1; Mon, 24 Apr 2023 16:02:57 -0400 X-MC-Unique: 0qDOmkryODypTVXjqufBwA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 510D8811E7C; Mon, 24 Apr 2023 20:02:56 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9ADEE1121318; Mon, 24 Apr 2023 20:02:55 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 09/20] tests/docker: add python3-venv dependency Date: Mon, 24 Apr 2023 16:02:37 -0400 Message-Id: <20230424200248.1183394-10-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Several debian-based tests need the python3-venv dependency as a consequence of Debian debundling the "ensurepip" module normally included with Python. As mkvenv.py stands as of this commit, Debian requires EITHER: (A) setuptools and pip, or (B) ensurepip mkvenv is a few seconds faster if you have setuptools and pip, so developers should prefer the first requirement. For the purposes of CI, the time-save is a wash; it's only a matter of who is responsible for installing pip and when; the timing is about the same. Arbitrarily, I chose adding ensurepip to the test configuration because it is normally part of the Python stdlib, and always having it allows us a more consistent cross-platform environment. Signed-off-by: John Snow Reviewed-by: Daniel P. Berrangé --- tests/docker/dockerfiles/debian-all-test-cross.docker | 3 ++- tests/docker/dockerfiles/debian-hexagon-cross.docker | 3 ++- tests/docker/dockerfiles/debian-riscv64-cross.docker | 3 ++- tests/docker/dockerfiles/debian-tricore-cross.docker | 3 ++- 4 files changed, 8 insertions(+), 4 deletions(-) diff --git a/tests/docker/dockerfiles/debian-all-test-cross.docker b/tests/docker/dockerfiles/debian-all-test-cross.docker index 981e9bdc7b..f9f401544a 100644 --- a/tests/docker/dockerfiles/debian-all-test-cross.docker +++ b/tests/docker/dockerfiles/debian-all-test-cross.docker @@ -57,7 +57,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \ gcc-sh4-linux-gnu \ libc6-dev-sh4-cross \ gcc-sparc64-linux-gnu \ - libc6-dev-sparc64-cross + libc6-dev-sparc64-cross \ + python3-venv ENV QEMU_CONFIGURE_OPTS --disable-system --disable-docs --disable-tools ENV DEF_TARGET_LIST aarch64-linux-user,alpha-linux-user,arm-linux-user,hppa-linux-user,i386-linux-user,m68k-linux-user,mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,ppc-linux-user,ppc64-linux-user,ppc64le-linux-user,riscv64-linux-user,s390x-linux-user,sh4-linux-user,sparc64-linux-user diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker b/tests/docker/dockerfiles/debian-hexagon-cross.docker index b99d99f943..c2cfb6a5d0 100644 --- a/tests/docker/dockerfiles/debian-hexagon-cross.docker +++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker @@ -20,7 +20,8 @@ RUN apt-get update && \ bison \ flex \ git \ - ninja-build && \ + ninja-build \ + python3-venv && \ # Install QEMU build deps for use in CI DEBIAN_FRONTEND=noninteractive eatmydata \ apt build-dep -yy --arch-only qemu diff --git a/tests/docker/dockerfiles/debian-riscv64-cross.docker b/tests/docker/dockerfiles/debian-riscv64-cross.docker index 803afb9573..081404e014 100644 --- a/tests/docker/dockerfiles/debian-riscv64-cross.docker +++ b/tests/docker/dockerfiles/debian-riscv64-cross.docker @@ -28,7 +28,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata apt install -yy \ libglib2.0-dev \ ninja-build \ pkg-config \ - python3 + python3 \ + python3-venv # Add ports and riscv64 architecture RUN echo "deb http://ftp.ports.debian.org/debian-ports/ sid main" >> /etc/apt/sources.list diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker index cfd2faf9a8..269bfa8d42 100644 --- a/tests/docker/dockerfiles/debian-tricore-cross.docker +++ b/tests/docker/dockerfiles/debian-tricore-cross.docker @@ -33,7 +33,8 @@ RUN apt update && \ pkgconf \ python3-pip \ python3-setuptools \ - python3-wheel + python3-wheel \ + python3-venv RUN curl -#SL https://github.com/bkoppelmann/package_940/releases/download/tricore-toolchain-9.40/tricore-toolchain-9.4.0.tar.gz \ | tar -xzC /usr/local/ From patchwork Mon Apr 24 20:02:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222544 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BD1F9C77B7E for ; Mon, 24 Apr 2023 20:04:13 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Ov-0007o5-Vl; Mon, 24 Apr 2023 16:03:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ot-0007kh-Ep for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:03 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Oq-0000x1-BQ for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366579; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ghbh4qRNVDBPRglbq18ow59ZrTEcqWklHbrforGx4nU=; b=E10OH0Iak09ttM8XOA8lvFnmXf+8uR6g9MjcEuAOaF+gJwmPlg0ShD3ZI8axG10H08P8ZU dp8w4t9dgmlJyoqev1Yc1rc1zB558g1ZPvtQOPBp3/Y3aB9RT9Y9jdALi1Lyi8n3V6faaV LlNZwPcoJ3BQtYFkneGBuiIeqlFjqFI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-417-VGoanxCfOT-pTZ5AEtgtoQ-1; Mon, 24 Apr 2023 16:02:57 -0400 X-MC-Unique: VGoanxCfOT-pTZ5AEtgtoQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 125CE101A551; Mon, 24 Apr 2023 20:02:57 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5E2AA1121319; Mon, 24 Apr 2023 20:02:56 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 10/20] tests/vm: Configure netbsd to use Python 3.10 Date: Mon, 24 Apr 2023 16:02:38 -0400 Message-Id: <20230424200248.1183394-11-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org NetBSD removes some packages from the Python stdlib, but only re-packages them for Python 3.10. Switch to using Python 3.10. Signed-off-by: John Snow Reviewed-by: Daniel P. Berrangé --- tests/vm/netbsd | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/vm/netbsd b/tests/vm/netbsd index 0b9536ca17..13eae109c0 100755 --- a/tests/vm/netbsd +++ b/tests/vm/netbsd @@ -30,6 +30,7 @@ class NetBSDVM(basevm.BaseVM): "git-base", "pkgconf", "xz", + "python310", "ninja-build", # gnu tools From patchwork Mon Apr 24 20:02:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222550 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EEEC6C7618E for ; Mon, 24 Apr 2023 20:04:54 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Ox-0007pU-41; Mon, 24 Apr 2023 16:03:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ou-0007m1-Gp for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:04 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ot-0000xx-3n for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366582; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qCN5N1sd5u3Iw2MtpMYHLDxOFq022o0H7fXfF2oJZDM=; b=dtmpzRX3Sr5ZNKMKfHMkmBbW8pn7zZxnoYi/YmqYEA15QoSnhw/icvntG9XAQYvOZROghv VS/IystF+yCAyMeobgFNc8uDpOP0zv7vsT7j8W4KgSAq8dLzOpWaW/qZZLPC6hk3uf2Zus dRG1HT/RNISKFOFHabdMhyFUvOSvLKI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-91-bFncF_WKMG63sIb3_P-yNw-1; Mon, 24 Apr 2023 16:02:58 -0400 X-MC-Unique: bFncF_WKMG63sIb3_P-yNw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C5DA33C0C881; Mon, 24 Apr 2023 20:02:57 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1E5131121319; Mon, 24 Apr 2023 20:02:57 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 11/20] tests/vm: add py310-expat to NetBSD Date: Mon, 24 Apr 2023 16:02:39 -0400 Message-Id: <20230424200248.1183394-12-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org NetBSD cannot successfully run "ensurepip" without access to the pyexpat module, which NetBSD debundles. Like the Debian patch, it would be strictly faster long term to install pip/setuptools, and I recommend developers at their workstations take that approach instead. For the purposes of a throwaway VM, there's not really a speed difference for who is responsible for installing pip; us (needs py310-pip) or Python (needs py310-expat). Signed-off-by: John Snow Reviewed-by: Daniel P. Berrangé --- tests/vm/netbsd | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/vm/netbsd b/tests/vm/netbsd index 13eae109c0..c7e3f1e735 100755 --- a/tests/vm/netbsd +++ b/tests/vm/netbsd @@ -31,6 +31,7 @@ class NetBSDVM(basevm.BaseVM): "pkgconf", "xz", "python310", + "py310-expat", "ninja-build", # gnu tools From patchwork Mon Apr 24 20:02:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222560 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9F6C5C77B61 for ; Mon, 24 Apr 2023 20:08:18 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Oy-0007rW-JU; Mon, 24 Apr 2023 16:03:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ow-0007p8-R4 for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:06 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ov-0000zx-Af for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366584; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wUtFzNljOU7mzy+l7MmVIk6x++rYT/1ibuCHaDggM+s=; b=h0fRg+5ZAghV/jH9wabeU6kk0si9DReFKcykj2ZElJ3plAKDedT8zMDfE0jIe4y2L8+Yup /RpQfTveyUfXGe8ta0S3sYhQpZlsrz8Pa4cKFPb0+Y/VVAY1kBQa+zpHDU9hV2TDueAVnL xe+q9ovBncjaYWzMyCYT2iK88jzd4cU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-49-9Ie0OmxcMlC3uAns0KiVAA-1; Mon, 24 Apr 2023 16:02:59 -0400 X-MC-Unique: 9Ie0OmxcMlC3uAns0KiVAA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8969687B2A6; Mon, 24 Apr 2023 20:02:58 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id D2CE51121318; Mon, 24 Apr 2023 20:02:57 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 12/20] scripts/make-release: download meson==0.61.5 .whl Date: Mon, 24 Apr 2023 16:02:40 -0400 Message-Id: <20230424200248.1183394-13-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org In preference to vendoring meson source, vendor a built distributable. This has two benefits: (1) We can get rid of a git submodule, (2) Installing built meson into a venv doesn't require any extra dependencies. RFC: The alternative approach is to just check in the .whl file into the git tree directly, and have it available for both git and tarball installations. That approach slightly changes the necessity of some subsequent patches, but otherwise either way will work. Owing to how "mkvenv ensure" will prefer vendored files prior to connecting to PyPI, checking in a vendored meson file in this manner means we will generally never use PyPI to install meson ever. ("Vote now on your phones.") Signed-off-by: John Snow --- scripts/make-release | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/scripts/make-release b/scripts/make-release index 44a9d86a04..a59bad11f9 100755 --- a/scripts/make-release +++ b/scripts/make-release @@ -41,6 +41,17 @@ git submodule update --init --single-branch BaseTools/Source/C/BrotliCompress/brotli \ CryptoPkg/Library/OpensslLib/openssl \ MdeModulePkg/Library/BrotliCustomDecompressLib/brotli) + +# Handle vendoring Python dependencies: +mkdir python/wheels +pushd python/wheels +pip download meson==0.61.5 +sha256sum -c < X-Patchwork-Id: 13222551 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 642FCC7618E for ; Mon, 24 Apr 2023 20:05:14 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Ox-0007qa-Gp; Mon, 24 Apr 2023 16:03:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ov-0007nk-Jn for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:05 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ou-0000y9-24 for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366583; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=haSGhn7+xY1tMHPJbNzMnLnBuxDDJ9oSFBnOrY3sQsg=; b=O+vNrDu8iUKmnAMPInDJANB33XVz+u94xtHr+uu1j0fjmVoSFEpqCi+xlxObEMc95zeVTB vujmOdrsI0rAKZWkZVxwFqDjixosisHmZoXeIHClfBPQ8DtHkAQXnjl5Fr0MU2Y8WIlgFe 2Eg9/Stc6pIoYylBOnVQ34782HCTTvQ= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-102-7XstacTSNwi-nxAHOEDt1Q-1; Mon, 24 Apr 2023 16:02:59 -0400 X-MC-Unique: 7XstacTSNwi-nxAHOEDt1Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4C5C2280BF61; Mon, 24 Apr 2023 20:02:59 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 975331121318; Mon, 24 Apr 2023 20:02:58 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 13/20] configure: create a python venv unconditionally Date: Mon, 24 Apr 2023 16:02:41 -0400 Message-Id: <20230424200248.1183394-14-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch changes the configure script so that it always creates and uses a python virtual environment unconditionally. Meson bootstrapping is temporarily altered to force the use of meson from git or vendored source. The next commit restores the use of distribution-vendored Meson. Signed-off-by: John Snow --- configure | 36 +++++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/configure b/configure index 77c03315f8..462fe604d6 100755 --- a/configure +++ b/configure @@ -625,7 +625,6 @@ check_py_version() { python= first_python= if test -z "${PYTHON}"; then - explicit_python=no # A bare 'python' is traditionally python 2.x, but some distros # have it as python 3.x, so check in both places. for binary in python3 python python3.11 python3.10 python3.9 python3.8 python3.7 python3.6; do @@ -644,7 +643,6 @@ else # Same as above, but only check the environment variable. has "${PYTHON}" || error_exit "The PYTHON environment variable does not point to an executable" python=$(command -v "$PYTHON") - explicit_python=yes if check_py_version "$python"; then # This one is good. first_python= @@ -729,7 +727,7 @@ for opt do ;; --install=*) ;; - --python=*) python="$optarg" ; explicit_python=yes + --python=*) python="$optarg" ;; --skip-meson) skip_meson=yes ;; @@ -1089,8 +1087,34 @@ if ! check_py_version "$python"; then "Use --python=/path/to/python to specify a supported Python." fi -# Resolve PATH + suppress writing compiled files -python="$(command -v "$python") -B" +# Resolve PATH +python="$(command -v "$python")" +explicit_python=yes + +# Create a Python virtual environment using our configured python. +# The stdout of this script will be the location of a symlink that +# points to the configured Python. +# Entry point scripts for pip, meson, and sphinx are generated if those +# packages are present. + +# Defaults assumed for now: +# - venv is cleared if it exists already; +# - venv is allowed to use system packages; +# - all setup is performed **offline**; +# - No packages are installed by default; +# - pip is not installed into the venv when possible, +# but ensurepip is called as a fallback when necessary. + +echo "python determined to be '$python'" +echo "python version: $($python --version)" + +python="$($python -B "${source_path}/python/scripts/mkvenv.py" create --gen pip,meson,sphinx pyvenv)" +if test "$?" -ne 0 ; then + error_exit "python venv creation failed" +fi + +# Suppress writing compiled files +python="$python -B" has_meson() { local python_dir=$(dirname "$python") @@ -1145,6 +1169,8 @@ case "$meson" in *) meson=$(command -v "$meson") ;; esac +echo "MKVENV ok!" + # Probe for ninja if test -z "$ninja"; then From patchwork Mon Apr 24 20:02:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C386FC7618E for ; Mon, 24 Apr 2023 20:04:23 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Oz-0007sA-Aj; Mon, 24 Apr 2023 16:03:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ow-0007oM-4S for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:06 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ou-0000yJ-A2 for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366583; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HtFYttd8CqMAVMoMy3j2CRbrsXLDDEtEXCCLyjF0Gl4=; b=b7t4QcqoPFf7sOKPbM7s6/HQYokbiRdSO+yofkUWNdfrlT8Qsps4eCZy+XKh1vHidNEfOr mFDItXjI7ztGFjNIXQMZFNugosYMEccE3yi9+A+YG/OtWC3eAnSdXK+Ti5G8v9LWxSth3K d2MOVe/58oE2OLbmQN0+Nh8zEyH//Q0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-550-R3rtj1TON_O2RoKRNcFplw-1; Mon, 24 Apr 2023 16:03:00 -0400 X-MC-Unique: R3rtj1TON_O2RoKRNcFplw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0D271185A794; Mon, 24 Apr 2023 20:03:00 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 58CD21121319; Mon, 24 Apr 2023 20:02:59 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 14/20] configure: use 'mkvenv ensure meson' to bootstrap meson Date: Mon, 24 Apr 2023 16:02:42 -0400 Message-Id: <20230424200248.1183394-15-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This commit changes how we detect and install meson. It notably removes '--meson='. The previous patch creates a lightweight Python virtual environment unconditionally using the user's configured $python that inherits system packages. If Meson is installed there and meets our minimum version requirements, we will use that Meson. In the event that Meson is installed but *not for the chosen Python interpreter*, not found, or of insufficient version, we will attempt to install Meson from vendored source into the newly created Python virtual environment. This vendored installation is considered to replace the mechanism from prior tarball distributions. This commit restores the ability to use a system meson, but in turn temporarily removes the ability to use a meson as obtained from the internet at configure-time (git submodules, as it stood prior to this patch); that ability will be restored in the next commit. As a result of this patch, the Python interpreter we use for both our own build scripts *and* Meson extensions are always known to be the exact same Python. As a further benefit, there will also be a symlink available in the build directory that points to the correct, configured python and can be used by e.g. manual tests to invoke the correct, configured Python unambiguously. Signed-off-by: John Snow --- configure | 72 ++++++++--------------------- .gitlab-ci.d/buildtest-template.yml | 4 +- 2 files changed, 21 insertions(+), 55 deletions(-) diff --git a/configure b/configure index 462fe604d6..e9947369b2 100755 --- a/configure +++ b/configure @@ -731,8 +731,6 @@ for opt do ;; --skip-meson) skip_meson=yes ;; - --meson=*) meson="$optarg" - ;; --ninja=*) ninja="$optarg" ;; --smbd=*) smbd="$optarg" @@ -1016,7 +1014,6 @@ Advanced options (experts only): --cross-prefix-ARCH=PREFIX cross compiler prefix when building ARCH guest test cases --make=MAKE use specified make [$make] --python=PYTHON use specified python [$python] - --meson=MESON use specified meson [$meson] --ninja=NINJA use specified ninja [$ninja] --smbd=SMBD use specified smbd [$smbd] --with-git=GIT use specified git [$git] @@ -1089,7 +1086,8 @@ fi # Resolve PATH python="$(command -v "$python")" -explicit_python=yes +# This variable is intended to be used only for error messages: +target_python=$python # Create a Python virtual environment using our configured python. # The stdout of this script will be the location of a symlink that @@ -1101,7 +1099,6 @@ explicit_python=yes # - venv is cleared if it exists already; # - venv is allowed to use system packages; # - all setup is performed **offline**; -# - No packages are installed by default; # - pip is not installed into the venv when possible, # but ensurepip is called as a fallback when necessary. @@ -1116,58 +1113,27 @@ fi # Suppress writing compiled files python="$python -B" -has_meson() { - local python_dir=$(dirname "$python") - # PEP405: pyvenv.cfg is either adjacent to the Python executable - # or one directory above - if test -f $python_dir/pyvenv.cfg || test -f $python_dir/../pyvenv.cfg; then - # Ensure that Meson and Python come from the same virtual environment - test -x "$python_dir/meson" && - test "$(command -v meson)" -ef "$python_dir/meson" - else - has meson - fi -} -if test -z "$meson"; then - if test "$explicit_python" = no && has_meson && version_ge "$(meson --version)" 0.61.5; then - meson=meson - elif test "$git_submodules_action" != 'ignore' ; then - meson=git - elif test -e "${source_path}/meson/meson.py" ; then - meson=internal - else - if test "$explicit_python" = yes; then - error_exit "--python requires using QEMU's embedded Meson distribution, but it was not found." - else - error_exit "Meson not found. Use --meson=/path/to/meson" - fi +if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \ + --dir "${source_path}/python/wheels" \ + "meson>=0.61.5" ; +then + # We're very out of luck. Try to give a good diagnostic. + if test -e pyvenv/bin/meson; then + echo "Meson is too old: $(pyvenv/bin/meson --version)" + elif has meson ; then + echo "Meson was found installed on your system," \ + "but not for the configured Python interpreter ($target_python)." + echo "(Hint: check '$(which meson)' to see which interpreter its shebang uses.)" fi -else - # Meson uses its own Python interpreter to invoke other Python scripts, - # but the user wants to use the one they specified with --python. - # - # We do not want to override the distro Python interpreter (and sometimes - # cannot: for example in Homebrew /usr/bin/meson is a bash script), so - # just require --meson=git|internal together with --python. - if test "$explicit_python" = yes; then - case "$meson" in - git | internal) ;; - *) error_exit "--python requires using QEMU's embedded Meson distribution." ;; - esac - fi -fi - -if test "$meson" = git; then - git_submodules="${git_submodules} meson" + exit 1 fi -case "$meson" in - git | internal) - meson="$python ${source_path}/meson/meson.py" - ;; - *) meson=$(command -v "$meson") ;; -esac +# At this point, we expect Meson to be installed and available. +# We expect mkvenv or pip to have created pyvenv/bin/meson for us. +# We ignore PATH completely here: we want to use the venv's Meson +# *exclusively*. +meson="$(cd pyvenv/bin; pwd)/meson" echo "MKVENV ok!" diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml index a6cfe9be97..7edb50b760 100644 --- a/.gitlab-ci.d/buildtest-template.yml +++ b/.gitlab-ci.d/buildtest-template.yml @@ -12,12 +12,12 @@ - mkdir build - cd build - ../configure --enable-werror --disable-docs --enable-fdt=system - ${LD_JOBS:+--meson=git} ${TARGETS:+--target-list="$TARGETS"} + ${TARGETS:+--target-list="$TARGETS"} $CONFIGURE_ARGS || { cat config.log meson-logs/meson-log.txt && exit 1; } - if test -n "$LD_JOBS"; then - ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ; + pyvenv/bin/meson configure . -Dbackend_max_links="$LD_JOBS" ; fi || exit 1; - make -j"$JOBS" - if test -n "$MAKE_CHECK_ARGS"; From patchwork Mon Apr 24 20:02:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A9270C7618E for ; Mon, 24 Apr 2023 20:05:38 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Ox-0007qs-Oj; Mon, 24 Apr 2023 16:03:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ov-0007nZ-F5 for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:05 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ot-0000y5-VQ for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366583; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2xhmH6Vc5KqGvVDk+uJmEGXYVREIzNMxeSfbzKX+8gw=; b=ckd+piwc6vYw6XFvsfd0VHBmTQRzX3xbvjNUTAiom8CA/FWl+hB8FboDzw9/ehVSBvrGAS RtXaIToVL6V+N1rON6h8lM5nAbooUr0uSCqtt8ZatxRFExyTTSThnOMaiMss/mu7TCUtA/ LkU0/PYFx64RY398A4Hon0L4oJXewCw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-563-8AqWwVHUMraMwh3EzKGPFQ-1; Mon, 24 Apr 2023 16:03:01 -0400 X-MC-Unique: 8AqWwVHUMraMwh3EzKGPFQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C5F2E88B7A1; Mon, 24 Apr 2023 20:03:00 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1C4611121318; Mon, 24 Apr 2023 20:03:00 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 15/20] configure: add --enable-pypi and --disable-pypi Date: Mon, 24 Apr 2023 16:02:43 -0400 Message-Id: <20230424200248.1183394-16-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org In the event that there's no vendored source present and no sufficient version of meson is found, we will attempt to connect to PyPI to install the package ... only if '--disable-pypi' was not passed. This mechanism is intended to replace the online functionality of the meson git submodule. While --enable-pypi is the default, vendored source will always be preferred when found, making PyPI a fallback. This should ensure that configure-time venv building "just works" for almost everyone in almost every circumstance. Signed-off-by: John Snow --- configure | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/configure b/configure index e9947369b2..7421bb8364 100755 --- a/configure +++ b/configure @@ -623,6 +623,7 @@ check_py_version() { } python= +pypi="enabled" first_python= if test -z "${PYTHON}"; then # A bare 'python' is traditionally python 2.x, but some distros @@ -883,6 +884,10 @@ for opt do --with-git-submodules=*) git_submodules_action="$optarg" ;; + --disable-pypi) pypi="disabled" + ;; + --enable-pypi) pypi="enabled" + ;; --enable-plugins) if test "$mingw32" = "yes"; then error_exit "TCG plugins not currently supported on Windows platforms" else @@ -1098,7 +1103,9 @@ target_python=$python # Defaults assumed for now: # - venv is cleared if it exists already; # - venv is allowed to use system packages; -# - all setup is performed **offline**; +# - all setup can be performed offline; +# - missing packages may be fetched from PyPI, +# unless --disable-pypi is passed. # - pip is not installed into the venv when possible, # but ensurepip is called as a fallback when necessary. @@ -1114,7 +1121,13 @@ fi python="$python -B" +mkvenv_flags="" +if test "$pypi" = "enabled" ; then + mkvenv_flags="--online" +fi + if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \ + $mkvenv_flags \ --dir "${source_path}/python/wheels" \ "meson>=0.61.5" ; then From patchwork Mon Apr 24 20:02:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222548 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 54C0FC77B61 for ; Mon, 24 Apr 2023 20:04:32 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2Oz-0007td-Vq; Mon, 24 Apr 2023 16:03:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Oz-0007rx-2n for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:09 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ow-00010P-PE for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366586; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wnEfX62FKB2h78OhdKkFkfQWaEpLq7uesHX0XNyBTIQ=; b=WGGqjbHsYmtkQRVCmqvBeIRWMep6JUum2Ib1MT0zE7uvFECg1NbfY110mtGpid9jS36Ux2 8CVZY36V6dqNFA4Gqp6YQxmgebG7Sha3snC4fWVyQbyhcmnNazESlaYMBTzOW2ZEhOwID/ uVAFZ22UjJDAOHCVtSvMCT47/oP4mcM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-433-8Ieea1BKNTycXHywsh9DOQ-1; Mon, 24 Apr 2023 16:03:02 -0400 X-MC-Unique: 8Ieea1BKNTycXHywsh9DOQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8BFE787B2A0; Mon, 24 Apr 2023 20:03:01 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id D82B0112131E; Mon, 24 Apr 2023 20:03:00 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 16/20] tests: Use configure-provided pyvenv for tests Date: Mon, 24 Apr 2023 16:02:44 -0400 Message-Id: <20230424200248.1183394-17-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch changes how the avocado tests are provided, ever so slightly. Instead of creating a new testing venv, use the configure-provided 'pyvenv' instead, and install optional packages into that. Note 1: This doesn't respect the "disable-pypi" setting; the avocado tests remain an unconditional PyPI hit when you run "make check-avocado". Future series may endeavor to offer tighter integration, for now, this is a cheap win that doesn't regress anything. Note 2: At the time of writing, avocado tests require avocado-framework<90 whereas the qemu.qmp self-tests rely on avocado-framework>=90. This collision is avoided for now because the qemu.git/python/qemu/ code does not need avocado at *runtime*; it does not install avocado-framework as a necessary dependency and is skipped in this circumstance. Nevertheless, we do want to address that discrepancy in the future so that it will be possible to re-use the same venv for qemu.git/python/qemu self-tests to introduce them to make check as "make check-python". Signed-off-by: John Snow --- docs/devel/acpi-bits.rst | 6 +++--- docs/devel/testing.rst | 14 +++++++------- .gitlab-ci.d/buildtest.yml | 6 +++--- scripts/ci/org.centos/stream/8/x86_64/test-avocado | 4 ++-- scripts/device-crash-test | 2 +- tests/Makefile.include | 10 +++++----- tests/requirements.txt | 7 +++++-- 7 files changed, 26 insertions(+), 23 deletions(-) diff --git a/docs/devel/acpi-bits.rst b/docs/devel/acpi-bits.rst index 9eb4b9e666..0c40359109 100644 --- a/docs/devel/acpi-bits.rst +++ b/docs/devel/acpi-bits.rst @@ -61,19 +61,19 @@ Under ``tests/avocado/`` as the root we have: :: $ make check-venv (needed only the first time to create the venv) - $ ./tests/venv/bin/avocado run -t acpi tests/avocado + $ ./pyvenv/bin/avocado run -t acpi tests/avocado The above will run all acpi avocado tests including this one. In order to run the individual tests, perform the following: :: - $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py --tap - + $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py --tap - The above will produce output in tap format. You can omit "--tap -" in the end and it will produce output like the following: :: - $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py + $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits JOB ID : eab225724da7b64c012c65705dc2fa14ab1defef JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-10T17.58-eab2257/job.log diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 4071e72710..50664d9eb9 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -882,9 +882,9 @@ You can run the avocado tests simply by executing: make check-avocado -This involves the automatic creation of Python virtual environment -within the build tree (at ``tests/venv``) which will have all the -right dependencies, and will save tests results also within the +This involves the automatic installation, from PyPI, of all the +necessary avocado-framework dependencies into the QEMU venv within the +build tree (at ``./pyvenv``). Test results are also saved within the build tree (at ``tests/results``). Note: the build environment must be using a Python 3 stack, and have @@ -941,7 +941,7 @@ may be invoked by running: .. code:: - tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/ + pyvenv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/ Note that if ``make check-avocado`` was not executed before, it is possible to create the Python virtual environment with the dependencies @@ -956,20 +956,20 @@ a test file. To run tests from a single file within the build tree, use: .. code:: - tests/venv/bin/avocado run tests/avocado/$TESTFILE + pyvenv/bin/avocado run tests/avocado/$TESTFILE To run a single test within a test file, use: .. code:: - tests/venv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME + pyvenv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME Valid test names are visible in the output from any previous execution of Avocado or ``make check-avocado``, and can also be queried using: .. code:: - tests/venv/bin/avocado list tests/avocado + pyvenv/bin/avocado list tests/avocado Manual Installation ~~~~~~~~~~~~~~~~~~~ diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml index ba6f551752..53de9f23c4 100644 --- a/.gitlab-ci.d/buildtest.yml +++ b/.gitlab-ci.d/buildtest.yml @@ -103,7 +103,7 @@ crash-test-debian: script: - cd build - make check-venv - - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-i386 + - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-i386 build-system-fedora: extends: @@ -146,8 +146,8 @@ crash-test-fedora: script: - cd build - make check-venv - - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc - - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32 + - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc + - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32 build-system-centos: extends: diff --git a/scripts/ci/org.centos/stream/8/x86_64/test-avocado b/scripts/ci/org.centos/stream/8/x86_64/test-avocado index d2c0e5fb4c..7bb5b317b6 100755 --- a/scripts/ci/org.centos/stream/8/x86_64/test-avocado +++ b/scripts/ci/org.centos/stream/8/x86_64/test-avocado @@ -4,7 +4,7 @@ # KVM and x86_64, or tests that are generic enough to be valid for all # targets. Such a test list can be generated with: # -# ./tests/venv/bin/avocado list --filter-by-tags-include-empty \ +# ./pyvenv/bin/avocado list --filter-by-tags-include-empty \ # --filter-by-tags-include-empty-key -t accel:kvm,arch:x86_64 \ # tests/avocado/ # @@ -22,7 +22,7 @@ # - tests/avocado/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types # make get-vm-images -./tests/venv/bin/avocado run \ +./pyvenv/bin/avocado run \ --job-results-dir=tests/results/ \ tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_i440fx_kvm \ tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_kvm \ diff --git a/scripts/device-crash-test b/scripts/device-crash-test index 73bcb98693..ef6ac262fa 100755 --- a/scripts/device-crash-test +++ b/scripts/device-crash-test @@ -43,7 +43,7 @@ except ModuleNotFoundError as exc: print(f"Module '{exc.name}' not found.") print(" Try 'make check-venv' from your build directory,") print(" and then one way to run this script is like so:") - print(f' > $builddir/tests/venv/bin/python3 "{path}"') + print(f' > $builddir/pyvenv/bin/python3 "{path}"') sys.exit(1) logger = logging.getLogger('device-crash-test') diff --git a/tests/Makefile.include b/tests/Makefile.include index 9422ddaece..3f39c607cc 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -89,7 +89,8 @@ distclean-tcg: $(DISTCLEAN_TCG_TARGET_RULES) # Build up our target list from the filtered list of ninja targets TARGETS=$(patsubst libqemu-%.fa, %, $(filter libqemu-%.fa, $(ninja-targets))) -TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv +TESTS_VENV_DIR=$(BUILD_DIR)/pyvenv +TESTS_VENV_TOKEN=$(BUILD_DIR)/pyvenv/tests.group TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results TESTS_PYTHON=$(TESTS_VENV_DIR)/bin/python3 @@ -111,8 +112,7 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \ $(TESTS_PYTHON) -m pip -q --disable-pip-version-check $1, \ "VENVPIP","$1") -$(TESTS_VENV_DIR): $(TESTS_VENV_REQ) - $(call quiet-command, $(PYTHON) -m venv $@, VENV, $@) +$(TESTS_VENV_TOKEN): $(TESTS_VENV_REQ) $(call quiet-venv-pip,install -e "$(SRC_PATH)/python/") $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) $(call quiet-command, touch $@) @@ -121,7 +121,7 @@ $(TESTS_RESULTS_DIR): $(call quiet-command, mkdir -p $@, \ MKDIR, $@) -check-venv: $(TESTS_VENV_DIR) +check-venv: $(TESTS_VENV_TOKEN) FEDORA_31_ARCHES_TARGETS=$(patsubst %-softmmu,%, $(filter %-softmmu,$(TARGETS))) FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(FEDORA_31_ARCHES_TARGETS)) @@ -163,7 +163,7 @@ check: check-build: run-ninja check-clean: - rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR) + rm -rf $(TESTS_RESULTS_DIR) clean: check-clean clean-tcg distclean: distclean-tcg diff --git a/tests/requirements.txt b/tests/requirements.txt index 0ba561b6bd..07e713ef5a 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -1,6 +1,9 @@ # Add Python module requirements, one per line, to be installed -# in the tests/venv Python virtual environment. For more info, +# in the qemu build_dir/pyvenv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 -# Note that qemu.git/python/ is always implicitly installed. +# +# Note that qemu.git/python/ is implicitly installed to this venv when +# 'make check-venv' is run, and will persist until configure is run +# again. avocado-framework==88.1 pycdlib==1.11.0 From patchwork Mon Apr 24 20:02:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 666AEC77B7E for ; Mon, 24 Apr 2023 20:05:17 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2P1-0007vG-9i; Mon, 24 Apr 2023 16:03:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Oz-0007sP-H8 for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:09 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ow-00010Y-UI for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366586; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YNP+fL23B81VoPR+iHM5CGssB9x+3X4xqlu1OkuBfxU=; b=aTVRJ+24AgYRZLWhM04LfOc2XN3JtiX+Q1hTD5gmpE1x66FiDUTQj0tidtAcQta+v8B2/z eI/3l9EubTTqpM8vrmh/YfMoknkDUYzal6rF5Z8wj27nF3bEhJxELLwwWttd9DSTqR9lQZ TufCGLS2QrbVFmiei6KYQIosqiaIkpc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-495-nUSzrq90PoaTenqd1zhgTg-1; Mon, 24 Apr 2023 16:03:02 -0400 X-MC-Unique: nUSzrq90PoaTenqd1zhgTg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 58CC685A5A3; Mon, 24 Apr 2023 20:03:02 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9CB261121319; Mon, 24 Apr 2023 20:03:01 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 17/20] configure: move --enable-docs and --disable-docs back to configure Date: Mon, 24 Apr 2023 16:02:45 -0400 Message-Id: <20230424200248.1183394-18-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Move this option back from meson into configure for the purposes of using the configuration value to bootstrap Sphinx in different ways based on this value. Signed-off-by: John Snow --- configure | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/configure b/configure index 7421bb8364..ae55de1408 100755 --- a/configure +++ b/configure @@ -279,6 +279,7 @@ debug_tcg="no" sanitizers="no" tsan="no" fortify_source="yes" +docs="auto" EXESUF="" modules="no" prefix="/usr/local" @@ -752,6 +753,10 @@ for opt do ;; --disable-debug-info) meson_option_add -Ddebug=false ;; + --enable-docs) docs=enabled + ;; + --disable-docs) docs=disabled + ;; --enable-modules) modules="yes" ;; @@ -2638,6 +2643,7 @@ if test "$skip_meson" = no; then # QEMU options test "$cfi" != false && meson_option_add "-Dcfi=$cfi" + test "$docs" != auto && meson_option_add "-Ddocs=$docs" test "$fdt" != auto && meson_option_add "-Dfdt=$fdt" test -n "${LIB_FUZZING_ENGINE+xxx}" && meson_option_add "-Dfuzzing_engine=$LIB_FUZZING_ENGINE" test "$qemu_suffix" != qemu && meson_option_add "-Dqemu_suffix=$qemu_suffix" From patchwork Mon Apr 24 20:02:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222561 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8903CC7618E for ; Mon, 24 Apr 2023 20:08:29 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2P3-0007yy-DN; Mon, 24 Apr 2023 16:03:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2P0-0007uf-QN for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:10 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Ox-00010j-NW for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366587; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5ZVxxSZ7/C5X1V8cQvXR6zh46ie6lD6B7GimcdwGhcs=; b=CEaroXkf+fb+a8Q9KhPn2Lb1Ldu1JgNETvvpHBNaxLrg0zJ6yAaOHJPVBSEc4zHzTyL3op kZyRTAOTJAtvX6Ks1mkZ9CdCF64W/H2+eaiSL6pKMLmdY9yCfzr4gUOL5JLmVeZ80dJe2g X8+X0c29SFaa0sB6OIF8b7FOdeo30Rc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-604-lEF_6W8KPBWjkJ1YEx1AWw-1; Mon, 24 Apr 2023 16:03:03 -0400 X-MC-Unique: lEF_6W8KPBWjkJ1YEx1AWw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2105C88B7A1; Mon, 24 Apr 2023 20:03:03 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6AD7F1121318; Mon, 24 Apr 2023 20:03:02 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 18/20] mkvenv: add diagnose() method for ensure() failures Date: Mon, 24 Apr 2023 16:02:46 -0400 Message-Id: <20230424200248.1183394-19-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This is a routine that is designed to print some usable info for human beings back out to the terminal if/when "mkvenv ensure" fails to locate or install a package during configure time, such as meson or sphinx. Since we are requiring that "meson" and "sphinx" are installed to the same Python environment as QEMU is configured to build with, this can produce some surprising failures when things are mismatched. This method is here to try and ease that sting by offering some actionable diagnosis. Signed-off-by: John Snow --- python/scripts/mkvenv.py | 153 +++++++++++++++++++++++++++++++++++---- 1 file changed, 140 insertions(+), 13 deletions(-) diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py index 937664ea9c..1bc4bc01b1 100644 --- a/python/scripts/mkvenv.py +++ b/python/scripts/mkvenv.py @@ -64,6 +64,7 @@ import os from pathlib import Path import re +import shutil import site import stat import subprocess @@ -543,6 +544,103 @@ def checkpip() -> None: logging.debug("Pip is now (hopefully) repaired!") +def diagnose( + dep_spec: str, + online: bool, + wheels_dir: Optional[Union[str, Path]], + prog: Optional[str], +) -> str: + """ + Offer a summary to the user as to why a package failed to be installed. + + :param dep_spec: The package we tried to ensure, e.g. 'meson>=0.61.5' + :param online: Did we allow PyPI access? + :param prog: + Optionally, a shell program name that can be used as a + bellwether to detect if this program is installed elsewhere on + the system. This is used to offer advice when a program is + detected for a different python version. + :param wheels_dir: + Optionally, a directory that was searched for vendored packages. + """ + # pylint: disable=too-many-branches + + # Parse name out of PEP-508 depspec. + # See https://peps.python.org/pep-0508/#names + match = re.match( + r"^([A-Z0-9]([A-Z0-9._-]*[A-Z0-9])?)", dep_spec, re.IGNORECASE + ) + if not match: + raise ValueError( + f"dep_spec '{dep_spec}'" + " does not appear to contain a valid package name" + ) + pkg_name = match.group(0) + pkg_version = None + + has_importlib = False + try: + # Python 3.8+ stdlib + # pylint: disable=import-outside-toplevel + from importlib.metadata import PackageNotFoundError, version + + has_importlib = True + try: + pkg_version = version(pkg_name) + except PackageNotFoundError: + pass + except ModuleNotFoundError: + pass + + lines = [] + + if pkg_version: + lines.append( + f"Python package '{pkg_name}' version '{pkg_version}' was found," + " but isn't suitable." + ) + elif has_importlib: + lines.append( + f"Python package '{pkg_name}' was not found nor installed." + ) + else: + lines.append( + f"Python package '{pkg_name}' is either not found or" + " not a suitable version." + ) + + if wheels_dir: + lines.append( + "No suitable version found in, or failed to install from" + f" '{wheels_dir}'." + ) + else: + lines.append("No local package directory was searched.") + + if online: + lines.append("A suitable version could not be obtained from PyPI.") + else: + lines.append( + "mkvenv was configured to operate offline and did not check PyPI." + ) + + if prog and not pkg_version: + which = shutil.which(prog) + if which: + pypath = Path(sys.executable).resolve() + lines.append( + f"'{prog}' was detected on your system at '{which}', " + f"but the Python package '{pkg_name}' was not found by this " + f"Python interpreter ('{pypath}'). " + f"Typically this means that '{prog}' has been installed " + "against a different Python interpreter on your system." + ) + + lines = [f" • {line}" for line in lines] + lines.insert(0, f"Could not ensure availability of '{dep_spec}':") + return os.linesep.join(lines) + + def pip_install( args: Sequence[str], online: bool = False, @@ -573,23 +671,11 @@ def pip_install( subprocess.run(full_args, check=True) -def ensure( +def _do_ensure( dep_spec: str, online: bool = False, wheels_dir: Optional[Union[str, Path]] = None, ) -> None: - """ - Use pip to ensure we have the package specified by @dep_spec. - - If the package is already installed, do nothing. If online and - wheels_dir are both provided, prefer packages found in wheels_dir - first before connecting to PyPI. - - :param dep_spec: - A PEP 508 dependency specification. e.g. 'meson>=0.61.5'. - :param online: If True, fall back to PyPI. - :param wheels_dir: If specified, search this path for packages. - """ # This first install command will: # (A) Do nothing, if we already have a suitable package. # (B) Install the package from vendored source, if possible. @@ -603,11 +689,42 @@ def ensure( # The package is missing or isn't a suitable version, # and we weren't able to install a suitable vendored package. if online: + logger.info("offline ensure failed, trying PyPI ...") pip_install([dep_spec], online=True) else: raise +def ensure( + dep_spec: str, + online: bool = False, + wheels_dir: Optional[Union[str, Path]] = None, + prog: Optional[str] = None, +) -> None: + """ + Use pip to ensure we have the package specified by @dep_spec. + + If the package is already installed, do nothing. If online and + wheels_dir are both provided, prefer packages found in wheels_dir + first before connecting to PyPI. + + :param dep_spec: + A PEP 508 dependency specification. e.g. 'meson>=0.61.5'. + :param online: If True, fall back to PyPI. + :param wheels_dir: If specified, search this path for packages. + :param prog: + If specified, use this program name for error diagnostics that will + be presented to the user. e.g., 'sphinx-build' can be used as a + bellwether for the presence of 'sphinx'. + """ + print(f"MKVENV ensure {dep_spec}", file=sys.stderr) + try: + _do_ensure(dep_spec, online, wheels_dir) + except subprocess.CalledProcessError as exc: + # Well, that's not good. + raise Ouch(diagnose(dep_spec, online, wheels_dir, prog)) from exc + + def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None: """ This is intended to be run *inside the venv* after it is created. @@ -671,6 +788,15 @@ def _add_ensure_subcommand(subparsers: Any) -> None: action="store", help="Path to vendored packages where we may install from.", ) + subparser.add_argument( + "--diagnose", + type=str, + action="store", + help=( + "Name of a shell utility to use for " + "diagnostics if this command fails." + ), + ) subparser.add_argument( "dep_spec", type=str, @@ -727,6 +853,7 @@ def _normalize_gen() -> None: dep_spec=args.dep_spec, online=args.online, wheels_dir=args.dir, + prog=args.diagnose, ) logger.debug("mkvenv.py %s: exiting", args.command) except Ouch as exc: From patchwork Mon Apr 24 20:02:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222546 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 495DFC7618E for ; Mon, 24 Apr 2023 20:04:17 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2P2-0007yV-P2; Mon, 24 Apr 2023 16:03:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2P0-0007uY-MU for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:10 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2Oy-00010t-Km for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366588; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TR90EIS18UVyGds2jFRXjl24pA0rHLnN+HKJ4k91mQc=; b=eRpCJ2zOO6cLZfVFhMhiioSeg3Psrpkfw4JqMl1flel6G5FxkkXy8Cb0zO04G72WZkx7IH Rm/2srvCCoAZTQ4L4GVp9wpUDSwDlMC1myYp0MXIqoy02NafH8IdLxNiu6mJNI18DVCEHO jBNMV7o2L24dMXfW3qXjaWpP8Gh49u0= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-460-zhe__F8EP_u0QQ15lGpyyQ-1; Mon, 24 Apr 2023 16:03:04 -0400 X-MC-Unique: zhe__F8EP_u0QQ15lGpyyQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D85F6381494C; Mon, 24 Apr 2023 20:03:03 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2F5FD1121318; Mon, 24 Apr 2023 20:03:03 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 19/20] configure: use --diagnose option with meson ensure Date: Mon, 24 Apr 2023 16:02:47 -0400 Message-Id: <20230424200248.1183394-20-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Rely on the diagnosis information from mkvenv so it can be removed from the configure script. Signed-off-by: John Snow --- configure | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/configure b/configure index ae55de1408..0881fffc14 100755 --- a/configure +++ b/configure @@ -1096,8 +1096,6 @@ fi # Resolve PATH python="$(command -v "$python")" -# This variable is intended to be used only for error messages: -target_python=$python # Create a Python virtual environment using our configured python. # The stdout of this script will be the location of a symlink that @@ -1134,16 +1132,9 @@ fi if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \ $mkvenv_flags \ --dir "${source_path}/python/wheels" \ + --diagnose "meson" \ "meson>=0.61.5" ; then - # We're very out of luck. Try to give a good diagnostic. - if test -e pyvenv/bin/meson; then - echo "Meson is too old: $(pyvenv/bin/meson --version)" - elif has meson ; then - echo "Meson was found installed on your system," \ - "but not for the configured Python interpreter ($target_python)." - echo "(Hint: check '$(which meson)' to see which interpreter its shebang uses.)" - fi exit 1 fi From patchwork Mon Apr 24 20:02:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13222559 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 84789C77B61 for ; Mon, 24 Apr 2023 20:08:11 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pr2P3-0007zB-TV; Mon, 24 Apr 2023 16:03:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2P2-0007xW-5U for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:12 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pr2P0-00012G-Op for qemu-devel@nongnu.org; Mon, 24 Apr 2023 16:03:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682366590; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/3kmnqHRNkPrVSSzR0s0yiKhFrynEX8NWBk93M6ZT6s=; b=JrzWaamXQMKo+zlzRoqmkf5tx8y/1bzaS8srYl4S/1yOSAlxw5PjiP7dfKNAcYhPxiUEaX 8rSlc2us99zu89C5+lAHsJ9T33Ugar1yxGDLKvD1PIU/iEtH5F7AGRdoeD4Bi43/qFl18a B8b4mqdYXFSFyhWYCNukcJvcWKtGlAY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-159-WBEKJe3RNB-aCuhnv2xRXQ-1; Mon, 24 Apr 2023 16:03:05 -0400 X-MC-Unique: WBEKJe3RNB-aCuhnv2xRXQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9A26FA0F382; Mon, 24 Apr 2023 20:03:04 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.213]) by smtp.corp.redhat.com (Postfix) with ESMTP id E43E41121318; Mon, 24 Apr 2023 20:03:03 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Beraldo Leal , John Snow , Kyle Evans , Paolo Bonzini , Thomas Huth , Daniel Berrange , Reinoud Zandijk , Wainer dos Santos Moschetta , Cleber Rosa , Ryo ONODERA , =?utf-8?q?P?= =?utf-8?q?hilippe_Mathieu-Daud=C3=A9?= , Ani Sinha , "Michael S. Tsirkin" , =?utf-8?q?Alex_Benn=C3=A9e?= Subject: [RFC PATCH v3 20/20] configure: bootstrap sphinx with mkvenv Date: Mon, 24 Apr 2023 16:02:48 -0400 Message-Id: <20230424200248.1183394-21-jsnow@redhat.com> In-Reply-To: <20230424200248.1183394-1-jsnow@redhat.com> References: <20230424200248.1183394-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org When docs are explicitly requested, require Sphinx>=1.6.0. When docs are explicitly disabled, don't bother to check for Sphinx at all. If docs are set to "auto", attempt to locate Sphinx, but continue onward if it wasn't located. For the case that --enable-pypi is set to 'enabled' (the default) but docs are set to 'auto' (also the default), do not actually consult PyPI to install Sphinx. Only perform this action when docs are requested explicitly. Signed-off-by: John Snow --- configure | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/configure b/configure index 0881fffc14..a247b9491c 100755 --- a/configure +++ b/configure @@ -1122,14 +1122,14 @@ fi # Suppress writing compiled files python="$python -B" - +mkvenv="$python ${source_path}/python/scripts/mkvenv.py" mkvenv_flags="" if test "$pypi" = "enabled" ; then mkvenv_flags="--online" fi -if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \ +if ! $mkvenv ensure \ $mkvenv_flags \ --dir "${source_path}/python/wheels" \ --diagnose "meson" \ @@ -1144,6 +1144,29 @@ fi # *exclusively*. meson="$(cd pyvenv/bin; pwd)/meson" +# Conditionally ensure Sphinx is installed. + +mkvenv_flags="" +if test "$pypi" = "enabled" -a "$docs" = "enabled" ; then + mkvenv_flags="--online" +fi + +if test "$docs" != "disabled" ; then + if ! $mkvenv ensure \ + $mkvenv_flags \ + --diagnose "sphinx-build" \ + "sphinx>=1.6.0" ; + then + if test "$docs" = "enabled" ; then + exit 1 + fi + echo "Sphinx not found/usable, disabling docs." + docs=disabled + else + docs=enabled + fi +fi + echo "MKVENV ok!" # Probe for ninja