| Message ID | 20230424200248.1183394-1-jsnow@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | configure: create a python venv and ensure meson, sphinx | expand |
On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote: > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409 > (All green, except Python self-tests, see below) > > This patch series creates a mandatory python virtual environment > ("venv") during configure time and uses it to ensure the availability of > meson and sphinx. > > See https://www.qemu.org/2023/03/24/python/ for details. The summary is > that the goal of this series is to ensure that the `python` used to run > meson is the same `python` used to run Sphinx, tests, and any build-time > python scripting we have. As it stands, meson and sphinx (and their > extensions) *may* run in a different python environment than the one > configured and chosen by the user at configure/build time. > > The effective change of this series is that QEMU will now > unconditionally create a venv at configure-time and will ensure that > meson (and sphinx, if docs are enabled) are available through that venv. > > Some important points as a pre-emptive "FAQ": > > - This venv is unconditionally created and lives at {build_dir}/pyvenv. > > - The python interpreter used by this venv is always the one identified > by configure. (Which in turn is always the one specified by --python > or $PYTHON) > > - *almost* all python scripts in qemu.git executed as part of the build > system, meson, sphinx, avocado tests, vm tests or CI are always > executed within this venv. > > (iotests are not yet integrated; I plan to tackle this separately as a > follow-up in order to have a more tightly focused scope on that > series.) > > - It remains possible to build and test fully offline. > (In most cases, you just need meson and sphinx from your distro's repo.) > > - Distribution packaged 'meson' and 'sphinx' are still utilized whenever > possible as the highest preference. > > - Vendored versions of e.g. 'meson' are always preferred to PyPI > versions for speed, repeatability and ensuring tarball builds work > as-is offline. > > (Sphinx will not be vendored, just like it already isn't.) > > - Missing dependencies, when possible, are fetched and installed > on-demand automatically to make developer environments "just work". > > - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, > Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere > > - No new dependencies (...for most platforms. Debian and NetBSD get an > asterisk.) > > - The meson git submodule is unused after this series and can be removed. > > For reviewers, here's how the series is broken up: > > Patch 1 is a testing pre-req. Note that even with this patch, > 'check-python-minreqs' and 'check-python-tox' CI jobs will both still > fail on origin/master because this series requires 3.7+, but > origin/master is currently still 3.6+. > > - python: update pylint configuration > > Patches 2-8 add the mkvenv script. The first patch checks in the barest > essentials, and each subsequent patch adds a workaround or feature one > at a time. > > - python: add mkvenv.py > - mkvenv: add console script entry point generation > - mkvenv: Add better error message for missing pyexapt module > - mkvenv: generate console entry shims from inside the venv > - mkvenv: work around broken pip installations on Debian 10 > - mkvenv: add nested venv workaround > - mkvenv: add ensure subcommand > > Patches 9-11 modify our testing configuration to add new dependencies as > needed. > > - tests/docker: add python3-venv dependency > - tests/vm: Configure netbsd to use Python 3.10 > - tests/vm: add py310-expat to NetBSD > > Patch 12 changes how we package release tarballs. > > - scripts/make-release: download meson==0.61.5 .whl > > Patches 13-16 wire mkvenv into configure and tests. > > - configure: create a python venv unconditionally > - configure: use 'mkvenv ensure meson' to bootstrap meson > - configure: add --enable-pypi and --disable-pypi > - tests: Use configure-provided pyvenv for tests > > Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these > changes could be folded earlier in the series (like the diagnose() > patch), but I'm keeping it separate for review for now. > > - configure: move --enable-docs and --disable-docs back to configure > - mkvenv: add diagnose() method for ensure() failures > - configure: use --diagnose option with meson ensure > - configure: bootstrap sphinx with mkvenv I'm not sure this last bit is working. I uninstalled meson and python3-sphinx from my F38 host and ran configure --target-list=x86_64-softmmu and got this: $ ./configure --target-list=x86_64-softmmu Using './build' as the directory for build output python determined to be '/usr/bin/python3' python version: Python 3.11.3 MKVENV pyvenv Configured python as '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B' MKVENV ensure meson>=0.61.5 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is ignored: it is neither a file nor a directory. ERROR: Could not find a version that satisfies the requirement meson>=0.61.5 (from versions: none) ERROR: No matching distribution found for meson>=0.61.5 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' MKVENV ensure sphinx>=1.6.0 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' ERROR: Could not find a version that satisfies the requirement sphinx>=1.6.0 (from versions: none) ERROR: No matching distribution found for sphinx>=1.6.0 *** Ouch! *** Could not ensure availability of 'sphinx>=1.6.0': • Python package 'sphinx' was not found nor installed. • No local package directory was searched. • mkvenv was configured to operate offline and did not check PyPI. Sphinx not found/usable, disabling docs. MKVENV ok! It says mkvenv was configured to run offline, but I didn't do anything. I thought the intention was for developers it would live download from PyPI ? On a system where i already have meson/sphinx installed, it all just worked fine AFAICT. With regards, Daniel
On Tue, Apr 25, 2023, 1:17 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote: > > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409 > > (All green, except Python self-tests, see below) > > > > This patch series creates a mandatory python virtual environment > > ("venv") during configure time and uses it to ensure the availability of > > meson and sphinx. > > > > See https://www.qemu.org/2023/03/24/python/ for details. The summary is > > that the goal of this series is to ensure that the `python` used to run > > meson is the same `python` used to run Sphinx, tests, and any build-time > > python scripting we have. As it stands, meson and sphinx (and their > > extensions) *may* run in a different python environment than the one > > configured and chosen by the user at configure/build time. > > > > The effective change of this series is that QEMU will now > > unconditionally create a venv at configure-time and will ensure that > > meson (and sphinx, if docs are enabled) are available through that venv. > > > > Some important points as a pre-emptive "FAQ": > > > > - This venv is unconditionally created and lives at {build_dir}/pyvenv. > > > > - The python interpreter used by this venv is always the one identified > > by configure. (Which in turn is always the one specified by --python > > or $PYTHON) > > > > - *almost* all python scripts in qemu.git executed as part of the build > > system, meson, sphinx, avocado tests, vm tests or CI are always > > executed within this venv. > > > > (iotests are not yet integrated; I plan to tackle this separately as a > > follow-up in order to have a more tightly focused scope on that > > series.) > > > > - It remains possible to build and test fully offline. > > (In most cases, you just need meson and sphinx from your distro's > repo.) > > > > - Distribution packaged 'meson' and 'sphinx' are still utilized whenever > > possible as the highest preference. > > > > - Vendored versions of e.g. 'meson' are always preferred to PyPI > > versions for speed, repeatability and ensuring tarball builds work > > as-is offline. > > > > (Sphinx will not be vendored, just like it already isn't.) > > > > - Missing dependencies, when possible, are fetched and installed > > on-demand automatically to make developer environments "just work". > > > > - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, > > Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere > > > > - No new dependencies (...for most platforms. Debian and NetBSD get an > > asterisk.) > > > > - The meson git submodule is unused after this series and can be removed. > > > > For reviewers, here's how the series is broken up: > > > > Patch 1 is a testing pre-req. Note that even with this patch, > > 'check-python-minreqs' and 'check-python-tox' CI jobs will both still > > fail on origin/master because this series requires 3.7+, but > > origin/master is currently still 3.6+. > > > > - python: update pylint configuration > > > > Patches 2-8 add the mkvenv script. The first patch checks in the barest > > essentials, and each subsequent patch adds a workaround or feature one > > at a time. > > > > - python: add mkvenv.py > > - mkvenv: add console script entry point generation > > - mkvenv: Add better error message for missing pyexapt module > > - mkvenv: generate console entry shims from inside the venv > > - mkvenv: work around broken pip installations on Debian 10 > > - mkvenv: add nested venv workaround > > - mkvenv: add ensure subcommand > > > > Patches 9-11 modify our testing configuration to add new dependencies as > > needed. > > > > - tests/docker: add python3-venv dependency > > - tests/vm: Configure netbsd to use Python 3.10 > > - tests/vm: add py310-expat to NetBSD > > > > Patch 12 changes how we package release tarballs. > > > > - scripts/make-release: download meson==0.61.5 .whl > > > > Patches 13-16 wire mkvenv into configure and tests. > > > > - configure: create a python venv unconditionally > > - configure: use 'mkvenv ensure meson' to bootstrap meson > > - configure: add --enable-pypi and --disable-pypi > > - tests: Use configure-provided pyvenv for tests > > > > Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these > > changes could be folded earlier in the series (like the diagnose() > > patch), but I'm keeping it separate for review for now. > > > > - configure: move --enable-docs and --disable-docs back to configure > > - mkvenv: add diagnose() method for ensure() failures > > - configure: use --diagnose option with meson ensure > > - configure: bootstrap sphinx with mkvenv > > I'm not sure this last bit is working. > > I uninstalled meson and python3-sphinx from my F38 host and ran > configure --target-list=x86_64-softmmu and got this: > > $ ./configure --target-list=x86_64-softmmu > Using './build' as the directory for build output > python determined to be '/usr/bin/python3' > python version: Python 3.11.3 > MKVENV pyvenv > Configured python as > '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B' > MKVENV ensure meson>=0.61.5 > WARNING: Skipping > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > invalid metadata entry 'name' > WARNING: Skipping > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > invalid metadata entry 'name' > WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is > ignored: it is neither a file nor a directory. > ERROR: Could not find a version that satisfies the requirement > meson>=0.61.5 (from versions: none) > ERROR: No matching distribution found for meson>=0.61.5 > WARNING: Skipping > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > invalid metadata entry 'name' > WARNING: Skipping > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > invalid metadata entry 'name' > WARNING: Skipping > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > invalid metadata entry 'name' > MKVENV ensure sphinx>=1.6.0 > WARNING: Skipping > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > invalid metadata entry 'name' > WARNING: Skipping > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > invalid metadata entry 'name' > ERROR: Could not find a version that satisfies the requirement > sphinx>=1.6.0 (from versions: none) > ERROR: No matching distribution found for sphinx>=1.6.0 > > *** Ouch! *** > > Could not ensure availability of 'sphinx>=1.6.0': > • Python package 'sphinx' was not found nor installed. > • No local package directory was searched. > • mkvenv was configured to operate offline and did not check PyPI. > > > Sphinx not found/usable, disabling docs. > MKVENV ok! > > > > It says mkvenv was configured to run offline, but I didn't do > anything. I thought the intention was for developers it would > live download from PyPI ? > Ah. So... with enable pypi being the default and with docs set to "auto", I actually fall back to not installing sphinx from pypi *by default*. It tries to locate it on your system and will enable docs if it can, but it doesn't try too hard and doesn't get upset if it fails. Try passing --enable-docs to convince the build system you'd really definitely like docs, and it'll force the pypi access. > > On a system where i already have meson/sphinx installed, it > all just worked fine AFAICT. > > With regards, > Daniel > -- > |: https://berrange.com -o- > https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- > https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- > https://www.instagram.com/dberrange :| > >
On Tue, Apr 25, 2023, 1:22 PM John Snow <jsnow@redhat.com> wrote: > > > On Tue, Apr 25, 2023, 1:17 PM Daniel P. Berrangé <berrange@redhat.com> > wrote: > >> On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote: >> > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409 >> > (All green, except Python self-tests, see below) >> > >> > This patch series creates a mandatory python virtual environment >> > ("venv") during configure time and uses it to ensure the availability of >> > meson and sphinx. >> > >> > See https://www.qemu.org/2023/03/24/python/ for details. The summary is >> > that the goal of this series is to ensure that the `python` used to run >> > meson is the same `python` used to run Sphinx, tests, and any build-time >> > python scripting we have. As it stands, meson and sphinx (and their >> > extensions) *may* run in a different python environment than the one >> > configured and chosen by the user at configure/build time. >> > >> > The effective change of this series is that QEMU will now >> > unconditionally create a venv at configure-time and will ensure that >> > meson (and sphinx, if docs are enabled) are available through that venv. >> > >> > Some important points as a pre-emptive "FAQ": >> > >> > - This venv is unconditionally created and lives at {build_dir}/pyvenv. >> > >> > - The python interpreter used by this venv is always the one identified >> > by configure. (Which in turn is always the one specified by --python >> > or $PYTHON) >> > >> > - *almost* all python scripts in qemu.git executed as part of the build >> > system, meson, sphinx, avocado tests, vm tests or CI are always >> > executed within this venv. >> > >> > (iotests are not yet integrated; I plan to tackle this separately as a >> > follow-up in order to have a more tightly focused scope on that >> > series.) >> > >> > - It remains possible to build and test fully offline. >> > (In most cases, you just need meson and sphinx from your distro's >> repo.) >> > >> > - Distribution packaged 'meson' and 'sphinx' are still utilized whenever >> > possible as the highest preference. >> > >> > - Vendored versions of e.g. 'meson' are always preferred to PyPI >> > versions for speed, repeatability and ensuring tarball builds work >> > as-is offline. >> > >> > (Sphinx will not be vendored, just like it already isn't.) >> > >> > - Missing dependencies, when possible, are fetched and installed >> > on-demand automatically to make developer environments "just work". >> > >> > - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, >> > Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere >> > >> > - No new dependencies (...for most platforms. Debian and NetBSD get an >> > asterisk.) >> > >> > - The meson git submodule is unused after this series and can be >> removed. >> > >> > For reviewers, here's how the series is broken up: >> > >> > Patch 1 is a testing pre-req. Note that even with this patch, >> > 'check-python-minreqs' and 'check-python-tox' CI jobs will both still >> > fail on origin/master because this series requires 3.7+, but >> > origin/master is currently still 3.6+. >> > >> > - python: update pylint configuration >> > >> > Patches 2-8 add the mkvenv script. The first patch checks in the barest >> > essentials, and each subsequent patch adds a workaround or feature one >> > at a time. >> > >> > - python: add mkvenv.py >> > - mkvenv: add console script entry point generation >> > - mkvenv: Add better error message for missing pyexapt module >> > - mkvenv: generate console entry shims from inside the venv >> > - mkvenv: work around broken pip installations on Debian 10 >> > - mkvenv: add nested venv workaround >> > - mkvenv: add ensure subcommand >> > >> > Patches 9-11 modify our testing configuration to add new dependencies as >> > needed. >> > >> > - tests/docker: add python3-venv dependency >> > - tests/vm: Configure netbsd to use Python 3.10 >> > - tests/vm: add py310-expat to NetBSD >> > >> > Patch 12 changes how we package release tarballs. >> > >> > - scripts/make-release: download meson==0.61.5 .whl >> > >> > Patches 13-16 wire mkvenv into configure and tests. >> > >> > - configure: create a python venv unconditionally >> > - configure: use 'mkvenv ensure meson' to bootstrap meson >> > - configure: add --enable-pypi and --disable-pypi >> > - tests: Use configure-provided pyvenv for tests >> > >> > Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these >> > changes could be folded earlier in the series (like the diagnose() >> > patch), but I'm keeping it separate for review for now. >> > >> > - configure: move --enable-docs and --disable-docs back to configure >> > - mkvenv: add diagnose() method for ensure() failures >> > - configure: use --diagnose option with meson ensure >> > - configure: bootstrap sphinx with mkvenv >> >> I'm not sure this last bit is working. >> >> I uninstalled meson and python3-sphinx from my F38 host and ran >> configure --target-list=x86_64-softmmu and got this: >> >> $ ./configure --target-list=x86_64-softmmu >> Using './build' as the directory for build output >> python determined to be '/usr/bin/python3' >> python version: Python 3.11.3 >> MKVENV pyvenv >> Configured python as >> '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B' >> MKVENV ensure meson>=0.61.5 >> WARNING: Skipping >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to >> invalid metadata entry 'name' >> WARNING: Skipping >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to >> invalid metadata entry 'name' >> WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is >> ignored: it is neither a file nor a directory. >> ERROR: Could not find a version that satisfies the requirement >> meson>=0.61.5 (from versions: none) >> ERROR: No matching distribution found for meson>=0.61.5 >> WARNING: Skipping >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to >> invalid metadata entry 'name' >> WARNING: Skipping >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to >> invalid metadata entry 'name' >> WARNING: Skipping >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to >> invalid metadata entry 'name' >> MKVENV ensure sphinx>=1.6.0 >> WARNING: Skipping >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to >> invalid metadata entry 'name' >> WARNING: Skipping >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to >> invalid metadata entry 'name' >> ERROR: Could not find a version that satisfies the requirement >> sphinx>=1.6.0 (from versions: none) >> ERROR: No matching distribution found for sphinx>=1.6.0 >> >> *** Ouch! *** >> >> Could not ensure availability of 'sphinx>=1.6.0': >> • Python package 'sphinx' was not found nor installed. >> • No local package directory was searched. >> • mkvenv was configured to operate offline and did not check PyPI. >> >> >> Sphinx not found/usable, disabling docs. >> MKVENV ok! >> >> >> >> It says mkvenv was configured to run offline, but I didn't do >> anything. I thought the intention was for developers it would >> live download from PyPI ? >> > > Ah. So... with enable pypi being the default and with docs set to "auto", > I actually fall back to not installing sphinx from pypi *by default*. It > tries to locate it on your system and will enable docs if it can, but it > doesn't try too hard and doesn't get upset if it fails. > (Though all of those errors and warnings sure are noisy for meaning "we couldn't find an optional package". mkvenv just doesn't distinguish between optional and required right now so it just leaves it to the caller to interpret. Any suggestions for improving this?) > Try passing --enable-docs to convince the build system you'd really > definitely like docs, and it'll force the pypi access. > I can make it try PyPI in this double-default case too, I was just being very conservative about when we tried PyPI - this solution is fairly reluctant by design to do it. If we all agree that it shouldn't be so reluctant, and anyone who doesnt want it at all should just pass --disable-pypi, I can make that change easily. I wonder if I should make an "auto" setting for the pypi access which represents this behavior ("only if I have to"), and make "enable" more aggressive (consult pypi even for optional features.) I was just being conservative on the first pass. Trying to strike a balance between convenience, speed, and least surprise. > >> >> On a system where i already have meson/sphinx installed, it >> all just worked fine AFAICT. >> >> With regards, >> Daniel >> -- >> |: https://berrange.com -o- >> https://www.flickr.com/photos/dberrange :| >> |: https://libvirt.org -o- >> https://fstop138.berrange.com :| >> |: https://entangle-photo.org -o- >> https://www.instagram.com/dberrange :| >> >>
On Tue, Apr 25, 2023 at 01:22:38PM -0400, John Snow wrote: > On Tue, Apr 25, 2023, 1:17 PM Daniel P. Berrangé <berrange@redhat.com> > wrote: > > > On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote: > > > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409 > > > (All green, except Python self-tests, see below) > > > > > > This patch series creates a mandatory python virtual environment > > > ("venv") during configure time and uses it to ensure the availability of > > > meson and sphinx. > > > > > > See https://www.qemu.org/2023/03/24/python/ for details. The summary is > > > that the goal of this series is to ensure that the `python` used to run > > > meson is the same `python` used to run Sphinx, tests, and any build-time > > > python scripting we have. As it stands, meson and sphinx (and their > > > extensions) *may* run in a different python environment than the one > > > configured and chosen by the user at configure/build time. > > > > > > The effective change of this series is that QEMU will now > > > unconditionally create a venv at configure-time and will ensure that > > > meson (and sphinx, if docs are enabled) are available through that venv. > > > > > > Some important points as a pre-emptive "FAQ": > > > > > > - This venv is unconditionally created and lives at {build_dir}/pyvenv. > > > > > > - The python interpreter used by this venv is always the one identified > > > by configure. (Which in turn is always the one specified by --python > > > or $PYTHON) > > > > > > - *almost* all python scripts in qemu.git executed as part of the build > > > system, meson, sphinx, avocado tests, vm tests or CI are always > > > executed within this venv. > > > > > > (iotests are not yet integrated; I plan to tackle this separately as a > > > follow-up in order to have a more tightly focused scope on that > > > series.) > > > > > > - It remains possible to build and test fully offline. > > > (In most cases, you just need meson and sphinx from your distro's > > repo.) > > > > > > - Distribution packaged 'meson' and 'sphinx' are still utilized whenever > > > possible as the highest preference. > > > > > > - Vendored versions of e.g. 'meson' are always preferred to PyPI > > > versions for speed, repeatability and ensuring tarball builds work > > > as-is offline. > > > > > > (Sphinx will not be vendored, just like it already isn't.) > > > > > > - Missing dependencies, when possible, are fetched and installed > > > on-demand automatically to make developer environments "just work". > > > > > > - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, > > > Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere > > > > > > - No new dependencies (...for most platforms. Debian and NetBSD get an > > > asterisk.) > > > > > > - The meson git submodule is unused after this series and can be removed. > > > > > > For reviewers, here's how the series is broken up: > > > > > > Patch 1 is a testing pre-req. Note that even with this patch, > > > 'check-python-minreqs' and 'check-python-tox' CI jobs will both still > > > fail on origin/master because this series requires 3.7+, but > > > origin/master is currently still 3.6+. > > > > > > - python: update pylint configuration > > > > > > Patches 2-8 add the mkvenv script. The first patch checks in the barest > > > essentials, and each subsequent patch adds a workaround or feature one > > > at a time. > > > > > > - python: add mkvenv.py > > > - mkvenv: add console script entry point generation > > > - mkvenv: Add better error message for missing pyexapt module > > > - mkvenv: generate console entry shims from inside the venv > > > - mkvenv: work around broken pip installations on Debian 10 > > > - mkvenv: add nested venv workaround > > > - mkvenv: add ensure subcommand > > > > > > Patches 9-11 modify our testing configuration to add new dependencies as > > > needed. > > > > > > - tests/docker: add python3-venv dependency > > > - tests/vm: Configure netbsd to use Python 3.10 > > > - tests/vm: add py310-expat to NetBSD > > > > > > Patch 12 changes how we package release tarballs. > > > > > > - scripts/make-release: download meson==0.61.5 .whl > > > > > > Patches 13-16 wire mkvenv into configure and tests. > > > > > > - configure: create a python venv unconditionally > > > - configure: use 'mkvenv ensure meson' to bootstrap meson > > > - configure: add --enable-pypi and --disable-pypi > > > - tests: Use configure-provided pyvenv for tests > > > > > > Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these > > > changes could be folded earlier in the series (like the diagnose() > > > patch), but I'm keeping it separate for review for now. > > > > > > - configure: move --enable-docs and --disable-docs back to configure > > > - mkvenv: add diagnose() method for ensure() failures > > > - configure: use --diagnose option with meson ensure > > > - configure: bootstrap sphinx with mkvenv > > > > I'm not sure this last bit is working. > > > > I uninstalled meson and python3-sphinx from my F38 host and ran > > configure --target-list=x86_64-softmmu and got this: > > > > $ ./configure --target-list=x86_64-softmmu > > Using './build' as the directory for build output > > python determined to be '/usr/bin/python3' > > python version: Python 3.11.3 > > MKVENV pyvenv > > Configured python as > > '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B' > > MKVENV ensure meson>=0.61.5 > > WARNING: Skipping > > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > > invalid metadata entry 'name' > > WARNING: Skipping > > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > > invalid metadata entry 'name' > > WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is > > ignored: it is neither a file nor a directory. > > ERROR: Could not find a version that satisfies the requirement > > meson>=0.61.5 (from versions: none) > > ERROR: No matching distribution found for meson>=0.61.5 > > WARNING: Skipping > > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > > invalid metadata entry 'name' > > WARNING: Skipping > > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > > invalid metadata entry 'name' > > WARNING: Skipping > > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > > invalid metadata entry 'name' > > MKVENV ensure sphinx>=1.6.0 > > WARNING: Skipping > > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > > invalid metadata entry 'name' > > WARNING: Skipping > > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > > invalid metadata entry 'name' > > ERROR: Could not find a version that satisfies the requirement > > sphinx>=1.6.0 (from versions: none) > > ERROR: No matching distribution found for sphinx>=1.6.0 > > > > *** Ouch! *** > > > > Could not ensure availability of 'sphinx>=1.6.0': > > • Python package 'sphinx' was not found nor installed. > > • No local package directory was searched. > > • mkvenv was configured to operate offline and did not check PyPI. > > > > > > Sphinx not found/usable, disabling docs. > > MKVENV ok! > > > > > > > > It says mkvenv was configured to run offline, but I didn't do > > anything. I thought the intention was for developers it would > > live download from PyPI ? > > > > Ah. So... with enable pypi being the default and with docs set to "auto", I > actually fall back to not installing sphinx from pypi *by default*. It > tries to locate it on your system and will enable docs if it can, but it > doesn't try too hard and doesn't get upset if it fails. Ah right, yes, I think that behaviour makes sense. > Try passing --enable-docs to convince the build system you'd really > definitely like docs, and it'll force the pypi access. Ok will try. With regards, Daniel
On Tue, Apr 25, 2023 at 01:34:52PM -0400, John Snow wrote: > On Tue, Apr 25, 2023, 1:22 PM John Snow <jsnow@redhat.com> wrote: > > > > > > > On Tue, Apr 25, 2023, 1:17 PM Daniel P. Berrangé <berrange@redhat.com> > > wrote: > > > >> On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote: > >> > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409 > >> > (All green, except Python self-tests, see below) > >> > > >> > This patch series creates a mandatory python virtual environment > >> > ("venv") during configure time and uses it to ensure the availability of > >> > meson and sphinx. > >> > > >> > See https://www.qemu.org/2023/03/24/python/ for details. The summary is > >> > that the goal of this series is to ensure that the `python` used to run > >> > meson is the same `python` used to run Sphinx, tests, and any build-time > >> > python scripting we have. As it stands, meson and sphinx (and their > >> > extensions) *may* run in a different python environment than the one > >> > configured and chosen by the user at configure/build time. > >> > > >> > The effective change of this series is that QEMU will now > >> > unconditionally create a venv at configure-time and will ensure that > >> > meson (and sphinx, if docs are enabled) are available through that venv. > >> > > >> > Some important points as a pre-emptive "FAQ": > >> > > >> > - This venv is unconditionally created and lives at {build_dir}/pyvenv. > >> > > >> > - The python interpreter used by this venv is always the one identified > >> > by configure. (Which in turn is always the one specified by --python > >> > or $PYTHON) > >> > > >> > - *almost* all python scripts in qemu.git executed as part of the build > >> > system, meson, sphinx, avocado tests, vm tests or CI are always > >> > executed within this venv. > >> > > >> > (iotests are not yet integrated; I plan to tackle this separately as a > >> > follow-up in order to have a more tightly focused scope on that > >> > series.) > >> > > >> > - It remains possible to build and test fully offline. > >> > (In most cases, you just need meson and sphinx from your distro's > >> repo.) > >> > > >> > - Distribution packaged 'meson' and 'sphinx' are still utilized whenever > >> > possible as the highest preference. > >> > > >> > - Vendored versions of e.g. 'meson' are always preferred to PyPI > >> > versions for speed, repeatability and ensuring tarball builds work > >> > as-is offline. > >> > > >> > (Sphinx will not be vendored, just like it already isn't.) > >> > > >> > - Missing dependencies, when possible, are fetched and installed > >> > on-demand automatically to make developer environments "just work". > >> > > >> > - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, > >> > Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere > >> > > >> > - No new dependencies (...for most platforms. Debian and NetBSD get an > >> > asterisk.) > >> > > >> > - The meson git submodule is unused after this series and can be > >> removed. > >> > > >> > For reviewers, here's how the series is broken up: > >> > > >> > Patch 1 is a testing pre-req. Note that even with this patch, > >> > 'check-python-minreqs' and 'check-python-tox' CI jobs will both still > >> > fail on origin/master because this series requires 3.7+, but > >> > origin/master is currently still 3.6+. > >> > > >> > - python: update pylint configuration > >> > > >> > Patches 2-8 add the mkvenv script. The first patch checks in the barest > >> > essentials, and each subsequent patch adds a workaround or feature one > >> > at a time. > >> > > >> > - python: add mkvenv.py > >> > - mkvenv: add console script entry point generation > >> > - mkvenv: Add better error message for missing pyexapt module > >> > - mkvenv: generate console entry shims from inside the venv > >> > - mkvenv: work around broken pip installations on Debian 10 > >> > - mkvenv: add nested venv workaround > >> > - mkvenv: add ensure subcommand > >> > > >> > Patches 9-11 modify our testing configuration to add new dependencies as > >> > needed. > >> > > >> > - tests/docker: add python3-venv dependency > >> > - tests/vm: Configure netbsd to use Python 3.10 > >> > - tests/vm: add py310-expat to NetBSD > >> > > >> > Patch 12 changes how we package release tarballs. > >> > > >> > - scripts/make-release: download meson==0.61.5 .whl > >> > > >> > Patches 13-16 wire mkvenv into configure and tests. > >> > > >> > - configure: create a python venv unconditionally > >> > - configure: use 'mkvenv ensure meson' to bootstrap meson > >> > - configure: add --enable-pypi and --disable-pypi > >> > - tests: Use configure-provided pyvenv for tests > >> > > >> > Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these > >> > changes could be folded earlier in the series (like the diagnose() > >> > patch), but I'm keeping it separate for review for now. > >> > > >> > - configure: move --enable-docs and --disable-docs back to configure > >> > - mkvenv: add diagnose() method for ensure() failures > >> > - configure: use --diagnose option with meson ensure > >> > - configure: bootstrap sphinx with mkvenv > >> > >> I'm not sure this last bit is working. > >> > >> I uninstalled meson and python3-sphinx from my F38 host and ran > >> configure --target-list=x86_64-softmmu and got this: > >> > >> $ ./configure --target-list=x86_64-softmmu > >> Using './build' as the directory for build output > >> python determined to be '/usr/bin/python3' > >> python version: Python 3.11.3 > >> MKVENV pyvenv > >> Configured python as > >> '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B' > >> MKVENV ensure meson>=0.61.5 > >> WARNING: Skipping > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > >> invalid metadata entry 'name' > >> WARNING: Skipping > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > >> invalid metadata entry 'name' > >> WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is > >> ignored: it is neither a file nor a directory. > >> ERROR: Could not find a version that satisfies the requirement > >> meson>=0.61.5 (from versions: none) > >> ERROR: No matching distribution found for meson>=0.61.5 > >> WARNING: Skipping > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > >> invalid metadata entry 'name' > >> WARNING: Skipping > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > >> invalid metadata entry 'name' > >> WARNING: Skipping > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > >> invalid metadata entry 'name' > >> MKVENV ensure sphinx>=1.6.0 > >> WARNING: Skipping > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > >> invalid metadata entry 'name' > >> WARNING: Skipping > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > >> invalid metadata entry 'name' > >> ERROR: Could not find a version that satisfies the requirement > >> sphinx>=1.6.0 (from versions: none) > >> ERROR: No matching distribution found for sphinx>=1.6.0 > >> > >> *** Ouch! *** > >> > >> Could not ensure availability of 'sphinx>=1.6.0': > >> • Python package 'sphinx' was not found nor installed. > >> • No local package directory was searched. > >> • mkvenv was configured to operate offline and did not check PyPI. > >> > >> > >> Sphinx not found/usable, disabling docs. > >> MKVENV ok! > >> > >> > >> > >> It says mkvenv was configured to run offline, but I didn't do > >> anything. I thought the intention was for developers it would > >> live download from PyPI ? > >> > > > > Ah. So... with enable pypi being the default and with docs set to "auto", > > I actually fall back to not installing sphinx from pypi *by default*. It > > tries to locate it on your system and will enable docs if it can, but it > > doesn't try too hard and doesn't get upset if it fails. > > > > (Though all of those errors and warnings sure are noisy for meaning "we > couldn't find an optional package". mkvenv just doesn't distinguish between > optional and required right now so it just leaves it to the caller to > interpret. Any suggestions for improving this?) > > > > Try passing --enable-docs to convince the build system you'd really > > definitely like docs, and it'll force the pypi access. > > > > I can make it try PyPI in this double-default case too, I was just being > very conservative about when we tried PyPI - this solution is fairly > reluctant by design to do it. > > If we all agree that it shouldn't be so reluctant, and anyone who doesnt > want it at all should just pass --disable-pypi, I can make that change > easily. > > I wonder if I should make an "auto" setting for the pypi access which > represents this behavior ("only if I have to"), and make "enable" more > aggressive (consult pypi even for optional features.) > > I was just being conservative on the first pass. Trying to strike a balance > between convenience, speed, and least surprise. How about having --enable-pypi never|auto|force with the following semantics for --enable-docs + --enable-pypi * docs=no - pypi never used * docs=auto + pypi=never => docs only enable if sphinx is already installed locally, otherwise disabled * docs=auto + pypi=auto => docs enable if sphinx is already installed locally, or can download from pypi as fallback * docs=auto + pypi=force => always download sphinx from pypi * docs=yes + pypi=never => ERROR if sphinx is not already installed locally * docs=yes + pypi=auto => docs enable if sphinx is already installed locally, or can download from pypi as fallback * docs=yes + pypi=force => always download sphinx from pypi So eg distros could use pypi=never, devs would use pypi=auto mostly, while CI might use pypi=force to test specific versions indepenant of distros ? With regards, Daniel
On Tue, Apr 25, 2023 at 2:10 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Tue, Apr 25, 2023 at 01:34:52PM -0400, John Snow wrote: > > On Tue, Apr 25, 2023, 1:22 PM John Snow <jsnow@redhat.com> wrote: > > > > > > > > > > > On Tue, Apr 25, 2023, 1:17 PM Daniel P. Berrangé <berrange@redhat.com> > > > wrote: > > > > > >> On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote: > > >> > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409 > > >> > (All green, except Python self-tests, see below) > > >> > > > >> > This patch series creates a mandatory python virtual environment > > >> > ("venv") during configure time and uses it to ensure the availability of > > >> > meson and sphinx. > > >> > > > >> > See https://www.qemu.org/2023/03/24/python/ for details. The summary is > > >> > that the goal of this series is to ensure that the `python` used to run > > >> > meson is the same `python` used to run Sphinx, tests, and any build-time > > >> > python scripting we have. As it stands, meson and sphinx (and their > > >> > extensions) *may* run in a different python environment than the one > > >> > configured and chosen by the user at configure/build time. > > >> > > > >> > The effective change of this series is that QEMU will now > > >> > unconditionally create a venv at configure-time and will ensure that > > >> > meson (and sphinx, if docs are enabled) are available through that venv. > > >> > > > >> > Some important points as a pre-emptive "FAQ": > > >> > > > >> > - This venv is unconditionally created and lives at {build_dir}/pyvenv. > > >> > > > >> > - The python interpreter used by this venv is always the one identified > > >> > by configure. (Which in turn is always the one specified by --python > > >> > or $PYTHON) > > >> > > > >> > - *almost* all python scripts in qemu.git executed as part of the build > > >> > system, meson, sphinx, avocado tests, vm tests or CI are always > > >> > executed within this venv. > > >> > > > >> > (iotests are not yet integrated; I plan to tackle this separately as a > > >> > follow-up in order to have a more tightly focused scope on that > > >> > series.) > > >> > > > >> > - It remains possible to build and test fully offline. > > >> > (In most cases, you just need meson and sphinx from your distro's > > >> repo.) > > >> > > > >> > - Distribution packaged 'meson' and 'sphinx' are still utilized whenever > > >> > possible as the highest preference. > > >> > > > >> > - Vendored versions of e.g. 'meson' are always preferred to PyPI > > >> > versions for speed, repeatability and ensuring tarball builds work > > >> > as-is offline. > > >> > > > >> > (Sphinx will not be vendored, just like it already isn't.) > > >> > > > >> > - Missing dependencies, when possible, are fetched and installed > > >> > on-demand automatically to make developer environments "just work". > > >> > > > >> > - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, > > >> > Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere > > >> > > > >> > - No new dependencies (...for most platforms. Debian and NetBSD get an > > >> > asterisk.) > > >> > > > >> > - The meson git submodule is unused after this series and can be > > >> removed. > > >> > > > >> > For reviewers, here's how the series is broken up: > > >> > > > >> > Patch 1 is a testing pre-req. Note that even with this patch, > > >> > 'check-python-minreqs' and 'check-python-tox' CI jobs will both still > > >> > fail on origin/master because this series requires 3.7+, but > > >> > origin/master is currently still 3.6+. > > >> > > > >> > - python: update pylint configuration > > >> > > > >> > Patches 2-8 add the mkvenv script. The first patch checks in the barest > > >> > essentials, and each subsequent patch adds a workaround or feature one > > >> > at a time. > > >> > > > >> > - python: add mkvenv.py > > >> > - mkvenv: add console script entry point generation > > >> > - mkvenv: Add better error message for missing pyexapt module > > >> > - mkvenv: generate console entry shims from inside the venv > > >> > - mkvenv: work around broken pip installations on Debian 10 > > >> > - mkvenv: add nested venv workaround > > >> > - mkvenv: add ensure subcommand > > >> > > > >> > Patches 9-11 modify our testing configuration to add new dependencies as > > >> > needed. > > >> > > > >> > - tests/docker: add python3-venv dependency > > >> > - tests/vm: Configure netbsd to use Python 3.10 > > >> > - tests/vm: add py310-expat to NetBSD > > >> > > > >> > Patch 12 changes how we package release tarballs. > > >> > > > >> > - scripts/make-release: download meson==0.61.5 .whl > > >> > > > >> > Patches 13-16 wire mkvenv into configure and tests. > > >> > > > >> > - configure: create a python venv unconditionally > > >> > - configure: use 'mkvenv ensure meson' to bootstrap meson > > >> > - configure: add --enable-pypi and --disable-pypi > > >> > - tests: Use configure-provided pyvenv for tests > > >> > > > >> > Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these > > >> > changes could be folded earlier in the series (like the diagnose() > > >> > patch), but I'm keeping it separate for review for now. > > >> > > > >> > - configure: move --enable-docs and --disable-docs back to configure > > >> > - mkvenv: add diagnose() method for ensure() failures > > >> > - configure: use --diagnose option with meson ensure > > >> > - configure: bootstrap sphinx with mkvenv > > >> > > >> I'm not sure this last bit is working. > > >> > > >> I uninstalled meson and python3-sphinx from my F38 host and ran > > >> configure --target-list=x86_64-softmmu and got this: > > >> > > >> $ ./configure --target-list=x86_64-softmmu > > >> Using './build' as the directory for build output > > >> python determined to be '/usr/bin/python3' > > >> python version: Python 3.11.3 > > >> MKVENV pyvenv > > >> Configured python as > > >> '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B' > > >> MKVENV ensure meson>=0.61.5 > > >> WARNING: Skipping > > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > > >> invalid metadata entry 'name' > > >> WARNING: Skipping > > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > > >> invalid metadata entry 'name' > > >> WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is > > >> ignored: it is neither a file nor a directory. > > >> ERROR: Could not find a version that satisfies the requirement > > >> meson>=0.61.5 (from versions: none) > > >> ERROR: No matching distribution found for meson>=0.61.5 > > >> WARNING: Skipping > > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > > >> invalid metadata entry 'name' > > >> WARNING: Skipping > > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > > >> invalid metadata entry 'name' > > >> WARNING: Skipping > > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > > >> invalid metadata entry 'name' > > >> MKVENV ensure sphinx>=1.6.0 > > >> WARNING: Skipping > > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > > >> invalid metadata entry 'name' > > >> WARNING: Skipping > > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to > > >> invalid metadata entry 'name' > > >> ERROR: Could not find a version that satisfies the requirement > > >> sphinx>=1.6.0 (from versions: none) > > >> ERROR: No matching distribution found for sphinx>=1.6.0 > > >> > > >> *** Ouch! *** > > >> > > >> Could not ensure availability of 'sphinx>=1.6.0': > > >> • Python package 'sphinx' was not found nor installed. > > >> • No local package directory was searched. > > >> • mkvenv was configured to operate offline and did not check PyPI. > > >> > > >> > > >> Sphinx not found/usable, disabling docs. > > >> MKVENV ok! > > >> > > >> > > >> > > >> It says mkvenv was configured to run offline, but I didn't do > > >> anything. I thought the intention was for developers it would > > >> live download from PyPI ? > > >> > > > > > > Ah. So... with enable pypi being the default and with docs set to "auto", > > > I actually fall back to not installing sphinx from pypi *by default*. It > > > tries to locate it on your system and will enable docs if it can, but it > > > doesn't try too hard and doesn't get upset if it fails. > > > > > > > (Though all of those errors and warnings sure are noisy for meaning "we > > couldn't find an optional package". mkvenv just doesn't distinguish between > > optional and required right now so it just leaves it to the caller to > > interpret. Any suggestions for improving this?) > > > > > > > Try passing --enable-docs to convince the build system you'd really > > > definitely like docs, and it'll force the pypi access. > > > > > > > I can make it try PyPI in this double-default case too, I was just being > > very conservative about when we tried PyPI - this solution is fairly > > reluctant by design to do it. > > > > If we all agree that it shouldn't be so reluctant, and anyone who doesnt > > want it at all should just pass --disable-pypi, I can make that change > > easily. > > > > I wonder if I should make an "auto" setting for the pypi access which > > represents this behavior ("only if I have to"), and make "enable" more > > aggressive (consult pypi even for optional features.) > > > > I was just being conservative on the first pass. Trying to strike a balance > > between convenience, speed, and least surprise. > > How about having --enable-pypi never|auto|force with the following > semantics for --enable-docs + --enable-pypi > > > * docs=no - pypi never used > > * docs=auto + pypi=never => docs only enable if sphinx is already > installed locally, otherwise disabled > > * docs=auto + pypi=auto => docs enable if sphinx is already > installed locally, or can download from > pypi as fallback > As long as this doesn't cause needless trouble for existing configure invocations baked into scripts and the like. It's a bit more aggressive about enabling docs than we have been in the past, but maybe that's not a bad thing. > * docs=auto + pypi=force => always download sphinx from pypi > So if you already have Sphinx, this should perform an upgrade to the latest version? > > * docs=yes + pypi=never => ERROR if sphinx is not already > installed locally > > * docs=yes + pypi=auto => docs enable if sphinx is already > installed locally, or can download from > pypi as fallback > > * docs=yes + pypi=force => always download sphinx from pypi Yeah, there's room for settings like these, and the above mostly makes sense to me. Paolo and I had written up a more elaborate set of flags at one point under the premise of integrating testing, too -- for instance, pre-loading the testing dependencies for iotests (local qemu package, qemu.qmp, etc) or for avocado tests (avocado-framework and other deps needed for those tests). I wanted to try and check in a "minimum viable" version of this patch set first close to the 8.1 tree opening to iron out any bugs in the core feature and then I'd work to expand the flags and capability of the system as differential patches. So, I think changing --enable-pypi too much is beyond the scope of the current patchset, but it's on my mind for what will follow almost immediately after it. With that in mind, we can brainstorm a little here and throw some darts at the wall: The version of flags we originally came up with was this: --python=... # runtime used to create venv --enable-pip-groups=testing,devel,avocado,meson,sphinx --enable-pip=now # install all python deps now --enable-pip=on-demand # install qemu.git/meson/sphinx, delay the rest --enable-pip=no # offline --{enable,disable}-isolated-venv # let venv use system/distro if disable I may have pulled us in a slightly different direction with the version of the patches I actually sent here today, but the key ideas here were: - The ability to opt into or out of different package groups at configure-time. meson is always required, docs/sphinx is optional, and the other three groups are testing related. (testing: iotests and qemu.qmp, devel: python self-tests, avocado: avocado tests.) - The ability to decide when packages get installed to the venv; either at configure time ("now") or as-needed ("on-demand") or functionally never ("no") - The ability to enable or disable venv isolation (turning on or off system packages). I think I abandoned the idea of configuring the venv isolation and have it set to "always off". We may revisit this later, but I think for this series it's fine to have glossed over it. I also skipped over the idea of having package installation being "now", "on-demand" or "no" -- this patch-set more or less is using a hybrid of "now" and "on-demand" where meson and sphinx are "now" but avocado is "on-demand". Unifying that discrepancy can occur in subsequent patches as it closely resembles what we had in 8.0 and earlier. I also didn't implement configurable package groups at all yet. meson and sphinx are just implicitly selected; everything else is not yet thoroughly integrated. The testing portions of this -- avocado, qemu.qmp, and python self-tests -- are not as integrated as the build portions, sphinx and meson. That will likely be the focus of my next patchset when I move to integrate iotests and actually set about the work of finally dropping the qemu.qmp package from the qemu.git source tree. I guess before I spend *too* much time writing that series (for the fourth time...!) I wanted to try and solidify the basics here to give me a nice solid foundation. --js > > > > So eg distros could use pypi=never, devs would use pypi=auto mostly, > while CI might use pypi=force to test specific versions indepenant > of distros ? > Sounds reasonable in general, I think, but I have some questions about what 'force' does, exactly. > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
On 4/24/23 22:02, John Snow wrote: > Some important points as a pre-emptive "FAQ": > > - This venv is unconditionally created and lives at {build_dir}/pyvenv. > > - The python interpreter used by this venv is always the one identified > by configure. (Which in turn is always the one specified by --python > or $PYTHON) > > -*almost* all python scripts in qemu.git executed as part of the build > system, meson, sphinx, avocado tests, vm tests or CI are always > executed within this venv. > > (iotests are not yet integrated; I plan to tackle this separately as a > follow-up in order to have a more tightly focused scope on that > series.) > > - It remains possible to build and test fully offline. > (In most cases, you just need meson and sphinx from your distro's repo.) > > - Distribution packaged 'meson' and 'sphinx' are still utilized whenever > possible as the highest preference. > > - Vendored versions of e.g. 'meson' are always preferred to PyPI > versions for speed, repeatability and ensuring tarball builds work > as-is offline. > > (Sphinx will not be vendored, just like it already isn't.) > > - Missing dependencies, when possible, are fetched and installed > on-demand automatically to make developer environments "just work". > > - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, > Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere > > - No new dependencies (...for most platforms. Debian and NetBSD get an > asterisk.) > > - The meson git submodule is unused after this series and can be removed. Thanks, this looks pretty good. Some changes I'd make for the non-RFC version: - I think we should just check in the meson wheel (which also removes the need for patch 12, so it can be done in its stead) and remove the submodule - The verbosity of mkvenv.py can be tuned down and most prints replaced with logger.info() or logger.debug() - While I agree with keeping patch 18 separate, I would move it earlier so that patch 19 can be squashed into patch 14 - I am ambivalent about keeping --enable/--disable-pypi in the first committed patchset, but in any case I would move patches 16 and 20 before patch 15 Paolo
On Tue, Apr 25, 2023 at 02:58:53PM -0400, John Snow wrote: > On Tue, Apr 25, 2023 at 2:10 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > How about having --enable-pypi never|auto|force with the following > > semantics for --enable-docs + --enable-pypi > > > > > > * docs=no - pypi never used > > > > * docs=auto + pypi=never => docs only enable if sphinx is already > > installed locally, otherwise disabled > > > > * docs=auto + pypi=auto => docs enable if sphinx is already > > installed locally, or can download from > > pypi as fallback > > > > As long as this doesn't cause needless trouble for existing configure > invocations baked into scripts and the like. It's a bit more > aggressive about enabling docs than we have been in the past, but > maybe that's not a bad thing. > > > * docs=auto + pypi=force => always download sphinx from pypi > > > > So if you already have Sphinx, this should perform an upgrade to the > latest version? Essentially I meant 'force' to mean *never* use the host python installation packages. Always install all the deps in the venv, even if they exist in the host with sufficient version met. > > * docs=yes + pypi=never => ERROR if sphinx is not already > > installed locally > > > > * docs=yes + pypi=auto => docs enable if sphinx is already > > installed locally, or can download from > > pypi as fallback > > > > * docs=yes + pypi=force => always download sphinx from pypi > > Yeah, there's room for settings like these, and the above mostly makes > sense to me. > > Paolo and I had written up a more elaborate set of flags at one point > under the premise of integrating testing, too -- for instance, > pre-loading the testing dependencies for iotests (local qemu package, > qemu.qmp, etc) or for avocado tests (avocado-framework and other deps > needed for those tests). I wanted to try and check in a "minimum > viable" version of this patch set first close to the 8.1 tree opening > to iron out any bugs in the core feature and then I'd work to expand > the flags and capability of the system as differential patches. > > So, I think changing --enable-pypi too much is beyond the scope of the > current patchset, but it's on my mind for what will follow almost > immediately after it. With that in mind, we can brainstorm a little > here and throw some darts at the wall: Yep, I don't consider it a pre-requisite for this series. > > The version of flags we originally came up with was this: > > --python=... # runtime used to create venv > --enable-pip-groups=testing,devel,avocado,meson,sphinx > --enable-pip=now # install all python deps now > --enable-pip=on-demand # install qemu.git/meson/sphinx, delay the rest > --enable-pip=no # offline > --{enable,disable}-isolated-venv # let venv use system/distro if disable This feels like a bit of overkill to me, and would create a hell of a lot of combinations to test if you expand the matrix of options. > I may have pulled us in a slightly different direction with the > version of the patches I actually sent here today, but the key ideas > here were: > > - The ability to opt into or out of different package groups at > configure-time. meson is always required, docs/sphinx is optional, and > the other three groups are testing related. (testing: iotests and > qemu.qmp, devel: python self-tests, avocado: avocado tests.) I think this is especially overkill. While you can probably come up with some scenarios where you could use this fine grained selection, I don't think it would be very compelling. > - The ability to decide when packages get installed to the venv; > either at configure time ("now") or as-needed ("on-demand") or > functionally never ("no") The distinction between now vs on-demand is really just about avoiding the overhead of downloading bits that you might not end up using. eg not downloading avocado unless you will be running 'make check-avocado'. That's saving a few 10s or 100s of KB of download, but doesn't feel like it'd make a noticable difference in the overall QEMU build time. THe 'now' and 'no' options look sufficient (corresponding to the 'auto' and 'never' options I suggested above) > - The ability to enable or disable venv isolation (turning on or off > system packages). Corresponds to the 'force' option I suggested above. > > I think I abandoned the idea of configuring the venv isolation and > have it set to "always off". We may revisit this later, but I think > for this series it's fine to have glossed over it. > I also skipped over the idea of having package installation being > "now", "on-demand" or "no" -- this patch-set more or less is using a > hybrid of "now" and "on-demand" where meson and sphinx are "now" but > avocado is "on-demand". Unifying that discrepancy can occur in > subsequent patches as it closely resembles what we had in 8.0 and > earlier. I guess the hybrid of 'now' and 'on-demand' could equally well map to what I suggested by 'auto'. > > So eg distros could use pypi=never, devs would use pypi=auto mostly, > > while CI might use pypi=force to test specific versions indepenant > > of distros ? > > > > Sounds reasonable in general, I think, but I have some questions about > what 'force' does, exactly. Turning of system packages in the venv, and always installing everything from pip. With regards, Daniel
On 4/26/23 10:21, Daniel P. Berrangé wrote: >> So if you already have Sphinx, this should perform an upgrade to the >> latest version? > > Essentially I meant 'force' to mean*never* use the host python > installation packages. Always install all the deps in the venv, > even if they exist in the host with sufficient version met. I think this is essentially --enable-isolated-venv. I don't think there is a usecase for "let the venv use system packages, but override them with pip right away". >> >> --python=... # runtime used to create venv >> --enable-pip-groups=testing,devel,avocado,meson,sphinx >> --enable-pip=now # install all python deps now >> --enable-pip=on-demand # install qemu.git/meson/sphinx, delay the rest >> --enable-pip=no # offline >> --{enable,disable}-isolated-venv # let venv use system/distro if disable > > This feels like a bit of overkill to me, and would create a hell > of a lot of combinations to test if you expand the matrix of > options. Yeah, this is a bit overkill. I think we can reduce it to three cases, corresponding to: - --enable-pypi --enable-isolated-venv - use pip to install everything, including for options in "auto" state (e.g. would install sphinx without --enable-docs) - --enable-pypi --disable-isolated-venv - use pip to install missing packages. TBD whether to do so for options in "auto" state or only for "enabled" (i.e., TBD whether to install sphinx without --enable-docs). - --disable-pypi (only meaningful for --disable-isolated-venv) - apart from vendored wheels, just use system site packages (same as QEMU <= 8.0) I think we want to hash out this detail first, and thus we should leave online mode out of the non-RFC version. It can be implemented together with isolated mode. Paolo
On 4/26/23 10:05, Paolo Bonzini wrote: >> > > Thanks, this looks pretty good. Some changes I'd make for the non-RFC > version: > > - I think we should just check in the meson wheel (which also removes > the need for patch 12, so it can be done in its stead) and remove the > submodule > > - The verbosity of mkvenv.py can be tuned down and most prints replaced > with logger.info() or logger.debug() > > - While I agree with keeping patch 18 separate, I would move it earlier > so that patch 19 can be squashed into patch 14 > > - I am ambivalent about keeping --enable/--disable-pypi in the first > committed patchset, but in any case I would move patches 16 and 20 > before patch 15 Just one extra thing, since we're changing so much of Python handling and since the code is written, I would keep the Debian 10 workarounds for now, and only drop them after we drop support for 3.6. Paolo
On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote: > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409 > (All green, except Python self-tests, see below) > > This patch series creates a mandatory python virtual environment > ("venv") during configure time and uses it to ensure the availability of > meson and sphinx. > > See https://www.qemu.org/2023/03/24/python/ for details. The summary is > that the goal of this series is to ensure that the `python` used to run > meson is the same `python` used to run Sphinx, tests, and any build-time > python scripting we have. As it stands, meson and sphinx (and their > extensions) *may* run in a different python environment than the one > configured and chosen by the user at configure/build time. I mentioned this when we were chatting on IRC, but to repeat for the general audience.. I think it'd be useful for support purposes if the configure script summary printed details of how we setup the venv. eg perhaps a summary for each python module of whether we added it to the venv, or relied on te site packages: python venv: meson (site), sphinx (venv), avocado (venv) With regards, Daniel
On 4/26/23 10:53, Daniel P. Berrangé wrote: > I think it'd be useful for support purposes if the configure > script summary printed details of how we setup the venv. > eg perhaps a summary for each python module of whether we > added it to the venv, or relied on te site packages: > > python venv: meson (site), sphinx (venv), avocado (venv) Yes, I agree John did a great job with the errors but a little more work is needed to cleanup of mkvenv.py's logging of the "good" case. The installation case is already covered by "pip install"'s output, but we need to have something like: mkvenv: Creating {isolated|non-isolated} virtual environment [based on /home/pbonzini/myvenv] and when creating the console-scripts shims: mkvenv: Found avocado v90.0 Paolo
On Wed, Apr 26, 2023, 4:49 AM Paolo Bonzini <pbonzini@redhat.com> wrote: > On 4/26/23 10:05, Paolo Bonzini wrote: > >> > > > > Thanks, this looks pretty good. Some changes I'd make for the non-RFC > > version: > > > > - I think we should just check in the meson wheel (which also removes > > the need for patch 12, so it can be done in its stead) and remove the > > submodule > OK, if there's no objections. As a consequence, meson will likely never be downloaded from PyPI with the patch set written as-is. No real problem with that, just a difference. > > > - The verbosity of mkvenv.py can be tuned down and most prints replaced > > with logger.info() or logger.debug() >
On Wed, Apr 26, 2023, 5:08 AM Paolo Bonzini <pbonzini@redhat.com> wrote: > On 4/26/23 10:53, Daniel P. Berrangé wrote: > > I think it'd be useful for support purposes if the configure > > script summary printed details of how we setup the venv. > > eg perhaps a summary for each python module of whether we > > added it to the venv, or relied on te site packages: > > > > python venv: meson (site), sphinx (venv), avocado (venv) > OK, I'll try to work this out. At the moment "already had the package" or "installed it from vendored source" is the same pip call, so it's hard to differentiate post-hoc without some further analysis. One reason this is hard is because I don't have a good method for quickly determining "Do we already have $package that fulfills $depspec" except by invoking pip, and that's costly, 500ms at the absolute fastest I have yet managed. 1.2s in more typical cases. Owowowow. So, for speed, I combine "do we already have it" and "can we install it from vendored source" into a single call to amortize pip startup cost. With Python 3.8+ I might be able to inspect the package post-hoc to find where it's installed to and determine if it came from system packages or not. It's easy to tell the user if we used PyPI or not, though. I'll experiment. > Yes, I agree John did a great job with the errors but a little more work > is needed to cleanup of mkvenv.py's logging of the "good" case. In the good case, it worked! you should be happy! :D (kidding.) The > installation case is already covered by "pip install"'s output, but we > need to have something like: > > mkvenv: Creating {isolated|non-isolated} virtual environment [based on > /home/pbonzini/myvenv] > "based on ..." for nested venv case only? > and when creating the console-scripts shims: > > mkvenv: Found avocado v90.0 > Sure. Up to the user at that point to figure out if we used that package or not. e.g. "found meson 0.5.0" might occur even if we require >=1.0. Simple to implement, though. > Paolo > >
On 4/26/23 18:16, John Snow wrote: > > > - I am ambivalent about keeping --enable/--disable-pypi in the first > > committed patchset, but in any case I would move patches 16 and 20 > > before patch 15 > > I might be stubborn but I think I want to keep it in for now. If it > needs redesigned to fit with the other flags you want to add, I think > that's OK. > > if we vendor the whl directly in qemu.git we won't need PyPI for meson, > but it's still useful for Sphinx so I think I'm still leaning towards > keeping it. Yes, it's definitely useful. It's just that unifying $python with sphinx-build's interpreter is as a separate (and earlier) step than introducing PyPI. > I'll try to refactor to keep it at the tail end of the series. Cool, thanks. > Just one extra thing, since we're changing so much of Python handling > and since the code is written, I would keep the Debian 10 workarounds > for now, and only drop them after we drop support for 3.6. > > > This series was written assuming we get to drop 3.6 as a prereq. Is that > not the case? > > Or did you mean to write 3.7 there? Yes, 3.7. Paolo
On 4/26/23 18:32, John Snow wrote: > > mkvenv: Creating {isolated|non-isolated} virtual environment [based on > /home/pbonzini/myvenv] > > "based on ..." for nested venv case only? Yes. > > and when creating the console-scripts shims: > > mkvenv: Found avocado v90.0 > > Sure. Up to the user at that point to figure out if we used that package > or not. > > e.g. "found meson 0.5.0" might occur even if we require >=1.0. John and I discussed offlist and he'll try to remove --gen so that all system packages handling is done at "mkvenv ensure" time. The "Found" message can then be printed at "mkvenv ensure" time too. Thanks, Paolo > Simple to implement, though. >
On Mon, Apr 24, 2023, 4:02 PM John Snow <jsnow@redhat.com> wrote: > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409 > (All green, except Python self-tests, see below) > > This patch series creates a mandatory python virtual environment > ("venv") during configure time and uses it to ensure the availability of > meson and sphinx. > > See https://www.qemu.org/2023/03/24/python/ for details. The summary is > that the goal of this series is to ensure that the `python` used to run > meson is the same `python` used to run Sphinx, tests, and any build-time > python scripting we have. As it stands, meson and sphinx (and their > extensions) *may* run in a different python environment than the one > configured and chosen by the user at configure/build time. > > The effective change of this series is that QEMU will now > unconditionally create a venv at configure-time and will ensure that > meson (and sphinx, if docs are enabled) are available through that venv. > > Some important points as a pre-emptive "FAQ": > > - This venv is unconditionally created and lives at {build_dir}/pyvenv. > > - The python interpreter used by this venv is always the one identified > by configure. (Which in turn is always the one specified by --python > or $PYTHON) > > - *almost* all python scripts in qemu.git executed as part of the build > system, meson, sphinx, avocado tests, vm tests or CI are always > executed within this venv. > > (iotests are not yet integrated; I plan to tackle this separately as a > follow-up in order to have a more tightly focused scope on that > series.) > > - It remains possible to build and test fully offline. > (In most cases, you just need meson and sphinx from your distro's repo.) > > - Distribution packaged 'meson' and 'sphinx' are still utilized whenever > possible as the highest preference. > > - Vendored versions of e.g. 'meson' are always preferred to PyPI > versions for speed, repeatability and ensuring tarball builds work > as-is offline. > > (Sphinx will not be vendored, just like it already isn't.) > > - Missing dependencies, when possible, are fetched and installed > on-demand automatically to make developer environments "just work". > > - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, > Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere > > - No new dependencies (...for most platforms. Debian and NetBSD get an > asterisk.) > > - The meson git submodule is unused after this series and can be removed. > > For reviewers, here's how the series is broken up: > > Patch 1 is a testing pre-req. Note that even with this patch, > 'check-python-minreqs' and 'check-python-tox' CI jobs will both still > fail on origin/master because this series requires 3.7+, but > origin/master is currently still 3.6+. > > - python: update pylint configuration > > Patches 2-8 add the mkvenv script. The first patch checks in the barest > essentials, and each subsequent patch adds a workaround or feature one > at a time. > > - python: add mkvenv.py > - mkvenv: add console script entry point generation > - mkvenv: Add better error message for missing pyexapt module > - mkvenv: generate console entry shims from inside the venv > - mkvenv: work around broken pip installations on Debian 10 > - mkvenv: add nested venv workaround > - mkvenv: add ensure subcommand > > Patches 9-11 modify our testing configuration to add new dependencies as > needed. > > - tests/docker: add python3-venv dependency > - tests/vm: Configure netbsd to use Python 3.10 > - tests/vm: add py310-expat to NetBSD > > Patch 12 changes how we package release tarballs. > > - scripts/make-release: download meson==0.61.5 .whl > > Patches 13-16 wire mkvenv into configure and tests. > > - configure: create a python venv unconditionally > - configure: use 'mkvenv ensure meson' to bootstrap meson > - configure: add --enable-pypi and --disable-pypi > - tests: Use configure-provided pyvenv for tests > > Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these > changes could be folded earlier in the series (like the diagnose() > patch), but I'm keeping it separate for review for now. > > - configure: move --enable-docs and --disable-docs back to configure > - mkvenv: add diagnose() method for ensure() failures > - configure: use --diagnose option with meson ensure > - configure: bootstrap sphinx with mkvenv > > That's all for now, seeya! > --js > > John Snow (20): > python: update pylint configuration > python: add mkvenv.py > mkvenv: add console script entry point generation > mkvenv: Add better error message for missing pyexpat module > mkvenv: generate console entry shims from inside the venv > mkvenv: work around broken pip installations on Debian 10 > mkvenv: add nested venv workaround > mkvenv: add ensure subcommand > tests/docker: add python3-venv dependency > tests/vm: Configure netbsd to use Python 3.10 > tests/vm: add py310-expat to NetBSD > scripts/make-release: download meson==0.61.5 .whl > configure: create a python venv unconditionally > configure: use 'mkvenv ensure meson' to bootstrap meson > configure: add --enable-pypi and --disable-pypi > tests: Use configure-provided pyvenv for tests > configure: move --enable-docs and --disable-docs back to configure > mkvenv: add diagnose() method for ensure() failures > configure: use --diagnose option with meson ensure > configure: bootstrap sphinx with mkvenv > > docs/devel/acpi-bits.rst | 6 +- > docs/devel/testing.rst | 14 +- > configure | 139 +-- > .gitlab-ci.d/buildtest-template.yml | 4 +- > .gitlab-ci.d/buildtest.yml | 6 +- > python/scripts/mkvenv.py | 871 ++++++++++++++++++ > python/setup.cfg | 10 + > python/tests/flake8.sh | 1 + > python/tests/isort.sh | 1 + > python/tests/mypy.sh | 1 + > python/tests/pylint.sh | 1 + > .../org.centos/stream/8/x86_64/test-avocado | 4 +- > scripts/device-crash-test | 2 +- > scripts/make-release | 11 + > tests/Makefile.include | 10 +- > .../dockerfiles/debian-all-test-cross.docker | 3 +- > .../dockerfiles/debian-hexagon-cross.docker | 3 +- > .../dockerfiles/debian-riscv64-cross.docker | 3 +- > .../dockerfiles/debian-tricore-cross.docker | 3 +- > tests/requirements.txt | 7 +- > tests/vm/netbsd | 2 + > 21 files changed, 1016 insertions(+), 86 deletions(-) > create mode 100644 python/scripts/mkvenv.py > > -- > 2.39.2 > Just a note to say I'm working on the respin here - Paolo suggested I remove the "--gen" arguments to mkvenv and generate the entry point scripts conditionally at ensure time instead. Good idea! I ran into some problems with msys2, though, where the entry point scripts native to the system are binary .exe files I don't know how to recreate. I have some research to do on windows perhaps to understand what's expected on that platform and how to achieve it. I'm hoping the fix is as easy as "You can still use console scripts, but you need to use the .py or .pyw suffix". I'll know shortly! If I don't smooth over all the issues before Wednesday, though, I'm going to be away from my desk for the rest of the week and may need to ask Paolo for help finishing up the polish so we can land this in a timely fashion so we have a lot of runway to fix build issues before 8.1 rc0. --js >
GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409 (All green, except Python self-tests, see below) This patch series creates a mandatory python virtual environment ("venv") during configure time and uses it to ensure the availability of meson and sphinx. See https://www.qemu.org/2023/03/24/python/ for details. The summary is that the goal of this series is to ensure that the `python` used to run meson is the same `python` used to run Sphinx, tests, and any build-time python scripting we have. As it stands, meson and sphinx (and their extensions) *may* run in a different python environment than the one configured and chosen by the user at configure/build time. The effective change of this series is that QEMU will now unconditionally create a venv at configure-time and will ensure that meson (and sphinx, if docs are enabled) are available through that venv. Some important points as a pre-emptive "FAQ": - This venv is unconditionally created and lives at {build_dir}/pyvenv. - The python interpreter used by this venv is always the one identified by configure. (Which in turn is always the one specified by --python or $PYTHON) - *almost* all python scripts in qemu.git executed as part of the build system, meson, sphinx, avocado tests, vm tests or CI are always executed within this venv. (iotests are not yet integrated; I plan to tackle this separately as a follow-up in order to have a more tightly focused scope on that series.) - It remains possible to build and test fully offline. (In most cases, you just need meson and sphinx from your distro's repo.) - Distribution packaged 'meson' and 'sphinx' are still utilized whenever possible as the highest preference. - Vendored versions of e.g. 'meson' are always preferred to PyPI versions for speed, repeatability and ensuring tarball builds work as-is offline. (Sphinx will not be vendored, just like it already isn't.) - Missing dependencies, when possible, are fetched and installed on-demand automatically to make developer environments "just work". - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere - No new dependencies (...for most platforms. Debian and NetBSD get an asterisk.) - The meson git submodule is unused after this series and can be removed. For reviewers, here's how the series is broken up: Patch 1 is a testing pre-req. Note that even with this patch, 'check-python-minreqs' and 'check-python-tox' CI jobs will both still fail on origin/master because this series requires 3.7+, but origin/master is currently still 3.6+. - python: update pylint configuration Patches 2-8 add the mkvenv script. The first patch checks in the barest essentials, and each subsequent patch adds a workaround or feature one at a time. - python: add mkvenv.py - mkvenv: add console script entry point generation - mkvenv: Add better error message for missing pyexapt module - mkvenv: generate console entry shims from inside the venv - mkvenv: work around broken pip installations on Debian 10 - mkvenv: add nested venv workaround - mkvenv: add ensure subcommand Patches 9-11 modify our testing configuration to add new dependencies as needed. - tests/docker: add python3-venv dependency - tests/vm: Configure netbsd to use Python 3.10 - tests/vm: add py310-expat to NetBSD Patch 12 changes how we package release tarballs. - scripts/make-release: download meson==0.61.5 .whl Patches 13-16 wire mkvenv into configure and tests. - configure: create a python venv unconditionally - configure: use 'mkvenv ensure meson' to bootstrap meson - configure: add --enable-pypi and --disable-pypi - tests: Use configure-provided pyvenv for tests Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these changes could be folded earlier in the series (like the diagnose() patch), but I'm keeping it separate for review for now. - configure: move --enable-docs and --disable-docs back to configure - mkvenv: add diagnose() method for ensure() failures - configure: use --diagnose option with meson ensure - configure: bootstrap sphinx with mkvenv That's all for now, seeya! --js John Snow (20): python: update pylint configuration python: add mkvenv.py mkvenv: add console script entry point generation mkvenv: Add better error message for missing pyexpat module mkvenv: generate console entry shims from inside the venv mkvenv: work around broken pip installations on Debian 10 mkvenv: add nested venv workaround mkvenv: add ensure subcommand tests/docker: add python3-venv dependency tests/vm: Configure netbsd to use Python 3.10 tests/vm: add py310-expat to NetBSD scripts/make-release: download meson==0.61.5 .whl configure: create a python venv unconditionally configure: use 'mkvenv ensure meson' to bootstrap meson configure: add --enable-pypi and --disable-pypi tests: Use configure-provided pyvenv for tests configure: move --enable-docs and --disable-docs back to configure mkvenv: add diagnose() method for ensure() failures configure: use --diagnose option with meson ensure configure: bootstrap sphinx with mkvenv docs/devel/acpi-bits.rst | 6 +- docs/devel/testing.rst | 14 +- configure | 139 +-- .gitlab-ci.d/buildtest-template.yml | 4 +- .gitlab-ci.d/buildtest.yml | 6 +- python/scripts/mkvenv.py | 871 ++++++++++++++++++ python/setup.cfg | 10 + python/tests/flake8.sh | 1 + python/tests/isort.sh | 1 + python/tests/mypy.sh | 1 + python/tests/pylint.sh | 1 + .../org.centos/stream/8/x86_64/test-avocado | 4 +- scripts/device-crash-test | 2 +- scripts/make-release | 11 + tests/Makefile.include | 10 +- .../dockerfiles/debian-all-test-cross.docker | 3 +- .../dockerfiles/debian-hexagon-cross.docker | 3 +- .../dockerfiles/debian-riscv64-cross.docker | 3 +- .../dockerfiles/debian-tricore-cross.docker | 3 +- tests/requirements.txt | 7 +- tests/vm/netbsd | 2 + 21 files changed, 1016 insertions(+), 86 deletions(-) create mode 100644 python/scripts/mkvenv.py