| Message ID | 20231109155855.844630-1-pbonzini@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | docs: document what configure does with virtual environments | expand |
Whoops, didn't mean to reply off-list. On Thu, Nov 9, 2023 at 4:34 PM John Snow <jsnow@redhat.com> wrote: > > On Thu, Nov 9, 2023 at 10:59 AM Paolo Bonzini <pbonzini@redhat.com> wrote: > > > > Given the recent confusion around how QEMU detects the system > > Meson installation, and/or decides to install its own, it is > > time to fill in the "Python virtual environments and the QEMU > > build system" section of the documentation. > > > > As a curiosity, a first and partial draft of the text was generated > > by an LLM[1]. It required quite a bit of editing and probably did not > > save much time, but some expressions do remain in the finished text. > > boo :p > > > > > [1] https://chat.openai.com/share/42c1500d-71c1-480b-bab9-7ccc2c155365 > > > > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> > > --- > > docs/devel/build-system.rst | 54 ++++++++++++++++++++++++++++++++++--- > > pythondeps.toml | 3 ++- > > 2 files changed, 53 insertions(+), 4 deletions(-) > > > > diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst > > index 21f78da7d1d..4def2e55e73 100644 > > --- a/docs/devel/build-system.rst > > +++ b/docs/devel/build-system.rst > > @@ -122,10 +122,49 @@ functioning. These are performed using a few more helper functions: > > indicated by $TMPC. > > > > > > -Python virtual environments and the QEMU build system > > ------------------------------------------------------ > > +Python virtual environments and the build process > > +------------------------------------------------- > > + > > +An important part of configure is to create a Python virtual environment > > +(venv) during the configuration phase, using the Python interpreter that > > +``configure`` identified, or that was requested via the ``--python`` > > +command line option and the ``$PYTHON`` variable from the environment. > > Which takes precedence, in what order? "and" makes it sound as if both > --python and $PYTHON are considered, but I don't think that's actually > true. > > > +The venv resides in the ``pyvenv`` directory in the build tree, > > +and provides consistency in how the build process runs Python code. > > +In particular it avoids a potential mismatch, where Meson and Sphinx > > I think you can drop the comma. This is so pedantic that if you left > it in to spite me, I'd not blame you. :) > > > +binaries on the PATH might operate in a different Python environment > > +than the one chosen by the user during the build process. > > + > > +At this stage, ``configure`` also queries the chosen Python interpreter > > +about QEMU's build dependencies. ``configure`` does *not* > > +pick the ``meson``, ``sphinx-build`` or ``avocado`` binaries in the PATH; > > +likewise, there are no options such as ``--meson`` or ``sphinx-build``. > > should we say ``--sphinx-build``? > > I also might say "does not ^necessarily pick the ..." because they > could be the same, it just isn't the criteria it uses to choose them. > > > +If QEMU does not find a dependency, check that it was installed in the > > +right ``site-packages`` directory or with the right ``pip`` program. > > I don't actually know what this means. >_> > > > + > > +If the package is available as a system package for the chosen > > technically, I think if the package is available at all for the chosen > interpreter. We supported nested venvs, so it's more than just system > packages. > > > +interpreter, ``configure`` prepares a small script that invokes it > > +from the venv itself. If not, ``configure`` can also optionally > > +install dependencies in the virtual environment with ``pip``. > > What may not be clear here is that this can refer to installing from > bundled sources in the source tree. IIRC, it's used in preference to > internet sources, isn't it? (Unless that changed.) > > > +Downloading is triggered only when a ``configure`` option (currently, > > +only ``--enable-docs``) is explicitly enabled but the dependencies are > > +not present, and can also be disabled with ``--disable-download``.[#pip]_ > > + > > +.. [#pip_] Avocado can also be installed with ``pip`` in the virtual > > + environment when running ``make check-avocado``. In this > > + case, it is not currently possible to block the downloading. > > + > > Similarly to my above comment: if downloads are disabled, we do have > some vendored packages we'll attempt to use. > > > +The required versions of the packages are stored in a configuration file > > +``pythondeps.toml``. The format is custom to QEMU, but it is documented > > +at the top of the file itself and it should be easy to understand. The > > +requirements should make it possible to use the version that is packaged > > +that is provided by supported distros. > > + > > +When dependencies are downloaded, instead, ``configure`` uses a "known > > +good" version that is also listed in ``pythondeps.toml``. In this > > +scenario, ``pythondeps.toml`` behaves like the "lock file" used by > > +``cargo``, ``poetry`` or other dependency management systems. > > > > -TBD > > > > Stage 2: Meson > > ============== > > @@ -376,6 +415,15 @@ This is needed to obey the --python= option passed to the configure > > script, which may point to something other than the first python3 > > binary on the path. > > > > +By the time Meson runs, Python dependencies are available in the virtual > > +environment and should be invoked though the scripts that ``configure`` > > through > > > +places under ``pyvenv``. One way to do so is as follows, using Meson's > > +``find_program`` function:: > > + > > + sphinx_build = find_program( > > + fs.parent(python.full_path()) / 'sphinx-build', > > + required: get_option('docs')) > > + > > > > Stage 3: Make > > ============= > > diff --git a/pythondeps.toml b/pythondeps.toml > > index 0a35ebcf9f0..4beadfd96f5 100644 > > --- a/pythondeps.toml > > +++ b/pythondeps.toml > > @@ -10,7 +10,8 @@ > > # - accepted: accepted versions when using a system package > > # - installed: fixed version to install in the virtual environment > > # if a system package is not found; if not specified, > > -# the minimum and maximum > > +# defaults to the same as "accepted" or, if also missing, > > +# to the newest version available on PyPI. > > # - canary: if specified, use this program name to present more > > # precise error diagnostics to the user. For example, > > # 'sphinx-build' can be used as a bellwether for the > > -- > > 2.41.0 > >
On Thu, Nov 9, 2023 at 10:35 PM John Snow <jsnow@redhat.com> wrote: > > > +The venv resides in the ``pyvenv`` directory in the build tree, > > > +and provides consistency in how the build process runs Python code. > > > +In particular it avoids a potential mismatch, where Meson and Sphinx > > > > I think you can drop the comma. This is so pedantic that if you left > > it in to spite me, I'd not blame you. :) I'll keep it then. :) > > should we say ``--sphinx-build``? Yes, typo. > > I also might say "does not ^necessarily pick the ..." because they > > could be the same, it just isn't the criteria it uses to choose them. I'll replace "pick" with "look for". > > > +If QEMU does not find a dependency, check that it was installed in the > > > +right ``site-packages`` directory or with the right ``pip`` program. > > > > I don't actually know what this means. >_> It's meant to explain what happened with homebrew. Rephrased in v2: This avoids a potential mismatch, where Meson and Sphinx binaries on the PATH might operate in a different Python environment than the one chosen by the user during the build process. On the other hand, it introduces a potential source of confusion where the user installs a dependency but ``configure`` is not able to find it. When this happens, the dependency was installed in the ``site-packages`` directory of another interpreter, or with the wrong ``pip`` program. Paolo
diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst index 21f78da7d1d..4def2e55e73 100644 --- a/docs/devel/build-system.rst +++ b/docs/devel/build-system.rst @@ -122,10 +122,49 @@ functioning. These are performed using a few more helper functions: indicated by $TMPC. -Python virtual environments and the QEMU build system ------------------------------------------------------ +Python virtual environments and the build process +------------------------------------------------- + +An important part of configure is to create a Python virtual environment +(venv) during the configuration phase, using the Python interpreter that +``configure`` identified, or that was requested via the ``--python`` +command line option and the ``$PYTHON`` variable from the environment. +The venv resides in the ``pyvenv`` directory in the build tree, +and provides consistency in how the build process runs Python code. +In particular it avoids a potential mismatch, where Meson and Sphinx +binaries on the PATH might operate in a different Python environment +than the one chosen by the user during the build process. + +At this stage, ``configure`` also queries the chosen Python interpreter +about QEMU's build dependencies. ``configure`` does *not* +pick the ``meson``, ``sphinx-build`` or ``avocado`` binaries in the PATH; +likewise, there are no options such as ``--meson`` or ``sphinx-build``. +If QEMU does not find a dependency, check that it was installed in the +right ``site-packages`` directory or with the right ``pip`` program. + +If the package is available as a system package for the chosen +interpreter, ``configure`` prepares a small script that invokes it +from the venv itself. If not, ``configure`` can also optionally +install dependencies in the virtual environment with ``pip``. +Downloading is triggered only when a ``configure`` option (currently, +only ``--enable-docs``) is explicitly enabled but the dependencies are +not present, and can also be disabled with ``--disable-download``.[#pip]_ + +.. [#pip_] Avocado can also be installed with ``pip`` in the virtual + environment when running ``make check-avocado``. In this + case, it is not currently possible to block the downloading. + +The required versions of the packages are stored in a configuration file +``pythondeps.toml``. The format is custom to QEMU, but it is documented +at the top of the file itself and it should be easy to understand. The +requirements should make it possible to use the version that is packaged +that is provided by supported distros. + +When dependencies are downloaded, instead, ``configure`` uses a "known +good" version that is also listed in ``pythondeps.toml``. In this +scenario, ``pythondeps.toml`` behaves like the "lock file" used by +``cargo``, ``poetry`` or other dependency management systems. -TBD Stage 2: Meson ============== @@ -376,6 +415,15 @@ This is needed to obey the --python= option passed to the configure script, which may point to something other than the first python3 binary on the path. +By the time Meson runs, Python dependencies are available in the virtual +environment and should be invoked though the scripts that ``configure`` +places under ``pyvenv``. One way to do so is as follows, using Meson's +``find_program`` function:: + + sphinx_build = find_program( + fs.parent(python.full_path()) / 'sphinx-build', + required: get_option('docs')) + Stage 3: Make ============= diff --git a/pythondeps.toml b/pythondeps.toml index 0a35ebcf9f0..4beadfd96f5 100644 --- a/pythondeps.toml +++ b/pythondeps.toml @@ -10,7 +10,8 @@ # - accepted: accepted versions when using a system package # - installed: fixed version to install in the virtual environment # if a system package is not found; if not specified, -# the minimum and maximum +# defaults to the same as "accepted" or, if also missing, +# to the newest version available on PyPI. # - canary: if specified, use this program name to present more # precise error diagnostics to the user. For example, # 'sphinx-build' can be used as a bellwether for the
Given the recent confusion around how QEMU detects the system Meson installation, and/or decides to install its own, it is time to fill in the "Python virtual environments and the QEMU build system" section of the documentation. As a curiosity, a first and partial draft of the text was generated by an LLM[1]. It required quite a bit of editing and probably did not save much time, but some expressions do remain in the finished text. [1] https://chat.openai.com/share/42c1500d-71c1-480b-bab9-7ccc2c155365 Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> --- docs/devel/build-system.rst | 54 ++++++++++++++++++++++++++++++++++--- pythondeps.toml | 3 ++- 2 files changed, 53 insertions(+), 4 deletions(-)