From patchwork Mon Jul 11 23:01:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12914340 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 24BCFC433EF for ; Mon, 11 Jul 2022 23:04:46 +0000 (UTC) Received: from localhost ([::1]:56350 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oB2SK-000369-Ti for qemu-devel@archiver.kernel.org; Mon, 11 Jul 2022 19:04:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34116) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oB2Pk-0000gX-T6 for qemu-devel@nongnu.org; Mon, 11 Jul 2022 19:02:07 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:49238) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oB2Ph-00022U-Nz for qemu-devel@nongnu.org; Mon, 11 Jul 2022 19:02:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1657580520; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=QJuOkhEXAiXjH4eYGnIaD9sRWCWZFi+PQLkERePzj1U=; b=Z9BsMVqgTiQ5XLtuQPI3rRI0v3/k9qmZPOlyC8M362T4GeuEx2f0eVRCUhEomJpG4etxXn CWAbsKNu22/H/nNYEvmZpg2p/eIajkj6p9VABX2MCmUzECOIwv2IKKwmIfhQdKXntcgEWh J+pNIz0iIBPhZWQfy1jZQyJUuUfGx9k= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-55-4i_7WstmN4-lyZQtNrWdNQ-1; Mon, 11 Jul 2022 19:01:57 -0400 X-MC-Unique: 4i_7WstmN4-lyZQtNrWdNQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B3E8F1C06ECA; Mon, 11 Jul 2022 23:01:56 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.16.25]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8B9F32166B26; Mon, 11 Jul 2022 23:01:55 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Daniel Berrange , Kevin Wolf , Beraldo Leal , John Snow , =?utf-8?q?Al?= =?utf-8?q?ex_Benn=C3=A9e?= , Thomas Huth , Wainer dos Santos Moschetta , Kyle Evans , Warner Losh , qemu-block@nongnu.org, Paolo Bonzini , Cleber Rosa , Michael Tsirkin , Ani Sinha , Hanna Reitz , =?utf-8?q?Phi?= =?utf-8?q?lippe_Mathieu-Daud=C3=A9?= Subject: [RFC PATCH v3 0/7] tests: run python tests under a venv Date: Mon, 11 Jul 2022 19:01:48 -0400 Message-Id: <20220711230155.953788-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Hi, here's another RFC for bringing external Python dependencies to the QEMU test suite. This is mostly a refresh of a version I sent out before, but mixes in my VM improvement test as an optional pre-requisite to improve VM test stability to ensure that the BSDs all pass with the new infrastructure. (And our oldest supported Debian and Ubuntu targets, too.) (Note: this requires dropping support for Ubuntu 18.04, which ships with a version of setuptools that is simply too old.) This patchset is still not without some problems that I am working on, but progress has been slow. Problems I am aware of: - This version of the patch series does not itself enforce any offline-only behavior for venv creation, but downstreams can modify any call to 'mkvenv' to pass '--offline'. I am working on a configure file toggle to swap the default behavior when running tests. - iotests will now actually never run mypy or pylint tests by default anymore, because the bootstrapper won't select those packages by default, and the virtual environment won't utilize the system packages, so iotest 297 will just "skip" all of the time now. The reason we don't want to install these packages by default is because we don't want to add dependencies on mypy and pylint for downstream builds. With these patches, 297 would still work if you manually opened up the testing venv and installed suitable mypy/pylint packages there. I could also add a new optional dependency group, and one could theoretically invoke a once-per-build-dir command of 'make check-venv-iotests' to help make the process only semi-manual, but it's still annoying. Ideally, the python checks in qemu.git/python/ can handle the same tests as 297 does -- but we need to give a shorthand invocation like "make check-python" that is excluded from the default "make check" to allow block developers to quickly opt-in to the same tests. I've covered some of the problems here on-list before: https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg03661.html ...But I haven't quite solved them yet. - iotests can now self-bootstrap the venv when it isn't present, but this self-bootstrapping has some issues in that because it skips the Makefile magic, it cannot update the venv when changes are made to the venv configuration piece. That's all for now. Work on the configure file integration is ongoing. I don't know if I'll beat soft freeze (It's looking unlikely given the amount of IRL issues I am juggling right now), but I'm hoping to push as much of this forward as I can to try and get some testing in for RC phase to determine what problems might exist that I haven't noticed yet. --js John Snow (7): tests: create optional tests/venv dependency groups tests: pythonize test venv creation tests: Remove spurious pip warnings on Ubuntu20.04 tests/vm: add venv pre-requisites to VM building recipes tests: add 'check-venv' as a dependency of 'make check' iotests: use tests/venv for running tests iotests: self-bootstrap testing venv tests/Makefile.include | 32 +++--- tests/mkvenv.py | 187 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/testenv.py | 25 +++-- tests/requirements.txt | 6 -- tests/setup.cfg | 20 ++++ tests/setup.py | 16 +++ tests/vm/netbsd | 1 + 7 files changed, 262 insertions(+), 25 deletions(-) create mode 100644 tests/mkvenv.py delete mode 100644 tests/requirements.txt create mode 100644 tests/setup.cfg create mode 100644 tests/setup.py