From patchwork Thu May 11 03:54:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13237477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7447FC77B7F for ; Thu, 11 May 2023 03:55:58 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pwxO9-0007W9-Ho; Wed, 10 May 2023 23:54:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pwxO7-0007Vp-OW for qemu-devel@nongnu.org; Wed, 10 May 2023 23:54:43 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pwxO5-0005Iw-E2 for qemu-devel@nongnu.org; Wed, 10 May 2023 23:54:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1683777280; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=xJlg9JyzyR7szAd7WwY8KbUFR6rj7CNXRSYZjvK/b5M=; b=fiZSzG36T7Q/C2kEoQeYI/EmjBHFfeEiXiWoGLsQI6+VzazyWbtLQOfoNq5aH3tHFEAx/e 5XIEn3+3Dd+XQDS0R0qnMltYeQh+f28fvxT0c8vdsJ9ztFSKJrDEmK43u9nB5zsAcdc5/L y7ROg07+j0bsuobLeA+VFOGs4H2yh2c= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-296-BHyl-oHHOy2R58kecM_L_Q-1; Wed, 10 May 2023 23:54:37 -0400 X-MC-Unique: BHyl-oHHOy2R58kecM_L_Q-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A7128101A531; Thu, 11 May 2023 03:54:36 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.34.188]) by smtp.corp.redhat.com (Postfix) with ESMTP id D545C40C2076; Thu, 11 May 2023 03:54:35 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Warner Losh , Peter Maydell , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Ani Sinha , Beraldo Leal , Markus Armbruster , Ryo ONODERA , Paolo Bonzini , Kyle Evans , =?utf-8?q?Alex_Benn=C3=A9e?= , Michael Roth , Reinoud Zandijk , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Cleber Rosa , Thomas Huth , "Michael S. Tsirkin" , John Snow , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Wainer dos Santos Moschetta Subject: [PATCH 00/27] configure: create a python venv and ensure meson, sphinx Date: Wed, 10 May 2023 23:54:08 -0400 Message-Id: <20230511035435.734312-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch series creates a mandatory python virtual environment ("venv") during configure time and uses it to ensure the availability of meson and sphinx. See https://www.qemu.org/2023/03/24/python/ for motivations. The summary is that the goal of this series is to ensure that the `python` used to run meson is the same `python` used to run Sphinx, tests, and any build-time python scripting we have. As it stands, meson and sphinx (and their extensions) *may* run in a different python environment than the one configured and chosen by the user at configure/build time. The effective change of this series is that QEMU will now unconditionally create a venv at configure-time and will ensure that meson (and sphinx, if docs are enabled) are available through that venv. Some important points as a pre-emptive "FAQ": - This venv is unconditionally created and lives at {build_dir}/pyvenv. - The python interpreter used by this venv is always the one identified by configure. (Which in turn is always the one specified by --python or $PYTHON) - *almost* all python scripts in qemu.git executed as part of the build system, meson, sphinx, avocado tests, vm tests or CI are always executed within this venv. (iotests are not yet integrated; I plan to tackle this separately as a follow-up in order to have a more tightly focused scope on that series.) - It remains possible to build and test fully offline. (In most cases, you just need meson and sphinx from your distro's repo.) - Distribution packaged 'meson' and 'sphinx' are still utilized whenever possible as the highest preference. - Vendored versions of e.g. 'meson' are always preferred to PyPI versions for speed, repeatability and ensuring tarball builds work as-is offline. (Sphinx will not be vendored, just like it already isn't.) - Missing dependencies, when possible, are fetched and installed on-demand automatically to make developer environments "just work". - Works for Python 3.6 and up, on Fedora, OpenSuSE, Red Hat, CentOS, Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere - No new dependencies (...for most platforms. Debian and NetBSD get an asterisk.) - The meson git submodule is unused after this series and can be removed. ... I could write more, and I'm sure there's a few things that need buffing out or changing here and there, but I think it's largely good. So, enjoy! ~js John Snow (22): python: update pylint configuration python: add mkvenv.py mkvenv: add better error message for missing pyexpat module mkvenv: add nested venv workaround mkvenv: add ensure subcommand mkvenv: add diagnose() method for ensure() failures mkvenv: add console script entry point generation mkvenv: work around broken pip installations on Debian 10 tests/docker: add python3-venv dependency tests/vm: Configure netbsd to use Python 3.10 tests/vm: add py310-expat to NetBSD python: add vendor.py utility configure: create a python venv unconditionally python/wheels: add vendored meson package configure: use 'mkvenv ensure meson' to bootstrap meson qemu.git: drop meson git submodule tests: Use configure-provided pyvenv for tests configure: move --enable-docs and --disable-docs back to configure configure: bootstrap sphinx with mkvenv configure: add --enable-pypi and --disable-pypi configure: Add courtesy hint to Python version failure message mkvenv.py: experiment; use distlib to generate script entry points Paolo Bonzini (5): python: shut up "pip install" during "make check-minreqs" mkvenv: create pip binary in virtual environment Python: Drop support for Python 3.6 mkvenv: mark command as required python: bump some of the dependencies docs/about/build-platforms.rst | 2 +- docs/conf.py | 9 - docs/devel/acpi-bits.rst | 6 +- docs/devel/testing.rst | 14 +- docs/meson.build | 2 +- configure | 155 ++-- .gitlab-ci.d/buildtest-template.yml | 4 +- .gitlab-ci.d/buildtest.yml | 6 +- .gitmodules | 3 - meson | 1 - meson_options.txt | 2 - python/Makefile | 19 +- python/scripts/mkvenv.py | 858 ++++++++++++++++++ python/scripts/vendor.py | 74 ++ python/setup.cfg | 27 +- python/tests/flake8.sh | 1 + python/tests/isort.sh | 1 + python/tests/minreqs.txt | 16 +- python/tests/mypy.sh | 1 + python/tests/pylint.sh | 1 + python/wheels/meson-0.61.5-py3-none-any.whl | Bin 0 -> 862509 bytes .../org.centos/stream/8/x86_64/test-avocado | 4 +- scripts/device-crash-test | 2 +- scripts/meson-buildoptions.sh | 3 - scripts/qapi/mypy.ini | 2 +- tests/Makefile.include | 10 +- .../dockerfiles/debian-all-test-cross.docker | 3 +- .../dockerfiles/debian-hexagon-cross.docker | 3 +- .../dockerfiles/debian-riscv64-cross.docker | 3 +- .../dockerfiles/debian-tricore-cross.docker | 3 +- tests/requirements.txt | 7 +- tests/vm/netbsd | 2 + 32 files changed, 1108 insertions(+), 136 deletions(-) delete mode 160000 meson create mode 100644 python/scripts/mkvenv.py create mode 100755 python/scripts/vendor.py create mode 100644 python/wheels/meson-0.61.5-py3-none-any.whl