From patchwork Wed Jun 21 00:21:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13286508 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1DACFEB64D7 for ; Wed, 21 Jun 2023 00:22:49 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBlbX-00041e-P1; Tue, 20 Jun 2023 20:21:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBlbW-00040E-DF for qemu-devel@nongnu.org; Tue, 20 Jun 2023 20:21:46 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBlbQ-00006p-S8 for qemu-devel@nongnu.org; Tue, 20 Jun 2023 20:21:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687306898; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IbqHUNUUzyICKNg1KPxJjBeG4KgfXfoJFkJJCGALlEY=; b=ZmTQiTt8acOVh0uDev7e8vrV9DozVoAP4sm0eog7V4HLF4sqEKaPiYxgNJgt/8JUOSKFEy yDYp6F+qRrvJdfD+uqSSrzG9giZh4Ow3MR+8PkGj0ec0BcUcwPhqmrgB/lfN74a8GkpjD3 30AgjbIbHJhDOT7pZimStsZR2PzDm3s= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-88-ZTshDb7HOiuNIGhGbmf0XQ-1; Tue, 20 Jun 2023 20:21:35 -0400 X-MC-Unique: ZTshDb7HOiuNIGhGbmf0XQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 851048D6E26; Wed, 21 Jun 2023 00:21:35 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.32.44]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2E7F8112132C; Wed, 21 Jun 2023 00:21:35 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Thomas Huth , qemu-block@nongnu.org, Beraldo Leal , Hanna Reitz , =?utf-8?q?Alex_Benn=C3=A9e?= , John Snow , Kevin Wolf , Cleber Rosa , Paolo Bonzini Subject: [PATCH RFC 1/6] experiment: add mkvenv install Date: Tue, 20 Jun 2023 20:21:16 -0400 Message-Id: <20230621002121.1609612-2-jsnow@redhat.com> In-Reply-To: <20230621002121.1609612-1-jsnow@redhat.com> References: <20230621002121.1609612-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This is just so I can do "mkvenv install './python'" or "mkvenv install file:python" to install the in-tree packages to pyvenv. It probably isn't quite appropriate to bypass do_ensure in its entirety like this because we miss out on a lot of error handling, but as a quick proof of concept it works just fine. Signed-off-by: John Snow --- python/scripts/mkvenv.py | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py index a47f1eaf5d..ea8df34111 100644 --- a/python/scripts/mkvenv.py +++ b/python/scripts/mkvenv.py @@ -940,6 +940,35 @@ def _add_ensure_subcommand(subparsers: Any) -> None: ) +def _add_install_subcommand(subparsers: Any) -> None: + subparser = subparsers.add_parser( + "install", help="Install the specified package." + ) + subparser.add_argument( + "--online", + action="store_true", + help="Install packages from PyPI, if necessary.", + ) + subparser.add_argument( + "--dir", + type=str, + action="store", + help="Path to vendored packages where we may install from.", + ) + subparser.add_argument( + '--editable', + action="store_true", + help="Should package(s) be installed in editable mode?" + ) + subparser.add_argument( + "dep_specs", + type=str, + action="store", + help="PEP 508 Dependency specification, e.g. 'meson>=0.61.5'", + nargs="+", + ) + + def main() -> int: """CLI interface to make_qemu_venv. See module docstring.""" if os.environ.get("DEBUG") or os.environ.get("GITLAB_CI"): @@ -964,6 +993,7 @@ def main() -> int: _add_create_subcommand(subparsers) _add_post_init_subcommand(subparsers) _add_ensure_subcommand(subparsers) + _add_install_subcommand(subparsers) args = parser.parse_args() try: @@ -982,6 +1012,16 @@ def main() -> int: wheels_dir=args.dir, prog=args.diagnose, ) + if args.command == "install": + print(f"mkvenv: installing {', '.join(args.dep_specs)}", file=sys.stderr) + pip_args = list(args.dep_specs) + if args.editable: + pip_args.insert(0, "--editable") + pip_install( + args=pip_args, + online=args.online, + wheels_dir=args.dir + ) logger.debug("mkvenv.py %s: exiting", args.command) except Ouch as exc: print("\n*** Ouch! ***\n", file=sys.stderr) From patchwork Wed Jun 21 00:21:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13286513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BCD80EB64D8 for ; Wed, 21 Jun 2023 00:23:36 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBlbV-0003zI-Tx; Tue, 20 Jun 2023 20:21:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBlbU-0003yT-6l for qemu-devel@nongnu.org; Tue, 20 Jun 2023 20:21:44 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBlbQ-000093-N3 for qemu-devel@nongnu.org; Tue, 20 Jun 2023 20:21:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687306900; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7tmXMJY+e4M0cUKQ5yliUcPrNFZ1qAAb8NZJYE/gZ7o=; b=AKQECWenCV9fKTUuH/MgIS5sC9D7mpVj8K/c8cRUb+KyDtW5XOS3cUzt8JyDU3fB1e6Mq/ eLkBJTgqRwNxAJpGVWog0q/F6wNn463kmXoavOHXMYf78JJuc6L7xlObcum4I/YHT4kUT6 Kw793KtsCp/Iws0haZIki5AsxN7o+8w= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-134-UaBkaYbiNzafKZ9VAPAM1w-1; Tue, 20 Jun 2023 20:21:36 -0400 X-MC-Unique: UaBkaYbiNzafKZ9VAPAM1w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EA88A8028B2; Wed, 21 Jun 2023 00:21:35 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.32.44]) by smtp.corp.redhat.com (Postfix) with ESMTP id 93948112132C; Wed, 21 Jun 2023 00:21:35 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Thomas Huth , qemu-block@nongnu.org, Beraldo Leal , Hanna Reitz , =?utf-8?q?Alex_Benn=C3=A9e?= , John Snow , Kevin Wolf , Cleber Rosa , Paolo Bonzini Subject: [PATCH RFC 2/6] build, tests: Add qemu in-tree packages to pyvenv at configure time. Date: Tue, 20 Jun 2023 20:21:17 -0400 Message-Id: <20230621002121.1609612-3-jsnow@redhat.com> In-Reply-To: <20230621002121.1609612-1-jsnow@redhat.com> References: <20230621002121.1609612-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org though, ouch: on my machine this takes 3-4 entire seconds to do. I wish it wasn't so slow, but we can't rely on these packages not having any dependencies any more. We could theoretically use a .pth hack when creating the venv to automatically include this directory as an "installed packages" location, but when we go to drop qemu.qmp in the future, that will break - I think we need to *install* this package. Signed-off-by: John Snow --- configure | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/configure b/configure index 01a53576a7..d2e0abc068 100755 --- a/configure +++ b/configure @@ -250,6 +250,7 @@ git_submodules_action="update" git="git" debug_tcg="no" docs="auto" +tests="enabled" EXESUF="" prefix="/usr/local" qemu_suffix="qemu" @@ -639,6 +640,10 @@ for opt do ;; --disable-docs) docs=disabled ;; + --enable-tests) tests=enabled + ;; + --disable-tests) tests=disabled + ;; --cpu=*) ;; --target-list=*) target_list="$optarg" @@ -985,6 +990,32 @@ if test "$docs" != "disabled" ; then fi fi +# Optionally pre-load the testing pre-requisites. This is for iotests, +# vmtests, and anything else that uses Python qemu.* packages. Note that +# our in-tree qemu packages are currently pure python with zero external +# dependencies. For this reason, it excludes the Avocado dependencies +# which are installed on-demand at time of use instead. + +mkvenv_flags="" +if test "$pypi" = "enabled" ; then + mkvenv_flags="--online" +fi + +if test "$tests" = "enabled" ; then + if ! $mkvenv install \ + $mkvenv_flags \ + --editable \ + --dir "${source_path}/python/wheels" \ + "${source_path}/python/"; + then + echo "There was a problem installing the in-tree python packages for testing." + exit 1 + fi + touch pyvenv/tests.group +fi + +echo "mkvenv: done for now, ciao!" + # Probe for ninja if test -z "$ninja"; then From patchwork Wed Jun 21 00:21:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13286511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8B3EBEB64D7 for ; Wed, 21 Jun 2023 00:23:35 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBlba-00042k-4c; Tue, 20 Jun 2023 20:21:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBlbV-0003z9-0x for qemu-devel@nongnu.org; Tue, 20 Jun 2023 20:21:45 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBlbQ-00006w-Pj for qemu-devel@nongnu.org; Tue, 20 Jun 2023 20:21:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687306898; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iYQaPAZHDGfmc/iGdIP4cW70XOXR2Y92QJpitsAFuNc=; b=AzwmTgLfH3gKNVnwHsW5Rqk+dWswoYYk3IfPz2fBF30gMs6aQDl6LH2d7fTAgpqkkA0J2k rM3xjyCCaDiAInhuMwnBhGI5B/vKYQ8DPF7HFx/1Kyjm9wFoMd2Zu1/Zq5JCat9wtDzWdn HT/SJy4l8UUlr1AlULyvt6JER41NMtU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-134-l8Gli7PSOjuoqNN1WzWOZQ-1; Tue, 20 Jun 2023 20:21:36 -0400 X-MC-Unique: l8Gli7PSOjuoqNN1WzWOZQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5AC6C29ABA0B; Wed, 21 Jun 2023 00:21:36 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.32.44]) by smtp.corp.redhat.com (Postfix) with ESMTP id 02283112132C; Wed, 21 Jun 2023 00:21:35 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Thomas Huth , qemu-block@nongnu.org, Beraldo Leal , Hanna Reitz , =?utf-8?q?Alex_Benn=C3=A9e?= , John Snow , Kevin Wolf , Cleber Rosa , Paolo Bonzini Subject: [PATCH RFC 3/6] iotests: get rid of '..' in path environment output Date: Tue, 20 Jun 2023 20:21:18 -0400 Message-Id: <20230621002121.1609612-4-jsnow@redhat.com> In-Reply-To: <20230621002121.1609612-1-jsnow@redhat.com> References: <20230621002121.1609612-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Resolve the build_root before we append more items onto it so that the environment output is more concise with less parent directory confetti in it. Signed-off-by: John Snow --- tests/qemu-iotests/testenv.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py index 9a37ad9152..e67ebd254b 100644 --- a/tests/qemu-iotests/testenv.py +++ b/tests/qemu-iotests/testenv.py @@ -216,7 +216,7 @@ def __init__(self, source_dir: str, build_dir: str, self.source_iotests = source_dir self.build_iotests = build_dir - self.build_root = os.path.join(self.build_iotests, '..', '..') + self.build_root = Path(self.build_iotests).parent.parent self.init_directories() From patchwork Wed Jun 21 00:21:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13286507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9354FEB64D7 for ; Wed, 21 Jun 2023 00:22:26 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBlbb-00042y-2k; Tue, 20 Jun 2023 20:21:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBlbW-00040I-Do for qemu-devel@nongnu.org; Tue, 20 Jun 2023 20:21:46 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBlbR-00009w-2v for qemu-devel@nongnu.org; Tue, 20 Jun 2023 20:21:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687306900; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LkU7CMyTAXo6v8bUrSdszIMVMWjQzMx6I+wCcy8oGI4=; b=ErcUBrjyIphKR2C5OjZXjbu2E689eJDo7Re3WLUOKvw01mW5Q7A2cXKOauetdbLwJjPBXg NLdxB0Rjg1jtCGR//bGC9ySwJSm5NJTgPr4HIs18fMzPE7/NKl2TUt4BN9w42TmQEUcrSd q6eTfuFtPL2q1Rl5XouTFF8/iE8vHBk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-464-bUcv_dKzOyaXjPhwH74t6Q-1; Tue, 20 Jun 2023 20:21:37 -0400 X-MC-Unique: bUcv_dKzOyaXjPhwH74t6Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C166F88CC49; Wed, 21 Jun 2023 00:21:36 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.32.44]) by smtp.corp.redhat.com (Postfix) with ESMTP id 69424112132C; Wed, 21 Jun 2023 00:21:36 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Thomas Huth , qemu-block@nongnu.org, Beraldo Leal , Hanna Reitz , =?utf-8?q?Alex_Benn=C3=A9e?= , John Snow , Kevin Wolf , Cleber Rosa , Paolo Bonzini Subject: [PATCH RFC 4/6] iotests: use the correct python to run linters Date: Tue, 20 Jun 2023 20:21:19 -0400 Message-Id: <20230621002121.1609612-5-jsnow@redhat.com> In-Reply-To: <20230621002121.1609612-1-jsnow@redhat.com> References: <20230621002121.1609612-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Whichever python is used to run iotest 297 should be the one used to actually run the linters. Signed-off-by: John Snow --- tests/qemu-iotests/linters.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/linters.py b/tests/qemu-iotests/linters.py index 65c4c4e827..9fb3fd1449 100644 --- a/tests/qemu-iotests/linters.py +++ b/tests/qemu-iotests/linters.py @@ -68,7 +68,7 @@ def run_linter( :raise CalledProcessError: If the linter process exits with failure. """ subprocess.run( - ('python3', '-m', tool, *args), + (sys.executable, '-m', tool, *args), env=env, check=True, stdout=subprocess.PIPE if suppress_output else None, From patchwork Wed Jun 21 00:21:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13286509 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AA51FEB64D7 for ; Wed, 21 Jun 2023 00:22:53 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBlbY-000425-E7; Tue, 20 Jun 2023 20:21:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBlbW-00040Q-Et for qemu-devel@nongnu.org; Tue, 20 Jun 2023 20:21:46 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBlbS-0000AO-DE for qemu-devel@nongnu.org; Tue, 20 Jun 2023 20:21:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687306901; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PlwB+WM1b7QfqPY4EB+HTeRP+apwkJM6BwxEO67ro5k=; b=FkioT9ll0CXwHSpOh5EgrG4VH4BBQRWw8GZSMWcrxbbEXSiYm25GFsugK++gHIqvO2h0vz ONvP/7HdWwvIdLwS1Wile8Fpn2xFOANwtZwFtTTaD+VNDQv9z61zoYNLpi6c85Ho5q/ONh DGdGhVQ0+U2UfjPMYvn9fs2h0ixx1a4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-595-vK6XCe1hNO6uY2GqojsTdw-1; Tue, 20 Jun 2023 20:21:37 -0400 X-MC-Unique: vK6XCe1hNO6uY2GqojsTdw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3250A381079C; Wed, 21 Jun 2023 00:21:37 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.32.44]) by smtp.corp.redhat.com (Postfix) with ESMTP id D01AA112132C; Wed, 21 Jun 2023 00:21:36 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Thomas Huth , qemu-block@nongnu.org, Beraldo Leal , Hanna Reitz , =?utf-8?q?Alex_Benn=C3=A9e?= , John Snow , Kevin Wolf , Cleber Rosa , Paolo Bonzini Subject: [PATCH RFC 5/6] iotests: use pyvenv/bin/python3 to launch child test processes Date: Tue, 20 Jun 2023 20:21:20 -0400 Message-Id: <20230621002121.1609612-6-jsnow@redhat.com> In-Reply-To: <20230621002121.1609612-1-jsnow@redhat.com> References: <20230621002121.1609612-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Now that there's a fancy venv set up for us by configure, we should take care to use it even when check is invoked directly. ./check will now use the pyvenv environment when launching python tests, which allows those tests to find and access the 'qemu.*' packages without PYTHONPATH modifications. RFC: This patch now means that ./check may launch test subprocesses using a different Python than the one used to launch it. If that isn't acceptable, we might need a launcher shim whose job it is to sit above "check" and just chooses the correct Python. ...Or maybe it's fine the way it is. Comments welcome, sorry for my indecision. Signed-off-by: John Snow --- tests/qemu-iotests/testenv.py | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py index e67ebd254b..1b095d70f2 100644 --- a/tests/qemu-iotests/testenv.py +++ b/tests/qemu-iotests/testenv.py @@ -138,7 +138,20 @@ def init_binaries(self) -> None: PYTHON (for bash tests) QEMU_PROG, QEMU_IMG_PROG, QEMU_IO_PROG, QEMU_NBD_PROG, QSD_PROG """ - self.python = sys.executable + # The python we want to use to launch tests. + self.python: str = str( + Path(self.build_root).joinpath('pyvenv', 'bin', 'python3') + ) + # RFC: Do I need to amend '.exe' for windows, or nah? + + if self.python != sys.executable: + print( + "Note: " + f"check was launched with a Python ({sys.executable}) " + f"that doesn't match QEMU's configured Python ({self.python})." + " QEMU's Python will be used for individual test processes.", + file=sys.stderr + ) def root(*names: str) -> str: return os.path.join(self.build_root, *names) From patchwork Wed Jun 21 00:21:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 13286512 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 602BFEB64DB for ; Wed, 21 Jun 2023 00:23:36 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qBlba-00042m-IK; Tue, 20 Jun 2023 20:21:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBlbW-00040L-Di for qemu-devel@nongnu.org; Tue, 20 Jun 2023 20:21:46 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qBlbS-0000AM-BT for qemu-devel@nongnu.org; Tue, 20 Jun 2023 20:21:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687306901; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dLtrgsESiA1y9/tsmlAWVcd6ujxkZYiJAKc2n+jmnXg=; b=edJx8n0R55O6UZe/opndeMuqHtnmhrD3JO+hmIvAZBsGv8ujnDQbOm+A2Z/35smypWi//4 tqsr0hDn6wYXIGcifPTHIlR41k/QM1fSpdDGrldvWQLKhn2vxyOpJpXxTE5hjuZDKOTkZ/ kTiRKfY+ddXOoJ8Wv0XwJVAGvkCnlXk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-539-DkcmP4kqOS2NJt4bvxsd-w-1; Tue, 20 Jun 2023 20:21:38 -0400 X-MC-Unique: DkcmP4kqOS2NJt4bvxsd-w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9B0AF185A78B; Wed, 21 Jun 2023 00:21:37 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.32.44]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4141E112132C; Wed, 21 Jun 2023 00:21:37 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Cc: Thomas Huth , qemu-block@nongnu.org, Beraldo Leal , Hanna Reitz , =?utf-8?q?Alex_Benn=C3=A9e?= , John Snow , Kevin Wolf , Cleber Rosa , Paolo Bonzini Subject: [PATCH RFC 6/6] iotests: don't add qemu.git/python to PYTHONPATH Date: Tue, 20 Jun 2023 20:21:21 -0400 Message-Id: <20230621002121.1609612-7-jsnow@redhat.com> In-Reply-To: <20230621002121.1609612-1-jsnow@redhat.com> References: <20230621002121.1609612-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org qemu.* should be provided by the configure-time venv, now. Signed-off-by: John Snow --- tests/qemu-iotests/testenv.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py index 1b095d70f2..6441145701 100644 --- a/tests/qemu-iotests/testenv.py +++ b/tests/qemu-iotests/testenv.py @@ -108,12 +108,8 @@ def init_directories(self) -> None: SAMPLE_IMG_DIR """ - # Path where qemu goodies live in this source tree. - qemu_srctree_path = Path(__file__, '../../../python').resolve() - self.pythonpath = os.pathsep.join(filter(None, ( self.source_iotests, - str(qemu_srctree_path), os.getenv('PYTHONPATH'), )))