From patchwork Wed Apr 20 12:40:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12820178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8C6FFC433F5 for ; Wed, 20 Apr 2022 12:49:50 +0000 (UTC) Received: from localhost ([::1]:41042 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nh9mH-0007aO-GB for qemu-devel@archiver.kernel.org; Wed, 20 Apr 2022 08:49:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56034) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9e1-0002CQ-LG for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:17 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:35530) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9dy-00043n-Qo for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1650458473; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gY7FxXxkC1X0zRLnFGAjAA2kO9J4keZNznHr2+7XkvY=; b=cRzgR9iEkWYmZOx2AxWfZX/y9g9qvPCkIcfN0YeiCt3io79acaOaxO847WA66RJ3rxz6A3 dIgZ1GvYfMdWlXGLTrU80oJcNFVhaJnwG1dwtzPMQ4iOVPSndVVL72bYUH01NdQx1mFT1g vy2EpENZLxNEIoYLsIi4Rke1Wmzc96c= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-502-UKCcla0HMBWtfLksHMj77Q-1; Wed, 20 Apr 2022 08:41:10 -0400 X-MC-Unique: UKCcla0HMBWtfLksHMj77Q-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3885E18E0068; Wed, 20 Apr 2022 12:41:07 +0000 (UTC) Received: from localhost (unknown [10.39.193.151]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E39D440EC005; Wed, 20 Apr 2022 12:41:06 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 1/8] tests/qemu-iotests/meson.build: Improve the indentation Date: Wed, 20 Apr 2022 14:40:57 +0200 Message-Id: <20220420124104.795836-2-hreitz@redhat.com> In-Reply-To: <20220420124104.795836-1-hreitz@redhat.com> References: <20220420124104.795836-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Hanna Reitz , Richard Henderson , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Thomas Huth By using subdir_done(), we can get rid of one level of indentation in this file. This will make it easier to add more conditions to skip the iotests in future patches. Reviewed-by: Hanna Reitz Reviewed-by: Philippe Mathieu-Daudé Signed-off-by: Thomas Huth Message-Id: <20220223093840.2515281-3-thuth@redhat.com> Signed-off-by: Hanna Reitz --- tests/qemu-iotests/meson.build | 61 ++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 29 deletions(-) diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build index 9747bb68a5..92f09251a6 100644 --- a/tests/qemu-iotests/meson.build +++ b/tests/qemu-iotests/meson.build @@ -1,30 +1,33 @@ -if have_tools and targetos != 'windows' and not get_option('gprof') - qemu_iotests_binaries = [qemu_img, qemu_io, qemu_nbd, qsd] - qemu_iotests_env = {'PYTHON': python.full_path()} - qemu_iotests_formats = { - 'qcow2': 'quick', - 'raw': 'slow', - 'qed': 'thorough', - 'vmdk': 'thorough', - 'vpc': 'thorough' - } - - foreach k, v : emulators - if k.startswith('qemu-system-') - qemu_iotests_binaries += v - endif - endforeach - foreach format, speed: qemu_iotests_formats - if speed == 'quick' - suites = 'block' - else - suites = ['block-' + speed, speed] - endif - test('qemu-iotests ' + format, sh, args: [files('../check-block.sh'), format], - depends: qemu_iotests_binaries, env: qemu_iotests_env, - protocol: 'tap', - suite: suites, - timeout: 0, - is_parallel: false) - endforeach +if not have_tools or targetos == 'windows' or get_option('gprof') + subdir_done() endif + +qemu_iotests_binaries = [qemu_img, qemu_io, qemu_nbd, qsd] +qemu_iotests_env = {'PYTHON': python.full_path()} +qemu_iotests_formats = { + 'qcow2': 'quick', + 'raw': 'slow', + 'qed': 'thorough', + 'vmdk': 'thorough', + 'vpc': 'thorough' +} + +foreach k, v : emulators + if k.startswith('qemu-system-') + qemu_iotests_binaries += v + endif +endforeach + +foreach format, speed: qemu_iotests_formats + if speed == 'quick' + suites = 'block' + else + suites = ['block-' + speed, speed] + endif + test('qemu-iotests ' + format, sh, args: [files('../check-block.sh'), format], + depends: qemu_iotests_binaries, env: qemu_iotests_env, + protocol: 'tap', + suite: suites, + timeout: 0, + is_parallel: false) +endforeach From patchwork Wed Apr 20 12:40:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12820170 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9BC1BC433EF for ; Wed, 20 Apr 2022 12:43:12 +0000 (UTC) Received: from localhost ([::1]:54260 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nh9fr-0005EX-DB for qemu-devel@archiver.kernel.org; Wed, 20 Apr 2022 08:43:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55998) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9e0-0002BP-ID for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:16 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:22053) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9dy-00043l-C3 for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1650458473; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Opt3O5zhg+NWCDjkfFDUQ0F2LnpdKGyQDG8VjNIROQ0=; b=BZgfPW8izf1pJ3wiilEkL4seQsQxI9aZhbWtH7J9lnNZF2Lj/ShzQbjwFipzfmY8rjq4bK iOv69gtn4l7OlBxb2ezrgHkNHvp4GTfYNiiVKEPeQ4nAsvOnZZANdhVQlcPEu/KPoV/ydF Qh91HvC/Yk5AvQMLEbMeDB1Ty8I668g= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-3-fguGV2uEMEOiqQ8qBoelNA-1; Wed, 20 Apr 2022 08:41:11 -0400 X-MC-Unique: fguGV2uEMEOiqQ8qBoelNA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B7C2186B8A7; Wed, 20 Apr 2022 12:41:08 +0000 (UTC) Received: from localhost (unknown [10.39.193.151]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 725372166B48; Wed, 20 Apr 2022 12:41:08 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 2/8] tests/qemu-iotests: Move the bash and sanitizer checks to meson.build Date: Wed, 20 Apr 2022 14:40:58 +0200 Message-Id: <20220420124104.795836-3-hreitz@redhat.com> In-Reply-To: <20220420124104.795836-1-hreitz@redhat.com> References: <20220420124104.795836-1-hreitz@redhat.com> MIME-Version: 1.0 Content-type: text/plain X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Hanna Reitz , Richard Henderson , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Thomas Huth We want to get rid of check-block.sh in the long run, so let's move the checks for the bash version and sanitizers from check-block.sh into the meson.build file instead. Signed-off-by: Thomas Huth Message-Id: <20220223093840.2515281-4-thuth@redhat.com> Signed-off-by: Hanna Reitz --- tests/check-block.sh | 26 -------------------------- tests/qemu-iotests/meson.build | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 26 deletions(-) diff --git a/tests/check-block.sh b/tests/check-block.sh index f59496396c..5de2c1ba0b 100755 --- a/tests/check-block.sh +++ b/tests/check-block.sh @@ -18,36 +18,10 @@ skip() { exit 0 } -# Disable tests with any sanitizer except for specific ones -SANITIZE_FLAGS=$( grep "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ) -ALLOWED_SANITIZE_FLAGS="safe-stack cfi-icall" -#Remove all occurrencies of allowed Sanitize flags -for j in ${ALLOWED_SANITIZE_FLAGS}; do - TMP_FLAGS=${SANITIZE_FLAGS} - SANITIZE_FLAGS="" - for i in ${TMP_FLAGS}; do - if ! echo ${i} | grep -q "${j}" 2>/dev/null; then - SANITIZE_FLAGS="${SANITIZE_FLAGS} ${i}" - fi - done -done -if echo ${SANITIZE_FLAGS} | grep -q "\-fsanitize" 2>/dev/null; then - # Have a sanitize flag that is not allowed, stop - skip "Sanitizers are enabled ==> Not running the qemu-iotests." -fi - if [ -z "$(find . -name 'qemu-system-*' -print)" ]; then skip "No qemu-system binary available ==> Not running the qemu-iotests." fi -if ! command -v bash >/dev/null 2>&1 ; then - skip "bash not available ==> Not running the qemu-iotests." -fi - -if LANG=C bash --version | grep -q 'GNU bash, version [123]' ; then - skip "bash version too old ==> Not running the qemu-iotests." -fi - cd tests/qemu-iotests # QEMU_CHECK_BLOCK_AUTO is used to disable some unstable sub-tests diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build index 92f09251a6..323a4acb6a 100644 --- a/tests/qemu-iotests/meson.build +++ b/tests/qemu-iotests/meson.build @@ -2,6 +2,20 @@ if not have_tools or targetos == 'windows' or get_option('gprof') subdir_done() endif +foreach cflag: config_host['QEMU_CFLAGS'].split() + if cflag.startswith('-fsanitize') and \ + not cflag.contains('safe-stack') and not cflag.contains('cfi-icall') + message('Sanitizers are enabled ==> Disabled the qemu-iotests.') + subdir_done() + endif +endforeach + +bash = find_program('bash', required: false, version: '>= 4.0') +if not bash.found() + message('bash >= v4.0 not available ==> Disabled the qemu-iotests.') + subdir_done() +endif + qemu_iotests_binaries = [qemu_img, qemu_io, qemu_nbd, qsd] qemu_iotests_env = {'PYTHON': python.full_path()} qemu_iotests_formats = { From patchwork Wed Apr 20 12:40:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12820175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B3CEFC433F5 for ; Wed, 20 Apr 2022 12:47:49 +0000 (UTC) Received: from localhost ([::1]:35810 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nh9kK-00041o-6I for qemu-devel@archiver.kernel.org; Wed, 20 Apr 2022 08:47:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56002) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9e0-0002BT-KD for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:16 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:37557) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9dy-00043q-Qd for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1650458474; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=URyTzH693l8cpHBjASSKF6GjzautDHhirWhdMg4qe5A=; b=CDrLQrwQ52xAxPond5cx2hw/jN7WZar8frK8VwEuYSvPhuZU+V1gU5XlYKJKtYobebjO/6 6QmtkbGEtw6j+BVN7y7m+JwBor8QgWYOf5RlLSNPvsUjq0J0L7P4cJkMurwJTK+tPZvPBa yILKlkouvSbY0jk3ibofXvK7BmoHroI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-29-nXSMSj0rMjaGiFg7rlwqBg-1; Wed, 20 Apr 2022 08:41:12 -0400 X-MC-Unique: nXSMSj0rMjaGiFg7rlwqBg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 48F001066567; Wed, 20 Apr 2022 12:41:10 +0000 (UTC) Received: from localhost (unknown [10.39.193.151]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 001B340EC005; Wed, 20 Apr 2022 12:41:09 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 3/8] iotests.py: Add supports_qcow2_zstd_compression() Date: Wed, 20 Apr 2022 14:40:59 +0200 Message-Id: <20220420124104.795836-4-hreitz@redhat.com> In-Reply-To: <20220420124104.795836-1-hreitz@redhat.com> References: <20220420124104.795836-1-hreitz@redhat.com> MIME-Version: 1.0 Content-type: text/plain X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Hanna Reitz , Richard Henderson , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Hanna Reitz Message-Id: <20220323105522.53660-2-hreitz@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index fcec3e51e5..fe10a6cf05 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -1471,6 +1471,26 @@ def verify_working_luks(): if not working: notrun(reason) +def supports_qcow2_zstd_compression() -> bool: + img_file = f'{test_dir}/qcow2-zstd-test.qcow2' + res = qemu_img('create', '-f', 'qcow2', '-o', 'compression_type=zstd', + img_file, '0', + check=False) + try: + os.remove(img_file) + except OSError: + pass + + if res.returncode == 1 and \ + "'compression-type' does not accept value 'zstd'" in res.stdout: + return False + else: + return True + +def verify_qcow2_zstd_compression(): + if not supports_qcow2_zstd_compression(): + notrun('zstd compression not supported') + def qemu_pipe(*args: str) -> str: """ Run qemu with an option to print something and exit (e.g. a help option). From patchwork Wed Apr 20 12:41:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12820174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 03AB7C433F5 for ; Wed, 20 Apr 2022 12:47:32 +0000 (UTC) Received: from localhost ([::1]:35308 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nh9k3-0003hP-UK for qemu-devel@archiver.kernel.org; Wed, 20 Apr 2022 08:47:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56074) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9e3-0002Dl-4E for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:20 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:48222) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9e1-000455-CH for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1650458476; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IcGgb7YudwgqGCPiPOi1JwHbHd51sPSxbL0UOVRL7e8=; b=fWys51Bdh9nzDltJUIhO0s5obSpqLwOUtfWkYPYxH+2/4tI7APBaGUBaZ3SxbwnjQcZSCP u6N462KMv6iKl9lePlLX2Mn7hDxpMUjp3tuleG1Ti61XIZ58RhHQG5lFPshAksCo0oQ/a+ Dl08WRQSO5NbZl7rsOeisOfV7HjijQ8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-164-OHdpYEecPaOwJXGV3-Et9Q-1; Wed, 20 Apr 2022 08:41:13 -0400 X-MC-Unique: OHdpYEecPaOwJXGV3-Et9Q-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F151F89C5D7; Wed, 20 Apr 2022 12:41:11 +0000 (UTC) Received: from localhost (unknown [10.39.193.151]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AC79E2024CB6; Wed, 20 Apr 2022 12:41:11 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 4/8] iotests/065: Check for zstd support Date: Wed, 20 Apr 2022 14:41:00 +0200 Message-Id: <20220420124104.795836-5-hreitz@redhat.com> In-Reply-To: <20220420124104.795836-1-hreitz@redhat.com> References: <20220420124104.795836-1-hreitz@redhat.com> MIME-Version: 1.0 Content-type: text/plain X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Hanna Reitz , Richard Henderson , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Some test cases run in iotest 065 want to run with zstd compression just for added coverage. Run them with zlib if there is no zstd support compiled in. Reported-by: Thomas Huth Fixes: 12a936171d71f839dc907ff ("iotest 065: explicit compression type") Signed-off-by: Hanna Reitz Message-Id: <20220323105522.53660-3-hreitz@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/065 | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065 index ba94e19349..b724c89c7c 100755 --- a/tests/qemu-iotests/065 +++ b/tests/qemu-iotests/065 @@ -24,7 +24,7 @@ import os import re import json import iotests -from iotests import qemu_img, qemu_img_info +from iotests import qemu_img, qemu_img_info, supports_qcow2_zstd_compression import unittest test_img = os.path.join(iotests.test_dir, 'test.img') @@ -95,11 +95,17 @@ class TestQCow2(TestQemuImgInfo): class TestQCow3NotLazy(TestQemuImgInfo): '''Testing a qcow2 version 3 image with lazy refcounts disabled''' - img_options = 'compat=1.1,lazy_refcounts=off,compression_type=zstd' + if supports_qcow2_zstd_compression(): + compression_type = 'zstd' + else: + compression_type = 'zlib' + + img_options = 'compat=1.1,lazy_refcounts=off' + img_options += f',compression_type={compression_type}' json_compare = { 'compat': '1.1', 'lazy-refcounts': False, 'refcount-bits': 16, 'corrupt': False, - 'compression-type': 'zstd', 'extended-l2': False } - human_compare = [ 'compat: 1.1', 'compression type: zstd', + 'compression-type': compression_type, 'extended-l2': False } + human_compare = [ 'compat: 1.1', f'compression type: {compression_type}', 'lazy refcounts: false', 'refcount bits: 16', 'corrupt: false', 'extended l2: false' ] @@ -126,11 +132,17 @@ class TestQCow3NotLazyQMP(TestQMP): class TestQCow3LazyQMP(TestQMP): '''Testing a qcow2 version 3 image with lazy refcounts enabled, opening with lazy refcounts disabled''' - img_options = 'compat=1.1,lazy_refcounts=on,compression_type=zstd' + if supports_qcow2_zstd_compression(): + compression_type = 'zstd' + else: + compression_type = 'zlib' + + img_options = 'compat=1.1,lazy_refcounts=on' + img_options += f',compression_type={compression_type}' qemu_options = 'lazy-refcounts=off' compare = { 'compat': '1.1', 'lazy-refcounts': True, 'refcount-bits': 16, 'corrupt': False, - 'compression-type': 'zstd', 'extended-l2': False } + 'compression-type': compression_type, 'extended-l2': False } TestImageInfoSpecific = None TestQemuImgInfo = None From patchwork Wed Apr 20 12:41:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12820179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 543D9C433F5 for ; Wed, 20 Apr 2022 12:53:12 +0000 (UTC) Received: from localhost ([::1]:43820 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nh9pX-000126-9u for qemu-devel@archiver.kernel.org; Wed, 20 Apr 2022 08:53:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56082) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9e3-0002Dz-NK for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:20 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:42715) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9e1-00045J-Va for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1650458477; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Q257GkIszh0EbBU+w+1rQ2Urypi7uipTGe9FX2iKXtw=; b=F1IcJ2oc3r6JxdKYSRkBGmyC9MDdf+T6/UX1xwFWmILgR03M75H4JLDbcrxOOjUNonfjDT /Cgdymzoz/+XeHgo+6cmwuIw2AyWbzvA0QNKKM6nyhAVvAH0y1qIKhQ6E090aCWmftuAGz QxfIN5ON/wx+rnhXtEpKIDdx2nBUxWM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-332-R2YQO4amM8-pw7wBapv9Jw-1; Wed, 20 Apr 2022 08:41:14 -0400 X-MC-Unique: R2YQO4amM8-pw7wBapv9Jw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8767A18F0240; Wed, 20 Apr 2022 12:41:13 +0000 (UTC) Received: from localhost (unknown [10.39.193.151]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 41279111E3EA; Wed, 20 Apr 2022 12:41:13 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 5/8] iotests/303: Check for zstd support Date: Wed, 20 Apr 2022 14:41:01 +0200 Message-Id: <20220420124104.795836-6-hreitz@redhat.com> In-Reply-To: <20220420124104.795836-1-hreitz@redhat.com> References: <20220420124104.795836-1-hreitz@redhat.com> MIME-Version: 1.0 Content-type: text/plain X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Hanna Reitz , Richard Henderson , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" 303 runs two test cases, one of which requires zstd support. Unfortunately, given that this is not a unittest-style test, we cannot easily skip that single case, and instead can only skip the whole test. (Alternatively, we could split this test into a zlib and a zstd part, but that seems excessive, given that this test is not in auto and thus likely only run by developers who have zstd support compiled in.) Fixes: 677e0bae686e7c670a71d1f ("iotest 303: explicit compression type") Signed-off-by: Hanna Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <20220323105522.53660-4-hreitz@redhat.com> --- tests/qemu-iotests/303 | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/303 b/tests/qemu-iotests/303 index 93aa5ce9b7..40e947f26c 100755 --- a/tests/qemu-iotests/303 +++ b/tests/qemu-iotests/303 @@ -21,10 +21,12 @@ import iotests import subprocess -from iotests import qemu_img_create, qemu_io, file_path, log, filter_qemu_io +from iotests import qemu_img_create, qemu_io, file_path, log, filter_qemu_io, \ + verify_qcow2_zstd_compression iotests.script_initialize(supported_fmts=['qcow2'], unsupported_imgopts=['refcount_bits', 'compat']) +verify_qcow2_zstd_compression() disk = file_path('disk') chunk = 1024 * 1024 From patchwork Wed Apr 20 12:41:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12820171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 93538C433F5 for ; Wed, 20 Apr 2022 12:43:14 +0000 (UTC) Received: from localhost ([::1]:54408 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nh9ft-0005NW-J6 for qemu-devel@archiver.kernel.org; Wed, 20 Apr 2022 08:43:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56122) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9e4-0002FL-No for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:22 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:35092) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9e1-00045L-Vt for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1650458477; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HOVPnb7qLsKScZDmafIoqnna6aPyF+7U8JLaxW3X37A=; b=EjuPraKslfuf8TFqOlEzCa44lM94BqDheX+PwvA/tjTlSDQgpjhqqyIVsEosu/n+WfTqtt u76Y1M2mZU5uEInzdgyMmRc7AElYP3sfStjnLRKpFM9Fkt09sqMEd3c9jLBIophjoW2yo8 ZB2DT0ZltB/zA/29PkAMhXWWn3e9Jeg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-531-7dxdNRjmPDaBmqaYfoFbrg-1; Wed, 20 Apr 2022 08:41:15 -0400 X-MC-Unique: 7dxdNRjmPDaBmqaYfoFbrg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5A2B618F0261; Wed, 20 Apr 2022 12:41:15 +0000 (UTC) Received: from localhost (unknown [10.39.193.151]) by smtp.corp.redhat.com (Postfix) with ESMTPS id CB3F2111E3EA; Wed, 20 Apr 2022 12:41:14 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 6/8] qcow2: Improve refcount structure rebuilding Date: Wed, 20 Apr 2022 14:41:02 +0200 Message-Id: <20220420124104.795836-7-hreitz@redhat.com> In-Reply-To: <20220420124104.795836-1-hreitz@redhat.com> References: <20220420124104.795836-1-hreitz@redhat.com> MIME-Version: 1.0 Content-type: text/plain X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Hanna Reitz , Richard Henderson , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" When rebuilding the refcount structures (when qemu-img check -r found errors with refcount = 0, but reference count > 0), the new refcount table defaults to being put at the image file end[1]. There is no good reason for that except that it means we will not have to rewrite any refblocks we already wrote to disk. Changing the code to rewrite those refblocks is not too difficult, though, so let us do that. That is beneficial for images on block devices, where we cannot really write beyond the end of the image file. Use this opportunity to add extensive comments to the code, and refactor it a bit, getting rid of the backwards-jumping goto. [1] Unless there is something allocated in the area pointed to by the last refblock, so we have to write that refblock. In that case, we try to put the reftable in there. Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1519071 Closes: https://gitlab.com/qemu-project/qemu/-/issues/941 Reviewed-by: Eric Blake Signed-off-by: Hanna Reitz Message-Id: <20220405134652.19278-2-hreitz@redhat.com> --- block/qcow2-refcount.c | 332 +++++++++++++++++++++++++++++------------ 1 file changed, 235 insertions(+), 97 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index b91499410c..c5669eaa51 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -2438,111 +2438,140 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs, } /* - * Creates a new refcount structure based solely on the in-memory information - * given through *refcount_table. All necessary allocations will be reflected - * in that array. + * Helper function for rebuild_refcount_structure(). * - * On success, the old refcount structure is leaked (it will be covered by the - * new refcount structure). + * Scan the range of clusters [first_cluster, end_cluster) for allocated + * clusters and write all corresponding refblocks to disk. The refblock + * and allocation data is taken from the in-memory refcount table + * *refcount_table[] (of size *nb_clusters), which is basically one big + * (unlimited size) refblock for the whole image. + * + * For these refblocks, clusters are allocated using said in-memory + * refcount table. Care is taken that these allocations are reflected + * in the refblocks written to disk. + * + * The refblocks' offsets are written into a reftable, which is + * *on_disk_reftable_ptr[] (of size *on_disk_reftable_entries_ptr). If + * that reftable is of insufficient size, it will be resized to fit. + * This reftable is not written to disk. + * + * (If *on_disk_reftable_ptr is not NULL, the entries within are assumed + * to point to existing valid refblocks that do not need to be allocated + * again.) + * + * Return whether the on-disk reftable array was resized (true/false), + * or -errno on error. */ -static int rebuild_refcount_structure(BlockDriverState *bs, - BdrvCheckResult *res, - void **refcount_table, - int64_t *nb_clusters) +static int rebuild_refcounts_write_refblocks( + BlockDriverState *bs, void **refcount_table, int64_t *nb_clusters, + int64_t first_cluster, int64_t end_cluster, + uint64_t **on_disk_reftable_ptr, uint32_t *on_disk_reftable_entries_ptr + ) { BDRVQcow2State *s = bs->opaque; - int64_t first_free_cluster = 0, reftable_offset = -1, cluster = 0; + int64_t cluster; int64_t refblock_offset, refblock_start, refblock_index; - uint32_t reftable_size = 0; - uint64_t *on_disk_reftable = NULL; + int64_t first_free_cluster = 0; + uint64_t *on_disk_reftable = *on_disk_reftable_ptr; + uint32_t on_disk_reftable_entries = *on_disk_reftable_entries_ptr; void *on_disk_refblock; - int ret = 0; - struct { - uint64_t reftable_offset; - uint32_t reftable_clusters; - } QEMU_PACKED reftable_offset_and_clusters; - - qcow2_cache_empty(bs, s->refcount_block_cache); + bool reftable_grown = false; + int ret; -write_refblocks: - for (; cluster < *nb_clusters; cluster++) { + for (cluster = first_cluster; cluster < end_cluster; cluster++) { + /* Check all clusters to find refblocks that contain non-zero entries */ if (!s->get_refcount(*refcount_table, cluster)) { continue; } + /* + * This cluster is allocated, so we need to create a refblock + * for it. The data we will write to disk is just the + * respective slice from *refcount_table, so it will contain + * accurate refcounts for all clusters belonging to this + * refblock. After we have written it, we will therefore skip + * all remaining clusters in this refblock. + */ + refblock_index = cluster >> s->refcount_block_bits; refblock_start = refblock_index << s->refcount_block_bits; - /* Don't allocate a cluster in a refblock already written to disk */ - if (first_free_cluster < refblock_start) { - first_free_cluster = refblock_start; - } - refblock_offset = alloc_clusters_imrt(bs, 1, refcount_table, - nb_clusters, &first_free_cluster); - if (refblock_offset < 0) { - fprintf(stderr, "ERROR allocating refblock: %s\n", - strerror(-refblock_offset)); - res->check_errors++; - ret = refblock_offset; - goto fail; - } + if (on_disk_reftable_entries > refblock_index && + on_disk_reftable[refblock_index]) + { + /* + * We can get here after a `goto write_refblocks`: We have a + * reftable from a previous run, and the refblock is already + * allocated. No need to allocate it again. + */ + refblock_offset = on_disk_reftable[refblock_index]; + } else { + int64_t refblock_cluster_index; - if (reftable_size <= refblock_index) { - uint32_t old_reftable_size = reftable_size; - uint64_t *new_on_disk_reftable; + /* Don't allocate a cluster in a refblock already written to disk */ + if (first_free_cluster < refblock_start) { + first_free_cluster = refblock_start; + } + refblock_offset = alloc_clusters_imrt(bs, 1, refcount_table, + nb_clusters, + &first_free_cluster); + if (refblock_offset < 0) { + fprintf(stderr, "ERROR allocating refblock: %s\n", + strerror(-refblock_offset)); + return refblock_offset; + } - reftable_size = ROUND_UP((refblock_index + 1) * REFTABLE_ENTRY_SIZE, - s->cluster_size) / REFTABLE_ENTRY_SIZE; - new_on_disk_reftable = g_try_realloc(on_disk_reftable, - reftable_size * - REFTABLE_ENTRY_SIZE); - if (!new_on_disk_reftable) { - res->check_errors++; - ret = -ENOMEM; - goto fail; + refblock_cluster_index = refblock_offset / s->cluster_size; + if (refblock_cluster_index >= end_cluster) { + /* + * We must write the refblock that holds this refblock's + * refcount + */ + end_cluster = refblock_cluster_index + 1; } - on_disk_reftable = new_on_disk_reftable; - memset(on_disk_reftable + old_reftable_size, 0, - (reftable_size - old_reftable_size) * REFTABLE_ENTRY_SIZE); + if (on_disk_reftable_entries <= refblock_index) { + on_disk_reftable_entries = + ROUND_UP((refblock_index + 1) * REFTABLE_ENTRY_SIZE, + s->cluster_size) / REFTABLE_ENTRY_SIZE; + on_disk_reftable = + g_try_realloc(on_disk_reftable, + on_disk_reftable_entries * + REFTABLE_ENTRY_SIZE); + if (!on_disk_reftable) { + return -ENOMEM; + } - /* The offset we have for the reftable is now no longer valid; - * this will leak that range, but we can easily fix that by running - * a leak-fixing check after this rebuild operation */ - reftable_offset = -1; - } else { - assert(on_disk_reftable); - } - on_disk_reftable[refblock_index] = refblock_offset; + memset(on_disk_reftable + *on_disk_reftable_entries_ptr, 0, + (on_disk_reftable_entries - + *on_disk_reftable_entries_ptr) * + REFTABLE_ENTRY_SIZE); - /* If this is apparently the last refblock (for now), try to squeeze the - * reftable in */ - if (refblock_index == (*nb_clusters - 1) >> s->refcount_block_bits && - reftable_offset < 0) - { - uint64_t reftable_clusters = size_to_clusters(s, reftable_size * - REFTABLE_ENTRY_SIZE); - reftable_offset = alloc_clusters_imrt(bs, reftable_clusters, - refcount_table, nb_clusters, - &first_free_cluster); - if (reftable_offset < 0) { - fprintf(stderr, "ERROR allocating reftable: %s\n", - strerror(-reftable_offset)); - res->check_errors++; - ret = reftable_offset; - goto fail; + *on_disk_reftable_ptr = on_disk_reftable; + *on_disk_reftable_entries_ptr = on_disk_reftable_entries; + + reftable_grown = true; + } else { + assert(on_disk_reftable); } + on_disk_reftable[refblock_index] = refblock_offset; } + /* Refblock is allocated, write it to disk */ + ret = qcow2_pre_write_overlap_check(bs, 0, refblock_offset, s->cluster_size, false); if (ret < 0) { fprintf(stderr, "ERROR writing refblock: %s\n", strerror(-ret)); - goto fail; + return ret; } - /* The size of *refcount_table is always cluster-aligned, therefore the - * write operation will not overflow */ + /* + * The refblock is simply a slice of *refcount_table. + * Note that the size of *refcount_table is always aligned to + * whole clusters, so the write operation will not result in + * out-of-bounds accesses. + */ on_disk_refblock = (void *)((char *) *refcount_table + refblock_index * s->cluster_size); @@ -2550,23 +2579,99 @@ write_refblocks: s->cluster_size); if (ret < 0) { fprintf(stderr, "ERROR writing refblock: %s\n", strerror(-ret)); - goto fail; + return ret; } - /* Go to the end of this refblock */ + /* This refblock is done, skip to its end */ cluster = refblock_start + s->refcount_block_size - 1; } - if (reftable_offset < 0) { - uint64_t post_refblock_start, reftable_clusters; + return reftable_grown; +} + +/* + * Creates a new refcount structure based solely on the in-memory information + * given through *refcount_table (this in-memory information is basically just + * the concatenation of all refblocks). All necessary allocations will be + * reflected in that array. + * + * On success, the old refcount structure is leaked (it will be covered by the + * new refcount structure). + */ +static int rebuild_refcount_structure(BlockDriverState *bs, + BdrvCheckResult *res, + void **refcount_table, + int64_t *nb_clusters) +{ + BDRVQcow2State *s = bs->opaque; + int64_t reftable_offset = -1; + int64_t reftable_length = 0; + int64_t reftable_clusters; + int64_t refblock_index; + uint32_t on_disk_reftable_entries = 0; + uint64_t *on_disk_reftable = NULL; + int ret = 0; + int reftable_size_changed = 0; + struct { + uint64_t reftable_offset; + uint32_t reftable_clusters; + } QEMU_PACKED reftable_offset_and_clusters; + + qcow2_cache_empty(bs, s->refcount_block_cache); + + /* + * For each refblock containing entries, we try to allocate a + * cluster (in the in-memory refcount table) and write its offset + * into on_disk_reftable[]. We then write the whole refblock to + * disk (as a slice of the in-memory refcount table). + * This is done by rebuild_refcounts_write_refblocks(). + * + * Once we have scanned all clusters, we try to find space for the + * reftable. This will dirty the in-memory refcount table (i.e. + * make it differ from the refblocks we have already written), so we + * need to run rebuild_refcounts_write_refblocks() again for the + * range of clusters where the reftable has been allocated. + * + * This second run might make the reftable grow again, in which case + * we will need to allocate another space for it, which is why we + * repeat all this until the reftable stops growing. + * + * (This loop will terminate, because with every cluster the + * reftable grows, it can accomodate a multitude of more refcounts, + * so that at some point this must be able to cover the reftable + * and all refblocks describing it.) + * + * We then convert the reftable to big-endian and write it to disk. + * + * Note that we never free any reftable allocations. Doing so would + * needlessly complicate the algorithm: The eventual second check + * run we do will clean up all leaks we have caused. + */ + + reftable_size_changed = + rebuild_refcounts_write_refblocks(bs, refcount_table, nb_clusters, + 0, *nb_clusters, + &on_disk_reftable, + &on_disk_reftable_entries); + if (reftable_size_changed < 0) { + res->check_errors++; + ret = reftable_size_changed; + goto fail; + } + + /* + * There was no reftable before, so rebuild_refcounts_write_refblocks() + * must have increased its size (from 0 to something). + */ + assert(reftable_size_changed); + + do { + int64_t reftable_start_cluster, reftable_end_cluster; + int64_t first_free_cluster = 0; + + reftable_length = on_disk_reftable_entries * REFTABLE_ENTRY_SIZE; + reftable_clusters = size_to_clusters(s, reftable_length); - post_refblock_start = ROUND_UP(*nb_clusters, s->refcount_block_size); - reftable_clusters = - size_to_clusters(s, reftable_size * REFTABLE_ENTRY_SIZE); - /* Not pretty but simple */ - if (first_free_cluster < post_refblock_start) { - first_free_cluster = post_refblock_start; - } reftable_offset = alloc_clusters_imrt(bs, reftable_clusters, refcount_table, nb_clusters, &first_free_cluster); @@ -2578,24 +2683,55 @@ write_refblocks: goto fail; } - goto write_refblocks; - } + /* + * We need to update the affected refblocks, so re-run the + * write_refblocks loop for the reftable's range of clusters. + */ + assert(offset_into_cluster(s, reftable_offset) == 0); + reftable_start_cluster = reftable_offset / s->cluster_size; + reftable_end_cluster = reftable_start_cluster + reftable_clusters; + reftable_size_changed = + rebuild_refcounts_write_refblocks(bs, refcount_table, nb_clusters, + reftable_start_cluster, + reftable_end_cluster, + &on_disk_reftable, + &on_disk_reftable_entries); + if (reftable_size_changed < 0) { + res->check_errors++; + ret = reftable_size_changed; + goto fail; + } + + /* + * If the reftable size has changed, we will need to find a new + * allocation, repeating the loop. + */ + } while (reftable_size_changed); - for (refblock_index = 0; refblock_index < reftable_size; refblock_index++) { + /* The above loop must have run at least once */ + assert(reftable_offset >= 0); + + /* + * All allocations are done, all refblocks are written, convert the + * reftable to big-endian and write it to disk. + */ + + for (refblock_index = 0; refblock_index < on_disk_reftable_entries; + refblock_index++) + { cpu_to_be64s(&on_disk_reftable[refblock_index]); } - ret = qcow2_pre_write_overlap_check(bs, 0, reftable_offset, - reftable_size * REFTABLE_ENTRY_SIZE, + ret = qcow2_pre_write_overlap_check(bs, 0, reftable_offset, reftable_length, false); if (ret < 0) { fprintf(stderr, "ERROR writing reftable: %s\n", strerror(-ret)); goto fail; } - assert(reftable_size < INT_MAX / REFTABLE_ENTRY_SIZE); + assert(reftable_length < INT_MAX); ret = bdrv_pwrite(bs->file, reftable_offset, on_disk_reftable, - reftable_size * REFTABLE_ENTRY_SIZE); + reftable_length); if (ret < 0) { fprintf(stderr, "ERROR writing reftable: %s\n", strerror(-ret)); goto fail; @@ -2604,7 +2740,7 @@ write_refblocks: /* Enter new reftable into the image header */ reftable_offset_and_clusters.reftable_offset = cpu_to_be64(reftable_offset); reftable_offset_and_clusters.reftable_clusters = - cpu_to_be32(size_to_clusters(s, reftable_size * REFTABLE_ENTRY_SIZE)); + cpu_to_be32(reftable_clusters); ret = bdrv_pwrite_sync(bs->file, offsetof(QCowHeader, refcount_table_offset), &reftable_offset_and_clusters, @@ -2614,12 +2750,14 @@ write_refblocks: goto fail; } - for (refblock_index = 0; refblock_index < reftable_size; refblock_index++) { + for (refblock_index = 0; refblock_index < on_disk_reftable_entries; + refblock_index++) + { be64_to_cpus(&on_disk_reftable[refblock_index]); } s->refcount_table = on_disk_reftable; s->refcount_table_offset = reftable_offset; - s->refcount_table_size = reftable_size; + s->refcount_table_size = on_disk_reftable_entries; update_max_refcount_table_index(s); return 0; From patchwork Wed Apr 20 12:41:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12820180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 936FDC4332F for ; Wed, 20 Apr 2022 12:54:49 +0000 (UTC) Received: from localhost ([::1]:46992 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nh9r6-0003HK-LZ for qemu-devel@archiver.kernel.org; Wed, 20 Apr 2022 08:54:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56202) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9e7-0002KP-Vv for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:24 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:56433) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9e5-00048v-GM for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1650458481; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O5IuLuVWlvcqQWXSk0cvHcOsuCEISPNagguu1t0IxWQ=; b=SGzx9sbMo6n7C35aqS1I1RrnleH/SOvkiY1tuzfAQD742LNpbZo4b+i8FtPqjCYZ3X5RJW xuF29dOfzaCeSTnZHybxGi2z3nLi8aKGwYKiJ71b3O456h0sd/9w+zslNaFYDNEd0fhFd0 H1bU/kH3h2ufxJqS/oNSLI/pbw+wdVg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-67-n996PKpoP7eSJ8yzBJyusw-1; Wed, 20 Apr 2022 08:41:17 -0400 X-MC-Unique: n996PKpoP7eSJ8yzBJyusw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2B513857AA6; Wed, 20 Apr 2022 12:41:17 +0000 (UTC) Received: from localhost (unknown [10.39.193.151]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9BD5E4104970; Wed, 20 Apr 2022 12:41:16 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 7/8] iotests/108: Test new refcount rebuild algorithm Date: Wed, 20 Apr 2022 14:41:03 +0200 Message-Id: <20220420124104.795836-8-hreitz@redhat.com> In-Reply-To: <20220420124104.795836-1-hreitz@redhat.com> References: <20220420124104.795836-1-hreitz@redhat.com> MIME-Version: 1.0 Content-type: text/plain X-Scanned-By: MIMEDefang 2.84 on 10.11.54.1 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Hanna Reitz , Richard Henderson , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" One clear problem with how qcow2's refcount structure rebuild algorithm used to be before "qcow2: Improve refcount structure rebuilding" was that it is prone to failure for qcow2 images on block devices: There is generally unused space after the actual image, and if that exceeds what one refblock covers, the old algorithm would invariably write the reftable past the block device's end, which cannot work. The new algorithm does not have this problem. Test it with three tests: (1) Create an image with more empty space at the end than what one refblock covers, see whether rebuilding the refcount structures results in a change in the image file length. (It should not.) (2) Leave precisely enough space somewhere at the beginning of the image for the new reftable (and the refblock for that place), see whether the new algorithm puts the reftable there. (It should.) (3) Test the original problem: Create (something like) a block device with a fixed size, then create a qcow2 image in there, write some data, and then have qemu-img check rebuild the refcount structures. Before HEAD^, the reftable would have been written past the image file end, i.e. outside of what the block device provides, which cannot work. HEAD^ should have fixed that. ("Something like a block device" means a loop device if we can use one ("sudo -n losetup" works), or a FUSE block export with growable=false otherwise.) Reviewed-by: Eric Blake Signed-off-by: Hanna Reitz Message-Id: <20220405134652.19278-3-hreitz@redhat.com> --- tests/qemu-iotests/108 | 259 ++++++++++++++++++++++++++++++++++++- tests/qemu-iotests/108.out | 81 ++++++++++++ 2 files changed, 339 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/108 b/tests/qemu-iotests/108 index 56339ab2c5..688d3ae8f6 100755 --- a/tests/qemu-iotests/108 +++ b/tests/qemu-iotests/108 @@ -30,13 +30,20 @@ status=1 # failure is the default! _cleanup() { - _cleanup_test_img + _cleanup_test_img + if [ -f "$TEST_DIR/qsd.pid" ]; then + qsd_pid=$(cat "$TEST_DIR/qsd.pid") + kill -KILL "$qsd_pid" + fusermount -u "$TEST_DIR/fuse-export" &>/dev/null + fi + rm -f "$TEST_DIR/fuse-export" } trap "_cleanup; exit \$status" 0 1 2 3 15 # get standard environment, filters and checks . ./common.rc . ./common.filter +. ./common.qemu # This tests qcow2-specific low-level functionality _supported_fmt qcow2 @@ -47,6 +54,22 @@ _supported_os Linux # files _unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)' data_file +# This test either needs sudo -n losetup or FUSE exports to work +if sudo -n losetup &>/dev/null; then + loopdev=true +else + loopdev=false + + # QSD --export fuse will either yield "Parameter 'id' is missing" + # or "Invalid parameter 'fuse'", depending on whether there is + # FUSE support or not. + error=$($QSD --export fuse 2>&1) + if [[ $error = *"'fuse'"* ]]; then + _notrun 'Passwordless sudo for losetup or FUSE support required, but' \ + 'neither is available' + fi +fi + echo echo '=== Repairing an image without any refcount table ===' echo @@ -138,6 +161,240 @@ _make_test_img 64M poke_file "$TEST_IMG" $((0x10008)) "\xff\xff\xff\xff\xff\xff\x00\x00" _check_test_img -r all +echo +echo '=== Check rebuilt reftable location ===' + +# In an earlier version of the refcount rebuild algorithm, the +# reftable was generally placed at the image end (unless something was +# allocated in the area covered by the refblock right before the image +# file end, then we would try to place the reftable in that refblock). +# This was later changed so the reftable would be placed in the +# earliest possible location. Test this. + +echo +echo '--- Does the image size increase? ---' +echo + +# First test: Just create some image, write some data to it, and +# resize it so there is free space at the end of the image (enough +# that it spans at least one full refblock, which for cluster_size=512 +# images, spans 128k). With the old algorithm, the reftable would +# have then been placed at the end of the image file, but with the new +# one, it will be put in that free space. +# We want to check whether the size of the image file increases due to +# rebuilding the refcount structures (it should not). + +_make_test_img -o 'cluster_size=512' 1M +# Write something +$QEMU_IO -c 'write 0 64k' "$TEST_IMG" | _filter_qemu_io + +# Add free space +file_len=$(stat -c '%s' "$TEST_IMG") +truncate -s $((file_len + 256 * 1024)) "$TEST_IMG" + +# Corrupt the image by saying the image header was not allocated +rt_offset=$(peek_file_be "$TEST_IMG" 48 8) +rb_offset=$(peek_file_be "$TEST_IMG" $rt_offset 8) +poke_file "$TEST_IMG" $rb_offset "\x00\x00" + +# Check whether rebuilding the refcount structures increases the image +# file size +file_len=$(stat -c '%s' "$TEST_IMG") +echo +# The only leaks there can be are the old refcount structures that are +# leaked during rebuilding, no need to clutter the output with them +_check_test_img -r all | grep -v '^Repairing cluster.*refcount=1 reference=0' +echo +post_repair_file_len=$(stat -c '%s' "$TEST_IMG") + +if [[ $file_len -eq $post_repair_file_len ]]; then + echo 'OK: Image size did not change' +else + echo 'ERROR: Image size differs' \ + "($file_len before, $post_repair_file_len after)" +fi + +echo +echo '--- Will the reftable occupy a hole specifically left for it? ---' +echo + +# Note: With cluster_size=512, every refblock covers 128k. +# The reftable covers 8M per reftable cluster. + +# Create an image that requires two reftable clusters (just because +# this is more interesting than a single-clustered reftable). +_make_test_img -o 'cluster_size=512' 9M +$QEMU_IO -c 'write 0 8M' "$TEST_IMG" | _filter_qemu_io + +# Writing 8M will have resized the reftable. Unfortunately, doing so +# will leave holes in the file, so we need to fill them up so we can +# be sure the whole file is allocated. Do that by writing +# consecutively smaller chunks starting from 8 MB, until the file +# length increases even with a chunk size of 512. Then we must have +# filled all holes. +ofs=$((8 * 1024 * 1024)) +block_len=$((16 * 1024)) +while [[ $block_len -ge 512 ]]; do + file_len=$(stat -c '%s' "$TEST_IMG") + while [[ $(stat -c '%s' "$TEST_IMG") -eq $file_len ]]; do + # Do not include this in the reference output, it does not + # really matter which qemu-io calls we do here exactly + $QEMU_IO -c "write $ofs $block_len" "$TEST_IMG" >/dev/null + ofs=$((ofs + block_len)) + done + block_len=$((block_len / 2)) +done + +# Fill up to 9M (do not include this in the reference output either, +# $ofs is random for all we know) +$QEMU_IO -c "write $ofs $((9 * 1024 * 1024 - ofs))" "$TEST_IMG" >/dev/null + +# Make space as follows: +# - For the first refblock: Right at the beginning of the image (this +# refblock is placed in the first place possible), +# - For the reftable somewhere soon afterwards, still near the +# beginning of the image (i.e. covered by the first refblock); the +# reftable too is placed in the first place possible, but only after +# all refblocks have been placed) +# No space is needed for the other refblocks, because no refblock is +# put before the space it covers. In this test case, we do not mind +# if they are placed at the image file's end. + +# Before we make that space, we have to find out the host offset of +# the area that belonged to the two data clusters at guest offset 4k, +# because we expect the reftable to be placed there, and we will have +# to verify that it is. + +l1_offset=$(peek_file_be "$TEST_IMG" 40 8) +l2_offset=$(peek_file_be "$TEST_IMG" $l1_offset 8) +l2_offset=$((l2_offset & 0x00fffffffffffe00)) +data_4k_offset=$(peek_file_be "$TEST_IMG" \ + $((l2_offset + 4096 / 512 * 8)) 8) +data_4k_offset=$((data_4k_offset & 0x00fffffffffffe00)) + +$QEMU_IO -c "discard 0 512" -c "discard 4k 1k" "$TEST_IMG" | _filter_qemu_io + +# Corrupt the image by saying the image header was not allocated +rt_offset=$(peek_file_be "$TEST_IMG" 48 8) +rb_offset=$(peek_file_be "$TEST_IMG" $rt_offset 8) +poke_file "$TEST_IMG" $rb_offset "\x00\x00" + +echo +# The only leaks there can be are the old refcount structures that are +# leaked during rebuilding, no need to clutter the output with them +_check_test_img -r all | grep -v '^Repairing cluster.*refcount=1 reference=0' +echo + +# Check whether the reftable was put where we expected +rt_offset=$(peek_file_be "$TEST_IMG" 48 8) +if [[ $rt_offset -eq $data_4k_offset ]]; then + echo 'OK: Reftable is where we expect it' +else + echo "ERROR: Reftable is at $rt_offset, but was expected at $data_4k_offset" +fi + +echo +echo '--- Rebuilding refcount structures on block devices ---' +echo + +# A block device cannot really grow, at least not during qemu-img +# check. As mentioned in the above cases, rebuilding the refcount +# structure may lead to new refcount structures being written after +# the end of the image, and in the past that happened even if there +# was more than sufficient space in the image. Such post-EOF writes +# will not work on block devices, so test that the new algorithm +# avoids it. + +# If we have passwordless sudo and losetup, we can use those to create +# a block device. Otherwise, we can resort to qemu's FUSE export to +# create a file that isn't growable, which effectively tests the same +# thing. + +_cleanup_test_img +truncate -s $((64 * 1024 * 1024)) "$TEST_IMG" + +if $loopdev; then + export_mp=$(sudo -n losetup --show -f "$TEST_IMG") + export_mp_driver=host_device + sudo -n chmod go+rw "$export_mp" +else + # Create non-growable FUSE export that is a bit like an empty + # block device + export_mp="$TEST_DIR/fuse-export" + export_mp_driver=file + touch "$export_mp" + + $QSD \ + --blockdev file,node-name=export-node,filename="$TEST_IMG" \ + --export fuse,id=fuse-export,node-name=export-node,mountpoint="$export_mp",writable=on,growable=off \ + --pidfile "$TEST_DIR/qsd.pid" \ + --daemonize +fi + +# Now create a qcow2 image on the device -- unfortunately, qemu-img +# create force-creates the file, so we have to resort to the +# blockdev-create job. +_launch_qemu \ + --blockdev $export_mp_driver,node-name=file,filename="$export_mp" + +_send_qemu_cmd \ + $QEMU_HANDLE \ + '{ "execute": "qmp_capabilities" }' \ + 'return' + +# Small cluster size again, so the image needs multiple refblocks +_send_qemu_cmd \ + $QEMU_HANDLE \ + '{ "execute": "blockdev-create", + "arguments": { + "job-id": "create", + "options": { + "driver": "qcow2", + "file": "file", + "size": '$((64 * 1024 * 1024))', + "cluster-size": 512 + } } }' \ + '"concluded"' + +_send_qemu_cmd \ + $QEMU_HANDLE \ + '{ "execute": "job-dismiss", "arguments": { "id": "create" } }' \ + 'return' + +_send_qemu_cmd \ + $QEMU_HANDLE \ + '{ "execute": "quit" }' \ + 'return' + +wait=y _cleanup_qemu +echo + +# Write some data +$QEMU_IO -c 'write 0 64k' "$export_mp" | _filter_qemu_io + +# Corrupt the image by saying the image header was not allocated +rt_offset=$(peek_file_be "$export_mp" 48 8) +rb_offset=$(peek_file_be "$export_mp" $rt_offset 8) +poke_file "$export_mp" $rb_offset "\x00\x00" + +# Repairing such a simple case should just work +# (We used to put the reftable at the end of the image file, which can +# never work for non-growable devices.) +echo +TEST_IMG="$export_mp" _check_test_img -r all \ + | grep -v '^Repairing cluster.*refcount=1 reference=0' + +if $loopdev; then + sudo -n losetup -d "$export_mp" +else + qsd_pid=$(cat "$TEST_DIR/qsd.pid") + kill -TERM "$qsd_pid" + # Wait for process to exit (cannot `wait` because the QSD is daemonized) + while [ -f "$TEST_DIR/qsd.pid" ]; do + true + done +fi + # success, all done echo '*** done' rm -f $seq.full diff --git a/tests/qemu-iotests/108.out b/tests/qemu-iotests/108.out index 75bab8dc84..b5401d788d 100644 --- a/tests/qemu-iotests/108.out +++ b/tests/qemu-iotests/108.out @@ -105,6 +105,87 @@ The following inconsistencies were found and repaired: 0 leaked clusters 1 corruptions +Double checking the fixed image now... +No errors were found on the image. + +=== Check rebuilt reftable location === + +--- Does the image size increase? --- + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 +wrote 65536/65536 bytes at offset 0 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +ERROR cluster 0 refcount=0 reference=1 +Rebuilding refcount structure +The following inconsistencies were found and repaired: + + 0 leaked clusters + 1 corruptions + +Double checking the fixed image now... +No errors were found on the image. + +OK: Image size did not change + +--- Will the reftable occupy a hole specifically left for it? --- + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=9437184 +wrote 8388608/8388608 bytes at offset 0 +8 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +discard 512/512 bytes at offset 0 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +discard 1024/1024 bytes at offset 4096 +1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +ERROR cluster 0 refcount=0 reference=1 +Rebuilding refcount structure +The following inconsistencies were found and repaired: + + 0 leaked clusters + 1 corruptions + +Double checking the fixed image now... +No errors were found on the image. + +OK: Reftable is where we expect it + +--- Rebuilding refcount structures on block devices --- + +{ "execute": "qmp_capabilities" } +{"return": {}} +{ "execute": "blockdev-create", + "arguments": { + "job-id": "create", + "options": { + "driver": "IMGFMT", + "file": "file", + "size": 67108864, + "cluster-size": 512 + } } } +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "create"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "create"}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "create"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "create"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "create"}} +{ "execute": "job-dismiss", "arguments": { "id": "create" } } +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "create"}} +{"return": {}} +{ "execute": "quit" } +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} + +wrote 65536/65536 bytes at offset 0 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +ERROR cluster 0 refcount=0 reference=1 +Rebuilding refcount structure +The following inconsistencies were found and repaired: + + 0 leaked clusters + 1 corruptions + Double checking the fixed image now... No errors were found on the image. *** done From patchwork Wed Apr 20 12:41:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12820181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A5D4DC433EF for ; Wed, 20 Apr 2022 12:57:24 +0000 (UTC) Received: from localhost ([::1]:50666 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nh9tb-00065M-Pz for qemu-devel@archiver.kernel.org; Wed, 20 Apr 2022 08:57:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56210) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9e8-0002Km-L3 for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:24 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:46770) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nh9e6-00049J-Sn for qemu-devel@nongnu.org; Wed, 20 Apr 2022 08:41:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1650458482; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f1lbctqjoYWtE2nKk+RStis+7Aqzle8szyXHu/XF1gM=; b=XxRmeMI6zYwDa6dNXUk8zY55WjWvqCj9c/4kjs0WYGmvkXjeraFZD0n4xOxDBb9cXSrjQF xact4LzHDIoIzhDI8smpbOUNCZdTxWdwZNFkRjD68KTU7/io0DTF0ydazUozopgbv2W/O+ SywHRkwSDEM3YdlZ+chQtnLP5wZmpo0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-643-g5iFGWYdMCe0GhPsDJunEg-1; Wed, 20 Apr 2022 08:41:19 -0400 X-MC-Unique: g5iFGWYdMCe0GhPsDJunEg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BAF2A802819; Wed, 20 Apr 2022 12:41:18 +0000 (UTC) Received: from localhost (unknown [10.39.193.151]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 72D7340EC001; Wed, 20 Apr 2022 12:41:18 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 8/8] qcow2: Add errp to rebuild_refcount_structure() Date: Wed, 20 Apr 2022 14:41:04 +0200 Message-Id: <20220420124104.795836-9-hreitz@redhat.com> In-Reply-To: <20220420124104.795836-1-hreitz@redhat.com> References: <20220420124104.795836-1-hreitz@redhat.com> MIME-Version: 1.0 Content-type: text/plain X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Hanna Reitz , Richard Henderson , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Instead of fprint()-ing error messages in rebuild_refcount_structure() and its rebuild_refcounts_write_refblocks() helper, pass them through an Error object to qcow2_check_refcounts() (which will then print it). Suggested-by: Eric Blake Signed-off-by: Hanna Reitz Message-Id: <20220405134652.19278-4-hreitz@redhat.com> Reviewed-by: Eric Blake --- block/qcow2-refcount.c | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index c5669eaa51..ed0ecfaa89 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -2465,7 +2465,8 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs, static int rebuild_refcounts_write_refblocks( BlockDriverState *bs, void **refcount_table, int64_t *nb_clusters, int64_t first_cluster, int64_t end_cluster, - uint64_t **on_disk_reftable_ptr, uint32_t *on_disk_reftable_entries_ptr + uint64_t **on_disk_reftable_ptr, uint32_t *on_disk_reftable_entries_ptr, + Error **errp ) { BDRVQcow2State *s = bs->opaque; @@ -2516,8 +2517,8 @@ static int rebuild_refcounts_write_refblocks( nb_clusters, &first_free_cluster); if (refblock_offset < 0) { - fprintf(stderr, "ERROR allocating refblock: %s\n", - strerror(-refblock_offset)); + error_setg_errno(errp, -refblock_offset, + "ERROR allocating refblock"); return refblock_offset; } @@ -2539,6 +2540,7 @@ static int rebuild_refcounts_write_refblocks( on_disk_reftable_entries * REFTABLE_ENTRY_SIZE); if (!on_disk_reftable) { + error_setg(errp, "ERROR allocating reftable memory"); return -ENOMEM; } @@ -2562,7 +2564,7 @@ static int rebuild_refcounts_write_refblocks( ret = qcow2_pre_write_overlap_check(bs, 0, refblock_offset, s->cluster_size, false); if (ret < 0) { - fprintf(stderr, "ERROR writing refblock: %s\n", strerror(-ret)); + error_setg_errno(errp, -ret, "ERROR writing refblock"); return ret; } @@ -2578,7 +2580,7 @@ static int rebuild_refcounts_write_refblocks( ret = bdrv_pwrite(bs->file, refblock_offset, on_disk_refblock, s->cluster_size); if (ret < 0) { - fprintf(stderr, "ERROR writing refblock: %s\n", strerror(-ret)); + error_setg_errno(errp, -ret, "ERROR writing refblock"); return ret; } @@ -2601,7 +2603,8 @@ static int rebuild_refcounts_write_refblocks( static int rebuild_refcount_structure(BlockDriverState *bs, BdrvCheckResult *res, void **refcount_table, - int64_t *nb_clusters) + int64_t *nb_clusters, + Error **errp) { BDRVQcow2State *s = bs->opaque; int64_t reftable_offset = -1; @@ -2652,7 +2655,7 @@ static int rebuild_refcount_structure(BlockDriverState *bs, rebuild_refcounts_write_refblocks(bs, refcount_table, nb_clusters, 0, *nb_clusters, &on_disk_reftable, - &on_disk_reftable_entries); + &on_disk_reftable_entries, errp); if (reftable_size_changed < 0) { res->check_errors++; ret = reftable_size_changed; @@ -2676,8 +2679,8 @@ static int rebuild_refcount_structure(BlockDriverState *bs, refcount_table, nb_clusters, &first_free_cluster); if (reftable_offset < 0) { - fprintf(stderr, "ERROR allocating reftable: %s\n", - strerror(-reftable_offset)); + error_setg_errno(errp, -reftable_offset, + "ERROR allocating reftable"); res->check_errors++; ret = reftable_offset; goto fail; @@ -2695,7 +2698,7 @@ static int rebuild_refcount_structure(BlockDriverState *bs, reftable_start_cluster, reftable_end_cluster, &on_disk_reftable, - &on_disk_reftable_entries); + &on_disk_reftable_entries, errp); if (reftable_size_changed < 0) { res->check_errors++; ret = reftable_size_changed; @@ -2725,7 +2728,7 @@ static int rebuild_refcount_structure(BlockDriverState *bs, ret = qcow2_pre_write_overlap_check(bs, 0, reftable_offset, reftable_length, false); if (ret < 0) { - fprintf(stderr, "ERROR writing reftable: %s\n", strerror(-ret)); + error_setg_errno(errp, -ret, "ERROR writing reftable"); goto fail; } @@ -2733,7 +2736,7 @@ static int rebuild_refcount_structure(BlockDriverState *bs, ret = bdrv_pwrite(bs->file, reftable_offset, on_disk_reftable, reftable_length); if (ret < 0) { - fprintf(stderr, "ERROR writing reftable: %s\n", strerror(-ret)); + error_setg_errno(errp, -ret, "ERROR writing reftable"); goto fail; } @@ -2746,7 +2749,7 @@ static int rebuild_refcount_structure(BlockDriverState *bs, &reftable_offset_and_clusters, sizeof(reftable_offset_and_clusters)); if (ret < 0) { - fprintf(stderr, "ERROR setting reftable: %s\n", strerror(-ret)); + error_setg_errno(errp, -ret, "ERROR setting reftable"); goto fail; } @@ -2814,11 +2817,13 @@ int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res, if (rebuild && (fix & BDRV_FIX_ERRORS)) { BdrvCheckResult old_res = *res; int fresh_leaks = 0; + Error *local_err = NULL; fprintf(stderr, "Rebuilding refcount structure\n"); ret = rebuild_refcount_structure(bs, res, &refcount_table, - &nb_clusters); + &nb_clusters, &local_err); if (ret < 0) { + error_report_err(local_err); goto fail; }