From patchwork Wed Sep 15 17:52:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497027 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1790C433EF for ; Wed, 15 Sep 2021 17:55:00 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A9CB3610E9 for ; Wed, 15 Sep 2021 17:55:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A9CB3610E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:47560 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZ7b-00009O-KW for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 13:54:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51434) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6B-0005YL-EP for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:31 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:25874) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ68-0006M8-5S for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728407; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3nJoyOt3XCEys/6FJOHc9A0/TJWtjz849euL84GZ8UA=; b=EsHLsfvvIu3JDpiNxii0quRyegTtdUekFZ0zn9/p+mEcw5S0EQh2wAN1jQ40k6n9MAJ5cc Q2yucjPpoUPOWLvp3idQLja7RLQnla63ho4wSkXCXeur2tj/RXCaQcs9lt9DeY2LfgpNDb WteFFIW43GZMF8zJvEw39gEYeYzUZNA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-78-DQw785tvNZSKh97fLyQvCg-1; Wed, 15 Sep 2021 13:53:24 -0400 X-MC-Unique: DQw785tvNZSKh97fLyQvCg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0EDEE84A5E1; Wed, 15 Sep 2021 17:53:23 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 99F4C604CC; Wed, 15 Sep 2021 17:53:22 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 01/32] gluster: Align block-status tail Date: Wed, 15 Sep 2021 19:52:47 +0200 Message-Id: <20210915175318.853225-2-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Max Reitz gluster's block-status implementation is basically a copy of that in block/file-posix.c, there is only one thing missing, and that is aligning trailing data extents to the request alignment (as added by commit 9c3db310ff0). Note that 9c3db310ff0 mentions that "there seems to be no other block driver that sets request_alignment and [...]", but while block/gluster.c does indeed not set request_alignment, block/io.c's bdrv_refresh_limits() will still default to an alignment of 512 because block/gluster.c does not provide a byte-aligned read function. Therefore, unaligned tails can conceivably occur, and so we should apply the change from 9c3db310ff0 to gluster's block-status implementation. Reported-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Max Reitz Message-Id: <20210805143603.59503-1-mreitz@redhat.com> Reviewed-by: Eric Blake Signed-off-by: Hanna Reitz --- block/gluster.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/block/gluster.c b/block/gluster.c index e8ee14c8e9..48a04417cf 100644 --- a/block/gluster.c +++ b/block/gluster.c @@ -1477,6 +1477,8 @@ static int coroutine_fn qemu_gluster_co_block_status(BlockDriverState *bs, off_t data = 0, hole = 0; int ret = -EINVAL; + assert(QEMU_IS_ALIGNED(offset | bytes, bs->bl.request_alignment)); + if (!s->fd) { return ret; } @@ -1501,6 +1503,20 @@ static int coroutine_fn qemu_gluster_co_block_status(BlockDriverState *bs, /* On a data extent, compute bytes to the end of the extent, * possibly including a partial sector at EOF. */ *pnum = MIN(bytes, hole - offset); + + /* + * We are not allowed to return partial sectors, though, so + * round up if necessary. + */ + if (!QEMU_IS_ALIGNED(*pnum, bs->bl.request_alignment)) { + int64_t file_length = qemu_gluster_getlength(bs); + if (file_length > 0) { + /* Ignore errors, this is just a safeguard */ + assert(hole == file_length); + } + *pnum = ROUND_UP(*pnum, bs->bl.request_alignment); + } + ret = BDRV_BLOCK_DATA; } else { /* On a hole, compute bytes to the beginning of the next extent. */ From patchwork Wed Sep 15 17:52:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EF0EC433EF for ; Wed, 15 Sep 2021 17:57:57 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CBA6661130 for ; Wed, 15 Sep 2021 17:57:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org CBA6661130 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:56068 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZAR-0005vr-St for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 13:57:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51436) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6B-0005Yq-Qn for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:31 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:54954) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6A-0006Ow-2N for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728409; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KYQWoN0xAA6zjtNIq8rQopg0qXx6HZ93Cl16X2G9H/Y=; b=ipJsCyIwC49QutPAlRmsq9EtNlNO0ZBJ38dMfKuhJE7YlR8tX3b7blQO3SZ5CWGXbm1h4G H1CmE8/BWwLPI0NQhvM2MkFh97FLt2SA2L9DbXrJkeM7qYzce0aN8DFkRnoVSgoxFK/W1A I/QJuAC+3kiRw2uUqA2NZqJDfEBkpbA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-562-WAu5s3ycPyWt8-ka8ZiB9g-1; Wed, 15 Sep 2021 13:53:26 -0400 X-MC-Unique: WAu5s3ycPyWt8-ka8ZiB9g-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5381E1006AAB; Wed, 15 Sep 2021 17:53:25 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EEF8A5C1C5; Wed, 15 Sep 2021 17:53:24 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 02/32] block: Drop BDS comment regarding bdrv_append() Date: Wed, 15 Sep 2021 19:52:48 +0200 Message-Id: <20210915175318.853225-3-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" There is a comment above the BDS definition stating care must be taken to consider handling newly added fields in bdrv_append(). Actually, this comment should have said "bdrv_swap()" as of 4ddc07cac (nine years ago), and in any case, bdrv_swap() was dropped in 8e419aefa (six years ago). So no such care is necessary anymore. Signed-off-by: Hanna Reitz Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Kevin Wolf Message-Id: <20210812084148.14458-2-hreitz@redhat.com> --- include/block/block_int.h | 6 ------ 1 file changed, 6 deletions(-) diff --git a/include/block/block_int.h b/include/block/block_int.h index f1a54db0f8..12e5750fe8 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -839,12 +839,6 @@ struct BdrvChild { QLIST_ENTRY(BdrvChild) next_parent; }; -/* - * Note: the function bdrv_append() copies and swaps contents of - * BlockDriverStates, so if you add new fields to this struct, please - * inspect bdrv_append() to determine if the new fields need to be - * copied as well. - */ struct BlockDriverState { /* Protected by big QEMU lock or read-only after opening. No special * locking needed during I/O... From patchwork Wed Sep 15 17:52:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497031 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAC22C433EF for ; Wed, 15 Sep 2021 17:55:17 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4686560E8B for ; Wed, 15 Sep 2021 17:55:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4686560E8B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:48834 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZ7s-00012F-FB for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 13:55:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51520) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6H-0005pq-Ee for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:37 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:24420) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6D-0006SE-Gk for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728412; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=M8migMfuF3Nd3k/ibP+nD7eZPRTtft4ZfoWzq+korcA=; b=Apnc8Ml79LdnxOGXxxWAtcHg30OkBhhMp+2i7G1umKjR2nozDrM4/lLxlfD86tRIp/ii0k YTnKe/1/t29emMO0N8IPAICtGqVQPA93cyUD70NEOfv5Q+vSKSylBQK6HYnC9V+FaB7CDG krwXm7I22hOHmmZm87QzkTfx9reMA5g= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-524-Mq8KUxu7PO2T4nRul6yrPQ-1; Wed, 15 Sep 2021 13:53:30 -0400 X-MC-Unique: Mq8KUxu7PO2T4nRul6yrPQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B30FE1006AA4; Wed, 15 Sep 2021 17:53:27 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2AAA61972E; Wed, 15 Sep 2021 17:53:27 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 03/32] block: block-status cache for data regions Date: Wed, 15 Sep 2021 19:52:49 +0200 Message-Id: <20210915175318.853225-4-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" As we have attempted before (https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06451.html, "file-posix: Cache lseek result for data regions"; https://lists.nongnu.org/archive/html/qemu-block/2021-02/msg00934.html, "file-posix: Cache next hole"), this patch seeks to reduce the number of SEEK_DATA/HOLE operations the file-posix driver has to perform. The main difference is that this time it is implemented as part of the general block layer code. The problem we face is that on some filesystems or in some circumstances, SEEK_DATA/HOLE is unreasonably slow. Given the implementation is outside of qemu, there is little we can do about its performance. We have already introduced the want_zero parameter to bdrv_co_block_status() to reduce the number of SEEK_DATA/HOLE calls unless we really want zero information; but sometimes we do want that information, because for files that consist largely of zero areas, special-casing those areas can give large performance boosts. So the real problem is with files that consist largely of data, so that inquiring the block status does not gain us much performance, but where such an inquiry itself takes a lot of time. To address this, we want to cache data regions. Most of the time, when bad performance is reported, it is in places where the image is iterated over from start to end (qemu-img convert or the mirror job), so a simple yet effective solution is to cache only the current data region. (Note that only caching data regions but not zero regions means that returning false information from the cache is not catastrophic: Treating zeroes as data is fine. While we try to invalidate the cache on zero writes and discards, such incongruences may still occur when there are other processes writing to the image.) We only use the cache for nodes without children (i.e. protocol nodes), because that is where the problem is: Drivers that rely on block-status implementations outside of qemu (e.g. SEEK_DATA/HOLE). Resolves: https://gitlab.com/qemu-project/qemu/-/issues/307 Signed-off-by: Hanna Reitz Message-Id: <20210812084148.14458-3-hreitz@redhat.com> Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy [hreitz: Added `local_file == bs` assertion, as suggested by Vladimir] Signed-off-by: Hanna Reitz --- include/block/block_int.h | 50 ++++++++++++++++++++++++ block.c | 80 +++++++++++++++++++++++++++++++++++++++ block/io.c | 68 +++++++++++++++++++++++++++++++-- 3 files changed, 195 insertions(+), 3 deletions(-) diff --git a/include/block/block_int.h b/include/block/block_int.h index 12e5750fe8..437d746733 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -34,6 +34,7 @@ #include "qemu/hbitmap.h" #include "block/snapshot.h" #include "qemu/throttle.h" +#include "qemu/rcu.h" #define BLOCK_FLAG_LAZY_REFCOUNTS 8 @@ -839,6 +840,24 @@ struct BdrvChild { QLIST_ENTRY(BdrvChild) next_parent; }; +/* + * Allows bdrv_co_block_status() to cache one data region for a + * protocol node. + * + * @valid: Whether the cache is valid (should be accessed with atomic + * functions so this can be reset by RCU readers) + * @data_start: Offset where we know (or strongly assume) is data + * @data_end: Offset where the data region ends (which is not necessarily + * the start of a zeroed region) + */ +typedef struct BdrvBlockStatusCache { + struct rcu_head rcu; + + bool valid; + int64_t data_start; + int64_t data_end; +} BdrvBlockStatusCache; + struct BlockDriverState { /* Protected by big QEMU lock or read-only after opening. No special * locking needed during I/O... @@ -1004,6 +1023,11 @@ struct BlockDriverState { /* BdrvChild links to this node may never be frozen */ bool never_freeze; + + /* Lock for block-status cache RCU writers */ + CoMutex bsc_modify_lock; + /* Always non-NULL, but must only be dereferenced under an RCU read guard */ + BdrvBlockStatusCache *block_status_cache; }; struct BlockBackendRootState { @@ -1429,4 +1453,30 @@ static inline BlockDriverState *bdrv_primary_bs(BlockDriverState *bs) */ void bdrv_drain_all_end_quiesce(BlockDriverState *bs); +/** + * Check whether the given offset is in the cached block-status data + * region. + * + * If it is, and @pnum is not NULL, *pnum is set to + * `bsc.data_end - offset`, i.e. how many bytes, starting from + * @offset, are data (according to the cache). + * Otherwise, *pnum is not touched. + */ +bool bdrv_bsc_is_data(BlockDriverState *bs, int64_t offset, int64_t *pnum); + +/** + * If [offset, offset + bytes) overlaps with the currently cached + * block-status region, invalidate the cache. + * + * (To be used by I/O paths that cause data regions to be zero or + * holes.) + */ +void bdrv_bsc_invalidate_range(BlockDriverState *bs, + int64_t offset, int64_t bytes); + +/** + * Mark the range [offset, offset + bytes) as a data region. + */ +void bdrv_bsc_fill(BlockDriverState *bs, int64_t offset, int64_t bytes); + #endif /* BLOCK_INT_H */ diff --git a/block.c b/block.c index b2b66263f9..00982edf45 100644 --- a/block.c +++ b/block.c @@ -49,6 +49,8 @@ #include "qemu/timer.h" #include "qemu/cutils.h" #include "qemu/id.h" +#include "qemu/range.h" +#include "qemu/rcu.h" #include "block/coroutines.h" #ifdef CONFIG_BSD @@ -401,6 +403,9 @@ BlockDriverState *bdrv_new(void) qemu_co_queue_init(&bs->flush_queue); + qemu_co_mutex_init(&bs->bsc_modify_lock); + bs->block_status_cache = g_new0(BdrvBlockStatusCache, 1); + for (i = 0; i < bdrv_drain_all_count; i++) { bdrv_drained_begin(bs); } @@ -4694,6 +4699,8 @@ static void bdrv_close(BlockDriverState *bs) bs->explicit_options = NULL; qobject_unref(bs->full_open_options); bs->full_open_options = NULL; + g_free(bs->block_status_cache); + bs->block_status_cache = NULL; bdrv_release_named_dirty_bitmaps(bs); assert(QLIST_EMPTY(&bs->dirty_bitmaps)); @@ -7684,3 +7691,76 @@ BlockDriverState *bdrv_backing_chain_next(BlockDriverState *bs) { return bdrv_skip_filters(bdrv_cow_bs(bdrv_skip_filters(bs))); } + +/** + * Check whether [offset, offset + bytes) overlaps with the cached + * block-status data region. + * + * If so, and @pnum is not NULL, set *pnum to `bsc.data_end - offset`, + * which is what bdrv_bsc_is_data()'s interface needs. + * Otherwise, *pnum is not touched. + */ +static bool bdrv_bsc_range_overlaps_locked(BlockDriverState *bs, + int64_t offset, int64_t bytes, + int64_t *pnum) +{ + BdrvBlockStatusCache *bsc = qatomic_rcu_read(&bs->block_status_cache); + bool overlaps; + + overlaps = + qatomic_read(&bsc->valid) && + ranges_overlap(offset, bytes, bsc->data_start, + bsc->data_end - bsc->data_start); + + if (overlaps && pnum) { + *pnum = bsc->data_end - offset; + } + + return overlaps; +} + +/** + * See block_int.h for this function's documentation. + */ +bool bdrv_bsc_is_data(BlockDriverState *bs, int64_t offset, int64_t *pnum) +{ + RCU_READ_LOCK_GUARD(); + + return bdrv_bsc_range_overlaps_locked(bs, offset, 1, pnum); +} + +/** + * See block_int.h for this function's documentation. + */ +void bdrv_bsc_invalidate_range(BlockDriverState *bs, + int64_t offset, int64_t bytes) +{ + RCU_READ_LOCK_GUARD(); + + if (bdrv_bsc_range_overlaps_locked(bs, offset, bytes, NULL)) { + qatomic_set(&bs->block_status_cache->valid, false); + } +} + +/** + * See block_int.h for this function's documentation. + */ +void bdrv_bsc_fill(BlockDriverState *bs, int64_t offset, int64_t bytes) +{ + BdrvBlockStatusCache *new_bsc = g_new(BdrvBlockStatusCache, 1); + BdrvBlockStatusCache *old_bsc; + + *new_bsc = (BdrvBlockStatusCache) { + .valid = true, + .data_start = offset, + .data_end = offset + bytes, + }; + + QEMU_LOCK_GUARD(&bs->bsc_modify_lock); + + old_bsc = qatomic_rcu_read(&bs->block_status_cache); + qatomic_rcu_set(&bs->block_status_cache, new_bsc); + if (old_bsc) { + g_free_rcu(old_bsc, rcu); + } +} diff --git a/block/io.c b/block/io.c index a19942718b..99ee182ca4 100644 --- a/block/io.c +++ b/block/io.c @@ -1883,6 +1883,9 @@ static int coroutine_fn bdrv_co_do_pwrite_zeroes(BlockDriverState *bs, return -ENOTSUP; } + /* Invalidate the cached block-status data range if this write overlaps */ + bdrv_bsc_invalidate_range(bs, offset, bytes); + assert(alignment % bs->bl.request_alignment == 0); head = offset % alignment; tail = (offset + bytes) % alignment; @@ -2447,9 +2450,65 @@ static int coroutine_fn bdrv_co_block_status(BlockDriverState *bs, aligned_bytes = ROUND_UP(offset + bytes, align) - aligned_offset; if (bs->drv->bdrv_co_block_status) { - ret = bs->drv->bdrv_co_block_status(bs, want_zero, aligned_offset, - aligned_bytes, pnum, &local_map, - &local_file); + /* + * Use the block-status cache only for protocol nodes: Format + * drivers are generally quick to inquire the status, but protocol + * drivers often need to get information from outside of qemu, so + * we do not have control over the actual implementation. There + * have been cases where inquiring the status took an unreasonably + * long time, and we can do nothing in qemu to fix it. + * This is especially problematic for images with large data areas, + * because finding the few holes in them and giving them special + * treatment does not gain much performance. Therefore, we try to + * cache the last-identified data region. + * + * Second, limiting ourselves to protocol nodes allows us to assume + * the block status for data regions to be DATA | OFFSET_VALID, and + * that the host offset is the same as the guest offset. + * + * Note that it is possible that external writers zero parts of + * the cached regions without the cache being invalidated, and so + * we may report zeroes as data. This is not catastrophic, + * however, because reporting zeroes as data is fine. + */ + if (QLIST_EMPTY(&bs->children) && + bdrv_bsc_is_data(bs, aligned_offset, pnum)) + { + ret = BDRV_BLOCK_DATA | BDRV_BLOCK_OFFSET_VALID; + local_file = bs; + local_map = aligned_offset; + } else { + ret = bs->drv->bdrv_co_block_status(bs, want_zero, aligned_offset, + aligned_bytes, pnum, &local_map, + &local_file); + + /* + * Note that checking QLIST_EMPTY(&bs->children) is also done when + * the cache is queried above. Technically, we do not need to check + * it here; the worst that can happen is that we fill the cache for + * non-protocol nodes, and then it is never used. However, filling + * the cache requires an RCU update, so double check here to avoid + * such an update if possible. + */ + if (ret == (BDRV_BLOCK_DATA | BDRV_BLOCK_OFFSET_VALID) && + QLIST_EMPTY(&bs->children)) + { + /* + * When a protocol driver reports BLOCK_OFFSET_VALID, the + * returned local_map value must be the same as the offset we + * have passed (aligned_offset), and local_bs must be the node + * itself. + * Assert this, because we follow this rule when reading from + * the cache (see the `local_file = bs` and + * `local_map = aligned_offset` assignments above), and the + * result the cache delivers must be the same as the driver + * would deliver. + */ + assert(local_file == bs); + assert(local_map == aligned_offset); + bdrv_bsc_fill(bs, aligned_offset, *pnum); + } + } } else { /* Default code for filters */ @@ -3002,6 +3061,9 @@ int coroutine_fn bdrv_co_pdiscard(BdrvChild *child, int64_t offset, return 0; } + /* Invalidate the cached block-status data range if this discard overlaps */ + bdrv_bsc_invalidate_range(bs, offset, bytes); + /* Discard is advisory, but some devices track and coalesce * unaligned requests, so we must pass everything down rather than * round here. Still, most devices will just silently ignore From patchwork Wed Sep 15 17:52:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAE4CC433F5 for ; Wed, 15 Sep 2021 18:00:10 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7C452610E9 for ; Wed, 15 Sep 2021 18:00:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7C452610E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:36390 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZCb-0003Fo-Jp for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:00:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51516) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6H-0005pX-Cl for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:37 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:43620) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6D-0006Rw-Kl for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728412; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xEmsIsVZJ1b5hOCrP/lM8Zro4QPWjantR9Q31P7XTqs=; b=TgLHxRh1Hxa6SOo3WJBjAKV2Dsqn8zxek8EvPrxCRydGd1wkseGIJNC6Ijwmusk2K8AzBl mAH15ZUOq6uMytWku/oIFnH5y2gR7X7FEXlunUbeHdRcpMf71B8E54OAmxJc+TTkb37hZU l+rgB0ijQ80MG6cgkS3u83BwnB5tla8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-227-Di7TYAicO26HA0Kz2FFSLQ-1; Wed, 15 Sep 2021 13:53:30 -0400 X-MC-Unique: Di7TYAicO26HA0Kz2FFSLQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C96F684A5E0; Wed, 15 Sep 2021 17:53:29 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7202D5D9D3; Wed, 15 Sep 2021 17:53:29 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 04/32] block: Clarify that @bytes is no limit on *pnum Date: Wed, 15 Sep 2021 19:52:50 +0200 Message-Id: <20210915175318.853225-5-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" .bdrv_co_block_status() implementations are free to return a *pnum that exceeds @bytes, because bdrv_co_block_status() in block/io.c will clamp *pnum as necessary. On the other hand, if drivers' implementations return values for *pnum that are as large as possible, our recently introduced block-status cache will become more effective. So, make a note in block_int.h that @bytes is no upper limit for *pnum. Suggested-by: Eric Blake Signed-off-by: Hanna Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <20210812084148.14458-4-hreitz@redhat.com> Reviewed-by: Eric Blake --- include/block/block_int.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/include/block/block_int.h b/include/block/block_int.h index 437d746733..5451f89b8d 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -348,6 +348,15 @@ struct BlockDriver { * clamped to bdrv_getlength() and aligned to request_alignment, * as well as non-NULL pnum, map, and file; in turn, the driver * must return an error or set pnum to an aligned non-zero value. + * + * Note that @bytes is just a hint on how big of a region the + * caller wants to inspect. It is not a limit on *pnum. + * Implementations are free to return larger values of *pnum if + * doing so does not incur a performance penalty. + * + * block/io.c's bdrv_co_block_status() will utilize an unclamped + * *pnum value for the block-status cache on protocol nodes, prior + * to clamping *pnum for return to its caller. */ int coroutine_fn (*bdrv_co_block_status)(BlockDriverState *bs, bool want_zero, int64_t offset, int64_t bytes, int64_t *pnum, From patchwork Wed Sep 15 17:52:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497053 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45361C433F5 for ; Wed, 15 Sep 2021 17:59:38 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0E00B61130 for ; Wed, 15 Sep 2021 17:59:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 0E00B61130 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:34070 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZC5-0001iX-1o for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 13:59:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51542) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6I-0005rh-0e for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:38 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:26107) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6F-0006Tq-0m for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728414; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Qo+uPRCXtU9RGueubkJAbnnVqop39YkPM/NlvMn3tI8=; b=ZQqN8o0QQASj0KA3tEqCYtPIg6WudlREmI9Uquy7bvzdL8xdY8789pHz5pEbZ1fnlaRqr3 DYdf4kWjgukzgakqFat9v7jKA6nhRwctiyuopko3RZ6PYMr8UsAT+i3hAuvn8YJx4S0vrN aJs3A5n6no4MnRA5JDypbqiIpmGIFWI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-532-v_OkWvJFNl-kTynpiVSHkw-1; Wed, 15 Sep 2021 13:53:32 -0400 X-MC-Unique: v_OkWvJFNl-kTynpiVSHkw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id DCCF61006AA2; Wed, 15 Sep 2021 17:53:31 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7D7ED10016FB; Wed, 15 Sep 2021 17:53:31 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 05/32] block/file-posix: Do not force-cap *pnum Date: Wed, 15 Sep 2021 19:52:51 +0200 Message-Id: <20210915175318.853225-6-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" bdrv_co_block_status() does it for us, we do not need to do it here. The advantage of not capping *pnum is that bdrv_co_block_status() can cache larger data regions than requested by its caller. Signed-off-by: Hanna Reitz Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Kevin Wolf Message-Id: <20210812084148.14458-5-hreitz@redhat.com> --- block/file-posix.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/block/file-posix.c b/block/file-posix.c index cb9bffe047..9f35e5631a 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -2744,7 +2744,8 @@ static int find_allocation(BlockDriverState *bs, off_t start, * the specified offset) that are known to be in the same * allocated/unallocated state. * - * 'bytes' is the max value 'pnum' should be set to. + * 'bytes' is a soft cap for 'pnum'. If the information is free, 'pnum' may + * well exceed it. */ static int coroutine_fn raw_co_block_status(BlockDriverState *bs, bool want_zero, @@ -2782,7 +2783,7 @@ static int coroutine_fn raw_co_block_status(BlockDriverState *bs, } else if (data == offset) { /* On a data extent, compute bytes to the end of the extent, * possibly including a partial sector at EOF. */ - *pnum = MIN(bytes, hole - offset); + *pnum = hole - offset; /* * We are not allowed to return partial sectors, though, so @@ -2801,7 +2802,7 @@ static int coroutine_fn raw_co_block_status(BlockDriverState *bs, } else { /* On a hole, compute bytes to the beginning of the next extent. */ assert(hole == offset); - *pnum = MIN(bytes, data - offset); + *pnum = data - offset; ret = BDRV_BLOCK_ZERO; } *map = offset; From patchwork Wed Sep 15 17:52:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E25A3C433EF for ; Wed, 15 Sep 2021 17:58:16 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AF29B61130 for ; Wed, 15 Sep 2021 17:58:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AF29B61130 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:57420 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZAl-0006t4-R3 for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 13:58:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51564) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6J-0005wP-40 for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:39 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:41769) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6H-0006Wf-E5 for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728416; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c3OEWBgbx1FlL+tgX3YVs4AuUhVFeFNXIkziX9jmhqY=; b=LIjPsNcJm6nY1n8448FB3r64481qYarInl2hiVj6lpf+hvfUu78+tZBVVj5e0pvT3IJO4e 4etlbPugnlXW3ZRbR1qW9B5aWF0CgT3boy5n3siZ/S8NHb1zju1vYcirZ5fwYizH8jmRhk pSh9dr07D7GpbPlem2yfThpoPEJGHME= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-45-Y_8_xzGdPYyObgPJ8CJqdQ-1; Wed, 15 Sep 2021 13:53:35 -0400 X-MC-Unique: Y_8_xzGdPYyObgPJ8CJqdQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E7F0384A5E6; Wed, 15 Sep 2021 17:53:33 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8EAD460C2B; Wed, 15 Sep 2021 17:53:33 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 06/32] block/gluster: Do not force-cap *pnum Date: Wed, 15 Sep 2021 19:52:52 +0200 Message-Id: <20210915175318.853225-7-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" bdrv_co_block_status() does it for us, we do not need to do it here. The advantage of not capping *pnum is that bdrv_co_block_status() can cache larger data regions than requested by its caller. Signed-off-by: Hanna Reitz Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Kevin Wolf Message-Id: <20210812084148.14458-6-hreitz@redhat.com> --- block/gluster.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/block/gluster.c b/block/gluster.c index 48a04417cf..d51938e447 100644 --- a/block/gluster.c +++ b/block/gluster.c @@ -1461,7 +1461,8 @@ exit: * the specified offset) that are known to be in the same * allocated/unallocated state. * - * 'bytes' is the max value 'pnum' should be set to. + * 'bytes' is a soft cap for 'pnum'. If the information is free, 'pnum' may + * well exceed it. * * (Based on raw_co_block_status() from file-posix.c.) */ @@ -1502,7 +1503,7 @@ static int coroutine_fn qemu_gluster_co_block_status(BlockDriverState *bs, } else if (data == offset) { /* On a data extent, compute bytes to the end of the extent, * possibly including a partial sector at EOF. */ - *pnum = MIN(bytes, hole - offset); + *pnum = hole - offset; /* * We are not allowed to return partial sectors, though, so @@ -1521,7 +1522,7 @@ static int coroutine_fn qemu_gluster_co_block_status(BlockDriverState *bs, } else { /* On a hole, compute bytes to the beginning of the next extent. */ assert(hole == offset); - *pnum = MIN(bytes, data - offset); + *pnum = data - offset; ret = BDRV_BLOCK_ZERO; } From patchwork Wed Sep 15 17:52:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F70FC433EF for ; Wed, 15 Sep 2021 18:03:54 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EF11C611C4 for ; Wed, 15 Sep 2021 18:03:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org EF11C611C4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:44932 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZGD-0000ce-5n for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:03:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51616) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6K-00062s-Ve for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:41 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:56052) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6J-0006Y6-1m for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728418; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aMUtWYK/2KwKWgY8RiKkut/IQ2b4UDkPmQRpHRS/4Sk=; b=Djhw5vlyuNNMqNgxfSAO8VwvCaJNhIpx/UKAxYsjGzxh9sVmUcPcfABYbE7OyERC15mZxp Os5r9lpJjB9nMuraEeOjozIWilyBX5QAbfxIjBh8kim5FPVXLf+r/k6JZm1sljjVRuUpqH Rh2/h6eoPc/9TmO9XxDwnENfmiCSvmk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-179-iQ6rC0evNAuMEJ33HHxfPw-1; Wed, 15 Sep 2021 13:53:37 -0400 X-MC-Unique: iQ6rC0evNAuMEJ33HHxfPw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0416F84A5E1; Wed, 15 Sep 2021 17:53:36 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9E0CB19736; Wed, 15 Sep 2021 17:53:35 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 07/32] block/iscsi: Do not force-cap *pnum Date: Wed, 15 Sep 2021 19:52:53 +0200 Message-Id: <20210915175318.853225-8-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" bdrv_co_block_status() does it for us, we do not need to do it here. The advantage of not capping *pnum is that bdrv_co_block_status() can cache larger data regions than requested by its caller. Signed-off-by: Hanna Reitz Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Kevin Wolf Message-Id: <20210812084148.14458-7-hreitz@redhat.com> --- block/iscsi.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/block/iscsi.c b/block/iscsi.c index 4d2a416ce7..852384086b 100644 --- a/block/iscsi.c +++ b/block/iscsi.c @@ -781,9 +781,6 @@ retry: iscsi_allocmap_set_allocated(iscsilun, offset, *pnum); } - if (*pnum > bytes) { - *pnum = bytes; - } out_unlock: qemu_mutex_unlock(&iscsilun->mutex); g_free(iTask.err_str); From patchwork Wed Sep 15 17:52:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDB18C433F5 for ; Wed, 15 Sep 2021 18:01:07 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 782B8610E9 for ; Wed, 15 Sep 2021 18:01:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 782B8610E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:37834 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZDW-0004De-GD for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:01:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51698) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6Q-0006HY-Oo for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:47 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:23854) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6N-0006c8-4n for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728422; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yWcmsjCWbbXTbiBRs6U1UqCBtizGkxWfW7vTjH3r/M8=; b=bCLkmIXp5+tuSnGZBIOR/QJShwDpZNw6siGIl7AjgAzCE1MhcFuTZCQaLS+O4JZ4Kn0mM9 rztTRjc0vfwXgbWgS02Z1H6VFBkj3UE5d0MgrwonO3vszPWhUSZa4fHDmsUXxKO3iHxyt4 77odZhgbTEAcOWpKoCdDQotaPklncRw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-427-SF_k5w07Od6uxSCLMhIt0w-1; Wed, 15 Sep 2021 13:53:39 -0400 X-MC-Unique: SF_k5w07Od6uxSCLMhIt0w-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 184171966325; Wed, 15 Sep 2021 17:53:38 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B88F660583; Wed, 15 Sep 2021 17:53:37 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 08/32] iotests: Fix unspecified-encoding pylint warnings Date: Wed, 15 Sep 2021 19:52:54 +0200 Message-Id: <20210915175318.853225-9-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" As of recently, pylint complains when `open()` calls are missing an `encoding=` specified. Everything we have should be UTF-8 (and in fact, everything should be UTF-8, period (exceptions apply)), so use that. Signed-off-by: Hanna Reitz Message-Id: <20210824153540.177128-2-hreitz@redhat.com> Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: John Snow --- tests/qemu-iotests/297 | 2 +- tests/qemu-iotests/iotests.py | 8 +++++--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297 index 345b617b34..1ee15dd866 100755 --- a/tests/qemu-iotests/297 +++ b/tests/qemu-iotests/297 @@ -46,7 +46,7 @@ def is_python_file(filename): if filename.endswith('.py'): return True - with open(filename) as f: + with open(filename, encoding='utf-8') as f: try: first_line = f.readline() return re.match('^#!.*python', first_line) is not None diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 11276f380a..d8c64d4c11 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -610,7 +610,7 @@ def _post_shutdown(self) -> None: return valgrind_filename = f"{test_dir}/{self._popen.pid}.valgrind" if self.exitcode() == 99: - with open(valgrind_filename) as f: + with open(valgrind_filename, encoding='utf-8') as f: print(f.read()) else: os.remove(valgrind_filename) @@ -1121,7 +1121,8 @@ def notrun(reason): # Each test in qemu-iotests has a number ("seq") seq = os.path.basename(sys.argv[0]) - with open('%s/%s.notrun' % (output_dir, seq), 'w') as outfile: + with open('%s/%s.notrun' % (output_dir, seq), 'w', encoding='utf-8') \ + as outfile: outfile.write(reason + '\n') logger.warning("%s not run: %s", seq, reason) sys.exit(0) @@ -1135,7 +1136,8 @@ def case_notrun(reason): # Each test in qemu-iotests has a number ("seq") seq = os.path.basename(sys.argv[0]) - with open('%s/%s.casenotrun' % (output_dir, seq), 'a') as outfile: + with open('%s/%s.casenotrun' % (output_dir, seq), 'a', encoding='utf-8') \ + as outfile: outfile.write(' [case not run] ' + reason + '\n') def _verify_image_format(supported_fmts: Sequence[str] = (), From patchwork Wed Sep 15 17:52:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497029 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8299FC433F5 for ; Wed, 15 Sep 2021 17:55:14 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 32A9560E8B for ; Wed, 15 Sep 2021 17:55:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 32A9560E8B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:48756 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZ7p-0000z5-Br for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 13:55:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51728) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6S-0006K8-E3 for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:50 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:60449) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6N-0006c5-52 for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728422; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=joD4WMD27Tc41sGzuYQQdUF+MK11b76h19wdcVkn+wY=; b=TypZiSsM4Q0Qbj3KU4pTMyKHE3pMvdAxDEE32z5GXqgOQ2rXFwSA1TazjCZvnranGEW3eP 2edjHPNeYFyINGiv8p6MYNZqbE/xZXe9GEHiAF9RUlHP/kaRTF2YS3M6zR913H7WYCyiLX l8/q62dBMBO9ChukeUGHWl5/po9qoa8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-512-bAUZDa1OM1SNJXGaTa7SfQ-1; Wed, 15 Sep 2021 13:53:41 -0400 X-MC-Unique: bAUZDa1OM1SNJXGaTa7SfQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2DF461084683; Wed, 15 Sep 2021 17:53:40 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C918E19736; Wed, 15 Sep 2021 17:53:39 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 09/32] iotests: Fix use-{list,dict}-literal warnings Date: Wed, 15 Sep 2021 19:52:55 +0200 Message-Id: <20210915175318.853225-10-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" pylint proposes using `[]` instead of `list()` and `{}` instead of `dict()`, because it is faster. That seems simple enough, so heed its advice. Signed-off-by: Hanna Reitz Message-Id: <20210824153540.177128-3-hreitz@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index d8c64d4c11..ce06cf5630 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -703,7 +703,7 @@ def hmp_qemu_io(self, drive: str, cmd: str, def flatten_qmp_object(self, obj, output=None, basestr=''): if output is None: - output = dict() + output = {} if isinstance(obj, list): for i, item in enumerate(obj): self.flatten_qmp_object(item, output, basestr + str(i) + '.') @@ -716,7 +716,7 @@ def flatten_qmp_object(self, obj, output=None, basestr=''): def qmp_to_opts(self, obj): obj = self.flatten_qmp_object(obj) - output_list = list() + output_list = [] for key in obj: output_list += [key + '=' + obj[key]] return ','.join(output_list) From patchwork Wed Sep 15 17:52:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C81A7C433EF for ; Wed, 15 Sep 2021 18:04:35 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 45355611C4 for ; Wed, 15 Sep 2021 18:04:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 45355611C4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:46704 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZGs-0001r8-D1 for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:04:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51748) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6U-0006Kq-9I for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:50 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:25634) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6Q-0006fG-E5 for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728424; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xabGNgbrdboR/jzlUlFWKXCDyhQKRxtHQ9tiGKd09RI=; b=ZoFMHJjZ6kmTOdqT7JIXA8oYLs5zTVa9jhBr66mhYSXIVLwNMnG3m+dPLJn9vH+sb+9UcK 1t6LzIwgbDpy9krGjXS4NTbIHUAhOaeVI+npiqQ9demW/fvUaLrfGWB+HFUvj45E2c29d9 ceLYWvWwYjm0CWPYphT0jhl87ebHLRM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-315-Vf_Zy9HlNtaPB74veHFKYg-1; Wed, 15 Sep 2021 13:53:43 -0400 X-MC-Unique: Vf_Zy9HlNtaPB74veHFKYg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3C8D0802947; Wed, 15 Sep 2021 17:53:42 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D638C196F1; Wed, 15 Sep 2021 17:53:41 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 10/32] iotests/297: Drop 169 and 199 from the skip list Date: Wed, 15 Sep 2021 19:52:56 +0200 Message-Id: <20210915175318.853225-11-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" 169 and 199 have been renamed and moved to tests/ (commit a44be0334be: "iotests: rename and move 169 and 199 tests"), so we can drop them from the skip list. Signed-off-by: Hanna Reitz Reviewed-by: Willian Rampazzo Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Kevin Wolf Message-Id: <20210902094017.32902-2-hreitz@redhat.com> --- tests/qemu-iotests/297 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297 index 1ee15dd866..a0c0cf9e5d 100755 --- a/tests/qemu-iotests/297 +++ b/tests/qemu-iotests/297 @@ -29,7 +29,7 @@ import iotests SKIP_FILES = ( '030', '040', '041', '044', '045', '055', '056', '057', '065', '093', '096', '118', '124', '132', '136', '139', '147', '148', '149', - '151', '152', '155', '163', '165', '169', '194', '196', '199', '202', + '151', '152', '155', '163', '165', '194', '196', '202', '203', '205', '206', '207', '208', '210', '211', '212', '213', '216', '218', '219', '224', '228', '234', '235', '236', '237', '238', '240', '242', '245', '246', '248', '255', '256', '257', '258', '260', From patchwork Wed Sep 15 17:52:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AF44C433FE for ; Wed, 15 Sep 2021 17:58:46 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D429261209 for ; Wed, 15 Sep 2021 17:58:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D429261209 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:58696 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZBF-0007iv-1y for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 13:58:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51802) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6W-0006Lg-4T for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:53 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:52467) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6R-0006h8-Kw for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728426; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=puXKn0rMLPVcQpzKeggCaZfrT6XDpFSqRFBPAxDi+6w=; b=jGpOwMUA5prIqD58dQhk6wv4Kv0arP1GZn4hOT4ITSibGsABrCaDGtshY+JKxMVn+5idVk O3X6a80AjpRliLix9p6RGzewD3LtwCfj8FePCDIQyLeX1n1LPtvpb5xKxSj430a8kqp8X3 t1gYue6tOhCyoV9SknoiqRUHfCygXtg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-521-5__-etKlNyeVxos94a2Zsg-1; Wed, 15 Sep 2021 13:53:45 -0400 X-MC-Unique: 5__-etKlNyeVxos94a2Zsg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 496A21966320; Wed, 15 Sep 2021 17:53:44 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E3F6F5D9D3; Wed, 15 Sep 2021 17:53:43 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 11/32] migrate-bitmaps-postcopy-test: Fix pylint warnings Date: Wed, 15 Sep 2021 19:52:57 +0200 Message-Id: <20210915175318.853225-12-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" pylint complains that discards1_sha256 and all_discards_sha256 are first set in non-__init__ methods. These variables are not really class-variables anyway, so let them instead be returned by start_postcopy(), thus silencing pylint. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Hanna Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <20210902094017.32902-3-hreitz@redhat.com> --- .../tests/migrate-bitmaps-postcopy-test | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test b/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test index 584062b412..00ebb5c251 100755 --- a/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test +++ b/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test @@ -132,10 +132,10 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase): result = self.vm_a.qmp('x-debug-block-dirty-bitmap-sha256', node='drive0', name='bitmap0') - self.discards1_sha256 = result['return']['sha256'] + discards1_sha256 = result['return']['sha256'] # Check, that updating the bitmap by discards works - assert self.discards1_sha256 != empty_sha256 + assert discards1_sha256 != empty_sha256 # We want to calculate resulting sha256. Do it in bitmap0, so, disable # other bitmaps @@ -148,7 +148,7 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase): result = self.vm_a.qmp('x-debug-block-dirty-bitmap-sha256', node='drive0', name='bitmap0') - self.all_discards_sha256 = result['return']['sha256'] + all_discards_sha256 = result['return']['sha256'] # Now, enable some bitmaps, to be updated during migration for i in range(2, nb_bitmaps, 2): @@ -173,10 +173,11 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase): event_resume = self.vm_b.event_wait('RESUME') self.vm_b_events.append(event_resume) - return event_resume + return (event_resume, discards1_sha256, all_discards_sha256) def test_postcopy_success(self): - event_resume = self.start_postcopy() + event_resume, discards1_sha256, all_discards_sha256 = \ + self.start_postcopy() # enabled bitmaps should be updated apply_discards(self.vm_b, discards2) @@ -217,7 +218,7 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase): for i in range(0, nb_bitmaps, 5): result = self.vm_b.qmp('x-debug-block-dirty-bitmap-sha256', node='drive0', name='bitmap{}'.format(i)) - sha = self.discards1_sha256 if i % 2 else self.all_discards_sha256 + sha = discards1_sha256 if i % 2 else all_discards_sha256 self.assert_qmp(result, 'return/sha256', sha) def test_early_shutdown_destination(self): From patchwork Wed Sep 15 17:52:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497059 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7933C433F5 for ; Wed, 15 Sep 2021 18:01:35 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 69E2160F3A for ; Wed, 15 Sep 2021 18:01:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 69E2160F3A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:39096 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZDx-0005Ay-FB for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:01:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51814) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6W-0006Li-Dy for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:53 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:47609) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6U-0006iv-1M for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728428; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dxr3gZZ7wURu4pguvCBx0LSWFl8P90YLyEzkpe9p68c=; b=YB/j1cefu6YjHrDohxs9mPEjKKKKCbIx98c2fsePbbPdKxnVv7Ief2C9/UU0PiJ3oOnAub NfmVTj5/BrtCX1oCTybCH99nh82c2E9ntypzIeCsOla/JQgtoiQXeTVFN+LVqNWGAk1En2 QoxlMyr6ystcXe10wyaZu2NgVDdIYAU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-96-F389TsNhMdWcOa8-dFpCoQ-1; Wed, 15 Sep 2021 13:53:47 -0400 X-MC-Unique: F389TsNhMdWcOa8-dFpCoQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 60DDD1966320; Wed, 15 Sep 2021 17:53:46 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 087A71972E; Wed, 15 Sep 2021 17:53:45 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 12/32] migrate-bitmaps-test: Fix pylint warnings Date: Wed, 15 Sep 2021 19:52:58 +0200 Message-Id: <20210915175318.853225-13-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" There are a couple of things pylint takes issue with: - The "time" import is unused - The import order (iotests should come last) - get_bitmap_hash() doesn't use @self and so should be a function - Semicolons at the end of some lines - Parentheses after "if" - Some lines are too long (80 characters instead of 79) - inject_test_case()'s @name parameter shadows a top-level @name variable - "lambda self: mc(self)" were equivalent to just "mc", but in inject_test_case(), it is not equivalent, so add a comment and disable the warning locally - Always put two empty lines after a function - f'exec: cat > /dev/null' does not need to be an f-string Fix them. Signed-off-by: Hanna Reitz Message-Id: <20210902094017.32902-4-hreitz@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/migrate-bitmaps-test | 43 +++++++++++-------- 1 file changed, 25 insertions(+), 18 deletions(-) diff --git a/tests/qemu-iotests/tests/migrate-bitmaps-test b/tests/qemu-iotests/tests/migrate-bitmaps-test index a5c7bc83e0..dc431c35b3 100755 --- a/tests/qemu-iotests/tests/migrate-bitmaps-test +++ b/tests/qemu-iotests/tests/migrate-bitmaps-test @@ -20,11 +20,10 @@ # import os -import iotests -import time import itertools import operator import re +import iotests from iotests import qemu_img, qemu_img_create, Timeout @@ -37,6 +36,12 @@ mig_cmd = 'exec: cat > ' + mig_file incoming_cmd = 'exec: cat ' + mig_file +def get_bitmap_hash(vm): + result = vm.qmp('x-debug-block-dirty-bitmap-sha256', + node='drive0', name='bitmap0') + return result['return']['sha256'] + + class TestDirtyBitmapMigration(iotests.QMPTestCase): def tearDown(self): self.vm_a.shutdown() @@ -62,21 +67,16 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): params['persistent'] = True result = vm.qmp('block-dirty-bitmap-add', **params) - self.assert_qmp(result, 'return', {}); - - def get_bitmap_hash(self, vm): - result = vm.qmp('x-debug-block-dirty-bitmap-sha256', - node='drive0', name='bitmap0') - return result['return']['sha256'] + self.assert_qmp(result, 'return', {}) def check_bitmap(self, vm, sha256): result = vm.qmp('x-debug-block-dirty-bitmap-sha256', node='drive0', name='bitmap0') if sha256: - self.assert_qmp(result, 'return/sha256', sha256); + self.assert_qmp(result, 'return/sha256', sha256) else: self.assert_qmp(result, 'error/desc', - "Dirty bitmap 'bitmap0' not found"); + "Dirty bitmap 'bitmap0' not found") def do_test_migration_resume_source(self, persistent, migrate_bitmaps): granularity = 512 @@ -97,7 +97,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): self.add_bitmap(self.vm_a, granularity, persistent) for r in regions: self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % r) - sha256 = self.get_bitmap_hash(self.vm_a) + sha256 = get_bitmap_hash(self.vm_a) result = self.vm_a.qmp('migrate', uri=mig_cmd) while True: @@ -106,7 +106,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): break while True: result = self.vm_a.qmp('query-status') - if (result['return']['status'] == 'postmigrate'): + if result['return']['status'] == 'postmigrate': break # test that bitmap is still here @@ -164,7 +164,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): self.add_bitmap(self.vm_a, granularity, persistent) for r in regions: self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % r) - sha256 = self.get_bitmap_hash(self.vm_a) + sha256 = get_bitmap_hash(self.vm_a) if pre_shutdown: self.vm_a.shutdown() @@ -214,16 +214,22 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): self.check_bitmap(self.vm_b, sha256 if persistent else False) -def inject_test_case(klass, name, method, *args, **kwargs): +def inject_test_case(klass, suffix, method, *args, **kwargs): mc = operator.methodcaller(method, *args, **kwargs) - setattr(klass, 'test_' + method + name, lambda self: mc(self)) + # We want to add a function attribute to `klass`, so that it is + # correctly converted to a method on instantiation. The + # methodcaller object `mc` is a callable, not a function, so we + # need the lambda to turn it into a function. + # pylint: disable=unnecessary-lambda + setattr(klass, 'test_' + method + suffix, lambda self: mc(self)) + for cmb in list(itertools.product((True, False), repeat=5)): name = ('_' if cmb[0] else '_not_') + 'persistent_' name += ('_' if cmb[1] else '_not_') + 'migbitmap_' name += '_online' if cmb[2] else '_offline' name += '_shared' if cmb[3] else '_nonshared' - if (cmb[4]): + if cmb[4]: name += '__pre_shutdown' inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration', @@ -270,7 +276,8 @@ class TestDirtyBitmapBackingMigration(iotests.QMPTestCase): self.assert_qmp(result, 'return', {}) # Check that the bitmaps are there - for node in self.vm.qmp('query-named-block-nodes', flat=True)['return']: + nodes = self.vm.qmp('query-named-block-nodes', flat=True)['return'] + for node in nodes: if 'node0' in node['node-name']: self.assert_qmp(node, 'dirty-bitmaps[0]/name', 'bmap0') @@ -287,7 +294,7 @@ class TestDirtyBitmapBackingMigration(iotests.QMPTestCase): """ Continue the source after migration. """ - result = self.vm.qmp('migrate', uri=f'exec: cat > /dev/null') + result = self.vm.qmp('migrate', uri='exec: cat > /dev/null') self.assert_qmp(result, 'return', {}) with Timeout(10, 'Migration timeout'): From patchwork Wed Sep 15 17:52:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C039C433EF for ; Wed, 15 Sep 2021 18:07:24 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 19D23611C6 for ; Wed, 15 Sep 2021 18:07:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 19D23611C6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:55276 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZJb-0007iU-7O for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:07:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51838) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6X-0006MN-VP for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:54 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:58763) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6V-0006lP-Tr for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728431; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QqIe7dS/ryYf9qQ+IGWc/6eeiCKykochrunSgl7XrQk=; b=MybJtlB+Zfex2pszrRPSd/qIoIXfe4xZ0RulCnDUnIEobKaa7TK39qo5iE2RoJFqgZBaBz p7t6gD/iw9/IAPigdVHLtANmTWWm3/KOw97tBSw64y+rWxu1jkK0OQVa/7cwo1EZ5oUvEZ rMPy41jwRbK533glYs7kMJqXpyQtjd8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-333-UYSjoIWCPferI3DiSyZZJg-1; Wed, 15 Sep 2021 13:53:49 -0400 X-MC-Unique: UYSjoIWCPferI3DiSyZZJg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 961FC5074B; Wed, 15 Sep 2021 17:53:48 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 384136A908; Wed, 15 Sep 2021 17:53:48 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 13/32] mirror-top-perms: Fix AbnormalShutdown path Date: Wed, 15 Sep 2021 19:52:59 +0200 Message-Id: <20210915175318.853225-14-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The AbnormalShutdown exception class is not in qemu.machine, but in qemu.machine.machine. (qemu.machine.AbnormalShutdown was enough for Python to find it in order to run this test, but pylint complains about it.) Signed-off-by: Hanna Reitz Message-Id: <20210902094017.32902-5-hreitz@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/mirror-top-perms | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/tests/mirror-top-perms b/tests/qemu-iotests/tests/mirror-top-perms index 451a0666f8..2fc8dd66e0 100755 --- a/tests/qemu-iotests/tests/mirror-top-perms +++ b/tests/qemu-iotests/tests/mirror-top-perms @@ -47,7 +47,7 @@ class TestMirrorTopPerms(iotests.QMPTestCase): def tearDown(self): try: self.vm.shutdown() - except qemu.machine.AbnormalShutdown: + except qemu.machine.machine.AbnormalShutdown: pass if self.vm_b is not None: From patchwork Wed Sep 15 17:53:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B435C433F5 for ; Wed, 15 Sep 2021 18:07:01 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 275D860F51 for ; Wed, 15 Sep 2021 18:07:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 275D860F51 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:53482 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZJE-0006P3-9d for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:07:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51916) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6c-0006RQ-UK for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:59 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:42250) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6a-0006pO-TG for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728435; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lkOkeoxp19pmD7HeGfldqA9NqC7NzqGhqxfyK41gAp4=; b=Vq0C2Og8CdotwZHDn2RkblZr2AO3KKymU8twGNVEIga1+9+iHSpmCX454mgHbNzMdcPsyQ I/MMt+SLUOuZ325RBepU+Yc2Q8ZI2oIiB1lZRadbm20X9ipojff+1tNRdTALAP4eSIWyH4 w9i+LE5+UnjOaBm7Hrgdd3JJ1hkgA/w= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-362-6rzER8G8NXKF_OLsnfC95g-1; Wed, 15 Sep 2021 13:53:51 -0400 X-MC-Unique: 6rzER8G8NXKF_OLsnfC95g-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id AF8911084684; Wed, 15 Sep 2021 17:53:50 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 57BE25C1C5; Wed, 15 Sep 2021 17:53:50 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 14/32] iotests/297: Cover tests/ Date: Wed, 15 Sep 2021 19:53:00 +0200 Message-Id: <20210915175318.853225-15-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" 297 so far does not check the named tests, which reside in the tests/ directory (i.e. full path tests/qemu-iotests/tests). Fix it. Thanks to the previous two commits, all named tests pass its scrutiny, so we do not have to add anything to SKIP_FILES. Signed-off-by: Hanna Reitz Reviewed-by: Willian Rampazzo Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Kevin Wolf Message-Id: <20210902094017.32902-6-hreitz@redhat.com> --- tests/qemu-iotests/297 | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297 index a0c0cf9e5d..b04cba5366 100755 --- a/tests/qemu-iotests/297 +++ b/tests/qemu-iotests/297 @@ -55,8 +55,9 @@ def is_python_file(filename): def run_linters(): - files = [filename for filename in (set(os.listdir('.')) - set(SKIP_FILES)) - if is_python_file(filename)] + named_tests = [f'tests/{entry}' for entry in os.listdir('tests')] + check_tests = set(os.listdir('.') + named_tests) - set(SKIP_FILES) + files = [filename for filename in check_tests if is_python_file(filename)] iotests.logger.debug('Files to be checked:') iotests.logger.debug(', '.join(sorted(files))) From patchwork Wed Sep 15 17:53:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8442C433F5 for ; Wed, 15 Sep 2021 18:04:59 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A5666611C4 for ; Wed, 15 Sep 2021 18:04:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A5666611C4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:48606 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZHG-00035r-S6 for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:04:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51882) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6b-0006Qz-Pl for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:59 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:49981) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6Z-0006p2-Ut for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:53:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728435; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FQTGBN/4iuV8wISjMgE98z+muO+KQ01nh5WbJ/PiZ54=; b=HWdiJVTfj5NnpzBzzWwpqO6nulC07oF68o3F0ZXKoJ234LCvPrYCxExlhKmtsIOz4mc5mQ StYEQrlPQEukvixTdsOt+N2uyNiAwJKH66lFcG+tMB3W7fPCALzayk3CyosDHjUiuB82vZ Tdd3947VItqc7x5qRiInQcQNp/SiAxE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-406-_zMm7k7HPLWHxoTp8-r2hw-1; Wed, 15 Sep 2021 13:53:53 -0400 X-MC-Unique: _zMm7k7HPLWHxoTp8-r2hw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C205C5074C; Wed, 15 Sep 2021 17:53:52 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 690441B5B7; Wed, 15 Sep 2021 17:53:52 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 15/32] block/mirror: fix NULL pointer dereference in mirror_wait_on_conflicts() Date: Wed, 15 Sep 2021 19:53:01 +0200 Message-Id: <20210915175318.853225-16-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Stefano Garzarella In mirror_iteration() we call mirror_wait_on_conflicts() with `self` parameter set to NULL. Starting from commit d44dae1a7c we dereference `self` pointer in mirror_wait_on_conflicts() without checks if it is not NULL. Backtrace: Program terminated with signal SIGSEGV, Segmentation fault. #0 mirror_wait_on_conflicts (self=0x0, s=, offset=, bytes=) at ../block/mirror.c:172 172 self->waiting_for_op = op; [Current thread is 1 (Thread 0x7f0908931ec0 (LWP 380249))] (gdb) bt #0 mirror_wait_on_conflicts (self=0x0, s=, offset=, bytes=) at ../block/mirror.c:172 #1 0x00005610c5d9d631 in mirror_run (job=0x5610c76a2c00, errp=) at ../block/mirror.c:491 #2 0x00005610c5d58726 in job_co_entry (opaque=0x5610c76a2c00) at ../job.c:917 #3 0x00005610c5f046c6 in coroutine_trampoline (i0=, i1=) at ../util/coroutine-ucontext.c:173 #4 0x00007f0909975820 in ?? () at ../sysdeps/unix/sysv/linux/x86_64/__start_context.S:91 from /usr/lib64/libc.so.6 Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2001404 Fixes: d44dae1a7c ("block/mirror: fix active mirror dead-lock in mirror_wait_on_conflicts") Signed-off-by: Stefano Garzarella Message-Id: <20210910124533.288318-1-sgarzare@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Hanna Reitz --- block/mirror.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/block/mirror.c b/block/mirror.c index 98fc66eabf..85b781bc21 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -160,18 +160,25 @@ static void coroutine_fn mirror_wait_on_conflicts(MirrorOp *self, if (ranges_overlap(self_start_chunk, self_nb_chunks, op_start_chunk, op_nb_chunks)) { - /* - * If the operation is already (indirectly) waiting for us, or - * will wait for us as soon as it wakes up, then just go on - * (instead of producing a deadlock in the former case). - */ - if (op->waiting_for_op) { - continue; + if (self) { + /* + * If the operation is already (indirectly) waiting for us, + * or will wait for us as soon as it wakes up, then just go + * on (instead of producing a deadlock in the former case). + */ + if (op->waiting_for_op) { + continue; + } + + self->waiting_for_op = op; } - self->waiting_for_op = op; qemu_co_queue_wait(&op->waiting_requests, NULL); - self->waiting_for_op = NULL; + + if (self) { + self->waiting_for_op = NULL; + } + break; } } From patchwork Wed Sep 15 17:53:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497099 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46A44C433F5 for ; Wed, 15 Sep 2021 18:08:06 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BF03B611C8 for ; Wed, 15 Sep 2021 18:08:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org BF03B611C8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:57036 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZKG-0000Th-Vq for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:08:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51940) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6e-0006Se-Dh for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:01 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:49258) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6c-0006rH-9E for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728437; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7f8jWXwv/UeM2qL7zOLHTHGXiOIsI8zl10DcCWQrbtA=; b=CNu7Vs9ex96+Yu9UtVCmSxluHATeVJ7T9eQ0jmvS9EdSOSKAMqwRTV3iNhW0/cb+RsjZ9l fAqaxy3kEv0b/ZDnJsSoCQsTJjYTh+gHTPIfwPC8BSLHTvCRdqk4ZU1T/0y0ROmiRzelUc OL8tn6o/Rjd+8zRDbzQf87uTX6I7/iA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-470-1Ao9B-FnMyC97usb_g3grw-1; Wed, 15 Sep 2021 13:53:55 -0400 X-MC-Unique: 1Ao9B-FnMyC97usb_g3grw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id F2B151966321; Wed, 15 Sep 2021 17:53:54 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 828BA6B54B; Wed, 15 Sep 2021 17:53:54 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 16/32] tests: add migrate-during-backup Date: Wed, 15 Sep 2021 19:53:02 +0200 Message-Id: <20210915175318.853225-17-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Add a simple test which tries to run migration during backup. bdrv_inactivate_all() should fail. But due to bug (see next commit with fix) it doesn't, nodes are inactivated and continued backup crashes on assertion "assert(!(bs->open_flags & BDRV_O_INACTIVE));" in bdrv_co_write_req_prepare(). Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20210911120027.8063-2-vsementsov@virtuozzo.com> Signed-off-by: Hanna Reitz --- .../qemu-iotests/tests/migrate-during-backup | 97 +++++++++++++++++++ .../tests/migrate-during-backup.out | 5 + 2 files changed, 102 insertions(+) create mode 100755 tests/qemu-iotests/tests/migrate-during-backup create mode 100644 tests/qemu-iotests/tests/migrate-during-backup.out diff --git a/tests/qemu-iotests/tests/migrate-during-backup b/tests/qemu-iotests/tests/migrate-during-backup new file mode 100755 index 0000000000..1046904c5a --- /dev/null +++ b/tests/qemu-iotests/tests/migrate-during-backup @@ -0,0 +1,97 @@ +#!/usr/bin/env python3 +# group: migration disabled +# +# Copyright (c) 2021 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +import os +import iotests +from iotests import qemu_img_create, qemu_io + + +disk_a = os.path.join(iotests.test_dir, 'disk_a') +disk_b = os.path.join(iotests.test_dir, 'disk_b') +size = '1M' +mig_file = os.path.join(iotests.test_dir, 'mig_file') +mig_cmd = 'exec: cat > ' + mig_file + + +class TestMigrateDuringBackup(iotests.QMPTestCase): + def tearDown(self): + self.vm.shutdown() + os.remove(disk_a) + os.remove(disk_b) + os.remove(mig_file) + + def setUp(self): + qemu_img_create('-f', iotests.imgfmt, disk_a, size) + qemu_img_create('-f', iotests.imgfmt, disk_b, size) + qemu_io('-c', f'write 0 {size}', disk_a) + + self.vm = iotests.VM().add_drive(disk_a) + self.vm.launch() + result = self.vm.qmp('blockdev-add', { + 'node-name': 'target', + 'driver': iotests.imgfmt, + 'file': { + 'driver': 'file', + 'filename': disk_b + } + }) + self.assert_qmp(result, 'return', {}) + + def test_migrate(self): + result = self.vm.qmp('blockdev-backup', device='drive0', + target='target', sync='full', + speed=1, x_perf={ + 'max-workers': 1, + 'max-chunk': 64 * 1024 + }) + self.assert_qmp(result, 'return', {}) + + result = self.vm.qmp('job-pause', id='drive0') + self.assert_qmp(result, 'return', {}) + + result = self.vm.qmp('migrate-set-capabilities', + capabilities=[{'capability': 'events', + 'state': True}]) + self.assert_qmp(result, 'return', {}) + result = self.vm.qmp('migrate', uri=mig_cmd) + self.assert_qmp(result, 'return', {}) + + e = self.vm.events_wait((('MIGRATION', + {'data': {'status': 'completed'}}), + ('MIGRATION', + {'data': {'status': 'failed'}}))) + + # Don't assert that e is 'failed' now: this way we'll miss + # possible crash when backup continues :) + + result = self.vm.qmp('block-job-set-speed', device='drive0', + speed=0) + self.assert_qmp(result, 'return', {}) + result = self.vm.qmp('job-resume', id='drive0') + self.assert_qmp(result, 'return', {}) + + # For future: if something changes so that both migration + # and backup pass, let's not miss that moment, as it may + # be a bug as well as improvement. + self.assert_qmp(e, 'data/status', 'failed') + + +if __name__ == '__main__': + iotests.main(supported_fmts=['qcow2'], + supported_protocols=['file']) diff --git a/tests/qemu-iotests/tests/migrate-during-backup.out b/tests/qemu-iotests/tests/migrate-during-backup.out new file mode 100644 index 0000000000..ae1213e6f8 --- /dev/null +++ b/tests/qemu-iotests/tests/migrate-during-backup.out @@ -0,0 +1,5 @@ +. +---------------------------------------------------------------------- +Ran 1 tests + +OK From patchwork Wed Sep 15 17:53:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497107 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86ECBC433F5 for ; Wed, 15 Sep 2021 18:11:33 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1662260041 for ; Wed, 15 Sep 2021 18:11:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 1662260041 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:37070 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZNc-00068D-8b for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:11:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51968) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6f-0006UE-Mu for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:01 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:22260) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6e-0006tA-3h for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728439; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JJLyditKkv+WSfGTBJalfCfVjtKWxjAtVBFYv2pOuv0=; b=I36BH/XY8GrFHzKz27WoJ1vgONMtWED+/fqFVNxvVQdhI+KN+IazEEr7KEGbUapZSvSAFk mr7u6kZmhhxuaqvS93GMCWCRl+6FKG9nUzAHhQZaJgzcKu7oOQeHj6PDdO/c5QJ8LnXs6b Q3auXBcwaV6PUtophvZMLOVCKAHWhhI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-393-axuihbpCOn6TQjxSnznaKg-1; Wed, 15 Sep 2021 13:53:58 -0400 X-MC-Unique: axuihbpCOn6TQjxSnznaKg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EB2021966320; Wed, 15 Sep 2021 17:53:56 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9498F7771A; Wed, 15 Sep 2021 17:53:56 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 17/32] block: bdrv_inactivate_recurse(): check for permissions and fix crash Date: Wed, 15 Sep 2021 19:53:03 +0200 Message-Id: <20210915175318.853225-18-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy We must not inactivate child when parent has write permissions on it. Calling .bdrv_inactivate() doesn't help: actually only qcow2 has this handler and it is used to flush caches, not for permission manipulations. So, let's simply check cumulative parent permissions before inactivating the node. This commit fixes a crash when we do migration during backup: prior to the commit nothing prevents all nodes inactivation at migration finish and following backup write to the target crashes on assertion "assert(!(bs->open_flags & BDRV_O_INACTIVE));" in bdrv_co_write_req_prepare(). After the commit, we rely on the fact that copy-before-write filter keeps write permission on target node to be able to write to it. So inactivation fails and migration fails as expected. Corresponding test now passes, so, enable it. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz Message-Id: <20210911120027.8063-3-vsementsov@virtuozzo.com> Signed-off-by: Hanna Reitz --- block.c | 8 ++++++++ tests/qemu-iotests/tests/migrate-during-backup | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/block.c b/block.c index 00982edf45..5ce08a79fd 100644 --- a/block.c +++ b/block.c @@ -6326,6 +6326,7 @@ static int bdrv_inactivate_recurse(BlockDriverState *bs) { BdrvChild *child, *parent; int ret; + uint64_t cumulative_perms, cumulative_shared_perms; if (!bs->drv) { return -ENOMEDIUM; @@ -6356,6 +6357,13 @@ static int bdrv_inactivate_recurse(BlockDriverState *bs) } } + bdrv_get_cumulative_perm(bs, &cumulative_perms, + &cumulative_shared_perms); + if (cumulative_perms & (BLK_PERM_WRITE | BLK_PERM_WRITE_UNCHANGED)) { + /* Our inactive parents still need write access. Inactivation failed. */ + return -EPERM; + } + bs->open_flags |= BDRV_O_INACTIVE; /* diff --git a/tests/qemu-iotests/tests/migrate-during-backup b/tests/qemu-iotests/tests/migrate-during-backup index 1046904c5a..34103229ee 100755 --- a/tests/qemu-iotests/tests/migrate-during-backup +++ b/tests/qemu-iotests/tests/migrate-during-backup @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# group: migration disabled +# group: migration # # Copyright (c) 2021 Virtuozzo International GmbH # From patchwork Wed Sep 15 17:53:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497117 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50373C433F5 for ; Wed, 15 Sep 2021 18:16:32 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DA46C611C8 for ; Wed, 15 Sep 2021 18:16:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org DA46C611C8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:45872 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZSQ-000429-SR for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:16:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52068) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6o-0006vd-5X for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:10 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:27113) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6k-0006yo-Cs for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728445; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tp9T3gUpiqm09jJvfntCGMWWH/0DFA5yq3rv2TX7cNc=; b=fVf6YvLY5+mUimaAi+vHUDP0VItLkrb6iWue1SmYO05G3JPHph2FAawAssQGFN0UiuXBC3 4BgOneedw91oU6ynun+PC7gOsWuvksRWfVi/rJlP/l1td9K7ZZJZtTVJurhLuCgsj3i5SM bj5Lr9AL/jH8Jm6Ohepwgc3W6jAZgr8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-512-JFOYivlhPoKjWS5WcJ2Nbg-1; Wed, 15 Sep 2021 13:54:00 -0400 X-MC-Unique: JFOYivlhPoKjWS5WcJ2Nbg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3246C1084683; Wed, 15 Sep 2021 17:53:59 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A3ACB77BF5; Wed, 15 Sep 2021 17:53:58 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 18/32] simplebench: add img_bench_templater.py Date: Wed, 15 Sep 2021 19:53:04 +0200 Message-Id: <20210915175318.853225-19-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Add simple grammar-parsing template benchmark. New tool consume test template written in bash with some special grammar injections and produces multiple tests, run them and finally print a performance comparison table of different tests produced from one template. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20210824101517.59802-2-vsementsov@virtuozzo.com> Reviewed-by: Hanna Reitz Signed-off-by: Hanna Reitz --- scripts/simplebench/img_bench_templater.py | 95 ++++++++++++++++++++++ scripts/simplebench/table_templater.py | 62 ++++++++++++++ 2 files changed, 157 insertions(+) create mode 100755 scripts/simplebench/img_bench_templater.py create mode 100644 scripts/simplebench/table_templater.py diff --git a/scripts/simplebench/img_bench_templater.py b/scripts/simplebench/img_bench_templater.py new file mode 100755 index 0000000000..f8e1540ada --- /dev/null +++ b/scripts/simplebench/img_bench_templater.py @@ -0,0 +1,95 @@ +#!/usr/bin/env python3 +# +# Process img-bench test templates +# +# Copyright (c) 2021 Virtuozzo International GmbH. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + + +import sys +import subprocess +import re +import json + +import simplebench +from results_to_text import results_to_text +from table_templater import Templater + + +def bench_func(env, case): + test = templater.gen(env['data'], case['data']) + + p = subprocess.run(test, shell=True, stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, universal_newlines=True) + + if p.returncode == 0: + try: + m = re.search(r'Run completed in (\d+.\d+) seconds.', p.stdout) + return {'seconds': float(m.group(1))} + except Exception: + return {'error': f'failed to parse qemu-img output: {p.stdout}'} + else: + return {'error': f'qemu-img failed: {p.returncode}: {p.stdout}'} + + +if __name__ == '__main__': + if len(sys.argv) > 1: + print(""" +Usage: img_bench_templater.py < path/to/test-template.sh + +This script generates performance tests from a test template (example below), +runs them, and displays the results in a table. The template is read from +stdin. It must be written in bash and end with a `qemu-img bench` invocation +(whose result is parsed to get the test instance’s result). + +Use the following syntax in the template to create the various different test +instances: + + column templating: {var1|var2|...} - test will use different values in + different columns. You may use several {} constructions in the test, in this + case product of all choice-sets will be used. + + row templating: [var1|var2|...] - similar thing to define rows (test-cases) + +Test template example: + +Assume you want to compare two qemu-img binaries, called qemu-img-old and +qemu-img-new in your build directory in two test-cases with 4K writes and 64K +writes. The template may look like this: + +qemu_img=/path/to/qemu/build/qemu-img-{old|new} +$qemu_img create -f qcow2 /ssd/x.qcow2 1G +$qemu_img bench -c 100 -d 8 [-s 4K|-s 64K] -w -t none -n /ssd/x.qcow2 + +When passing this to stdin of img_bench_templater.py, the resulting comparison +table will contain two columns (for two binaries) and two rows (for two +test-cases). + +In addition to displaying the results, script also stores results in JSON +format into results.json file in current directory. +""") + sys.exit() + + templater = Templater(sys.stdin.read()) + + envs = [{'id': ' / '.join(x), 'data': x} for x in templater.columns] + cases = [{'id': ' / '.join(x), 'data': x} for x in templater.rows] + + result = simplebench.bench(bench_func, envs, cases, count=5, + initial_run=False) + print(results_to_text(result)) + with open('results.json', 'w') as f: + json.dump(result, f, indent=4) diff --git a/scripts/simplebench/table_templater.py b/scripts/simplebench/table_templater.py new file mode 100644 index 0000000000..950f3b3024 --- /dev/null +++ b/scripts/simplebench/table_templater.py @@ -0,0 +1,62 @@ +# Parser for test templates +# +# Copyright (c) 2021 Virtuozzo International GmbH. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +import itertools +from lark import Lark + +grammar = """ +start: ( text | column_switch | row_switch )+ + +column_switch: "{" text ["|" text]+ "}" +row_switch: "[" text ["|" text]+ "]" +text: /[^|{}\[\]]+/ +""" + +parser = Lark(grammar) + +class Templater: + def __init__(self, template): + self.tree = parser.parse(template) + + c_switches = [] + r_switches = [] + for x in self.tree.children: + if x.data == 'column_switch': + c_switches.append([el.children[0].value for el in x.children]) + elif x.data == 'row_switch': + r_switches.append([el.children[0].value for el in x.children]) + + self.columns = list(itertools.product(*c_switches)) + self.rows = list(itertools.product(*r_switches)) + + def gen(self, column, row): + i = 0 + j = 0 + result = [] + + for x in self.tree.children: + if x.data == 'text': + result.append(x.children[0].value) + elif x.data == 'column_switch': + result.append(column[i]) + i += 1 + elif x.data == 'row_switch': + result.append(row[j]) + j += 1 + + return ''.join(result) From patchwork Wed Sep 15 17:53:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B34CC433EF for ; Wed, 15 Sep 2021 18:05:51 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1F5D1611C4 for ; Wed, 15 Sep 2021 18:05:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 1F5D1611C4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:50706 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZI6-0004W1-8g for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:05:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52028) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6l-0006qM-AN for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:07 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:54621) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6j-0006y2-MO for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728445; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/5k33/WjhnjqrPOeMVHgyyNGwRokJxqIE8rdiNy64aY=; b=cw7145y1FnjYF7jc+FT84hpOnZ+zwCmsRtJVamEUUpI7aWsuw1macFpEm8eouqbik2c6So xrjA2Di+HfXnbsvwEJtt61B+jENN7KxV74FcsnlBHtMCPoNXYQSZKDwRbxpNlMn9qPYcNB TB6nQem2Fzu1lKM79Ct6xs7wxTE6MhE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-241-Ghs7VUoTNGK2-mGU7iHFWg-1; Wed, 15 Sep 2021 13:54:02 -0400 X-MC-Unique: Ghs7VUoTNGK2-mGU7iHFWg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 40E5E802928; Wed, 15 Sep 2021 17:54:01 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DF06810016FB; Wed, 15 Sep 2021 17:54:00 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 19/32] qcow2: refactor handle_dependencies() loop body Date: Wed, 15 Sep 2021 19:53:05 +0200 Message-Id: <20210915175318.853225-20-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy No logic change, just prepare for the following commit. While being here do also small grammar fix in a comment. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Reviewed-by: Hanna Reitz Message-Id: <20210824101517.59802-3-vsementsov@virtuozzo.com> Signed-off-by: Hanna Reitz --- block/qcow2-cluster.c | 49 ++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 21 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index bd0597842f..9917e5c28c 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -1400,29 +1400,36 @@ static int handle_dependencies(BlockDriverState *bs, uint64_t guest_offset, if (end <= old_start || start >= old_end) { /* No intersection */ - } else { - if (start < old_start) { - /* Stop at the start of a running allocation */ - bytes = old_start - start; - } else { - bytes = 0; - } + continue; + } - /* Stop if already an l2meta exists. After yielding, it wouldn't - * be valid any more, so we'd have to clean up the old L2Metas - * and deal with requests depending on them before starting to - * gather new ones. Not worth the trouble. */ - if (bytes == 0 && *m) { - *cur_bytes = 0; - return 0; - } + /* Conflict */ - if (bytes == 0) { - /* Wait for the dependency to complete. We need to recheck - * the free/allocated clusters when we continue. */ - qemu_co_queue_wait(&old_alloc->dependent_requests, &s->lock); - return -EAGAIN; - } + if (start < old_start) { + /* Stop at the start of a running allocation */ + bytes = old_start - start; + } else { + bytes = 0; + } + + /* + * Stop if an l2meta already exists. After yielding, it wouldn't + * be valid any more, so we'd have to clean up the old L2Metas + * and deal with requests depending on them before starting to + * gather new ones. Not worth the trouble. + */ + if (bytes == 0 && *m) { + *cur_bytes = 0; + return 0; + } + + if (bytes == 0) { + /* + * Wait for the dependency to complete. We need to recheck + * the free/allocated clusters when we continue. + */ + qemu_co_queue_wait(&old_alloc->dependent_requests, &s->lock); + return -EAGAIN; } } From patchwork Wed Sep 15 17:53:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497113 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D728CC433F5 for ; Wed, 15 Sep 2021 18:16:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6E525611C4 for ; Wed, 15 Sep 2021 18:16:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 6E525611C4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:44222 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZRy-0002od-6P for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:16:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52138) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6r-000760-JM for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:13 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:56950) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6m-00070K-3S for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728447; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GKhGGeL/HQSlt9pmGXMeq7t9aX9YoIvPwLp/0+X6+2Q=; b=dnkkFxfVT4ZSs9YhvP/qu54tL1NmwSSXWu1a7COqFKDlvF1qah+GCFbSYCscAeBvHWFkVF gGdVVLIz+xd37STNLprABr60W1NQxB5ofDlVTg2MuFS2k+GOZnewleQC+l/7Vg9DgK9L7L 0+s2OZkpZcXAD0hhz8GdR8cGNgGCUiY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-204-tmNFrr1ZPnaOB8qgNT5mBg-1; Wed, 15 Sep 2021 13:54:04 -0400 X-MC-Unique: tmNFrr1ZPnaOB8qgNT5mBg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8697784A5EB; Wed, 15 Sep 2021 17:54:03 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 028381972E; Wed, 15 Sep 2021 17:54:02 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 20/32] qcow2: handle_dependencies(): relax conflict detection Date: Wed, 15 Sep 2021 19:53:06 +0200 Message-Id: <20210915175318.853225-21-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy There is no conflict and no dependency if we have parallel writes to different subclusters of one cluster when the cluster itself is already allocated. So, relax extra dependency. Measure performance: First, prepare build/qemu-img-old and build/qemu-img-new images. cd scripts/simplebench ./img_bench_templater.py Paste the following to stdin of running script: qemu_img=../../build/qemu-img-{old|new} $qemu_img create -f qcow2 -o extended_l2=on /ssd/x.qcow2 1G $qemu_img bench -c 100000 -d 8 [-s 2K|-s 2K -o 512|-s $((1024*2+512))] \ -w -t none -n /ssd/x.qcow2 The result: All results are in seconds ------------------ --------- --------- old new -s 2K 6.7 ± 15% 6.2 ± 12% -7% -s 2K -o 512 13 ± 3% 11 ± 5% -16% -s $((1024*2+512)) 9.5 ± 4% 8.4 -12% ------------------ --------- --------- So small writes are more independent now and that helps to keep deeper io queue which improves performance. 271 iotest output becomes racy for three allocation in one cluster. Second and third writes may finish in different order. Second and third requests don't depend on each other any more. Still they both depend on first request anyway. Filter out second and third write offsets to cover both possible outputs. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20210824101517.59802-4-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake Reviewed-by: Hanna Reitz [hreitz: s/ an / and /] Signed-off-by: Hanna Reitz --- block/qcow2-cluster.c | 11 +++++++++++ tests/qemu-iotests/271 | 5 ++++- tests/qemu-iotests/271.out | 4 ++-- 3 files changed, 17 insertions(+), 3 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 9917e5c28c..c1c43a891b 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -1403,6 +1403,17 @@ static int handle_dependencies(BlockDriverState *bs, uint64_t guest_offset, continue; } + if (old_alloc->keep_old_clusters && + (end <= l2meta_cow_start(old_alloc) || + start >= l2meta_cow_end(old_alloc))) + { + /* + * Clusters intersect but COW areas don't. And cluster itself is + * already allocated. So, there is no actual conflict. + */ + continue; + } + /* Conflict */ if (start < old_start) { diff --git a/tests/qemu-iotests/271 b/tests/qemu-iotests/271 index 599b849cc6..2775b4d130 100755 --- a/tests/qemu-iotests/271 +++ b/tests/qemu-iotests/271 @@ -893,7 +893,10 @@ EOF } _make_test_img -o extended_l2=on 1M -_concurrent_io | $QEMU_IO | _filter_qemu_io +# Second and third writes in _concurrent_io() are independent and may finish in +# different order. So, filter offset out to match both possible variants. +_concurrent_io | $QEMU_IO | _filter_qemu_io | \ + $SED -e 's/\(20480\|40960\)/OFFSET/' _concurrent_verify | $QEMU_IO | _filter_qemu_io # success, all done diff --git a/tests/qemu-iotests/271.out b/tests/qemu-iotests/271.out index 81043ba4d7..5be780de76 100644 --- a/tests/qemu-iotests/271.out +++ b/tests/qemu-iotests/271.out @@ -719,8 +719,8 @@ blkdebug: Suspended request 'A' blkdebug: Resuming request 'A' wrote 2048/2048 bytes at offset 30720 2 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 2048/2048 bytes at offset 20480 +wrote 2048/2048 bytes at offset OFFSET 2 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 2048/2048 bytes at offset 40960 +wrote 2048/2048 bytes at offset OFFSET 2 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) *** done From patchwork Wed Sep 15 17:53:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497105 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 652CFC433F5 for ; Wed, 15 Sep 2021 18:11:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1A17B611C6 for ; Wed, 15 Sep 2021 18:11:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 1A17B611C6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:36154 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZN8-0005Vs-0H for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:11:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52128) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6q-00071L-C0 for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:12 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:55440) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6n-00071E-LG for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728448; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1ZNUvK+igyqmEZ+P+tnZfm/uqiSbIePHbkyWvCn0fFU=; b=CBQOXs3pCtRGEhhbXkhRuVHdpwAvZg0DS8c15jhlY2zZPa+wfFvpXqGgdSTIUx9/QweIZq ycBUP+7SGEFvkt93wC8FTCMmVcg/GL5SgpxecUkMzuYFDAlsz8GeKxwDgfO6Wi7sG3WA1D wj6e7AcNU0l4Zm4EpVy3T1iun4HdObo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-526-mHKnlEr8O2-rXWrY30nviA-1; Wed, 15 Sep 2021 13:54:06 -0400 X-MC-Unique: mHKnlEr8O2-rXWrY30nviA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7C79B1084683; Wed, 15 Sep 2021 17:54:05 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1FDC3604CC; Wed, 15 Sep 2021 17:54:04 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 21/32] qemu-img: Allow target be aligned to sector size Date: Wed, 15 Sep 2021 19:53:07 +0200 Message-Id: <20210915175318.853225-22-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" We cannot write to images opened with O_DIRECT unless we allow them to be resized so they are aligned to the sector size: Since 9c60a5d1978, bdrv_node_refresh_perm() ensures that for nodes whose length is not aligned to the request alignment and where someone has taken a WRITE permission, the RESIZE permission is taken, too). Let qemu-img convert pass the BDRV_O_RESIZE flag (which causes blk_new_open() to take the RESIZE permission) when using cache=none for the target, so that when writing to it, it can be aligned to the target sector size. Without this patch, an error is returned: $ qemu-img convert -f raw -O raw -t none foo.img /mnt/tmp/foo.img qemu-img: Could not open '/mnt/tmp/foo.img': Cannot get 'write' permission without 'resize': Image size is not a multiple of request alignment Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1994266 Signed-off-by: Hanna Reitz Message-Id: <20210819101200.64235-1-hreitz@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy --- qemu-img.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/qemu-img.c b/qemu-img.c index d77f3e76a9..e43a71a794 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -2628,6 +2628,14 @@ static int img_convert(int argc, char **argv) goto out; } + if (flags & BDRV_O_NOCACHE) { + /* + * If we open the target with O_DIRECT, it may be necessary to + * extend its size to align to the physical sector size. + */ + flags |= BDRV_O_RESIZE; + } + if (skip_create) { s.target = img_open(tgt_image_opts, out_filename, out_fmt, flags, writethrough, s.quiet, false); From patchwork Wed Sep 15 17:53:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1B51C433F5 for ; Wed, 15 Sep 2021 18:21:13 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4ED1560F26 for ; Wed, 15 Sep 2021 18:21:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4ED1560F26 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:55162 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZWy-00020S-Gh for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:21:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52172) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6s-0007A2-In for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:14 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:57925) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6q-00074R-RS for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728452; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2KlHe7tGu//cdJNIfMj1TR4njO/RkTY1YGV0mEbgXyc=; b=WEEJFY+pq0mr0SrtBM1xgYtzVuyUo61wgAQs+c7K8Hu2Fkd8lCWrAC7VdikEGCEt3kGx1t 1HSTBTR8bvln+5q5MdjtAWM37kOpHkqBxUG9XvlCP16GE1acm14Ti8tofgO0/24yxpxfSV jAD1Cx4dCFBHNoJTQ9y96hKKSGbix30= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-449-KD41pAWWPOG8t8X8lRjwQg-1; Wed, 15 Sep 2021 13:54:08 -0400 X-MC-Unique: KD41pAWWPOG8t8X8lRjwQg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 97704802B9F; Wed, 15 Sep 2021 17:54:07 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3D5FE604CC; Wed, 15 Sep 2021 17:54:07 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 22/32] qcow2-refcount: improve style of check_refcounts_l2() Date: Wed, 15 Sep 2021 19:53:08 +0200 Message-Id: <20210915175318.853225-23-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy - don't use same name for size in bytes and in entries - use g_autofree for l2_table - add whitespace - fix block comment style Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Reviewed-by: Hanna Reitz Message-Id: <20210914122454.141075-2-vsementsov@virtuozzo.com> Signed-off-by: Hanna Reitz --- block/qcow2-refcount.c | 47 +++++++++++++++++++++--------------------- 1 file changed, 24 insertions(+), 23 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 8e649b008e..2734338625 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -1601,23 +1601,22 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, int flags, BdrvCheckMode fix, bool active) { BDRVQcow2State *s = bs->opaque; - uint64_t *l2_table, l2_entry; + uint64_t l2_entry; uint64_t next_contiguous_offset = 0; - int i, l2_size, nb_csectors, ret; + int i, nb_csectors, ret; + size_t l2_size_bytes = s->l2_size * l2_entry_size(s); + g_autofree uint64_t *l2_table = g_malloc(l2_size_bytes); /* Read L2 table from disk */ - l2_size = s->l2_size * l2_entry_size(s); - l2_table = g_malloc(l2_size); - - ret = bdrv_pread(bs->file, l2_offset, l2_table, l2_size); + ret = bdrv_pread(bs->file, l2_offset, l2_table, l2_size_bytes); if (ret < 0) { fprintf(stderr, "ERROR: I/O error in check_refcounts_l2\n"); res->check_errors++; - goto fail; + return ret; } /* Do the actual checks */ - for(i = 0; i < s->l2_size; i++) { + for (i = 0; i < s->l2_size; i++) { l2_entry = get_l2_entry(s, l2_table, i); switch (qcow2_get_cluster_type(bs, l2_entry)) { @@ -1647,14 +1646,15 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, l2_entry & QCOW2_COMPRESSED_SECTOR_MASK, nb_csectors * QCOW2_COMPRESSED_SECTOR_SIZE); if (ret < 0) { - goto fail; + return ret; } if (flags & CHECK_FRAG_INFO) { res->bfi.allocated_clusters++; res->bfi.compressed_clusters++; - /* Compressed clusters are fragmented by nature. Since they + /* + * Compressed clusters are fragmented by nature. Since they * take up sub-sector space but we only have sector granularity * I/O we need to re-read the same sectors even for adjacent * compressed clusters. @@ -1700,9 +1700,11 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, if (ret < 0) { fprintf(stderr, "ERROR: Overlap check failed\n"); res->check_errors++; - /* Something is seriously wrong, so abort checking - * this L2 table */ - goto fail; + /* + * Something is seriously wrong, so abort checking + * this L2 table. + */ + return ret; } ret = bdrv_pwrite_sync(bs->file, l2e_offset, @@ -1712,13 +1714,17 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, fprintf(stderr, "ERROR: Failed to overwrite L2 " "table entry: %s\n", strerror(-ret)); res->check_errors++; - /* Do not abort, continue checking the rest of this - * L2 table's entries */ + /* + * Do not abort, continue checking the rest of this + * L2 table's entries. + */ } else { res->corruptions--; res->corruptions_fixed++; - /* Skip marking the cluster as used - * (it is unused now) */ + /* + * Skip marking the cluster as used + * (it is unused now). + */ continue; } } @@ -1743,7 +1749,7 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, refcount_table_size, offset, s->cluster_size); if (ret < 0) { - goto fail; + return ret; } } break; @@ -1758,12 +1764,7 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, } } - g_free(l2_table); return 0; - -fail: - g_free(l2_table); - return ret; } /* From patchwork Wed Sep 15 17:53:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497131 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F56DC433EF for ; Wed, 15 Sep 2021 18:19:37 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2E5A6611C4 for ; Wed, 15 Sep 2021 18:19:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2E5A6611C4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:52712 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZVQ-0000Fm-5H for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:19:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52182) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6s-0007BS-Vj for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:15 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:58302) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6r-00074n-4e for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728452; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e8Kv6b2MENRFnYss7QQVqjazj1kPqRZEvFM3G/HIyuE=; b=H9d+559tc1QUwwTlFim1R37oYP3fObdJm/x7c4+ItF99oksibIuEQzSo3IoYqzTbwh1bdR f2bXeFfJy1Ez6V05B8XgBAO0bmUKqq5EylifqO9g5we9xSLi9yc6Q2Wic4n/ehPl8BibDP c6uUhk86oMbBMcYd++Ic6kc2PzwuubU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-435-XM04Ge3YPYWpSNelKHttOw-1; Wed, 15 Sep 2021 13:54:11 -0400 X-MC-Unique: XM04Ge3YPYWpSNelKHttOw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EE6D784A5E4; Wed, 15 Sep 2021 17:54:09 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 576D319736; Wed, 15 Sep 2021 17:54:09 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 23/32] qcow2: compressed read: simplify cluster descriptor passing Date: Wed, 15 Sep 2021 19:53:09 +0200 Message-Id: <20210915175318.853225-24-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Let's pass the whole L2 entry and not bother with L2E_COMPRESSED_OFFSET_SIZE_MASK. It also helps further refactoring that adds generic qcow2_parse_compressed_l2_entry() helper. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Reviewed-by: Alberto Garcia Reviewed-by: Hanna Reitz Message-Id: <20210914122454.141075-3-vsementsov@virtuozzo.com> Signed-off-by: Hanna Reitz --- block/qcow2.h | 1 - block/qcow2-cluster.c | 5 ++--- block/qcow2.c | 12 +++++++----- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index 0fe5f74ed3..42a0058ab7 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -588,7 +588,6 @@ typedef enum QCow2MetadataOverlap { #define L1E_OFFSET_MASK 0x00fffffffffffe00ULL #define L2E_OFFSET_MASK 0x00fffffffffffe00ULL -#define L2E_COMPRESSED_OFFSET_SIZE_MASK 0x3fffffffffffffffULL #define REFT_OFFSET_MASK 0xfffffffffffffe00ULL diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index c1c43a891b..3d53657c26 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -556,8 +556,7 @@ static int coroutine_fn do_perform_cow_write(BlockDriverState *bs, * offset needs to be aligned to a cluster boundary. * * If the cluster is unallocated then *host_offset will be 0. - * If the cluster is compressed then *host_offset will contain the - * complete compressed cluster descriptor. + * If the cluster is compressed then *host_offset will contain the l2 entry. * * On entry, *bytes is the maximum number of contiguous bytes starting at * offset that we are interested in. @@ -660,7 +659,7 @@ int qcow2_get_host_offset(BlockDriverState *bs, uint64_t offset, ret = -EIO; goto fail; } - *host_offset = l2_entry & L2E_COMPRESSED_OFFSET_SIZE_MASK; + *host_offset = l2_entry; break; case QCOW2_SUBCLUSTER_ZERO_PLAIN: case QCOW2_SUBCLUSTER_UNALLOCATED_PLAIN: diff --git a/block/qcow2.c b/block/qcow2.c index 9f1b6461c8..e5d8ab679e 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -74,7 +74,7 @@ typedef struct { static int coroutine_fn qcow2_co_preadv_compressed(BlockDriverState *bs, - uint64_t cluster_descriptor, + uint64_t l2_entry, uint64_t offset, uint64_t bytes, QEMUIOVector *qiov, @@ -2205,7 +2205,7 @@ typedef struct Qcow2AioTask { BlockDriverState *bs; QCow2SubclusterType subcluster_type; /* only for read */ - uint64_t host_offset; /* or full descriptor in compressed clusters */ + uint64_t host_offset; /* or l2_entry for compressed read */ uint64_t offset; uint64_t bytes; QEMUIOVector *qiov; @@ -4693,7 +4693,7 @@ qcow2_co_pwritev_compressed_part(BlockDriverState *bs, static int coroutine_fn qcow2_co_preadv_compressed(BlockDriverState *bs, - uint64_t cluster_descriptor, + uint64_t l2_entry, uint64_t offset, uint64_t bytes, QEMUIOVector *qiov, @@ -4705,8 +4705,10 @@ qcow2_co_preadv_compressed(BlockDriverState *bs, uint8_t *buf, *out_buf; int offset_in_cluster = offset_into_cluster(s, offset); - coffset = cluster_descriptor & s->cluster_offset_mask; - nb_csectors = ((cluster_descriptor >> s->csize_shift) & s->csize_mask) + 1; + assert(qcow2_get_cluster_type(bs, l2_entry) == QCOW2_CLUSTER_COMPRESSED); + + coffset = l2_entry & s->cluster_offset_mask; + nb_csectors = ((l2_entry >> s->csize_shift) & s->csize_mask) + 1; csize = nb_csectors * QCOW2_COMPRESSED_SECTOR_SIZE - (coffset & ~QCOW2_COMPRESSED_SECTOR_MASK); From patchwork Wed Sep 15 17:53:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497163 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4AC7C433EF for ; Wed, 15 Sep 2021 18:24:34 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 72EE360EFF for ; Wed, 15 Sep 2021 18:24:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 72EE360EFF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:35802 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZaD-0007xl-Lj for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:24:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52252) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6x-0007RB-S2 for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:20 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:27634) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6v-00078B-0o for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728456; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Dq0zmom0FvUhfkvzz53bwNPvankO7VvUk2WY/lGZZgA=; b=Cy7amBpad9XzFvIMQvih4Z/jKCxc6+HjiwbDrQNDXiOMpU7tHe7jSs8uvfdtWQehr72Pb7 SrTyQ5QZ5hZWsR5KHjRA4y950bAdx4hrxENHjBTlrmcFUk1NqXURyOZ0ABIfJFP9Zrb8L1 z4CmxuZRDOsmsDoZa5N6ytPJ+ezximk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-592-aBu1VH21ODezo87jjuUkvA-1; Wed, 15 Sep 2021 13:54:13 -0400 X-MC-Unique: aBu1VH21ODezo87jjuUkvA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 14CDF1966320; Wed, 15 Sep 2021 17:54:12 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8564F5C1C5; Wed, 15 Sep 2021 17:54:11 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 24/32] qcow2: introduce qcow2_parse_compressed_l2_entry() helper Date: Wed, 15 Sep 2021 19:53:10 +0200 Message-Id: <20210915175318.853225-25-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Add helper to parse compressed l2_entry and use it everywhere instead of open-coding. Note, that in most places we move to precise coffset/csize instead of sector-aligned. Still it should work good enough for updating refcounts. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Reviewed-by: Hanna Reitz Message-Id: <20210914122454.141075-4-vsementsov@virtuozzo.com> Signed-off-by: Hanna Reitz --- block/qcow2.h | 3 ++- block/qcow2-cluster.c | 15 +++++++++++++++ block/qcow2-refcount.c | 36 +++++++++++++++++------------------- block/qcow2.c | 9 ++------- 4 files changed, 36 insertions(+), 27 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index 42a0058ab7..c0e1e83796 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -110,7 +110,6 @@ /* Defined in the qcow2 spec (compressed cluster descriptor) */ #define QCOW2_COMPRESSED_SECTOR_SIZE 512U -#define QCOW2_COMPRESSED_SECTOR_MASK (~(QCOW2_COMPRESSED_SECTOR_SIZE - 1ULL)) /* Must be at least 2 to cover COW */ #define MIN_L2_CACHE_SIZE 2 /* cache entries */ @@ -913,6 +912,8 @@ int qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs, uint64_t offset, int compressed_size, uint64_t *host_offset); +void qcow2_parse_compressed_l2_entry(BlockDriverState *bs, uint64_t l2_entry, + uint64_t *coffset, int *csize); int qcow2_alloc_cluster_link_l2(BlockDriverState *bs, QCowL2Meta *m); void qcow2_alloc_cluster_abort(BlockDriverState *bs, QCowL2Meta *m); diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 3d53657c26..4ebb49a087 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -2480,3 +2480,18 @@ fail: g_free(l1_table); return ret; } + +void qcow2_parse_compressed_l2_entry(BlockDriverState *bs, uint64_t l2_entry, + uint64_t *coffset, int *csize) +{ + BDRVQcow2State *s = bs->opaque; + int nb_csectors; + + assert(qcow2_get_cluster_type(bs, l2_entry) == QCOW2_CLUSTER_COMPRESSED); + + *coffset = l2_entry & s->cluster_offset_mask; + + nb_csectors = ((l2_entry >> s->csize_shift) & s->csize_mask) + 1; + *csize = nb_csectors * QCOW2_COMPRESSED_SECTOR_SIZE - + (*coffset & (QCOW2_COMPRESSED_SECTOR_SIZE - 1)); +} diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 2734338625..66cbb94ef9 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -1177,11 +1177,11 @@ void qcow2_free_any_cluster(BlockDriverState *bs, uint64_t l2_entry, switch (ctype) { case QCOW2_CLUSTER_COMPRESSED: { - int64_t offset = (l2_entry & s->cluster_offset_mask) - & QCOW2_COMPRESSED_SECTOR_MASK; - int size = QCOW2_COMPRESSED_SECTOR_SIZE * - (((l2_entry >> s->csize_shift) & s->csize_mask) + 1); - qcow2_free_clusters(bs, offset, size, type); + uint64_t coffset; + int csize; + + qcow2_parse_compressed_l2_entry(bs, l2_entry, &coffset, &csize); + qcow2_free_clusters(bs, coffset, csize, type); } break; case QCOW2_CLUSTER_NORMAL: @@ -1247,7 +1247,7 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs, bool l1_allocated = false; int64_t old_entry, old_l2_offset; unsigned slice, slice_size2, n_slices; - int i, j, l1_modified = 0, nb_csectors; + int i, j, l1_modified = 0; int ret; assert(addend >= -1 && addend <= 1); @@ -1318,14 +1318,14 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs, switch (qcow2_get_cluster_type(bs, entry)) { case QCOW2_CLUSTER_COMPRESSED: - nb_csectors = ((entry >> s->csize_shift) & - s->csize_mask) + 1; if (addend != 0) { - uint64_t coffset = (entry & s->cluster_offset_mask) - & QCOW2_COMPRESSED_SECTOR_MASK; + uint64_t coffset; + int csize; + + qcow2_parse_compressed_l2_entry(bs, entry, + &coffset, &csize); ret = update_refcount( - bs, coffset, - nb_csectors * QCOW2_COMPRESSED_SECTOR_SIZE, + bs, coffset, csize, abs(addend), addend < 0, QCOW2_DISCARD_SNAPSHOT); if (ret < 0) { @@ -1603,7 +1603,7 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, BDRVQcow2State *s = bs->opaque; uint64_t l2_entry; uint64_t next_contiguous_offset = 0; - int i, nb_csectors, ret; + int i, ret; size_t l2_size_bytes = s->l2_size * l2_entry_size(s); g_autofree uint64_t *l2_table = g_malloc(l2_size_bytes); @@ -1617,6 +1617,8 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, /* Do the actual checks */ for (i = 0; i < s->l2_size; i++) { + uint64_t coffset; + int csize; l2_entry = get_l2_entry(s, l2_table, i); switch (qcow2_get_cluster_type(bs, l2_entry)) { @@ -1638,13 +1640,9 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, } /* Mark cluster as used */ - nb_csectors = ((l2_entry >> s->csize_shift) & - s->csize_mask) + 1; - l2_entry &= s->cluster_offset_mask; + qcow2_parse_compressed_l2_entry(bs, l2_entry, &coffset, &csize); ret = qcow2_inc_refcounts_imrt( - bs, res, refcount_table, refcount_table_size, - l2_entry & QCOW2_COMPRESSED_SECTOR_MASK, - nb_csectors * QCOW2_COMPRESSED_SECTOR_SIZE); + bs, res, refcount_table, refcount_table_size, coffset, csize); if (ret < 0) { return ret; } diff --git a/block/qcow2.c b/block/qcow2.c index e5d8ab679e..02f9f3e636 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -4700,17 +4700,12 @@ qcow2_co_preadv_compressed(BlockDriverState *bs, size_t qiov_offset) { BDRVQcow2State *s = bs->opaque; - int ret = 0, csize, nb_csectors; + int ret = 0, csize; uint64_t coffset; uint8_t *buf, *out_buf; int offset_in_cluster = offset_into_cluster(s, offset); - assert(qcow2_get_cluster_type(bs, l2_entry) == QCOW2_CLUSTER_COMPRESSED); - - coffset = l2_entry & s->cluster_offset_mask; - nb_csectors = ((l2_entry >> s->csize_shift) & s->csize_mask) + 1; - csize = nb_csectors * QCOW2_COMPRESSED_SECTOR_SIZE - - (coffset & ~QCOW2_COMPRESSED_SECTOR_MASK); + qcow2_parse_compressed_l2_entry(bs, l2_entry, &coffset, &csize); buf = g_try_malloc(csize); if (!buf) { From patchwork Wed Sep 15 17:53:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497139 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9427C433EF for ; Wed, 15 Sep 2021 18:22:50 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4F42B60EFF for ; Wed, 15 Sep 2021 18:22:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4F42B60EFF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:33024 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZYX-00066i-2A for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:22:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52254) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6y-0007Rg-ME for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:20 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:21150) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6v-00078b-Eo for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728456; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p10qoiSK5/pk5iPahEiY2qK+VuyB0VLu/CcUgo2jSOc=; b=dn4x69+NW0ZUWPlMJcLYLRbzZcvyxyRBdJ2pCHkQLa+9Pus6T7+nq3S+rNT4yyC38XghEO McpPSZmy1wLbJtLRIg5zIXfz1vHRmHmtQAZJ890/KSwkH5CmCwCychPRY8w23GN/kfXZUc kvs0qKPfI1BlOyg41VcJix3ffGYYH/Y= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-381-C3KZvZg1OsSCFqCu5tqXsw-1; Wed, 15 Sep 2021 13:54:15 -0400 X-MC-Unique: C3KZvZg1OsSCFqCu5tqXsw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 269141084685; Wed, 15 Sep 2021 17:54:14 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id BC0B610016FB; Wed, 15 Sep 2021 17:54:13 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 25/32] qcow2-refcount: introduce fix_l2_entry_by_zero() Date: Wed, 15 Sep 2021 19:53:11 +0200 Message-Id: <20210915175318.853225-26-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Split fix_l2_entry_by_zero() out of check_refcounts_l2() to be reused in further patch. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Reviewed-by: Hanna Reitz Message-Id: <20210914122454.141075-5-vsementsov@virtuozzo.com> Signed-off-by: Hanna Reitz --- block/qcow2-refcount.c | 87 +++++++++++++++++++++++++++++------------- 1 file changed, 60 insertions(+), 27 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 66cbb94ef9..184b96ad63 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -1587,6 +1587,54 @@ enum { CHECK_FRAG_INFO = 0x2, /* update BlockFragInfo counters */ }; +/* + * Fix L2 entry by making it QCOW2_CLUSTER_ZERO_PLAIN. + * + * This function decrements res->corruptions on success, so the caller is + * responsible to increment res->corruptions prior to the call. + * + * On failure in-memory @l2_table may be modified. + */ +static int fix_l2_entry_by_zero(BlockDriverState *bs, BdrvCheckResult *res, + uint64_t l2_offset, + uint64_t *l2_table, int l2_index, bool active, + bool *metadata_overlap) +{ + BDRVQcow2State *s = bs->opaque; + int ret; + int idx = l2_index * (l2_entry_size(s) / sizeof(uint64_t)); + uint64_t l2e_offset = l2_offset + (uint64_t)l2_index * l2_entry_size(s); + int ign = active ? QCOW2_OL_ACTIVE_L2 : QCOW2_OL_INACTIVE_L2; + uint64_t l2_entry = has_subclusters(s) ? 0 : QCOW_OFLAG_ZERO; + + set_l2_entry(s, l2_table, l2_index, l2_entry); + ret = qcow2_pre_write_overlap_check(bs, ign, l2e_offset, l2_entry_size(s), + false); + if (metadata_overlap) { + *metadata_overlap = ret < 0; + } + if (ret < 0) { + fprintf(stderr, "ERROR: Overlap check failed\n"); + goto fail; + } + + ret = bdrv_pwrite_sync(bs->file, l2e_offset, &l2_table[idx], + l2_entry_size(s)); + if (ret < 0) { + fprintf(stderr, "ERROR: Failed to overwrite L2 " + "table entry: %s\n", strerror(-ret)); + goto fail; + } + + res->corruptions--; + res->corruptions_fixed++; + return 0; + +fail: + res->check_errors++; + return ret; +} + /* * Increases the refcount in the given refcount table for the all clusters * referenced in the L2 table. While doing so, performs some checks on L2 @@ -1606,6 +1654,7 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, int i, ret; size_t l2_size_bytes = s->l2_size * l2_entry_size(s); g_autofree uint64_t *l2_table = g_malloc(l2_size_bytes); + bool metadata_overlap; /* Read L2 table from disk */ ret = bdrv_pread(bs->file, l2_offset, l2_table, l2_size_bytes); @@ -1685,19 +1734,10 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR", offset); if (fix & BDRV_FIX_ERRORS) { - int idx = i * (l2_entry_size(s) / sizeof(uint64_t)); - uint64_t l2e_offset = - l2_offset + (uint64_t)i * l2_entry_size(s); - int ign = active ? QCOW2_OL_ACTIVE_L2 : - QCOW2_OL_INACTIVE_L2; - - l2_entry = has_subclusters(s) ? 0 : QCOW_OFLAG_ZERO; - set_l2_entry(s, l2_table, i, l2_entry); - ret = qcow2_pre_write_overlap_check(bs, ign, - l2e_offset, l2_entry_size(s), false); - if (ret < 0) { - fprintf(stderr, "ERROR: Overlap check failed\n"); - res->check_errors++; + ret = fix_l2_entry_by_zero(bs, res, l2_offset, + l2_table, i, active, + &metadata_overlap); + if (metadata_overlap) { /* * Something is seriously wrong, so abort checking * this L2 table. @@ -1705,26 +1745,19 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, return ret; } - ret = bdrv_pwrite_sync(bs->file, l2e_offset, - &l2_table[idx], - l2_entry_size(s)); - if (ret < 0) { - fprintf(stderr, "ERROR: Failed to overwrite L2 " - "table entry: %s\n", strerror(-ret)); - res->check_errors++; - /* - * Do not abort, continue checking the rest of this - * L2 table's entries. - */ - } else { - res->corruptions--; - res->corruptions_fixed++; + if (ret == 0) { /* * Skip marking the cluster as used * (it is unused now). */ continue; } + + /* + * Failed to fix. + * Do not abort, continue checking the rest of this + * L2 table's entries. + */ } } else { fprintf(stderr, "ERROR offset=%" PRIx64 ": Data cluster is " From patchwork Wed Sep 15 17:53:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B93BC433F5 for ; Wed, 15 Sep 2021 18:28:36 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E5299610A4 for ; Wed, 15 Sep 2021 18:28:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E5299610A4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:44370 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZe7-0005JQ-5Q for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:28:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52276) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ70-0007XG-R0 for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:22 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:38057) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ6y-0007BK-Qr for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728460; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Wk6xFgIjoymCAZ9mM2AceqT6xK7ZGXGDRVC8mDtLZog=; b=QiwajBLReImsWo2IXRSokcnyL0zwALIYIpJbJVxZJu3zVM6e10Dy5cwmu3VqEh+iyE8hUO YWEcMVqyKMsMhNOhqnTL5Ly2RSxHSix3TgWLrW9QHLNE+r3OCxwmORu6hBn2Q6Y0FD8Ter EqkKNS/KwxeGS3FdRVeVeno6I960l84= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-123-X1RZ2bNAP4SPehSsDL6Khw-1; Wed, 15 Sep 2021 13:54:17 -0400 X-MC-Unique: X1RZ2bNAP4SPehSsDL6Khw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 42EDD1006ACA; Wed, 15 Sep 2021 17:54:16 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DB6B019736; Wed, 15 Sep 2021 17:54:15 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 26/32] qcow2-refcount: fix_l2_entry_by_zero(): also zero L2 entry bitmap Date: Wed, 15 Sep 2021 19:53:12 +0200 Message-Id: <20210915175318.853225-27-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy We'll reuse the function to fix wrong L2 entry bitmap. Support it now. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Reviewed-by: Hanna Reitz Message-Id: <20210914122454.141075-6-vsementsov@virtuozzo.com> Signed-off-by: Hanna Reitz --- block/qcow2-refcount.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 184b96ad63..f48c5e1b5d 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -1588,7 +1588,8 @@ enum { }; /* - * Fix L2 entry by making it QCOW2_CLUSTER_ZERO_PLAIN. + * Fix L2 entry by making it QCOW2_CLUSTER_ZERO_PLAIN (or making all its present + * subclusters QCOW2_SUBCLUSTER_ZERO_PLAIN). * * This function decrements res->corruptions on success, so the caller is * responsible to increment res->corruptions prior to the call. @@ -1605,9 +1606,20 @@ static int fix_l2_entry_by_zero(BlockDriverState *bs, BdrvCheckResult *res, int idx = l2_index * (l2_entry_size(s) / sizeof(uint64_t)); uint64_t l2e_offset = l2_offset + (uint64_t)l2_index * l2_entry_size(s); int ign = active ? QCOW2_OL_ACTIVE_L2 : QCOW2_OL_INACTIVE_L2; - uint64_t l2_entry = has_subclusters(s) ? 0 : QCOW_OFLAG_ZERO; - set_l2_entry(s, l2_table, l2_index, l2_entry); + if (has_subclusters(s)) { + uint64_t l2_bitmap = get_l2_bitmap(s, l2_table, l2_index); + + /* Allocated subclusters become zero */ + l2_bitmap |= l2_bitmap << 32; + l2_bitmap &= QCOW_L2_BITMAP_ALL_ZEROES; + + set_l2_bitmap(s, l2_table, l2_index, l2_bitmap); + set_l2_entry(s, l2_table, l2_index, 0); + } else { + set_l2_entry(s, l2_table, l2_index, QCOW_OFLAG_ZERO); + } + ret = qcow2_pre_write_overlap_check(bs, ign, l2e_offset, l2_entry_size(s), false); if (metadata_overlap) { From patchwork Wed Sep 15 17:53:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497119 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F04FDC433EF for ; Wed, 15 Sep 2021 18:17:22 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9835B611C4 for ; Wed, 15 Sep 2021 18:17:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9835B611C4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:47044 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZTF-0004pb-QW for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:17:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52312) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ72-0007c7-K2 for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:25 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:27699) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ70-0007DF-Qh for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728462; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qlZ6RTQTKYnjfEbjtw6/W5ZOlmU2l9EmeF+xWRIr0vk=; b=IL+75nto3Ti+mbjwZBepStNFyPFrpp7ogfulfxksWOUjnn8ABH3eBg4/vVwq1EInE0TI+P fNx2hCy9DR8DPw2LcrdEw+plXPL+n4zOc65PCkJ4KprYpGiYYKv1PBTMmtEC9NPvFIQv7W u30nQ0kSVtISZuepHQkLHt0MIIHmTFM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-268-_xPs65RHPWasscFsiU8BlQ-1; Wed, 15 Sep 2021 13:54:19 -0400 X-MC-Unique: _xPs65RHPWasscFsiU8BlQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 53D40801B3D; Wed, 15 Sep 2021 17:54:18 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EE3C37A8CA; Wed, 15 Sep 2021 17:54:17 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 27/32] qcow2-refcount: check_refcounts_l2(): check l2_bitmap Date: Wed, 15 Sep 2021 19:53:13 +0200 Message-Id: <20210915175318.853225-28-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Check subcluster bitmap of the l2 entry for different types of clusters: - for compressed it must be zero - for allocated check consistency of two parts of the bitmap - for unallocated all subclusters should be unallocated (or zero-plain) Signed-off-by: Vladimir Sementsov-Ogievskiy Tested-by: Kirill Tkhai Message-Id: <20210914122454.141075-7-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake Reviewed-by: Hanna Reitz Signed-off-by: Hanna Reitz --- block/qcow2-refcount.c | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index f48c5e1b5d..9a5ae3cac4 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -1661,7 +1661,7 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, int flags, BdrvCheckMode fix, bool active) { BDRVQcow2State *s = bs->opaque; - uint64_t l2_entry; + uint64_t l2_entry, l2_bitmap; uint64_t next_contiguous_offset = 0; int i, ret; size_t l2_size_bytes = s->l2_size * l2_entry_size(s); @@ -1681,6 +1681,7 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, uint64_t coffset; int csize; l2_entry = get_l2_entry(s, l2_table, i); + l2_bitmap = get_l2_bitmap(s, l2_table, i); switch (qcow2_get_cluster_type(bs, l2_entry)) { case QCOW2_CLUSTER_COMPRESSED: @@ -1700,6 +1701,14 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, break; } + if (l2_bitmap) { + fprintf(stderr, "ERROR compressed cluster %d with non-zero " + "subcluster allocation bitmap, entry=0x%" PRIx64 "\n", + i, l2_entry); + res->corruptions++; + break; + } + /* Mark cluster as used */ qcow2_parse_compressed_l2_entry(bs, l2_entry, &coffset, &csize); ret = qcow2_inc_refcounts_imrt( @@ -1727,13 +1736,19 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, { uint64_t offset = l2_entry & L2E_OFFSET_MASK; + if ((l2_bitmap >> 32) & l2_bitmap) { + res->corruptions++; + fprintf(stderr, "ERROR offset=%" PRIx64 ": Allocated " + "cluster has corrupted subcluster allocation bitmap\n", + offset); + } + /* Correct offsets are cluster aligned */ if (offset_into_cluster(s, offset)) { bool contains_data; res->corruptions++; if (has_subclusters(s)) { - uint64_t l2_bitmap = get_l2_bitmap(s, l2_table, i); contains_data = (l2_bitmap & QCOW_L2_BITMAP_ALL_ALLOC); } else { contains_data = !(l2_entry & QCOW_OFLAG_ZERO); @@ -1799,7 +1814,16 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, } case QCOW2_CLUSTER_ZERO_PLAIN: + /* Impossible when image has subclusters */ + assert(!l2_bitmap); + break; + case QCOW2_CLUSTER_UNALLOCATED: + if (l2_bitmap & QCOW_L2_BITMAP_ALL_ALLOC) { + res->corruptions++; + fprintf(stderr, "ERROR: Unallocated " + "cluster has non-zero subcluster allocation map\n"); + } break; default: From patchwork Wed Sep 15 17:53:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D92F8C433EF for ; Wed, 15 Sep 2021 18:32:53 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9476160F38 for ; Wed, 15 Sep 2021 18:32:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9476160F38 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:53126 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZiG-0002pI-LJ for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:32:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52338) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ75-0007g4-33 for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:27 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:20377) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ71-0007E5-Nc for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728463; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OQEYhPE9nXPUBTQOUn4zz3K5hWB46g2ANnP4UKtpmOE=; b=JevSxG/v74mUiAzBW/UkTofNRiBXdEPzwoFKoxhHcsICgM80a7seB5A4YeRbmWsB5kpJNK GORi1g2HfhdmrgJpNUNy8xpZ1yQJSmvcOYogBd7aCUcFNgtz4cjxXdEFLqToMaLJMWV5V1 lRytzJdun/LYOYG5skAdaPZCjeJRdVs= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-203-KhLD0gXNMnGpFmZQyJGyfw-1; Wed, 15 Sep 2021 13:54:21 -0400 X-MC-Unique: KhLD0gXNMnGpFmZQyJGyfw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7D96B80124F; Wed, 15 Sep 2021 17:54:20 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2437460583; Wed, 15 Sep 2021 17:54:19 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 28/32] qcow2-refcount: check_refcounts_l2(): check reserved bits Date: Wed, 15 Sep 2021 19:53:14 +0200 Message-Id: <20210915175318.853225-29-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Tested-by: Kirill Tkhai Reviewed-by: Hanna Reitz Message-Id: <20210914122454.141075-8-vsementsov@virtuozzo.com> [hreitz: Separated `type` declaration from statements] Signed-off-by: Hanna Reitz --- block/qcow2.h | 1 + block/qcow2-refcount.c | 14 +++++++++++++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/block/qcow2.h b/block/qcow2.h index c0e1e83796..b8b1093b61 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -587,6 +587,7 @@ typedef enum QCow2MetadataOverlap { #define L1E_OFFSET_MASK 0x00fffffffffffe00ULL #define L2E_OFFSET_MASK 0x00fffffffffffe00ULL +#define L2E_STD_RESERVED_MASK 0x3f000000000001feULL #define REFT_OFFSET_MASK 0xfffffffffffffe00ULL diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 9a5ae3cac4..bdac7b1780 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -1680,10 +1680,22 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, for (i = 0; i < s->l2_size; i++) { uint64_t coffset; int csize; + QCow2ClusterType type; + l2_entry = get_l2_entry(s, l2_table, i); l2_bitmap = get_l2_bitmap(s, l2_table, i); + type = qcow2_get_cluster_type(bs, l2_entry); + + if (type != QCOW2_CLUSTER_COMPRESSED) { + /* Check reserved bits of Standard Cluster Descriptor */ + if (l2_entry & L2E_STD_RESERVED_MASK) { + fprintf(stderr, "ERROR found l2 entry with reserved bits set: " + "%" PRIx64 "\n", l2_entry); + res->corruptions++; + } + } - switch (qcow2_get_cluster_type(bs, l2_entry)) { + switch (type) { case QCOW2_CLUSTER_COMPRESSED: /* Compressed clusters don't have QCOW_OFLAG_COPIED */ if (l2_entry & QCOW_OFLAG_COPIED) { From patchwork Wed Sep 15 17:53:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F41F1C433EF for ; Wed, 15 Sep 2021 18:28:05 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7219061056 for ; Wed, 15 Sep 2021 18:28:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7219061056 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:43250 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZdc-0004aK-Kr for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:28:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52382) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ77-0007pI-Np for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:29 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:59307) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ75-0007HM-T4 for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728467; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BZQPTEYCOY/RoJosPIYDTQ+iWISPbSjxflDa2pjzPDU=; b=Ra0DHcBolgCVJitOLqd9a6TOzypW6fM9HYKfzkUz4VHgBp127poURescIK1H2BgMCCdRGt B+Pi5CKq0s3xg/aioWJfqsTJ02OpNH9B5DhZvLUVPzsvi7Spw/FV0T5DN6V/BlTF+kktI9 W41EVnqcU5Bi6d6En+iTyMPYojKweu4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-121-bEWG2NzAOqqA6cJg5DKalA-1; Wed, 15 Sep 2021 13:54:23 -0400 X-MC-Unique: bEWG2NzAOqqA6cJg5DKalA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 962BD8064A5; Wed, 15 Sep 2021 17:54:22 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3AC9110074FD; Wed, 15 Sep 2021 17:54:22 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 29/32] qcow2-refcount: improve style of check_refcounts_l1() Date: Wed, 15 Sep 2021 19:53:15 +0200 Message-Id: <20210915175318.853225-30-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy - use g_autofree for l1_table - better name for size in bytes variable - reduce code blocks nesting - whitespaces, braces, newlines Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz Message-Id: <20210914122454.141075-9-vsementsov@virtuozzo.com> Signed-off-by: Hanna Reitz --- block/qcow2-refcount.c | 98 +++++++++++++++++++++--------------------- 1 file changed, 50 insertions(+), 48 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index bdac7b1780..3c89e09599 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -1862,71 +1862,73 @@ static int check_refcounts_l1(BlockDriverState *bs, int flags, BdrvCheckMode fix, bool active) { BDRVQcow2State *s = bs->opaque; - uint64_t *l1_table = NULL, l2_offset, l1_size2; + size_t l1_size_bytes = l1_size * L1E_SIZE; + g_autofree uint64_t *l1_table = NULL; + uint64_t l2_offset; int i, ret; - l1_size2 = l1_size * L1E_SIZE; + if (!l1_size) { + return 0; + } /* Mark L1 table as used */ ret = qcow2_inc_refcounts_imrt(bs, res, refcount_table, refcount_table_size, - l1_table_offset, l1_size2); + l1_table_offset, l1_size_bytes); if (ret < 0) { - goto fail; + return ret; + } + + l1_table = g_try_malloc(l1_size_bytes); + if (l1_table == NULL) { + res->check_errors++; + return -ENOMEM; } /* Read L1 table entries from disk */ - if (l1_size2 > 0) { - l1_table = g_try_malloc(l1_size2); - if (l1_table == NULL) { - ret = -ENOMEM; - res->check_errors++; - goto fail; - } - ret = bdrv_pread(bs->file, l1_table_offset, l1_table, l1_size2); - if (ret < 0) { - fprintf(stderr, "ERROR: I/O error in check_refcounts_l1\n"); - res->check_errors++; - goto fail; - } - for(i = 0;i < l1_size; i++) - be64_to_cpus(&l1_table[i]); + ret = bdrv_pread(bs->file, l1_table_offset, l1_table, l1_size_bytes); + if (ret < 0) { + fprintf(stderr, "ERROR: I/O error in check_refcounts_l1\n"); + res->check_errors++; + return ret; + } + + for (i = 0; i < l1_size; i++) { + be64_to_cpus(&l1_table[i]); } /* Do the actual checks */ - for(i = 0; i < l1_size; i++) { - l2_offset = l1_table[i]; - if (l2_offset) { - /* Mark L2 table as used */ - l2_offset &= L1E_OFFSET_MASK; - ret = qcow2_inc_refcounts_imrt(bs, res, - refcount_table, refcount_table_size, - l2_offset, s->cluster_size); - if (ret < 0) { - goto fail; - } + for (i = 0; i < l1_size; i++) { + if (!l1_table[i]) { + continue; + } - /* L2 tables are cluster aligned */ - if (offset_into_cluster(s, l2_offset)) { - fprintf(stderr, "ERROR l2_offset=%" PRIx64 ": Table is not " - "cluster aligned; L1 entry corrupted\n", l2_offset); - res->corruptions++; - } + l2_offset = l1_table[i] & L1E_OFFSET_MASK; - /* Process and check L2 entries */ - ret = check_refcounts_l2(bs, res, refcount_table, - refcount_table_size, l2_offset, flags, - fix, active); - if (ret < 0) { - goto fail; - } + /* Mark L2 table as used */ + ret = qcow2_inc_refcounts_imrt(bs, res, + refcount_table, refcount_table_size, + l2_offset, s->cluster_size); + if (ret < 0) { + return ret; + } + + /* L2 tables are cluster aligned */ + if (offset_into_cluster(s, l2_offset)) { + fprintf(stderr, "ERROR l2_offset=%" PRIx64 ": Table is not " + "cluster aligned; L1 entry corrupted\n", l2_offset); + res->corruptions++; + } + + /* Process and check L2 entries */ + ret = check_refcounts_l2(bs, res, refcount_table, + refcount_table_size, l2_offset, flags, + fix, active); + if (ret < 0) { + return ret; } } - g_free(l1_table); - return 0; -fail: - g_free(l1_table); - return ret; + return 0; } /* From patchwork Wed Sep 15 17:53:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E3D5C433EF for ; Wed, 15 Sep 2021 18:37:18 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 36A9861157 for ; Wed, 15 Sep 2021 18:37:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 36A9861157 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:34806 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZmX-0000yK-BM for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:37:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52380) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ77-0007oB-DN for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:29 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:59861) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ75-0007HI-K7 for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728467; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Jq24j4hmzhxp6qwU39w7PkxHqqQF+fATDblPaviuxOQ=; b=NvmjqMpZogO/ax3MR26p6rZBhCFeGHIgh1nqmOm63MhAVt4VxmfgaOlBjntK1EA7Akp/Gt 6OIbUs6cGQh+zh0/N+E9oLpDnaYcALyrTXEByw52tUy64DsnlXR1BGhS6je7PyvRgoC5oy zelfwcO0OFOEpO4JU0UJlXbjutKlWQo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-88-aQUawDfCPtuC8YNP55L6sA-1; Wed, 15 Sep 2021 13:54:25 -0400 X-MC-Unique: aQUawDfCPtuC8YNP55L6sA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A846A1966327; Wed, 15 Sep 2021 17:54:24 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4A7C83AC0; Wed, 15 Sep 2021 17:54:24 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 30/32] qcow2-refcount: check_refcounts_l1(): check reserved bits Date: Wed, 15 Sep 2021 19:53:16 +0200 Message-Id: <20210915175318.853225-31-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Tested-by: Kirill Tkhai Reviewed-by: Hanna Reitz Message-Id: <20210914122454.141075-10-vsementsov@virtuozzo.com> Signed-off-by: Hanna Reitz --- block/qcow2.h | 1 + block/qcow2-refcount.c | 6 ++++++ 2 files changed, 7 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h index b8b1093b61..58fd7f1678 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -586,6 +586,7 @@ typedef enum QCow2MetadataOverlap { (QCOW2_OL_CACHED | QCOW2_OL_INACTIVE_L2) #define L1E_OFFSET_MASK 0x00fffffffffffe00ULL +#define L1E_RESERVED_MASK 0x7f000000000001ffULL #define L2E_OFFSET_MASK 0x00fffffffffffe00ULL #define L2E_STD_RESERVED_MASK 0x3f000000000001feULL diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 3c89e09599..1c246b9227 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -1902,6 +1902,12 @@ static int check_refcounts_l1(BlockDriverState *bs, continue; } + if (l1_table[i] & L1E_RESERVED_MASK) { + fprintf(stderr, "ERROR found L1 entry with reserved bits set: " + "%" PRIx64 "\n", l1_table[i]); + res->corruptions++; + } + l2_offset = l1_table[i] & L1E_OFFSET_MASK; /* Mark L2 table as used */ From patchwork Wed Sep 15 17:53:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34FA6C433EF for ; Wed, 15 Sep 2021 18:21:23 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EC59260F26 for ; Wed, 15 Sep 2021 18:21:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org EC59260F26 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:56100 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZX7-0002de-WA for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:21:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52406) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ79-0007wT-Hj for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:31 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:51318) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ77-0007JE-Qt for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728469; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C56eKEevVNFSB3qbkBOrLbpsNKhnoUq4V0uA9Yj62eI=; b=KomaMfgs88GBBgGjVE5jQpk40Pv7seHxj3ScO9I2ez4a2xxbZilS6ZfzRy1FVT1QnhnZ3x Pa+Qn3Mg6h4JfQjqdQDNEK7B4miu6UIkDJXk1TcZBZb7r9Zp7pma8Q2BCfRLrxfAAYErb1 0zAeJw4FHQ6EgoMLdQ97ZSWOPZXZmUk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-291-W-MZ_lJSPVu8D3pH5LzC_A-1; Wed, 15 Sep 2021 13:54:27 -0400 X-MC-Unique: W-MZ_lJSPVu8D3pH5LzC_A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C733A1084684; Wed, 15 Sep 2021 17:54:26 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 727185C1C5; Wed, 15 Sep 2021 17:54:26 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 31/32] qcow2-refcount: check_refblocks(): add separate message for reserved Date: Wed, 15 Sep 2021 19:53:17 +0200 Message-Id: <20210915175318.853225-32-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Split checking for reserved bits out of aligned offset check. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Tested-by: Kirill Tkhai Reviewed-by: Hanna Reitz Message-Id: <20210914122454.141075-11-vsementsov@virtuozzo.com> Signed-off-by: Hanna Reitz --- block/qcow2.h | 1 + block/qcow2-refcount.c | 10 +++++++++- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/block/qcow2.h b/block/qcow2.h index 58fd7f1678..fd48a89d45 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -591,6 +591,7 @@ typedef enum QCow2MetadataOverlap { #define L2E_STD_RESERVED_MASK 0x3f000000000001feULL #define REFT_OFFSET_MASK 0xfffffffffffffe00ULL +#define REFT_RESERVED_MASK 0x1ffULL #define INV_OFFSET (-1ULL) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 1c246b9227..4614572252 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -2089,9 +2089,17 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, for(i = 0; i < s->refcount_table_size; i++) { uint64_t offset, cluster; - offset = s->refcount_table[i]; + offset = s->refcount_table[i] & REFT_OFFSET_MASK; cluster = offset >> s->cluster_bits; + if (s->refcount_table[i] & REFT_RESERVED_MASK) { + fprintf(stderr, "ERROR refcount table entry %" PRId64 " has " + "reserved bits set\n", i); + res->corruptions++; + *rebuild = true; + continue; + } + /* Refcount blocks are cluster aligned */ if (offset_into_cluster(s, offset)) { fprintf(stderr, "ERROR refcount block %" PRId64 " is not " From patchwork Wed Sep 15 17:53:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 12497211 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 277F7C433F5 for ; Wed, 15 Sep 2021 18:41:22 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 99D24610E8 for ; Wed, 15 Sep 2021 18:41:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 99D24610E8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:44102 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQZqS-0007Cm-NW for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 14:41:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52442) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ7C-00085r-5b for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:34 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:44786) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQZ7A-0007Lz-9i for qemu-devel@nongnu.org; Wed, 15 Sep 2021 13:54:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631728471; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cI3nAh5JkMF6I7Xd3q3+a+2VAwYvpUUFHuWx29LH5B0=; b=OVfIbP7qv56REofXQfRbrMvuxbsocd7b3o3VAa+jFc29QSPMQZVsf8Sc3Et4A3/sstgoHl +RT+Wc8alKcFHdax6LmEd0dbiG+lKGg4c8ulwRypb4tmTZcj5uVq9kwAAENOilovH8ZY6w YLeBe0254SJ26xy/JpACiFQStLYrgpc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-100-eOJDYLEZPjOa4Uv1wenwyA-1; Wed, 15 Sep 2021 13:54:30 -0400 X-MC-Unique: eOJDYLEZPjOa4Uv1wenwyA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 09AF0100B3AF; Wed, 15 Sep 2021 17:54:29 +0000 (UTC) Received: from localhost (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9FF1F1B5B7; Wed, 15 Sep 2021 17:54:28 +0000 (UTC) From: Hanna Reitz To: qemu-block@nongnu.org Subject: [PULL 32/32] qemu-img: Add -F shorthand to convert Date: Wed, 15 Sep 2021 19:53:18 +0200 Message-Id: <20210915175318.853225-33-hreitz@redhat.com> In-Reply-To: <20210915175318.853225-1-hreitz@redhat.com> References: <20210915175318.853225-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , Hanna Reitz , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Eric Blake Although we have long supported 'qemu-img convert -o backing_file=foo,backing_fmt=bar', the fact that we have a shortcut -B for backing_file but none for backing_fmt has made it more likely that users accidentally run into: qemu-img: warning: Deprecated use of backing file without explicit backing format when using -B instead of -o. For similarity with other qemu-img commands, such as create and compare, add '-F $fmt' as the shorthand for '-o backing_fmt=$fmt'. Update iotest 122 for coverage of both spellings. Signed-off-by: Eric Blake Message-Id: <20210913131735.1948339-1-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Maxim Levitsky Signed-off-by: Hanna Reitz --- docs/tools/qemu-img.rst | 4 ++-- qemu-img.c | 10 +++++++--- qemu-img-cmds.hx | 2 +- tests/qemu-iotests/122 | 2 +- 4 files changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst index fe6c30d509..d58980aef8 100644 --- a/docs/tools/qemu-img.rst +++ b/docs/tools/qemu-img.rst @@ -415,7 +415,7 @@ Command description: 4 Error on reading data -.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps [--skip-broken-bitmaps]] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-r RATE_LIMIT] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME +.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps [--skip-broken-bitmaps]] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE [-F backing_fmt]] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-r RATE_LIMIT] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME Convert the disk image *FILENAME* or a snapshot *SNAPSHOT_PARAM* to disk image *OUTPUT_FILENAME* using format *OUTPUT_FMT*. It can @@ -439,7 +439,7 @@ Command description: You can use the *BACKING_FILE* option to force the output image to be created as a copy on write image of the specified base image; the *BACKING_FILE* should have the same content as the input's base image, - however the path, image format, etc may differ. + however the path, image format (as given by *BACKING_FMT*), etc may differ. If a relative path name is given, the backing file is looked up relative to the directory containing *OUTPUT_FILENAME*. diff --git a/qemu-img.c b/qemu-img.c index e43a71a794..f036a1d428 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -2183,7 +2183,8 @@ static int img_convert(int argc, char **argv) int c, bs_i, flags, src_flags = BDRV_O_NO_SHARE; const char *fmt = NULL, *out_fmt = NULL, *cache = "unsafe", *src_cache = BDRV_DEFAULT_CACHE, *out_baseimg = NULL, - *out_filename, *out_baseimg_param, *snapshot_name = NULL; + *out_filename, *out_baseimg_param, *snapshot_name = NULL, + *backing_fmt = NULL; BlockDriver *drv = NULL, *proto_drv = NULL; BlockDriverInfo bdi; BlockDriverState *out_bs; @@ -2223,7 +2224,7 @@ static int img_convert(int argc, char **argv) {"skip-broken-bitmaps", no_argument, 0, OPTION_SKIP_BROKEN}, {0, 0, 0, 0} }; - c = getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WUr:", + c = getopt_long(argc, argv, ":hf:O:B:CcF:o:l:S:pt:T:qnm:WUr:", long_options, NULL); if (c == -1) { break; @@ -2253,6 +2254,9 @@ static int img_convert(int argc, char **argv) case 'c': s.compressed = true; break; + case 'F': + backing_fmt = optarg; + break; case 'o': if (accumulate_options(&options, optarg) < 0) { goto fail_getopt; @@ -2521,7 +2525,7 @@ static int img_convert(int argc, char **argv) qemu_opt_set_number(opts, BLOCK_OPT_SIZE, s.total_sectors * BDRV_SECTOR_SIZE, &error_abort); - ret = add_old_style_options(out_fmt, opts, out_baseimg, NULL); + ret = add_old_style_options(out_fmt, opts, out_baseimg, backing_fmt); if (ret < 0) { goto out; } diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx index b3620f29e5..4c4d94ab22 100644 --- a/qemu-img-cmds.hx +++ b/qemu-img-cmds.hx @@ -46,7 +46,7 @@ SRST ERST DEF("convert", img_convert, - "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-r rate_limit] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename") + "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file [-F backing_fmt]] [-o options] [-l snapshot_param] [-S sparse_size] [-r rate_limit] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename") SRST .. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-r RATE_LIMIT] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME ERST diff --git a/tests/qemu-iotests/122 b/tests/qemu-iotests/122 index 5d550ed13e..efb260d822 100755 --- a/tests/qemu-iotests/122 +++ b/tests/qemu-iotests/122 @@ -67,7 +67,7 @@ echo _make_test_img -b "$TEST_IMG".base -F $IMGFMT $QEMU_IO -c "write -P 0 0 3M" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir -$QEMU_IMG convert -O $IMGFMT -B "$TEST_IMG".base -o backing_fmt=$IMGFMT \ +$QEMU_IMG convert -O $IMGFMT -B "$TEST_IMG".base -F $IMGFMT \ "$TEST_IMG" "$TEST_IMG".orig $QEMU_IO -c "read -P 0 0 3M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir $QEMU_IMG convert -O $IMGFMT -c -B "$TEST_IMG".base -o backing_fmt=$IMGFMT \