From patchwork Mon Jul 27 21:58:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nir Soffer X-Patchwork-Id: 11687705 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3BE22138A for ; Mon, 27 Jul 2020 22:00:12 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C4351206D7 for ; Mon, 27 Jul 2020 22:00:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SXJZqhuY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C4351206D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:56624 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k0BAH-0000tu-U1 for patchwork-qemu-devel@patchwork.kernel.org; Mon, 27 Jul 2020 18:00:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47628) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k0B97-0007p6-W1; Mon, 27 Jul 2020 17:58:58 -0400 Received: from mail-wm1-x341.google.com ([2a00:1450:4864:20::341]:33977) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1k0B94-0004Kh-Pq; Mon, 27 Jul 2020 17:58:57 -0400 Received: by mail-wm1-x341.google.com with SMTP id g10so14241786wmc.1; Mon, 27 Jul 2020 14:58:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0SYxtxkPqDhdL0SY95r16v3wmBbeg7zalBIYXgGxdzI=; b=SXJZqhuYQFD6+kitOGX+vVAzArtn0FoQWQ384D1ugzh3mZsrv0yTH0AEIHDeWcvMKD 1dLj7J8RIDE31kf87P+DZpF15RFHp3x1sVGk0b1BBpIrLGxdHNOYwqBwWgVnfzvKLgn1 EIMURh6/QPRQAB/RomQJ3V9ASWxTvU0cAd/wkpTRDJzUv3jt4FdXy3/coME6UBvqu8m2 TiC+9T59ks8kcGsfqqAv/Lvj9ttbTdrE6lige+bqqE3yE6pxY3iJwv6SQ9x1GmKjxCo3 suUPuMX1KY6I+bzkFaP0Okhw7AqQp1EPe3n4yR6A0468mnwqr2JoOJwz4FdFNZHpSoNz 8ezA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0SYxtxkPqDhdL0SY95r16v3wmBbeg7zalBIYXgGxdzI=; b=n5NTEe+F58WJbXeku2DuNZdGhm3UQtEIdI5uZC9eYNNvzTM7+8pGNNShn5XLPCh+uX /0uZa8K1pmP7Uj7hltuYff5rLjcdNW4k+W2nIHu3YMblVAKrnmIxyigNrpEJzDisEGR7 BuIz5hvlbjL2vNCk7kWTfag6F3DXX7/cb6kqvJh78bdGmnC0eniM2ZmgZUhf480tWbKJ qfN6mIMHpmhi+AKS5suxV8RvtCVf/TxBumNuiozx4A5S2PWKkYBa1cfr/UzvAE4muu5l kw23ptXpygjjv4bEgnRwfSEKuV0dUFI7k8QoXaJL9F/w43+3Vo2+YUzo/iC12MwmlLwW cVsA== X-Gm-Message-State: AOAM532hYJjQjncSOcOq3ZZ2ZrDsTDlkTxUBV+XqHU1pCrrpdVxsmiyu lYPX6LHrOfqqiq788qtYT7GVdNqBpVE= X-Google-Smtp-Source: ABdhPJznSiwH/8PIl6exw5a9+QIc+heo6Q8YXHi+CjIAEhAbPJ0S2ZKiCqzBdBS+SATRzApMEeoWuw== X-Received: by 2002:a1c:2095:: with SMTP id g143mr1056121wmg.78.1595887132256; Mon, 27 Jul 2020 14:58:52 -0700 (PDT) Received: from localhost.localdomain (109-186-134-209.bb.netvision.net.il. [109.186.134.209]) by smtp.gmail.com with ESMTPSA id p14sm14492940wrx.90.2020.07.27.14.58.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jul 2020 14:58:51 -0700 (PDT) From: Nir Soffer X-Google-Original-From: Nir Soffer To: qemu-devel@nongnu.org, qemu-block@nongnu.org Subject: [PATCH v2 1/4] block: nbd: Fix convert qcow2 compressed to nbd Date: Tue, 28 Jul 2020 00:58:43 +0300 Message-Id: <20200727215846.395443-2-nsoffer@redhat.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200727215846.395443-1-nsoffer@redhat.com> References: <20200727215846.395443-1-nsoffer@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::341; envelope-from=nirsof@gmail.com; helo=mail-wm1-x341.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Nir Soffer , vsementsov@virtuozzo.com, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" When converting to qcow2 compressed format, the last step is a special zero length compressed write, ending in call to bdrv_co_truncate(). This call always fails for the nbd driver since it does not implement bdrv_co_truncate(). For block devices, which have the same limits, the call succeeds since file driver implements bdrv_co_truncate(). If the caller asked to truncate to the same or smaller size with exact=false, the truncate succeeds. Implement the same logic for nbd. Example failing without this change: In one shell starts qemu-nbd: $ truncate -s 1g test.tar $ qemu-nbd --socket=/tmp/nbd.sock --persistent --format=raw --offset 1536 test.tar In another shell convert an image to qcow2 compressed via NBD: $ echo "disk data" > disk.raw $ truncate -s 1g disk.raw $ qemu-img convert -f raw -O qcow2 -c disk1.raw nbd+unix:///?socket=/tmp/nbd.sock; echo $? 1 qemu-img failed, but the conversion was successful: $ qemu-img info nbd+unix:///?socket=/tmp/nbd.sock image: nbd+unix://?socket=/tmp/nbd.sock file format: qcow2 virtual size: 1 GiB (1073741824 bytes) ... $ qemu-img check nbd+unix:///?socket=/tmp/nbd.sock No errors were found on the image. 1/16384 = 0.01% allocated, 100.00% fragmented, 100.00% compressed clusters Image end offset: 393216 $ qemu-img compare disk.raw nbd+unix:///?socket=/tmp/nbd.sock Images are identical. Fixes: https://bugzilla.redhat.com/1860627 Signed-off-by: Nir Soffer Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- block/nbd.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/block/nbd.c b/block/nbd.c index 65a4f56924..dcb0b03641 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -1966,6 +1966,33 @@ static void nbd_close(BlockDriverState *bs) nbd_clear_bdrvstate(s); } +/* + * NBD cannot truncate, but if the caller asks to truncate to the same size, or + * to a smaller size with exact=false, there is no reason to fail the + * operation. + * + * Preallocation mode is ignored since it does not seems useful to fail when + * when never change anything. + */ +static int coroutine_fn nbd_co_truncate(BlockDriverState *bs, int64_t offset, + bool exact, PreallocMode prealloc, + BdrvRequestFlags flags, Error **errp) +{ + BDRVNBDState *s = bs->opaque; + + if (offset != s->info.size && exact) { + error_setg(errp, "Cannot resize NBD nodes"); + return -ENOTSUP; + } + + if (offset > s->info.size) { + error_setg(errp, "Cannot grow NBD nodes"); + return -EINVAL; + } + + return 0; +} + static int64_t nbd_getlength(BlockDriverState *bs) { BDRVNBDState *s = bs->opaque; @@ -2045,6 +2072,7 @@ static BlockDriver bdrv_nbd = { .bdrv_co_flush_to_os = nbd_co_flush, .bdrv_co_pdiscard = nbd_client_co_pdiscard, .bdrv_refresh_limits = nbd_refresh_limits, + .bdrv_co_truncate = nbd_co_truncate, .bdrv_getlength = nbd_getlength, .bdrv_detach_aio_context = nbd_client_detach_aio_context, .bdrv_attach_aio_context = nbd_client_attach_aio_context, @@ -2072,6 +2100,7 @@ static BlockDriver bdrv_nbd_tcp = { .bdrv_co_flush_to_os = nbd_co_flush, .bdrv_co_pdiscard = nbd_client_co_pdiscard, .bdrv_refresh_limits = nbd_refresh_limits, + .bdrv_co_truncate = nbd_co_truncate, .bdrv_getlength = nbd_getlength, .bdrv_detach_aio_context = nbd_client_detach_aio_context, .bdrv_attach_aio_context = nbd_client_attach_aio_context, @@ -2099,6 +2128,7 @@ static BlockDriver bdrv_nbd_unix = { .bdrv_co_flush_to_os = nbd_co_flush, .bdrv_co_pdiscard = nbd_client_co_pdiscard, .bdrv_refresh_limits = nbd_refresh_limits, + .bdrv_co_truncate = nbd_co_truncate, .bdrv_getlength = nbd_getlength, .bdrv_detach_aio_context = nbd_client_detach_aio_context, .bdrv_attach_aio_context = nbd_client_attach_aio_context, From patchwork Mon Jul 27 21:58:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nir Soffer X-Patchwork-Id: 11687713 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 26F1B722 for ; Mon, 27 Jul 2020 22:04:26 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F0121206D7 for ; Mon, 27 Jul 2020 22:04:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OPaNkLQI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F0121206D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:37766 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k0BEP-0004wh-7y for patchwork-qemu-devel@patchwork.kernel.org; Mon, 27 Jul 2020 18:04:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47640) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k0B98-0007qb-RD; Mon, 27 Jul 2020 17:58:58 -0400 Received: from mail-wm1-x331.google.com ([2a00:1450:4864:20::331]:35322) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1k0B96-0004Kz-Sj; Mon, 27 Jul 2020 17:58:58 -0400 Received: by mail-wm1-x331.google.com with SMTP id 184so16199759wmb.0; Mon, 27 Jul 2020 14:58:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=U6hr5AyADbn9Pq4va0Kp0ppxdlnRLsEkuw/N630unvk=; b=OPaNkLQImNPB91Xp65msKF4wh53iVS+JJA/S/v6Qe9vUxLe0+sRogE/zG/HAVoOkZv 1Uw80DAMMUBejNzQ0X4OlbheuZCu1WlCtdUZuoEKl+UzQisNcD5gPpWTD+nr1TR5/8Ud ecR+SR8n0bgUHGIU+dM6PirRn53GpObJGwpB/f2KCXFzNgf+ojCWqkdh9LWt4X1fpBAo fprhy7JfGUgsbfi2eMvexiOfiS/1en39C6fuZ8djOaGzB8iKzAZSJFJFiPyW6FmFZdET Kz7W38N15/lNBfg8WzR5z33Ba0tmdr+4aNh9nBVP+d8D6d66+bmRBqXfI9kzIL0PikvD 1NWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=U6hr5AyADbn9Pq4va0Kp0ppxdlnRLsEkuw/N630unvk=; b=C8JxF82hvLD3i+OSBdY06vYvHnLYOGOvs2I0EF+/Ura9DpQ8kRITt1UeAXG8N8pZJX qCLB7X4CUrWgzLmRjGU4tn6hRrqYebV2apOvHnwXs8GD87w3ArL1isRw9QjOspvR5XU1 YLabnheRQCmbFqfLrM9hgkjUDNyQHlcukgoXncLYE9ziRyhZjcFJrsx6Cz8dxybW20F1 /RnkpqYZHaTpojea7a9QSRPPE8nhLqiV09He+H7agBkrthTPJ7vYX0Fa1IJJ/lyRCF8t pEqeYzP2e/3YFWU0uMnW4ZCcQnm1uKjTvEJYE6IbVwTcVd09p+slWh9gH4s/qmT8eWUN 6+xQ== X-Gm-Message-State: AOAM530wObNlS8hCRbFmAOQz3ld7SkiLQG5ZQkweD9hbBVOrjYBcgA/T ZAzQd+gcD8yxPRmLaK4+g/Lg8mN9H7M= X-Google-Smtp-Source: ABdhPJzzJkOiNiH2updFBSao9M9jdWD8scJP1ywieRsTYM7lhECGDK7vfX8wML8mWPHGt/bGTxrXbg== X-Received: by 2002:a05:600c:230e:: with SMTP id 14mr1019957wmo.3.1595887133959; Mon, 27 Jul 2020 14:58:53 -0700 (PDT) Received: from localhost.localdomain (109-186-134-209.bb.netvision.net.il. [109.186.134.209]) by smtp.gmail.com with ESMTPSA id p14sm14492940wrx.90.2020.07.27.14.58.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jul 2020 14:58:53 -0700 (PDT) From: Nir Soffer X-Google-Original-From: Nir Soffer To: qemu-devel@nongnu.org, qemu-block@nongnu.org Subject: [PATCH v2 2/4] iotests: Make qemu_nbd_popen() a contextmanager Date: Tue, 28 Jul 2020 00:58:44 +0300 Message-Id: <20200727215846.395443-3-nsoffer@redhat.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200727215846.395443-1-nsoffer@redhat.com> References: <20200727215846.395443-1-nsoffer@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::331; envelope-from=nirsof@gmail.com; helo=mail-wm1-x331.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Nir Soffer , vsementsov@virtuozzo.com, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Instead of duplicating the code to wait until the server is ready and remember to terminate the server and wait for it, make it possible to use like this: with qemu_nbd_popen('-k', sock, image): # Access image via qemu-nbd socket... Only test 264 used this helper, but I had to modify the output since it did not consistently when starting and stopping qemu-nbd. Signed-off-by: Nir Soffer Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/264 | 76 +++++++++++++---------------------- tests/qemu-iotests/264.out | 2 + tests/qemu-iotests/iotests.py | 28 ++++++++++++- 3 files changed, 56 insertions(+), 50 deletions(-) diff --git a/tests/qemu-iotests/264 b/tests/qemu-iotests/264 index 304a7443d7..666f164ed8 100755 --- a/tests/qemu-iotests/264 +++ b/tests/qemu-iotests/264 @@ -36,48 +36,32 @@ wait_step = 0.2 qemu_img_create('-f', iotests.imgfmt, disk_a, str(size)) qemu_img_create('-f', iotests.imgfmt, disk_b, str(size)) -srv = qemu_nbd_popen('-k', nbd_sock, '-f', iotests.imgfmt, disk_b) -# Wait for NBD server availability -t = 0 -ok = False -while t < wait_limit: - ok = qemu_io_silent_check('-f', 'raw', '-c', 'read 0 512', nbd_uri) - if ok: - break - time.sleep(wait_step) - t += wait_step +with qemu_nbd_popen('-k', nbd_sock, '-f', iotests.imgfmt, disk_b): + vm = iotests.VM().add_drive(disk_a) + vm.launch() + vm.hmp_qemu_io('drive0', 'write 0 {}'.format(size)) + + vm.qmp_log('blockdev-add', filters=[iotests.filter_qmp_testfiles], + **{'node_name': 'backup0', + 'driver': 'raw', + 'file': {'driver': 'nbd', + 'server': {'type': 'unix', 'path': nbd_sock}, + 'reconnect-delay': 10}}) + vm.qmp_log('blockdev-backup', device='drive0', sync='full', target='backup0', + speed=(1 * 1024 * 1024)) + + # Wait for some progress + t = 0 + while t < wait_limit: + jobs = vm.qmp('query-block-jobs')['return'] + if jobs and jobs[0]['offset'] > 0: + break + time.sleep(wait_step) + t += wait_step -assert ok - -vm = iotests.VM().add_drive(disk_a) -vm.launch() -vm.hmp_qemu_io('drive0', 'write 0 {}'.format(size)) - -vm.qmp_log('blockdev-add', filters=[iotests.filter_qmp_testfiles], - **{'node_name': 'backup0', - 'driver': 'raw', - 'file': {'driver': 'nbd', - 'server': {'type': 'unix', 'path': nbd_sock}, - 'reconnect-delay': 10}}) -vm.qmp_log('blockdev-backup', device='drive0', sync='full', target='backup0', - speed=(1 * 1024 * 1024)) - -# Wait for some progress -t = 0 -while t < wait_limit: - jobs = vm.qmp('query-block-jobs')['return'] if jobs and jobs[0]['offset'] > 0: - break - time.sleep(wait_step) - t += wait_step - -if jobs and jobs[0]['offset'] > 0: - log('Backup job is started') - -log('Kill NBD server') -srv.kill() -srv.wait() + log('Backup job is started') jobs = vm.qmp('query-block-jobs')['return'] if jobs and jobs[0]['offset'] < jobs[0]['len']: @@ -88,12 +72,8 @@ vm.qmp_log('block-job-set-speed', device='drive0', speed=0) # Emulate server down time for 1 second time.sleep(1) -log('Start NBD server') -srv = qemu_nbd_popen('-k', nbd_sock, '-f', iotests.imgfmt, disk_b) - -e = vm.event_wait('BLOCK_JOB_COMPLETED') -log('Backup completed: {}'.format(e['data']['offset'])) - -vm.qmp_log('blockdev-del', node_name='backup0') -srv.kill() -vm.shutdown() +with qemu_nbd_popen('-k', nbd_sock, '-f', iotests.imgfmt, disk_b): + e = vm.event_wait('BLOCK_JOB_COMPLETED') + log('Backup completed: {}'.format(e['data']['offset'])) + vm.qmp_log('blockdev-del', node_name='backup0') + vm.shutdown() diff --git a/tests/qemu-iotests/264.out b/tests/qemu-iotests/264.out index 3000944b09..c45b1e81ef 100644 --- a/tests/qemu-iotests/264.out +++ b/tests/qemu-iotests/264.out @@ -1,3 +1,4 @@ +Start NBD server {"execute": "blockdev-add", "arguments": {"driver": "raw", "file": {"driver": "nbd", "reconnect-delay": 10, "server": {"path": "TEST_DIR/PID-nbd-sock", "type": "unix"}}, "node-name": "backup0"}} {"return": {}} {"execute": "blockdev-backup", "arguments": {"device": "drive0", "speed": 1048576, "sync": "full", "target": "backup0"}} @@ -11,3 +12,4 @@ Start NBD server Backup completed: 5242880 {"execute": "blockdev-del", "arguments": {"node-name": "backup0"}} {"return": {}} +Kill NBD server diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 3590ed78a0..8f79668435 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -28,10 +28,13 @@ import signal import struct import subprocess import sys +import time from typing import (Any, Callable, Dict, Iterable, List, Optional, Sequence, Tuple, TypeVar) import unittest +from contextlib import contextmanager + # pylint: disable=import-error, wrong-import-position sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python')) from qemu import qtest @@ -270,9 +273,30 @@ def qemu_nbd_early_pipe(*args): return subp.returncode, output if subp.returncode else '' +@contextmanager def qemu_nbd_popen(*args): - '''Run qemu-nbd in daemon mode and return the parent's exit code''' - return subprocess.Popen(qemu_nbd_args + ['--persistent'] + list(args)) + '''Context manager running qemu-nbd within the context''' + pid_file = file_path("pid") + + cmd = list(qemu_nbd_args) + cmd.extend(('--persistent', '--pid-file', pid_file)) + cmd.extend(args) + + log('Start NBD server') + p = subprocess.Popen(cmd) + try: + while not os.path.exists(pid_file): + if p.poll() is not None: + raise RuntimeError( + "qemu-nbd terminated with exit code {}: {}" + .format(p.returncode, ' '.join(cmd))) + + time.sleep(0.01) + yield + finally: + log('Kill NBD server') + p.kill() + p.wait() def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt): '''Return True if two image files are identical''' From patchwork Mon Jul 27 21:58:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nir Soffer X-Patchwork-Id: 11687711 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 205CE722 for ; Mon, 27 Jul 2020 22:03:10 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EB0BC206D7 for ; Mon, 27 Jul 2020 22:03:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LoV9N6lR" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EB0BC206D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:34254 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k0BDB-0003V9-5Q for patchwork-qemu-devel@patchwork.kernel.org; Mon, 27 Jul 2020 18:03:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47646) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k0B99-0007rv-Qc; Mon, 27 Jul 2020 17:58:59 -0400 Received: from mail-wr1-x444.google.com ([2a00:1450:4864:20::444]:41675) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1k0B98-0004LH-8c; Mon, 27 Jul 2020 17:58:59 -0400 Received: by mail-wr1-x444.google.com with SMTP id r2so11236799wrs.8; Mon, 27 Jul 2020 14:58:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TXJ4qVIr+/EEDMVg8MI9Tdsi5K/g1R0LNvzFvLMTjsg=; b=LoV9N6lRbqWkxZYPpTG5rJk+yuSUEPwij9p8OzuIKNh7TK1njNLkA8qKjRvWahlW4G D66gp4j4lABN75So0IvvcXfD/QOKR5OiDPqRfh8sKoeD8P1kVGN2DydXnybcw9wB7JrC 7mDKtSMf76icj3cy3ERM1IUmNQGZrGbFCwZ75ocoU5FqBHaTjD/cbfDzgZ+cMVHwl7UH +s6AYhUEyoxYCmFfCIQo1xIctewKOivvHpx32eWKf4waJeN12GXqPhy1lKbVdX4j297J z1uUL62EFOgKnmCILPW/tSEOB3hNwgGR/k2tilJWlL7wviLMbDC98xzj1C3jv09gna9C 37uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TXJ4qVIr+/EEDMVg8MI9Tdsi5K/g1R0LNvzFvLMTjsg=; b=fAAUYCzAULP5VqaI0fOgKjOeMzaCxyZolTGfbHReZUxIXGyryIjp/UliSy0SWE13I1 EMThviKw57dhG/6balAl3LgKpK6eyF9JMHc28Sa3v8nLD4GzbyAgyTi89x9sS3LzxS/7 fVXL/HFx/UtQBURSQlCw+Sxs7AqP+2FmOGTTa/65ajcwznRp/CYw1NOBnGtWusmU314C iWFRp6EXfEJ2smdDumYbATwhdD2YzIuJOcbfg4chKNZJ4B/Iqol6o+n6kzyWXDWSQHeJ +3fT11qAfeY5Hr39eBuH5/7LKKfuQxk9uIfs09vl6s0p5vW9CyBFlMjjJBGKgiAJSKiB gaOg== X-Gm-Message-State: AOAM533jr06qVxhSOWFiY+vBJjivPZRTE8NKPUtCyIMFn+j2aabNA6Rs 7jwdOLWiN26EIDkkzhCRbn41DbfA1xc= X-Google-Smtp-Source: ABdhPJzCMPVhhnKdq1MA2DUfhEY3qAQ2DOjs/BJcpQrgQWrCCLK/DcYLGj3vCDh9Ofi2OhaD4A2k+A== X-Received: by 2002:a5d:6681:: with SMTP id l1mr20988152wru.47.1595887135886; Mon, 27 Jul 2020 14:58:55 -0700 (PDT) Received: from localhost.localdomain (109-186-134-209.bb.netvision.net.il. [109.186.134.209]) by smtp.gmail.com with ESMTPSA id p14sm14492940wrx.90.2020.07.27.14.58.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jul 2020 14:58:55 -0700 (PDT) From: Nir Soffer X-Google-Original-From: Nir Soffer To: qemu-devel@nongnu.org, qemu-block@nongnu.org Subject: [PATCH v2 3/4] iotests: Add more qemu_img helpers Date: Tue, 28 Jul 2020 00:58:45 +0300 Message-Id: <20200727215846.395443-4-nsoffer@redhat.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200727215846.395443-1-nsoffer@redhat.com> References: <20200727215846.395443-1-nsoffer@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::444; envelope-from=nirsof@gmail.com; helo=mail-wr1-x444.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Nir Soffer , vsementsov@virtuozzo.com, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Add 2 helpers for measuring and checking images: - qemu_img_measure() - qemu_img_check() Both use --output-json and parse the returned json to make easy to use in other tests. I'm going to use them in a new test, and I hope they will be useful in may other tests. Signed-off-by: Nir Soffer Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 8f79668435..717b5b652c 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -141,6 +141,12 @@ def qemu_img_create(*args): return qemu_img(*args) +def qemu_img_measure(*args): + return json.loads(qemu_img_pipe("measure", "--output", "json", *args)) + +def qemu_img_check(*args): + return json.loads(qemu_img_pipe("check", "--output", "json", *args)) + def qemu_img_verbose(*args): '''Run qemu-img without suppressing its output and return the exit code''' exitcode = subprocess.call(qemu_img_args + list(args)) From patchwork Mon Jul 27 21:58:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Nir Soffer X-Patchwork-Id: 11687707 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AF904138A for ; Mon, 27 Jul 2020 22:00:35 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 85D1C206D7 for ; Mon, 27 Jul 2020 22:00:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BVoqSPcq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 85D1C206D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57414 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k0BAg-0001IR-Qm for patchwork-qemu-devel@patchwork.kernel.org; Mon, 27 Jul 2020 18:00:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47690) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k0B9F-00083S-4i; Mon, 27 Jul 2020 17:59:05 -0400 Received: from mail-wm1-x333.google.com ([2a00:1450:4864:20::333]:39569) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1k0B9D-0004Ls-8s; Mon, 27 Jul 2020 17:59:04 -0400 Received: by mail-wm1-x333.google.com with SMTP id t142so9835649wmt.4; Mon, 27 Jul 2020 14:59:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YxkYB4mtsedDfq3OADLqoKGrrr9VRyJd5kW4UXFdOtQ=; b=BVoqSPcqpqZih8B18GX/smub9v22DY/4t+1BkGIb6gRsp6Fu3GuGP/HjLukHiHPW7e oJbBXrwx0noJvLliGkFGiYSKmn3Wz/Nw+i4soy/aw3da6Z30UAlH1p4VzUhpe/ts2l7e T2FP0vQJZn826hVzwk08dRF4RneWOajzjXYQg0O/w1KlclcD6yX7Mfgk6E6h6UAGXuc3 46FPRH8SLx0QGb7xORHN41Kt8EvhiqjtY5Avi7OAfXakd5KS7zTcVG3dK2/z4imCA06L +oJvRwGUl/EsWYY8xWtWRk6WcodiVwCMoxOfBGVqeK+7M8KSf9TdzpISraAUThZHICRl mRWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YxkYB4mtsedDfq3OADLqoKGrrr9VRyJd5kW4UXFdOtQ=; b=OQPp8hF3Z2u6C+n6zzmMGgkS/pSV8O9qAlo3rvNFZmNmtG3TR62jAf1dMI1l9V9xDV B666WSH9/l0nwkdradSWCl5v+2Rsk6SczzKzR/ReDhJLHPRQPauG1X+b2qlGzVrzIbyX xPRprJlgy+EVWu2uiil3QJR8iNojwZAg5qKp87UWTtW57dQLDbSTQ+Dvsl8yBhNdUWgL DB72T127O9V86h4hinFAYrn5B1g1zezET+J6i0Jqn6w/8TBQu8iJBDwXoBvxOZD9CaTM e0RQWPL3ZuOUsNL0NnbxDY1DrCrLiY45EgiJuA9ISjCwAjabS6m0y1ubYBECGyoME7xm Ntww== X-Gm-Message-State: AOAM531gWanmXqPSlMOUtrAaysXoedVvHC5XohttxTLEMH49HDY62q+D OrvOtMSbSrxPijYJByRwWtE90vhzxb0= X-Google-Smtp-Source: ABdhPJyikwlpDKbC++I50eFvhJjNBkQERvAavZvfWk7Dh1HFVcEpvDqnT4LGF0oMY2ZK4qvKBayEvQ== X-Received: by 2002:a1c:f219:: with SMTP id s25mr1067199wmc.2.1595887140970; Mon, 27 Jul 2020 14:59:00 -0700 (PDT) Received: from localhost.localdomain (109-186-134-209.bb.netvision.net.il. [109.186.134.209]) by smtp.gmail.com with ESMTPSA id p14sm14492940wrx.90.2020.07.27.14.58.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jul 2020 14:58:56 -0700 (PDT) From: Nir Soffer X-Google-Original-From: Nir Soffer To: qemu-devel@nongnu.org, qemu-block@nongnu.org Subject: [PATCH v2 4/4] iotests: Test convert to qcow2 compressed to NBD Date: Tue, 28 Jul 2020 00:58:46 +0300 Message-Id: <20200727215846.395443-5-nsoffer@redhat.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200727215846.395443-1-nsoffer@redhat.com> References: <20200727215846.395443-1-nsoffer@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::333; envelope-from=nirsof@gmail.com; helo=mail-wm1-x333.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, FREEMAIL_REPLY=1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Nir Soffer , vsementsov@virtuozzo.com, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Add test for "qemu-img convert -O qcow2 -c" to NBD target. The tests     create a OVA file and write compressed qcow2 disk content directly into the OVA file via qemu-nbd. Signed-off-by: Nir Soffer Tested-by: Eric Blake Reviewed-by: Eric Blake --- tests/qemu-iotests/302 | 127 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/302.out | 31 +++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 159 insertions(+) create mode 100755 tests/qemu-iotests/302 create mode 100644 tests/qemu-iotests/302.out diff --git a/tests/qemu-iotests/302 b/tests/qemu-iotests/302 new file mode 100755 index 0000000000..a8506bda15 --- /dev/null +++ b/tests/qemu-iotests/302 @@ -0,0 +1,127 @@ +#!/usr/bin/env python3 +# +# Tests converting qcow2 compressed to NBD +# +# Copyright (c) 2020 Nir Soffer +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# +# owner=nirsof@gmail.com + +import io +import tarfile + +import iotests + +from iotests import ( + file_path, + qemu_img, + qemu_img_check, + qemu_img_create, + qemu_img_log, + qemu_img_measure, + qemu_io, + qemu_nbd_popen, +) + +iotests.script_initialize(supported_fmts=["qcow2"]) + +# Create source disk. Using qcow2 to enable strict comparing later, and +# avoid issues with random filesystem on CI environment. +src_disk = file_path("disk.qcow2") +qemu_img_create("-f", iotests.imgfmt, src_disk, "1g") +qemu_io("-f", iotests.imgfmt, "-c", "write 1m 64k", src_disk) + +# The use case is writing qcow2 image directly into an ova file, which +# is a tar file with specific layout. This is tricky since we don't know the +# size of the image before compressing, so we have to do: +# 1. Add an ovf file. +# 2. Find the offset of the next member data. +# 3. Make room for image data, allocating for the worst case. +# 4. Write compressed image data into the tar. +# 5. Add a tar entry with the actual image size. +# 6. Shrink the tar to the actual size, aligned to 512 bytes. + +tar_file = file_path("test.ova") + +with tarfile.open(tar_file, "w") as tar: + + # 1. Add an ovf file. + + ovf_data = b"" + ovf = tarfile.TarInfo("vm.ovf") + ovf.size = len(ovf_data) + tar.addfile(ovf, io.BytesIO(ovf_data)) + + # 2. Find the offset of the next member data. + + offset = tar.fileobj.tell() + 512 + + # 3. Make room for image data, allocating for the worst case. + + measure = qemu_img_measure("-O", "qcow2", src_disk) + tar.fileobj.truncate(offset + measure["required"]) + + # 4. Write compressed image data into the tar. + + nbd_sock = file_path("nbd-sock", base_dir=iotests.sock_dir) + nbd_uri = "nbd+unix:///exp?socket=" + nbd_sock + + # Use raw format to allow creating qcow2 directly into tar file. + with qemu_nbd_popen( + "--socket", nbd_sock, + "--export-name", "exp", + "--format", "raw", + "--offset", str(offset), + tar_file): + + iotests.log("=== Target image info ===") + qemu_img_log("info", nbd_uri) + + qemu_img( + "convert", + "-f", iotests.imgfmt, + "-O", "qcow2", + "-c", + src_disk, + nbd_uri) + + iotests.log("=== Converted image info ===") + qemu_img_log("info", nbd_uri) + + iotests.log("=== Converted image check ===") + qemu_img_log("check", nbd_uri) + + iotests.log("=== Comparing to source disk ===") + qemu_img_log("compare", src_disk, nbd_uri) + + actual_size = qemu_img_check(nbd_uri)["image-end-offset"] + + # 5. Add a tar entry with the actual image size. + + disk = tarfile.TarInfo("disk") + disk.size = actual_size + tar.addfile(disk) + + # 6. Shrink the tar to the actual size, aligned to 512 bytes. + + tar_size = offset + (disk.size + 511) & ~511 + tar.fileobj.seek(tar_size) + tar.fileobj.truncate(tar_size) + +with tarfile.open(tar_file) as tar: + members = [{"name": m.name, "size": m.size, "offset": m.offset_data} + for m in tar] + iotests.log("=== OVA file contents ===") + iotests.log(members) diff --git a/tests/qemu-iotests/302.out b/tests/qemu-iotests/302.out new file mode 100644 index 0000000000..e37d3a1030 --- /dev/null +++ b/tests/qemu-iotests/302.out @@ -0,0 +1,31 @@ +Start NBD server +=== Target image info === +image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock +file format: raw +virtual size: 448 KiB (458752 bytes) +disk size: unavailable + +=== Converted image info === +image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock +file format: qcow2 +virtual size: 1 GiB (1073741824 bytes) +disk size: unavailable +cluster_size: 65536 +Format specific information: + compat: 1.1 + compression type: zlib + lazy refcounts: false + refcount bits: 16 + corrupt: false + +=== Converted image check === +No errors were found on the image. +1/16384 = 0.01% allocated, 100.00% fragmented, 100.00% compressed clusters +Image end offset: 393216 + +=== Comparing to source disk === +Images are identical. + +Kill NBD server +=== OVA file contents === +[{"name": "vm.ovf", "offset": 512, "size": 6}, {"name": "disk", "offset": 1536, "size": 393216}] diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index 1d0252e1f0..1e1cb27bc8 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -308,3 +308,4 @@ 297 meta 299 auto quick 301 backing quick +302 quick