From patchwork Tue Dec 18 10:00:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yongji Xie X-Patchwork-Id: 10735309 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3EF796C2 for ; Tue, 18 Dec 2018 10:10:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2F2F92A589 for ; Tue, 18 Dec 2018 10:10:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2D53A2A643; Tue, 18 Dec 2018 10:10:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 872252A598 for ; Tue, 18 Dec 2018 10:10:12 +0000 (UTC) Received: from localhost ([::1]:52912 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gZCKJ-0006GS-Kr for patchwork-qemu-devel@patchwork.kernel.org; Tue, 18 Dec 2018 05:10:11 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53265) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gZCBm-0007rH-Fp for qemu-devel@nongnu.org; Tue, 18 Dec 2018 05:01:26 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gZCBe-0001t4-TT for qemu-devel@nongnu.org; Tue, 18 Dec 2018 05:01:20 -0500 Received: from mail-pl1-x642.google.com ([2607:f8b0:4864:20::642]:34421) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gZCBU-0001oe-J7 for qemu-devel@nongnu.org; Tue, 18 Dec 2018 05:01:06 -0500 Received: by mail-pl1-x642.google.com with SMTP id w4so7603520plz.1 for ; Tue, 18 Dec 2018 02:01:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=rpCrrMTMjO8Pf2TduvQzJoDEGAQRbHHxcLC8HFS/jb4=; b=lznQMdNX9xYyD9nyL3YjZg1qqrso6t14Jaz6csUq/DAHbr6IINWZCB5BXuffozYtmQ nhOdFVIU80EEbGz8kpNgxxDAtzUE6Zz6VE7k5Abrl2G3eCiq1qaaS4ebNmruDafgXvEQ sKc+XoUMIkv7D5b8MxEpP209Fc7/ahVPzNjqKyAvAdw6hj2mEhC49gVViALKjDUDEbog 6p455kSt/H2gNJFkYd8tQZKES5I4qJVdHwqS3Akzony3t4AYmk/ePrbDj/IjbZmjLR+Y xjvjBVXRsAUhoBNgIqRQYT7cMO12qgbtR3NRKFbLQWucPDLbTIav4if2bIQoEwhSGuVs fZ4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=rpCrrMTMjO8Pf2TduvQzJoDEGAQRbHHxcLC8HFS/jb4=; b=SERGkN4XARuw1fyROfFk7OaXO2aT1KQzJhXVU5Ea5IOo33R2vpi4pBR8rE7WXlId9x Ym+F55EMKPpf2SXpeM7ZXLtn+gYRvaJOzHD6ZaUsEZnZtbHp6+tKybwm6Hi8fcESmK0d i4qrYMzzYVa3q0qblKNl7QOl9f7EupYEO2eQMGkU7pCU53+P1ucnMKmAxS/241KX3stJ Txrxs1Qlq2LpYBuo8A75ATNH+wA1PdndafDaMyBvzOCkqf7yYcOcU9yS/7/F8U6yrXZY t/LoCtlJYG3CpR7mHbEWccM1mhYF68a01mYVN0EEtQiQdrvGZZtbdIG+9zB+h2DAmgQG 832w== X-Gm-Message-State: AA+aEWb/YNJWDTAJHBxAM/9oEsBQXn9Hy52tXqiMR8AdOluA5sTaBD4G rK7IEpnbnhrCNfRRkNeU5QE= X-Google-Smtp-Source: AFSGD/U7UH1ujFOuuXdFVgI3faiTu4kM2lV91Xi5qkj/n/1leFEDjFfNwqTbZqEItIwBFNcMOcxNmQ== X-Received: by 2002:a17:902:bd4a:: with SMTP id b10mr15904886plx.232.1545127262946; Tue, 18 Dec 2018 02:01:02 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id l22sm23723308pfj.179.2018.12.18.02.01.01 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 18 Dec 2018 02:01:02 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, marcandre.lureau@redhat.com, jasowang@redhat.com, yury-kotov@yandex-team.ru, maxime.coquelin@redhat.com Date: Tue, 18 Dec 2018 18:00:01 +0800 Message-Id: <20181218100002.11219-7-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181218100002.11219-1-xieyongji@baidu.com> References: <20181218100002.11219-1-xieyongji@baidu.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::642 Subject: [Qemu-devel] [PATCH v2 for-4.0 6/7] vhost-user-blk: Add support to reconnect backend X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Xie Yongji Since we now support the message VHOST_USER_GET_SHM_SIZE and VHOST_USER_SET_SHM_FD. The backend is able to restart safely because it can record inflight I/O in shared memory. This patch allows qemu to reconnect the backend after connection closed. Signed-off-by: Xie Yongji Signed-off-by: Ni Xun Signed-off-by: Zhang Yu --- hw/block/vhost-user-blk.c | 183 ++++++++++++++++++++++++----- include/hw/virtio/vhost-user-blk.h | 4 + 2 files changed, 160 insertions(+), 27 deletions(-) diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c index 27028cf996..80f2e2d765 100644 --- a/hw/block/vhost-user-blk.c +++ b/hw/block/vhost-user-blk.c @@ -101,7 +101,7 @@ const VhostDevConfigOps blk_ops = { .vhost_dev_config_notifier = vhost_user_blk_handle_config_change, }; -static void vhost_user_blk_start(VirtIODevice *vdev) +static int vhost_user_blk_start(VirtIODevice *vdev) { VHostUserBlk *s = VHOST_USER_BLK(vdev); BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(vdev))); @@ -110,13 +110,13 @@ static void vhost_user_blk_start(VirtIODevice *vdev) if (!k->set_guest_notifiers) { error_report("binding does not support guest notifiers"); - return; + return -ENOSYS; } ret = vhost_dev_enable_notifiers(&s->dev, vdev); if (ret < 0) { error_report("Error enabling host notifiers: %d", -ret); - return; + return ret; } ret = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, true); @@ -147,12 +147,13 @@ static void vhost_user_blk_start(VirtIODevice *vdev) vhost_virtqueue_mask(&s->dev, vdev, i, false); } - return; + return ret; err_guest_notifiers: k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false); err_host_notifiers: vhost_dev_disable_notifiers(&s->dev, vdev); + return ret; } static void vhost_user_blk_stop(VirtIODevice *vdev) @@ -171,7 +172,6 @@ static void vhost_user_blk_stop(VirtIODevice *vdev) ret = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false); if (ret < 0) { error_report("vhost guest notifier cleanup failed: %d", ret); - return; } vhost_dev_disable_notifiers(&s->dev, vdev); @@ -181,21 +181,43 @@ static void vhost_user_blk_set_status(VirtIODevice *vdev, uint8_t status) { VHostUserBlk *s = VHOST_USER_BLK(vdev); bool should_start = status & VIRTIO_CONFIG_S_DRIVER_OK; + int ret; if (!vdev->vm_running) { should_start = false; } - if (s->dev.started == should_start) { + if (s->should_start == should_start) { + return; + } + + if (!s->connected || s->dev.started == should_start) { + s->should_start = should_start; return; } if (should_start) { - vhost_user_blk_start(vdev); + s->should_start = true; + /* + * make sure vhost_user_blk_handle_output() ignores fake + * guest kick by vhost_dev_enable_notifiers() + */ + barrier(); + ret = vhost_user_blk_start(vdev); + if (ret < 0) { + error_report("vhost-user-blk: vhost start failed: %s", + strerror(-ret)); + qemu_chr_fe_disconnect(&s->chardev); + } } else { vhost_user_blk_stop(vdev); + /* + * make sure vhost_user_blk_handle_output() ignore fake + * guest kick by vhost_dev_disable_notifiers() + */ + barrier(); + s->should_start = false; } - } static uint64_t vhost_user_blk_get_features(VirtIODevice *vdev, @@ -225,13 +247,22 @@ static uint64_t vhost_user_blk_get_features(VirtIODevice *vdev, static void vhost_user_blk_handle_output(VirtIODevice *vdev, VirtQueue *vq) { VHostUserBlk *s = VHOST_USER_BLK(vdev); - int i; + int i, ret; if (!(virtio_host_has_feature(vdev, VIRTIO_F_VERSION_1) && !virtio_vdev_has_feature(vdev, VIRTIO_F_VERSION_1))) { return; } + if (s->should_start) { + return; + } + s->should_start = true; + + if (!s->connected) { + return; + } + if (s->dev.started) { return; } @@ -239,7 +270,13 @@ static void vhost_user_blk_handle_output(VirtIODevice *vdev, VirtQueue *vq) /* Some guests kick before setting VIRTIO_CONFIG_S_DRIVER_OK so start * vhost here instead of waiting for .set_status(). */ - vhost_user_blk_start(vdev); + ret = vhost_user_blk_start(vdev); + if (ret < 0) { + error_report("vhost-user-blk: vhost start failed: %s", + strerror(-ret)); + qemu_chr_fe_disconnect(&s->chardev); + return; + } /* Kick right away to begin processing requests already in vring */ for (i = 0; i < s->dev.nvqs; i++) { @@ -259,12 +296,105 @@ static void vhost_user_blk_reset(VirtIODevice *vdev) vhost_dev_reset_shm(s->shm); } +static int vhost_user_blk_connect(DeviceState *dev) +{ + VirtIODevice *vdev = VIRTIO_DEVICE(dev); + VHostUserBlk *s = VHOST_USER_BLK(vdev); + int ret = 0; + + if (s->connected) { + return 0; + } + s->connected = true; + + s->dev.nvqs = s->num_queues; + s->dev.vqs = s->vqs; + s->dev.vq_index = 0; + s->dev.backend_features = 0; + + vhost_dev_set_config_notifier(&s->dev, &blk_ops); + + ret = vhost_dev_init(&s->dev, s->vhost_user, VHOST_BACKEND_TYPE_USER, 0); + if (ret < 0) { + error_report("vhost-user-blk: vhost initialization failed: %s", + strerror(-ret)); + return ret; + } + + /* restore vhost state */ + if (s->should_start) { + ret = vhost_user_blk_start(vdev); + if (ret < 0) { + error_report("vhost-user-blk: vhost start failed: %s", + strerror(-ret)); + return ret; + } + } + + return 0; +} + +static void vhost_user_blk_disconnect(DeviceState *dev) +{ + VirtIODevice *vdev = VIRTIO_DEVICE(dev); + VHostUserBlk *s = VHOST_USER_BLK(vdev); + + if (!s->connected) { + return; + } + s->connected = false; + + if (s->dev.started) { + vhost_user_blk_stop(vdev); + } + + vhost_dev_cleanup(&s->dev); +} + +static gboolean vhost_user_blk_watch(GIOChannel *chan, GIOCondition cond, + void *opaque) +{ + DeviceState *dev = opaque; + VirtIODevice *vdev = VIRTIO_DEVICE(dev); + VHostUserBlk *s = VHOST_USER_BLK(vdev); + + qemu_chr_fe_disconnect(&s->chardev); + + return true; +} + +static void vhost_user_blk_event(void *opaque, int event) +{ + DeviceState *dev = opaque; + VirtIODevice *vdev = VIRTIO_DEVICE(dev); + VHostUserBlk *s = VHOST_USER_BLK(vdev); + + switch (event) { + case CHR_EVENT_OPENED: + if (vhost_user_blk_connect(dev) < 0) { + qemu_chr_fe_disconnect(&s->chardev); + return; + } + s->watch = qemu_chr_fe_add_watch(&s->chardev, G_IO_HUP, + vhost_user_blk_watch, dev); + break; + case CHR_EVENT_CLOSED: + vhost_user_blk_disconnect(dev); + if (s->watch) { + g_source_remove(s->watch); + s->watch = 0; + } + break; + } +} + static void vhost_user_blk_device_realize(DeviceState *dev, Error **errp) { VirtIODevice *vdev = VIRTIO_DEVICE(dev); VHostUserBlk *s = VHOST_USER_BLK(vdev); VhostUserState *user; int i, ret; + Error *err = NULL; if (!s->chardev.chr) { error_setg(errp, "vhost-user-blk: chardev is mandatory"); @@ -299,23 +429,21 @@ static void vhost_user_blk_device_realize(DeviceState *dev, Error **errp) } s->shm = g_new0(struct vhost_shm, 1); - - s->dev.nvqs = s->num_queues; - s->dev.vqs = g_new(struct vhost_virtqueue, s->dev.nvqs); - s->dev.vq_index = 0; - s->dev.backend_features = 0; - - vhost_dev_set_config_notifier(&s->dev, &blk_ops); - - ret = vhost_dev_init(&s->dev, s->vhost_user, VHOST_BACKEND_TYPE_USER, 0); - if (ret < 0) { - error_setg(errp, "vhost-user-blk: vhost initialization failed: %s", - strerror(-ret)); - goto virtio_err; + s->vqs = g_new(struct vhost_virtqueue, s->num_queues); + s->watch = 0; + s->should_start = false; + s->connected = false; + + while (qemu_chr_fe_wait_connected(&s->chardev, &err) < 0) { + error_report_err(err); + err = NULL; + sleep(1); } + qemu_chr_fe_set_handlers(&s->chardev, NULL, NULL, vhost_user_blk_event, + NULL, (void *)dev, NULL, true); ret = vhost_dev_get_config(&s->dev, (uint8_t *)&s->blkcfg, - sizeof(struct virtio_blk_config)); + sizeof(struct virtio_blk_config)); if (ret < 0) { error_setg(errp, "vhost-user-blk: get block config failed"); goto vhost_err; @@ -335,8 +463,7 @@ static void vhost_user_blk_device_realize(DeviceState *dev, Error **errp) vhost_err: vhost_dev_cleanup(&s->dev); -virtio_err: - g_free(s->dev.vqs); + g_free(s->vqs); g_free(s->shm); virtio_cleanup(vdev); @@ -351,9 +478,11 @@ static void vhost_user_blk_device_unrealize(DeviceState *dev, Error **errp) VHostUserBlk *s = VHOST_USER_BLK(dev); vhost_user_blk_set_status(vdev, 0); + qemu_chr_fe_set_handlers(&s->chardev, NULL, NULL, NULL, + NULL, NULL, NULL, false); vhost_dev_cleanup(&s->dev); vhost_dev_free_shm(s->shm); - g_free(s->dev.vqs); + g_free(s->vqs); g_free(s->shm); virtio_cleanup(vdev); diff --git a/include/hw/virtio/vhost-user-blk.h b/include/hw/virtio/vhost-user-blk.h index bb706d70b3..c17d47402b 100644 --- a/include/hw/virtio/vhost-user-blk.h +++ b/include/hw/virtio/vhost-user-blk.h @@ -38,6 +38,10 @@ typedef struct VHostUserBlk { struct vhost_dev dev; struct vhost_shm *shm; VhostUserState *vhost_user; + struct vhost_virtqueue *vqs; + guint watch; + bool should_start; + bool connected; } VHostUserBlk; #endif