From patchwork Wed Jul 3 05:56:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chao Gao X-Patchwork-Id: 11028931 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0FC5C1398 for ; Wed, 3 Jul 2019 05:53:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F2CD428872 for ; Wed, 3 Jul 2019 05:53:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E588B288A6; Wed, 3 Jul 2019 05:53:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 12BBE28872 for ; Wed, 3 Jul 2019 05:53:45 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hiYBc-0008L9-CB; Wed, 03 Jul 2019 05:52:08 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hiYBa-0008L4-S9 for xen-devel@lists.xenproject.org; Wed, 03 Jul 2019 05:52:06 +0000 X-Inumbo-ID: af9563f1-9d56-11e9-8980-bc764e045a96 Received: from mga11.intel.com (unknown [192.55.52.93]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTPS id af9563f1-9d56-11e9-8980-bc764e045a96; Wed, 03 Jul 2019 05:52:05 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 02 Jul 2019 22:52:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.63,446,1557212400"; d="scan'208";a="154649872" Received: from gao-cwp.sh.intel.com ([10.239.159.26]) by orsmga007.jf.intel.com with ESMTP; 02 Jul 2019 22:52:03 -0700 From: Chao Gao To: xen-devel@lists.xenproject.org Date: Wed, 3 Jul 2019 13:56:13 +0800 Message-Id: <1562133373-19208-1-git-send-email-chao.gao@intel.com> X-Mailer: git-send-email 1.9.1 Subject: [Xen-devel] [PATCH v2] libxl_qmp: wait for completion of device removal X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Anthony PERARD , Ian Jackson , Wei Liu , Chao Gao MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP To remove a device from a domain, a qmp command is sent to qemu. But it is handled by qemu asychronously. Even the qmp command is claimed to be done, the actual handling in qemu side may happen later. This behavior brings two questions: 1. Attaching a device back to a domain right after detaching the device from that domain would fail with error: libxl: error: libxl_qmp.c:341:qmp_handle_error_response: Domain 1:received an error message from QMP server: Duplicate ID 'pci-pt-60_00.0' for device 2. Accesses to PCI configuration space in Qemu may overlap with later device reset issued by 'xl' or by pciback. In order to avoid mentioned questions, wait for the completion of device removal by querying all pci devices using qmp command and ensuring the target device isn't listed. Only retry 5 times to avoid 'xl' potentially being blocked by qemu. Signed-off-by: Chao Gao Reviewed-by: Anthony PERARD Signed-off-by: Ian Jackson --- Changes in v2: - Break out early if we found an error during querying pci devices. - Print a message to warn admin that device removal may not be done in device model's side. --- tools/libxl/libxl_qmp.c | 61 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 60 insertions(+), 1 deletion(-) diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c index 42c8ab8d8d..9c4480a2b1 100644 --- a/tools/libxl/libxl_qmp.c +++ b/tools/libxl/libxl_qmp.c @@ -916,6 +916,38 @@ out: return rc; } +static int pci_del_callback(libxl__qmp_handler *qmp, + const libxl__json_object *response, void *opaque) +{ + const char *asked_id = opaque; + const libxl__json_object *bus = NULL; + GC_INIT(qmp->ctx); + int i, j, rc = 0; + + for (i = 0; (bus = libxl__json_array_get(response, i)); i++) { + const libxl__json_object *devices = NULL; + const libxl__json_object *device = NULL; + const libxl__json_object *o = NULL; + const char *id = NULL; + + devices = libxl__json_map_get("devices", bus, JSON_ARRAY); + + for (j = 0; (device = libxl__json_array_get(devices, j)); j++) { + o = libxl__json_map_get("qdev_id", device, JSON_STRING); + id = libxl__json_object_get_string(o); + + if (id && strcmp(asked_id, id) == 0) { + rc = 1; + goto out; + } + } + } + +out: + GC_FREE; + return rc; +} + static int qmp_run_command(libxl__gc *gc, int domid, const char *cmd, libxl__json_object *args, qmp_callback_t callback, void *opaque) @@ -1000,9 +1032,36 @@ int libxl__qmp_pci_add(libxl__gc *gc, int domid, libxl_device_pci *pcidev) static int qmp_device_del(libxl__gc *gc, int domid, char *id) { libxl__json_object *args = NULL; + libxl__qmp_handler *qmp = NULL; + int rc = 0; + + qmp = libxl__qmp_initialize(gc, domid); + if (!qmp) + return ERROR_FAIL; qmp_parameters_add_string(gc, &args, "id", id); - return qmp_run_command(gc, domid, "device_del", args, NULL, NULL); + rc = qmp_synchronous_send(qmp, "device_del", args, + NULL, NULL, qmp->timeout); + if (rc == 0) { + unsigned int retry = 0; + + do { + rc = qmp_synchronous_send(qmp, "query-pci", NULL, + pci_del_callback, id, qmp->timeout); + if (rc != 1) { + break; + } + sleep(1); + } while (retry++ < 5); + + if (rc != 0) { + LOGD(WARN, qmp->domid, + "device model may not complete removing device %s", id); + } + } + + libxl__qmp_close(qmp); + return rc; } int libxl__qmp_pci_del(libxl__gc *gc, int domid, libxl_device_pci *pcidev)