From patchwork Wed Dec 6 17:23:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13482090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 75632C10F05 for ; Wed, 6 Dec 2023 17:25:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rAvcs-00016Q-HF; Wed, 06 Dec 2023 12:23:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvcq-00015o-Hl for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:23:56 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvco-0003Yw-IW for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:23:55 -0500 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6EYRwR014599; Wed, 6 Dec 2023 17:23:52 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=54WE3zIc9WakuDR0Wl+eRiPQLJuKyyI4IJzRdInpsHI=; b=hqbw0IvC5pe9eSNtOnkvS77IgkONZobeFOTsJkZKmPqKMJJD5fP+/AgGs7vJ/rzB9rNB uk//AMlk0VGjy8MRDc9JUnCLvjU6Pb/OfTsHUACWQN8AeNMpYIzKYDp2+jAwvzFw11v+ TO57rajblvcsGbRxeQq7sGJSe8G0Z54H5iO6VixF7juvPJQ79DZqGS0DGJl1o0SfsxUr jrSalDDofVjN/w0dYfyWFqZtPKceR8a1Dk2U6DGmf1u+LXlwaSdDrBO+C9FFHPZGjzFQ N3sKEg+W6jq/5lGddXzBctk5cXh3B26KuLqdjUdEZDnxMTnFq8dGYoODmK3UmM+kvYis 4A== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdc19xpu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:23:52 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6GkgLX037791; Wed, 6 Dec 2023 17:23:51 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3utanc79w3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:23:45 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B6HMwkK002907; Wed, 6 Dec 2023 17:23:45 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3utanc79rv-2; Wed, 06 Dec 2023 17:23:45 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Juan Quintela , Peter Xu , Paolo Bonzini , Thomas Huth , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Fabiano Rosas , Leonardo Bras , Steve Sistare Subject: [PATCH V7 01/12] cpus: vm_was_suspended Date: Wed, 6 Dec 2023 09:23:26 -0800 Message-Id: <1701883417-356268-2-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> References: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-06_15,2023-12-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 malwarescore=0 bulkscore=0 mlxlogscore=999 phishscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312060140 X-Proofpoint-GUID: tj45wyInZKSCC3ka12a9zDPm6iY9LhMQ X-Proofpoint-ORIG-GUID: tj45wyInZKSCC3ka12a9zDPm6iY9LhMQ Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add a state variable to remember if a vm previously transitioned into a suspended state. Signed-off-by: Steve Sistare Reviewed-by: Peter Xu --- include/sysemu/runstate.h | 2 ++ system/cpus.c | 15 +++++++++++++++ 2 files changed, 17 insertions(+) diff --git a/include/sysemu/runstate.h b/include/sysemu/runstate.h index c8c2bd8..88a67e2 100644 --- a/include/sysemu/runstate.h +++ b/include/sysemu/runstate.h @@ -51,6 +51,8 @@ int vm_prepare_start(bool step_pending); int vm_stop(RunState state); int vm_stop_force_state(RunState state); int vm_shutdown(void); +void vm_set_suspended(bool suspended); +bool vm_get_suspended(void); typedef enum WakeupReason { /* Always keep QEMU_WAKEUP_REASON_NONE = 0 */ diff --git a/system/cpus.c b/system/cpus.c index a444a74..9f631ab 100644 --- a/system/cpus.c +++ b/system/cpus.c @@ -259,6 +259,21 @@ void cpu_interrupt(CPUState *cpu, int mask) } } +/* + * True if the vm was previously suspended, and has not been woken or reset. + */ +static int vm_was_suspended; + +void vm_set_suspended(bool suspended) +{ + vm_was_suspended = suspended; +} + +bool vm_get_suspended(void) +{ + return vm_was_suspended; +} + static int do_vm_stop(RunState state, bool send_stop) { int ret = 0; From patchwork Wed Dec 6 17:23:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13482091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0462CC4167B for ; Wed, 6 Dec 2023 17:25:44 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rAvcy-00017b-Vh; Wed, 06 Dec 2023 12:24:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvcx-000174-JY for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:03 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvcv-0003eL-Ac for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:03 -0500 Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6EYmru004992; Wed, 6 Dec 2023 17:24:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=4Zc2LVEgq+4U//t2CEw5FDFAE3EppjozVvjLAIWMH8Y=; b=Mna5KRQJJdNBs4AjY53NHggUEZDgB5fMFFGaR+3AC/9Y0EFA5Ehr57qHmAHF59VfgjKV VLFx0sOrxNSLYffzUovwCrP5iGmotaBszBTp59rna+bDw7RGIIRRpqWN1G67zXvFRWgt 0Y0//9yRBq5KNXUlzYQEWXO90Nmum/pwPEnDcUEmw/bN9JZ7dOId/a+j2akihbSRhDm2 RckzM2Bkk43eEVLRh76TmqVmvb4jYK/4EPkEfSbFwAQtn1QyTcRmnhf0wY2krTmAHTCe 2PcJjogRABsV/Bf75KcY35aKjfUt5oosHHeZLvz9jMFUgAvRB9NiMn6y1bMCJyKQkh7p /g== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd0hj0ug-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:23:59 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6GnhkF037795; Wed, 6 Dec 2023 17:23:58 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3utanc79wv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:23:47 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B6HMwkM002907; Wed, 6 Dec 2023 17:23:46 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3utanc79rv-3; Wed, 06 Dec 2023 17:23:46 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Juan Quintela , Peter Xu , Paolo Bonzini , Thomas Huth , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Fabiano Rosas , Leonardo Bras , Steve Sistare Subject: [PATCH V7 02/12] cpus: stop vm in suspended runstate Date: Wed, 6 Dec 2023 09:23:27 -0800 Message-Id: <1701883417-356268-3-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> References: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-06_15,2023-12-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 malwarescore=0 bulkscore=0 mlxlogscore=999 phishscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312060140 X-Proofpoint-GUID: dMu3hKz6r_Ywd8k1BR0hatHfB2woCVPG X-Proofpoint-ORIG-GUID: dMu3hKz6r_Ywd8k1BR0hatHfB2woCVPG Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Currently, a vm in the suspended state is not completely stopped. The VCPUs have been paused, but the cpu clock still runs, and runstate notifiers for the transition to stopped have not been called. This causes problems for live migration. Stale cpu timers_state is saved to the migration stream, causing time errors in the guest when it wakes from suspend, and state that would have been modified by runstate notifiers is wrong. Modify vm_stop to completely stop the vm if the current state is suspended, transition to RUN_STATE_PAUSED, and remember that the machine was suspended. Modify vm_start to restore the suspended state. This affects all callers of vm_stop and vm_start, notably, the qapi stop and cont commands. For example: (qemu) info status VM status: paused (suspended) (qemu) stop (qemu) info status VM status: paused (qemu) system_wakeup Error: Unable to wake up: guest is not in suspended state (qemu) cont (qemu) info status VM status: paused (suspended) (qemu) system_wakeup (qemu) info status VM status: running Suggested-by: Peter Xu Signed-off-by: Steve Sistare Reviewed-by: Peter Xu --- include/sysemu/runstate.h | 5 +++++ qapi/misc.json | 10 ++++++++-- system/cpus.c | 23 +++++++++++++++-------- system/runstate.c | 3 +++ 4 files changed, 31 insertions(+), 10 deletions(-) diff --git a/include/sysemu/runstate.h b/include/sysemu/runstate.h index 88a67e2..867e53c 100644 --- a/include/sysemu/runstate.h +++ b/include/sysemu/runstate.h @@ -40,6 +40,11 @@ static inline bool shutdown_caused_by_guest(ShutdownCause cause) return cause >= SHUTDOWN_CAUSE_GUEST_SHUTDOWN; } +static inline bool runstate_is_live(RunState state) +{ + return state == RUN_STATE_RUNNING || state == RUN_STATE_SUSPENDED; +} + void vm_start(void); /** diff --git a/qapi/misc.json b/qapi/misc.json index cda2eff..efb8d44 100644 --- a/qapi/misc.json +++ b/qapi/misc.json @@ -134,7 +134,7 @@ ## # @stop: # -# Stop all guest VCPU execution. +# Stop all guest VCPU and VM execution. # # Since: 0.14 # @@ -143,6 +143,9 @@ # the guest remains paused once migration finishes, as if the -S # option was passed on the command line. # +# In the "suspended" state, it will completely stop the VM and +# cause a transition to the "paused" state. (Since 9.0) +# # Example: # # -> { "execute": "stop" } @@ -153,7 +156,7 @@ ## # @cont: # -# Resume guest VCPU execution. +# Resume guest VCPU and VM execution. # # Since: 0.14 # @@ -165,6 +168,9 @@ # guest starts once migration finishes, removing the effect of the # -S command line option if it was passed. # +# If the VM was previously suspended, and not been reset or woken, +# this command will transition back to the "suspended" state. (Since 9.0) +# # Example: # # -> { "execute": "cont" } diff --git a/system/cpus.c b/system/cpus.c index 9f631ab..f162435 100644 --- a/system/cpus.c +++ b/system/cpus.c @@ -277,11 +277,15 @@ bool vm_get_suspended(void) static int do_vm_stop(RunState state, bool send_stop) { int ret = 0; + RunState oldstate = runstate_get(); - if (runstate_is_running()) { + if (runstate_is_live(oldstate)) { + vm_was_suspended = (oldstate == RUN_STATE_SUSPENDED); runstate_set(state); cpu_disable_ticks(); - pause_all_vcpus(); + if (oldstate == RUN_STATE_RUNNING) { + pause_all_vcpus(); + } vm_state_notify(0, state); if (send_stop) { qapi_event_send_stop(); @@ -694,11 +698,13 @@ int vm_stop(RunState state) /** * Prepare for (re)starting the VM. - * Returns -1 if the vCPUs are not to be restarted (e.g. if they are already - * running or in case of an error condition), 0 otherwise. + * Returns 0 if the vCPUs should be restarted, -1 on an error condition, + * and 1 otherwise. */ int vm_prepare_start(bool step_pending) { + int ret = vm_was_suspended ? 1 : 0; + RunState state = vm_was_suspended ? RUN_STATE_SUSPENDED : RUN_STATE_RUNNING; RunState requested; qemu_vmstop_requested(&requested); @@ -729,9 +735,10 @@ int vm_prepare_start(bool step_pending) qapi_event_send_resume(); cpu_enable_ticks(); - runstate_set(RUN_STATE_RUNNING); - vm_state_notify(1, RUN_STATE_RUNNING); - return 0; + runstate_set(state); + vm_state_notify(1, state); + vm_was_suspended = false; + return ret; } void vm_start(void) @@ -745,7 +752,7 @@ void vm_start(void) current state is forgotten forever */ int vm_stop_force_state(RunState state) { - if (runstate_is_running()) { + if (runstate_is_live(runstate_get())) { return vm_stop(state); } else { int ret; diff --git a/system/runstate.c b/system/runstate.c index ea9d6c2..e2fa204 100644 --- a/system/runstate.c +++ b/system/runstate.c @@ -108,6 +108,7 @@ static const RunStateTransition runstate_transitions_def[] = { { RUN_STATE_PAUSED, RUN_STATE_POSTMIGRATE }, { RUN_STATE_PAUSED, RUN_STATE_PRELAUNCH }, { RUN_STATE_PAUSED, RUN_STATE_COLO}, + { RUN_STATE_PAUSED, RUN_STATE_SUSPENDED}, { RUN_STATE_POSTMIGRATE, RUN_STATE_RUNNING }, { RUN_STATE_POSTMIGRATE, RUN_STATE_FINISH_MIGRATE }, @@ -161,6 +162,7 @@ static const RunStateTransition runstate_transitions_def[] = { { RUN_STATE_SUSPENDED, RUN_STATE_FINISH_MIGRATE }, { RUN_STATE_SUSPENDED, RUN_STATE_PRELAUNCH }, { RUN_STATE_SUSPENDED, RUN_STATE_COLO}, + { RUN_STATE_SUSPENDED, RUN_STATE_PAUSED}, { RUN_STATE_WATCHDOG, RUN_STATE_RUNNING }, { RUN_STATE_WATCHDOG, RUN_STATE_FINISH_MIGRATE }, @@ -502,6 +504,7 @@ void qemu_system_reset(ShutdownCause reason) qapi_event_send_reset(shutdown_caused_by_guest(reason), reason); } cpu_synchronize_all_post_reset(); + vm_set_suspended(false); } /* From patchwork Wed Dec 6 17:23:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13482093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2F258C4167B for ; Wed, 6 Dec 2023 17:26:53 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rAvdM-0001PI-JW; Wed, 06 Dec 2023 12:24:28 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvdJ-0001Ny-1F for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:25 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvdF-0003iE-NS for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:24 -0500 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6EY31W022603; Wed, 6 Dec 2023 17:24:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=ZT7aZFi5hiX+6FcwJHLVWgrKIok3nx5ofBwRwPQtJy8=; b=ni8RfH947SLmTPodsxZa09B6lLsbr3JDRkTQnc+IYEnkqSJZEP53nqo4K8TEnOQiYS1s dBE7L9pw0//xfz5ZozvFL67/n6ZwRdUyqMLeTGHkKokbCUHWc0MWs2sMgtsGW1XAIzgq OYJUDWrf71srqlA41cxRMeoTg4chKxtZXCugvnN1DPJAFJ5cfCshZ3f0oKWdrz6VvcZL wH0PzvoethZG7cMSVr2XmuSaHQbRcQBkfwdLIMiaqXki1oLN3RsR1fxDd/wbbYpHqrQk Jv/Kdg+FIXVWQFd7LEESvypEVITaGe78KijeZxe53CVBVIQCsz+g7wjuCwKkR87Fl5+G 0w== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd0mj0k4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:24:09 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6GVbQI037850; Wed, 6 Dec 2023 17:23:59 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3utanc7a0r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:23:58 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B6HMwkS002907; Wed, 6 Dec 2023 17:23:50 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3utanc79rv-6; Wed, 06 Dec 2023 17:23:50 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Juan Quintela , Peter Xu , Paolo Bonzini , Thomas Huth , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Fabiano Rosas , Leonardo Bras , Steve Sistare Subject: [PATCH V7 05/12] migration: propagate suspended runstate Date: Wed, 6 Dec 2023 09:23:30 -0800 Message-Id: <1701883417-356268-6-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> References: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-06_16,2023-12-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 malwarescore=0 bulkscore=0 mlxlogscore=999 phishscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312060140 X-Proofpoint-GUID: r8tHlFWcLlc_GTZlFPOHiOh3qZzoi_uK X-Proofpoint-ORIG-GUID: r8tHlFWcLlc_GTZlFPOHiOh3qZzoi_uK Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org If the outgoing machine was previously suspended, propagate that to the incoming side via global_state, so a subsequent vm_start restores the suspended state. To maintain backward and forward compatibility, reclaim some space from the runstate member. Signed-off-by: Steve Sistare Reviewed-by: Peter Xu --- migration/global_state.c | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/migration/global_state.c b/migration/global_state.c index 4e2a9d8..d4f61a1 100644 --- a/migration/global_state.c +++ b/migration/global_state.c @@ -22,7 +22,16 @@ typedef struct { uint32_t size; - uint8_t runstate[100]; + + /* + * runstate was 100 bytes, zero padded, but we trimmed it to add a + * few fields and maintain backwards compatibility. + */ + uint8_t runstate[32]; + uint8_t has_vm_was_suspended; + uint8_t vm_was_suspended; + uint8_t unused[66]; + RunState state; bool received; } GlobalState; @@ -35,6 +44,10 @@ static void global_state_do_store(RunState state) assert(strlen(state_str) < sizeof(global_state.runstate)); strpadcpy((char *)global_state.runstate, sizeof(global_state.runstate), state_str, '\0'); + global_state.has_vm_was_suspended = true; + global_state.vm_was_suspended = vm_get_suspended(); + + memset(global_state.unused, 0, sizeof(global_state.unused)); } void global_state_store(void) @@ -68,6 +81,12 @@ static bool global_state_needed(void *opaque) return true; } + /* If the suspended state must be remembered, it is needed */ + + if (vm_get_suspended()) { + return true; + } + /* If state is running or paused, it is not needed */ if (strcmp(runstate, "running") == 0 || @@ -85,6 +104,7 @@ static int global_state_post_load(void *opaque, int version_id) Error *local_err = NULL; int r; char *runstate = (char *)s->runstate; + bool vm_was_suspended = s->has_vm_was_suspended && s->vm_was_suspended; s->received = true; trace_migrate_global_state_post_load(runstate); @@ -93,7 +113,7 @@ static int global_state_post_load(void *opaque, int version_id) sizeof(s->runstate)) == sizeof(s->runstate)) { /* * This condition should never happen during migration, because - * all runstate names are shorter than 100 bytes (the size of + * all runstate names are shorter than 32 bytes (the size of * s->runstate). However, a malicious stream could overflow * the qapi_enum_parse() call, so we force the last character * to a NUL byte. @@ -110,6 +130,14 @@ static int global_state_post_load(void *opaque, int version_id) } s->state = r; + /* + * global_state is saved on the outgoing side before forcing a stopped + * state, so it may have saved state=suspended and vm_was_suspended=0. + * Now we are in a paused state, and when we later call vm_start, it must + * restore the suspended state, so we must set vm_was_suspended=1 here. + */ + vm_set_suspended(vm_was_suspended || r == RUN_STATE_SUSPENDED); + return 0; } @@ -134,6 +162,9 @@ static const VMStateDescription vmstate_globalstate = { .fields = (VMStateField[]) { VMSTATE_UINT32(size, GlobalState), VMSTATE_BUFFER(runstate, GlobalState), + VMSTATE_UINT8(has_vm_was_suspended, GlobalState), + VMSTATE_UINT8(vm_was_suspended, GlobalState), + VMSTATE_BUFFER(unused, GlobalState), VMSTATE_END_OF_LIST() }, }; From patchwork Wed Dec 6 17:23:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13482089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 90EFDC4167B for ; Wed, 6 Dec 2023 17:25:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rAvd8-00019z-97; Wed, 06 Dec 2023 12:24:14 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvd6-00019M-1t for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:12 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvd4-0003hh-1l for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:11 -0500 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6EYjWK014754; Wed, 6 Dec 2023 17:24:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=Tdz/0665DIlKQYsKJPHoEEW9YQgWa4jTt33Mn6oJUDU=; b=maD3mz+cohdc3QqjldBOBAm/wMA4vJw/+leFyO7gRgm2k5jveCFqYgGJl8u4bWJNVnRk U5R9iKpARb5UBFzqjM1Jes1UDxzDkjeUJDe4JZp2/8RLZZluTbiAEge3CT4EFL/LNU/C mS7lwV0+fWgLBUxjSYGLWhoiFnlBGyZh6bxJNOngg7CnI0a7HQkmD7xie+Em35/kylAR ZXQe87JttLOCMAaymcc6INyGgLHhWcss6EgMa3hNk2g+26myDtmeG6MDNaiYbPPjsxVl 5I8Tcd3f4w/qq3NNMspyAuP28CZJdOxU8s69oze51I/j1Ntl3UmSTW3hbdWhWNQEUMJ+ Mw== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdda1y99-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:24:07 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6GnOh7038489; Wed, 6 Dec 2023 17:23:59 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3utanc7a6q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:23:59 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B6HMwkU002907; Wed, 6 Dec 2023 17:23:59 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3utanc79rv-7; Wed, 06 Dec 2023 17:23:58 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Juan Quintela , Peter Xu , Paolo Bonzini , Thomas Huth , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Fabiano Rosas , Leonardo Bras , Steve Sistare Subject: [PATCH V7 06/12] migration: preserve suspended runstate Date: Wed, 6 Dec 2023 09:23:31 -0800 Message-Id: <1701883417-356268-7-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> References: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-06_16,2023-12-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 malwarescore=0 bulkscore=0 mlxlogscore=999 phishscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312060140 X-Proofpoint-GUID: nyUvwgaWqTJ0KkClgeUSlEM0LJHxnMVF X-Proofpoint-ORIG-GUID: nyUvwgaWqTJ0KkClgeUSlEM0LJHxnMVF Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org A guest that is migrated in the suspended state automaticaly wakes and continues execution. This is wrong; the guest should end migration in the same state it started. The root cause is that the outgoing migration code automatically wakes the guest, then saves the RUNNING runstate in global_state_store(), hence the incoming migration code thinks the guest is running and continues the guest if autostart is true. On the outgoing side, delete the call to qemu_system_wakeup_request(). Now that vm_stop completely stops a vm in the suspended state (from the preceding patches), the existing call to vm_stop_force_state is sufficient to correctly migrate all vmstate. On the incoming side, call vm_start if the pre-migration state was running or suspended. For the latter, vm_start correctly restores the suspended state, and a future system_wakeup monitor request will cause the vm to resume running. Signed-off-by: Steve Sistare Reviewed-by: Peter Xu --- migration/migration.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 3ce04b2..8124811 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -604,7 +604,7 @@ static void process_incoming_migration_bh(void *opaque) */ if (!migrate_late_block_activate() || (autostart && (!global_state_received() || - global_state_get_runstate() == RUN_STATE_RUNNING))) { + runstate_is_live(global_state_get_runstate())))) { /* Make sure all file formats throw away their mutable metadata. * If we get an error here, just don't restart the VM yet. */ bdrv_activate_all(&local_err); @@ -628,7 +628,7 @@ static void process_incoming_migration_bh(void *opaque) dirty_bitmap_mig_before_vm_start(); if (!global_state_received() || - global_state_get_runstate() == RUN_STATE_RUNNING) { + runstate_is_live(global_state_get_runstate())) { if (autostart) { vm_start(); } else { @@ -2416,7 +2416,6 @@ static int postcopy_start(MigrationState *ms, Error **errp) migration_downtime_start(ms); - qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER, NULL); global_state_store(); ret = migration_stop_vm(RUN_STATE_FINISH_MIGRATE); if (ret < 0) { @@ -2615,7 +2614,6 @@ static int migration_completion_precopy(MigrationState *s, qemu_mutex_lock_iothread(); migration_downtime_start(s); - qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER, NULL); s->vm_old_state = runstate_get(); global_state_store(); @@ -3136,7 +3134,7 @@ static void migration_iteration_finish(MigrationState *s) case MIGRATION_STATUS_FAILED: case MIGRATION_STATUS_CANCELLED: case MIGRATION_STATUS_CANCELLING: - if (s->vm_old_state == RUN_STATE_RUNNING) { + if (runstate_is_live(s->vm_old_state)) { if (!runstate_check(RUN_STATE_SHUTDOWN)) { vm_start(); } From patchwork Wed Dec 6 17:23:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13482087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 980BEC4167B for ; Wed, 6 Dec 2023 17:25:16 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rAvd3-00018R-T0; Wed, 06 Dec 2023 12:24:09 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvd1-00018G-DD for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:07 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvcz-0003ez-Gy for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:07 -0500 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6EYkJx014763; Wed, 6 Dec 2023 17:24:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=7r47Pue9XL+J9wl9bbIYyHu5FgRAV/eyOGMK/N66L0o=; b=oPevSKtDkZWEeI/Dy7wHBNNy2eS1ceqK0Kcd+RSwHSNmN+hLTKbGL4LF56KNkEMBGuZm MzgLFBiCeE9cZH9rR8t5dL64xw72mThnkKyvUYbiJCvwAj+OuRtGdLsPQV7fmHnH4TGT 7dkCfwfLSaxWGyLATmDV6FEwnZQhHLsp7GEMqKwCwffS9iaBbFA7ArKf4UAyckkqzPcX uKJSLTcElH/jy8w8htg2nibUz7iKv3dwFFUMDFSbU9JzquNrbhQ4ybs6yaPVwcWjvysI 1U10hZHjvwSqqBM991x1HETcrLN5vn0VvlEz2h6xlbJ/HsKtKsleesGklW9DbU/HDx37 gQ== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdda1y9d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:24:03 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6GiPRF037847; Wed, 6 Dec 2023 17:24:01 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3utanc7a86-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:24:01 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B6HMwkW002907; Wed, 6 Dec 2023 17:24:00 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3utanc79rv-8; Wed, 06 Dec 2023 17:24:00 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Juan Quintela , Peter Xu , Paolo Bonzini , Thomas Huth , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Fabiano Rosas , Leonardo Bras , Steve Sistare Subject: [PATCH V7 07/12] migration: preserve suspended for snapshot Date: Wed, 6 Dec 2023 09:23:32 -0800 Message-Id: <1701883417-356268-8-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> References: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-06_16,2023-12-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 malwarescore=0 bulkscore=0 mlxlogscore=999 phishscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312060140 X-Proofpoint-GUID: VWbflXzBkVNoGAChzN6woP40V4SU4YuW X-Proofpoint-ORIG-GUID: VWbflXzBkVNoGAChzN6woP40V4SU4YuW Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Restoring a snapshot can break a suspended guest. Snapshots suffer from the same suspended-state issues that affect live migration, plus they must handle an additional problematic scenario, which is that a running vm must remain running if it loads a suspended snapshot. To save, the existing vm_stop call now completely stops the suspended state. Finish with vm_resume to leave the vm in the state it had prior to the save, correctly restoring the suspended state. To load, if the snapshot is not suspended, then vm_stop + vm_resume correctly handles all states, and leaves the vm in the state it had prior to the load. However, if the snapshot is suspended, restoration is trickier. First, call vm_resume to restore the state to suspended so the current state matches the saved state. Then, if the pre-load state is running, call wakeup to resume running. Prior to these changes, the vm_stop to RUN_STATE_SAVE_VM and RUN_STATE_RESTORE_VM did not change runstate if the current state was suspended, but now it does, so allow these transitions. Signed-off-by: Steve Sistare Reviewed-by: Peter Xu --- include/migration/snapshot.h | 7 +++++++ migration/migration-hmp-cmds.c | 8 +++++--- migration/savevm.c | 23 +++++++++++++---------- system/runstate.c | 5 +++++ system/vl.c | 2 ++ 5 files changed, 32 insertions(+), 13 deletions(-) diff --git a/include/migration/snapshot.h b/include/migration/snapshot.h index e72083b..9e4dcaa 100644 --- a/include/migration/snapshot.h +++ b/include/migration/snapshot.h @@ -16,6 +16,7 @@ #define QEMU_MIGRATION_SNAPSHOT_H #include "qapi/qapi-builtin-types.h" +#include "qapi/qapi-types-run-state.h" /** * save_snapshot: Save an internal snapshot. @@ -61,4 +62,10 @@ bool delete_snapshot(const char *name, bool has_devices, strList *devices, Error **errp); +/** + * load_snapshot_resume: Restore runstate after loading snapshot. + * @state: state to restore + */ +void load_snapshot_resume(RunState state); + #endif diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index 86ae832..c8d70bc 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -399,15 +399,17 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) void hmp_loadvm(Monitor *mon, const QDict *qdict) { - int saved_vm_running = runstate_is_running(); + RunState saved_state = runstate_get(); + const char *name = qdict_get_str(qdict, "name"); Error *err = NULL; vm_stop(RUN_STATE_RESTORE_VM); - if (load_snapshot(name, NULL, false, NULL, &err) && saved_vm_running) { - vm_start(); + if (load_snapshot(name, NULL, false, NULL, &err)) { + load_snapshot_resume(saved_state); } + hmp_handle_error(mon, err); } diff --git a/migration/savevm.c b/migration/savevm.c index eec5503..26866f0 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -3046,7 +3046,7 @@ bool save_snapshot(const char *name, bool overwrite, const char *vmstate, QEMUSnapshotInfo sn1, *sn = &sn1; int ret = -1, ret2; QEMUFile *f; - int saved_vm_running; + RunState saved_state = runstate_get(); uint64_t vm_state_size; g_autoptr(GDateTime) now = g_date_time_new_now_local(); AioContext *aio_context; @@ -3094,8 +3094,6 @@ bool save_snapshot(const char *name, bool overwrite, const char *vmstate, } aio_context = bdrv_get_aio_context(bs); - saved_vm_running = runstate_is_running(); - global_state_store(); vm_stop(RUN_STATE_SAVE_VM); @@ -3163,9 +3161,7 @@ bool save_snapshot(const char *name, bool overwrite, const char *vmstate, bdrv_drain_all_end(); - if (saved_vm_running) { - vm_start(); - } + vm_resume(saved_state); return ret == 0; } @@ -3339,6 +3335,14 @@ err_drain: return false; } +void load_snapshot_resume(RunState state) +{ + vm_resume(state); + if (state == RUN_STATE_RUNNING && runstate_get() == RUN_STATE_SUSPENDED) { + qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER, &error_abort); + } +} + bool delete_snapshot(const char *name, bool has_devices, strList *devices, Error **errp) { @@ -3403,16 +3407,15 @@ static void snapshot_load_job_bh(void *opaque) { Job *job = opaque; SnapshotJob *s = container_of(job, SnapshotJob, common); - int orig_vm_running; + RunState orig_state = runstate_get(); job_progress_set_remaining(&s->common, 1); - orig_vm_running = runstate_is_running(); vm_stop(RUN_STATE_RESTORE_VM); s->ret = load_snapshot(s->tag, s->vmstate, true, s->devices, s->errp); - if (s->ret && orig_vm_running) { - vm_start(); + if (s->ret) { + load_snapshot_resume(orig_state); } job_progress_update(&s->common, 1); diff --git a/system/runstate.c b/system/runstate.c index e2fa204..ca9eb54 100644 --- a/system/runstate.c +++ b/system/runstate.c @@ -77,6 +77,7 @@ typedef struct { static const RunStateTransition runstate_transitions_def[] = { { RUN_STATE_PRELAUNCH, RUN_STATE_INMIGRATE }, + { RUN_STATE_PRELAUNCH, RUN_STATE_SUSPENDED }, { RUN_STATE_DEBUG, RUN_STATE_RUNNING }, { RUN_STATE_DEBUG, RUN_STATE_FINISH_MIGRATE }, @@ -132,6 +133,7 @@ static const RunStateTransition runstate_transitions_def[] = { { RUN_STATE_RESTORE_VM, RUN_STATE_RUNNING }, { RUN_STATE_RESTORE_VM, RUN_STATE_PRELAUNCH }, + { RUN_STATE_RESTORE_VM, RUN_STATE_SUSPENDED }, { RUN_STATE_COLO, RUN_STATE_RUNNING }, { RUN_STATE_COLO, RUN_STATE_PRELAUNCH }, @@ -150,6 +152,7 @@ static const RunStateTransition runstate_transitions_def[] = { { RUN_STATE_RUNNING, RUN_STATE_COLO}, { RUN_STATE_SAVE_VM, RUN_STATE_RUNNING }, + { RUN_STATE_SAVE_VM, RUN_STATE_SUSPENDED }, { RUN_STATE_SHUTDOWN, RUN_STATE_PAUSED }, { RUN_STATE_SHUTDOWN, RUN_STATE_FINISH_MIGRATE }, @@ -163,6 +166,8 @@ static const RunStateTransition runstate_transitions_def[] = { { RUN_STATE_SUSPENDED, RUN_STATE_PRELAUNCH }, { RUN_STATE_SUSPENDED, RUN_STATE_COLO}, { RUN_STATE_SUSPENDED, RUN_STATE_PAUSED}, + { RUN_STATE_SUSPENDED, RUN_STATE_SAVE_VM }, + { RUN_STATE_SUSPENDED, RUN_STATE_RESTORE_VM }, { RUN_STATE_WATCHDOG, RUN_STATE_RUNNING }, { RUN_STATE_WATCHDOG, RUN_STATE_FINISH_MIGRATE }, diff --git a/system/vl.c b/system/vl.c index 2bcd9ef..2fbbbba 100644 --- a/system/vl.c +++ b/system/vl.c @@ -2706,7 +2706,9 @@ void qmp_x_exit_preconfig(Error **errp) qemu_machine_creation_done(); if (loadvm) { + RunState state = autostart ? RUN_STATE_RUNNING : runstate_get(); load_snapshot(loadvm, NULL, false, NULL, &error_fatal); + load_snapshot_resume(state); } if (replay_mode != REPLAY_MODE_NONE) { replay_vmstate_init(); From patchwork Wed Dec 6 17:23:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13482092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BE2F3C10DC1 for ; Wed, 6 Dec 2023 17:26:02 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rAvd7-00019j-LB; Wed, 06 Dec 2023 12:24:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvd5-000193-6B for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:11 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvd2-0003gg-6G for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:10 -0500 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6EYjWJ014754; Wed, 6 Dec 2023 17:24:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=16Hz/0QpTsbArtLyqdF3glLAD2N1aY4+squSC785lPc=; b=c0dkXayx/q7VjrIdmU1LW0CxMEMvQF0QYSSYcv/Ju8QyTOFzNqj/eX+Sh9GIBDjEewCT 0LOLM97DDPzgFjaNuRhSRJYfhNlMkmi7hAqp+wpJt47dpIIZuOUwZ7oHPCaqXS0NqLjH qSnpwzCdmfVyz9fmSyX1nCFc8tga2oJKXIyuTSL3c/lVUqrjukaImOnUJtpfCD/o5DZf qKqB/9ZnzNF+/WxWPfaycr6D9OuaaIYY+4GmhhAY7u/K7cG7bn9Fr7aoM33yp/nWkex9 h7P3E23JfaPAEBYD0dJHuItPjAqI105qJvNN6882c2HeYv0OR5bWgz3b3HRY46lT+DpN CA== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdda1y9g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:24:06 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6GVbQS037850; Wed, 6 Dec 2023 17:24:02 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3utanc7a8y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:24:02 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B6HMwkY002907; Wed, 6 Dec 2023 17:24:02 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3utanc79rv-9; Wed, 06 Dec 2023 17:24:01 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Juan Quintela , Peter Xu , Paolo Bonzini , Thomas Huth , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Fabiano Rosas , Leonardo Bras , Steve Sistare Subject: [PATCH V7 08/12] migration: preserve suspended for bg_migration Date: Wed, 6 Dec 2023 09:23:33 -0800 Message-Id: <1701883417-356268-9-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> References: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-06_16,2023-12-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 malwarescore=0 bulkscore=0 mlxlogscore=999 phishscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312060140 X-Proofpoint-GUID: ljTOkteSg0UnJO69nWL_DHycDCtuVQUy X-Proofpoint-ORIG-GUID: ljTOkteSg0UnJO69nWL_DHycDCtuVQUy Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Do not wake a suspended guest during bg_migration, and restore the prior state at finish rather than unconditionally running. Allow the additional state transitions that occur. Signed-off-by: Steve Sistare Reviewed-by: Fabiano Rosas Reviewed-by: Peter Xu --- migration/migration.c | 7 +------ system/runstate.c | 1 + 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 8124811..2cc7e2a 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -3390,7 +3390,7 @@ static void bg_migration_vm_start_bh(void *opaque) qemu_bh_delete(s->vm_start_bh); s->vm_start_bh = NULL; - vm_start(); + vm_resume(s->vm_old_state); migration_downtime_end(s); } @@ -3462,11 +3462,6 @@ static void *bg_migration_thread(void *opaque) qemu_mutex_lock_iothread(); - /* - * If VM is currently in suspended state, then, to make a valid runstate - * transition in vm_stop_force_state() we need to wakeup it up. - */ - qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER, NULL); s->vm_old_state = runstate_get(); global_state_store(); diff --git a/system/runstate.c b/system/runstate.c index ca9eb54..621a023 100644 --- a/system/runstate.c +++ b/system/runstate.c @@ -168,6 +168,7 @@ static const RunStateTransition runstate_transitions_def[] = { { RUN_STATE_SUSPENDED, RUN_STATE_PAUSED}, { RUN_STATE_SUSPENDED, RUN_STATE_SAVE_VM }, { RUN_STATE_SUSPENDED, RUN_STATE_RESTORE_VM }, + { RUN_STATE_SUSPENDED, RUN_STATE_SHUTDOWN }, { RUN_STATE_WATCHDOG, RUN_STATE_RUNNING }, { RUN_STATE_WATCHDOG, RUN_STATE_FINISH_MIGRATE }, From patchwork Wed Dec 6 17:23:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13482085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 16BE4C4167B for ; Wed, 6 Dec 2023 17:24:56 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rAvd5-00018s-2Z; Wed, 06 Dec 2023 12:24:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvd3-00018i-7f for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:09 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvd0-0003gS-Vb for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:08 -0500 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6EYQcl014591; Wed, 6 Dec 2023 17:24:05 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2023-11-20; bh=poJGV6l16x1wEB9PjmRYReQy9aqyUwxSHd5jrD+iWpY=; b=lD/V9n439E+bkqf1+CTc7Gg13vxjHJ79d33o3XNLrjaWUSxuYzPXUlLHKnIpHKrdepiP 8Tw/Rmvs2ydofqiMqMiCaU0fRhRj7V7wGI3Yn+tMEAcPL6xRdlMdI34dl3OWbqZTuxMx ho6hC+b+XVSYreRgKu1B3rmU/NojcWwrw9QIrvw9tJlY+8PL0kJeSaXq6Mtq3c8wrfmC tMZklFVv+XZPyEENaJXxmtOYSbLeft9vpOBg8Le0LmNh6Ji0tscO+5dyubwaC0lo9b4E GqJ/bqc8Z2Ot17UgZ6GVeRBQjHIS2B5RzEtAf/+vuZ9iC1U/KcEfTipSK4vtFpvbWxbW Ug== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdc19xqt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:24:05 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6GwOtV037820; Wed, 6 Dec 2023 17:24:04 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3utanc7a9y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:24:04 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B6HMwka002907; Wed, 6 Dec 2023 17:24:03 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3utanc79rv-10; Wed, 06 Dec 2023 17:24:03 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Juan Quintela , Peter Xu , Paolo Bonzini , Thomas Huth , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Fabiano Rosas , Leonardo Bras , Steve Sistare Subject: [PATCH V7 09/12] tests/qtest: migration events Date: Wed, 6 Dec 2023 09:23:34 -0800 Message-Id: <1701883417-356268-10-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> References: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-06_16,2023-12-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 malwarescore=0 bulkscore=0 mlxlogscore=999 phishscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312060140 X-Proofpoint-GUID: -l-ymZQF89YN16qSCms9OSNZ-TzlVVKM X-Proofpoint-ORIG-GUID: -l-ymZQF89YN16qSCms9OSNZ-TzlVVKM Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Define a state object to capture events seen by migration tests, to allow more events to be captured in a subsequent patch, and simplify event checking in wait_for_migration_pass. No functional change. Signed-off-by: Steve Sistare Reviewed-by: Fabiano Rosas Reviewed-by: Daniel P. Berrangé --- tests/qtest/migration-helpers.c | 24 ++++------- tests/qtest/migration-helpers.h | 9 ++-- tests/qtest/migration-test.c | 91 +++++++++++++++++++---------------------- 3 files changed, 56 insertions(+), 68 deletions(-) diff --git a/tests/qtest/migration-helpers.c b/tests/qtest/migration-helpers.c index 24fb7b3..fd3b94e 100644 --- a/tests/qtest/migration-helpers.c +++ b/tests/qtest/migration-helpers.c @@ -24,26 +24,16 @@ */ #define MIGRATION_STATUS_WAIT_TIMEOUT 120 -bool migrate_watch_for_stop(QTestState *who, const char *name, - QDict *event, void *opaque) -{ - bool *seen = opaque; - - if (g_str_equal(name, "STOP")) { - *seen = true; - return true; - } - - return false; -} - -bool migrate_watch_for_resume(QTestState *who, const char *name, +bool migrate_watch_for_events(QTestState *who, const char *name, QDict *event, void *opaque) { - bool *seen = opaque; + QTestMigrationState *state = opaque; - if (g_str_equal(name, "RESUME")) { - *seen = true; + if (g_str_equal(name, "STOP")) { + state->stop_seen = true; + return true; + } else if (g_str_equal(name, "RESUME")) { + state->resume_seen = true; return true; } diff --git a/tests/qtest/migration-helpers.h b/tests/qtest/migration-helpers.h index e31dc85..3d32699 100644 --- a/tests/qtest/migration-helpers.h +++ b/tests/qtest/migration-helpers.h @@ -15,9 +15,12 @@ #include "libqtest.h" -bool migrate_watch_for_stop(QTestState *who, const char *name, - QDict *event, void *opaque); -bool migrate_watch_for_resume(QTestState *who, const char *name, +typedef struct QTestMigrationState { + bool stop_seen; + bool resume_seen; +} QTestMigrationState; + +bool migrate_watch_for_events(QTestState *who, const char *name, QDict *event, void *opaque); G_GNUC_PRINTF(3, 4) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 0fbaa6a..05c0740 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -43,8 +43,8 @@ unsigned start_address; unsigned end_address; static bool uffd_feature_thread_id; -static bool got_src_stop; -static bool got_dst_resume; +static QTestMigrationState src_state; +static QTestMigrationState dst_state; /* * An initial 3 MB offset is used as that corresponds @@ -230,6 +230,20 @@ static void wait_for_serial(const char *side) } while (true); } +static void wait_for_stop(QTestState *who, QTestMigrationState *state) +{ + if (!state->stop_seen) { + qtest_qmp_eventwait(who, "STOP"); + } +} + +static void wait_for_resume(QTestState *who, QTestMigrationState *state) +{ + if (!state->resume_seen) { + qtest_qmp_eventwait(who, "RESUME"); + } +} + /* * It's tricky to use qemu's migration event capability with qtest, * events suddenly appearing confuse the qmp()/hmp() responses. @@ -277,21 +291,19 @@ static void read_blocktime(QTestState *who) qobject_unref(rsp_return); } +/* + * Wait for two changes in the migration pass count, but bail if we stop. + */ static void wait_for_migration_pass(QTestState *who) { - uint64_t initial_pass = get_migration_pass(who); - uint64_t pass; + uint64_t pass, prev_pass = 0, changes = 0; - /* Wait for the 1st sync */ - while (!got_src_stop && !initial_pass) { - usleep(1000); - initial_pass = get_migration_pass(who); - } - - do { + while (changes < 2 && !src_state.stop_seen) { usleep(1000); pass = get_migration_pass(who); - } while (pass == initial_pass && !got_src_stop); + changes += (pass != prev_pass); + prev_pass = pass; + } } static void check_guests_ram(QTestState *who) @@ -617,10 +629,7 @@ static void migrate_postcopy_start(QTestState *from, QTestState *to) { qtest_qmp_assert_success(from, "{ 'execute': 'migrate-start-postcopy' }"); - if (!got_src_stop) { - qtest_qmp_eventwait(from, "STOP"); - } - + wait_for_stop(from, &src_state); qtest_qmp_eventwait(to, "RESUME"); } @@ -756,8 +765,8 @@ static int test_migrate_start(QTestState **from, QTestState **to, } } - got_src_stop = false; - got_dst_resume = false; + dst_state = (QTestMigrationState) { }; + src_state = (QTestMigrationState) { }; if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) { memory_size = "150M"; @@ -848,8 +857,8 @@ static int test_migrate_start(QTestState **from, QTestState **to, if (!args->only_target) { *from = qtest_init_with_env(QEMU_ENV_SRC, cmd_source); qtest_qmp_set_event_callback(*from, - migrate_watch_for_stop, - &got_src_stop); + migrate_watch_for_events, + &src_state); } cmd_target = g_strdup_printf("-accel kvm%s -accel tcg " @@ -869,8 +878,8 @@ static int test_migrate_start(QTestState **from, QTestState **to, ignore_stderr); *to = qtest_init_with_env(QEMU_ENV_DST, cmd_target); qtest_qmp_set_event_callback(*to, - migrate_watch_for_resume, - &got_dst_resume); + migrate_watch_for_events, + &dst_state); /* * Remove shmem file immediately to avoid memory leak in test failed case. @@ -1717,9 +1726,7 @@ static void test_precopy_common(MigrateCommon *args) */ if (args->result == MIG_TEST_SUCCEED) { qtest_qmp_assert_success(from, "{ 'execute' : 'stop'}"); - if (!got_src_stop) { - qtest_qmp_eventwait(from, "STOP"); - } + wait_for_stop(from, &src_state); migrate_ensure_converge(from); } } @@ -1765,9 +1772,8 @@ static void test_precopy_common(MigrateCommon *args) */ wait_for_migration_complete(from); - if (!got_src_stop) { - qtest_qmp_eventwait(from, "STOP"); - } + wait_for_stop(from, &src_state); + } else { wait_for_migration_complete(from); /* @@ -1780,9 +1786,7 @@ static void test_precopy_common(MigrateCommon *args) qtest_qmp_assert_success(to, "{ 'execute' : 'cont'}"); } - if (!got_dst_resume) { - qtest_qmp_eventwait(to, "RESUME"); - } + wait_for_resume(to, &dst_state); wait_for_serial("dest_serial"); } @@ -1821,9 +1825,7 @@ static void test_file_common(MigrateCommon *args, bool stop_src) if (stop_src) { qtest_qmp_assert_success(from, "{ 'execute' : 'stop'}"); - if (!got_src_stop) { - qtest_qmp_eventwait(from, "STOP"); - } + wait_for_stop(from, &src_state); } if (args->result == MIG_TEST_QMP_ERROR) { @@ -1844,10 +1846,7 @@ static void test_file_common(MigrateCommon *args, bool stop_src) if (stop_src) { qtest_qmp_assert_success(to, "{ 'execute' : 'cont'}"); } - - if (!got_dst_resume) { - qtest_qmp_eventwait(to, "RESUME"); - } + wait_for_resume(to, &dst_state); wait_for_serial("dest_serial"); @@ -1966,9 +1965,7 @@ static void test_ignore_shared(void) migrate_wait_for_dirty_mem(from, to); - if (!got_src_stop) { - qtest_qmp_eventwait(from, "STOP"); - } + wait_for_stop(from, &src_state); qtest_qmp_eventwait(to, "RESUME"); @@ -2503,7 +2500,7 @@ static void test_migrate_auto_converge(void) break; } usleep(20); - g_assert_false(got_src_stop); + g_assert_false(src_state.stop_seen); } while (true); /* The first percentage of throttling should be at least init_pct */ g_assert_cmpint(percentage, >=, init_pct); @@ -2842,9 +2839,7 @@ static void test_multifd_tcp_cancel(void) migrate_ensure_converge(from); - if (!got_src_stop) { - qtest_qmp_eventwait(from, "STOP"); - } + wait_for_stop(from, &src_state); qtest_qmp_eventwait(to2, "RESUME"); wait_for_serial("dest_serial"); @@ -3177,7 +3172,7 @@ static void test_migrate_dirty_limit(void) throttle_us_per_full = read_migrate_property_int(from, "dirty-limit-throttle-time-per-round"); usleep(100); - g_assert_false(got_src_stop); + g_assert_false(src_state.stop_seen); } /* Now cancel migrate and wait for dirty limit throttle switch off */ @@ -3189,7 +3184,7 @@ static void test_migrate_dirty_limit(void) throttle_us_per_full = read_migrate_property_int(from, "dirty-limit-throttle-time-per-round"); usleep(100); - g_assert_false(got_src_stop); + g_assert_false(src_state.stop_seen); } while (throttle_us_per_full != 0 && --max_try_count); /* Assert dirty limit is not in service */ @@ -3218,7 +3213,7 @@ static void test_migrate_dirty_limit(void) throttle_us_per_full = read_migrate_property_int(from, "dirty-limit-throttle-time-per-round"); usleep(100); - g_assert_false(got_src_stop); + g_assert_false(src_state.stop_seen); } /* From patchwork Wed Dec 6 17:23:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13482088 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CC070C4167B for ; Wed, 6 Dec 2023 17:25:20 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rAvdI-0001Nf-Ur; Wed, 06 Dec 2023 12:24:24 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvdH-0001NS-NF for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:23 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvd2-0003gj-B4 for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:23 -0500 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6EYTaI014679; Wed, 6 Dec 2023 17:24:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=7eSfxs7j/v/8uF1eJVhEWFbbl0HfYQ8PuywKsoPyzTU=; b=FS/91ddhHlu8sNBvTMULARl9H759g1r/3gFjj0VU19lAYggqHof1ud2CQjfSTh02mJei ZAdgJ+QlyHuT95UsiEyb4n4y8vtzxUNM1WIQLBW0cQ6xxq3AOSkqI21+NogxKbEwTarx ZJuHuJIX510WkU/W5fXi+CLIchv/z3vuQ6MjTROqO3QAKEWSygquttdGRskVvFQPRz1g Fjy7fo7JmBWU1H+1tGgsfAVxpc1iNInIGYDwenpft+glY0j0kpmq+ylVvmWr3lMTYe2H qI5XARQQubZibFXEHP6aPFV+e159lPJL4y/3M8VHsBUvF9iZbrG9RsGgE+po/GAZjyjK hQ== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdc19xqw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:24:06 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6GhKJ5037813; Wed, 6 Dec 2023 17:24:05 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3utanc7aax-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:24:05 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B6HMwkc002907; Wed, 6 Dec 2023 17:24:04 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3utanc79rv-11; Wed, 06 Dec 2023 17:24:04 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Juan Quintela , Peter Xu , Paolo Bonzini , Thomas Huth , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Fabiano Rosas , Leonardo Bras , Steve Sistare Subject: [PATCH V7 10/12] tests/qtest: option to suspend during migration Date: Wed, 6 Dec 2023 09:23:35 -0800 Message-Id: <1701883417-356268-11-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> References: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-06_16,2023-12-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 malwarescore=0 bulkscore=0 mlxlogscore=999 phishscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312060140 X-Proofpoint-GUID: AlUUXQ3ESpryZQbxzh40IQ29NZAc77ew X-Proofpoint-ORIG-GUID: AlUUXQ3ESpryZQbxzh40IQ29NZAc77ew Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add an option to suspend the src in a-b-bootblock.S, which puts the guest in S3 state after one round of writing to memory. The option is enabled by poking a 1 into the suspend_me word in the boot block prior to starting the src vm. Generate symbol offsets in a-b-bootblock.h so that the suspend_me offset is known. Generate the bootblock for each test, because suspend_me may differ for each. Signed-off-by: Steve Sistare Acked-by: Peter Xu Reviewed-by: Fabiano Rosas --- tests/migration/i386/Makefile | 5 ++-- tests/migration/i386/a-b-bootblock.S | 50 +++++++++++++++++++++++++++++++++--- tests/migration/i386/a-b-bootblock.h | 26 +++++++++++++------ tests/qtest/migration-test.c | 12 ++++++--- 4 files changed, 77 insertions(+), 16 deletions(-) diff --git a/tests/migration/i386/Makefile b/tests/migration/i386/Makefile index 5c03241..37a72ae 100644 --- a/tests/migration/i386/Makefile +++ b/tests/migration/i386/Makefile @@ -4,9 +4,10 @@ .PHONY: all clean all: a-b-bootblock.h -a-b-bootblock.h: x86.bootsect +a-b-bootblock.h: x86.bootsect x86.o echo "$$__note" > header.tmp xxd -i $< | sed -e 's/.*int.*//' >> header.tmp + nm x86.o | awk '{print "#define SYM_"$$3" 0x"$$1}' >> header.tmp mv header.tmp $@ x86.bootsect: x86.boot @@ -16,7 +17,7 @@ x86.boot: x86.o $(CROSS_PREFIX)objcopy -O binary $< $@ x86.o: a-b-bootblock.S - $(CROSS_PREFIX)gcc -m32 -march=i486 -c $< -o $@ + $(CROSS_PREFIX)gcc -I.. -m32 -march=i486 -c $< -o $@ clean: @rm -rf *.boot *.o *.bootsect diff --git a/tests/migration/i386/a-b-bootblock.S b/tests/migration/i386/a-b-bootblock.S index 6bb9999..6f39eb6 100644 --- a/tests/migration/i386/a-b-bootblock.S +++ b/tests/migration/i386/a-b-bootblock.S @@ -9,6 +9,23 @@ # # Author: dgilbert@redhat.com +#include "migration-test.h" + +#define ACPI_ENABLE 0xf1 +#define ACPI_PORT_SMI_CMD 0xb2 +#define ACPI_PM_BASE 0x600 +#define PM1A_CNT_OFFSET 4 + +#define ACPI_SCI_ENABLE 0x0001 +#define ACPI_SLEEP_TYPE 0x0400 +#define ACPI_SLEEP_ENABLE 0x2000 +#define SLEEP (ACPI_SCI_ENABLE + ACPI_SLEEP_TYPE + ACPI_SLEEP_ENABLE) + +#define LOW_ADDR X86_TEST_MEM_START +#define HIGH_ADDR X86_TEST_MEM_END + +/* Save the suspended status at an address that is not written in the loop. */ +#define suspended (X86_TEST_MEM_START + 4) .code16 .org 0x7c00 @@ -35,8 +52,8 @@ start: # at 0x7c00 ? mov %eax,%ds # Start from 1MB -.set TEST_MEM_START, (1024*1024) -.set TEST_MEM_END, (100*1024*1024) +.set TEST_MEM_START, X86_TEST_MEM_START +.set TEST_MEM_END, X86_TEST_MEM_END mov $65,%ax mov $0x3f8,%dx @@ -69,7 +86,30 @@ innerloop: mov $0x3f8,%dx outb %al,%dx - jmp mainloop + # should this test suspend? + mov (suspend_me),%eax + cmp $0,%eax + je mainloop + + # are we waking after suspend? do not suspend again. + mov $suspended,%eax + mov (%eax),%eax + cmp $1,%eax + je mainloop + + # enable acpi + mov $ACPI_ENABLE,%al + outb %al,$ACPI_PORT_SMI_CMD + + # suspend to ram + mov $suspended,%eax + movl $1,(%eax) + mov $SLEEP,%ax + mov $(ACPI_PM_BASE + PM1A_CNT_OFFSET),%dx + outw %ax,%dx + # not reached. The wakeup causes reset and restart at 0x7c00, and we + # do not save and restore registers as a real kernel would do. + # GDT magic from old (GPLv2) Grub startup.S .p2align 2 /* force 4-byte alignment */ @@ -95,6 +135,10 @@ gdtdesc: .word 0x27 /* limit */ .long gdt /* addr */ + /* test launcher can poke a 1 here to exercise suspend */ +suspend_me: + .int 0 + /* I'm a bootable disk */ .org 0x7dfe .byte 0x55 diff --git a/tests/migration/i386/a-b-bootblock.h b/tests/migration/i386/a-b-bootblock.h index 5b52391..c83f871 100644 --- a/tests/migration/i386/a-b-bootblock.h +++ b/tests/migration/i386/a-b-bootblock.h @@ -4,7 +4,7 @@ * the header and the assembler differences in your patch submission. */ unsigned char x86_bootsect[] = { - 0xfa, 0x0f, 0x01, 0x16, 0x8c, 0x7c, 0x66, 0xb8, 0x01, 0x00, 0x00, 0x00, + 0xfa, 0x0f, 0x01, 0x16, 0xb8, 0x7c, 0x66, 0xb8, 0x01, 0x00, 0x00, 0x00, 0x0f, 0x22, 0xc0, 0x66, 0xea, 0x20, 0x7c, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xe4, 0x92, 0x0c, 0x02, 0xe6, 0x92, 0xb8, 0x10, 0x00, 0x00, 0x00, 0x8e, 0xd8, 0x66, 0xb8, 0x41, @@ -13,13 +13,13 @@ unsigned char x86_bootsect[] = { 0x40, 0x06, 0x7c, 0xf1, 0xb8, 0x00, 0x00, 0x10, 0x00, 0xfe, 0x00, 0x05, 0x00, 0x10, 0x00, 0x00, 0x3d, 0x00, 0x00, 0x40, 0x06, 0x7c, 0xf2, 0xfe, 0xc3, 0x80, 0xe3, 0x3f, 0x75, 0xe6, 0x66, 0xb8, 0x42, 0x00, 0x66, 0xba, - 0xf8, 0x03, 0xee, 0xeb, 0xdb, 0x8d, 0x76, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0xff, 0xff, 0x00, 0x00, 0x00, 0x9a, 0xcf, 0x00, - 0xff, 0xff, 0x00, 0x00, 0x00, 0x92, 0xcf, 0x00, 0x27, 0x00, 0x74, 0x7c, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0xf8, 0x03, 0xee, 0xa1, 0xbe, 0x7c, 0x00, 0x00, 0x83, 0xf8, 0x00, 0x74, + 0xd3, 0xb8, 0x04, 0x00, 0x10, 0x00, 0x8b, 0x00, 0x83, 0xf8, 0x01, 0x74, + 0xc7, 0xb0, 0xf1, 0xe6, 0xb2, 0xb8, 0x04, 0x00, 0x10, 0x00, 0xc7, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x66, 0xb8, 0x01, 0x24, 0x66, 0xba, 0x04, 0x06, + 0x66, 0xef, 0x66, 0x90, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0xff, 0xff, 0x00, 0x00, 0x00, 0x9a, 0xcf, 0x00, 0xff, 0xff, 0x00, 0x00, + 0x00, 0x92, 0xcf, 0x00, 0x27, 0x00, 0xa0, 0x7c, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, @@ -49,3 +49,13 @@ unsigned char x86_bootsect[] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x55, 0xaa }; +#define SYM_do_zero 0x00007c3d +#define SYM_gdt 0x00007ca0 +#define SYM_gdtdesc 0x00007cb8 +#define SYM_innerloop 0x00007c51 +#define SYM_mainloop 0x00007c4c +#define SYM_pre_zero 0x00007c38 +#define SYM_start 0x00007c00 +#define SYM_suspend_me 0x00007cbe +#define SYM_TEST_MEM_END 0x06400000 +#define SYM_TEST_MEM_START 0x00100000 diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 05c0740..e10d5a4 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -133,7 +133,7 @@ static char *bootpath; #include "tests/migration/aarch64/a-b-kernel.h" #include "tests/migration/s390x/a-b-bios.h" -static void bootfile_create(char *dir) +static void bootfile_create(char *dir, bool suspend_me) { const char *arch = qtest_get_arch(); unsigned char *content; @@ -143,6 +143,7 @@ static void bootfile_create(char *dir) if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) { /* the assembled x86 boot sector should be exactly one sector large */ g_assert(sizeof(x86_bootsect) == 512); + x86_bootsect[SYM_suspend_me - SYM_start] = suspend_me; content = x86_bootsect; len = sizeof(x86_bootsect); } else if (g_str_equal(arch, "s390x")) { @@ -646,6 +647,8 @@ typedef struct { bool use_dirty_ring; const char *opts_source; const char *opts_target; + /* suspend the src before migrating to dest. */ + bool suspend_me; } MigrateStart; /* @@ -767,6 +770,8 @@ static int test_migrate_start(QTestState **from, QTestState **to, dst_state = (QTestMigrationState) { }; src_state = (QTestMigrationState) { }; + bootfile_create(tmpfs, args->suspend_me); + if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) { memory_size = "150M"; @@ -2980,7 +2985,9 @@ static int64_t get_limit_rate(QTestState *who) static QTestState *dirtylimit_start_vm(void) { QTestState *vm = NULL; - g_autofree gchar * + g_autofree gchar *cmd = NULL; + + bootfile_create(tmpfs, false); cmd = g_strdup_printf("-accel kvm,dirty-ring-size=4096 " "-name dirtylimit-test,debug-threads=on " "-m 150M -smp 1 " @@ -3329,7 +3336,6 @@ int main(int argc, char **argv) g_get_tmp_dir(), err->message); } g_assert(tmpfs); - bootfile_create(tmpfs); module_call_init(MODULE_INIT_QOM); From patchwork Wed Dec 6 17:23:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13482086 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 248D0C10F05 for ; Wed, 6 Dec 2023 17:25:14 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rAvdK-0001Oq-9L; Wed, 06 Dec 2023 12:24:26 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvdI-0001Nb-Om for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:24 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rAvdF-0003iA-M8 for qemu-devel@nongnu.org; Wed, 06 Dec 2023 12:24:24 -0500 Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6EYmrw004992; Wed, 6 Dec 2023 17:24:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=r983GWFdT+mQeD40xwqGLFF/xeF7Vfd8XQz3MHvnlV8=; b=jFum5f8HHi3Bjb9F/gLxFQKcq+Yd35ljwD9nhiqDX31jeYnrDQryC1zDQglEHTcOyAK/ 9K9lYVbrjWsQKY/yavjW1xlAPqq9QsLoikugFWitNBDb9UKiBnyR8JJNqevgxDJZNPJ8 YpTe5D4/TdPR0n8ZypVASAn9fALV5JO8r5oJYTtqTFKbLhPEKmIedM31VjMzBGLn6duB tXHvxypIyoLFvcXcOn9CK89EJgjzjMxmLDhVenZrrG7AuADC886MLwzfn9MyBGnSalzI hjdsY/0DFIqLk9HwIK2/Gjtno+bZSDq2wFkQpYggrIVbJAlT8gTtyHn6DqH9n/FCI9Py yw== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd0hj0v2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:24:09 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B6G5Nro037885; Wed, 6 Dec 2023 17:24:07 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3utanc7acq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Dec 2023 17:24:07 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B6HMwkg002907; Wed, 6 Dec 2023 17:24:07 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3utanc79rv-13; Wed, 06 Dec 2023 17:24:06 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Juan Quintela , Peter Xu , Paolo Bonzini , Thomas Huth , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Fabiano Rosas , Leonardo Bras , Steve Sistare Subject: [PATCH V7 12/12] tests/qtest: postcopy migration with suspend Date: Wed, 6 Dec 2023 09:23:37 -0800 Message-Id: <1701883417-356268-13-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> References: <1701883417-356268-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-06_16,2023-12-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 malwarescore=0 bulkscore=0 mlxlogscore=999 phishscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312060140 X-Proofpoint-GUID: Wdyj3J_hbywrehqyIMcYinmQFExDV7NN X-Proofpoint-ORIG-GUID: Wdyj3J_hbywrehqyIMcYinmQFExDV7NN Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add a test case to verify that the suspended state is handled correctly by live migration postcopy. The test suspends the src, migrates, then wakes the dest. Signed-off-by: Steve Sistare Reviewed-by: Peter Xu --- tests/qtest/migration-test.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index f57a978..b7c094e 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -1347,6 +1347,7 @@ static int migrate_postcopy_prepare(QTestState **from_ptr, /* Wait for the first serial output from the source */ wait_for_serial("src_serial"); + wait_for_suspend(from, &src_state); g_autofree char *uri = migrate_get_socket_address(to, "socket-address"); migrate_qmp(from, uri, "{}"); @@ -1364,6 +1365,11 @@ static void migrate_postcopy_complete(QTestState *from, QTestState *to, { wait_for_migration_complete(from); + if (args->start.suspend_me) { + /* wakeup succeeds only if guest is suspended */ + qtest_qmp_assert_success(to, "{'execute': 'system_wakeup'}"); + } + /* Make sure we get at least one "B" on destination */ wait_for_serial("dest_serial"); @@ -1397,6 +1403,15 @@ static void test_postcopy(void) test_postcopy_common(&args); } +static void test_postcopy_suspend(void) +{ + MigrateCommon args = { + .start.suspend_me = true, + }; + + test_postcopy_common(&args); +} + static void test_postcopy_compress(void) { MigrateCommon args = { @@ -3412,7 +3427,10 @@ int main(int argc, char **argv) qtest_add_func("/migration/postcopy/recovery/double-failures", test_postcopy_recovery_double_fail); #endif /* _WIN32 */ - + if (is_x86) { + qtest_add_func("/migration/postcopy/suspend", + test_postcopy_suspend); + } } qtest_add_func("/migration/bad_dest", test_baddest);