[COLO-Frame,(Base),v21,12/17] COLO: Add 'x-colo-lost-heartbeat' command to trigger failover

Commit Message

Zhanghailiang Oct. 18, 2016, 12:10 p.m. UTC

We leave users to choose whatever heartbeat solution they want,
if the heartbeat is lost, or other errors they detect, they can use
experimental command 'x_colo_lost_heartbeat' to tell COLO to do failover,
COLO will do operations accordingly.

For example, if the command is sent to the PVM, the Primary side will
exit COLO mode and take over operation. If sent to the Secondary, the
secondary will run failover work, then take over server operation to
become the new Primary.

Cc: Luiz Capitulino <[e-mail]>
Cc: Eric Blake <[e-mail]>
Cc: Markus Armbruster <[e-mail]>
Signed-off-by: zhanghailiang <[e-mail]>
Signed-off-by: Li Zhijian <[e-mail]>
Reviewed-by: Dr. David Alan Gilbert <[e-mail]>
---
v19:
- Fix title and comment
v16:
- Fix compile broken due to missing osdep.h
v13:
- Add Reviewed-by tag
v11:
- Add more comments for x-colo-lost-heartbeat command (Eric's suggestion)
- Return 'enum' instead of 'int' for get_colo_mode() (Eric's suggestion)
v10:
- Rename command colo_lost_hearbeat to experimental 'x_colo_lost_heartbeat'
---
 docs/qmp-commands.txt        | 10 ++++++++++
 hmp-commands.hx              | 15 +++++++++++++++
 hmp.c                        |  8 ++++++++
 hmp.h                        |  1 +
 include/migration/colo.h     |  3 +++
 include/migration/failover.h | 20 ++++++++++++++++++++
 migration/Makefile.objs      |  2 +-
 migration/colo-comm.c        | 11 +++++++++++
 migration/colo-failover.c    | 42 ++++++++++++++++++++++++++++++++++++++++++
 migration/colo.c             |  1 +
 qapi-schema.json             | 29 +++++++++++++++++++++++++++++
 stubs/migration-colo.c       |  8 ++++++++
 12 files changed, 149 insertions(+), 1 deletion(-)
 create mode 100644 include/migration/failover.h
 create mode 100644 migration/colo-failover.c

Comments

Amit Shah Oct. 26, 2016, 5:51 a.m. UTC | #1

On (Tue) 18 Oct 2016 [20:10:08], zhanghailiang wrote:
> We leave users to choose whatever heartbeat solution they want,
> if the heartbeat is lost, or other errors they detect, they can use
> experimental command 'x_colo_lost_heartbeat' to tell COLO to do failover,
> COLO will do operations accordingly.
> 
> For example, if the command is sent to the PVM, the Primary side will
> exit COLO mode and take over operation.

Primary should already be in control, so there's nothing special
needed to 'take over operation'?  At max, it should not do periodic
syncs anymore till it hears from a (new) secondary.

> If sent to the Secondary, the
> secondary will run failover work, then take over server operation to
> become the new Primary.
> 
> Cc: Luiz Capitulino <[e-mail]>
> Cc: Eric Blake <[e-mail]>
> Cc: Markus Armbruster <[e-mail]>
> Signed-off-by: zhanghailiang <[e-mail]>
> Signed-off-by: Li Zhijian <[e-mail]>
> Reviewed-by: Dr. David Alan Gilbert <[e-mail]>

Reviewed-by: Amit Shah <[e-mail]>


		Amit

Dr. David Alan Gilbert Oct. 26, 2016, 1:59 p.m. UTC | #2

* Amit Shah ([e-mail]) wrote:
> On (Tue) 18 Oct 2016 [20:10:08], zhanghailiang wrote:
> > We leave users to choose whatever heartbeat solution they want,
> > if the heartbeat is lost, or other errors they detect, they can use
> > experimental command 'x_colo_lost_heartbeat' to tell COLO to do failover,
> > COLO will do operations accordingly.
> > 
> > For example, if the command is sent to the PVM, the Primary side will
> > exit COLO mode and take over operation.
> 
> Primary should already be in control, so there's nothing special
> needed to 'take over operation'?  At max, it should not do periodic
> syncs anymore till it hears from a (new) secondary.

But it has to stop waiting for an ack from the secondary, and stop sending
copies of block data to it etc.

Dave

> > If sent to the Secondary, the
> > secondary will run failover work, then take over server operation to
> > become the new Primary.
> > 
> > Cc: Luiz Capitulino <[e-mail]>
> > Cc: Eric Blake <[e-mail]>
> > Cc: Markus Armbruster <[e-mail]>
> > Signed-off-by: zhanghailiang <[e-mail]>
> > Signed-off-by: Li Zhijian <[e-mail]>
> > Reviewed-by: Dr. David Alan Gilbert <[e-mail]>
> 
> Reviewed-by: Amit Shah <[e-mail]>
> 
> 
> 		Amit
--
Dr. David Alan Gilbert / [e-mail] / Manchester, UK

Zhanghailiang Oct. 26, 2016, 2:50 p.m. UTC | #3

On 2016/10/26 13:51, Amit Shah wrote:
> On (Tue) 18 Oct 2016 [20:10:08], zhanghailiang wrote:
>> We leave users to choose whatever heartbeat solution they want,
>> if the heartbeat is lost, or other errors they detect, they can use
>> experimental command 'x_colo_lost_heartbeat' to tell COLO to do failover,
>> COLO will do operations accordingly.
>>
>> For example, if the command is sent to the PVM, the Primary side will
>> exit COLO mode and take over operation.
>
> Primary should already be in control, so there's nothing special
> needed to 'take over operation'?  At max, it should not do periodic

Maybe the description here is not accurate which makes you confuse,
It should be 'does cleanup work and PVM will take over the service work',
I will fix it in next version.

> syncs anymore till it hears from a (new) secondary.
>
>> If sent to the Secondary, the
>> secondary will run failover work, then take over server operation to
>> become the new Primary.
>>
>> Cc: Luiz Capitulino <[e-mail]>
>> Cc: Eric Blake <[e-mail]>
>> Cc: Markus Armbruster <[e-mail]>
>> Signed-off-by: zhanghailiang <[e-mail]>
>> Signed-off-by: Li Zhijian <[e-mail]>
>> Reviewed-by: Dr. David Alan Gilbert <[e-mail]>
>
> Reviewed-by: Amit Shah <[e-mail]>
>
>
> 		Amit
>
> .
>

Zhanghailiang Oct. 26, 2016, 3:32 p.m. UTC | #4

On 2016/10/26 21:59, Dr. David Alan Gilbert wrote:
> * Amit Shah ([e-mail]) wrote:
>> On (Tue) 18 Oct 2016 [20:10:08], zhanghailiang wrote:
>>> We leave users to choose whatever heartbeat solution they want,
>>> if the heartbeat is lost, or other errors they detect, they can use
>>> experimental command 'x_colo_lost_heartbeat' to tell COLO to do failover,
>>> COLO will do operations accordingly.
>>>
>>> For example, if the command is sent to the PVM, the Primary side will
>>> exit COLO mode and take over operation.
>>
>> Primary should already be in control, so there's nothing special
>> needed to 'take over operation'?  At max, it should not do periodic
>> syncs anymore till it hears from a (new) secondary.
>
> But it has to stop waiting for an ack from the secondary, and stop sending
> copies of block data to it etc.
>

Yes, you are right, we still need to do this cleanup work :)

Thanks,
hailiang

> Dave
>
>>> If sent to the Secondary, the
>>> secondary will run failover work, then take over server operation to
>>> become the new Primary.
>>>
>>> Cc: Luiz Capitulino <[e-mail]>
>>> Cc: Eric Blake <[e-mail]>
>>> Cc: Markus Armbruster <[e-mail]>
>>> Signed-off-by: zhanghailiang <[e-mail]>
>>> Signed-off-by: Li Zhijian <[e-mail]>
>>> Reviewed-by: Dr. David Alan Gilbert <[e-mail]>
>>
>> Reviewed-by: Amit Shah <[e-mail]>
>>
>>
>> 		Amit
> --
> Dr. David Alan Gilbert / [e-mail] / Manchester, UK
>
> .
>

Patch

diff --git a/docs/qmp-commands.txt b/docs/qmp-commands.txt
index 13ab523..5296b30 100644
--- a/docs/qmp-commands.txt
+++ b/docs/qmp-commands.txt
@@ -554,6 +554,16 @@  Example:
 -> { "execute": "migrate_set_downtime", "arguments": { "value": 0.1 } }
 <- { "return": {} }
 
+x-colo-lost-heartbeat
+--------------------
+
+Tell COLO that heartbeat is lost, a failover or takeover is needed.
+
+Example:
+
+-> { "execute": "x-colo-lost-heartbeat" }
+<- { "return": {} }
+
 client_migrate_info
 -------------------
 
diff --git a/hmp-commands.hx b/hmp-commands.hx
index 06bef47..8819281 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -1040,6 +1040,21 @@  migration (or once already in postcopy).
 ETEXI
 
     {
+        .name       = "x_colo_lost_heartbeat",
+        .args_type  = "",
+        .params     = "",
+        .help       = "Tell COLO that heartbeat is lost,\n\t\t\t"
+                      "a failover or takeover is needed.",
+        .cmd = hmp_x_colo_lost_heartbeat,
+    },
+
+STEXI
+@item x_colo_lost_heartbeat
+@findex x_colo_lost_heartbeat
+Tell COLO that heartbeat is lost, a failover or takeover is needed.
+ETEXI
+
+    {
         .name       = "client_migrate_info",
         .args_type  = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
         .params     = "protocol hostname port tls-port cert-subject",
diff --git a/hmp.c b/hmp.c
index 759f4f4..909ec43 100644
--- a/hmp.c
+++ b/hmp.c
@@ -1450,6 +1450,14 @@  void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict)
     hmp_handle_error(mon, &err);
 }
 
+void hmp_x_colo_lost_heartbeat(Monitor *mon, const QDict *qdict)
+{
+    Error *err = NULL;
+
+    qmp_x_colo_lost_heartbeat(&err);
+    hmp_handle_error(mon, &err);
+}
+
 void hmp_set_password(Monitor *mon, const QDict *qdict)
 {
     const char *protocol  = qdict_get_str(qdict, "protocol");
diff --git a/hmp.h b/hmp.h
index 184769c..05daf7c 100644
--- a/hmp.h
+++ b/hmp.h
@@ -72,6 +72,7 @@  void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
 void hmp_client_migrate_info(Monitor *mon, const QDict *qdict);
 void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict);
+void hmp_x_colo_lost_heartbeat(Monitor *mon, const QDict *qdict);
 void hmp_set_password(Monitor *mon, const QDict *qdict);
 void hmp_expire_password(Monitor *mon, const QDict *qdict);
 void hmp_eject(Monitor *mon, const QDict *qdict);
diff --git a/include/migration/colo.h b/include/migration/colo.h
index b40676c..e9ac2c3 100644
--- a/include/migration/colo.h
+++ b/include/migration/colo.h
@@ -17,6 +17,7 @@ 
 #include "migration/migration.h"
 #include "qemu/coroutine_int.h"
 #include "qemu/thread.h"
+#include "qemu/main-loop.h"
 
 bool colo_supported(void);
 void colo_info_init(void);
@@ -29,4 +30,6 @@  bool migration_incoming_enable_colo(void);
 void migration_incoming_exit_colo(void);
 void *colo_process_incoming_thread(void *opaque);
 bool migration_incoming_in_colo_state(void);
+
+COLOMode get_colo_mode(void);
 #endif
diff --git a/include/migration/failover.h b/include/migration/failover.h
new file mode 100644
index 0000000..3274735
--- /dev/null
+++ b/include/migration/failover.h
@@ -0,0 +1,20 @@ 
+/*
+ *  COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ *  (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (c) 2016 HUAWEI TECHNOLOGIES CO.,LTD.
+ * Copyright (c) 2016 FUJITSU LIMITED
+ * Copyright (c) 2016 Intel Corporation
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_FAILOVER_H
+#define QEMU_FAILOVER_H
+
+#include "qemu-common.h"
+
+void failover_request_active(Error **errp);
+
+#endif
diff --git a/migration/Makefile.objs b/migration/Makefile.objs
index 4bbe9ab..3f3e237 100644
--- a/migration/Makefile.objs
+++ b/migration/Makefile.objs
@@ -1,7 +1,7 @@ 
 common-obj-y += migration.o socket.o fd.o exec.o
 common-obj-y += tls.o
-common-obj-$(CONFIG_COLO) += colo.o
 common-obj-y += colo-comm.o
+common-obj-$(CONFIG_COLO) += colo.o colo-failover.o
 common-obj-y += vmstate.o
 common-obj-y += qemu-file.o
 common-obj-y += qemu-file-channel.o
diff --git a/migration/colo-comm.c b/migration/colo-comm.c
index 3af9333..a69a5f7 100644
--- a/migration/colo-comm.c
+++ b/migration/colo-comm.c
@@ -21,6 +21,17 @@  typedef struct {
 
 static COLOInfo colo_info;
 
+COLOMode get_colo_mode(void)
+{
+    if (migration_in_colo_state()) {
+        return COLO_MODE_PRIMARY;
+    } else if (migration_incoming_in_colo_state()) {
+        return COLO_MODE_SECONDARY;
+    } else {
+        return COLO_MODE_UNKNOWN;
+    }
+}
+
 static void colo_info_pre_save(void *opaque)
 {
     COLOInfo *s = opaque;
diff --git a/migration/colo-failover.c b/migration/colo-failover.c
new file mode 100644
index 0000000..e31fc10
--- /dev/null
+++ b/migration/colo-failover.c
@@ -0,0 +1,42 @@ 
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD.
+ * Copyright (c) 2016 FUJITSU LIMITED
+ * Copyright (c) 2016 Intel Corporation
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "migration/colo.h"
+#include "migration/failover.h"
+#include "qmp-commands.h"
+#include "qapi/qmp/qerror.h"
+
+static QEMUBH *failover_bh;
+
+static void colo_failover_bh(void *opaque)
+{
+    qemu_bh_delete(failover_bh);
+    failover_bh = NULL;
+    /* TODO: Do failover work */
+}
+
+void failover_request_active(Error **errp)
+{
+    failover_bh = qemu_bh_new(colo_failover_bh, NULL);
+    qemu_bh_schedule(failover_bh);
+}
+
+void qmp_x_colo_lost_heartbeat(Error **errp)
+{
+    if (get_colo_mode() == COLO_MODE_UNKNOWN) {
+        error_setg(errp, QERR_FEATURE_DISABLED, "colo");
+        return;
+    }
+
+    failover_request_active(errp);
+}
diff --git a/migration/colo.c b/migration/colo.c
index 7c7d169..45b13cd 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -18,6 +18,7 @@ 
 #include "trace.h"
 #include "qemu/error-report.h"
 #include "qapi/error.h"
+#include "migration/failover.h"
 
 #define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024)
 
diff --git a/qapi-schema.json b/qapi-schema.json
index b6950be..6367685 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -822,6 +822,35 @@ 
             'vmstate-send', 'vmstate-size', 'vmstate-received',
             'vmstate-loaded' ] }
 
+##
+# @COLOMode
+#
+# The colo mode
+#
+# @unknown: unknown mode
+#
+# @primary: master side
+#
+# @secondary: slave side
+#
+# Since: 2.8
+##
+{ 'enum': 'COLOMode',
+  'data': [ 'unknown', 'primary', 'secondary'] }
+
+##
+# @x-colo-lost-heartbeat
+#
+# Tell qemu that heartbeat is lost, request it to do takeover procedures.
+# If this command is sent to the PVM, the Primary side will exit COLO mode.
+# If sent to the Secondary, the Secondary side will run failover work,
+# then takes over server operation to become the service VM.
+#
+# Since: 2.8
+##
+{ 'command': 'x-colo-lost-heartbeat' }
+
+##
 # @MouseInfo:
 #
 # Information about a mouse device.
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
index 7b72395..7811764 100644
--- a/stubs/migration-colo.c
+++ b/stubs/migration-colo.c
@@ -12,6 +12,7 @@ 
 
 #include "qemu/osdep.h"
 #include "migration/colo.h"
+#include "qmp-commands.h"
 
 bool colo_supported(void)
 {
@@ -36,3 +37,10 @@  void *colo_process_incoming_thread(void *opaque)
 {
     return NULL;
 }
+
+void qmp_x_colo_lost_heartbeat(Error **errp)
+{
+    error_setg(errp, "COLO is not supported, please rerun configure"
+                     " with --enable-colo option in order to support"
+                     " COLO feature");
+}