[04/12] x86/xen: add system core suspend and resume callbacks

Commit Message

Anchal Agarwal May 19, 2020, 11:26 p.m. UTC

From: Munehisa Kamata <[e-mail]>

Add Xen PVHVM specific system core callbacks for PM suspend and
hibernation support. The callbacks suspend and resume Xen
primitives,like shared_info, pvclock and grant table. Note that
Xen suspend can handle them in a different manner, but system
core callbacks are called from the context. So if the callbacks
are called from Xen suspend context, return immediately.

Signed-off-by: Agarwal Anchal <[e-mail]>
Signed-off-by: Munehisa Kamata <[e-mail]>
---
 arch/x86/xen/enlighten_hvm.c |  1 +
 arch/x86/xen/suspend.c       | 53 ++++++++++++++++++++++++++++++++++++
 include/xen/xen-ops.h        |  3 ++
 3 files changed, 57 insertions(+)

Comments

Boris Ostrovsky May 30, 2020, 11:10 p.m. UTC | #1

On 5/19/20 7:26 PM, Anchal Agarwal wrote:
> From: Munehisa Kamata <[e-mail]>
>
> Add Xen PVHVM specific system core callbacks for PM suspend and
> hibernation support. The callbacks suspend and resume Xen
> primitives,like shared_info, pvclock and grant table. Note that
> Xen suspend can handle them in a different manner, but system
> core callbacks are called from the context.


I don't think I understand that last sentence.


>  So if the callbacks
> are called from Xen suspend context, return immediately.
>


> +
> +static int xen_syscore_suspend(void)
> +{
> +	struct xen_remove_from_physmap xrfp;
> +	int ret;
> +
> +	/* Xen suspend does similar stuffs in its own logic */
> +	if (xen_suspend_mode_is_xen_suspend())
> +		return 0;
> +
> +	xrfp.domid = DOMID_SELF;
> +	xrfp.gpfn = __pa(HYPERVISOR_shared_info) >> PAGE_SHIFT;
> +
> +	ret = HYPERVISOR_memory_op(XENMEM_remove_from_physmap, &xrfp);
> +	if (!ret)
> +		HYPERVISOR_shared_info = &xen_dummy_shared_info;
> +
> +	return ret;
> +}
> +
> +static void xen_syscore_resume(void)
> +{
> +	/* Xen suspend does similar stuffs in its own logic */
> +	if (xen_suspend_mode_is_xen_suspend())
> +		return;
> +
> +	/* No need to setup vcpu_info as it's already moved off */
> +	xen_hvm_map_shared_info();
> +
> +	pvclock_resume();
> +
> +	gnttab_resume();


Do you call gnttab_suspend() in pm suspend path?


> +}
> +
> +/*
> + * These callbacks will be called with interrupts disabled and when having only
> + * one CPU online.
> + */
> +static struct syscore_ops xen_hvm_syscore_ops = {
> +	.suspend = xen_syscore_suspend,
> +	.resume = xen_syscore_resume
> +};
> +
> +void __init xen_setup_syscore_ops(void)
> +{
> +	if (xen_hvm_domain())


Have you tested this (the whole feature, not just this patch) with PVH
guest BTW? And PVH dom0 for that matter?


-boris


> +		register_syscore_ops(&xen_hvm_syscore_ops);
> +}

Anchal Agarwal June 3, 2020, 10:40 p.m. UTC | #2

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



    On 5/19/20 7:26 PM, Anchal Agarwal wrote:
    > From: Munehisa Kamata <[e-mail]>
    >
    > Add Xen PVHVM specific system core callbacks for PM suspend and
    > hibernation support. The callbacks suspend and resume Xen
    > primitives,like shared_info, pvclock and grant table. Note that
    > Xen suspend can handle them in a different manner, but system
    > core callbacks are called from the context.


    I don't think I understand that last sentence.

Looks like it may have cryptic meaning of stating that xen_suspend calls syscore_suspend from xen_suspend
So, if these syscore ops gets called  during xen_suspend do not do anything. Check if the mode is in xen suspend 
and return from there. These syscore_ops are specifically for domU hibernation.
I must admit, I may have overlooked lack of explanation of some implicit details in the original commit msg. 

    >  So if the callbacks
    > are called from Xen suspend context, return immediately.
    >


    > +
    > +static int xen_syscore_suspend(void)
    > +{
    > +     struct xen_remove_from_physmap xrfp;
    > +     int ret;
    > +
    > +     /* Xen suspend does similar stuffs in its own logic */
    > +     if (xen_suspend_mode_is_xen_suspend())
    > +             return 0;
    > +
    > +     xrfp.domid = DOMID_SELF;
    > +     xrfp.gpfn = __pa(HYPERVISOR_shared_info) >> PAGE_SHIFT;
    > +
    > +     ret = HYPERVISOR_memory_op(XENMEM_remove_from_physmap, &xrfp);
    > +     if (!ret)
    > +             HYPERVISOR_shared_info = &xen_dummy_shared_info;
    > +
    > +     return ret;
    > +}
    > +
    > +static void xen_syscore_resume(void)
    > +{
    > +     /* Xen suspend does similar stuffs in its own logic */
    > +     if (xen_suspend_mode_is_xen_suspend())
    > +             return;
    > +
    > +     /* No need to setup vcpu_info as it's already moved off */
    > +     xen_hvm_map_shared_info();
    > +
    > +     pvclock_resume();
    > +
    > +     gnttab_resume();


    Do you call gnttab_suspend() in pm suspend path?
No, since it does nothing for HVM guests. The unmap_frames is only applicable for PV guests right?

    > +}
    > +
    > +/*
    > + * These callbacks will be called with interrupts disabled and when having only
    > + * one CPU online.
    > + */
    > +static struct syscore_ops xen_hvm_syscore_ops = {
    > +     .suspend = xen_syscore_suspend,
    > +     .resume = xen_syscore_resume
    > +};
    > +
    > +void __init xen_setup_syscore_ops(void)
    > +{
    > +     if (xen_hvm_domain())


    Have you tested this (the whole feature, not just this patch) with PVH
    guest BTW? And PVH dom0 for that matter?

No I haven't. The whole series is just tested with hvm/pvhvm guests.

    -boris
Thanks,
Anchal

    > +             register_syscore_ops(&xen_hvm_syscore_ops);
    > +}

Boris Ostrovsky June 5, 2020, 9:24 p.m. UTC | #3

On 6/3/20 6:40 PM, Agarwal, Anchal wrote:
>     CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
>     On 5/19/20 7:26 PM, Anchal Agarwal wrote:
>     > From: Munehisa Kamata <[e-mail]>
>     >
>     > Add Xen PVHVM specific system core callbacks for PM suspend and
>     > hibernation support. The callbacks suspend and resume Xen
>     > primitives,like shared_info, pvclock and grant table. Note that
>     > Xen suspend can handle them in a different manner, but system
>     > core callbacks are called from the context.
>
>
>     I don't think I understand that last sentence.
>
> Looks like it may have cryptic meaning of stating that xen_suspend calls syscore_suspend from xen_suspend
> So, if these syscore ops gets called  during xen_suspend do not do anything. Check if the mode is in xen suspend 
> and return from there. These syscore_ops are specifically for domU hibernation.
> I must admit, I may have overlooked lack of explanation of some implicit details in the original commit msg. 
>
>     >  So if the callbacks
>     > are called from Xen suspend context, return immediately.
>     >
>
>
>     > +
>     > +static int xen_syscore_suspend(void)
>     > +{
>     > +     struct xen_remove_from_physmap xrfp;
>     > +     int ret;
>     > +
>     > +     /* Xen suspend does similar stuffs in its own logic */
>     > +     if (xen_suspend_mode_is_xen_suspend())
>     > +             return 0;


With your explanation now making this clearer, is this check really
necessary? From what I see we are in XEN_SUSPEND mode when
lock_system_sleep() lock is taken, meaning that we can't initialize
hibernation.


>     > +
>     > +     xrfp.domid = DOMID_SELF;
>     > +     xrfp.gpfn = __pa(HYPERVISOR_shared_info) >> PAGE_SHIFT;
>     > +
>     > +     ret = HYPERVISOR_memory_op(XENMEM_remove_from_physmap, &xrfp);
>     > +     if (!ret)
>     > +             HYPERVISOR_shared_info = &xen_dummy_shared_info;
>     > +
>     > +     return ret;
>     > +}
>     > +
>     > +static void xen_syscore_resume(void)
>     > +{
>     > +     /* Xen suspend does similar stuffs in its own logic */
>     > +     if (xen_suspend_mode_is_xen_suspend())
>     > +             return;
>     > +
>     > +     /* No need to setup vcpu_info as it's already moved off */
>     > +     xen_hvm_map_shared_info();
>     > +
>     > +     pvclock_resume();
>     > +
>     > +     gnttab_resume();
>
>
>     Do you call gnttab_suspend() in pm suspend path?
> No, since it does nothing for HVM guests. The unmap_frames is only applicable for PV guests right?


You should call it nevertheless. It will decide whether or not anything
needs to be done.


-boris

Anchal Agarwal June 8, 2020, 5:09 p.m. UTC | #4

On Fri, Jun 05, 2020 at 05:24:37PM -0400, Boris Ostrovsky wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> 
> 
> 
> On 6/3/20 6:40 PM, Agarwal, Anchal wrote:
> >     CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> >
> >
> >
> >     On 5/19/20 7:26 PM, Anchal Agarwal wrote:
> >     > From: Munehisa Kamata <[e-mail]>
> >     >
> >     > Add Xen PVHVM specific system core callbacks for PM suspend and
> >     > hibernation support. The callbacks suspend and resume Xen
> >     > primitives,like shared_info, pvclock and grant table. Note that
> >     > Xen suspend can handle them in a different manner, but system
> >     > core callbacks are called from the context.
> >
> >
> >     I don't think I understand that last sentence.
> >
> > Looks like it may have cryptic meaning of stating that xen_suspend calls syscore_suspend from xen_suspend
> > So, if these syscore ops gets called  during xen_suspend do not do anything. Check if the mode is in xen suspend
> > and return from there. These syscore_ops are specifically for domU hibernation.
> > I must admit, I may have overlooked lack of explanation of some implicit details in the original commit msg.
> >
> >     >  So if the callbacks
> >     > are called from Xen suspend context, return immediately.
> >     >
> >
> >
> >     > +
> >     > +static int xen_syscore_suspend(void)
> >     > +{
> >     > +     struct xen_remove_from_physmap xrfp;
> >     > +     int ret;
> >     > +
> >     > +     /* Xen suspend does similar stuffs in its own logic */
> >     > +     if (xen_suspend_mode_is_xen_suspend())
> >     > +             return 0;
> 
> 
> With your explanation now making this clearer, is this check really
> necessary? From what I see we are in XEN_SUSPEND mode when
> lock_system_sleep() lock is taken, meaning that we can't initialize
> hibernation.
> 
I see. Sounds plausible. I will fix both the code and commit message
for better readability. Thanks for catching this.
> 
> >     > +
> >     > +     xrfp.domid = DOMID_SELF;
> >     > +     xrfp.gpfn = __pa(HYPERVISOR_shared_info) >> PAGE_SHIFT;
> >     > +
> >     > +     ret = HYPERVISOR_memory_op(XENMEM_remove_from_physmap, &xrfp);
> >     > +     if (!ret)
> >     > +             HYPERVISOR_shared_info = &xen_dummy_shared_info;
> >     > +
> >     > +     return ret;
> >     > +}
> >     > +
> >     > +static void xen_syscore_resume(void)
> >     > +{
> >     > +     /* Xen suspend does similar stuffs in its own logic */
> >     > +     if (xen_suspend_mode_is_xen_suspend())
> >     > +             return;
> >     > +
> >     > +     /* No need to setup vcpu_info as it's already moved off */
> >     > +     xen_hvm_map_shared_info();
> >     > +
> >     > +     pvclock_resume();
> >     > +
> >     > +     gnttab_resume();
> >
> >
> >     Do you call gnttab_suspend() in pm suspend path?
> > No, since it does nothing for HVM guests. The unmap_frames is only applicable for PV guests right?
> 
> 
> You should call it nevertheless. It will decide whether or not anything
> needs to be done.
Will fix it in V2.
> 
> 
> -boris
> 
Thanks,
Anchal
>

Patch

diff --git a/arch/x86/xen/enlighten_hvm.c b/arch/x86/xen/enlighten_hvm.c
index 75b1ec7a0fcd..138e71786e03 100644
--- a/arch/x86/xen/enlighten_hvm.c
+++ b/arch/x86/xen/enlighten_hvm.c
@@ -204,6 +204,7 @@  static void __init xen_hvm_guest_init(void)
 	if (xen_feature(XENFEAT_hvm_callback_vector))
 		xen_have_vector_callback = 1;
 
+	xen_setup_syscore_ops();
 	xen_hvm_smp_init();
 	WARN_ON(xen_cpuhp_setup(xen_cpu_up_prepare_hvm, xen_cpu_dead_hvm));
 	xen_unplug_emulated_devices();
diff --git a/arch/x86/xen/suspend.c b/arch/x86/xen/suspend.c
index 1d83152c761b..784c4484100b 100644
--- a/arch/x86/xen/suspend.c
+++ b/arch/x86/xen/suspend.c
@@ -2,17 +2,22 @@ 
 #include <linux/types.h>
 #include <linux/tick.h>
 #include <linux/percpu-defs.h>
+#include <linux/syscore_ops.h>
+#include <linux/kernel_stat.h>
 
 #include <xen/xen.h>
 #include <xen/interface/xen.h>
+#include <xen/interface/memory.h>
 #include <xen/grant_table.h>
 #include <xen/events.h>
+#include <xen/xen-ops.h>
 
 #include <asm/cpufeatures.h>
 #include <asm/msr-index.h>
 #include <asm/xen/hypercall.h>
 #include <asm/xen/page.h>
 #include <asm/fixmap.h>
+#include <asm/pvclock.h>
 
 #include "xen-ops.h"
 #include "mmu.h"
@@ -82,3 +87,51 @@  void xen_arch_suspend(void)
 
 	on_each_cpu(xen_vcpu_notify_suspend, NULL, 1);
 }
+
+static int xen_syscore_suspend(void)
+{
+	struct xen_remove_from_physmap xrfp;
+	int ret;
+
+	/* Xen suspend does similar stuffs in its own logic */
+	if (xen_suspend_mode_is_xen_suspend())
+		return 0;
+
+	xrfp.domid = DOMID_SELF;
+	xrfp.gpfn = __pa(HYPERVISOR_shared_info) >> PAGE_SHIFT;
+
+	ret = HYPERVISOR_memory_op(XENMEM_remove_from_physmap, &xrfp);
+	if (!ret)
+		HYPERVISOR_shared_info = &xen_dummy_shared_info;
+
+	return ret;
+}
+
+static void xen_syscore_resume(void)
+{
+	/* Xen suspend does similar stuffs in its own logic */
+	if (xen_suspend_mode_is_xen_suspend())
+		return;
+
+	/* No need to setup vcpu_info as it's already moved off */
+	xen_hvm_map_shared_info();
+
+	pvclock_resume();
+
+	gnttab_resume();
+}
+
+/*
+ * These callbacks will be called with interrupts disabled and when having only
+ * one CPU online.
+ */
+static struct syscore_ops xen_hvm_syscore_ops = {
+	.suspend = xen_syscore_suspend,
+	.resume = xen_syscore_resume
+};
+
+void __init xen_setup_syscore_ops(void)
+{
+	if (xen_hvm_domain())
+		register_syscore_ops(&xen_hvm_syscore_ops);
+}
diff --git a/include/xen/xen-ops.h b/include/xen/xen-ops.h
index 4ffe031adfc7..89b1e88712d6 100644
--- a/include/xen/xen-ops.h
+++ b/include/xen/xen-ops.h
@@ -43,6 +43,9 @@  int xen_setup_shutdown_event(void);
 bool xen_suspend_mode_is_xen_suspend(void);
 bool xen_suspend_mode_is_pm_suspend(void);
 bool xen_suspend_mode_is_pm_hibernation(void);
+
+void xen_setup_syscore_ops(void);
+
 extern unsigned long *xen_contiguous_bitmap;
 
 #if defined(CONFIG_XEN_PV) || defined(CONFIG_ARM) || defined(CONFIG_ARM64)