diff mbox

[RFC] ARM hibernation / suspend-to-disk support code

Message ID alpine.DEB.2.00.1105181257370.2374@localhost6.localdomain6 (mailing list archive)
State RFC, archived
Headers show

Commit Message

Frank Hofmann May 19, 2011, 5:31 p.m. UTC
Hi,

/me again ...

Sorry that this took a little ... holidays. And work. And distractions...

Anyway, here we go again, basic code to enable hibernation 
(suspend-to-disk) on ARM platforms.

Any comments highly welcome.



To use this, you need sleep.S modifications for your SoC type (to get 
__save/__restore_processor_state hooks). I've sent some of those for 
illustration earlier, they haven't changed, I've not included them here, 
so pick these changes up from:

http://68.183.106.108/lists/linux-pm/msg24020.html

The patch below only contains the _generic_ code.


This is tested on S5P6450 and OMAP3, with the sleep...S changes just 
mentioned - check the archives for those. Works both with normal swsusp and 
tuxonice (again, check the archives for the TOI modification needed).



Previously, I've reported OMAP3 video issues, after resume-from-disk. That 
isn't fully solved (it's a driver issue) but I've found a workaround: 
Trigger the resume from initramfs, after loading a logo image into the 
framebuffer and switching it on. That gets everything back without 
corruptions / wrong LCD reinitialization.

The OMAP video seems a bit of a diva; I've got one board type on which 
suspend/resume work perfectly but the omapdss driver spits out thousands 
of error interrupts during system startup (before the image is loaded), 
and the other board where all that is fine but the restore somehow garbles 
the LCD clocking (but the driver's sysfs files claim it's the same).


In short: This stuff really works now, for all I can say. And adding 
support for new type of ARM SoC doesn't touch the basic / generic code at 
all anymore either.




Anyway ...
About the patch, changes vs. all previous suggestions:

* Made the assembly sources as small as I responsibly could ;-)
   They compile for thumb2 (but I haven't tested that yet) as well.

* The page copy loop is now a C function. That also has the advantage
   that one can use cpu_switch_mm() - a macro - there for swapper_pg_dir,
   which makes resume via uswsusp ioctl or /sys/power/tuxonice/do_resume
   possible (only tested the latter, though).

* The SoC state save/restore is made to (re-)use the existing code in
   sleep....S  for the particular chip.
   OMAP3 and S5P64xx are provided as examples of that.

* The save/restore_processor_state() hooks are now used in the same way
   as e.g. existing powerpc code uses them (to trigger lazy saves before
   and perform cache flushes after).


Things that probably aren't perfect yet:

* The code currently reserves a full page for the saved "core" state.
   This is more than absolutely necessary; anyone think it's a problem ?

* it sets aside another half a page of __nosavedata page for use as
   temporary stack during the image copy (so that funcs can be called).

   Right now on ARM, that's not an issue because even with TuxOnIce in,
   there's less than 20 bytes of nosave stuff, so can as well put the
   rest of that page to good use ;-)

* I'd love to get rid of the include/asm-generic/vmlinux.lds.h change,
   as it seems that's not necessary in other architectures.
   Without that, the code gives a link error when building vmlinux
   though, and I'm unsure how to address that.

* The "integration" with the CPU sleep code is rather "backdoorish".
   While the hooks into ..._cpu_suspend aren't massive, and there's no
   code duplication, it'd be nicer to eventually have a cleaner way.

* An OMAPDSS restore troubleshooting HOWTO would be helpful ;-)


* The patch needs to be rebaselined against a current kernel;
   any preferences which tree to base this on ?



Thanks for all help with the little nits !
FrankH.

Comments

tip-bot for Dave Martin May 20, 2011, 11:37 a.m. UTC | #1
On Thu, May 19, 2011 at 06:31:28PM +0100, Frank Hofmann wrote:
> Hi,
> 
> /me again ...
> 
> Sorry that this took a little ... holidays. And work. And distractions...
> 
> Anyway, here we go again, basic code to enable hibernation
> (suspend-to-disk) on ARM platforms.
> 
> Any comments highly welcome.
> 
> 
> 
> To use this, you need sleep.S modifications for your SoC type (to
> get __save/__restore_processor_state hooks). I've sent some of those
> for illustration earlier, they haven't changed, I've not included
> them here, so pick these changes up from:
> 
> http://68.183.106.108/lists/linux-pm/msg24020.html
> 
> The patch below only contains the _generic_ code.
> 
> 
> This is tested on S5P6450 and OMAP3, with the sleep...S changes just
> mentioned - check the archives for those. Works both with normal
> swsusp and tuxonice (again, check the archives for the TOI
> modification needed).
> 
> 
> 
> Previously, I've reported OMAP3 video issues, after
> resume-from-disk. That isn't fully solved (it's a driver issue) but
> I've found a workaround: Trigger the resume from initramfs, after
> loading a logo image into the framebuffer and switching it on. That
> gets everything back without corruptions / wrong LCD
> reinitialization.
> 
> The OMAP video seems a bit of a diva; I've got one board type on
> which suspend/resume work perfectly but the omapdss driver spits out
> thousands of error interrupts during system startup (before the
> image is loaded), and the other board where all that is fine but the
> restore somehow garbles the LCD clocking (but the driver's sysfs
> files claim it's the same).
> 
> 
> In short: This stuff really works now, for all I can say. And adding
> support for new type of ARM SoC doesn't touch the basic / generic
> code at all anymore either.
> 
> 
> 
> 
> Anyway ...
> About the patch, changes vs. all previous suggestions:
> 
> * Made the assembly sources as small as I responsibly could ;-)
>   They compile for thumb2 (but I haven't tested that yet) as well.
> 
> * The page copy loop is now a C function. That also has the advantage
>   that one can use cpu_switch_mm() - a macro - there for swapper_pg_dir,
>   which makes resume via uswsusp ioctl or /sys/power/tuxonice/do_resume
>   possible (only tested the latter, though).
> 
> * The SoC state save/restore is made to (re-)use the existing code in
>   sleep....S  for the particular chip.
>   OMAP3 and S5P64xx are provided as examples of that.
> 
> * The save/restore_processor_state() hooks are now used in the same way
>   as e.g. existing powerpc code uses them (to trigger lazy saves before
>   and perform cache flushes after).
> 
> 
> Things that probably aren't perfect yet:
> 
> * The code currently reserves a full page for the saved "core" state.
>   This is more than absolutely necessary; anyone think it's a problem ?
> 
> * it sets aside another half a page of __nosavedata page for use as
>   temporary stack during the image copy (so that funcs can be called).
> 
>   Right now on ARM, that's not an issue because even with TuxOnIce in,
>   there's less than 20 bytes of nosave stuff, so can as well put the
>   rest of that page to good use ;-)
> 
> * I'd love to get rid of the include/asm-generic/vmlinux.lds.h change,
>   as it seems that's not necessary in other architectures.
>   Without that, the code gives a link error when building vmlinux
>   though, and I'm unsure how to address that.
> 
> * The "integration" with the CPU sleep code is rather "backdoorish".
>   While the hooks into ..._cpu_suspend aren't massive, and there's no
>   code duplication, it'd be nicer to eventually have a cleaner way.
> 
> * An OMAPDSS restore troubleshooting HOWTO would be helpful ;-)
> 
> 
> * The patch needs to be rebaselined against a current kernel;
>   any preferences which tree to base this on ?
> 
> 
> 
> Thanks for all help with the little nits !
> FrankH.

>  arch/arm/Kconfig                  |    3 +
>  arch/arm/include/asm/memory.h     |    1 +
>  arch/arm/include/asm/suspend.h    |    6 ++
>  arch/arm/kernel/cpu.c             |   65 ++++++++++++++++++++++++++
>  arch/arm/kernel/swsusp.S          |   92 +++++++++++++++++++++++++++++++++++++
>  arch/arm/kernel/vmlinux.lds.S     |    3 +-
>  include/asm-generic/vmlinux.lds.h |    2 +-
>  7 files changed, 170 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
> index 6b6786c..859dd86 100644
> --- a/arch/arm/Kconfig
> +++ b/arch/arm/Kconfig
> @@ -198,6 +198,9 @@ config VECTORS_BASE
>  config ARCH_HAS_CPU_IDLE_WAIT
>  	def_bool y
>  
> +config ARCH_HIBERNATION_POSSIBLE
> +	def_bool n
> +
>  source "init/Kconfig"
>  
>  source "kernel/Kconfig.freezer"
> diff --git a/arch/arm/include/asm/memory.h b/arch/arm/include/asm/memory.h
> index 5421d82..23e93a6 100644
> --- a/arch/arm/include/asm/memory.h
> +++ b/arch/arm/include/asm/memory.h
> @@ -191,6 +191,7 @@ static inline void *phys_to_virt(unsigned long x)
>   */
>  #define __pa(x)			__virt_to_phys((unsigned long)(x))
>  #define __va(x)			((void *)__phys_to_virt((unsigned long)(x)))
> +#define __pa_symbol(x)		__pa(RELOC_HIDE((unsigned long)(x),0))
>  #define pfn_to_kaddr(pfn)	__va((pfn) << PAGE_SHIFT)
>  
>  /*
> diff --git a/arch/arm/include/asm/suspend.h b/arch/arm/include/asm/suspend.h
> new file mode 100644
> index 0000000..7ab1fd2
> --- /dev/null
> +++ b/arch/arm/include/asm/suspend.h
> @@ -0,0 +1,6 @@
> +#ifndef __ASM_ARM_SUSPEND_H
> +#define __ASM_ARM_SUSPEND_H
> +
> +static inline int arch_prepare_suspend(void) { return 0; }
> +
> +#endif /* __ASM_ARM_SUSPEND_H */
> diff --git a/arch/arm/kernel/cpu.c b/arch/arm/kernel/cpu.c
> new file mode 100644
> index 0000000..764c8fa
> --- /dev/null
> +++ b/arch/arm/kernel/cpu.c
> @@ -0,0 +1,65 @@
> +/*
> + * Hibernation support specific for ARM
> + *
> + * Derived from work on ARM hibernation support by:
> + *
> + * Ubuntu project, hibernation support for mach-dove
> + * Copyright (C) 2010 Nokia Corporation (Hiroshi Doyu)
> + * Copyright (C) 2010 Texas Instruments, Inc. (Teerth Reddy et al.)
> + *	https://lkml.org/lkml/2010/6/18/4
> + *	https://lists.linux-foundation.org/pipermail/linux-pm/2010-June/027422.html
> + *	https://patchwork.kernel.org/patch/96442/
> + *
> + * Copyright (C) 2006 Rafael J. Wysocki <rjw@sisk.pl>
> + *
> + * License terms: GNU General Public License (GPL) version 2
> + */
> +
> +#include <linux/mm.h>
> +#include <linux/sched.h>
> +#include <linux/suspend.h>
> +#include <asm/tlbflush.h>
> +
> +extern const void __nosave_begin, __nosave_end;
> +
> +int pfn_is_nosave(unsigned long pfn)
> +{
> +	unsigned long nosave_begin_pfn = __pa_symbol(&__nosave_begin) >> PAGE_SHIFT;
> +	unsigned long nosave_end_pfn = PAGE_ALIGN(__pa_symbol(&__nosave_end)) >> PAGE_SHIFT;
> +
> +	return (pfn >= nosave_begin_pfn) && (pfn < nosave_end_pfn);
> +}
> +
> +void save_processor_state(void)
> +{
> +	flush_thread();
> +}
> +
> +void restore_processor_state(void)
> +{
> +	local_flush_tlb_all();
> +}
> +
> +u8 __swsusp_arch_ctx[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
> +u8 __swsusp_resume_stk[PAGE_SIZE/2] __nosavedata;
> +
> +/*
> + * The framework loads the hibernation image into this linked list,
> + * for swsusp_arch_resume() to copy back to the proper destinations.
> + *
> + * To make this work if resume is triggered from initramfs, the
> + * pagetables need to be switched to allow writes to kernel mem.
> + */
> +void notrace __swsusp_arch_restore_prepare(void)
> +{
> +	cpu_switch_mm(__virt_to_phys(swapper_pg_dir), current->active_mm);
> +}
> +
> +void notrace __swsusp_arch_restore_image(void)
> +{
> +	extern struct pbe *restore_pblist;
> +	struct pbe *pbe;
> +
> +	for (pbe = restore_pblist; pbe; pbe = pbe->next)
> +		copy_page(pbe->orig_address, pbe->address);
> +}
> diff --git a/arch/arm/kernel/swsusp.S b/arch/arm/kernel/swsusp.S
> new file mode 100644
> index 0000000..fb260a7
> --- /dev/null
> +++ b/arch/arm/kernel/swsusp.S
> @@ -0,0 +1,92 @@
> +/*
> + * Hibernation support specific for ARM
> + *
> + * Based on work by:
> + *
> + * Ubuntu project, hibernation support for mach-dove,
> + * Copyright (C) 2010 Nokia Corporation (Hiroshi Doyu)
> + * Copyright (C) 2010 Texas Instruments, Inc. (Teerth Reddy et al.)
> + *	https://lkml.org/lkml/2010/6/18/4
> + *	https://lists.linux-foundation.org/pipermail/linux-pm/2010-June/027422.html
> + *	https://patchwork.kernel.org/patch/96442/
> + *
> + * Copyright (C) 2006 Rafael J. Wysocki <rjw@sisk.pl>
> + *
> + * License terms: GNU General Public License (GPL) version 2
> + */
> +
> +#include <linux/linkage.h>
> +#include <asm/memory.h>
> +#include <asm/page.h>
> +#include <asm/cache.h>
> +#include <asm/ptrace.h>
> +
> +/*
> + * Save the current CPU state before suspend / poweroff.
> + */
> +ENTRY(swsusp_arch_suspend)
> +	ldr	r0, =__swsusp_arch_ctx
> +	mrs	r1, cpsr
> +	str	r1, [r0], #4		/* CPSR */
> +ARM(	msr	cpsr_c, #SYSTEM_MODE					)
> +THUMB(	mov	r2, #SYSTEM_MODE					)
> +THUMB(	msr	cpsr_c, r2						)

For Thumb-2 kernels, you can use the cps instruction to change the CPU
mode:
	cps	#SYSTEM_MODE

For ARM though, this instruction is only supported for ARMv6 and above,
so it's best to keep the msr form for compatibility for that case.

> +	stm	r0!, {r4-r12,lr}	/* nonvolatile regs */

Since r12 is allowed to be corrupted across a function call, we
probably don't need to save it.

> +	str	sp, [r0], #4
> +ARM(	msr	cpsr_c, #SVC_MODE					)
> +THUMB(	mov	r2, #SVC_MODE						)
> +THUMB(	msr	cpsr_c, r2						)
> +	mrs	r2, spsr
> +	stm	r0!, {r2,lr}		/* SVC SPSR, SVC regs */
> +	str	sp, [r0], #4
> +	msr	cpsr, r1		/* restore mode at entry */
> +	push	{lr}
> +	bl	__save_processor_state

<aside>

Structurally, we seem to have:

swsusp_arch_suspend {
	/* save some processor state */
	__save_processor_state();

	swsusp_save();
}

Is __save_processor_state() intended to encapsulate all the CPU-model-
specific state saving?  Excuse my ignorance of previous conversations...

</aside>

> +	pop	{lr}
> +	b	swsusp_save
> +ENDPROC(swsusp_arch_suspend)

I'm not too familiar with the suspend/resume stuff, so I may be asking
a dumb question here, but:

Where do we save/restore r8_FIQ..r13_FIQ, r13_IRQ, r13_UND and r13_ABT?
(I'm assuming there's no reason to save/restore r14 or SPSR for any
exception mode, since we presumably aren't going to suspend/resume
from inside an exception handler (?))

The exception stack pointers might conceivably be reinitialised to
sane values on resume, without necessarily needing to save/restore
them, providing my assumption in the previous paragraph is correct.

r8_FIQ..r12_FIQ can store arbitrary state used by the FIQ handler,
if FIQ is in use.  Can we expect any driver using FIQ to save/restore
this state itself, rather than doing it globally?  This may be
reasonable.

Cheers
---Dave
Russell King - ARM Linux May 20, 2011, 6:05 p.m. UTC | #2
On Fri, May 20, 2011 at 12:37:58PM +0100, Dave Martin wrote:
> Where do we save/restore r8_FIQ..r13_FIQ, r13_IRQ, r13_UND and r13_ABT?
> (I'm assuming there's no reason to save/restore r14 or SPSR for any
> exception mode, since we presumably aren't going to suspend/resume
> from inside an exception handler (?))

There is absolutely no need to save r13 for _any_ of the abort modes.  As
we have to set them up at boot time to fixed values, we keep that code
around, and in the resume paths we re-execute that initialization code.
cpu_init().

Out of the list you mention above, the only thing which isn't saved are the
FIQ registers, and that's a problem for S2RAM too, and should be done by
arch/arm/kernel/fiq.c hooking into the suspend paths.
tip-bot for Dave Martin May 23, 2011, 10:01 a.m. UTC | #3
On Fri, May 20, 2011 at 07:05:10PM +0100, Russell King - ARM Linux wrote:
> On Fri, May 20, 2011 at 12:37:58PM +0100, Dave Martin wrote:
> > Where do we save/restore r8_FIQ..r13_FIQ, r13_IRQ, r13_UND and r13_ABT?
> > (I'm assuming there's no reason to save/restore r14 or SPSR for any
> > exception mode, since we presumably aren't going to suspend/resume
> > from inside an exception handler (?))
> 
> There is absolutely no need to save r13 for _any_ of the abort modes.  As
> we have to set them up at boot time to fixed values, we keep that code
> around, and in the resume paths we re-execute that initialization code.
> cpu_init().

That seemed a sensible possibility, but I'm still not familiar with some
of the details.  It was, as I suspected, a dumb question from me...

> Out of the list you mention above, the only thing which isn't saved are the
> FIQ registers, and that's a problem for S2RAM too, and should be done by
> arch/arm/kernel/fiq.c hooking into the suspend paths.

The alternative view is that the driver using FIQ owns the state in the FIQ
mode registers and is therefore responsible for saving and restoring it
across suspend/resume.  If so, then any additional globally implemented
state save/restore of the FIQ mode state is unnecessary.

Do you have an opinion on which is the better model?

Requiring the driver to take responsibility might encourage the driver
writers to think about all the implications of save/restore on their
driver: saving/restoring the FIQ mode registers is unlikely to be enough
by itself.  Depending on the state of the driver, it might also be
unnecessary (though it's only a few words of state, so not necessarily
worth optimising).

Cheers
---Dave
Russell King - ARM Linux May 23, 2011, 10:12 a.m. UTC | #4
On Mon, May 23, 2011 at 11:01:32AM +0100, Dave Martin wrote:
> On Fri, May 20, 2011 at 07:05:10PM +0100, Russell King - ARM Linux wrote:
> > Out of the list you mention above, the only thing which isn't saved are the
> > FIQ registers, and that's a problem for S2RAM too, and should be done by
> > arch/arm/kernel/fiq.c hooking into the suspend paths.
> 
> The alternative view is that the driver using FIQ owns the state in the FIQ
> mode registers and is therefore responsible for saving and restoring it
> across suspend/resume.  If so, then any additional globally implemented
> state save/restore of the FIQ mode state is unnecessary.

This seems to be most sensible - if the FIQ is being used as a software-DMA,
then the hardware side of that needs to be cleanly shutdown (eg, waiting for
the DMA to complete before proceeding) to ensure no loss of data.  That
can't happen from within the FIQ code.

I also wonder about issues of secure/non-secure stuff here too - what
that means is that if we have a driver using FIQ mode, then we have
FIQ state to save, but if not there's probably no point (or even any
way) to save that state ourselves anyway.
tip-bot for Dave Martin May 23, 2011, 11:16 a.m. UTC | #5
On Mon, May 23, 2011 at 11:12:20AM +0100, Russell King - ARM Linux wrote:
> On Mon, May 23, 2011 at 11:01:32AM +0100, Dave Martin wrote:
> > On Fri, May 20, 2011 at 07:05:10PM +0100, Russell King - ARM Linux wrote:
> > > Out of the list you mention above, the only thing which isn't saved are the
> > > FIQ registers, and that's a problem for S2RAM too, and should be done by
> > > arch/arm/kernel/fiq.c hooking into the suspend paths.
> > 
> > The alternative view is that the driver using FIQ owns the state in the FIQ
> > mode registers and is therefore responsible for saving and restoring it
> > across suspend/resume.  If so, then any additional globally implemented
> > state save/restore of the FIQ mode state is unnecessary.
> 
> This seems to be most sensible - if the FIQ is being used as a software-DMA,
> then the hardware side of that needs to be cleanly shutdown (eg, waiting for
> the DMA to complete before proceeding) to ensure no loss of data.  That
> can't happen from within the FIQ code.

OK.  Frank suggested putting comments to this effect in <asm/fiq.h>.

I think something along these lines might be appropriate:

/*
 * The FIQ mode registers are not magically preserved across suspend/resume.
 *
 * Drivers which require these registers to be preserved across power
 * management operations must implement appropriate suspend/resume handlers
 * to save and restore them.
 */

Is the header file a good place for this, or is there some other better
place to put it?

> I also wonder about issues of secure/non-secure stuff here too - what
> that means is that if we have a driver using FIQ mode, then we have
> FIQ state to save, but if not there's probably no point (or even any
> way) to save that state ourselves anyway.

That argument may apply to a ton of state in the Secure World, not just
the FIQ registers

The likely model there is that the Secure World must hook somewhere into
Linux suspend/resume too, so that its hooks can be called at the
appropriate times.  This could involve a driver which saves/restores the
state of the Secure World, or low-level hooks which get called from
the platform power management code.

Either way, Linux doesn't need to be doing anything special except for
calling those hooks, just as for any other driver.

Cheers
---Dave
Russell King - ARM Linux May 23, 2011, 4:11 p.m. UTC | #6
On Mon, May 23, 2011 at 12:16:31PM +0100, Dave Martin wrote:
> On Mon, May 23, 2011 at 11:12:20AM +0100, Russell King - ARM Linux wrote:
> > On Mon, May 23, 2011 at 11:01:32AM +0100, Dave Martin wrote:
> > > On Fri, May 20, 2011 at 07:05:10PM +0100, Russell King - ARM Linux wrote:
> > > > Out of the list you mention above, the only thing which isn't saved are the
> > > > FIQ registers, and that's a problem for S2RAM too, and should be done by
> > > > arch/arm/kernel/fiq.c hooking into the suspend paths.
> > > 
> > > The alternative view is that the driver using FIQ owns the state in the FIQ
> > > mode registers and is therefore responsible for saving and restoring it
> > > across suspend/resume.  If so, then any additional globally implemented
> > > state save/restore of the FIQ mode state is unnecessary.
> > 
> > This seems to be most sensible - if the FIQ is being used as a software-DMA,
> > then the hardware side of that needs to be cleanly shutdown (eg, waiting for
> > the DMA to complete before proceeding) to ensure no loss of data.  That
> > can't happen from within the FIQ code.
> 
> OK.  Frank suggested putting comments to this effect in <asm/fiq.h>.
> 
> I think something along these lines might be appropriate:
> 
> /*
>  * The FIQ mode registers are not magically preserved across suspend/resume.
>  *
>  * Drivers which require these registers to be preserved across power
>  * management operations must implement appropriate suspend/resume handlers
>  * to save and restore them.
>  */
> 
> Is the header file a good place for this, or is there some other better
> place to put it?

I tend to suggest that the header file is the right place, because that's
where the interface is defined.  Other people argue that its more likely
to be seen in the implementation (fiq.c).  So I'm tempted to say both,
but lets go with fiq.h for the time being.

> That argument may apply to a ton of state in the Secure World, not just
> the FIQ registers

It very much does, and I know OMAP has some kind of SMC call to handle
this.
tip-bot for Dave Martin May 23, 2011, 4:38 p.m. UTC | #7
On Mon, May 23, 2011 at 05:11:49PM +0100, Russell King - ARM Linux wrote:
> On Mon, May 23, 2011 at 12:16:31PM +0100, Dave Martin wrote:
> > On Mon, May 23, 2011 at 11:12:20AM +0100, Russell King - ARM Linux wrote:
> > > On Mon, May 23, 2011 at 11:01:32AM +0100, Dave Martin wrote:
> > > > On Fri, May 20, 2011 at 07:05:10PM +0100, Russell King - ARM Linux wrote:
> > > > > Out of the list you mention above, the only thing which isn't saved are the
> > > > > FIQ registers, and that's a problem for S2RAM too, and should be done by
> > > > > arch/arm/kernel/fiq.c hooking into the suspend paths.
> > > > 
> > > > The alternative view is that the driver using FIQ owns the state in the FIQ
> > > > mode registers and is therefore responsible for saving and restoring it
> > > > across suspend/resume.  If so, then any additional globally implemented
> > > > state save/restore of the FIQ mode state is unnecessary.
> > > 
> > > This seems to be most sensible - if the FIQ is being used as a software-DMA,
> > > then the hardware side of that needs to be cleanly shutdown (eg, waiting for
> > > the DMA to complete before proceeding) to ensure no loss of data.  That
> > > can't happen from within the FIQ code.
> > 
> > OK.  Frank suggested putting comments to this effect in <asm/fiq.h>.
> > 
> > I think something along these lines might be appropriate:
> > 
> > /*
> >  * The FIQ mode registers are not magically preserved across suspend/resume.
> >  *
> >  * Drivers which require these registers to be preserved across power
> >  * management operations must implement appropriate suspend/resume handlers
> >  * to save and restore them.
> >  */
> > 
> > Is the header file a good place for this, or is there some other better
> > place to put it?
> 
> I tend to suggest that the header file is the right place, because that's
> where the interface is defined.  Other people argue that its more likely
> to be seen in the implementation (fiq.c).  So I'm tempted to say both,
> but lets go with fiq.h for the time being.

OK -- the {get,set}_fiq_regs patch is currently in your patch system.

If you have no objection, I'll submit a patch adding the above text to apply
on top of the other patch (or, if possible, orthogonally).

---Dave

> 
> > That argument may apply to a ton of state in the Secure World, not just
> > the FIQ registers
> 
> It very much does, and I know OMAP has some kind of SMC call to handle
> this.
Frank Hofmann May 24, 2011, 12:33 p.m. UTC | #8
On Mon, 23 May 2011, Dave Martin wrote:

> On Mon, May 23, 2011 at 05:11:49PM +0100, Russell King - ARM Linux wrote:
[ ... ]
>> I tend to suggest that the header file is the right place, because that's
>> where the interface is defined.  Other people argue that its more likely
>> to be seen in the implementation (fiq.c).  So I'm tempted to say both,
>> but lets go with fiq.h for the time being.
>
> OK -- the {get,set}_fiq_regs patch is currently in your patch system.
>
> If you have no objection, I'll submit a patch adding the above text to apply
> on top of the other patch (or, if possible, orthogonally).
>
> ---Dave

Thanks a lot !

>
>>
>>> That argument may apply to a ton of state in the Secure World, not just
>>> the FIQ registers
>>
>> It very much does, and I know OMAP has some kind of SMC call to handle
>> this.
>

Yes, _omap_sram_idle() does that bit, it gives a physical address to the 
OMAP ROM code to save/restore the "secure state" in, triggered via smc.

Anyway, architecturally it seems to be much cleaner to _allow_ device 
drivers (or machine-specific hooks) to save/restore _more_ state than 
whatever the "core suspend code" would do, instead of _forcing_ the core 
suspend code to do everything-and-the-kitchen-sink.
That's where things like FIQ or secure state would be.

FrankH.
Frank Hofmann May 24, 2011, 1:27 p.m. UTC | #9
Hi,


Regarding ARM hibernation / suspend-to-disk, now that the glue part of the 
code has "settled" it's probably time to think about the gory low-level 
things, what's been delegated to the __save/__restore_processor_state() 
hooks.


Russell King did the cpu_suspend / cpu_resume cleanup in Feb/2011,

 	http://lists.arm.linux.org.uk/lurker/message/20110211.161754.78371a42.en.html

which cleansed the mach-.../ sleep code in favour of moving these things 
into processor type support hooks in arch/arm/mm/proc-*.S instead.


The key thing here was the addition of cpu_do_suspend/cpu_do_resume hooks 
which save/restore state to/from a caller-supplied buffer, and then return 
to their callers. I.e. pure auxilliary, "no side effects" (other than, for 
resume, of course, what's desired).
The power-down / enter-low-power parts were separated from that.


From my point of view, all platforms that use these should be able to do 
the cpu-specific parts of hibernation via a simple addition, say, to 
arch/arm/kernel/sleep.S (that's already using <asm/glue-proc.h> so 
cpu_do_... exists), of something like:


#if defined(CONFIG_HAS_ARM_GENERIC_CPU_SUSPEND_SUPPORT) && defined(CONFIG_HIBERNATION)
ENTRY(__save_processor_state)
 	b	cpu_do_suspend
ENDPROC(__save_processor_state)

ENTRY(__restore_processor_state)
 	stmfd	sp!,{r4-r11,lr}
 	bl	cpu_do_resume
 	ldmfd	sp!,{r4-r11,pc}
ENDPROC(__restore_processor_state)
#endif


and "select HAS_ARM_GENERIC_CPU_SUSPEND_SUPPORT if CONFIG_PM" for the CPU 
architectures that compile suitable mm/proc-*.S in.


That would do because the cpu_do_suspend/resume hooks perform the same CP 
reg save/restore operations as the "illustration patches" I've posted for 
OMAP3 and samsung 64xx (in fact, the latter is one of the platforms 
changed via Russell's patch series).



From my point of view, this assumes the following:

- cpu_do_suspend() takes one argument, a virtual address of the buffer to
   save the state into. It is a "well-behaved" EABI-compliant func that
   preserves nonvolatile regs.
   It returns to LR.

- cpu_do_resume() assumes it can clobber registers; it takes one argument,
   the address (not necessarily virtual) of the buffer to restore state
   from.
   The function does not depend on the MMU being OFF (could be called with
   the MMU on, it'll "just clobber the state").
   It also returns to LR.

Do these assumptions hold ?


If so, then would it be ok to create a patch for 2.6.39 onwards (baselined 
against ARM devel-stable), along the lines just mentioned ?


How would one best test something as sweeping as that ?


The above suggestion still leaves a hole out for machine-specific 
implementations - unselect the GENERIC config and implement the hook 
yourself. Seems desireable ?


Regarding pre-2.6.38 kernels, are there any plans to backport the 
cpu_suspend changes to any -stable series ?



Thanks in advance for comments / suggestions / corrections,
FrankH.
diff mbox

Patch

 arch/arm/Kconfig                  |    3 +
 arch/arm/include/asm/memory.h     |    1 +
 arch/arm/include/asm/suspend.h    |    6 ++
 arch/arm/kernel/cpu.c             |   65 ++++++++++++++++++++++++++
 arch/arm/kernel/swsusp.S          |   92 +++++++++++++++++++++++++++++++++++++
 arch/arm/kernel/vmlinux.lds.S     |    3 +-
 include/asm-generic/vmlinux.lds.h |    2 +-
 7 files changed, 170 insertions(+), 2 deletions(-)

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 6b6786c..859dd86 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -198,6 +198,9 @@  config VECTORS_BASE
 config ARCH_HAS_CPU_IDLE_WAIT
 	def_bool y
 
+config ARCH_HIBERNATION_POSSIBLE
+	def_bool n
+
 source "init/Kconfig"
 
 source "kernel/Kconfig.freezer"
diff --git a/arch/arm/include/asm/memory.h b/arch/arm/include/asm/memory.h
index 5421d82..23e93a6 100644
--- a/arch/arm/include/asm/memory.h
+++ b/arch/arm/include/asm/memory.h
@@ -191,6 +191,7 @@  static inline void *phys_to_virt(unsigned long x)
  */
 #define __pa(x)			__virt_to_phys((unsigned long)(x))
 #define __va(x)			((void *)__phys_to_virt((unsigned long)(x)))
+#define __pa_symbol(x)		__pa(RELOC_HIDE((unsigned long)(x),0))
 #define pfn_to_kaddr(pfn)	__va((pfn) << PAGE_SHIFT)
 
 /*
diff --git a/arch/arm/include/asm/suspend.h b/arch/arm/include/asm/suspend.h
new file mode 100644
index 0000000..7ab1fd2
--- /dev/null
+++ b/arch/arm/include/asm/suspend.h
@@ -0,0 +1,6 @@ 
+#ifndef __ASM_ARM_SUSPEND_H
+#define __ASM_ARM_SUSPEND_H
+
+static inline int arch_prepare_suspend(void) { return 0; }
+
+#endif /* __ASM_ARM_SUSPEND_H */
diff --git a/arch/arm/kernel/cpu.c b/arch/arm/kernel/cpu.c
new file mode 100644
index 0000000..764c8fa
--- /dev/null
+++ b/arch/arm/kernel/cpu.c
@@ -0,0 +1,65 @@ 
+/*
+ * Hibernation support specific for ARM
+ *
+ * Derived from work on ARM hibernation support by:
+ *
+ * Ubuntu project, hibernation support for mach-dove
+ * Copyright (C) 2010 Nokia Corporation (Hiroshi Doyu)
+ * Copyright (C) 2010 Texas Instruments, Inc. (Teerth Reddy et al.)
+ *	https://lkml.org/lkml/2010/6/18/4
+ *	https://lists.linux-foundation.org/pipermail/linux-pm/2010-June/027422.html
+ *	https://patchwork.kernel.org/patch/96442/
+ *
+ * Copyright (C) 2006 Rafael J. Wysocki <rjw@sisk.pl>
+ *
+ * License terms: GNU General Public License (GPL) version 2
+ */
+
+#include <linux/mm.h>
+#include <linux/sched.h>
+#include <linux/suspend.h>
+#include <asm/tlbflush.h>
+
+extern const void __nosave_begin, __nosave_end;
+
+int pfn_is_nosave(unsigned long pfn)
+{
+	unsigned long nosave_begin_pfn = __pa_symbol(&__nosave_begin) >> PAGE_SHIFT;
+	unsigned long nosave_end_pfn = PAGE_ALIGN(__pa_symbol(&__nosave_end)) >> PAGE_SHIFT;
+
+	return (pfn >= nosave_begin_pfn) && (pfn < nosave_end_pfn);
+}
+
+void save_processor_state(void)
+{
+	flush_thread();
+}
+
+void restore_processor_state(void)
+{
+	local_flush_tlb_all();
+}
+
+u8 __swsusp_arch_ctx[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
+u8 __swsusp_resume_stk[PAGE_SIZE/2] __nosavedata;
+
+/*
+ * The framework loads the hibernation image into this linked list,
+ * for swsusp_arch_resume() to copy back to the proper destinations.
+ *
+ * To make this work if resume is triggered from initramfs, the
+ * pagetables need to be switched to allow writes to kernel mem.
+ */
+void notrace __swsusp_arch_restore_prepare(void)
+{
+	cpu_switch_mm(__virt_to_phys(swapper_pg_dir), current->active_mm);
+}
+
+void notrace __swsusp_arch_restore_image(void)
+{
+	extern struct pbe *restore_pblist;
+	struct pbe *pbe;
+
+	for (pbe = restore_pblist; pbe; pbe = pbe->next)
+		copy_page(pbe->orig_address, pbe->address);
+}
diff --git a/arch/arm/kernel/swsusp.S b/arch/arm/kernel/swsusp.S
new file mode 100644
index 0000000..fb260a7
--- /dev/null
+++ b/arch/arm/kernel/swsusp.S
@@ -0,0 +1,92 @@ 
+/*
+ * Hibernation support specific for ARM
+ *
+ * Based on work by:
+ *
+ * Ubuntu project, hibernation support for mach-dove,
+ * Copyright (C) 2010 Nokia Corporation (Hiroshi Doyu)
+ * Copyright (C) 2010 Texas Instruments, Inc. (Teerth Reddy et al.)
+ *	https://lkml.org/lkml/2010/6/18/4
+ *	https://lists.linux-foundation.org/pipermail/linux-pm/2010-June/027422.html
+ *	https://patchwork.kernel.org/patch/96442/
+ *
+ * Copyright (C) 2006 Rafael J. Wysocki <rjw@sisk.pl>
+ *
+ * License terms: GNU General Public License (GPL) version 2
+ */
+
+#include <linux/linkage.h>
+#include <asm/memory.h>
+#include <asm/page.h>
+#include <asm/cache.h>
+#include <asm/ptrace.h>
+
+/*
+ * Save the current CPU state before suspend / poweroff.
+ */
+ENTRY(swsusp_arch_suspend)
+	ldr	r0, =__swsusp_arch_ctx
+	mrs	r1, cpsr
+	str	r1, [r0], #4		/* CPSR */
+ARM(	msr	cpsr_c, #SYSTEM_MODE					)
+THUMB(	mov	r2, #SYSTEM_MODE					)
+THUMB(	msr	cpsr_c, r2						)
+	stm	r0!, {r4-r12,lr}	/* nonvolatile regs */
+	str	sp, [r0], #4
+ARM(	msr	cpsr_c, #SVC_MODE					)
+THUMB(	mov	r2, #SVC_MODE						)
+THUMB(	msr	cpsr_c, r2						)
+	mrs	r2, spsr
+	stm	r0!, {r2,lr}		/* SVC SPSR, SVC regs */
+	str	sp, [r0], #4
+	msr	cpsr, r1		/* restore mode at entry */
+	push	{lr}
+	bl	__save_processor_state
+	pop	{lr}
+	b	swsusp_save
+ENDPROC(swsusp_arch_suspend)
+
+/*
+ * Restore the memory image from the pagelists, and load the CPU registers
+ * from saved state.
+ */
+ENTRY(swsusp_arch_resume)
+	bl	__swsusp_arch_restore_prepare
+	/*
+	 * Switch stack to a nosavedata region to make sure image restore
+	 * doesn't clobber it underneath itself.
+	 */
+	ldr	sp, =(__swsusp_resume_stk + PAGE_SIZE / 2)
+	bl	__swsusp_arch_restore_image
+
+	/*
+	 * Restore the CPU registers.
+	 */
+ARM(	msr	cpsr_c, #SYSTEM_MODE					)
+THUMB(	mov	r2, #SYSTEM_MODE					)
+THUMB(	msr	cpsr_c, r2						)
+	ldr	r0, =__swsusp_arch_ctx
+	ldr	r1, [r0], #4
+	msr	cpsr_cxsf, r1
+	ldm	r0!, {r4-r12,lr}
+	ldr	sp, [r0], #4
+ARM(	msr	cpsr_c, #SVC_MODE					)
+THUMB(	mov	r2, #SVC_MODE						)
+THUMB(	msr	cpsr_c, r2						)
+	ldm	r0!, {r2,lr}
+	ldr	sp, [r0], #4
+	msr	spsr_cxsf, r2
+	msr	cpsr_c, r1
+
+	/*
+	 * From here on we have a valid stack again. Core state is
+	 * not restored yet, redirect to the machine-specific
+	 * implementation to get that done.
+	 * Resume has succeeded at this point; if the machine-specific
+	 * code wants to fail it needs to panic.
+	 */
+	mov	r1, #0
+	push	{r1,lr}
+	bl	__restore_processor_state	/* restore core state */
+	pop	{r0,pc}
+ENDPROC(swsusp_arch_resume)
diff --git a/arch/arm/kernel/vmlinux.lds.S b/arch/arm/kernel/vmlinux.lds.S
index 4957e13..e691c77 100644
--- a/arch/arm/kernel/vmlinux.lds.S
+++ b/arch/arm/kernel/vmlinux.lds.S
@@ -153,7 +153,6 @@  SECTIONS
 		__init_end = .;
 #endif
 
-		NOSAVE_DATA
 		CACHELINE_ALIGNED_DATA(32)
 
 		/*
@@ -176,6 +175,8 @@  SECTIONS
 	}
 	_edata_loc = __data_loc + SIZEOF(.data);
 
+	NOSAVE_DATA
+
 #ifdef CONFIG_HAVE_TCM
         /*
 	 * We align everything to a page boundary so we can
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index b6e818f..0d39ae0 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -171,7 +171,7 @@ 
 #define NOSAVE_DATA							\
 	. = ALIGN(PAGE_SIZE);						\
 	VMLINUX_SYMBOL(__nosave_begin) = .;				\
-	*(.data.nosave)							\
+	.data.nosave : { *(.data.nosave) }				\
 	. = ALIGN(PAGE_SIZE);						\
 	VMLINUX_SYMBOL(__nosave_end) = .;