| Message ID | 20241114090810.1961175-4-andrew.cooper3@citrix.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | x86/trampoline: Layout description improvements. | expand |
On 14.11.2024 10:08, Andrew Cooper wrote: > This is, to the best of my knowledge, accurate. I am providing no comment on > how sane I believe it to be. > > At the time of writing, the sizes of the regions are: > > offset size > AP: 0x0000 0x00b0 > S3: 0x00b0 0x0229 > Boot: 0x02d9 0x1697 > Heap: 0x1970 0xe690 > Stack: 0xf000 0x1000 > > and wakeup_stack overlays boot_edd_info. > > Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> > Reviewed-by: Frediano Ziglio <frediano.ziglio@cloud.com> Acked-by: Jan Beulich <jbeulich@suse.com>
On Thu Nov 14, 2024 at 9:08 AM GMT, Andrew Cooper wrote: > This is, to the best of my knowledge, accurate. I am providing no comment on > how sane I believe it to be. > > At the time of writing, the sizes of the regions are: > > offset size > AP: 0x0000 0x00b0 > S3: 0x00b0 0x0229 > Boot: 0x02d9 0x1697 > Heap: 0x1970 0xe690 > Stack: 0xf000 0x1000 > > and wakeup_stack overlays boot_edd_info. > > Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> > Reviewed-by: Frediano Ziglio <frediano.ziglio@cloud.com> > --- > CC: Jan Beulich <JBeulich@suse.com> > CC: Roger Pau Monné <roger.pau@citrix.com> > CC: Daniel P. Smith <dpsmith@apertussolutions.com> > CC: Frediano Ziglio <frediano.ziglio@cloud.com> > CC: Alejandro Vallejo <alejandro.vallejo@cloud.com> > > v2: > * Rebase over the introduction of trampoline_perm_end > * Fix the description of the boot stack position > --- > xen/arch/x86/include/asm/trampoline.h | 57 ++++++++++++++++++++++++++- > 1 file changed, 55 insertions(+), 2 deletions(-) > > diff --git a/xen/arch/x86/include/asm/trampoline.h b/xen/arch/x86/include/asm/trampoline.h > index 8c1e0b48c2c9..559111d2ccfc 100644 > --- a/xen/arch/x86/include/asm/trampoline.h > +++ b/xen/arch/x86/include/asm/trampoline.h > @@ -37,12 +37,65 @@ > * manually as part of placement. > */ > > +/* > + * Layout of the trampoline. Logical areas, in ascending order: > + * > + * 1) AP boot: > + * > + * The INIT-SIPI-SIPI entrypoint. This logic is stack-less so the identity > + * mapping (which must be executable) can at least be Read Only. > + * > + * 2) S3 resume: > + * > + * The S3 wakeup logic may need to interact with the BIOS, so needs a > + * stack. The stack pointer is set to trampoline_phys + 4k and clobbers an > + * arbitrary part of the the boot trampoline. The stack is only used with > + * paging disabled. > + * > + * 3) Boot trampoline: > + * > + * The boot trampoline collects data from the BIOS (E820/EDD/EDID/etc), so > + * needs a stack. The stack pointer is set to trampoline_phys + 64k, is 4k > + * in size, and only used with paging disabled. > + * > + * 4) Heap space: > + * > + * The first 1k of heap space is statically allocated scratch space for > + * VESA information. > + * > + * The remainder of the heap is used by reloc(), logic which is otherwise > + * outside of the trampoline, to collect the bootloader metadata (cmdline, Wh> + * module list, etc). It does so with a bump allocator starting from the > + * end of the heap and allocating backwards. > + * > + * 5) Boot stack: > + * > + * The boot stack is 4k in size at the end of the trampoline, taking the > + * total trampoline size to 64k. > + * > + * Therefore, when placed, it looks somewhat like this: > + * > + * +--- trampoline_phys > + * v > + * |<-------------------------------64K------------------------------->| > + * |<-----4K----->| |<---4K--->| > + * +-------+------+-+---------------------------------------+----------+ > + * | AP+S3 | Boot | Heap | Stack | > + * +-------+------+-+---------------------------------------+----------+ > + * ^ ^ <~~^ ^ <~~^ <~~^ > + * | | | +- trampoline_end[] | | > + * | | +--- wakeup_stack reloc() allocator -+ | > + * | +---------- trampoline_perm_end Boot Stack ------------+ > + * +------------------ trampoline_start[] > + */ Neat. Nothing like a pretty picture to properly explain things. > + > #include <xen/compiler.h> > #include <xen/types.h> > > /* > - * Start and end of the trampoline section, as linked into Xen. It is within > - * the .init section and reclaimed after boot. > + * Start and end of the trampoline section, as linked into Xen. This covers > + * the AP, S3 and Boot regions, but not the heap or stack. It is within the > + * .init section and reclaimed after boot. How can it be reclaimed after boot if it's used for S3 wakeups? I assume you meant that the heap and stack are reclaimed after boot, but from that wording it sounds like the other way around. > */ > /* SAF-0-safe */ > extern char trampoline_start[], trampoline_end[]; Cheers, Alejandro
On 14/11/2024 10:48 am, Alejandro Vallejo wrote: > On Thu Nov 14, 2024 at 9:08 AM GMT, Andrew Cooper wrote: >> diff --git a/xen/arch/x86/include/asm/trampoline.h b/xen/arch/x86/include/asm/trampoline.h >> index 8c1e0b48c2c9..559111d2ccfc 100644 >> --- a/xen/arch/x86/include/asm/trampoline.h >> +++ b/xen/arch/x86/include/asm/trampoline.h >> @@ -37,12 +37,65 @@ >> * manually as part of placement. >> */ >> >> +/* >> + * Layout of the trampoline. Logical areas, in ascending order: >> + * >> + * 1) AP boot: >> + * >> + * The INIT-SIPI-SIPI entrypoint. This logic is stack-less so the identity >> + * mapping (which must be executable) can at least be Read Only. >> + * >> + * 2) S3 resume: >> + * >> + * The S3 wakeup logic may need to interact with the BIOS, so needs a >> + * stack. The stack pointer is set to trampoline_phys + 4k and clobbers an >> + * arbitrary part of the the boot trampoline. The stack is only used with >> + * paging disabled. >> + * >> + * 3) Boot trampoline: >> + * >> + * The boot trampoline collects data from the BIOS (E820/EDD/EDID/etc), so >> + * needs a stack. The stack pointer is set to trampoline_phys + 64k, is 4k >> + * in size, and only used with paging disabled. >> + * >> + * 4) Heap space: >> + * >> + * The first 1k of heap space is statically allocated scratch space for >> + * VESA information. >> + * >> + * The remainder of the heap is used by reloc(), logic which is otherwise >> + * outside of the trampoline, to collect the bootloader metadata (cmdline, > Wh> + * module list, etc). It does so with a bump allocator starting from the >> + * end of the heap and allocating backwards. Was this a typo replying to the email? >> + * >> + * 5) Boot stack: >> + * >> + * The boot stack is 4k in size at the end of the trampoline, taking the >> + * total trampoline size to 64k. >> + * >> + * Therefore, when placed, it looks somewhat like this: >> + * >> + * +--- trampoline_phys >> + * v >> + * |<-------------------------------64K------------------------------->| >> + * |<-----4K----->| |<---4K--->| >> + * +-------+------+-+---------------------------------------+----------+ >> + * | AP+S3 | Boot | Heap | Stack | >> + * +-------+------+-+---------------------------------------+----------+ >> + * ^ ^ <~~^ ^ <~~^ <~~^ >> + * | | | +- trampoline_end[] | | >> + * | | +--- wakeup_stack reloc() allocator -+ | >> + * | +---------- trampoline_perm_end Boot Stack ------------+ >> + * +------------------ trampoline_start[] >> + */ > Neat. Nothing like a pretty picture to properly explain things. > >> + >> #include <xen/compiler.h> >> #include <xen/types.h> >> >> /* >> - * Start and end of the trampoline section, as linked into Xen. It is within >> - * the .init section and reclaimed after boot. >> + * Start and end of the trampoline section, as linked into Xen. This covers >> + * the AP, S3 and Boot regions, but not the heap or stack. It is within the >> + * .init section and reclaimed after boot. > How can it be reclaimed after boot if it's used for S3 wakeups? I assume you > meant that the heap and stack are reclaimed after boot, but from that wording > it sounds like the other way around. This is the one bit that is slightly problematic to represent. trampoline_{start,end}[] describe the AP/S3/Boot text/data *in the Xen image*, which is in the .init section. trampoline_phys is where trampoline_start[] ends up when placed. Maybe I should have "Note: trampoline_start[] and trampoline_end[] represent the shown boundaries as linked in Xen" at the bottom of the diagram? ~Andrew
On 14/11/2024 11:17 am, Andrew Cooper wrote: > On 14/11/2024 10:48 am, Alejandro Vallejo wrote: >> On Thu Nov 14, 2024 at 9:08 AM GMT, Andrew Cooper wrote: >>> diff --git a/xen/arch/x86/include/asm/trampoline.h b/xen/arch/x86/include/asm/trampoline.h >>> index 8c1e0b48c2c9..559111d2ccfc 100644 >>> --- a/xen/arch/x86/include/asm/trampoline.h >>> +++ b/xen/arch/x86/include/asm/trampoline.h >>> @@ -37,12 +37,65 @@ >>> * manually as part of placement. >>> */ >>> >>> +/* >>> + * Layout of the trampoline. Logical areas, in ascending order: >>> + * >>> + * 1) AP boot: >>> + * >>> + * The INIT-SIPI-SIPI entrypoint. This logic is stack-less so the identity >>> + * mapping (which must be executable) can at least be Read Only. >>> + * >>> + * 2) S3 resume: >>> + * >>> + * The S3 wakeup logic may need to interact with the BIOS, so needs a >>> + * stack. The stack pointer is set to trampoline_phys + 4k and clobbers an >>> + * arbitrary part of the the boot trampoline. The stack is only used with >>> + * paging disabled. >>> + * >>> + * 3) Boot trampoline: >>> + * >>> + * The boot trampoline collects data from the BIOS (E820/EDD/EDID/etc), so >>> + * needs a stack. The stack pointer is set to trampoline_phys + 64k, is 4k >>> + * in size, and only used with paging disabled. >>> + * >>> + * 4) Heap space: >>> + * >>> + * The first 1k of heap space is statically allocated scratch space for >>> + * VESA information. >>> + * >>> + * The remainder of the heap is used by reloc(), logic which is otherwise >>> + * outside of the trampoline, to collect the bootloader metadata (cmdline, >> Wh> + * module list, etc). It does so with a bump allocator starting from the >>> + * end of the heap and allocating backwards. > Was this a typo replying to the email? > > >>> + * >>> + * 5) Boot stack: >>> + * >>> + * The boot stack is 4k in size at the end of the trampoline, taking the >>> + * total trampoline size to 64k. >>> + * >>> + * Therefore, when placed, it looks somewhat like this: >>> + * >>> + * +--- trampoline_phys >>> + * v >>> + * |<-------------------------------64K------------------------------->| >>> + * |<-----4K----->| |<---4K--->| >>> + * +-------+------+-+---------------------------------------+----------+ >>> + * | AP+S3 | Boot | Heap | Stack | >>> + * +-------+------+-+---------------------------------------+----------+ >>> + * ^ ^ <~~^ ^ <~~^ <~~^ >>> + * | | | +- trampoline_end[] | | >>> + * | | +--- wakeup_stack reloc() allocator -+ | >>> + * | +---------- trampoline_perm_end Boot Stack ------------+ >>> + * +------------------ trampoline_start[] >>> + */ >> Neat. Nothing like a pretty picture to properly explain things. >> >>> + >>> #include <xen/compiler.h> >>> #include <xen/types.h> >>> >>> /* >>> - * Start and end of the trampoline section, as linked into Xen. It is within >>> - * the .init section and reclaimed after boot. >>> + * Start and end of the trampoline section, as linked into Xen. This covers >>> + * the AP, S3 and Boot regions, but not the heap or stack. It is within the >>> + * .init section and reclaimed after boot. >> How can it be reclaimed after boot if it's used for S3 wakeups? I assume you >> meant that the heap and stack are reclaimed after boot, but from that wording >> it sounds like the other way around. > This is the one bit that is slightly problematic to represent. > > trampoline_{start,end}[] describe the AP/S3/Boot text/data *in the Xen > image*, which is in the .init section. > > trampoline_phys is where trampoline_start[] ends up when placed. > > Maybe I should have "Note: trampoline_start[] and trampoline_end[] > represent the shown boundaries as linked in Xen" at the bottom of the > diagram? I'm going to go ahead and do this, and commit the series. ~Andrew
diff --git a/xen/arch/x86/include/asm/trampoline.h b/xen/arch/x86/include/asm/trampoline.h index 8c1e0b48c2c9..559111d2ccfc 100644 --- a/xen/arch/x86/include/asm/trampoline.h +++ b/xen/arch/x86/include/asm/trampoline.h @@ -37,12 +37,65 @@ * manually as part of placement. */ +/* + * Layout of the trampoline. Logical areas, in ascending order: + * + * 1) AP boot: + * + * The INIT-SIPI-SIPI entrypoint. This logic is stack-less so the identity + * mapping (which must be executable) can at least be Read Only. + * + * 2) S3 resume: + * + * The S3 wakeup logic may need to interact with the BIOS, so needs a + * stack. The stack pointer is set to trampoline_phys + 4k and clobbers an + * arbitrary part of the the boot trampoline. The stack is only used with + * paging disabled. + * + * 3) Boot trampoline: + * + * The boot trampoline collects data from the BIOS (E820/EDD/EDID/etc), so + * needs a stack. The stack pointer is set to trampoline_phys + 64k, is 4k + * in size, and only used with paging disabled. + * + * 4) Heap space: + * + * The first 1k of heap space is statically allocated scratch space for + * VESA information. + * + * The remainder of the heap is used by reloc(), logic which is otherwise + * outside of the trampoline, to collect the bootloader metadata (cmdline, + * module list, etc). It does so with a bump allocator starting from the + * end of the heap and allocating backwards. + * + * 5) Boot stack: + * + * The boot stack is 4k in size at the end of the trampoline, taking the + * total trampoline size to 64k. + * + * Therefore, when placed, it looks somewhat like this: + * + * +--- trampoline_phys + * v + * |<-------------------------------64K------------------------------->| + * |<-----4K----->| |<---4K--->| + * +-------+------+-+---------------------------------------+----------+ + * | AP+S3 | Boot | Heap | Stack | + * +-------+------+-+---------------------------------------+----------+ + * ^ ^ <~~^ ^ <~~^ <~~^ + * | | | +- trampoline_end[] | | + * | | +--- wakeup_stack reloc() allocator -+ | + * | +---------- trampoline_perm_end Boot Stack ------------+ + * +------------------ trampoline_start[] + */ + #include <xen/compiler.h> #include <xen/types.h> /* - * Start and end of the trampoline section, as linked into Xen. It is within - * the .init section and reclaimed after boot. + * Start and end of the trampoline section, as linked into Xen. This covers + * the AP, S3 and Boot regions, but not the heap or stack. It is within the + * .init section and reclaimed after boot. */ /* SAF-0-safe */ extern char trampoline_start[], trampoline_end[];
This is, to the best of my knowledge, accurate. I am providing no comment on how sane I believe it to be. At the time of writing, the sizes of the regions are: offset size AP: 0x0000 0x00b0 S3: 0x00b0 0x0229 Boot: 0x02d9 0x1697 Heap: 0x1970 0xe690 Stack: 0xf000 0x1000 and wakeup_stack overlays boot_edd_info. Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> Reviewed-by: Frediano Ziglio <frediano.ziglio@cloud.com> --- CC: Jan Beulich <JBeulich@suse.com> CC: Roger Pau Monné <roger.pau@citrix.com> CC: Daniel P. Smith <dpsmith@apertussolutions.com> CC: Frediano Ziglio <frediano.ziglio@cloud.com> CC: Alejandro Vallejo <alejandro.vallejo@cloud.com> v2: * Rebase over the introduction of trampoline_perm_end * Fix the description of the boot stack position --- xen/arch/x86/include/asm/trampoline.h | 57 ++++++++++++++++++++++++++- 1 file changed, 55 insertions(+), 2 deletions(-)