| Message ID | 20240315105902.160047-11-carlo.nonato@minervasys.tech (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Arm cache coloring | expand |
On 15.03.2024 11:58, Carlo Nonato wrote: > Add a new memory page allocator that implements the cache coloring mechanism. > The allocation algorithm enforces equal frequency distribution of cache > partitions, following the coloring configuration of a domain. This allows > for an even utilization of cache sets for every domain. > > Pages are stored in a color-indexed array of lists. Those lists are filled > by a simple init function which computes the color of each page. > When a domain requests a page, the allocator extract the page from the list > with the maximum number of free pages between those that the domain can > access, given its coloring configuration. Minor remark: I'm not a native speaker, but "between" here reads odd to me. I'd have expected perhaps "among". > --- a/docs/misc/xen-command-line.pandoc > +++ b/docs/misc/xen-command-line.pandoc > @@ -270,6 +270,20 @@ and not running softirqs. Reduce this if softirqs are not being run frequently > enough. Setting this to a high value may cause boot failure, particularly if > the NMI watchdog is also enabled. > > +### buddy-alloc-size (arm64) > +> `= <size>` > + > +> Default: `64M` > + > +Amount of memory reserved for the buddy allocator when colored allocator is > +active. This options is parsed only when LLC coloring support is enabled. Nit: s/parsed/used/ - the option is always parsed as long as LLC_COLORING=y. > @@ -1945,6 +1949,164 @@ static unsigned long avail_heap_pages( > return free_pages; > } > > +/************************* > + * COLORED SIDE-ALLOCATOR > + * > + * Pages are grouped by LLC color in lists which are globally referred to as the > + * color heap. Lists are populated in end_boot_allocator(). > + * After initialization there will be N lists where N is the number of > + * available colors on the platform. > + */ > +static struct page_list_head *__ro_after_init _color_heap; > +#define color_heap(color) (&_color_heap[color]) > + > +static unsigned long *__ro_after_init free_colored_pages; > + > +/* Memory required for buddy allocator to work with colored one */ > +#ifdef CONFIG_LLC_COLORING > +static unsigned long __initdata buddy_alloc_size = > + MB(CONFIG_BUDDY_ALLOCATOR_SIZE); > +size_param("buddy-alloc-size", buddy_alloc_size); > + > +#define domain_num_llc_colors(d) (d)->num_llc_colors > +#define domain_llc_color(d, i) (d)->llc_colors[i] > +#else > +static unsigned long __initdata buddy_alloc_size; > + > +#define domain_num_llc_colors(d) 0 > +#define domain_llc_color(d, i) 0 > +#endif > + > +static void free_color_heap_page(struct page_info *pg, bool need_scrub) > +{ > + unsigned int color = page_to_llc_color(pg); > + struct page_list_head *head = color_heap(color); > + > + spin_lock(&heap_lock); > + > + mark_page_free(pg, page_to_mfn(pg)); > + > + if ( need_scrub ) > + { > + pg->count_info |= PGC_need_scrub; > + poison_one_page(pg); > + } > + > + free_colored_pages[color]++; > + page_list_add(pg, head); May I please ask for a comment (or at least some wording in the description) as to the choice made here between head or tail insertion? When assuming that across a system there's no sharing of colors, preferably re-using cache-hot pages is certainly good. Whereas when colors can reasonably be expected to be shared, avoiding to quickly re-use a freed page can also have benefits. > +static struct page_info *alloc_color_heap_page(unsigned int memflags, > + const struct domain *d) > +{ > + struct page_info *pg = NULL; > + unsigned int i, color = 0; > + unsigned long max = 0; > + bool need_tlbflush = false; > + uint32_t tlbflush_timestamp = 0; > + bool need_scrub; > + > + if ( memflags >> _MEMF_bits ) > + return NULL; By mentioning MEMF_bits earlier on I meant to give an example. What about MEMF_node and in particular MEMF_exact_node? Certain other flags also aren't obvious as to being okay to silently ignore. > + spin_lock(&heap_lock); > + > + for ( i = 0; i < domain_num_llc_colors(d); i++ ) > + { > + unsigned long free = free_colored_pages[domain_llc_color(d, i)]; > + > + if ( free > max ) > + { > + color = domain_llc_color(d, i); > + pg = page_list_first(color_heap(color)); > + max = free; > + } > + } > + > + if ( !pg ) > + { > + spin_unlock(&heap_lock); > + return NULL; > + } > + > + need_scrub = pg->count_info & (PGC_need_scrub); > + pg->count_info = PGC_state_inuse | (pg->count_info & PGC_colored); Better PGC_preserved? > +static void __init init_color_heap_pages(struct page_info *pg, > + unsigned long nr_pages) > +{ > + unsigned int i; > + bool need_scrub = opt_bootscrub == BOOTSCRUB_IDLE; > + > + if ( buddy_alloc_size ) > + { > + unsigned long buddy_pages = min(PFN_DOWN(buddy_alloc_size), nr_pages); > + > + init_heap_pages(pg, buddy_pages); There's a corner case where init_heap_pages() would break when passed 0 as 2nd argument. I think you want to alter the enclosing if() to "if ( buddy_alloc_size >= PAGE_SIZE )" to be entirely certain to avoid that case. > +static void dump_color_heap(void) > +{ > + unsigned int color; > + > + printk("Dumping color heap info\n"); > + for ( color = 0; color < get_max_nr_llc_colors(); color++ ) > + if ( free_colored_pages[color] > 0 ) > + printk("Color heap[%u]: %lu pages\n", > + color, free_colored_pages[color]); > +} While having all of the code above from here outside of any #ifdef is helpful to prevent unintended breakage when changes are made and tested only on non-Arm64 targets, I'd still like to ask: Halfway recent compilers manage to eliminate everything? I'd like to avoid e.g. x86 being left with traces of coloring despite not being able at all to use it. > @@ -2485,7 +2660,10 @@ struct page_info *alloc_domheap_pages( > } > if ( assign_page(pg, order, d, memflags) ) > { > - free_heap_pages(pg, order, memflags & MEMF_no_scrub); > + if ( pg->count_info & PGC_colored ) > + free_color_heap_page(pg, memflags & MEMF_no_scrub); > + else > + free_heap_pages(pg, order, memflags & MEMF_no_scrub); > return NULL; > } > } > @@ -2568,7 +2746,10 @@ void free_domheap_pages(struct page_info *pg, unsigned int order) > scrub = 1; > } > > - free_heap_pages(pg, order, scrub); > + if ( pg->count_info & PGC_colored ) > + free_color_heap_page(pg, scrub); > + else > + free_heap_pages(pg, order, scrub); > } Instead of this, did you consider altering free_heap_pages() to forward to free_color_heap_page()? That would then also allow to have a single, central comment and/or assertion that the "order" value here isn't lost. Jan
Hi Jan, On Tue, Mar 19, 2024 at 5:43 PM Jan Beulich <jbeulich@suse.com> wrote: > > On 15.03.2024 11:58, Carlo Nonato wrote: > > Add a new memory page allocator that implements the cache coloring mechanism. > > The allocation algorithm enforces equal frequency distribution of cache > > partitions, following the coloring configuration of a domain. This allows > > for an even utilization of cache sets for every domain. > > > > Pages are stored in a color-indexed array of lists. Those lists are filled > > by a simple init function which computes the color of each page. > > When a domain requests a page, the allocator extract the page from the list > > with the maximum number of free pages between those that the domain can > > access, given its coloring configuration. > > Minor remark: I'm not a native speaker, but "between" here reads odd to > me. I'd have expected perhaps "among". Yes, I'm gonna change it. > > --- a/docs/misc/xen-command-line.pandoc > > +++ b/docs/misc/xen-command-line.pandoc > > @@ -270,6 +270,20 @@ and not running softirqs. Reduce this if softirqs are not being run frequently > > enough. Setting this to a high value may cause boot failure, particularly if > > the NMI watchdog is also enabled. > > > > +### buddy-alloc-size (arm64) > > +> `= <size>` > > + > > +> Default: `64M` > > + > > +Amount of memory reserved for the buddy allocator when colored allocator is > > +active. This options is parsed only when LLC coloring support is enabled. > > Nit: s/parsed/used/ - the option is always parsed as long as LLC_COLORING=y. > > > @@ -1945,6 +1949,164 @@ static unsigned long avail_heap_pages( > > return free_pages; > > } > > > > +/************************* > > + * COLORED SIDE-ALLOCATOR > > + * > > + * Pages are grouped by LLC color in lists which are globally referred to as the > > + * color heap. Lists are populated in end_boot_allocator(). > > + * After initialization there will be N lists where N is the number of > > + * available colors on the platform. > > + */ > > +static struct page_list_head *__ro_after_init _color_heap; > > +#define color_heap(color) (&_color_heap[color]) > > + > > +static unsigned long *__ro_after_init free_colored_pages; > > + > > +/* Memory required for buddy allocator to work with colored one */ > > +#ifdef CONFIG_LLC_COLORING > > +static unsigned long __initdata buddy_alloc_size = > > + MB(CONFIG_BUDDY_ALLOCATOR_SIZE); > > +size_param("buddy-alloc-size", buddy_alloc_size); > > + > > +#define domain_num_llc_colors(d) (d)->num_llc_colors > > +#define domain_llc_color(d, i) (d)->llc_colors[i] > > +#else > > +static unsigned long __initdata buddy_alloc_size; > > + > > +#define domain_num_llc_colors(d) 0 > > +#define domain_llc_color(d, i) 0 > > +#endif > > + > > +static void free_color_heap_page(struct page_info *pg, bool need_scrub) > > +{ > > + unsigned int color = page_to_llc_color(pg); > > + struct page_list_head *head = color_heap(color); > > + > > + spin_lock(&heap_lock); > > + > > + mark_page_free(pg, page_to_mfn(pg)); > > + > > + if ( need_scrub ) > > + { > > + pg->count_info |= PGC_need_scrub; > > + poison_one_page(pg); > > + } > > + > > + free_colored_pages[color]++; > > + page_list_add(pg, head); > > May I please ask for a comment (or at least some wording in the description) > as to the choice made here between head or tail insertion? When assuming > that across a system there's no sharing of colors, preferably re-using > cache-hot pages is certainly good. Whereas when colors can reasonably be > expected to be shared, avoiding to quickly re-use a freed page can also > have benefits. I'll add it. > > +static struct page_info *alloc_color_heap_page(unsigned int memflags, > > + const struct domain *d) > > +{ > > + struct page_info *pg = NULL; > > + unsigned int i, color = 0; > > + unsigned long max = 0; > > + bool need_tlbflush = false; > > + uint32_t tlbflush_timestamp = 0; > > + bool need_scrub; > > + > > + if ( memflags >> _MEMF_bits ) > > + return NULL; > > By mentioning MEMF_bits earlier on I meant to give an example. What > about MEMF_node and in particular MEMF_exact_node? Certain other flags > also aren't obvious as to being okay to silently ignore. You're right. > > + spin_lock(&heap_lock); > > + > > + for ( i = 0; i < domain_num_llc_colors(d); i++ ) > > + { > > + unsigned long free = free_colored_pages[domain_llc_color(d, i)]; > > + > > + if ( free > max ) > > + { > > + color = domain_llc_color(d, i); > > + pg = page_list_first(color_heap(color)); > > + max = free; > > + } > > + } > > + > > + if ( !pg ) > > + { > > + spin_unlock(&heap_lock); > > + return NULL; > > + } > > + > > + need_scrub = pg->count_info & (PGC_need_scrub); > > + pg->count_info = PGC_state_inuse | (pg->count_info & PGC_colored); > > Better PGC_preserved? Yeah. > > +static void __init init_color_heap_pages(struct page_info *pg, > > + unsigned long nr_pages) > > +{ > > + unsigned int i; > > + bool need_scrub = opt_bootscrub == BOOTSCRUB_IDLE; > > + > > + if ( buddy_alloc_size ) > > + { > > + unsigned long buddy_pages = min(PFN_DOWN(buddy_alloc_size), nr_pages); > > + > > + init_heap_pages(pg, buddy_pages); > > There's a corner case where init_heap_pages() would break when passed 0 > as 2nd argument. I don't see it. There's just a for-loop that would be skipped in that case... > I think you want to alter the enclosing if() to > "if ( buddy_alloc_size >= PAGE_SIZE )" to be entirely certain to avoid > that case. ... anyway, ok. > > +static void dump_color_heap(void) > > +{ > > + unsigned int color; > > + > > + printk("Dumping color heap info\n"); > > + for ( color = 0; color < get_max_nr_llc_colors(); color++ ) > > + if ( free_colored_pages[color] > 0 ) > > + printk("Color heap[%u]: %lu pages\n", > > + color, free_colored_pages[color]); > > +} > > While having all of the code above from here outside of any #ifdef is > helpful to prevent unintended breakage when changes are made and tested > only on non-Arm64 targets, I'd still like to ask: Halfway recent > compilers manage to eliminate everything? I'd like to avoid e.g. x86 > being left with traces of coloring despite not being able at all to use > it. I don't know the answer to this, sorry. > > @@ -2485,7 +2660,10 @@ struct page_info *alloc_domheap_pages( > > } > > if ( assign_page(pg, order, d, memflags) ) > > { > > - free_heap_pages(pg, order, memflags & MEMF_no_scrub); > > + if ( pg->count_info & PGC_colored ) > > + free_color_heap_page(pg, memflags & MEMF_no_scrub); > > + else > > + free_heap_pages(pg, order, memflags & MEMF_no_scrub); > > return NULL; > > } > > } > > @@ -2568,7 +2746,10 @@ void free_domheap_pages(struct page_info *pg, unsigned int order) > > scrub = 1; > > } > > > > - free_heap_pages(pg, order, scrub); > > + if ( pg->count_info & PGC_colored ) > > + free_color_heap_page(pg, scrub); > > + else > > + free_heap_pages(pg, order, scrub); > > } > > Instead of this, did you consider altering free_heap_pages() to forward > to free_color_heap_page()? That would then also allow to have a single, > central comment and/or assertion that the "order" value here isn't lost. Yes this can be easily done. > Jan Thanks.
On 21.03.2024 16:36, Carlo Nonato wrote: > On Tue, Mar 19, 2024 at 5:43 PM Jan Beulich <jbeulich@suse.com> wrote: >> On 15.03.2024 11:58, Carlo Nonato wrote: >>> +static void __init init_color_heap_pages(struct page_info *pg, >>> + unsigned long nr_pages) >>> +{ >>> + unsigned int i; >>> + bool need_scrub = opt_bootscrub == BOOTSCRUB_IDLE; >>> + >>> + if ( buddy_alloc_size ) >>> + { >>> + unsigned long buddy_pages = min(PFN_DOWN(buddy_alloc_size), nr_pages); >>> + >>> + init_heap_pages(pg, buddy_pages); >> >> There's a corner case where init_heap_pages() would break when passed 0 >> as 2nd argument. > > I don't see it. There's just a for-loop that would be skipped in that case... Look at the first comment in the function and the if() following it. I don't think that code would work very well when nr_pages == 0. >>> +static void dump_color_heap(void) >>> +{ >>> + unsigned int color; >>> + >>> + printk("Dumping color heap info\n"); >>> + for ( color = 0; color < get_max_nr_llc_colors(); color++ ) >>> + if ( free_colored_pages[color] > 0 ) >>> + printk("Color heap[%u]: %lu pages\n", >>> + color, free_colored_pages[color]); >>> +} >> >> While having all of the code above from here outside of any #ifdef is >> helpful to prevent unintended breakage when changes are made and tested >> only on non-Arm64 targets, I'd still like to ask: Halfway recent >> compilers manage to eliminate everything? I'd like to avoid e.g. x86 >> being left with traces of coloring despite not being able at all to use >> it. > > I don't know the answer to this, sorry. Yet it is important to have. Jan
diff --git a/docs/misc/cache-coloring.rst b/docs/misc/cache-coloring.rst index 028aecda28..50b6d94ffc 100644 --- a/docs/misc/cache-coloring.rst +++ b/docs/misc/cache-coloring.rst @@ -11,6 +11,9 @@ To compile LLC coloring support set ``CONFIG_LLC_COLORING=y``. If needed, change the maximum number of colors with ``CONFIG_NR_LLC_COLORS=<n>``. +If needed, change the buddy allocator reserved size with +``CONFIG_BUDDY_ALLOCATOR_SIZE=<n>``. + Runtime configuration is done via `Command line parameters`_. For DomUs follow `DomUs configuration`_. @@ -117,6 +120,8 @@ Specific documentation is available at `docs/misc/xen-command-line.pandoc`. +----------------------+-------------------------------+ | ``dom0-llc-colors`` | Dom0 color configuration | +----------------------+-------------------------------+ +| ``buddy-alloc-size`` | Buddy allocator reserved size | ++----------------------+-------------------------------+ Colors selection format *********************** @@ -204,6 +209,17 @@ the ``llc-colors`` option. For example: **Note:** If no color configuration is provided for a domain, the default one, which corresponds to all available colors is used instead. +Colored allocator and buddy allocator +************************************* + +The colored allocator distributes pages based on color configurations of +domains so that each domains only gets pages of its own colors. +The colored allocator is meant as an alternative to the buddy allocator because +its allocation policy is by definition incompatible with the generic one. Since +the Xen heap is not colored yet, we need to support the coexistence of the two +allocators and some memory must be left for the buddy one. Buddy memory +reservation is configured via Kconfig or via command-line. + Known issues and limitations **************************** @@ -214,3 +230,24 @@ In the domain configuration, "xen,static-mem" allows memory to be statically allocated to the domain. This isn't possible when LLC coloring is enabled, because that memory can't be guaranteed to use only colors assigned to the domain. + +Cache coloring is intended only for embedded systems +#################################################### + +The current implementation aims to satisfy the need of predictability in +embedded systems with small amount of memory to be managed in a colored way. +Given that, some shortcuts are taken in the development. Expect worse +performances on larger systems. + +Colored allocator can only make use of order-0 pages +#################################################### + +The cache coloring technique relies on memory mappings and on the smallest +mapping granularity to achieve the maximum number of colors (cache partitions) +possible. This granularity is what is normally called a page and, in Xen +terminology, the order-0 page is the smallest one. The fairly simple +colored allocator currently implemented, makes use only of such pages. +It must be said that a more complex one could, in theory, adopt higher order +pages if the colors selection contained adjacent colors. Two subsequent colors, +for example, can be represented by an order-1 page, four colors correspond to +an order-2 page, etc. diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc index 28035a214d..461403362f 100644 --- a/docs/misc/xen-command-line.pandoc +++ b/docs/misc/xen-command-line.pandoc @@ -270,6 +270,20 @@ and not running softirqs. Reduce this if softirqs are not being run frequently enough. Setting this to a high value may cause boot failure, particularly if the NMI watchdog is also enabled. +### buddy-alloc-size (arm64) +> `= <size>` + +> Default: `64M` + +Amount of memory reserved for the buddy allocator when colored allocator is +active. This options is parsed only when LLC coloring support is enabled. +The colored allocator is meant as an alternative to the buddy allocator, +because its allocation policy is by definition incompatible with the generic +one. Since the Xen heap systems is not colored yet, we need to support the +coexistence of the two allocators for now. This parameter, which is optional +and for expert only, it's used to set the amount of memory reserved to the +buddy allocator. + ### cet = List of [ shstk=<bool>, ibt=<bool> ] diff --git a/xen/arch/Kconfig b/xen/arch/Kconfig index a65c38e53e..6819a96f78 100644 --- a/xen/arch/Kconfig +++ b/xen/arch/Kconfig @@ -51,3 +51,11 @@ config NR_LLC_COLORS more than what's needed in the general case. Use only power of 2 values. 1024 is the number of colors that fit in a 4 KiB page when integers are 4 bytes long. + +config BUDDY_ALLOCATOR_SIZE + int "Buddy allocator reserved memory size (MiB)" + default "64" + depends on LLC_COLORING + help + Amount of memory reserved for the buddy allocator to serve Xen heap, + working alongside the colored one. diff --git a/xen/arch/arm/include/asm/mm.h b/xen/arch/arm/include/asm/mm.h index 48538b5337..68b7754bec 100644 --- a/xen/arch/arm/include/asm/mm.h +++ b/xen/arch/arm/include/asm/mm.h @@ -145,6 +145,11 @@ struct page_info #else #define PGC_static 0 #endif +#ifdef CONFIG_LLC_COLORING +/* Page is cache colored */ +#define _PGC_colored PG_shift(4) +#define PGC_colored PG_mask(1, 4) +#endif /* ... */ /* Page is broken? */ #define _PGC_broken PG_shift(7) diff --git a/xen/common/llc-coloring.c b/xen/common/llc-coloring.c index 77d24553e0..e34ba6b6ec 100644 --- a/xen/common/llc-coloring.c +++ b/xen/common/llc-coloring.c @@ -22,6 +22,9 @@ static unsigned int __ro_after_init max_nr_colors; static unsigned int __initdata dom0_colors[CONFIG_NR_LLC_COLORS]; static unsigned int __initdata dom0_num_colors; +#define mfn_color_mask (max_nr_colors - 1) +#define mfn_to_color(mfn) (mfn_x(mfn) & mfn_color_mask) + /* * Parse the coloring configuration given in the buf string, following the * syntax below. @@ -284,6 +287,16 @@ int __init domain_set_llc_colors_from_str(struct domain *d, const char *str) return 0; } +unsigned int page_to_llc_color(const struct page_info *pg) +{ + return mfn_to_color(page_to_mfn(pg)); +} + +unsigned int get_max_nr_llc_colors(void) +{ + return max_nr_colors; +} + /* * Local variables: * mode: C diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index 3adea713b7..8aab18d1fe 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -158,8 +158,12 @@ #define PGC_static 0 #endif -#define PGC_preserved (PGC_extra | PGC_static) -#define PGC_no_buddy_merge PGC_static +#ifndef PGC_colored +#define PGC_colored 0 +#endif + +#define PGC_preserved (PGC_extra | PGC_static | PGC_colored) +#define PGC_no_buddy_merge (PGC_static | PGC_colored) #ifndef PGT_TYPE_INFO_INITIALIZER #define PGT_TYPE_INFO_INITIALIZER 0 @@ -1945,6 +1949,164 @@ static unsigned long avail_heap_pages( return free_pages; } +/************************* + * COLORED SIDE-ALLOCATOR + * + * Pages are grouped by LLC color in lists which are globally referred to as the + * color heap. Lists are populated in end_boot_allocator(). + * After initialization there will be N lists where N is the number of + * available colors on the platform. + */ +static struct page_list_head *__ro_after_init _color_heap; +#define color_heap(color) (&_color_heap[color]) + +static unsigned long *__ro_after_init free_colored_pages; + +/* Memory required for buddy allocator to work with colored one */ +#ifdef CONFIG_LLC_COLORING +static unsigned long __initdata buddy_alloc_size = + MB(CONFIG_BUDDY_ALLOCATOR_SIZE); +size_param("buddy-alloc-size", buddy_alloc_size); + +#define domain_num_llc_colors(d) (d)->num_llc_colors +#define domain_llc_color(d, i) (d)->llc_colors[i] +#else +static unsigned long __initdata buddy_alloc_size; + +#define domain_num_llc_colors(d) 0 +#define domain_llc_color(d, i) 0 +#endif + +static void free_color_heap_page(struct page_info *pg, bool need_scrub) +{ + unsigned int color = page_to_llc_color(pg); + struct page_list_head *head = color_heap(color); + + spin_lock(&heap_lock); + + mark_page_free(pg, page_to_mfn(pg)); + + if ( need_scrub ) + { + pg->count_info |= PGC_need_scrub; + poison_one_page(pg); + } + + free_colored_pages[color]++; + page_list_add(pg, head); + + spin_unlock(&heap_lock); +} + +static struct page_info *alloc_color_heap_page(unsigned int memflags, + const struct domain *d) +{ + struct page_info *pg = NULL; + unsigned int i, color = 0; + unsigned long max = 0; + bool need_tlbflush = false; + uint32_t tlbflush_timestamp = 0; + bool need_scrub; + + if ( memflags >> _MEMF_bits ) + return NULL; + + spin_lock(&heap_lock); + + for ( i = 0; i < domain_num_llc_colors(d); i++ ) + { + unsigned long free = free_colored_pages[domain_llc_color(d, i)]; + + if ( free > max ) + { + color = domain_llc_color(d, i); + pg = page_list_first(color_heap(color)); + max = free; + } + } + + if ( !pg ) + { + spin_unlock(&heap_lock); + return NULL; + } + + need_scrub = pg->count_info & (PGC_need_scrub); + pg->count_info = PGC_state_inuse | (pg->count_info & PGC_colored); + free_colored_pages[color]--; + page_list_del(pg, color_heap(color)); + + if ( !(memflags & MEMF_no_tlbflush) ) + accumulate_tlbflush(&need_tlbflush, pg, &tlbflush_timestamp); + + init_free_page_fields(pg); + + spin_unlock(&heap_lock); + + if ( !(memflags & MEMF_no_scrub) ) + { + if ( need_scrub ) + scrub_one_page(pg); + else + check_one_page(pg); + } + + if ( need_tlbflush ) + filtered_flush_tlb_mask(tlbflush_timestamp); + + flush_page_to_ram(mfn_x(page_to_mfn(pg)), + !(memflags & MEMF_no_icache_flush)); + + return pg; +} + +static void __init init_color_heap_pages(struct page_info *pg, + unsigned long nr_pages) +{ + unsigned int i; + bool need_scrub = opt_bootscrub == BOOTSCRUB_IDLE; + + if ( buddy_alloc_size ) + { + unsigned long buddy_pages = min(PFN_DOWN(buddy_alloc_size), nr_pages); + + init_heap_pages(pg, buddy_pages); + nr_pages -= buddy_pages; + buddy_alloc_size -= buddy_pages << PAGE_SHIFT; + pg += buddy_pages; + } + + if ( !_color_heap ) + { + unsigned int max_nr_colors = get_max_nr_llc_colors(); + + _color_heap = xmalloc_array(struct page_list_head, max_nr_colors); + free_colored_pages = xzalloc_array(unsigned long, max_nr_colors); + if ( !_color_heap || !free_colored_pages ) + panic("Can't allocate colored heap. Buddy reserved size is too low"); + + for ( i = 0; i < max_nr_colors; i++ ) + INIT_PAGE_LIST_HEAD(color_heap(i)); + } + + for ( i = 0; i < nr_pages; i++ ) + { + pg[i].count_info = PGC_colored; + free_color_heap_page(&pg[i], need_scrub); + } +} + +static void dump_color_heap(void) +{ + unsigned int color; + + printk("Dumping color heap info\n"); + for ( color = 0; color < get_max_nr_llc_colors(); color++ ) + if ( free_colored_pages[color] > 0 ) + printk("Color heap[%u]: %lu pages\n", + color, free_colored_pages[color]); +} + void __init end_boot_allocator(void) { unsigned int i; @@ -1964,7 +2126,13 @@ void __init end_boot_allocator(void) for ( i = nr_bootmem_regions; i-- > 0; ) { struct bootmem_region *r = &bootmem_region_list[i]; - if ( r->s < r->e ) + + if ( r->s >= r->e ) + continue; + + if ( llc_coloring_enabled ) + init_color_heap_pages(mfn_to_page(_mfn(r->s)), r->e - r->s); + else init_heap_pages(mfn_to_page(_mfn(r->s)), r->e - r->s); } nr_bootmem_regions = 0; @@ -2460,7 +2628,14 @@ struct page_info *alloc_domheap_pages( if ( memflags & MEMF_no_owner ) memflags |= MEMF_no_refcount; - if ( !dma_bitsize ) + /* Only domains are supported for coloring */ + if ( d && llc_coloring_enabled ) + { + /* Colored allocation must be done on 0 order */ + if ( order || (pg = alloc_color_heap_page(memflags, d)) == NULL ) + return NULL; + } + else if ( !dma_bitsize ) memflags &= ~MEMF_no_dma; else if ( (dma_zone = bits_to_zone(dma_bitsize)) < zone_hi ) pg = alloc_heap_pages(dma_zone + 1, zone_hi, order, memflags, d); @@ -2485,7 +2660,10 @@ struct page_info *alloc_domheap_pages( } if ( assign_page(pg, order, d, memflags) ) { - free_heap_pages(pg, order, memflags & MEMF_no_scrub); + if ( pg->count_info & PGC_colored ) + free_color_heap_page(pg, memflags & MEMF_no_scrub); + else + free_heap_pages(pg, order, memflags & MEMF_no_scrub); return NULL; } } @@ -2568,7 +2746,10 @@ void free_domheap_pages(struct page_info *pg, unsigned int order) scrub = 1; } - free_heap_pages(pg, order, scrub); + if ( pg->count_info & PGC_colored ) + free_color_heap_page(pg, scrub); + else + free_heap_pages(pg, order, scrub); } if ( drop_dom_ref ) @@ -2677,6 +2858,9 @@ static void cf_check dump_heap(unsigned char key) continue; printk("Node %d has %lu unscrubbed pages\n", i, node_need_scrub[i]); } + + if ( llc_coloring_enabled ) + dump_color_heap(); } static __init int cf_check register_heap_trigger(void) diff --git a/xen/include/xen/llc-coloring.h b/xen/include/xen/llc-coloring.h index 49ebd1e712..7f8218bfb2 100644 --- a/xen/include/xen/llc-coloring.h +++ b/xen/include/xen/llc-coloring.h @@ -33,6 +33,10 @@ int domain_set_llc_colors(struct domain *d, const struct xen_domctl_set_llc_colors *config); int domain_set_llc_colors_from_str(struct domain *d, const char *str); +struct page_info; +unsigned int page_to_llc_color(const struct page_info *pg); +unsigned int get_max_nr_llc_colors(void); + #endif /* __COLORING_H__ */ /*