| Message ID | 20200413125303.423864-5-npiggin@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | huge vmalloc mappings | expand |
On Mon, Apr 13, 2020 at 10:53:03PM +1000, Nicholas Piggin wrote: > +static int vmap_pages_range_noflush(unsigned long start, unsigned long end, > + pgprot_t prot, struct page **pages, > + unsigned int page_shift) > +{ > + if (page_shift == PAGE_SIZE) { ... I think you meant 'page_shift == PAGE_SHIFT' Overall I like this series, although it's a bit biased towards CPUs which have page sizes which match PMD/PUD sizes. It doesn't offer the possibility of using 64kB page sizes on ARM, for example. But it's a step in the right direction.
On Mon, Apr 13, 2020 at 10:53:03PM +1000, Nicholas Piggin wrote: > For platforms that define HAVE_ARCH_HUGE_VMAP and support PMD vmap mappings, > have vmalloc attempt to allocate PMD-sized pages first, before falling back > to small pages. Allocations which use something other than PAGE_KERNEL > protections are not permitted to use huge pages yet, not all callers expect > this (e.g., module allocations vs strict module rwx). > > This gives a 6x reduction in dTLB misses for a `git diff` (of linux), from > 45600 to 6500 and a 2.2% reduction in cycles on a 2-node POWER9. > > This can result in more internal fragmentation and memory overhead for a > given allocation. It can also cause greater NUMA unbalance on hashdist > allocations. > > There may be other callers that expect small pages under vmalloc but use > PAGE_KERNEL, I'm not sure if it's feasible to catch them all. An > alternative would be a new function or flag which enables large mappings, > and use that in callers. Why do we even use vmalloc in this case rather than just doing a huge page allocation? What callers are you intersted in?
Excerpts from Matthew Wilcox's message of April 13, 2020 11:41 pm: > On Mon, Apr 13, 2020 at 10:53:03PM +1000, Nicholas Piggin wrote: >> +static int vmap_pages_range_noflush(unsigned long start, unsigned long end, >> + pgprot_t prot, struct page **pages, >> + unsigned int page_shift) >> +{ >> + if (page_shift == PAGE_SIZE) { > > ... I think you meant 'page_shift == PAGE_SHIFT' Thanks, good catch. I obviously didn't test the fallback path (the other path works for small pages, it just goes one at a time). > Overall I like this series, although it's a bit biased towards CPUs > which have page sizes which match PMD/PUD sizes. It doesn't offer the > possibility of using 64kB page sizes on ARM, for example. No, it's just an incremental step on existing huge vmap stuff in tree, so such a thing would be out of scope. > But it's a > step in the right direction. > I don't know about moving kernel maps away from a generic Linux page table format. I quite like moving to it and making it as generic as possible. On the other hand, I also would like to make some arch-specific allowances for certain special cases that may not fit within the standard page table format, but it might be a much more specific and limited interface than the general vmalloc stuff. Thanks, Nick
Excerpts from Christoph Hellwig's message of April 14, 2020 5:23 pm: > On Mon, Apr 13, 2020 at 10:53:03PM +1000, Nicholas Piggin wrote: >> For platforms that define HAVE_ARCH_HUGE_VMAP and support PMD vmap mappings, >> have vmalloc attempt to allocate PMD-sized pages first, before falling back >> to small pages. Allocations which use something other than PAGE_KERNEL >> protections are not permitted to use huge pages yet, not all callers expect >> this (e.g., module allocations vs strict module rwx). >> >> This gives a 6x reduction in dTLB misses for a `git diff` (of linux), from >> 45600 to 6500 and a 2.2% reduction in cycles on a 2-node POWER9. >> >> This can result in more internal fragmentation and memory overhead for a >> given allocation. It can also cause greater NUMA unbalance on hashdist >> allocations. >> >> There may be other callers that expect small pages under vmalloc but use >> PAGE_KERNEL, I'm not sure if it's feasible to catch them all. An >> alternative would be a new function or flag which enables large mappings, >> and use that in callers. > > Why do we even use vmalloc in this case rather than just doing a huge > page allocation? Which case? Usually the answer would be because you don't want to use contiguous physical memory and/or you don't want to use the linear mapping. > What callers are you intersted in? The dentry and inode caches for this test, obviously. Lots of other things could possibly benefit though, other system hashes like networking, but lot of other vmalloc callers that might benefit right away, some others could use some work to batch up allocation sizes to benefit. Thanks, Nick
Le 13/04/2020 à 15:41, Matthew Wilcox a écrit : > On Mon, Apr 13, 2020 at 10:53:03PM +1000, Nicholas Piggin wrote: >> +static int vmap_pages_range_noflush(unsigned long start, unsigned long end, >> + pgprot_t prot, struct page **pages, >> + unsigned int page_shift) >> +{ >> + if (page_shift == PAGE_SIZE) { > > ... I think you meant 'page_shift == PAGE_SHIFT' > > Overall I like this series, although it's a bit biased towards CPUs > which have page sizes which match PMD/PUD sizes. It doesn't offer the > possibility of using 64kB page sizes on ARM, for example. But it's a > step in the right direction. > I was going to ask more or less the same question, I would have liked to use 512kB hugepages on powerpc 8xx. Even the 8M hugepages (still on the 8xx), can they be used as well, taking into account that two PGD entries have to point to the same 8M page ? I sent out a series which tends to make the management of 512k and 8M pages closer to what Linux expects, in order to use them inside kernel, for Linear mappings and Kasan mappings for the moment. See https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=164620 It would be nice if we could amplify it a use it for ioremaps and vmallocs as well. Christophe
On Tue, Apr 14, 2020 at 10:13:44PM +1000, Nicholas Piggin wrote: > Which case? Usually the answer would be because you don't want to use > contiguous physical memory and/or you don't want to use the linear > mapping. But with huge pages you do by definition already use large contiguous areas. So you want allocations larger than "small" huge pages but not using gigantic pages using vmalloc?
On Tue, Apr 14, 2020 at 02:28:35PM +0200, Christophe Leroy wrote: > Le 13/04/2020 à 15:41, Matthew Wilcox a écrit : > > On Mon, Apr 13, 2020 at 10:53:03PM +1000, Nicholas Piggin wrote: > > > +static int vmap_pages_range_noflush(unsigned long start, unsigned long end, > > > + pgprot_t prot, struct page **pages, > > > + unsigned int page_shift) > > > +{ > > > + if (page_shift == PAGE_SIZE) { > > > > ... I think you meant 'page_shift == PAGE_SHIFT' > > > > Overall I like this series, although it's a bit biased towards CPUs > > which have page sizes which match PMD/PUD sizes. It doesn't offer the > > possibility of using 64kB page sizes on ARM, for example. But it's a > > step in the right direction. > > I was going to ask more or less the same question, I would have liked to use > 512kB hugepages on powerpc 8xx. > > Even the 8M hugepages (still on the 8xx), can they be used as well, taking > into account that two PGD entries have to point to the same 8M page ? > > I sent out a series which tends to make the management of 512k and 8M pages > closer to what Linux expects, in order to use them inside kernel, for Linear > mappings and Kasan mappings for the moment. See > https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=164620 > It would be nice if we could amplify it a use it for ioremaps and vmallocs > as well. I haven't been looking at vmalloc at all; I've been looking at the page cache. See: https://lore.kernel.org/linux-mm/20200212041845.25879-1-willy@infradead.org/ Once we have large pages in the page cache, I want to sort out the API for asking the CPU to insert a TLB entry. Right now, we use set_pte_at(), set_pmd_at() and set_pud_at(). I'm thinking something along the lines of: vm_fault_t vmf_set_page_at(struct vm_fault *vmf, struct page *page); and the architecture can insert whatever PTEs and/or TLB entries it likes based on compound_order(page) -- if, say, it's a 1MB page, it might choose to insert 2 * 512kB entries, or just the upper or lower 512kB entry (depending which half of the 1MB page the address sits in).
Excerpts from Christoph Hellwig's message of April 14, 2020 11:02 pm: > On Tue, Apr 14, 2020 at 10:13:44PM +1000, Nicholas Piggin wrote: >> Which case? Usually the answer would be because you don't want to use >> contiguous physical memory and/or you don't want to use the linear >> mapping. > > But with huge pages you do by definition already use large contiguous > areas. So you want allocations larger than "small" huge pages but not > using gigantic pages using vmalloc? Yes. Thanks, Nick
Hi Nick, On Mon, Apr 13, 2020 at 10:53:03PM +1000, Nicholas Piggin wrote: > For platforms that define HAVE_ARCH_HUGE_VMAP and support PMD vmap mappings, > have vmalloc attempt to allocate PMD-sized pages first, before falling back > to small pages. Allocations which use something other than PAGE_KERNEL > protections are not permitted to use huge pages yet, not all callers expect > this (e.g., module allocations vs strict module rwx). > > This gives a 6x reduction in dTLB misses for a `git diff` (of linux), from > 45600 to 6500 and a 2.2% reduction in cycles on a 2-node POWER9. I wonder if it's worth extending vmap() to handle higher order pages in a similar way? That might be helpful for tracing PMUs such as Arm SPE, where the CPU streams tracing data out to a virtually addressed buffer (see rb_alloc_aux_page()). > This can result in more internal fragmentation and memory overhead for a > given allocation. It can also cause greater NUMA unbalance on hashdist > allocations. > > There may be other callers that expect small pages under vmalloc but use > PAGE_KERNEL, I'm not sure if it's feasible to catch them all. An > alternative would be a new function or flag which enables large mappings, > and use that in callers. > > Signed-off-by: Nicholas Piggin <npiggin@gmail.com> > --- > include/linux/vmalloc.h | 2 + > mm/vmalloc.c | 135 +++++++++++++++++++++++++++++----------- > 2 files changed, 102 insertions(+), 35 deletions(-) > > diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h > index 291313a7e663..853b82eac192 100644 > --- a/include/linux/vmalloc.h > +++ b/include/linux/vmalloc.h > @@ -24,6 +24,7 @@ struct notifier_block; /* in notifier.h */ > #define VM_UNINITIALIZED 0x00000020 /* vm_struct is not fully initialized */ > #define VM_NO_GUARD 0x00000040 /* don't add guard page */ > #define VM_KASAN 0x00000080 /* has allocated kasan shadow memory */ > +#define VM_HUGE_PAGES 0x00000100 /* may use huge pages */ Please can you add a check for this in the arm64 change_memory_common() code? Other architectures might need something similar, but we need to forbid changing memory attributes for portions of the huge page. In general, I'm a bit wary of software table walkers tripping over this. For example, I don't think apply_to_existing_page_range() can handle huge mappings at all, but the one user (KASAN) only ever uses page mappings so it's ok there. > @@ -2325,9 +2356,11 @@ static struct vm_struct *__get_vm_area_node(unsigned long size, > if (unlikely(!size)) > return NULL; > > - if (flags & VM_IOREMAP) > - align = 1ul << clamp_t(int, get_count_order_long(size), > - PAGE_SHIFT, IOREMAP_MAX_ORDER); > + if (flags & VM_IOREMAP) { > + align = max(align, > + 1ul << clamp_t(int, get_count_order_long(size), > + PAGE_SHIFT, IOREMAP_MAX_ORDER)); > + } I don't follow this part. Please could you explain why you're potentially aligning above IOREMAP_MAX_ORDER? It doesn't seem to follow from the rest of the patch. Cheers, Will
Excerpts from Will Deacon's message of April 15, 2020 8:47 pm: > Hi Nick, > > On Mon, Apr 13, 2020 at 10:53:03PM +1000, Nicholas Piggin wrote: >> For platforms that define HAVE_ARCH_HUGE_VMAP and support PMD vmap mappings, >> have vmalloc attempt to allocate PMD-sized pages first, before falling back >> to small pages. Allocations which use something other than PAGE_KERNEL >> protections are not permitted to use huge pages yet, not all callers expect >> this (e.g., module allocations vs strict module rwx). >> >> This gives a 6x reduction in dTLB misses for a `git diff` (of linux), from >> 45600 to 6500 and a 2.2% reduction in cycles on a 2-node POWER9. > > I wonder if it's worth extending vmap() to handle higher order pages in > a similar way? That might be helpful for tracing PMUs such as Arm SPE, > where the CPU streams tracing data out to a virtually addressed buffer > (see rb_alloc_aux_page()). Yeah it becomes pretty trivial to do that with VM_HUGE_PAGES after this patch, I have something to do it but no callers ready yet, if you have an easy one we can add it. >> This can result in more internal fragmentation and memory overhead for a >> given allocation. It can also cause greater NUMA unbalance on hashdist >> allocations. >> >> There may be other callers that expect small pages under vmalloc but use >> PAGE_KERNEL, I'm not sure if it's feasible to catch them all. An >> alternative would be a new function or flag which enables large mappings, >> and use that in callers. >> >> Signed-off-by: Nicholas Piggin <npiggin@gmail.com> >> --- >> include/linux/vmalloc.h | 2 + >> mm/vmalloc.c | 135 +++++++++++++++++++++++++++++----------- >> 2 files changed, 102 insertions(+), 35 deletions(-) >> >> diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h >> index 291313a7e663..853b82eac192 100644 >> --- a/include/linux/vmalloc.h >> +++ b/include/linux/vmalloc.h >> @@ -24,6 +24,7 @@ struct notifier_block; /* in notifier.h */ >> #define VM_UNINITIALIZED 0x00000020 /* vm_struct is not fully initialized */ >> #define VM_NO_GUARD 0x00000040 /* don't add guard page */ >> #define VM_KASAN 0x00000080 /* has allocated kasan shadow memory */ >> +#define VM_HUGE_PAGES 0x00000100 /* may use huge pages */ > > Please can you add a check for this in the arm64 change_memory_common() > code? Other architectures might need something similar, but we need to > forbid changing memory attributes for portions of the huge page. Yeah good idea, I can look about adding some more checks. > > In general, I'm a bit wary of software table walkers tripping over this. > For example, I don't think apply_to_existing_page_range() can handle > huge mappings at all, but the one user (KASAN) only ever uses page mappings > so it's ok there. Right, I have something to warn for apply to page range (and looking at adding support for bigger pages). It doesn't even have a test and warn at the moment which isn't good practice IMO so we should add one even without huge vmap. > >> @@ -2325,9 +2356,11 @@ static struct vm_struct *__get_vm_area_node(unsigned long size, >> if (unlikely(!size)) >> return NULL; >> >> - if (flags & VM_IOREMAP) >> - align = 1ul << clamp_t(int, get_count_order_long(size), >> - PAGE_SHIFT, IOREMAP_MAX_ORDER); >> + if (flags & VM_IOREMAP) { >> + align = max(align, >> + 1ul << clamp_t(int, get_count_order_long(size), >> + PAGE_SHIFT, IOREMAP_MAX_ORDER)); >> + } > > > I don't follow this part. Please could you explain why you're potentially > aligning above IOREMAP_MAX_ORDER? It doesn't seem to follow from the rest > of the patch. Trying to remember. If the caller asks for a particular alignment we shouldn't reduce it. Should put it in another patch. Thanks, Nick
> static void *__vmalloc_node(unsigned long size, unsigned long align, > - gfp_t gfp_mask, pgprot_t prot, > - int node, const void *caller); > + gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, > + int node, const void *caller); > static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, > - pgprot_t prot, int node) > + pgprot_t prot, unsigned int page_shift, > + int node) > { > struct page **pages; > + unsigned long addr = (unsigned long)area->addr; > + unsigned long size = get_vm_area_size(area); > + unsigned int page_order = page_shift - PAGE_SHIFT; > unsigned int nr_pages, array_size, i; > const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; > const gfp_t alloc_mask = gfp_mask | __GFP_NOWARN; > const gfp_t highmem_mask = (gfp_mask & (GFP_DMA | GFP_DMA32)) ? > - 0 : > - __GFP_HIGHMEM; > + 0 : __GFP_HIGHMEM; > > - nr_pages = get_vm_area_size(area) >> PAGE_SHIFT; > + nr_pages = size >> page_shift; while try out this patchset, we encountered a BUG_ON in account_kernel_stack() in kernel/fork.c. BUG_ON(vm->nr_pages != THREAD_SIZE / PAGE_SIZE); which obviously should be updated accordingly. > array_size = (nr_pages * sizeof(struct page *)); > > /* Please note that the recursion is strictly bounded. */ > if (array_size > PAGE_SIZE) { > pages = __vmalloc_node(array_size, 1, nested_gfp|highmem_mask, > - PAGE_KERNEL, node, area->caller); > + PAGE_KERNEL, 0, node, area->caller); > } else { > pages = kmalloc_node(array_size, nested_gfp, node); > }
Excerpts from Zefan Li's message of July 1, 2020 5:10 pm: >> static void *__vmalloc_node(unsigned long size, unsigned long align, >> - gfp_t gfp_mask, pgprot_t prot, >> - int node, const void *caller); >> + gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, >> + int node, const void *caller); >> static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, >> - pgprot_t prot, int node) >> + pgprot_t prot, unsigned int page_shift, >> + int node) >> { >> struct page **pages; >> + unsigned long addr = (unsigned long)area->addr; >> + unsigned long size = get_vm_area_size(area); >> + unsigned int page_order = page_shift - PAGE_SHIFT; >> unsigned int nr_pages, array_size, i; >> const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; >> const gfp_t alloc_mask = gfp_mask | __GFP_NOWARN; >> const gfp_t highmem_mask = (gfp_mask & (GFP_DMA | GFP_DMA32)) ? >> - 0 : >> - __GFP_HIGHMEM; >> + 0 : __GFP_HIGHMEM; >> >> - nr_pages = get_vm_area_size(area) >> PAGE_SHIFT; >> + nr_pages = size >> page_shift; > > while try out this patchset, we encountered a BUG_ON in account_kernel_stack() > in kernel/fork.c. > > BUG_ON(vm->nr_pages != THREAD_SIZE / PAGE_SIZE); > > which obviously should be updated accordingly. Thanks for finding that. We may have to change this around a bit so nr_pages still appears to be in PAGE_SIZE units for anybody looking. Thanks, Nick
> +static int vmap_pages_range_noflush(unsigned long start, unsigned long end, > + pgprot_t prot, struct page **pages, > + unsigned int page_shift) > +{ > + if (page_shift == PAGE_SIZE) { Is this a typo of PAGE_SHIFT? > + return vmap_small_pages_range_noflush(start, end, prot, pages); > + } else { > + unsigned long addr = start; > + unsigned int i, nr = (end - start) >> page_shift; > + > + for (i = 0; i < nr; i++) { > + int err; > + > + err = vmap_range_noflush(addr, > + addr + (1UL << page_shift), > + __pa(page_address(pages[i])), prot, > + page_shift); > + if (err) > + return err; > + > + addr += 1UL << page_shift; > + } > + > + return 0; > + } > +} > +
Excerpts from Zefan Li's message of July 20, 2020 12:02 pm: >> +static int vmap_pages_range_noflush(unsigned long start, unsigned long end, >> + pgprot_t prot, struct page **pages, >> + unsigned int page_shift) >> +{ >> + if (page_shift == PAGE_SIZE) { > > Is this a typo of PAGE_SHIFT? Oh good catch, yeah that'll always be going via the one-at-a-time route and slow down the small page vmaps. Will fix. Thanks, Nick > >> + return vmap_small_pages_range_noflush(start, end, prot, pages); >> + } else { >> + unsigned long addr = start; >> + unsigned int i, nr = (end - start) >> page_shift; >> + >> + for (i = 0; i < nr; i++) { >> + int err; >> + >> + err = vmap_range_noflush(addr, >> + addr + (1UL << page_shift), >> + __pa(page_address(pages[i])), prot, >> + page_shift); >> + if (err) >> + return err; >> + >> + addr += 1UL << page_shift; >> + } >> + >> + return 0; >> + } >> +} >> + >
diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 291313a7e663..853b82eac192 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -24,6 +24,7 @@ struct notifier_block; /* in notifier.h */ #define VM_UNINITIALIZED 0x00000020 /* vm_struct is not fully initialized */ #define VM_NO_GUARD 0x00000040 /* don't add guard page */ #define VM_KASAN 0x00000080 /* has allocated kasan shadow memory */ +#define VM_HUGE_PAGES 0x00000100 /* may use huge pages */ /* * VM_KASAN is used slighly differently depending on CONFIG_KASAN_VMALLOC. @@ -58,6 +59,7 @@ struct vm_struct { unsigned long size; unsigned long flags; struct page **pages; + unsigned int page_order; unsigned int nr_pages; phys_addr_t phys_addr; const void *caller; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index c898d16ddd25..7b7e992c5ff1 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -436,7 +436,7 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, * * Ie. pte at addr+N*PAGE_SIZE shall point to pfn corresponding to pages[N] */ -static int vmap_pages_range_noflush(unsigned long start, unsigned long end, +static int vmap_small_pages_range_noflush(unsigned long start, unsigned long end, pgprot_t prot, struct page **pages) { pgd_t *pgd; @@ -457,13 +457,44 @@ static int vmap_pages_range_noflush(unsigned long start, unsigned long end, return nr; } +static int vmap_pages_range_noflush(unsigned long start, unsigned long end, + pgprot_t prot, struct page **pages, + unsigned int page_shift) +{ + if (page_shift == PAGE_SIZE) { + return vmap_small_pages_range_noflush(start, end, prot, pages); + } else { + unsigned long addr = start; + unsigned int i, nr = (end - start) >> page_shift; + + for (i = 0; i < nr; i++) { + int err; + + err = vmap_range_noflush(addr, + addr + (1UL << page_shift), + __pa(page_address(pages[i])), prot, + page_shift); + if (err) + return err; + + addr += 1UL << page_shift; + } + + return 0; + } +} + static int vmap_pages_range(unsigned long start, unsigned long end, - pgprot_t prot, struct page **pages) + pgprot_t prot, struct page **pages, + unsigned int page_shift) { int ret; - ret = vmap_pages_range_noflush(start, end, prot, pages); + BUG_ON(page_shift < PAGE_SHIFT); + + ret = vmap_pages_range_noflush(start, end, prot, pages, page_shift); flush_cache_vmap(start, end); + return ret; } @@ -2064,7 +2095,7 @@ void *vm_map_ram(struct page **pages, unsigned int count, int node, pgprot_t pro kasan_unpoison_vmalloc(mem, size); - if (vmap_pages_range(addr, addr + size, prot, pages) < 0) { + if (vmap_pages_range(addr, addr + size, prot, pages, PAGE_SHIFT) < 0) { vm_unmap_ram(mem, count); return NULL; } @@ -2230,7 +2261,7 @@ void __init vmalloc_init(void) int map_kernel_range_noflush(unsigned long addr, unsigned long size, pgprot_t prot, struct page **pages) { - return vmap_pages_range_noflush(addr, addr + size, prot, pages); + return vmap_pages_range_noflush(addr, addr + size, prot, pages, PAGE_SHIFT); } /** @@ -2277,7 +2308,7 @@ int map_vm_area(struct vm_struct *area, pgprot_t prot, struct page **pages) unsigned long end = addr + get_vm_area_size(area); int err; - err = vmap_pages_range(addr, end, prot, pages); + err = vmap_pages_range(addr, end, prot, pages, PAGE_SHIFT); return err > 0 ? 0 : err; } @@ -2325,9 +2356,11 @@ static struct vm_struct *__get_vm_area_node(unsigned long size, if (unlikely(!size)) return NULL; - if (flags & VM_IOREMAP) - align = 1ul << clamp_t(int, get_count_order_long(size), - PAGE_SHIFT, IOREMAP_MAX_ORDER); + if (flags & VM_IOREMAP) { + align = max(align, + 1ul << clamp_t(int, get_count_order_long(size), + PAGE_SHIFT, IOREMAP_MAX_ORDER)); + } area = kzalloc_node(sizeof(*area), gfp_mask & GFP_RECLAIM_MASK, node); if (unlikely(!area)) @@ -2534,7 +2567,7 @@ static void __vunmap(const void *addr, int deallocate_pages) struct page *page = area->pages[i]; BUG_ON(!page); - __free_pages(page, 0); + __free_pages(page, area->page_order); } atomic_long_sub(area->nr_pages, &nr_vmalloc_pages); @@ -2672,26 +2705,29 @@ void *vmap(struct page **pages, unsigned int count, EXPORT_SYMBOL(vmap); static void *__vmalloc_node(unsigned long size, unsigned long align, - gfp_t gfp_mask, pgprot_t prot, - int node, const void *caller); + gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, + int node, const void *caller); static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, - pgprot_t prot, int node) + pgprot_t prot, unsigned int page_shift, + int node) { struct page **pages; + unsigned long addr = (unsigned long)area->addr; + unsigned long size = get_vm_area_size(area); + unsigned int page_order = page_shift - PAGE_SHIFT; unsigned int nr_pages, array_size, i; const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; const gfp_t alloc_mask = gfp_mask | __GFP_NOWARN; const gfp_t highmem_mask = (gfp_mask & (GFP_DMA | GFP_DMA32)) ? - 0 : - __GFP_HIGHMEM; + 0 : __GFP_HIGHMEM; - nr_pages = get_vm_area_size(area) >> PAGE_SHIFT; + nr_pages = size >> page_shift; array_size = (nr_pages * sizeof(struct page *)); /* Please note that the recursion is strictly bounded. */ if (array_size > PAGE_SIZE) { pages = __vmalloc_node(array_size, 1, nested_gfp|highmem_mask, - PAGE_KERNEL, node, area->caller); + PAGE_KERNEL, 0, node, area->caller); } else { pages = kmalloc_node(array_size, nested_gfp, node); } @@ -2704,14 +2740,13 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, area->pages = pages; area->nr_pages = nr_pages; + area->page_order = page_order; for (i = 0; i < area->nr_pages; i++) { struct page *page; - if (node == NUMA_NO_NODE) - page = alloc_page(alloc_mask|highmem_mask); - else - page = alloc_pages_node(node, alloc_mask|highmem_mask, 0); + page = alloc_pages_node(node, + alloc_mask|highmem_mask, page_order); if (unlikely(!page)) { /* Successfully allocated i pages, free them in __vunmap() */ @@ -2725,8 +2760,9 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, } atomic_long_add(area->nr_pages, &nr_vmalloc_pages); - if (map_vm_area(area, prot, pages)) + if (vmap_pages_range(addr, addr + size, prot, pages, page_shift) < 0) goto fail; + return area->addr; fail: @@ -2760,22 +2796,39 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) { - struct vm_struct *area; + struct vm_struct *area = NULL; void *addr; unsigned long real_size = size; + unsigned long real_align = align; + unsigned int shift = PAGE_SHIFT; size = PAGE_ALIGN(size); if (!size || (size >> PAGE_SHIFT) > totalram_pages()) goto fail; - area = __get_vm_area_node(real_size, align, VM_ALLOC | VM_UNINITIALIZED | + if (IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMAP) && + (vm_flags & VM_HUGE_PAGES)) { + unsigned long size_per_node; + + size_per_node = size; + if (node == NUMA_NO_NODE) + size_per_node /= num_online_nodes(); + if (size_per_node >= PMD_SIZE) + shift = PMD_SHIFT; + } + +again: + align = max(real_align, 1UL << shift); + size = ALIGN(real_size, align); + + area = __get_vm_area_node(size, align, VM_ALLOC | VM_UNINITIALIZED | vm_flags, start, end, node, gfp_mask, caller); if (!area) goto fail; - addr = __vmalloc_area_node(area, gfp_mask, prot, node); + addr = __vmalloc_area_node(area, gfp_mask, prot, shift, node); if (!addr) - return NULL; + goto fail; /* * In this function, newly allocated vm_struct has VM_UNINITIALIZED @@ -2789,8 +2842,16 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, return addr; fail: - warn_alloc(gfp_mask, NULL, + if (shift > PAGE_SHIFT) { + shift = PAGE_SHIFT; + goto again; + } + + if (!area) { + /* Warn for area allocation, page allocations already warn */ + warn_alloc(gfp_mask, NULL, "vmalloc: allocation failure: %lu bytes", real_size); + } return NULL; } @@ -2825,16 +2886,19 @@ EXPORT_SYMBOL_GPL(__vmalloc_node_range); * Return: pointer to the allocated memory or %NULL on error */ static void *__vmalloc_node(unsigned long size, unsigned long align, - gfp_t gfp_mask, pgprot_t prot, - int node, const void *caller) + gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, + int node, const void *caller) { return __vmalloc_node_range(size, align, VMALLOC_START, VMALLOC_END, - gfp_mask, prot, 0, node, caller); + gfp_mask, prot, vm_flags, node, caller); } void *__vmalloc(unsigned long size, gfp_t gfp_mask, pgprot_t prot) { - return __vmalloc_node(size, 1, gfp_mask, prot, NUMA_NO_NODE, + unsigned long vm_flags = 0; + if (pgprot_val(prot) == pgprot_val(PAGE_KERNEL)) + vm_flags |= VM_HUGE_PAGES; + return __vmalloc_node(size, 1, gfp_mask, prot, vm_flags, NUMA_NO_NODE, __builtin_return_address(0)); } EXPORT_SYMBOL(__vmalloc); @@ -2842,7 +2906,7 @@ EXPORT_SYMBOL(__vmalloc); static inline void *__vmalloc_node_flags(unsigned long size, int node, gfp_t flags) { - return __vmalloc_node(size, 1, flags, PAGE_KERNEL, + return __vmalloc_node(size, 1, flags, PAGE_KERNEL, VM_HUGE_PAGES, node, __builtin_return_address(0)); } @@ -2850,7 +2914,8 @@ static inline void *__vmalloc_node_flags(unsigned long size, void *__vmalloc_node_flags_caller(unsigned long size, int node, gfp_t flags, void *caller) { - return __vmalloc_node(size, 1, flags, PAGE_KERNEL, node, caller); + return __vmalloc_node(size, 1, flags, PAGE_KERNEL, VM_HUGE_PAGES, + node, caller); } /** @@ -2925,7 +2990,7 @@ EXPORT_SYMBOL(vmalloc_user); */ void *vmalloc_node(unsigned long size, int node) { - return __vmalloc_node(size, 1, GFP_KERNEL, PAGE_KERNEL, + return __vmalloc_node(size, 1, GFP_KERNEL, PAGE_KERNEL, VM_HUGE_PAGES, node, __builtin_return_address(0)); } EXPORT_SYMBOL(vmalloc_node); @@ -3014,7 +3079,7 @@ void *vmalloc_exec(unsigned long size) */ void *vmalloc_32(unsigned long size) { - return __vmalloc_node(size, 1, GFP_VMALLOC32, PAGE_KERNEL, + return __vmalloc_node(size, 1, GFP_VMALLOC32, PAGE_KERNEL, 0, NUMA_NO_NODE, __builtin_return_address(0)); } EXPORT_SYMBOL(vmalloc_32);
For platforms that define HAVE_ARCH_HUGE_VMAP and support PMD vmap mappings, have vmalloc attempt to allocate PMD-sized pages first, before falling back to small pages. Allocations which use something other than PAGE_KERNEL protections are not permitted to use huge pages yet, not all callers expect this (e.g., module allocations vs strict module rwx). This gives a 6x reduction in dTLB misses for a `git diff` (of linux), from 45600 to 6500 and a 2.2% reduction in cycles on a 2-node POWER9. This can result in more internal fragmentation and memory overhead for a given allocation. It can also cause greater NUMA unbalance on hashdist allocations. There may be other callers that expect small pages under vmalloc but use PAGE_KERNEL, I'm not sure if it's feasible to catch them all. An alternative would be a new function or flag which enables large mappings, and use that in callers. Signed-off-by: Nicholas Piggin <npiggin@gmail.com> --- include/linux/vmalloc.h | 2 + mm/vmalloc.c | 135 +++++++++++++++++++++++++++++----------- 2 files changed, 102 insertions(+), 35 deletions(-)