| Message ID | 20200916183411.64756-3-david@redhat.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | mm: place pages to the freelist tail when onling and undoing isolation | expand |
On Wed, Sep 16, 2020 at 11:34 AM David Hildenbrand <david@redhat.com> wrote: > > __putback_isolated_page() already documents that pages will be placed to > the tail of the freelist - this is, however, not the case for > "order >= MAX_ORDER - 2" (see buddy_merge_likely()) - which should be > the case for all existing users. > > This change affects two users: > - free page reporting > - page isolation, when undoing the isolation. > > This behavior is desireable for pages that haven't really been touched I think "desirable" is misspelled here. > lately, so exactly the two users that don't actually read/write page > content, but rather move untouched pages. So in reality we were already dealing with this for page reporting, but not in the most direct way. If I recall we were adding the pages to the head of the list and then when we would go back to pull more pages we were doing list rotation in the report function so they were technically being added to the head, but usually would end up back on the tail anyway. If anything the benefit for page reporting is that it should be more direct this way as we will only have to rescan the pages now when we have consumed all of the reported ones on the list. > The new behavior is especially desirable for memory onlining, where we > allow allocation of newly onlined pages via undo_isolate_page_range() > in online_pages(). Right now, we always place them to the head of the > free list, resulting in undesireable behavior: Assume we add > individual memory chunks via add_memory() and online them right away to > the NORMAL zone. We create a dependency chain of unmovable allocations > e.g., via the memmap. The memmap of the next chunk will be placed onto > previous chunks - if the last block cannot get offlined+removed, all > dependent ones cannot get offlined+removed. While this can already be > observed with individual DIMMs, it's more of an issue for virtio-mem > (and I suspect also ppc DLPAR). > > Note: If we observe a degradation due to the changed page isolation > behavior (which I doubt), we can always make this configurable by the > instance triggering undo of isolation (e.g., alloc_contig_range(), > memory onlining, memory offlining). > > Cc: Andrew Morton <akpm@linux-foundation.org> > Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com> > Cc: Mel Gorman <mgorman@techsingularity.net> > Cc: Michal Hocko <mhocko@kernel.org> > Cc: Dave Hansen <dave.hansen@intel.com> > Cc: Vlastimil Babka <vbabka@suse.cz> > Cc: Wei Yang <richard.weiyang@linux.alibaba.com> > Cc: Oscar Salvador <osalvador@suse.de> > Cc: Mike Rapoport <rppt@kernel.org> > Cc: Scott Cheloha <cheloha@linux.ibm.com> > Cc: Michael Ellerman <mpe@ellerman.id.au> > Signed-off-by: David Hildenbrand <david@redhat.com> > --- > mm/page_alloc.c | 10 +++++++++- > 1 file changed, 9 insertions(+), 1 deletion(-) > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 91cefb8157dd..bba9a0f60c70 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -89,6 +89,12 @@ typedef int __bitwise fop_t; > */ > #define FOP_SKIP_REPORT_NOTIFY ((__force fop_t)BIT(0)) > > +/* > + * Place the freed page to the tail of the freelist after buddy merging. Will > + * get ignored with page shuffling enabled. > + */ > +#define FOP_TO_TAIL ((__force fop_t)BIT(1)) > + > /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ > static DEFINE_MUTEX(pcp_batch_high_lock); > #define MIN_PERCPU_PAGELIST_FRACTION (8) > @@ -1040,6 +1046,8 @@ static inline void __free_one_page(struct page *page, unsigned long pfn, > > if (is_shuffle_order(order)) > to_tail = shuffle_pick_tail(); > + else if (fop_flags & FOP_TO_TAIL) > + to_tail = true; > else > to_tail = buddy_merge_likely(pfn, buddy_pfn, page, order); > > @@ -3289,7 +3297,7 @@ void __putback_isolated_page(struct page *page, unsigned int order, int mt) > > /* Return isolated page to tail of freelist. */ > __free_one_page(page, page_to_pfn(page), zone, order, mt, > - FOP_SKIP_REPORT_NOTIFY); > + FOP_SKIP_REPORT_NOTIFY | FOP_TO_TAIL); > } > > /* The code looks good to me. Reviewed-by: Alexander Duyck <alexander.h.duyck@linux.intel.com>
On Wed, Sep 16, 2020 at 08:34:09PM +0200, David Hildenbrand wrote: >__putback_isolated_page() already documents that pages will be placed to >the tail of the freelist - this is, however, not the case for >"order >= MAX_ORDER - 2" (see buddy_merge_likely()) - which should be >the case for all existing users. > >This change affects two users: >- free page reporting >- page isolation, when undoing the isolation. > >This behavior is desireable for pages that haven't really been touched >lately, so exactly the two users that don't actually read/write page >content, but rather move untouched pages. > >The new behavior is especially desirable for memory onlining, where we >allow allocation of newly onlined pages via undo_isolate_page_range() >in online_pages(). Right now, we always place them to the head of the The code looks good, while I don't fully understand the log here. undo_isolate_page_range() is used in __offline_pages and alloc_contig_range. I don't connect them with online_pages(). Do I miss something? >free list, resulting in undesireable behavior: Assume we add >individual memory chunks via add_memory() and online them right away to >the NORMAL zone. We create a dependency chain of unmovable allocations >e.g., via the memmap. The memmap of the next chunk will be placed onto >previous chunks - if the last block cannot get offlined+removed, all >dependent ones cannot get offlined+removed. While this can already be >observed with individual DIMMs, it's more of an issue for virtio-mem >(and I suspect also ppc DLPAR). > >Note: If we observe a degradation due to the changed page isolation >behavior (which I doubt), we can always make this configurable by the >instance triggering undo of isolation (e.g., alloc_contig_range(), >memory onlining, memory offlining). > >Cc: Andrew Morton <akpm@linux-foundation.org> >Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com> >Cc: Mel Gorman <mgorman@techsingularity.net> >Cc: Michal Hocko <mhocko@kernel.org> >Cc: Dave Hansen <dave.hansen@intel.com> >Cc: Vlastimil Babka <vbabka@suse.cz> >Cc: Wei Yang <richard.weiyang@linux.alibaba.com> >Cc: Oscar Salvador <osalvador@suse.de> >Cc: Mike Rapoport <rppt@kernel.org> >Cc: Scott Cheloha <cheloha@linux.ibm.com> >Cc: Michael Ellerman <mpe@ellerman.id.au> >Signed-off-by: David Hildenbrand <david@redhat.com> >--- > mm/page_alloc.c | 10 +++++++++- > 1 file changed, 9 insertions(+), 1 deletion(-) > >diff --git a/mm/page_alloc.c b/mm/page_alloc.c >index 91cefb8157dd..bba9a0f60c70 100644 >--- a/mm/page_alloc.c >+++ b/mm/page_alloc.c >@@ -89,6 +89,12 @@ typedef int __bitwise fop_t; > */ > #define FOP_SKIP_REPORT_NOTIFY ((__force fop_t)BIT(0)) > >+/* >+ * Place the freed page to the tail of the freelist after buddy merging. Will >+ * get ignored with page shuffling enabled. >+ */ >+#define FOP_TO_TAIL ((__force fop_t)BIT(1)) >+ > /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ > static DEFINE_MUTEX(pcp_batch_high_lock); > #define MIN_PERCPU_PAGELIST_FRACTION (8) >@@ -1040,6 +1046,8 @@ static inline void __free_one_page(struct page *page, unsigned long pfn, > > if (is_shuffle_order(order)) > to_tail = shuffle_pick_tail(); >+ else if (fop_flags & FOP_TO_TAIL) >+ to_tail = true; > else > to_tail = buddy_merge_likely(pfn, buddy_pfn, page, order); > >@@ -3289,7 +3297,7 @@ void __putback_isolated_page(struct page *page, unsigned int order, int mt) > > /* Return isolated page to tail of freelist. */ > __free_one_page(page, page_to_pfn(page), zone, order, mt, >- FOP_SKIP_REPORT_NOTIFY); >+ FOP_SKIP_REPORT_NOTIFY | FOP_TO_TAIL); > } > > /* >-- >2.26.2
On Wed, Sep 16, 2020 at 08:34:09PM +0200, David Hildenbrand wrote: >__putback_isolated_page() already documents that pages will be placed to >the tail of the freelist - this is, however, not the case for >"order >= MAX_ORDER - 2" (see buddy_merge_likely()) - which should be >the case for all existing users. > >This change affects two users: >- free page reporting >- page isolation, when undoing the isolation. > >This behavior is desireable for pages that haven't really been touched >lately, so exactly the two users that don't actually read/write page >content, but rather move untouched pages. > >The new behavior is especially desirable for memory onlining, where we >allow allocation of newly onlined pages via undo_isolate_page_range() >in online_pages(). Right now, we always place them to the head of the >free list, resulting in undesireable behavior: Assume we add >individual memory chunks via add_memory() and online them right away to >the NORMAL zone. We create a dependency chain of unmovable allocations >e.g., via the memmap. The memmap of the next chunk will be placed onto >previous chunks - if the last block cannot get offlined+removed, all >dependent ones cannot get offlined+removed. While this can already be >observed with individual DIMMs, it's more of an issue for virtio-mem >(and I suspect also ppc DLPAR). > >Note: If we observe a degradation due to the changed page isolation >behavior (which I doubt), we can always make this configurable by the >instance triggering undo of isolation (e.g., alloc_contig_range(), >memory onlining, memory offlining). > >Cc: Andrew Morton <akpm@linux-foundation.org> >Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com> >Cc: Mel Gorman <mgorman@techsingularity.net> >Cc: Michal Hocko <mhocko@kernel.org> >Cc: Dave Hansen <dave.hansen@intel.com> >Cc: Vlastimil Babka <vbabka@suse.cz> >Cc: Wei Yang <richard.weiyang@linux.alibaba.com> >Cc: Oscar Salvador <osalvador@suse.de> >Cc: Mike Rapoport <rppt@kernel.org> >Cc: Scott Cheloha <cheloha@linux.ibm.com> >Cc: Michael Ellerman <mpe@ellerman.id.au> >Signed-off-by: David Hildenbrand <david@redhat.com> >--- > mm/page_alloc.c | 10 +++++++++- > 1 file changed, 9 insertions(+), 1 deletion(-) > >diff --git a/mm/page_alloc.c b/mm/page_alloc.c >index 91cefb8157dd..bba9a0f60c70 100644 >--- a/mm/page_alloc.c >+++ b/mm/page_alloc.c >@@ -89,6 +89,12 @@ typedef int __bitwise fop_t; > */ > #define FOP_SKIP_REPORT_NOTIFY ((__force fop_t)BIT(0)) > >+/* >+ * Place the freed page to the tail of the freelist after buddy merging. Will >+ * get ignored with page shuffling enabled. >+ */ >+#define FOP_TO_TAIL ((__force fop_t)BIT(1)) >+ > /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ > static DEFINE_MUTEX(pcp_batch_high_lock); > #define MIN_PERCPU_PAGELIST_FRACTION (8) >@@ -1040,6 +1046,8 @@ static inline void __free_one_page(struct page *page, unsigned long pfn, > > if (is_shuffle_order(order)) > to_tail = shuffle_pick_tail(); >+ else if (fop_flags & FOP_TO_TAIL) >+ to_tail = true; Take another look into this part. Maybe we can move this check at top? For online_page case, currently we have following call flow: online_page online_pages_range shuffle_zone This means we would always shuffle the newly added pages. Maybe we don't need to do the shuffle when adding them to the free_list? > else > to_tail = buddy_merge_likely(pfn, buddy_pfn, page, order); > >@@ -3289,7 +3297,7 @@ void __putback_isolated_page(struct page *page, unsigned int order, int mt) > > /* Return isolated page to tail of freelist. */ > __free_one_page(page, page_to_pfn(page), zone, order, mt, >- FOP_SKIP_REPORT_NOTIFY); >+ FOP_SKIP_REPORT_NOTIFY | FOP_TO_TAIL); > } > > /* >-- >2.26.2
On 18.09.20 04:07, Wei Yang wrote: > On Wed, Sep 16, 2020 at 08:34:09PM +0200, David Hildenbrand wrote: >> __putback_isolated_page() already documents that pages will be placed to >> the tail of the freelist - this is, however, not the case for >> "order >= MAX_ORDER - 2" (see buddy_merge_likely()) - which should be >> the case for all existing users. >> >> This change affects two users: >> - free page reporting >> - page isolation, when undoing the isolation. >> >> This behavior is desireable for pages that haven't really been touched >> lately, so exactly the two users that don't actually read/write page >> content, but rather move untouched pages. >> >> The new behavior is especially desirable for memory onlining, where we >> allow allocation of newly onlined pages via undo_isolate_page_range() >> in online_pages(). Right now, we always place them to the head of the > > The code looks good, while I don't fully understand the log here. > > undo_isolate_page_range() is used in __offline_pages and alloc_contig_range. I > don't connect them with online_pages(). Do I miss something? Yeah, please look at -mm / -next instead. See https://lkml.kernel.org/r/20200819175957.28465-11-david@redhat.com
On 18.09.20 04:16, Wei Yang wrote: > On Wed, Sep 16, 2020 at 08:34:09PM +0200, David Hildenbrand wrote: >> __putback_isolated_page() already documents that pages will be placed to >> the tail of the freelist - this is, however, not the case for >> "order >= MAX_ORDER - 2" (see buddy_merge_likely()) - which should be >> the case for all existing users. >> >> This change affects two users: >> - free page reporting >> - page isolation, when undoing the isolation. >> >> This behavior is desireable for pages that haven't really been touched >> lately, so exactly the two users that don't actually read/write page >> content, but rather move untouched pages. >> >> The new behavior is especially desirable for memory onlining, where we >> allow allocation of newly onlined pages via undo_isolate_page_range() >> in online_pages(). Right now, we always place them to the head of the >> free list, resulting in undesireable behavior: Assume we add >> individual memory chunks via add_memory() and online them right away to >> the NORMAL zone. We create a dependency chain of unmovable allocations >> e.g., via the memmap. The memmap of the next chunk will be placed onto >> previous chunks - if the last block cannot get offlined+removed, all >> dependent ones cannot get offlined+removed. While this can already be >> observed with individual DIMMs, it's more of an issue for virtio-mem >> (and I suspect also ppc DLPAR). >> >> Note: If we observe a degradation due to the changed page isolation >> behavior (which I doubt), we can always make this configurable by the >> instance triggering undo of isolation (e.g., alloc_contig_range(), >> memory onlining, memory offlining). >> >> Cc: Andrew Morton <akpm@linux-foundation.org> >> Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com> >> Cc: Mel Gorman <mgorman@techsingularity.net> >> Cc: Michal Hocko <mhocko@kernel.org> >> Cc: Dave Hansen <dave.hansen@intel.com> >> Cc: Vlastimil Babka <vbabka@suse.cz> >> Cc: Wei Yang <richard.weiyang@linux.alibaba.com> >> Cc: Oscar Salvador <osalvador@suse.de> >> Cc: Mike Rapoport <rppt@kernel.org> >> Cc: Scott Cheloha <cheloha@linux.ibm.com> >> Cc: Michael Ellerman <mpe@ellerman.id.au> >> Signed-off-by: David Hildenbrand <david@redhat.com> >> --- >> mm/page_alloc.c | 10 +++++++++- >> 1 file changed, 9 insertions(+), 1 deletion(-) >> >> diff --git a/mm/page_alloc.c b/mm/page_alloc.c >> index 91cefb8157dd..bba9a0f60c70 100644 >> --- a/mm/page_alloc.c >> +++ b/mm/page_alloc.c >> @@ -89,6 +89,12 @@ typedef int __bitwise fop_t; >> */ >> #define FOP_SKIP_REPORT_NOTIFY ((__force fop_t)BIT(0)) >> >> +/* >> + * Place the freed page to the tail of the freelist after buddy merging. Will >> + * get ignored with page shuffling enabled. >> + */ >> +#define FOP_TO_TAIL ((__force fop_t)BIT(1)) >> + >> /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ >> static DEFINE_MUTEX(pcp_batch_high_lock); >> #define MIN_PERCPU_PAGELIST_FRACTION (8) >> @@ -1040,6 +1046,8 @@ static inline void __free_one_page(struct page *page, unsigned long pfn, >> >> if (is_shuffle_order(order)) >> to_tail = shuffle_pick_tail(); >> + else if (fop_flags & FOP_TO_TAIL) >> + to_tail = true; > > Take another look into this part. Maybe we can move this check at top? > > For online_page case, currently we have following call flow: > > online_page > online_pages_range > shuffle_zone > > This means we would always shuffle the newly added pages. Maybe we don't need > to do the shuffle when adding them to the free_list? Yeah we don't, but it doesn't really buy us too much as the call paths I am touching are used by other mechanisms as well that need it (especially undoing page isolation).
On Fri, Sep 18, 2020 at 09:27:23AM +0200, David Hildenbrand wrote: >On 18.09.20 04:07, Wei Yang wrote: >> On Wed, Sep 16, 2020 at 08:34:09PM +0200, David Hildenbrand wrote: >>> __putback_isolated_page() already documents that pages will be placed to >>> the tail of the freelist - this is, however, not the case for >>> "order >= MAX_ORDER - 2" (see buddy_merge_likely()) - which should be >>> the case for all existing users. >>> >>> This change affects two users: >>> - free page reporting >>> - page isolation, when undoing the isolation. >>> >>> This behavior is desireable for pages that haven't really been touched >>> lately, so exactly the two users that don't actually read/write page >>> content, but rather move untouched pages. >>> >>> The new behavior is especially desirable for memory onlining, where we >>> allow allocation of newly onlined pages via undo_isolate_page_range() >>> in online_pages(). Right now, we always place them to the head of the >> >> The code looks good, while I don't fully understand the log here. >> >> undo_isolate_page_range() is used in __offline_pages and alloc_contig_range. I >> don't connect them with online_pages(). Do I miss something? > >Yeah, please look at -mm / -next instead. See > >https://lkml.kernel.org/r/20200819175957.28465-11-david@redhat.com > Thanks, I get the point. > >-- >Thanks, > >David / dhildenb
On 9/16/20 8:34 PM, David Hildenbrand wrote: > __putback_isolated_page() already documents that pages will be placed to > the tail of the freelist - this is, however, not the case for > "order >= MAX_ORDER - 2" (see buddy_merge_likely()) - which should be > the case for all existing users. I think here should be a sentence saying something along "Thus this patch introduces a FOP_TO_TAIL flag to really ensure moving pages to tail." > This change affects two users: > - free page reporting > - page isolation, when undoing the isolation. > > This behavior is desireable for pages that haven't really been touched > lately, so exactly the two users that don't actually read/write page > content, but rather move untouched pages. > > The new behavior is especially desirable for memory onlining, where we > allow allocation of newly onlined pages via undo_isolate_page_range() > in online_pages(). Right now, we always place them to the head of the > free list, resulting in undesireable behavior: Assume we add > individual memory chunks via add_memory() and online them right away to > the NORMAL zone. We create a dependency chain of unmovable allocations > e.g., via the memmap. The memmap of the next chunk will be placed onto > previous chunks - if the last block cannot get offlined+removed, all > dependent ones cannot get offlined+removed. While this can already be > observed with individual DIMMs, it's more of an issue for virtio-mem > (and I suspect also ppc DLPAR). > > Note: If we observe a degradation due to the changed page isolation > behavior (which I doubt), we can always make this configurable by the > instance triggering undo of isolation (e.g., alloc_contig_range(), > memory onlining, memory offlining). > > Cc: Andrew Morton <akpm@linux-foundation.org> > Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com> > Cc: Mel Gorman <mgorman@techsingularity.net> > Cc: Michal Hocko <mhocko@kernel.org> > Cc: Dave Hansen <dave.hansen@intel.com> > Cc: Vlastimil Babka <vbabka@suse.cz> > Cc: Wei Yang <richard.weiyang@linux.alibaba.com> > Cc: Oscar Salvador <osalvador@suse.de> > Cc: Mike Rapoport <rppt@kernel.org> > Cc: Scott Cheloha <cheloha@linux.ibm.com> > Cc: Michael Ellerman <mpe@ellerman.id.au> > Signed-off-by: David Hildenbrand <david@redhat.com> > --- > mm/page_alloc.c | 10 +++++++++- > 1 file changed, 9 insertions(+), 1 deletion(-) > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 91cefb8157dd..bba9a0f60c70 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -89,6 +89,12 @@ typedef int __bitwise fop_t; > */ > #define FOP_SKIP_REPORT_NOTIFY ((__force fop_t)BIT(0)) > > +/* > + * Place the freed page to the tail of the freelist after buddy merging. Will > + * get ignored with page shuffling enabled. > + */ > +#define FOP_TO_TAIL ((__force fop_t)BIT(1)) > + > /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ > static DEFINE_MUTEX(pcp_batch_high_lock); > #define MIN_PERCPU_PAGELIST_FRACTION (8) > @@ -1040,6 +1046,8 @@ static inline void __free_one_page(struct page *page, unsigned long pfn, > > if (is_shuffle_order(order)) > to_tail = shuffle_pick_tail(); > + else if (fop_flags & FOP_TO_TAIL) > + to_tail = true; Should we really let random shuffling decision have a larger priority than explicit FOP_TO_TAIL request? Wei Yang mentioned that there's a call to shuffle_zone() anyway to process a freshly added memory, so we don't need to do that also during the process of addition itself? Might help with your goal of reducing dependencies even on systems that do have shuffling enabled? Thanks, Vlastimil > else > to_tail = buddy_merge_likely(pfn, buddy_pfn, page, order); > > @@ -3289,7 +3297,7 @@ void __putback_isolated_page(struct page *page, unsigned int order, int mt) > > /* Return isolated page to tail of freelist. */ > __free_one_page(page, page_to_pfn(page), zone, order, mt, > - FOP_SKIP_REPORT_NOTIFY); > + FOP_SKIP_REPORT_NOTIFY | FOP_TO_TAIL); > } > > /* >
On 24.09.20 12:37, Vlastimil Babka wrote: > On 9/16/20 8:34 PM, David Hildenbrand wrote: >> __putback_isolated_page() already documents that pages will be placed to >> the tail of the freelist - this is, however, not the case for >> "order >= MAX_ORDER - 2" (see buddy_merge_likely()) - which should be >> the case for all existing users. > > I think here should be a sentence saying something along "Thus this patch > introduces a FOP_TO_TAIL flag to really ensure moving pages to tail." Agreed, thanks! > >> This change affects two users: >> - free page reporting >> - page isolation, when undoing the isolation. >> >> This behavior is desireable for pages that haven't really been touched >> lately, so exactly the two users that don't actually read/write page >> content, but rather move untouched pages. >> >> The new behavior is especially desirable for memory onlining, where we >> allow allocation of newly onlined pages via undo_isolate_page_range() >> in online_pages(). Right now, we always place them to the head of the >> free list, resulting in undesireable behavior: Assume we add >> individual memory chunks via add_memory() and online them right away to >> the NORMAL zone. We create a dependency chain of unmovable allocations >> e.g., via the memmap. The memmap of the next chunk will be placed onto >> previous chunks - if the last block cannot get offlined+removed, all >> dependent ones cannot get offlined+removed. While this can already be >> observed with individual DIMMs, it's more of an issue for virtio-mem >> (and I suspect also ppc DLPAR). >> >> Note: If we observe a degradation due to the changed page isolation >> behavior (which I doubt), we can always make this configurable by the >> instance triggering undo of isolation (e.g., alloc_contig_range(), >> memory onlining, memory offlining). >> >> Cc: Andrew Morton <akpm@linux-foundation.org> >> Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com> >> Cc: Mel Gorman <mgorman@techsingularity.net> >> Cc: Michal Hocko <mhocko@kernel.org> >> Cc: Dave Hansen <dave.hansen@intel.com> >> Cc: Vlastimil Babka <vbabka@suse.cz> >> Cc: Wei Yang <richard.weiyang@linux.alibaba.com> >> Cc: Oscar Salvador <osalvador@suse.de> >> Cc: Mike Rapoport <rppt@kernel.org> >> Cc: Scott Cheloha <cheloha@linux.ibm.com> >> Cc: Michael Ellerman <mpe@ellerman.id.au> >> Signed-off-by: David Hildenbrand <david@redhat.com> >> --- >> mm/page_alloc.c | 10 +++++++++- >> 1 file changed, 9 insertions(+), 1 deletion(-) >> >> diff --git a/mm/page_alloc.c b/mm/page_alloc.c >> index 91cefb8157dd..bba9a0f60c70 100644 >> --- a/mm/page_alloc.c >> +++ b/mm/page_alloc.c >> @@ -89,6 +89,12 @@ typedef int __bitwise fop_t; >> */ >> #define FOP_SKIP_REPORT_NOTIFY ((__force fop_t)BIT(0)) >> >> +/* >> + * Place the freed page to the tail of the freelist after buddy merging. Will >> + * get ignored with page shuffling enabled. >> + */ >> +#define FOP_TO_TAIL ((__force fop_t)BIT(1)) >> + >> /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ >> static DEFINE_MUTEX(pcp_batch_high_lock); >> #define MIN_PERCPU_PAGELIST_FRACTION (8) >> @@ -1040,6 +1046,8 @@ static inline void __free_one_page(struct page *page, unsigned long pfn, >> >> if (is_shuffle_order(order)) >> to_tail = shuffle_pick_tail(); >> + else if (fop_flags & FOP_TO_TAIL) >> + to_tail = true; > > Should we really let random shuffling decision have a larger priority than > explicit FOP_TO_TAIL request? Wei Yang mentioned that there's a call to > shuffle_zone() anyway to process a freshly added memory, so we don't need to do > that also during the process of addition itself? Might help with your goal of > reducing dependencies even on systems that do have shuffling enabled? So, we do have cases where generic_online_page() -> __free_pages_core() isn't called (see patch #4): generic_online_page() is used in two cases: 1. Direct memory onlining in online_pages(). Here, we call shuffle_zone(). 2. Deferred memory onlining in memory-ballooning-like mechanisms (HyperV balloon and virtio-mem), when parts of a section are kept fake-offline to be fake-onlined later on. While we shuffle in the fist instance the whole zone, we wouldn't shuffle in the second case. But maybe this should be tackled (just like when alloc_contig_free() a large contiguous range, memory offlining failing, alloc_contig_range() failing) by manually shuffling the zone again. That would be cleaner, and the right thing to do when exposing large, contiguous ranges again to the buddy. Thanks!
On Wed, Sep 16, 2020 at 08:34:09PM +0200, David Hildenbrand wrote: > __putback_isolated_page() already documents that pages will be placed to > the tail of the freelist - this is, however, not the case for > "order >= MAX_ORDER - 2" (see buddy_merge_likely()) - which should be > the case for all existing users. > > This change affects two users: > - free page reporting > - page isolation, when undoing the isolation. > > This behavior is desireable for pages that haven't really been touched > lately, so exactly the two users that don't actually read/write page > content, but rather move untouched pages. > > The new behavior is especially desirable for memory onlining, where we > allow allocation of newly onlined pages via undo_isolate_page_range() > in online_pages(). Right now, we always place them to the head of the > free list, resulting in undesireable behavior: Assume we add > individual memory chunks via add_memory() and online them right away to > the NORMAL zone. We create a dependency chain of unmovable allocations > e.g., via the memmap. The memmap of the next chunk will be placed onto > previous chunks - if the last block cannot get offlined+removed, all > dependent ones cannot get offlined+removed. While this can already be > observed with individual DIMMs, it's more of an issue for virtio-mem > (and I suspect also ppc DLPAR). > > Note: If we observe a degradation due to the changed page isolation > behavior (which I doubt), we can always make this configurable by the > instance triggering undo of isolation (e.g., alloc_contig_range(), > memory onlining, memory offlining). > > Cc: Andrew Morton <akpm@linux-foundation.org> > Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com> > Cc: Mel Gorman <mgorman@techsingularity.net> > Cc: Michal Hocko <mhocko@kernel.org> > Cc: Dave Hansen <dave.hansen@intel.com> > Cc: Vlastimil Babka <vbabka@suse.cz> > Cc: Wei Yang <richard.weiyang@linux.alibaba.com> > Cc: Oscar Salvador <osalvador@suse.de> > Cc: Mike Rapoport <rppt@kernel.org> > Cc: Scott Cheloha <cheloha@linux.ibm.com> > Cc: Michael Ellerman <mpe@ellerman.id.au> > Signed-off-by: David Hildenbrand <david@redhat.com> LGTM, the only thing is the shuffe_zone topic that Wei and Vlastimil rose. Feels a bit odd that takes precedence over something we explicitily demanded. With the comment Vlastimil suggested: Reviewed-by: Oscar Salvador <osalvador@suse.de>
On 25.09.20 15:19, Oscar Salvador wrote: > On Wed, Sep 16, 2020 at 08:34:09PM +0200, David Hildenbrand wrote: >> __putback_isolated_page() already documents that pages will be placed to >> the tail of the freelist - this is, however, not the case for >> "order >= MAX_ORDER - 2" (see buddy_merge_likely()) - which should be >> the case for all existing users. >> >> This change affects two users: >> - free page reporting >> - page isolation, when undoing the isolation. >> >> This behavior is desireable for pages that haven't really been touched >> lately, so exactly the two users that don't actually read/write page >> content, but rather move untouched pages. >> >> The new behavior is especially desirable for memory onlining, where we >> allow allocation of newly onlined pages via undo_isolate_page_range() >> in online_pages(). Right now, we always place them to the head of the >> free list, resulting in undesireable behavior: Assume we add >> individual memory chunks via add_memory() and online them right away to >> the NORMAL zone. We create a dependency chain of unmovable allocations >> e.g., via the memmap. The memmap of the next chunk will be placed onto >> previous chunks - if the last block cannot get offlined+removed, all >> dependent ones cannot get offlined+removed. While this can already be >> observed with individual DIMMs, it's more of an issue for virtio-mem >> (and I suspect also ppc DLPAR). >> >> Note: If we observe a degradation due to the changed page isolation >> behavior (which I doubt), we can always make this configurable by the >> instance triggering undo of isolation (e.g., alloc_contig_range(), >> memory onlining, memory offlining). >> >> Cc: Andrew Morton <akpm@linux-foundation.org> >> Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com> >> Cc: Mel Gorman <mgorman@techsingularity.net> >> Cc: Michal Hocko <mhocko@kernel.org> >> Cc: Dave Hansen <dave.hansen@intel.com> >> Cc: Vlastimil Babka <vbabka@suse.cz> >> Cc: Wei Yang <richard.weiyang@linux.alibaba.com> >> Cc: Oscar Salvador <osalvador@suse.de> >> Cc: Mike Rapoport <rppt@kernel.org> >> Cc: Scott Cheloha <cheloha@linux.ibm.com> >> Cc: Michael Ellerman <mpe@ellerman.id.au> >> Signed-off-by: David Hildenbrand <david@redhat.com> > > LGTM, the only thing is the shuffe_zone topic that Wei and Vlastimil rose. > Feels a bit odd that takes precedence over something we explicitily demanded. > Thanks, yeah I'll be changing that. > With the comment Vlastimil suggested: > > Reviewed-by: Oscar Salvador <osalvador@suse.de> >
diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 91cefb8157dd..bba9a0f60c70 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -89,6 +89,12 @@ typedef int __bitwise fop_t; */ #define FOP_SKIP_REPORT_NOTIFY ((__force fop_t)BIT(0)) +/* + * Place the freed page to the tail of the freelist after buddy merging. Will + * get ignored with page shuffling enabled. + */ +#define FOP_TO_TAIL ((__force fop_t)BIT(1)) + /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX(pcp_batch_high_lock); #define MIN_PERCPU_PAGELIST_FRACTION (8) @@ -1040,6 +1046,8 @@ static inline void __free_one_page(struct page *page, unsigned long pfn, if (is_shuffle_order(order)) to_tail = shuffle_pick_tail(); + else if (fop_flags & FOP_TO_TAIL) + to_tail = true; else to_tail = buddy_merge_likely(pfn, buddy_pfn, page, order); @@ -3289,7 +3297,7 @@ void __putback_isolated_page(struct page *page, unsigned int order, int mt) /* Return isolated page to tail of freelist. */ __free_one_page(page, page_to_pfn(page), zone, order, mt, - FOP_SKIP_REPORT_NOTIFY); + FOP_SKIP_REPORT_NOTIFY | FOP_TO_TAIL); } /*
__putback_isolated_page() already documents that pages will be placed to the tail of the freelist - this is, however, not the case for "order >= MAX_ORDER - 2" (see buddy_merge_likely()) - which should be the case for all existing users. This change affects two users: - free page reporting - page isolation, when undoing the isolation. This behavior is desireable for pages that haven't really been touched lately, so exactly the two users that don't actually read/write page content, but rather move untouched pages. The new behavior is especially desirable for memory onlining, where we allow allocation of newly onlined pages via undo_isolate_page_range() in online_pages(). Right now, we always place them to the head of the free list, resulting in undesireable behavior: Assume we add individual memory chunks via add_memory() and online them right away to the NORMAL zone. We create a dependency chain of unmovable allocations e.g., via the memmap. The memmap of the next chunk will be placed onto previous chunks - if the last block cannot get offlined+removed, all dependent ones cannot get offlined+removed. While this can already be observed with individual DIMMs, it's more of an issue for virtio-mem (and I suspect also ppc DLPAR). Note: If we observe a degradation due to the changed page isolation behavior (which I doubt), we can always make this configurable by the instance triggering undo of isolation (e.g., alloc_contig_range(), memory onlining, memory offlining). Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com> Cc: Mel Gorman <mgorman@techsingularity.net> Cc: Michal Hocko <mhocko@kernel.org> Cc: Dave Hansen <dave.hansen@intel.com> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: Wei Yang <richard.weiyang@linux.alibaba.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Mike Rapoport <rppt@kernel.org> Cc: Scott Cheloha <cheloha@linux.ibm.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Signed-off-by: David Hildenbrand <david@redhat.com> --- mm/page_alloc.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)