| Message ID | 20190321200157.29678-1-keith.busch@intel.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Page demotion for memory reclaim | expand |
On Thu, Mar 21, 2019 at 02:20:51PM -0700, Zi Yan wrote: > 1. The name of “page demotion” seems confusing to me, since I thought it was about large pages > demote to small pages as opposite to promoting small pages to THPs. Am I the only > one here? If you have a THP, we'll skip the page migration and fall through to split_huge_page_to_list(), then the smaller pages can be considered, migrated and reclaimed individually. Not that we couldn't try to migrate a THP directly. It was just simpler implementation for this first attempt. > 2. For the demotion path, a common case would be from high-performance memory, like HBM > or Multi-Channel DRAM, to DRAM, then to PMEM, and finally to disks, right? More general > case for demotion path would be derived from the memory performance description from HMAT[1], > right? Do you have any algorithm to form such a path from HMAT? Yes, I have a PoC for the kernel setting up a demotion path based on HMAT properties here: https://git.kernel.org/pub/scm/linux/kernel/git/kbusch/linux.git/commit/?h=mm-migrate&id=4d007659e1dd1b0dad49514348be4441fbe7cadb The above is just from an experimental branch. > 3. Do you have a plan for promoting pages from lower-level memory to higher-level memory, > like from PMEM to DRAM? Will this one-way demotion make all pages sink to PMEM and disk? Promoting previously demoted pages would require the application do something to make that happen if you turn demotion on with this series. Kernel auto-promotion is still being investigated, and it's a little trickier than reclaim. If it sinks to disk, though, the next access behavior is the same as before, without this series. > 4. In your patch 3, you created a new method migrate_demote_mapping() to migrate pages to > other memory node, is there any problem of reusing existing migrate_pages() interface? Yes, we may not want to migrate everything in the shrink_page_list() pages. We might want to keep a page, so we have to do those checks first. At the point we know we want to attempt migration, the page is already locked and not in a list, so it is just easier to directly invoke the new __unmap_and_move_locked() that migrate_pages() eventually also calls. > 5. In addition, you only migrate base pages, is there any performance concern on migrating THPs? > Is it too costly to migrate THPs? It was just easier to consider single pages first, so we let a THP split if possible. I'm not sure of the cost in migrating THPs directly.
On Thu, Mar 21, 2019 at 3:36 PM Keith Busch <keith.busch@intel.com> wrote: > > On Thu, Mar 21, 2019 at 02:20:51PM -0700, Zi Yan wrote: > > 1. The name of “page demotion” seems confusing to me, since I thought it was about large pages > > demote to small pages as opposite to promoting small pages to THPs. Am I the only > > one here? > > If you have a THP, we'll skip the page migration and fall through to > split_huge_page_to_list(), then the smaller pages can be considered, > migrated and reclaimed individually. Not that we couldn't try to migrate > a THP directly. It was just simpler implementation for this first attempt. > > > 2. For the demotion path, a common case would be from high-performance memory, like HBM > > or Multi-Channel DRAM, to DRAM, then to PMEM, and finally to disks, right? More general > > case for demotion path would be derived from the memory performance description from HMAT[1], > > right? Do you have any algorithm to form such a path from HMAT? > > Yes, I have a PoC for the kernel setting up a demotion path based on > HMAT properties here: > > https://git.kernel.org/pub/scm/linux/kernel/git/kbusch/linux.git/commit/?h=mm-migrate&id=4d007659e1dd1b0dad49514348be4441fbe7cadb > > The above is just from an experimental branch. > > > 3. Do you have a plan for promoting pages from lower-level memory to higher-level memory, > > like from PMEM to DRAM? Will this one-way demotion make all pages sink to PMEM and disk? > > Promoting previously demoted pages would require the application do > something to make that happen if you turn demotion on with this series. > Kernel auto-promotion is still being investigated, and it's a little > trickier than reclaim. Just FYI. I'm currently working on a patchset which tries to promotes page from second tier memory (i.e. PMEM) to DRAM via NUMA balancing. But, NUMA balancing can't deal with unmapped page cache, they have to be promoted from different path, i.e. mark_page_accessed(). And, I do agree with Keith, promotion is definitely trickier than reclaim since kernel can't recognize "hot" pages accurately. NUMA balancing is still corse-grained and inaccurate, but it is simple. If we would like to implement more sophisticated algorithm, in-kernel implementation might be not a good idea. Thanks, Yang > > If it sinks to disk, though, the next access behavior is the same as > before, without this series. > > > 4. In your patch 3, you created a new method migrate_demote_mapping() to migrate pages to > > other memory node, is there any problem of reusing existing migrate_pages() interface? > > Yes, we may not want to migrate everything in the shrink_page_list() > pages. We might want to keep a page, so we have to do those checks first. At > the point we know we want to attempt migration, the page is already > locked and not in a list, so it is just easier to directly invoke the > new __unmap_and_move_locked() that migrate_pages() eventually also calls. > > > 5. In addition, you only migrate base pages, is there any performance concern on migrating THPs? > > Is it too costly to migrate THPs? > > It was just easier to consider single pages first, so we let a THP split > if possible. I'm not sure of the cost in migrating THPs directly. >
On Thu, Mar 21, 2019 at 05:12:33PM -0700, Zi Yan wrote: > > Yes, we may not want to migrate everything in the shrink_page_list() > > pages. We might want to keep a page, so we have to do those checks first. At > > the point we know we want to attempt migration, the page is already > > locked and not in a list, so it is just easier to directly invoke the > > new __unmap_and_move_locked() that migrate_pages() eventually also calls. > > Right, I understand that you want to only migrate small pages to begin with. My question is > why not using the existing migrate_pages() in your patch 3. Like: > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index a5ad0b35ab8e..0a0753af357f 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -1261,6 +1261,20 @@ static unsigned long shrink_page_list(struct list_head *page_list, > ; /* try to reclaim the page below */ > } > > + if (!PageCompound(page)) { > + int next_nid = next_migration_node(page); > + int err; > + > + if (next_nid != TERMINAL_NODE) { > + LIST_HEAD(migrate_list); > + list_add(&migrate_list, &page->lru); > + err = migrate_pages(&migrate_list, alloc_new_node_page, NULL, > + next_nid, MIGRATE_ASYNC, MR_DEMOTION); > + if (err) > + putback_movable_pages(&migrate_list); > + } > + } > + > /* > * Anonymous process memory has backing store? > * Try to allocate it some swap space here. > > Because your new migrate_demote_mapping() basically does the same thing as the code above. > If you are not OK with the gfp flags in alloc_new_node_page(), you can just write your own > alloc_new_node_page(). :) The page is already locked, you can't call migrate_pages() with locked pages. You'd have to surround migrate_pages with unlock_page/try_lock_page, and I thought that looked odd. Further, it changes the flow if the subsequent try lock fails, and I'm trying to be careful about not introducing different behavior if migration fails. Patch 2/5 is included here so we can reuse the necessary code from a locked page context.