From patchwork Fri Aug 30 10:03:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13784868 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BC4FCA0EFB for ; Fri, 30 Aug 2024 10:04:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2DF596B00F5; Fri, 30 Aug 2024 06:04:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F3E36B00F6; Fri, 30 Aug 2024 06:04:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 045696B00F7; Fri, 30 Aug 2024 06:04:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D505B6B00F5 for ; Fri, 30 Aug 2024 06:04:47 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9EC7EAB079 for ; Fri, 30 Aug 2024 10:04:47 +0000 (UTC) X-FDA: 82508477814.01.84685C0 Received: from mail-oo1-f48.google.com (mail-oo1-f48.google.com [209.85.161.48]) by imf21.hostedemail.com (Postfix) with ESMTP id C16D61C0020 for ; Fri, 30 Aug 2024 10:04:45 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Y00m0Yug; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.161.48 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725012212; a=rsa-sha256; cv=none; b=69aUkUz7FSFKsepl1vHPGrgALFl+vXj8PId5y/i1lcEHqtO7Qk/tADONmvwCEU2lisd7uz iP9rLjNwVRMafM8UwAggo22GxnvyRHlD2q/1ko4L1+p1TmORojGE/rTFxwo54oa04wrrQJ fddM7CbnC2YDEvP4ikkALdCRv7W+JrQ= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Y00m0Yug; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.161.48 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725012212; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lsaItdHLrkOs94vXPLpVeca+5XkpGp6YNABO6738q6c=; b=ltSDZUZpwojUFcKDXAayiSlVkdAor1XLwizlK4G5phVGtRNu8Io3D8otgxRMmW6Js1r73n gxOyJtbIGjoiIk4ssu0K8rkIhVI4HZiXBdGQtJSTMhNrp8AAyIydydKvlDsuOQJs7yOHuV VhUC/MhFHXpMH0anl0pYiWu2tsMkn/U= Received: by mail-oo1-f48.google.com with SMTP id 006d021491bc7-5dccc6cdc96so995937eaf.0 for ; Fri, 30 Aug 2024 03:04:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725012285; x=1725617085; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lsaItdHLrkOs94vXPLpVeca+5XkpGp6YNABO6738q6c=; b=Y00m0YugGJ7cjEXl2v+SFaOhiSb6ggSUrPQvmDqo3Spf5iJJSrFflDNN3ZpMp4Rdec tN7g65kA0F0zyr/36AKc5lMiw+9ACwwwkuKNm7hstflPVoyR9RlQOHkbHaJAiqCJ8eP6 hhJE7P+rwGj6Zgc+IhdwScq7thDFCadO/nJvUJRIzO+sQEqcWe7PqTQ3F2f91Mor1njQ 7K6rom8pLh8t72Yhb8l+gshy9P7QnCI6NKkLbJrrxaud/S2Vc/oiwPDLyHsHcSbOkIam M3uG4ETRRsC327QgT6iB3ikEiOJNCffk77xJO8wxUrtqS/kdTAeX4RlF6u+9cLr8dV0i 9QAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725012285; x=1725617085; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lsaItdHLrkOs94vXPLpVeca+5XkpGp6YNABO6738q6c=; b=lmYWDKWXSW49r2uHEUw0gcXy6UW9xODTE/IFzI0lFdjyittr9cZP/Rt2+BQ3sP5OZY mZefLg9XMROcP2x5PMD9rmLuvvlDupH5lUwtAfH6Ylf+kaq17YQpvfzrnI2fblqeX+Rj yDzcoavceX4MqKS61zvuWgqhgG6BA8H8xXmamPIil2MnV0TJUjp9C8AdLDr88Iz/jjn2 XOiCDnVI2rDSHfI3Hjnsvef3NHDMwFVbJh+AXtykAPvgSshaoiR6vuMwL/SvvvnTdV+y QsRJDoln/Awea85NAGy4N6wm40CNLFQ2DK9wBUCjaZXfskytYYxCheLxhB7RclG7YABN 9s/Q== X-Forwarded-Encrypted: i=1; AJvYcCUcLI8xSx9D0w0nmkVlOw+mx5v9+9nAT5ivwCNsygVfqsFI/uWXxnVQ0LsRcAu66FQ6ANoM61guSA==@kvack.org X-Gm-Message-State: AOJu0YwdqPj4GCBUdGyFWbOCuZDWDyE9S3mLYm+Ap8GAxU38VNqLCbAc AO6oqaKSejzPphpqZyOVl1YqP/cxo/d+3ecm3PIVpkAKysfNa+95 X-Google-Smtp-Source: AGHT+IHVXyDvTPcFg0d6jozxHg1gXGpjAfSA3ZiESWLvV7o1PUmHfmhl+s0Gs537cY51gwYtkOJCDA== X-Received: by 2002:a05:6820:1c9a:b0:5d5:bce7:677 with SMTP id 006d021491bc7-5dfacf2ff24mr1741737eaf.7.1725012284525; Fri, 30 Aug 2024 03:04:44 -0700 (PDT) Received: from localhost (fwdproxy-ash-113.fbsv.net. [2a03:2880:20ff:71::face:b00c]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6c340c96856sm13189406d6.89.2024.08.30.03.04.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Aug 2024 03:04:43 -0700 (PDT) From: Usama Arif To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, roman.gushchin@linux.dev, yuzhao@google.com, david@redhat.com, npache@redhat.com, baohua@kernel.org, ryan.roberts@arm.com, rppt@kernel.org, willy@infradead.org, cerasuolodomenico@gmail.com, ryncsn@gmail.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kernel-team@meta.com, Shuang Zhai , Usama Arif Subject: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp Date: Fri, 30 Aug 2024 11:03:36 +0100 Message-ID: <20240830100438.3623486-3-usamaarif642@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20240830100438.3623486-1-usamaarif642@gmail.com> References: <20240830100438.3623486-1-usamaarif642@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: C16D61C0020 X-Stat-Signature: ir6zrmrmfctdwc4pu61x9mtnhjjxmu5s X-Rspam-User: X-HE-Tag: 1725012285-354852 X-HE-Meta: U2FsdGVkX18TK2OFYlwyeAFCyISkSXPyRkm303stmhXWOR5e3AhYAVksM6d/BklvjuMFCCNw7s52xcwxmqqfISt7K6coQHmd6NP+ZFopQGdMN3F9HVYMAaUii8fdmMiOCumYwkw3yg4IinDH+sb3/9lbptXeFQ6FmIyoj3faC9qkLXOeN2Yjc8o+XnkV+7+5sSOsZ5ucFOcpzsTfm7isaPPf9mtj0AE8NtixnzUD0g08uKJG+Nnv+EZvxpr4PnWIzoHzTJ4EUJH113NM58geN2CyJUEE2vq7N5Po5BiG1c/apfNgZ++u7h7V2mRYWasayAV56lQcPs6aTYEPJVy4c4hOw+O7FFIKS9l791DWdfz8T0Kqbk6i/rEcx0aKTABGNbmJ33FDGjh40kcmbMY1PZ0Ys1oIbxRCLzzIh5glnhNFiUI6WYm5/qEcHk1GrSKwGRWKXW4PzQ7TYcv7PQhzeGyAgcOm+wsCnqCuNTFH2ZnnMmaMHkn7N1E9E5yGzv+YD3wzzrQ3NDAVk0/Ocwi2RdjvFBKhn7fyPGX0sDht84qDC4N/fL42uJObGajAan1wx+ffpO4uxZwEYNh+BJ+73r4e41BrAzW82xsyLjg4UwYRAv4uHtlY0A8oDOnUvBzCLFW1v9ZT+QcN25+IynaoL9lr+cR1lL97UB92CQPYcnO0k7HDHVb612cSKR9FaV74lCG65obvqe0MMfWi7aBYHFiwhvAQhCoSKabk6bvScqEmEjqLWK8e5NZ6nrWDmezZu0UQgkVUBpF7VpdQ/HwKJhLeqEFVj9h0ftjh6MHIzZhxvQWNetEfQ8HsXh0fvYo0WvP2U2flGsXihJz9xzcmlRcxnCDZT/ATJNER/gyk7O4+TUz7JFDsbKY98Ww9iojMy4pnUN3JB1fLe3gXLhcBssSpOFWnkpIYDpW8MlZYCQaSafnIBdKvizyiTyhhXXRbYbkjbDFn43PGEgfQPHH wJVh8gLQ EMCd/c26mgzbA5fXLoh634fWeCVe44jFFrj1TzumZvowHKuuIFX1WdVvxdRqva6n05g2QzcbB4mrxAhp58SneQ0vZ3wyiw93jhy93IOj7qNXEYxOrPaCv36F52tdHiJ52a5IOeLgqbmTQgCj1i2SxGpMFqXzdlKvLU1nq4ZcoAQOtkUixzroMLMPoeJBWD1wllgsQRK8w1lW/CSAMGKOuy2IlqNv+luYXYmvHeIF+czUPskZjKG6z4Ch2a5NYUCMgyT97iWneluaQX9Mg0hefMlz04G7g8gGB9UDtn/eoJ5jQdFVCJ6SgKgKz7ugqB3YPuGyj2Yt1CB94HYHiKLetS+b3iJFEkXlBzsBPVG5ePRhlSBc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Yu Zhao Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages. This is also a prerequisite for THP low utilization shrinker which will be introduced in later patches, where underutilized THPs are split, and the zero-filled pages are freed saving memory. Signed-off-by: Yu Zhao Tested-by: Shuang Zhai Signed-off-by: Usama Arif --- include/linux/rmap.h | 7 ++++- mm/huge_memory.c | 8 ++--- mm/migrate.c | 72 ++++++++++++++++++++++++++++++++++++++------ mm/migrate_device.c | 4 +-- 4 files changed, 75 insertions(+), 16 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 91b5935e8485..d5e93e44322e 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -745,7 +745,12 @@ int folio_mkclean(struct folio *); int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, struct vm_area_struct *vma); -void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked); +enum rmp_flags { + RMP_LOCKED = 1 << 0, + RMP_USE_SHARED_ZEROPAGE = 1 << 1, +}; + +void remove_migration_ptes(struct folio *src, struct folio *dst, int flags); /* * rmap_walk_control: To control rmap traversing for specific needs diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0c48806ccb9a..af60684e7c70 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr, return false; } -static void remap_page(struct folio *folio, unsigned long nr) +static void remap_page(struct folio *folio, unsigned long nr, int flags) { int i = 0; @@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio, unsigned long nr) if (!folio_test_anon(folio)) return; for (;;) { - remove_migration_ptes(folio, folio, true); + remove_migration_ptes(folio, folio, RMP_LOCKED | flags); i += folio_nr_pages(folio); if (i >= nr) break; @@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, if (nr_dropped) shmem_uncharge(folio->mapping->host, nr_dropped); - remap_page(folio, nr); + remap_page(folio, nr, PageAnon(head) ? RMP_USE_SHARED_ZEROPAGE : 0); /* * set page to its compound_head when split to non order-0 pages, so @@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, if (mapping) xas_unlock(&xas); local_irq_enable(); - remap_page(folio, folio_nr_pages(folio)); + remap_page(folio, folio_nr_pages(folio), 0); ret = -EAGAIN; } diff --git a/mm/migrate.c b/mm/migrate.c index 6f9c62c746be..d039863e014b 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -204,13 +204,57 @@ bool isolate_folio_to_list(struct folio *folio, struct list_head *list) return true; } +static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw, + struct folio *folio, + unsigned long idx) +{ + struct page *page = folio_page(folio, idx); + bool contains_data; + pte_t newpte; + void *addr; + + VM_BUG_ON_PAGE(PageCompound(page), page); + VM_BUG_ON_PAGE(!PageAnon(page), page); + VM_BUG_ON_PAGE(!PageLocked(page), page); + VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page); + + if (folio_test_mlocked(folio) || (pvmw->vma->vm_flags & VM_LOCKED) || + mm_forbids_zeropage(pvmw->vma->vm_mm)) + return false; + + /* + * The pmd entry mapping the old thp was flushed and the pte mapping + * this subpage has been non present. If the subpage is only zero-filled + * then map it to the shared zeropage. + */ + addr = kmap_local_page(page); + contains_data = memchr_inv(addr, 0, PAGE_SIZE); + kunmap_local(addr); + + if (contains_data) + return false; + + newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address), + pvmw->vma->vm_page_prot)); + set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte); + + dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio)); + return true; +} + +struct rmap_walk_arg { + struct folio *folio; + bool map_unused_to_zeropage; +}; + /* * Restore a potential migration pte to a working pte entry */ static bool remove_migration_pte(struct folio *folio, - struct vm_area_struct *vma, unsigned long addr, void *old) + struct vm_area_struct *vma, unsigned long addr, void *arg) { - DEFINE_FOLIO_VMA_WALK(pvmw, old, vma, addr, PVMW_SYNC | PVMW_MIGRATION); + struct rmap_walk_arg *rmap_walk_arg = arg; + DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | PVMW_MIGRATION); while (page_vma_mapped_walk(&pvmw)) { rmap_t rmap_flags = RMAP_NONE; @@ -234,6 +278,9 @@ static bool remove_migration_pte(struct folio *folio, continue; } #endif + if (rmap_walk_arg->map_unused_to_zeropage && + try_to_map_unused_to_zeropage(&pvmw, folio, idx)) + continue; folio_get(folio); pte = mk_pte(new, READ_ONCE(vma->vm_page_prot)); @@ -312,14 +359,21 @@ static bool remove_migration_pte(struct folio *folio, * Get rid of all migration entries and replace them by * references to the indicated page. */ -void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked) +void remove_migration_ptes(struct folio *src, struct folio *dst, int flags) { + struct rmap_walk_arg rmap_walk_arg = { + .folio = src, + .map_unused_to_zeropage = flags & RMP_USE_SHARED_ZEROPAGE, + }; + struct rmap_walk_control rwc = { .rmap_one = remove_migration_pte, - .arg = src, + .arg = &rmap_walk_arg, }; - if (locked) + VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src != dst), src); + + if (flags & RMP_LOCKED) rmap_walk_locked(dst, &rwc); else rmap_walk(dst, &rwc); @@ -934,7 +988,7 @@ static int writeout(struct address_space *mapping, struct folio *folio) * At this point we know that the migration attempt cannot * be successful. */ - remove_migration_ptes(folio, folio, false); + remove_migration_ptes(folio, folio, 0); rc = mapping->a_ops->writepage(&folio->page, &wbc); @@ -1098,7 +1152,7 @@ static void migrate_folio_undo_src(struct folio *src, struct list_head *ret) { if (page_was_mapped) - remove_migration_ptes(src, src, false); + remove_migration_ptes(src, src, 0); /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); @@ -1336,7 +1390,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, lru_add_drain(); if (old_page_state & PAGE_WAS_MAPPED) - remove_migration_ptes(src, dst, false); + remove_migration_ptes(src, dst, 0); out_unlock_both: folio_unlock(dst); @@ -1474,7 +1528,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio, if (page_was_mapped) remove_migration_ptes(src, - rc == MIGRATEPAGE_SUCCESS ? dst : src, false); + rc == MIGRATEPAGE_SUCCESS ? dst : src, 0); unlock_put_anon: folio_unlock(dst); diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 8d687de88a03..9cf26592ac93 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -424,7 +424,7 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns, continue; folio = page_folio(page); - remove_migration_ptes(folio, folio, false); + remove_migration_ptes(folio, folio, 0); src_pfns[i] = 0; folio_unlock(folio); @@ -840,7 +840,7 @@ void migrate_device_finalize(unsigned long *src_pfns, dst = src; } - remove_migration_ptes(src, dst, false); + remove_migration_ptes(src, dst, 0); folio_unlock(src); if (folio_is_zone_device(src))