From patchwork Tue Aug 13 12:02:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13761887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 214AAC52D7C for ; Tue, 13 Aug 2024 12:03:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 90ACD6B00A2; Tue, 13 Aug 2024 08:03:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8BDF66B00A4; Tue, 13 Aug 2024 08:03:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FE1D6B00A3; Tue, 13 Aug 2024 08:03:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 382176B00A1 for ; Tue, 13 Aug 2024 08:03:51 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id EB8E6120809 for ; Tue, 13 Aug 2024 12:03:50 +0000 (UTC) X-FDA: 82447088220.03.53E6A80 Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) by imf10.hostedemail.com (Postfix) with ESMTP id F0949C0031 for ; Tue, 13 Aug 2024 12:03:48 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BhiIX1Cu; spf=pass (imf10.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.181 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723550594; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ml/fx4q0/GpQb3pUk4RTjdvQZB9z9qmUFwqEb9M1LyE=; b=TIitCocOsWbtJg59vTGDVV2ieRuioyfacEVCJLfY+aYAnPTUErw8nirmbyud3osJdfodMD 8reCRUj8BmLCM9ARfnLCQ+VpSYLiDvyiVHHXxvyqiH6xSuksywOEcaS3UJm6aSGtDS3MGs Uwf5uQU8BG5GNaU6xoAw/JuoNuqnl6M= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BhiIX1Cu; spf=pass (imf10.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.181 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723550594; a=rsa-sha256; cv=none; b=Fb/LK5Qs90rVwXBw4++xQnyTsWvPE2NI7NXevlXUAzX/Xr4uJFxVY+Zfp06W/sgLpXfdSL h4VdzetQOb0ilw3Zstd87NOzLB2nxwiGQAGNScCx+1ViCNW6R571/L2eZeHGQT9hM+P8j4 6sTs8ewUwm1jQidShszgYH6DBhoV4+A= Received: by mail-qk1-f181.google.com with SMTP id af79cd13be357-7a4df9dc840so72733385a.2 for ; Tue, 13 Aug 2024 05:03:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723550628; x=1724155428; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ml/fx4q0/GpQb3pUk4RTjdvQZB9z9qmUFwqEb9M1LyE=; b=BhiIX1CugPrbnDrXNn1Sj9pMhSb9wm3LT9O5hd/WtMMO/8IJgosrXTKFbmxtLDlXaO u/MTPnjlAT7cVIvEtOJHp7PyaBVq8AZycrfXlmooiYEU7mViiK4bmQBWERxYSnjeNkLD srkH75lo+DN81BIO6wCzibUheNXGg6PQDRhB6N8SH2cfZzw1i3pVu3gNB5Sh89HwAYoq YhKCoJamAl3Q/sOJVGnapRdcrp+6xGv0WCkyS3z6CJlIohlZhPW9FdZK8pALSLezz2Fh +o8RRE7BBzWjGNJVUFVHDWC7Y6UQNc0kEoKqiOPDkCQ+uHqCGAaWaf0USSdAPCwkbDmk XG2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723550628; x=1724155428; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ml/fx4q0/GpQb3pUk4RTjdvQZB9z9qmUFwqEb9M1LyE=; b=FJLOq6nNVECNbhxLBlgkfRMmDMmhF8WzrfDlPhpq5u2a0mVqr62cVTKooJF0II+wQW XYA/M0zAz0gxlFoC0ozhbU+0+I7Tj7a2+h2E+2P8toOBYY8zbX51HY4XQt4hjjoHarup kwgol+1JxJaCbXTLMbUCHlo4bxRLblep0LOQZ1ypXU7m75npBvXk9CTOfXRqSU5vwyYU RQbv1fpC0DDsS350MeOz4MY0sYcYzFbyrK+PCx570X5rJDMn+3c1XvcZbuSTzCzngU1i 72WFVEyt94S3rUtJH4vQ/DcMyhLbi2Tt3Q3DaIi5hZ6YAVGDWqLWV9U/IHXcteWBCHyk t7Iw== X-Forwarded-Encrypted: i=1; AJvYcCUpeNeN1v/gy7NyJDTwTDdsrjvjTkFHmCKWQqpovDlVnUhHuv9arAeSnbF5sM8YOCsWtnSRfpFYG0ywC1uitrp2caI= X-Gm-Message-State: AOJu0YyAH4zqHEoiEdoWbHwxHk0uNWocelK900nybt9FUpra9DeU7pwr aB8ftJI+1DbtXywDIiqUyRv+Xz7/QZPsWO7eTHLSfuAbG8VYI21U X-Google-Smtp-Source: AGHT+IGnLAxIgNZTQh4vIeUHubCTk+9Li27aBYorSBwfeBTlQEsQV26mOZEo4xLzaw1PA1P3cAHmZQ== X-Received: by 2002:a05:620a:31a6:b0:7a3:7920:5500 with SMTP id af79cd13be357-7a4e15537c4mr389032385a.32.1723550627906; Tue, 13 Aug 2024 05:03:47 -0700 (PDT) Received: from localhost (fwdproxy-ash-006.fbsv.net. [2a03:2880:20ff:6::face:b00c]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a4c7d62678sm333062785a.5.2024.08.13.05.03.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Aug 2024 05:03:47 -0700 (PDT) From: Usama Arif To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, roman.gushchin@linux.dev, yuzhao@google.com, david@redhat.com, baohua@kernel.org, ryan.roberts@arm.com, rppt@kernel.org, willy@infradead.org, cerasuolodomenico@gmail.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kernel-team@meta.com, Usama Arif Subject: [PATCH v3 4/6] mm: Introduce a pageflag for partially mapped folios Date: Tue, 13 Aug 2024 13:02:47 +0100 Message-ID: <20240813120328.1275952-5-usamaarif642@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20240813120328.1275952-1-usamaarif642@gmail.com> References: <20240813120328.1275952-1-usamaarif642@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: F0949C0031 X-Stat-Signature: 6iw3apj84ka4oyot7iff5tnz1nsohtbh X-HE-Tag: 1723550628-291014 X-HE-Meta: U2FsdGVkX18HEBb9NdGu8fPtdud3TrenzKBzgIVX4DKEbL9Udj+axcwRE0WlTK2Zf58tCWRiRdigzrEH8EyBncJ0s/zs1XLUYhaXvL7fU8KlLrN0OTsEibLb+/xnBV6XP+H3uBQwWaOl/W88iM8mrPXGqmgLUZTIkfJZ2TvB7ZgHKtKaClH5X+QAkvSAy//LgpGO704/RVChJ/9lx8nw9KAMkfKqrcU9JbyG7a34ASNH1ZXVlLHvpaS4eLsz+p/2ljC29Iab3X9EBvaH4qRu6ejW5bmbXYRX+bq1XWsTQeQ0Fir3WPapG5ek/QGYJ1M5TeuPinUgYO7LmKGPLQBPDmKgMUCV+HxwPCVwCAroUkJFgaWyIGwoApbgnk2g3gbDt5cY0VyZH1HbogRRHMVPo64fPSenKugiueca5h6iPX7GQxEpdAEEaKbCiuuuQr0Y8/GUDy4d8ZFfFDZWRoy3M4UO1hemxHX54d7qa5ijqGhR6zGMmFIlU04SbhAfJKOPsptJiQfQvwvilS6VHPmwnXpafLzSHr9+80Sb85WNTSYm7iOQ8vNRRrrtwK9zvsGEB7gaXpkYXtxg68RrldbFy0fyltKv3H2o7KkfVCC+I6H0lb2TJs3fN/l9eZ1e/v1ueLGP2+zKOxN+Ivr5edLFd8LlVPrGy9I4vT22H8oVNXvmMllpvdy7+RFJcHbuEJMw/scRrBG7epBOcwM6W00uc5GtmCzuFo3CF5owAtE2gQXXrBEiQlPi1YYc/CpjUpdkJKNJRxLKJatFzlD5DDfbZNsCCX1fX+UYdJ/jGk2O2gdr5xi7aEJ7zu2Q7n9m4FgDt0R+AkFySVeGgBfffsy1tkaELVugH0dwoic55N4/W2QqBNXh92auvFW+7bbAuGUATlAPFvc+zeCmd/Oaj7WKg2Nf0u5SmNakfjot541LDL9FcWSQ/3Ju3yICR9ZPfvNO/5po2+VUejZq+gGT+3M cRPwTeir qoZNnbohL3uuORaIbLZl9IEoFEczm2y/zzzwrauPheRK1yNA4EQn/lHuPjO8wn5yRjESfE18J3JxvGMzu+FKTA8O5A2hxnLVAuzJqEojD4FTno2E100+OYOX/VHDg1HstrvHFQDjyqCPUWI9mvphWYUF7VygqPmOhaeosP3KqhzorQla2dwRPPvA3WoRUrQpqwrQ957yX0LcdqBP43Y5BdZEHTdwjLLihK5jqdC/GAQFUD0oE1+AALF0RSGI6LY2b9xOvOMpvghfqmlY8QDAU6D1K8hK4qk/jLZ4MxWPbiE0DyqTatxDxnE5QyD5nu4+A906IriQkeJ9MauejMfoqGbRZjA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently folio->_deferred_list is used to keep track of partially_mapped folios that are going to be split under memory pressure. In the next patch, all THPs that are faulted in and collapsed by khugepaged are also going to be tracked using _deferred_list. This patch introduces a pageflag to be able to distinguish between partially mapped folios and others in the deferred_list at split time in deferred_split_scan. Its needed as __folio_remove_rmap decrements _mapcount, _large_mapcount and _entire_mapcount, hence it won't be possible to distinguish between partially mapped folios and others in deferred_split_scan. Eventhough it introduces an extra flag to track if the folio is partially mapped, there is no functional change intended with this patch and the flag is not useful in this patch itself, it will become useful in the next patch when _deferred_list has non partially mapped folios. Signed-off-by: Usama Arif --- include/linux/huge_mm.h | 4 ++-- include/linux/page-flags.h | 3 +++ mm/huge_memory.c | 21 +++++++++++++-------- mm/hugetlb.c | 1 + mm/internal.h | 4 +++- mm/memcontrol.c | 3 ++- mm/migrate.c | 3 ++- mm/page_alloc.c | 5 +++-- mm/rmap.c | 3 ++- mm/vmscan.c | 3 ++- 10 files changed, 33 insertions(+), 17 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 4c32058cacfe..969f11f360d2 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -321,7 +321,7 @@ static inline int split_huge_page(struct page *page) { return split_huge_page_to_list_to_order(page, NULL, 0); } -void deferred_split_folio(struct folio *folio); +void deferred_split_folio(struct folio *folio, bool partially_mapped); void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, unsigned long address, bool freeze, struct folio *folio); @@ -495,7 +495,7 @@ static inline int split_huge_page(struct page *page) { return 0; } -static inline void deferred_split_folio(struct folio *folio) {} +static inline void deferred_split_folio(struct folio *folio, bool partially_mapped) {} #define split_huge_pmd(__vma, __pmd, __address) \ do { } while (0) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a0a29bd092f8..cecc1bad7910 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -182,6 +182,7 @@ enum pageflags { /* At least one page in this folio has the hwpoison flag set */ PG_has_hwpoisoned = PG_active, PG_large_rmappable = PG_workingset, /* anon or file-backed */ + PG_partially_mapped, /* was identified to be partially mapped */ }; #define PAGEFLAGS_MASK ((1UL << NR_PAGEFLAGS) - 1) @@ -861,8 +862,10 @@ static inline void ClearPageCompound(struct page *page) ClearPageHead(page); } FOLIO_FLAG(large_rmappable, FOLIO_SECOND_PAGE) +FOLIO_FLAG(partially_mapped, FOLIO_SECOND_PAGE) #else FOLIO_FLAG_FALSE(large_rmappable) +FOLIO_FLAG_FALSE(partially_mapped) #endif #define PG_head_mask ((1UL << PG_head)) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6df0e9f4f56c..c024ab0f745c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3397,6 +3397,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, * page_deferred_list. */ list_del_init(&folio->_deferred_list); + folio_clear_partially_mapped(folio); } spin_unlock(&ds_queue->split_queue_lock); if (mapping) { @@ -3453,11 +3454,12 @@ void __folio_undo_large_rmappable(struct folio *folio) if (!list_empty(&folio->_deferred_list)) { ds_queue->split_queue_len--; list_del_init(&folio->_deferred_list); + folio_clear_partially_mapped(folio); } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } -void deferred_split_folio(struct folio *folio) +void deferred_split_folio(struct folio *folio, bool partially_mapped) { struct deferred_split *ds_queue = get_deferred_split_queue(folio); #ifdef CONFIG_MEMCG @@ -3485,14 +3487,17 @@ void deferred_split_folio(struct folio *folio) if (folio_test_swapcache(folio)) return; - if (!list_empty(&folio->_deferred_list)) - return; - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); + if (partially_mapped) + folio_set_partially_mapped(folio); + else + folio_clear_partially_mapped(folio); if (list_empty(&folio->_deferred_list)) { - if (folio_test_pmd_mappable(folio)) - count_vm_event(THP_DEFERRED_SPLIT_PAGE); - count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED); + if (partially_mapped) { + if (folio_test_pmd_mappable(folio)) + count_vm_event(THP_DEFERRED_SPLIT_PAGE); + count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED); + } list_add_tail(&folio->_deferred_list, &ds_queue->split_queue); ds_queue->split_queue_len++; #ifdef CONFIG_MEMCG @@ -3541,6 +3546,7 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, } else { /* We lost race with folio_put() */ list_del_init(&folio->_deferred_list); + folio_clear_partially_mapped(folio); ds_queue->split_queue_len--; } if (!--sc->nr_to_scan) @@ -3558,7 +3564,6 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, next: folio_put(folio); } - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); list_splice_tail(&list, &ds_queue->split_queue); spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1fdd9eab240c..2ae2d9a18e40 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1758,6 +1758,7 @@ static void __update_and_free_hugetlb_folio(struct hstate *h, free_gigantic_folio(folio, huge_page_order(h)); } else { INIT_LIST_HEAD(&folio->_deferred_list); + folio_clear_partially_mapped(folio); folio_put(folio); } } diff --git a/mm/internal.h b/mm/internal.h index 52f7fc4e8ac3..d64546b8d377 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -662,8 +662,10 @@ static inline void prep_compound_head(struct page *page, unsigned int order) atomic_set(&folio->_entire_mapcount, -1); atomic_set(&folio->_nr_pages_mapped, 0); atomic_set(&folio->_pincount, 0); - if (order > 1) + if (order > 1) { INIT_LIST_HEAD(&folio->_deferred_list); + folio_clear_partially_mapped(folio); + } } static inline void prep_compound_tail(struct page *head, int tail_idx) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e1ffd2950393..0fd95daecf9a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4669,7 +4669,8 @@ static void uncharge_folio(struct folio *folio, struct uncharge_gather *ug) VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); VM_BUG_ON_FOLIO(folio_order(folio) > 1 && !folio_test_hugetlb(folio) && - !list_empty(&folio->_deferred_list), folio); + !list_empty(&folio->_deferred_list) && + folio_test_partially_mapped(folio), folio); /* * Nobody should be changing or seriously looking at diff --git a/mm/migrate.c b/mm/migrate.c index 3288ac041d03..6e32098ac2dc 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1734,7 +1734,8 @@ static int migrate_pages_batch(struct list_head *from, * use _deferred_list. */ if (nr_pages > 2 && - !list_empty(&folio->_deferred_list)) { + !list_empty(&folio->_deferred_list) && + folio_test_partially_mapped(folio)) { if (!try_split_folio(folio, split_folios, mode)) { nr_failed++; stats->nr_thp_failed += is_thp; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 408ef3d25cf5..a145c550dd2a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -957,8 +957,9 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) break; case 2: /* the second tail page: deferred_list overlaps ->mapping */ - if (unlikely(!list_empty(&folio->_deferred_list))) { - bad_page(page, "on deferred list"); + if (unlikely(!list_empty(&folio->_deferred_list) && + folio_test_partially_mapped(folio))) { + bad_page(page, "partially mapped folio on deferred list"); goto out; } break; diff --git a/mm/rmap.c b/mm/rmap.c index a6b9cd0b2b18..9ad558c2bad0 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1579,7 +1579,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, */ if (partially_mapped && folio_test_anon(folio) && list_empty(&folio->_deferred_list)) - deferred_split_folio(folio); + deferred_split_folio(folio, true); + __folio_mod_stat(folio, -nr, -nr_pmdmapped); /* diff --git a/mm/vmscan.c b/mm/vmscan.c index 25e43bb3b574..25f4e8403f41 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1233,7 +1233,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, * Split partially mapped folios right away. * We can free the unmapped pages without IO. */ - if (data_race(!list_empty(&folio->_deferred_list)) && + if (data_race(!list_empty(&folio->_deferred_list) && + folio_test_partially_mapped(folio)) && split_folio_to_list(folio, folio_list)) goto activate_locked; }