From patchwork Wed Feb 26 12:03:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13992212 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08ED0C021BF for ; Wed, 26 Feb 2025 12:04:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A89E1280040; Wed, 26 Feb 2025 07:03:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A2E4528003F; Wed, 26 Feb 2025 07:03:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C35C280042; Wed, 26 Feb 2025 07:03:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 54AF028003F for ; Wed, 26 Feb 2025 07:03:57 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1A6711A13A7 for ; Wed, 26 Feb 2025 12:03:57 +0000 (UTC) X-FDA: 83161962114.29.BD73C4E Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf19.hostedemail.com (Postfix) with ESMTP id 437C41A000B for ; Wed, 26 Feb 2025 12:03:54 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf19.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740571435; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=m9YKYHmA3jceTDObJol10aeVRasWp5qci5u0K+W6z1I=; b=Q5syOiLDaEgd5LZa4HofnJ2l+JWbIRUFwKgFVnXRZEpXNuCvEWPKC6Jhenv5w6GBkap492 4WertU7l37cfcIgwb9SvFA1HZsqyBzSJirbgez7LGMJ24E8+F2qdv2ytBBtzlRtPpJ233f UvVo0Uj8jVK0bRyy75XQLPRyZ09dZHk= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf19.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740571435; a=rsa-sha256; cv=none; b=6fq2VCXJ6EhUc7KWQxA5hqzcFR+ETA37gWcneT6HlQPGQc8riuaOEXg6keCef45MciXqRO VhY7rfTbS2UcGXzCRqK0gnHL5hn9CFkngqcX6/5Uh6/2xe8eCBLEAKNq0QCfINyw7YgoMA 0YC0qmaVuIiXlBl17AXFPh+9fo9uy0A= X-AuditID: a67dfc5b-3e1ff7000001d7ae-61-67bf03238ab2 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, rjgolo@gmail.com Subject: [RFC PATCH v12 based on v6.14-rc4 22/25] mm/page_alloc: not allow to tlb shootdown if !preemptable() && non_luf_pages_ok() Date: Wed, 26 Feb 2025 21:03:33 +0900 Message-Id: <20250226120336.29565-22-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20250226120336.29565-1-byungchul@sk.com> References: <20250226113024.GA1935@system.software.com> <20250226120336.29565-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrKLMWRmVeSWpSXmKPExsXC9ZZnka4y8/50gxnnJS3mrF/DZvF5wz82 i6/rfzFbPP3Ux2JxedccNot7a/6zWpzftZbVYsfSfUwWlw4sYLI43nuAyWL+vc9sFps3TWW2 OD5lKqPF7x9z2Bz4PL639rF47Jx1l91jwaZSj80rtDw2repk89j0aRK7x7tz59g9Tsz4zeLx ft9VNo+tv+w8GqdeY/P4vEkugCeKyyYlNSezLLVI3y6BK+PM3WNMBV+sKiYvOcfWwPjOoIuR k0NCwETifPcONhi78fN3RhCbTUBd4saNn8wgtoiAmcTB1j/sXYxcHMwCy5gk9p5oYANxhAWa GSUe7GxmB6liEVCV2LDqE1gHL1BHe+t3Foip8hKrNxwAi3MCxT9NOwa2TUggWWLn7z9MIIMk BO6zSVyY2ckI0SApcXDFDZYJjLwLGBlWMQpl5pXlJmbmmOhlVOZlVugl5+duYgQG9rLaP9E7 GD9dCD7EKMDBqMTD++DM3nQh1sSy4srcQ4wSHMxKIrycmXvShXhTEiurUovy44tKc1KLDzFK c7AoifMafStPERJITyxJzU5NLUgtgskycXBKNTA6Jh2L+5d5aPMdMcF3RnlvfOfcqFK83Ndj cfbTv3ceX/5Mutws2Hld6PGHjS4nb7Myer4O47Es1olXrJp7aNJD/+67aw5qL/8m+oqzIu7d 8tlRabV6kVvOBwqdV1nIx37oo9/+xXq/Zx26cLmgXDSB7aeOU5/TdzXzC7INuxkCJm+6e6f5 4RYHJZbijERDLeai4kQAPohn22gCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrDLMWRmVeSWpSXmKPExsXC5WfdrKvMvD/d4Pk6AYs569ewWXze8I/N 4uv6X8wWTz/1sVgcnnuS1eLyrjlsFvfW/Ge1OL9rLavFjqX7mCwuHVjAZHG89wCTxfx7n9ks Nm+aymxxfMpURovfP+awOfB7fG/tY/HYOesuu8eCTaUem1doeWxa1cnmsenTJHaPd+fOsXuc mPGbxeP9vqtsHotffGDy2PrLzqNx6jU2j8+b5AJ4o7hsUlJzMstSi/TtErgyztw9xlTwxapi 8pJzbA2M7wy6GDk5JARMJBo/f2cEsdkE1CVu3PjJDGKLCJhJHGz9w97FyMXBLLCMSWLviQY2 EEdYoJlR4sHOZnaQKhYBVYkNqz6BdfACdbS3fmeBmCovsXrDAbA4J1D807RjbCC2kECyxM7f f5gmMHItYGRYxSiSmVeWm5iZY6pXnJ1RmZdZoZecn7uJERimy2r/TNzB+OWy+yFGAQ5GJR7e B2f2pguxJpYVV+YeYpTgYFYS4eXM3JMuxJuSWFmVWpQfX1Sak1p8iFGag0VJnNcrPDVBSCA9 sSQ1OzW1ILUIJsvEwSnVwBjbXBu5Qjt+3xPnwxxTyiYmnclbbXXK2dL8bGdChIijrjiXXSGD 4u6HrSl3c51eTWvqkVLUl4uaVJUoOnfV7SspE/ZdmtP7+KqbzqW33x6zvHRLcvNcm/ZvoVj9 TZuPJV+qw4KXfxQ5bfen8Oh6Y/djH54HKUgxCfv++5CbuqZmxR3T338XSCuxFGckGmoxFxUn AgA5z9K+TwIAAA== X-CFilter-Loop: Reflected X-Rspam-User: X-Stat-Signature: tn6bu6nofhdppbs46zoh4y6ynr1cbt57 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 437C41A000B X-HE-Tag: 1740571434-270048 X-HE-Meta: U2FsdGVkX1+t5u5jBkQhNehj1Fe7RvD3YROv/kJjUyjdcGem6Kyji8GIWcCJ3utFx285HiJtCOQtjRoFQnG0LkPgGSfe9Q5jqDrszaegqET6k+jtbPgWD4q67FUqEgzdGOanCF5hY6SZp59FgnmeeLQg/h2coJqKLKzu8u+ZRenSqPNKzI3COLkcx8YIkYvKupGFgVxl0iQlX3TqIqg85b9rkRnjHGqMWA6kMhfBRLKDDYEwRiDEbc5KiwGhaYqKWb4Riepnr3Ex2XJcL2uu/tdBCVacEGoeWX9hIsbpyBitjrEUr68zgLefw3KYtncHPIdcVHnwlZhgEf6Cza0CIBvzRXLfu9RIu79sPZqaJZkitFxQ5xIe1+N5Y0XbMjoZDCc22KW/DyLSg5TBLjI1O+ToibmKthsl48W4haovV013Kf7xJsXu312Kdf/6fS4hr+lNUxu4YO17N2C8fuBwCUqEy1VXxw15q5YV9ytu2y7LGcKYi9AFuRj+uXHhK6eckpeNSbLANRYweakv3te+9y4wPvh6ZB/FPqCDjxKNCuFcBOETxQ7J4yDzKA5v7KlF9vywtHjXuTy9f1T/Br9cdb1j50DmbRzs0OydFrti+srhm+bgFWAYgogH7LKZSdfwSMp76d4tsvdagztkZa1iUcdYC7UIK7kje6RdTIMnGKnUcuHH5HE/NO/wEcSzJm9evaWB64X0m25ohlUIK80avD9HdeWWiYV4qH6KDysGe3ykZTaiGcq8LwAoediE6SlRMQbxdMnCz0uQGHaue20iL/JN4D0fVDlTVwminVVqxJVpleA4aflmMRyn8Wj1V8Jdgym0UA01GXw8MtC6HVK3jHhYMshR7Gcq77uM4V8m8fy+Zti/QSaFI6FwHcq5glxwc9Yqz5ZxDS+Btrs3Hs7Reyp+IngaZQHXRn2bJEKXB46l2MDPCKVS4DaDE3Rkz3I+4VfbGnAfXZo9QsD4COv 3NHlJypA aZBnOp/jBpnKmGlNCh/HbJTxdpOOhFZXQu3bmJkVyCe9UBjMe8V383BXAll9AZWJpOt+FOp0tzE/j1Ak= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Do not perform tlb shootdown if the context is in preempt disable and there are already enough non luf pages, not to hurt preemptibility. Signed-off-by: Byungchul Park --- mm/compaction.c | 6 +++--- mm/internal.h | 5 +++-- mm/page_alloc.c | 27 +++++++++++++++------------ mm/page_isolation.c | 2 +- mm/page_reporting.c | 4 ++-- 5 files changed, 24 insertions(+), 20 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index b7a7a6feb9eac..aab400ec6a734 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -606,7 +606,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, page = pfn_to_page(blockpfn); - luf_takeoff_start(); + luf_takeoff_start(cc->zone); /* Isolate free pages. */ for (; blockpfn < end_pfn; blockpfn += stride, page += stride) { int isolated; @@ -1603,7 +1603,7 @@ static void fast_isolate_freepages(struct compact_control *cc) if (!area->nr_free) continue; - can_shootdown = luf_takeoff_start(); + can_shootdown = luf_takeoff_start(cc->zone); spin_lock_irqsave(&cc->zone->lock, flags); freelist = &area->free_list[MIGRATE_MOVABLE]; retry: @@ -2416,7 +2416,7 @@ static enum compact_result compact_finished(struct compact_control *cc) * luf_takeoff_{start,end}() is required to identify whether * this compaction context is tlb shootdownable for luf'd pages. */ - luf_takeoff_start(); + luf_takeoff_start(cc->zone); ret = __compact_finished(cc); luf_takeoff_end(cc->zone); diff --git a/mm/internal.h b/mm/internal.h index 6d7b3b389810e..b5f1928732498 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1591,7 +1591,7 @@ static inline void accept_page(struct page *page) #endif /* CONFIG_UNACCEPTED_MEMORY */ #if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) extern struct luf_batch luf_batch[]; -bool luf_takeoff_start(void); +bool luf_takeoff_start(struct zone *zone); void luf_takeoff_end(struct zone *zone); bool luf_takeoff_no_shootdown(void); bool luf_takeoff_check(struct zone *zone, struct page *page); @@ -1605,6 +1605,7 @@ static inline bool non_luf_pages_ok(struct zone *zone) return nr_free - nr_luf_pages > min_wm; } + unsigned short fold_unmap_luf(void); /* @@ -1691,7 +1692,7 @@ static inline bool can_luf_vma(struct vm_area_struct *vma) return true; } #else /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ -static inline bool luf_takeoff_start(void) { return false; } +static inline bool luf_takeoff_start(struct zone *zone) { return false; } static inline void luf_takeoff_end(struct zone *zone) {} static inline bool luf_takeoff_no_shootdown(void) { return true; } static inline bool luf_takeoff_check(struct zone *zone, struct page *page) { return true; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 917a257ea5706..2a2103df2d88e 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -623,22 +623,25 @@ compaction_capture(struct capture_control *capc, struct page *page, #endif /* CONFIG_COMPACTION */ #if defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) -static bool no_shootdown_context(void) +static bool no_shootdown_context(struct zone *zone) { /* - * If it performs with irq disabled, that might cause a deadlock. - * Avoid tlb shootdown in this case. + * Tries to avoid tlb shootdown if !preemptible(). However, it + * should be allowed under heavy memory pressure. */ + if (zone && non_luf_pages_ok(zone)) + return !(preemptible() && in_task()); + return !(!irqs_disabled() && in_task()); } /* * Can be called with zone lock released and irq enabled. */ -bool luf_takeoff_start(void) +bool luf_takeoff_start(struct zone *zone) { unsigned long flags; - bool no_shootdown = no_shootdown_context(); + bool no_shootdown = no_shootdown_context(zone); local_irq_save(flags); @@ -2591,7 +2594,7 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, * luf_takeoff_{start,end}() is required for * get_page_from_free_area() to use luf_takeoff_check(). */ - luf_takeoff_start(); + luf_takeoff_start(zone); spin_lock_irqsave(&zone->lock, flags); for (order = 0; order < NR_PAGE_ORDERS; order++) { struct free_area *area = &(zone->free_area[order]); @@ -2796,7 +2799,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, unsigned long flags; int i; - luf_takeoff_start(); + luf_takeoff_start(zone); spin_lock_irqsave(&zone->lock, flags); for (i = 0; i < count; ++i) { struct page *page = __rmqueue(zone, order, migratetype, @@ -3422,7 +3425,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, do { page = NULL; - luf_takeoff_start(); + luf_takeoff_start(zone); spin_lock_irqsave(&zone->lock, flags); if (alloc_flags & ALLOC_HIGHATOMIC) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); @@ -3567,7 +3570,7 @@ static struct page *rmqueue_pcplist(struct zone *preferred_zone, struct page *page; unsigned long __maybe_unused UP_flags; - luf_takeoff_start(); + luf_takeoff_start(NULL); /* spin_trylock may fail due to a parallel drain or IRQ reentrancy. */ pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); @@ -5190,7 +5193,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, if (unlikely(!zone)) goto failed; - luf_takeoff_start(); + luf_takeoff_start(NULL); /* spin_trylock may fail due to a parallel drain or IRQ reentrancy. */ pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); @@ -7294,7 +7297,7 @@ unsigned long __offline_isolated_pages(unsigned long start_pfn, offline_mem_sections(pfn, end_pfn); zone = page_zone(pfn_to_page(pfn)); - luf_takeoff_start(); + luf_takeoff_start(zone); spin_lock_irqsave(&zone->lock, flags); while (pfn < end_pfn) { page = pfn_to_page(pfn); @@ -7412,7 +7415,7 @@ bool take_page_off_buddy(struct page *page) unsigned int order; bool ret = false; - luf_takeoff_start(); + luf_takeoff_start(zone); spin_lock_irqsave(&zone->lock, flags); for (order = 0; order < NR_PAGE_ORDERS; order++) { struct page *page_head = page - (pfn & ((1 << order) - 1)); diff --git a/mm/page_isolation.c b/mm/page_isolation.c index c34659b58ca6c..f4055c0a2ea89 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -211,7 +211,7 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) struct page *buddy; zone = page_zone(page); - luf_takeoff_start(); + luf_takeoff_start(zone); spin_lock_irqsave(&zone->lock, flags); if (!is_migrate_isolate_page(page)) goto out; diff --git a/mm/page_reporting.c b/mm/page_reporting.c index b23d3ed34ec07..83b66e7f0d257 100644 --- a/mm/page_reporting.c +++ b/mm/page_reporting.c @@ -170,7 +170,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone, if (free_area_empty(area, mt)) return err; - can_shootdown = luf_takeoff_start(); + can_shootdown = luf_takeoff_start(zone); spin_lock_irq(&zone->lock); /* @@ -250,7 +250,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone, /* update budget to reflect call to report function */ budget--; - luf_takeoff_start(); + luf_takeoff_start(zone); /* reacquire zone lock and resume processing */ spin_lock_irq(&zone->lock);