From patchwork Tue Aug 15 03:26:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13353477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA989EB64DD for ; Tue, 15 Aug 2023 03:26:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2718900017; Mon, 14 Aug 2023 23:26:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DD81490000B; Mon, 14 Aug 2023 23:26:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC643900017; Mon, 14 Aug 2023 23:26:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BEBD390000B for ; Mon, 14 Aug 2023 23:26:55 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 951C91A0376 for ; Tue, 15 Aug 2023 03:26:55 +0000 (UTC) X-FDA: 81124902390.15.4BC7824 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id 9E7C380020 for ; Tue, 15 Aug 2023 03:26:53 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=uE35NeTc; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692070013; a=rsa-sha256; cv=none; b=h+1n6MndM8hMAFIlZFmbUb0v+np4oZdmm22fY55b+bky5xgtkX4DnnyTn67FeY4DZ/vnuI ye/g7wbAxzH4HcF+7mt9BwxSvlS/aVK5dI5kZe6Q1pljAaWEeqTc1yODiJjaBLU3L5SDly ZBnG4U7SYpfHpfcW8MwHeIP2vALasyM= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=uE35NeTc; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692070013; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6nmvblvWXlPo+vgOoHsa5fU8BOXGI6xh6C5MkcfJnu8=; b=EYGt67YbUvWOCIHQ7LQX9VDkwJc32MFb0fQdjG+kasKncFgqeHttqY2obVYWdNADW9+gJO WC/62Z384cUYpsNYCR28a2UXKESR+PlfJCdl1uUdLCHALD2kV0mN46+cfQuMhWhXd0IUD2 nSh9FItgtP4fjrHFhOb7uS5vfYRDARc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=6nmvblvWXlPo+vgOoHsa5fU8BOXGI6xh6C5MkcfJnu8=; b=uE35NeTccLrzc2mwOmGSkD+XiV ESkGD0Ap1IZcCKQLYz2mTohMvFcEgsxpgxjLqQijm+MiUs6O6t7Dz++YKZkO3zCN7DTLYymhVBzlX la2f/xf12C6Qt9sP+kQfLCltasZurSSVUNT1xG5G5F2UoOvFnT+r6oLLZWJLecXHqObYfP/Kvof4C 3zs5KJ8MNCBF5v71mHMxn6VsbQ6yXBdnz0qlhhTTrszc51EGcix058lyUYtVmuJbPpafczORHF1Eq wqVEcqU6W7SzXOix8V82PlEbXhBLU2HLDZfMotkCC4B/GPpb3cRUm35GNx2nkcu+xrfJ3KO68dPB9 s2EKOwhA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVkhm-005qaV-Bq; Tue, 15 Aug 2023 03:26:50 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Jens Axboe , io-uring@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 3/9] mm: Call free_transhuge_folio() directly from destroy_large_folio() Date: Tue, 15 Aug 2023 04:26:39 +0100 Message-Id: <20230815032645.1393700-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230815032645.1393700-1-willy@infradead.org> References: <20230815032645.1393700-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9E7C380020 X-Stat-Signature: 6cyuhtzczgp74sbbhrkptmwfgxfb8auz X-HE-Tag: 1692070013-200057 X-HE-Meta: U2FsdGVkX1/dySl3Do+80FXZLslpFIr4k3JS9yFARHT6U9mK0QaxDLpqfaXpnD+DdNORtLNf0ZgospkijgGrqtcpIuvs+vQ/61WYFYy9reUpUGg9SMdCuFH3bObA5Yrtx/f8xdrJQaigq0xegLS7GN0CUDam0f0kuwiQn43TtFdaKTYvPcNp3RCTM89teoVtuN7XEHM5cNuLDcLPoMq4Reyh3RQCZdoj79z08dBn56JJ7kktOjodtoFi5qLfwft1QigkbB8H0zphNhoIrNBW7cnXVAVYfy9SeAMT3/rpvP5xIa7pdgs8QHgibQ4pfi4R7TOxaa5smIoBB1cvMtcRW3GJkP5x5EEwjuJqkFrbo7Oqmc/XzWYWn6I3wjRVzxn5FzK3KAN9xTWM/GIe7XpWsPqE2/oc3azxvFKBTY4Yq7rIorgX0yJdK4vmpdKvUd86QwgNcXFNCqHXU/IIcg9X/oU0xWaKZRfNCdzKea4AV+O+OmA+x4Sp6UUMkOWYQ+zydhEhWTLikKWx3tqqQJgj1rPjE9B2K0Dc8FpmQXPDSTxxj6XRXyiuN2GY0Gs1IXMFvh2u1mczMVZFOR1J1+Qq+goehe6gfPy5STzuXQ09L5F/oYdVezJbZeiKeXnTcsGz0tzo0y25cmY7wUxWa/lif5imsG2U0zVXK5PPe/vyufs0MV+7QGWr+sjefbAoz4v34w6iXmHrn0X+vNTyHLVNdq7wC74pfFcnKpKoQzEjWX4a+Ps1UaCE04xRE2FUwH3CugRjaXa7rt/1r1Na9OVWXSsf524E+zVG78xFyFPZCuNIQ9qywFhwiG+3SxEbgA0tqHIPmKuEr+wG0t3FZSaju6j4wu9qx6kAgRrwZyioDJ7WUeu26Nr6SZPkhlFOj0LKGs2/ZFccdv78+RU6RPbTfTGXZUVYxb55g307WmiJYiTItAWtGZiWmoHMFRDYyv6oWcDg8G+wRHlLlMqhveu SdbfRRkg cmb3Fs9P79+9YPgrPsW90MJTQUmVkotOsrBFIOw6FjqXxQqVkV9UVT6M6KOBhbjw8sRL1OONF2F1f++exn705485mC7DODAEVdUPIGPF+p4AY06oMnokQV3hr8/8qfyKm1cQZbfb9JPUCeaHMFxkvKxZ2qXzYnxRNHpLBKydDRDGs51hCeOXGs6jsyRXVc0icCwoM8ZOrIwJ1V4hHG4A+JN2rjJHlkfn9USWH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Indirect calls are expensive, thanks to Spectre. Convert this one to a direct call, and pass a folio instead of the head page for type safety. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/huge_mm.h | 2 +- mm/huge_memory.c | 5 ++--- mm/page_alloc.c | 8 +++++--- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 20284387b841..24aee49a581a 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -144,7 +144,7 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); void prep_transhuge_page(struct page *page); -void free_transhuge_page(struct page *page); +void free_transhuge_folio(struct folio *folio); bool can_split_folio(struct folio *folio, int *pextra_pins); int split_huge_page_to_list(struct page *page, struct list_head *list); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 8480728fa220..516fe3c26ef3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2779,9 +2779,8 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) return ret; } -void free_transhuge_page(struct page *page) +void free_transhuge_folio(struct folio *folio) { - struct folio *folio = (struct folio *)page; struct deferred_split *ds_queue = get_deferred_split_queue(folio); unsigned long flags; @@ -2798,7 +2797,7 @@ void free_transhuge_page(struct page *page) } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } - free_compound_page(page); + free_compound_page(&folio->page); } void deferred_split_folio(struct folio *folio) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1f67d4968590..feb2e95cf021 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -287,9 +287,6 @@ const char * const migratetype_names[MIGRATE_TYPES] = { static compound_page_dtor * const compound_page_dtors[NR_COMPOUND_DTORS] = { [NULL_COMPOUND_DTOR] = NULL, [COMPOUND_PAGE_DTOR] = free_compound_page, -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - [TRANSHUGE_PAGE_DTOR] = free_transhuge_page, -#endif }; int min_free_kbytes = 1024; @@ -624,6 +621,11 @@ void destroy_large_folio(struct folio *folio) return; } + if (folio_test_transhuge(folio) && dtor == TRANSHUGE_PAGE_DTOR) { + free_transhuge_folio(folio); + return; + } + VM_BUG_ON_FOLIO(dtor >= NR_COMPOUND_DTORS, folio); compound_page_dtors[dtor](&folio->page); }