From patchwork Tue Aug 15 03:26:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13353482 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC2D8C04A6A for ; Tue, 15 Aug 2023 03:27:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7710290001D; Mon, 14 Aug 2023 23:27:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 74BB790000B; Mon, 14 Aug 2023 23:27:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6102A90001D; Mon, 14 Aug 2023 23:27:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5608E90000B for ; Mon, 14 Aug 2023 23:27:10 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 26603B1C1C for ; Tue, 15 Aug 2023 03:27:10 +0000 (UTC) X-FDA: 81124903020.30.ADF2BA7 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 7B2D11C0006 for ; Tue, 15 Aug 2023 03:27:08 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=GU31PFG+; dmarc=none; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692070028; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LklJPoQHz5gx0kkx3SxtlRYJZMYAdukfOsnGQ5XVKy4=; b=3OBvhYE38FG0+Uf/0chcyn9MIKjXVxAruLggzymqrZK54Kj9v5fJMZ9hFyOdj+8yyE8XHt 2/JYMznTBox9Y3lvygPPPQCou+7FEXVbF3bkb0Maamro6CWYCMaH67bF7s6q+k1MB+jIb3 J6kvG2oxWMgHvpSHPOxXFsq8EWr0TeU= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=GU31PFG+; dmarc=none; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692070028; a=rsa-sha256; cv=none; b=7qxrAqPasMHzPC6E+Nwqqd16G/oyci7+0x89RqmqSY+LPhvHlQrpwrP9O0aUe65faqbo/2 d7FvDZKkYM9tQV8gwRu8E/vK45eYBjK5TVagWWhAjfdguyqn9KjOaXeXcUWu8zqeLID8MU AA3eEODXzsm3jGlK3IFTSXZbRn26U/0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=LklJPoQHz5gx0kkx3SxtlRYJZMYAdukfOsnGQ5XVKy4=; b=GU31PFG+QORyW43Si703UzIVka mjKR/IuX+RvLygPYzwjw6rTHsyBo9ofDiCRQAP+DqpvYkUBes1SHvpYqwhbc1EQWayzS+9R+57E5d sX8G9JDumR6p4MmIoYneqnekW9XCf1UnRVlhXS4NIu5luZeraaL005PAFQh+BDLrisg6Wgb6BeBOR oQsB1Euafhr7n49QZ+PAcpaAHYUqxE0bdhrJ6xrZxezyUmxbTKaErzLHGhuSWf6FsI3CTeLKAQHNz VivihZYRv6iRluHL6MI0fPb3p7DMD2x+GnoI18HsrNBbK+CzxYG/tXYkPRzpMQR5tPMNPxGWoK8EZ +mMRKj9g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVkhm-005qaR-5N; Tue, 15 Aug 2023 03:26:50 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Jens Axboe , io-uring@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 1/9] io_uring: Stop calling free_compound_page() Date: Tue, 15 Aug 2023 04:26:37 +0100 Message-Id: <20230815032645.1393700-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230815032645.1393700-1-willy@infradead.org> References: <20230815032645.1393700-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 4p754d4tspkzwaxdm6hubnko5cd3kxrk X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 7B2D11C0006 X-HE-Tag: 1692070028-181820 X-HE-Meta: U2FsdGVkX18xXYG96iSkil6Rw72U8WbxiMYDZ3B/mUV/ecAiGjgjE62DTMF1Z3yhKuqahiHPHUbvKeaRRccRVKQDYufME0oWcwTF1A6j39LHsBdgE7OsnW+8ng7E8M3PUNUktEeK4T7EqF9KwNCgCCtt/JtmBzuuN0lfiK2Hhvkmr9+o0sN0w7toquB0fDkR7ikvgs47Sg0niOjf8AGahczbJaNKvY/TcbYYTL0+7P9ynMyWNxQX7PNxje3pZiaWuD1fw9pYJsEV4keo8GDx52HA/HHXSAl3/GlE0y4wgUcIvTqkuHQ4MlFrDwH6Ve1G7iT5iYAQoWC4VHGzuen+GXT+2aTopwmYKJE14T6wC/V2b5jODBuap94oy9paIvAm5IZDVOB+eb6lo81vLHIidKUD85ZeLWkytUj1TsSGy9wuUWiAwhsTbQY+2XL0Q/xEOxd8UOTg+zhWbuBnYqfadTcrhtxKnHmTQ116WkHvSCzsf9zWNz4l3+aA5M6pMDIs5qcfladUg5oMrI2qFnprzMOrmrir11R+FjtTvjpdvjRu0oczbQic2sBaKj09l6P1ZDiinS9J/xHOCObdTmJD+2XN2AvvBdEBPpvICsj3TbSsHCr0WXNbrXW2P2zQIBJAUyz7xIkNKNmbAUYD2ejghRblVfkG2o/uuPg+cLMrCQbKh5CBgPJ89+3a1FU5EFpJEB20le2GywoaYF7u3IY1H8UYdO0vfOnBD7CFQDQezCpvMN0JA0/c2bmR/K0u8NpqrOKKwdtNahvMnJte25sTts0LrQtrBBujGYwH8uaQj2cqKZBW5jrcgMSa2ELzm0e2qtxLwQX3cjvYinJixir/MpXdrJp/qYVj6fFEQnzXhZ5yjwE4fpoJBTeBQhY1uBqvAn7boPohJyOrC3KzE7eD4/YHGD7BJgINYpV5I1o0hfCGI3tDWbUoQ0q69KoHUSZ8oTq+tBkPVsMWrhyah0H ywwUdUIr IuKuWJZu/1DTHxEjwGvP0to/dCRW+qOx0wpS21LSQDzpBwmrTp0wtmfju4sifBUD8g/fHUmyvzmJV4el6/6q8B1fqi1s3KweP0/jC356BeUl2cXn+P9345z+loQj8ZF1myw2vhTh1TL2718GyKwlmJMSxV79mF4BSI/c2EPeUaMmTYMbGSpFbXwvmGuFWxvqP4Gx17uyhmU18USpGxYuKMwjGlV2TucA5fm6x X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: folio_put() is the standard way to write this, and it's not appreciably slower. This is an enabling patch for removing free_compound_page() entirely. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: David Hildenbrand Reviewed-by: Jens Axboe --- io_uring/io_uring.c | 6 +----- io_uring/kbuf.c | 6 +----- 2 files changed, 2 insertions(+), 10 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index dcf5fc7d2820..a5b9b5de7aff 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2664,14 +2664,10 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, static void io_mem_free(void *ptr) { - struct page *page; - if (!ptr) return; - page = virt_to_head_page(ptr); - if (put_page_testzero(page)) - free_compound_page(page); + folio_put(virt_to_folio(ptr)); } static void io_pages_free(struct page ***pages, int npages) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 2f0181521c98..556f4df25b0f 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -218,11 +218,7 @@ static int __io_remove_buffers(struct io_ring_ctx *ctx, if (bl->is_mapped) { i = bl->buf_ring->tail - bl->head; if (bl->is_mmap) { - struct page *page; - - page = virt_to_head_page(bl->buf_ring); - if (put_page_testzero(page)) - free_compound_page(page); + folio_put(virt_to_folio(bl->buf_ring)); bl->buf_ring = NULL; bl->is_mmap = 0; } else if (bl->buf_nr_pages) { From patchwork Tue Aug 15 03:26:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13353478 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F228C001DB for ; Tue, 15 Aug 2023 03:26:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2DD41900019; Mon, 14 Aug 2023 23:26:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 28D4D90000B; Mon, 14 Aug 2023 23:26:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17BEE900019; Mon, 14 Aug 2023 23:26:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 087BD90000B for ; Mon, 14 Aug 2023 23:26:59 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D5BDD1A03AE for ; Tue, 15 Aug 2023 03:26:58 +0000 (UTC) X-FDA: 81124902516.05.442FD32 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id D8955140012 for ; Tue, 15 Aug 2023 03:26:56 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=vFZnTKXS; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692070017; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uE1o3xYTfNoT/OXe1q8MbroOJ3TtLR0fpcc8smSdFiQ=; b=RlDP/qLRV1/Gvknz8OG/QPR8S6s9vMkr2mUpgXOWrUME3ALtQJgy5lYSURIYlk2IGYybAF xodTfRimjYyFjS0ghRaTta4od+jtP/8248Dfj1qX5tkJ2qPNEAvnEjlyNpXFRb3tMoWjET QAC/5O95+hmSOSYvrgvMfe8nzf9IcRs= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=vFZnTKXS; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692070017; a=rsa-sha256; cv=none; b=zCvF3LtEWNW8vho4TnBKGuW6ZM4SOSjvjdJzSdhlx8KftytzCg/rAjohB3eCOLceCRaMLt N6bEgHTW/9MkuHmtZbZKWNaxB4e92KxBbB4wyonvoLN8c89jnOQ2h29wu6Keljlqc1v0yA aSwbwGqVdT7EcwnZFZO1JjVCQwlyLu8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=uE1o3xYTfNoT/OXe1q8MbroOJ3TtLR0fpcc8smSdFiQ=; b=vFZnTKXSmWpfKeIh2BcvGDa7T3 HsNqT/kyBENsHsnvkcr+2U5F4CV6/WHN4UBv9jaDxFO7aaTEbBqJnkXv8nLS9xR/wQf1L2jpaS3/8 bz6mdqZ6LwPyTRb8PyuYWf/EOkpnsd3AFC0iaq9T9t23WbjukvSDJACX/xrjl3CnP1sgIfzl7hkVe J4deXtWhnatjTXkkMLTrxZg5ZHHanaI8y9PNwt142O/LmcypHAHfrDRH2ADYu2OErqmm6Md1++gXP bL13ExJQlZs1MWyz5PcNN3ggmfVNs4fw56j+MynJKpFh/3wte4odcC3GpWQBdFrbHhlXGH75gK6mH pCwRfLaw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVkhm-005qaT-8c; Tue, 15 Aug 2023 03:26:50 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Jens Axboe , io-uring@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 2/9] mm: Call the hugetlb destructor directly Date: Tue, 15 Aug 2023 04:26:38 +0100 Message-Id: <20230815032645.1393700-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230815032645.1393700-1-willy@infradead.org> References: <20230815032645.1393700-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D8955140012 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: f6113gpm58tq3ep41m1ntmsfu47nbjk1 X-HE-Tag: 1692070016-424502 X-HE-Meta: U2FsdGVkX1+En4p8HK08pf0k0MQL4wR9u75zkTtmRHhr1A1oYybRZxPSXvRB0xSMpuX1iHG5uBBuEeKtBP9f6PCgPam0vm00bRrVzgsz/1slDAqqcmijScxvQlKcs2dmtfjGzzQP0hpId/efTtLuoO76XDSCoKu/nmx/faPre2oQzPwG6d9VlOMA0kmMUls3e/P+OQdIWGFKvGtCGVjEqpNBZL55c0tBwE5rTcnoGeo4DnsnMOaq6Ai4OaFLEFk1ksIbYC4sf2/MpLyI8BrEgCDhzOTCLraSc6mPUbBohnAz6Z4in8h/z4F4Ff7gXrzwCfThPN+/NcDK2TTKVtzSXjT7hCOVsFkUU2RdJBw8uTLfs2wqUCx6vEiDl9YkxpcnEGfP4sIhvJFksMTWExuL0n3kPhNKXMnRR8v9s6XStqsf33nUtaUc7rARVD2C92zrMkRI/d0r1arYPCQtCoFQDpUE6ewKLCX5msCIZmxWNPlIc6G39i/WNNwy+ayXhjm46Ae0/I5Y0vpczSdPBgj61fvuJN9dt9fLcqEzrtcrjImhavm6wr3zxxA8ZfLwa4RB6z7Tkk57AEuEmPOB2LQ5ZKwyeAcFONW8HiNPn/kenVelrWdRDK8MeZ1TckoOyVWVPxWHQSZ3UiKQz0/iPsvejYJM1Xwxy2MzS6s1jHHUCYEcEzML4MWxTTgxoh93pRJic9K7k+Ouri8HIbkfpQhj5bfWvaZwIdBHs2eNRAbLlxWWqMHoGSGAh4mOwi6A8m82C2MkmH9lHLCvGKvf1YlALIbQjbueP8BaZBV6BfZR+gveoJZO6POwXS8TeEfJ88fX299qzWBt67U7sU3XT+INywyWChqCzW/RxtR7OrKCjb+S6YzWG091IgoSZ7lk2aS0x76mFS1XBqpT2ULP9uJKrk3JzM/CMiR5UgEWUA1cedC2KMXAuK1SDCb8+GQaCqAGRSlJulfHGUCvyWEtAQQ 8ZzRvI7Z QQV4V7wyIUSdzjuwpaGYBr5vyFNoOJ21kvCiLeRcRixCluLxKNLtFP2vm3GoyfRoAaqsB9xf12ewc5AOG5vjRMMEoHCrRWS8cRCGQbKZwXzLcGuQzRsxDt3mVGmuNVRb4C7rUjNPIP7QOP0TsjfaFhrYUHS+AZYj9pmIg+DojUJNLEpUomNl4mN8dIQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Indirect calls are expensive, thanks to Spectre. Convert this one to a direct call, and pass a folio instead of the head page to save a few more instructions. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/hugetlb.h | 3 ++- include/linux/mm.h | 6 +----- mm/hugetlb.c | 26 ++++++++++++-------------- mm/page_alloc.c | 8 +++++--- 4 files changed, 20 insertions(+), 23 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 0a393bc02f25..9555859537a3 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -26,6 +26,8 @@ typedef struct { unsigned long pd; } hugepd_t; #define __hugepd(x) ((hugepd_t) { (x) }) #endif +void free_huge_page(struct folio *folio); + #ifdef CONFIG_HUGETLB_PAGE #include @@ -165,7 +167,6 @@ int get_huge_page_for_hwpoison(unsigned long pfn, int flags, bool *migratable_cleared); void folio_putback_active_hugetlb(struct folio *folio); void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int reason); -void free_huge_page(struct page *page); void hugetlb_fix_reserve_counts(struct inode *inode); extern struct mutex *hugetlb_fault_mutex_table; u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx); diff --git a/include/linux/mm.h b/include/linux/mm.h index 19493d6a2bb8..7fb529dbff31 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1278,13 +1278,9 @@ typedef void compound_page_dtor(struct page *); enum compound_dtor_id { NULL_COMPOUND_DTOR, COMPOUND_PAGE_DTOR, -#ifdef CONFIG_HUGETLB_PAGE HUGETLB_PAGE_DTOR, -#endif -#ifdef CONFIG_TRANSPARENT_HUGEPAGE TRANSHUGE_PAGE_DTOR, -#endif - NR_COMPOUND_DTORS, + NR_COMPOUND_DTORS }; static inline void folio_set_compound_dtor(struct folio *folio, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e327a5a7602c..bc340f5dbbd4 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1875,13 +1875,12 @@ struct hstate *size_to_hstate(unsigned long size) return NULL; } -void free_huge_page(struct page *page) +void free_huge_page(struct folio *folio) { /* * Can't pass hstate in here because it is called from the * compound page destructor. */ - struct folio *folio = page_folio(page); struct hstate *h = folio_hstate(folio); int nid = folio_nid(folio); struct hugepage_subpool *spool = hugetlb_folio_subpool(folio); @@ -1936,7 +1935,7 @@ void free_huge_page(struct page *page) spin_unlock_irqrestore(&hugetlb_lock, flags); update_and_free_hugetlb_folio(h, folio, true); } else { - arch_clear_hugepage_flags(page); + arch_clear_hugepage_flags(&folio->page); enqueue_hugetlb_folio(h, folio); spin_unlock_irqrestore(&hugetlb_lock, flags); } @@ -2246,7 +2245,7 @@ static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, folio = alloc_fresh_hugetlb_folio(h, gfp_mask, node, nodes_allowed, node_alloc_noretry); if (folio) { - free_huge_page(&folio->page); /* free it into the hugepage allocator */ + free_huge_page(folio); /* free it into the hugepage allocator */ return 1; } } @@ -2435,7 +2434,7 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h, if (h->surplus_huge_pages >= h->nr_overcommit_huge_pages) { folio_set_hugetlb_temporary(folio); spin_unlock_irq(&hugetlb_lock); - free_huge_page(&folio->page); + free_huge_page(folio); return NULL; } @@ -2547,8 +2546,7 @@ static int gather_surplus_pages(struct hstate *h, long delta) __must_hold(&hugetlb_lock) { LIST_HEAD(surplus_list); - struct folio *folio; - struct page *page, *tmp; + struct folio *folio, *tmp; int ret; long i; long needed, allocated; @@ -2608,11 +2606,11 @@ static int gather_surplus_pages(struct hstate *h, long delta) ret = 0; /* Free the needed pages to the hugetlb pool */ - list_for_each_entry_safe(page, tmp, &surplus_list, lru) { + list_for_each_entry_safe(folio, tmp, &surplus_list, lru) { if ((--needed) < 0) break; /* Add the page to the hugetlb allocator */ - enqueue_hugetlb_folio(h, page_folio(page)); + enqueue_hugetlb_folio(h, folio); } free: spin_unlock_irq(&hugetlb_lock); @@ -2621,8 +2619,8 @@ static int gather_surplus_pages(struct hstate *h, long delta) * Free unnecessary surplus pages to the buddy allocator. * Pages have no ref count, call free_huge_page directly. */ - list_for_each_entry_safe(page, tmp, &surplus_list, lru) - free_huge_page(page); + list_for_each_entry_safe(folio, tmp, &surplus_list, lru) + free_huge_page(folio); spin_lock_irq(&hugetlb_lock); return ret; @@ -3232,7 +3230,7 @@ static void __init gather_bootmem_prealloc(void) if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { WARN_ON(folio_test_reserved(folio)); prep_new_hugetlb_folio(h, folio, folio_nid(folio)); - free_huge_page(page); /* add to the hugepage allocator */ + free_huge_page(folio); /* add to the hugepage allocator */ } else { /* VERY unlikely inflated ref count on a tail page */ free_gigantic_folio(folio, huge_page_order(h)); @@ -3264,7 +3262,7 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) &node_states[N_MEMORY], NULL); if (!folio) break; - free_huge_page(&folio->page); /* free it into the hugepage allocator */ + free_huge_page(folio); /* free it into the hugepage allocator */ } cond_resched(); } @@ -3658,7 +3656,7 @@ static int demote_free_hugetlb_folio(struct hstate *h, struct folio *folio) prep_compound_page(subpage, target_hstate->order); folio_change_private(inner_folio, NULL); prep_new_hugetlb_folio(target_hstate, inner_folio, nid); - free_huge_page(subpage); + free_huge_page(inner_folio); } mutex_unlock(&target_hstate->resize_lock); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8fe9ff917850..1f67d4968590 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -287,9 +287,6 @@ const char * const migratetype_names[MIGRATE_TYPES] = { static compound_page_dtor * const compound_page_dtors[NR_COMPOUND_DTORS] = { [NULL_COMPOUND_DTOR] = NULL, [COMPOUND_PAGE_DTOR] = free_compound_page, -#ifdef CONFIG_HUGETLB_PAGE - [HUGETLB_PAGE_DTOR] = free_huge_page, -#endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE [TRANSHUGE_PAGE_DTOR] = free_transhuge_page, #endif @@ -622,6 +619,11 @@ void destroy_large_folio(struct folio *folio) { enum compound_dtor_id dtor = folio->_folio_dtor; + if (folio_test_hugetlb(folio)) { + free_huge_page(folio); + return; + } + VM_BUG_ON_FOLIO(dtor >= NR_COMPOUND_DTORS, folio); compound_page_dtors[dtor](&folio->page); } From patchwork Tue Aug 15 03:26:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13353477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA989EB64DD for ; Tue, 15 Aug 2023 03:26:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2718900017; Mon, 14 Aug 2023 23:26:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DD81490000B; Mon, 14 Aug 2023 23:26:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC643900017; Mon, 14 Aug 2023 23:26:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BEBD390000B for ; Mon, 14 Aug 2023 23:26:55 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 951C91A0376 for ; Tue, 15 Aug 2023 03:26:55 +0000 (UTC) X-FDA: 81124902390.15.4BC7824 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id 9E7C380020 for ; Tue, 15 Aug 2023 03:26:53 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=uE35NeTc; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692070013; a=rsa-sha256; cv=none; b=h+1n6MndM8hMAFIlZFmbUb0v+np4oZdmm22fY55b+bky5xgtkX4DnnyTn67FeY4DZ/vnuI ye/g7wbAxzH4HcF+7mt9BwxSvlS/aVK5dI5kZe6Q1pljAaWEeqTc1yODiJjaBLU3L5SDly ZBnG4U7SYpfHpfcW8MwHeIP2vALasyM= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=uE35NeTc; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692070013; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6nmvblvWXlPo+vgOoHsa5fU8BOXGI6xh6C5MkcfJnu8=; b=EYGt67YbUvWOCIHQ7LQX9VDkwJc32MFb0fQdjG+kasKncFgqeHttqY2obVYWdNADW9+gJO WC/62Z384cUYpsNYCR28a2UXKESR+PlfJCdl1uUdLCHALD2kV0mN46+cfQuMhWhXd0IUD2 nSh9FItgtP4fjrHFhOb7uS5vfYRDARc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=6nmvblvWXlPo+vgOoHsa5fU8BOXGI6xh6C5MkcfJnu8=; b=uE35NeTccLrzc2mwOmGSkD+XiV ESkGD0Ap1IZcCKQLYz2mTohMvFcEgsxpgxjLqQijm+MiUs6O6t7Dz++YKZkO3zCN7DTLYymhVBzlX la2f/xf12C6Qt9sP+kQfLCltasZurSSVUNT1xG5G5F2UoOvFnT+r6oLLZWJLecXHqObYfP/Kvof4C 3zs5KJ8MNCBF5v71mHMxn6VsbQ6yXBdnz0qlhhTTrszc51EGcix058lyUYtVmuJbPpafczORHF1Eq wqVEcqU6W7SzXOix8V82PlEbXhBLU2HLDZfMotkCC4B/GPpb3cRUm35GNx2nkcu+xrfJ3KO68dPB9 s2EKOwhA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVkhm-005qaV-Bq; Tue, 15 Aug 2023 03:26:50 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Jens Axboe , io-uring@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 3/9] mm: Call free_transhuge_folio() directly from destroy_large_folio() Date: Tue, 15 Aug 2023 04:26:39 +0100 Message-Id: <20230815032645.1393700-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230815032645.1393700-1-willy@infradead.org> References: <20230815032645.1393700-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9E7C380020 X-Stat-Signature: 6cyuhtzczgp74sbbhrkptmwfgxfb8auz X-HE-Tag: 1692070013-200057 X-HE-Meta: U2FsdGVkX1/dySl3Do+80FXZLslpFIr4k3JS9yFARHT6U9mK0QaxDLpqfaXpnD+DdNORtLNf0ZgospkijgGrqtcpIuvs+vQ/61WYFYy9reUpUGg9SMdCuFH3bObA5Yrtx/f8xdrJQaigq0xegLS7GN0CUDam0f0kuwiQn43TtFdaKTYvPcNp3RCTM89teoVtuN7XEHM5cNuLDcLPoMq4Reyh3RQCZdoj79z08dBn56JJ7kktOjodtoFi5qLfwft1QigkbB8H0zphNhoIrNBW7cnXVAVYfy9SeAMT3/rpvP5xIa7pdgs8QHgibQ4pfi4R7TOxaa5smIoBB1cvMtcRW3GJkP5x5EEwjuJqkFrbo7Oqmc/XzWYWn6I3wjRVzxn5FzK3KAN9xTWM/GIe7XpWsPqE2/oc3azxvFKBTY4Yq7rIorgX0yJdK4vmpdKvUd86QwgNcXFNCqHXU/IIcg9X/oU0xWaKZRfNCdzKea4AV+O+OmA+x4Sp6UUMkOWYQ+zydhEhWTLikKWx3tqqQJgj1rPjE9B2K0Dc8FpmQXPDSTxxj6XRXyiuN2GY0Gs1IXMFvh2u1mczMVZFOR1J1+Qq+goehe6gfPy5STzuXQ09L5F/oYdVezJbZeiKeXnTcsGz0tzo0y25cmY7wUxWa/lif5imsG2U0zVXK5PPe/vyufs0MV+7QGWr+sjefbAoz4v34w6iXmHrn0X+vNTyHLVNdq7wC74pfFcnKpKoQzEjWX4a+Ps1UaCE04xRE2FUwH3CugRjaXa7rt/1r1Na9OVWXSsf524E+zVG78xFyFPZCuNIQ9qywFhwiG+3SxEbgA0tqHIPmKuEr+wG0t3FZSaju6j4wu9qx6kAgRrwZyioDJ7WUeu26Nr6SZPkhlFOj0LKGs2/ZFccdv78+RU6RPbTfTGXZUVYxb55g307WmiJYiTItAWtGZiWmoHMFRDYyv6oWcDg8G+wRHlLlMqhveu SdbfRRkg cmb3Fs9P79+9YPgrPsW90MJTQUmVkotOsrBFIOw6FjqXxQqVkV9UVT6M6KOBhbjw8sRL1OONF2F1f++exn705485mC7DODAEVdUPIGPF+p4AY06oMnokQV3hr8/8qfyKm1cQZbfb9JPUCeaHMFxkvKxZ2qXzYnxRNHpLBKydDRDGs51hCeOXGs6jsyRXVc0icCwoM8ZOrIwJ1V4hHG4A+JN2rjJHlkfn9USWH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Indirect calls are expensive, thanks to Spectre. Convert this one to a direct call, and pass a folio instead of the head page for type safety. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/huge_mm.h | 2 +- mm/huge_memory.c | 5 ++--- mm/page_alloc.c | 8 +++++--- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 20284387b841..24aee49a581a 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -144,7 +144,7 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); void prep_transhuge_page(struct page *page); -void free_transhuge_page(struct page *page); +void free_transhuge_folio(struct folio *folio); bool can_split_folio(struct folio *folio, int *pextra_pins); int split_huge_page_to_list(struct page *page, struct list_head *list); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 8480728fa220..516fe3c26ef3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2779,9 +2779,8 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) return ret; } -void free_transhuge_page(struct page *page) +void free_transhuge_folio(struct folio *folio) { - struct folio *folio = (struct folio *)page; struct deferred_split *ds_queue = get_deferred_split_queue(folio); unsigned long flags; @@ -2798,7 +2797,7 @@ void free_transhuge_page(struct page *page) } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } - free_compound_page(page); + free_compound_page(&folio->page); } void deferred_split_folio(struct folio *folio) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1f67d4968590..feb2e95cf021 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -287,9 +287,6 @@ const char * const migratetype_names[MIGRATE_TYPES] = { static compound_page_dtor * const compound_page_dtors[NR_COMPOUND_DTORS] = { [NULL_COMPOUND_DTOR] = NULL, [COMPOUND_PAGE_DTOR] = free_compound_page, -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - [TRANSHUGE_PAGE_DTOR] = free_transhuge_page, -#endif }; int min_free_kbytes = 1024; @@ -624,6 +621,11 @@ void destroy_large_folio(struct folio *folio) return; } + if (folio_test_transhuge(folio) && dtor == TRANSHUGE_PAGE_DTOR) { + free_transhuge_folio(folio); + return; + } + VM_BUG_ON_FOLIO(dtor >= NR_COMPOUND_DTORS, folio); compound_page_dtors[dtor](&folio->page); } From patchwork Tue Aug 15 03:26:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13353485 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EABEAEB64DD for ; Tue, 15 Aug 2023 03:27:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 833FB900020; Mon, 14 Aug 2023 23:27:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E48190000B; Mon, 14 Aug 2023 23:27:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D3AA900020; Mon, 14 Aug 2023 23:27:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6070190000B for ; Mon, 14 Aug 2023 23:27:19 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 422DCA08A2 for ; Tue, 15 Aug 2023 03:27:19 +0000 (UTC) X-FDA: 81124903398.06.7C30F87 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id 7E8F31C0021 for ; Tue, 15 Aug 2023 03:27:17 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=n+Cmbhnp; dmarc=none; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692070037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hTIBRejTHMVa47/OsAXgkrlKS1/xzRl/GmhV8qZATSk=; b=JqAmWUuknIpBRgcGbPF1XH+xbXwljjtPoSeprPl777LT3oF1vujAo5cX2HiRKnH4EIqhv9 5tteRpDB90HFGgPDKrrv+iYBUvgIJnYG65OMpTa1OP7WnswQgH+30y4zHEXZx18oehFPvr SnH3hB8uVW8Z9HndTO1GW7YS7IQco0U= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=n+Cmbhnp; dmarc=none; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692070037; a=rsa-sha256; cv=none; b=yLbVCvQsH1dLqnOv85jxBi0XXPqItP0yulgEa/ntt1IcGjVnFb7WlnrM2V7O3heiZKMgR5 hiheXaEfz72GnfjONa6lSlSKeL0tDYwaiRH6GdRlgw77fEXydAVqqz3FEfzsuiCd10KhtZ Pg1Qtilc2EH/RV8tuB2pFbj9qbUwhyM= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=hTIBRejTHMVa47/OsAXgkrlKS1/xzRl/GmhV8qZATSk=; b=n+CmbhnpgxNvlfkTNzd9P2ZaHX Cn45PUv3SBi6Yq/3tgjPgZhhAF2R2RadpQ+j8MfyJ+gBK8nKGwkGNhQ686uOAnB/GaaXr+OxhWr/p CASKUPLP7CqSS81qN2gJHZahP+7uSiKiEV4IW3JtnHcTDEa4hCgyR7CvAytGlgvXzWDCF3TVZEDZT w+BU72BcAA/EyE38WlK6CoB6yKsCJr6xxLzEE0WqV0pzP4I7ls5jzKE9jdghA3qHmXUIzBqX6yoQu GEGXj1HYkffhAfBUhFp2clFl6v+eHn7AFzflOwOs2BCJFy6D3/dLnWFXx+28EhmUUUebp0RPGbtkk /d4Hpkjw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVkhm-005qaX-Eo; Tue, 15 Aug 2023 03:26:50 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Jens Axboe , io-uring@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 4/9] mm: Make free_compound_page() static Date: Tue, 15 Aug 2023 04:26:40 +0100 Message-Id: <20230815032645.1393700-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230815032645.1393700-1-willy@infradead.org> References: <20230815032645.1393700-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 7E8F31C0021 X-Stat-Signature: jhky63ngru4rhxejmajcuwwx49eqnbha X-Rspam-User: X-HE-Tag: 1692070037-46113 X-HE-Meta: U2FsdGVkX1+KvhL/M9SgPg87F55pqsqH1AZJ764kSCV7ZiRY6XgtN5ZIUGZt4Bet5k4Ol2ygAZkgbEMqpYnopXIV4dnuoC4WlLEwJ2R9M/KbVCRF3aJIjmp4Ht5GXTN8KisHIWa3zI8csuXE7UuFnQsMoq74uDub9t1lRwAPeT577esCyDiUWnMHW1l4fuLSj4tXiKII44e8MV/ix4nErnt6w635LgHW5d+BHqxxTpfSmIKI1Z6rNYyAkT+DMbxzfNEwhJkhggPm2iJi3/J6n+iYTclxgk6cKm1w+3tYXZtkdYGWIlhgQGjdFYkOTxMUDjqMstrJA0DzPdYPSZkufGp7aVoKVrf7eMuhJ3GfH4AtkXZusXL/Zf9obY2FS8co5a9CsWuquNMhWDM4+KmRuDKlr+lYTIIy1/oP5yfIcW3g0sdMaYyypIuk48U33VVJKWJe1vNc6lo4dKn51ggq5syarhi7x2zIqIzCgZDwJ87p5zLZfpFKYFP/p/iUS/qxbRX/PZcaWWivXmSHJFX1agWh1hCCPJGopbS0jSXVPFFCHQklGRtC4nXK8SJONqy31Jf5fzjj2LnnJpY1brl/WAJvyI+JGprHpnpstU+rbwlh0npGc95nT31dhlz9apbdkh5daYespws5DdanPCQzNkEpd3WNEh2xALTENYBYv6+F3tA3yvtCaYAxeY7nvEoC+nDACF6cQgsjvO/cQmZ/R8VpXGzEevBdRCbqrJOp3dJFmaimYYaPAKhwz9Mx9pOHaA+IWMwkGLN9VtlJ1shCULOViUrWBwXBu1HDCUNSxiH1n1RKxa2fCIG6XqfssmogfgS41fkveprVT/S2e6z2OoWPqhdkDAcj2IwgTX/g0zG97jnHEd+zJLj2adjv3eaPl8dxAd58MjfSDTwPKYnzeBbrc0koKIJEUFPZmyBfWHGNqzOkGTUqjzVNNrQVUgT6749VlysV2Ioju8hsiw1 YndGNOJf UfRR3+Qu4w9L1KDjgZ0ToUl2hB5SbMKO9TGqMYPHmLiiL1EtEKHCupo/HjmV/S+EGvii3h+sNSqgB2rH+qWuM01XVWINv1vzbOD7Oqj26omabC2kNuH/CeKcWr6f7iUdsdO9KacfXmXWF45AfXOr6GlpfcFw+Jcsw1ydzGLkYfjI9mbf0z04F383qSxDB7pAjc0WAMv8Ua/i8MXAQBviZ3dwQ6ePfCJVZ7Om1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_compound_page() is the only remaining dynamic destructor. Call it unconditionally from destroy_large_folio() and convert it to take a folio. It used to be the last thing called from free_transhuge_folio(), and the call from destroy_large_folio() will take care of that case too. This was the last entry in the compound_page_dtors array, so delete it and reword the comments that referred to it. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: David Hildenbrand --- include/linux/mm.h | 11 +---------- mm/huge_memory.c | 1 - mm/page_alloc.c | 23 +++++++---------------- 3 files changed, 8 insertions(+), 27 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7fb529dbff31..cf6707a7069e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1267,14 +1267,7 @@ void folio_copy(struct folio *dst, struct folio *src); unsigned long nr_free_buffer_pages(void); -/* - * Compound pages have a destructor function. Provide a - * prototype for that function and accessor functions. - * These are _only_ valid on the head of a compound page. - */ -typedef void compound_page_dtor(struct page *); - -/* Keep the enum in sync with compound_page_dtors array in mm/page_alloc.c */ +/* Compound pages may have a special destructor */ enum compound_dtor_id { NULL_COMPOUND_DTOR, COMPOUND_PAGE_DTOR, @@ -1325,8 +1318,6 @@ static inline unsigned long thp_size(struct page *page) return PAGE_SIZE << thp_order(page); } -void free_compound_page(struct page *page); - #ifdef CONFIG_MMU /* * Do pte_mkwrite, but only if the vma says VM_WRITE. We do this when diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 516fe3c26ef3..99e36ad540c4 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2797,7 +2797,6 @@ void free_transhuge_folio(struct folio *folio) } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } - free_compound_page(&folio->page); } void deferred_split_folio(struct folio *folio) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index feb2e95cf021..804982faba4e 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -284,11 +284,6 @@ const char * const migratetype_names[MIGRATE_TYPES] = { #endif }; -static compound_page_dtor * const compound_page_dtors[NR_COMPOUND_DTORS] = { - [NULL_COMPOUND_DTOR] = NULL, - [COMPOUND_PAGE_DTOR] = free_compound_page, -}; - int min_free_kbytes = 1024; int user_min_free_kbytes = -1; static int watermark_boost_factor __read_mostly = 15000; @@ -587,17 +582,17 @@ static inline void free_the_page(struct page *page, unsigned int order) * The remaining PAGE_SIZE pages are called "tail pages". PageTail() is encoded * in bit 0 of page->compound_head. The rest of bits is pointer to head page. * - * The first tail page's ->compound_dtor holds the offset in array of compound - * page destructors. See compound_page_dtors. + * The first tail page's ->compound_dtor describes how to destroy the + * compound page. * * The first tail page's ->compound_order holds the order of allocation. * This usage means that zero-order pages may not be compound. */ -void free_compound_page(struct page *page) +static void free_compound_page(struct folio *folio) { - mem_cgroup_uncharge(page_folio(page)); - free_the_page(page, compound_order(page)); + mem_cgroup_uncharge(folio); + free_the_page(&folio->page, folio_order(folio)); } void prep_compound_page(struct page *page, unsigned int order) @@ -621,13 +616,9 @@ void destroy_large_folio(struct folio *folio) return; } - if (folio_test_transhuge(folio) && dtor == TRANSHUGE_PAGE_DTOR) { + if (folio_test_transhuge(folio) && dtor == TRANSHUGE_PAGE_DTOR) free_transhuge_folio(folio); - return; - } - - VM_BUG_ON_FOLIO(dtor >= NR_COMPOUND_DTORS, folio); - compound_page_dtors[dtor](&folio->page); + free_compound_page(folio); } static inline void set_buddy_order(struct page *page, unsigned int order) From patchwork Tue Aug 15 03:26:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13353479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2201EB64DD for ; Tue, 15 Aug 2023 03:27:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E68190001A; Mon, 14 Aug 2023 23:27:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 76EF090000B; Mon, 14 Aug 2023 23:27:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65DF190001A; Mon, 14 Aug 2023 23:27:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5BBB490000B for ; Mon, 14 Aug 2023 23:27:03 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2E41614025F for ; Tue, 15 Aug 2023 03:27:03 +0000 (UTC) X-FDA: 81124902726.08.7430D70 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id 84C3880002 for ; Tue, 15 Aug 2023 03:27:01 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=XGGquuzk; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692070021; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9TGxu02Zdrue3buZphZr1o0ZlYN5TBIdyw14ApHva+w=; b=qftxC8Fzi6+pAibp/nw1p3VVswWWrfVnVpvp9gaMryi5Swxw1DYYMPK6MZH6Vtz6tV6gJg 6XlMWFKasUnNdvkKVQlXfuQLZ74xLa1PE4sCKNH0pvGhlfcU0Gi7s/voh0WwALed7u8ROE o8Dqaa5F0n0tUYCCMi2NI9NaxkJ6r3E= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=XGGquuzk; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692070021; a=rsa-sha256; cv=none; b=RgjeGH/0l5RvDHPIKA8GjLT+/voiam0BpOOQ96LWgKSmxMJ+BmhK+0iIfqlr6lCDd59rFr gdvyS7ExTe8cJTiHeS8BP0JAUUJ2eCz1c3DY9v2Jkc4ZjDBuTR/14HGrKqDtMvNz0Betjx SHcWzScvND+FvK5G0YpwZRRmgN2wND8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=9TGxu02Zdrue3buZphZr1o0ZlYN5TBIdyw14ApHva+w=; b=XGGquuzkKlPlaPg5Kj620RMtNL wJlRYlv2SOYquRt/UO44pwc0Kxd+sofv+wwr+9IY2kWMhuMuQjkOOnOs3mIHfU1zlzJDNWC/U7k9g gFLBcqCXKvz/SN3ggNZWOFB+FcA13nj6oogxs6K7R97+MWgJixPSB3EpF4PyGnu6RPZHEyAeKk6/7 Qg4Pcq+yJE+xsb3RLlA4qQrL0SS6kFwjlqx+xj8bFps/NhfGtdnUx9cOAxrbQM/s02adcHyoS+v6i vQ7PmanJpRAdAVFHN/x8eHflCkJLeBZETAceSujqZDGcsDaa4tFMSkdUf6GChrorsPCS4bDn8pBQm gFYqTYuw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVkhm-005qaZ-I5; Tue, 15 Aug 2023 03:26:50 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Jens Axboe , io-uring@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 5/9] mm: Remove free_compound_page() Date: Tue, 15 Aug 2023 04:26:41 +0100 Message-Id: <20230815032645.1393700-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230815032645.1393700-1-willy@infradead.org> References: <20230815032645.1393700-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 84C3880002 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: mu9gm5i7jc35ti658pz56u7iot8xscbq X-HE-Tag: 1692070021-399333 X-HE-Meta: U2FsdGVkX19VJftTH9H/Mcv3rgDWBC+GSRj4dEE2l1Zhlnt6Jgdzegatc56qyj1nU43ga1iwtKeThD86vcDlu0P0tLerFRo4gAOcgYuE6eKySSBCFAVL27fiXfSp44N/FWEjXjVYedOfAjfkZaHRY9f27PiYg50s2wVvH9nuMS1yeHkgDwLYjvr0oSorxH9Su2+gVnev+sfBNbGN3phYU1M/2NJwxRgGAae7L7saaEubPh/LVzOTq159PYDI+WleqnNnmqJBGQWkxLCPhC3jtc0cglALTnIhPJXCebJ9IhgOS+PC0mf9x+0FfV5Fe9swH60PJTzBc5Ul6B62cjKeJIlaqnjRFWGxTkwT9BF9xhcoPlKSDnO/DGcCI4Qpfth81fMMF+nlIgCZm+USOp2tftwQBLnn1OxXIjvOCCeUGSXR/dc08dH4d9DXaAgjK7lrac8OWvJRj0sd5ZnEDX9un8cJCZlDCBHcbPu46bd2W2Qd3l0Otn2byYziNCjLjiDkJmYVPmbNljQ/N69Ekm3+0BJGAOFsNkV8clCbdeouCAoYRzIOgUvVmX34bes73V0dU8lh1rCDU5cVoFupZeicYUmrf2VbGAWehYQthTPTuNvYdpKACBCMafnz8gAdBkiurRrC6Q+lS04/16HdFhybVXuZP2dEC9WwWjkqY5mjoNhe77GbVqkiXbNUVhysEIa1Dd9hIQIKY9l0CxBg/Aggw+n1fSUtI1q7CZOyPpPOoUZMn2sLDAzvDmhjO9yO/EC9maNbcJ4il0K511+hZf5bftMIFhUYCejnSVVVilqtyB44ZdPzi2dIfGbU+g2VHcd70UNZ6w8vbVkI43F2/KJhRfpxv3O09Y46YWLx7mGDzAXRUPN5XNCazbRpxhrRgR5tPhvkvtlcqf0SguITe/pVnfEP0zxqvAQ6psSyctzSwtvI7jeYaHj6EN8UU/wWu6GECiPqrdeX8FE6YYyqSKh 8udfk31O VBC5Ci8VuFZJnGBQeZPn0OZIJC/a90a9bUXxOBBJfO0Pz9rWo8cNN/pE3WGS0QfqRNWPMfoXZGxiRlYYFOaXFaVced5bQnw1vr50cgnU3pbyhf+a+/0cBgYgqupuhHM8qdb24TvRavjOWhiLrWD3x5mARpJnrThYEh8ZKGWmI2iDhQmFTgJT33NvhdA7rPoqJK15If8CNYqlEEezpqh7ENyjKe7qDqqzNqPff X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Inline it into its one caller. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: David Hildenbrand --- mm/page_alloc.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 804982faba4e..21af71aea6eb 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -589,12 +589,6 @@ static inline void free_the_page(struct page *page, unsigned int order) * This usage means that zero-order pages may not be compound. */ -static void free_compound_page(struct folio *folio) -{ - mem_cgroup_uncharge(folio); - free_the_page(&folio->page, folio_order(folio)); -} - void prep_compound_page(struct page *page, unsigned int order) { int i; @@ -618,7 +612,8 @@ void destroy_large_folio(struct folio *folio) if (folio_test_transhuge(folio) && dtor == TRANSHUGE_PAGE_DTOR) free_transhuge_folio(folio); - free_compound_page(folio); + mem_cgroup_uncharge(folio); + free_the_page(&folio->page, folio_order(folio)); } static inline void set_buddy_order(struct page *page, unsigned int order) From patchwork Tue Aug 15 03:26:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13353481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC272EB64DD for ; Tue, 15 Aug 2023 03:27:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6119D90001C; Mon, 14 Aug 2023 23:27:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C38290000B; Mon, 14 Aug 2023 23:27:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4625890001C; Mon, 14 Aug 2023 23:27:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3622A90000B for ; Mon, 14 Aug 2023 23:27:08 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D74E9140724 for ; Tue, 15 Aug 2023 03:27:07 +0000 (UTC) X-FDA: 81124902894.12.95A5A6C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id 7C59314000F for ; Tue, 15 Aug 2023 03:27:05 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=umf5Qjlr; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692070026; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AkiwaMX9J2EStw5aS/8SkpjxSTrjAeNOcY6xWBx7phM=; b=t2yJNJ1ghHPHeYlM2s4OrDEiLJD6/vsaYQUH9+UPXtadrB+yoeKoYSDL+mkresmahKvEwB paKB0kuYascuqZx7Lx41isuco/KjI9IK+JrilpGX2D+TKEUfpVLb74zOwMoTCluty7efeQ RyNIkzE755v5o6vi+85+KgkJHBsvHGk= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=umf5Qjlr; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692070026; a=rsa-sha256; cv=none; b=eyi4q56wJa53qBuovHdRpuu4mlgyqVo5IChUJCjsVoaYDEo8XEhjerWipaH3TX0d2MSRRd qLdL7m6Dwwggusl11o67DnLq+Ui+AFTPr3gOnx1noedMkaCP47iBKk3gri2IUVsWLEP/d0 b5ZqLay9uY/HYCNJM1/1LYbyxLNQt64= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=AkiwaMX9J2EStw5aS/8SkpjxSTrjAeNOcY6xWBx7phM=; b=umf5Qjlr4TVGgREeM841PEBgao U/OYkolNBqfy6XznAkCSwddr8yp9BZo/N9H1CdaqwxN06gvrn6jvEqEPblAuMZ/waLUZPVkZ5q7wx OY71tWBf9TI6QkfLbu7dgqNXSXjR5cCPXH9ZMZrrX3Q83VTs2JfbkUELVZd2w5ggZLg/uOH7iCSh7 2YnG4iRVeFk3+o/A8i46rvMVdmIyfKMXspzjCxKR1/AKExDZARAfxbaXsajqLo8bw0bsZglLNg7re Wx2OkHUSPmqEJE+LQBQ3M/ETqVLMRr+AXX3FtZWRR6GV/lEPaWXQBv+151JJoN3WDoLIzN8AfnjbB 6hPPGVtg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVkhm-005qab-LN; Tue, 15 Aug 2023 03:26:50 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Jens Axboe , io-uring@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 6/9] mm: Remove HUGETLB_PAGE_DTOR Date: Tue, 15 Aug 2023 04:26:42 +0100 Message-Id: <20230815032645.1393700-7-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230815032645.1393700-1-willy@infradead.org> References: <20230815032645.1393700-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7C59314000F X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: ixs6w74erjfuqonhg93myuc3ak1o8j9s X-HE-Tag: 1692070025-925524 X-HE-Meta: U2FsdGVkX18ElK/Tv2rp2PwFbO9vQzp51HnIjxBweQLtOXpFZLY+7bck1ZM/AJMQltJLDfbveGJOADQK3SUwee2+F9IbBytcRNURgQBrRCoOgzeG0HjIvlEXkyvUtayl746Uf/W5to6RFpU8S/TzsQdBXSGSiB/ejnPG/ctBM29kYY+ASezNgJKLUY6AaLUoHgBdX9B8ZEKLapmvvFHWY9r5ACHqU4ErbcEaAZk2H2jwA9yPhkYChv8AZ+pysxvEgVaX/I5SsgJ50ACJlUHdJqdk3Gei7j0I57IksJ5ltWIEKH//WaTGriJdbgzLEdFiLKPByt+I/ADC2hKugFPXlb1kYeRSHoNyPpNtT4Bw47RrvQvLgjfgYgxYmIR5n5/T4RA7GvNxdomlOT3vEToKu5mCbxohR+0QKulVjyxruYZ5joitXFpYCnUS4G9ZmlICyx5o2QxKJaRyCMVR3EQ+/HUHU50K/uTgyAMNzdgC3RsoWKazk7c5eedBly+22AKi3IYF5Hn9Xz60BPhoM207R2GFaAxjLdaDpm9kv2iAmpaLsAgf4Y1WW1opWLmtThkjHgEKKY/dSwSvwEdeBOlvEpv0Qpgb5kC7O+xbdGWohyEHzoMPAOiCStCiNk9COWPVk863+loKUlteIBAKTqPTAzQf+PP4vh1T2hlCHu3hadcNmOjudSxtuni39SD9EBHz5+UezFQTOVoFSg1p03DbYX1YExHIzUjSdqofx3LSNqbTregPHJNpe4AVtMzQBXrRtyM8WIjczfBl4SUPr1gA2w4zUGEhlqeUGDM/unTBziwp7xM7IIc8sic2ThjdXSKN357iT9GuDNmrarLHHZVX/5eftI3GygVuILaPCOf9H+9siTqQzMlEE37+lnR7Z7Jilg/qUeE5nsxlYHy/usMYoeQqU2ieXkCGTvR+6UkXnuhrj1J645W8zmtkmY62BLARcAWiO4fpVLiuBxQCqqN ivRg0MaT QrLVZfs+4QaFI7S2ywx8ad0k9U9ym4Xc/xrDBKZs2ax5x2R8RTvogY14E5rY+NK0mm2kLCGRRi40lp8vxGOFsn3rguvEWnhXBQjWkOcrfVSxKPONL5qo0y6kvQsxK5nyrOH/D+vFXAIluZA6EO3kDA/QX9lRewvCGRxeDNq7+nx3NHMIP4kbAm7JkzqLL1pNZhjdRD7dcVyfkJr9hca6qxs5ugvLEAxcdwhPk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We can use a bit in page[1].flags to indicate that this folio belongs to hugetlb instead of using a value in page[1].dtors. That lets folio_test_hugetlb() become an inline function like it should be. We can also get rid of NULL_COMPOUND_DTOR. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: David Hildenbrand --- .../admin-guide/kdump/vmcoreinfo.rst | 10 +--- include/linux/mm.h | 2 - include/linux/page-flags.h | 21 +++++++- kernel/crash_core.c | 2 +- mm/hugetlb.c | 49 +++---------------- 5 files changed, 29 insertions(+), 55 deletions(-) diff --git a/Documentation/admin-guide/kdump/vmcoreinfo.rst b/Documentation/admin-guide/kdump/vmcoreinfo.rst index c18d94fa6470..baa1c355741d 100644 --- a/Documentation/admin-guide/kdump/vmcoreinfo.rst +++ b/Documentation/admin-guide/kdump/vmcoreinfo.rst @@ -325,8 +325,8 @@ NR_FREE_PAGES On linux-2.6.21 or later, the number of free pages is in vm_stat[NR_FREE_PAGES]. Used to get the number of free pages. -PG_lru|PG_private|PG_swapcache|PG_swapbacked|PG_slab|PG_hwpoision|PG_head_mask ------------------------------------------------------------------------------- +PG_lru|PG_private|PG_swapcache|PG_swapbacked|PG_slab|PG_hwpoision|PG_head_mask|PG_hugetlb +----------------------------------------------------------------------------------------- Page attributes. These flags are used to filter various unnecessary for dumping pages. @@ -338,12 +338,6 @@ More page attributes. These flags are used to filter various unnecessary for dumping pages. -HUGETLB_PAGE_DTOR ------------------ - -The HUGETLB_PAGE_DTOR flag denotes hugetlbfs pages. Makedumpfile -excludes these pages. - x86_64 ====== diff --git a/include/linux/mm.h b/include/linux/mm.h index cf6707a7069e..c8c8b1fd64d3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1269,9 +1269,7 @@ unsigned long nr_free_buffer_pages(void); /* Compound pages may have a special destructor */ enum compound_dtor_id { - NULL_COMPOUND_DTOR, COMPOUND_PAGE_DTOR, - HUGETLB_PAGE_DTOR, TRANSHUGE_PAGE_DTOR, NR_COMPOUND_DTORS }; diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 92a2063a0a23..01d98695e79f 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -180,6 +180,9 @@ enum pageflags { PG_has_hwpoisoned = PG_error, #endif + /* Is a hugetlb page. Stored in first tail page. */ + PG_hugetlb = PG_writeback, + /* non-lru isolated movable page */ PG_isolated = PG_reclaim, @@ -812,7 +815,23 @@ static inline void ClearPageCompound(struct page *page) #ifdef CONFIG_HUGETLB_PAGE int PageHuge(struct page *page); -bool folio_test_hugetlb(struct folio *folio); +SETPAGEFLAG(HugeTLB, hugetlb, PF_SECOND) +CLEARPAGEFLAG(HugeTLB, hugetlb, PF_SECOND) + +/** + * folio_test_hugetlb - Determine if the folio belongs to hugetlbfs + * @folio: The folio to test. + * + * Context: Any context. Caller should have a reference on the folio to + * prevent it from being turned into a tail page. + * Return: True for hugetlbfs folios, false for anon folios or folios + * belonging to other filesystems. + */ +static inline bool folio_test_hugetlb(struct folio *folio) +{ + return folio_test_large(folio) && + test_bit(PG_hugetlb, folio_flags(folio, 1)); +} #else TESTPAGEFLAG_FALSE(Huge, hugetlb) #endif diff --git a/kernel/crash_core.c b/kernel/crash_core.c index 90ce1dfd591c..dd5f87047d06 100644 --- a/kernel/crash_core.c +++ b/kernel/crash_core.c @@ -490,7 +490,7 @@ static int __init crash_save_vmcoreinfo_init(void) #define PAGE_BUDDY_MAPCOUNT_VALUE (~PG_buddy) VMCOREINFO_NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE); #ifdef CONFIG_HUGETLB_PAGE - VMCOREINFO_NUMBER(HUGETLB_PAGE_DTOR); + VMCOREINFO_NUMBER(PG_hugetlb); #define PAGE_OFFLINE_MAPCOUNT_VALUE (~PG_offline) VMCOREINFO_NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE); #endif diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bc340f5dbbd4..a1cebcee6503 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1585,25 +1585,7 @@ static inline void __clear_hugetlb_destructor(struct hstate *h, { lockdep_assert_held(&hugetlb_lock); - /* - * Very subtle - * - * For non-gigantic pages set the destructor to the normal compound - * page dtor. This is needed in case someone takes an additional - * temporary ref to the page, and freeing is delayed until they drop - * their reference. - * - * For gigantic pages set the destructor to the null dtor. This - * destructor will never be called. Before freeing the gigantic - * page destroy_compound_gigantic_folio will turn the folio into a - * simple group of pages. After this the destructor does not - * apply. - * - */ - if (hstate_is_gigantic(h)) - folio_set_compound_dtor(folio, NULL_COMPOUND_DTOR); - else - folio_set_compound_dtor(folio, COMPOUND_PAGE_DTOR); + folio_clear_hugetlb(folio); } /* @@ -1690,7 +1672,7 @@ static void add_hugetlb_folio(struct hstate *h, struct folio *folio, h->surplus_huge_pages_node[nid]++; } - folio_set_compound_dtor(folio, HUGETLB_PAGE_DTOR); + folio_set_hugetlb(folio); folio_change_private(folio, NULL); /* * We have to set hugetlb_vmemmap_optimized again as above @@ -1814,9 +1796,8 @@ static void free_hpage_workfn(struct work_struct *work) /* * The VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio) in * folio_hstate() is going to trigger because a previous call to - * remove_hugetlb_folio() will call folio_set_compound_dtor - * (folio, NULL_COMPOUND_DTOR), so do not use folio_hstate() - * directly. + * remove_hugetlb_folio() will clear the hugetlb bit, so do + * not use folio_hstate() directly. */ h = size_to_hstate(page_size(page)); @@ -1955,7 +1936,7 @@ static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) { hugetlb_vmemmap_optimize(h, &folio->page); INIT_LIST_HEAD(&folio->lru); - folio_set_compound_dtor(folio, HUGETLB_PAGE_DTOR); + folio_set_hugetlb(folio); hugetlb_set_folio_subpool(folio, NULL); set_hugetlb_cgroup(folio, NULL); set_hugetlb_cgroup_rsvd(folio, NULL); @@ -2070,28 +2051,10 @@ int PageHuge(struct page *page) if (!PageCompound(page)) return 0; folio = page_folio(page); - return folio->_folio_dtor == HUGETLB_PAGE_DTOR; + return folio_test_hugetlb(folio); } EXPORT_SYMBOL_GPL(PageHuge); -/** - * folio_test_hugetlb - Determine if the folio belongs to hugetlbfs - * @folio: The folio to test. - * - * Context: Any context. Caller should have a reference on the folio to - * prevent it from being turned into a tail page. - * Return: True for hugetlbfs folios, false for anon folios or folios - * belonging to other filesystems. - */ -bool folio_test_hugetlb(struct folio *folio) -{ - if (!folio_test_large(folio)) - return false; - - return folio->_folio_dtor == HUGETLB_PAGE_DTOR; -} -EXPORT_SYMBOL_GPL(folio_test_hugetlb); - /* * Find and lock address space (mapping) in write mode. * From patchwork Tue Aug 15 03:26:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13353486 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D1CDEB64DD for ; Tue, 15 Aug 2023 03:27:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AF31C900017; Mon, 14 Aug 2023 23:27:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AA37F90000B; Mon, 14 Aug 2023 23:27:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96CF4900017; Mon, 14 Aug 2023 23:27:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8A69990000B for ; Mon, 14 Aug 2023 23:27:22 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4F57A14025F for ; Tue, 15 Aug 2023 03:27:22 +0000 (UTC) X-FDA: 81124903524.20.1FA521E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 912071C0006 for ; Tue, 15 Aug 2023 03:27:20 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=jaNFDNXx; dmarc=none; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692070040; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9AFjGPtcNZvRdattiMeM+NhO363S0DBsMAr8j6pNUgk=; b=nDd6dvFDXAhEpthW+A0qQJoN/edrVXOf4i3izv6t02pvb6E0SDJS9kKqPazXt7tKbcRcQM hpnWP34z/IqoVsyOgkk3SGdbB7lGNZeC/zW94l+72XRS478ICNwHIK6VWeT04Uy4vRv4oa BSMe0+v9ldRngY7pfdZPt+wBD+wtiCA= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=jaNFDNXx; dmarc=none; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692070040; a=rsa-sha256; cv=none; b=1beh3vhiiI/aEOm2y9ikputLe1ded4Se3XGBAxK+myZeq/rwQXJRwT6V6jePEnQT6y+P8O yJy5sgUct+Q+ylgxHDv8yk8+nZWAYI+VZpJfGNUSRUwo6WqOGv1yICjjbxXM+FDO05rZw+ QrUG2m94nA0M7Nxox48pEEaN+ZDvG3w= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=9AFjGPtcNZvRdattiMeM+NhO363S0DBsMAr8j6pNUgk=; b=jaNFDNXx63oA6YktKdyny5R+FA RVh5LmTlc1op/Sst8fvW70x9QNRDwuwJOHdIIjJZWUf6guT7CgC+x3iC4VTc0Z5C7ZeV8vywxhRn2 iQv7KMpfL46Nk/RHw5syw8gNRhvNS9JKOkvCC4s91EKuD+3TD2PM7Lck/sH50znrz4zhJ66K/6xqW frQp6MmSz5oZtlGIHcjgVJoS3tN28ZkHgwNnW2HzRzSYdibignJU8xbNbAUpTqqzvPL/2uF4lwhl+ HhW8C6yHRX3CyVf75d3s/k2AEl+VW5wqASKAdQSwMdsKUrccMFJq47rXEAyQWAHYMWzz3V5pr2xvE 93mcRMSg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVkhm-005qad-Or; Tue, 15 Aug 2023 03:26:50 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Jens Axboe , io-uring@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 7/9] mm: Add deferred_list page flag Date: Tue, 15 Aug 2023 04:26:43 +0100 Message-Id: <20230815032645.1393700-8-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230815032645.1393700-1-willy@infradead.org> References: <20230815032645.1393700-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 912071C0006 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 413sqoxhddeot68gq1sbewooys4ndcz5 X-HE-Tag: 1692070040-363195 X-HE-Meta: U2FsdGVkX1+3pjPb654G1ZfNLZUkHxnSN2qJ+Zi9KlbOwcbWLi9/Y6IdoVlmSUlpxUAPoKre5ALQcwI+q02c5yyvZJr6QFCUABcOD5yqeuEtBfLfLkcxxi1nvxvb77SIBbK7sj5L3ybnACUp4sc0neNPazOBn3Yn9lFmG0UQSsJ1UVh7Jazauu6k75xm7lLUNf3FUp1MvIt2SQ4RMhjf8fiPc2/1QVyE4jELSfCBekAgfEhnoICnSTUGdo7hPs94OGXNgeAeaEniYqOzDVaNqohFYLsVXfVKKLI8vgrGMi22RcnrW6MoXgVM29cq4ioiRDKNvAmbIDAYDHO3q2D2u6yDsRbhmHrLn3rqLmHN2DtkRby/8/5ZtnVC0BI+/AYCAV+GbrOKwz1x3GOd1nbOX285MzT2LuzjUiwt9u0Oa3lvSiECiBjuK+zRgc9sei1ve1irvzrHpl6NyxySjmL43MUK5A4U723k8RqlJVPeLzR20OaKdvg7f8RXcX5jrkGsQIYi/HWw3WyPZTclIHhMyYvLOoyB6w6GtKD286Hsz+zEmVqWP01lk3/BabBjrbXsQE+AjZg/oUxkhxjsGe/hxqTsqbnE1VISAnYmGf8WIFp/k0slRp0CksVbQnF6obDYDWgaRKvJ9Q7u3oQMxKnE32qzKSC3R+ZyqJMuHXqfelcLOYPQ6HXY5eSVHIOWutnid6jtrOdwYIaYXpyQp/FFaVZ4R3ndF57YYNtPncFUgx4ZJIi4keD0SvEiZ7Do/oVhxLrVbfyB8tWJxVG1KoIlk9Bsy4T/BquVgpt4tG4RvBUUPSICw0eeqOVaw0yoOzBzYBwnupVwYz/hpwtNeeHeSa/4/XU/tKfFdict2LHELm45p3pl+SqOL0GGCBU8M6fgISvdxXCESekMCm+JUOcMjCvl0l4OgmWtJQ0U6V8uZi7oLe+XOK19c/LwXo6DfnpLREeVShmiWBfa4sr9qdL 1m+AWGaF ZWD5aAwhd2CfWR+Pqot7hZz3XXJlBtM8pdcne3OwTrlZj4Dw94lQ9XSOqOxgjW0/CNls9zTOYiXZNP7c3TM2wLvZuibfK+5HK5kkFkgnLapCKe6mUx3abjLeYnpfxT61P3D2GKBXN5Sdy6IXQFfBjtims/Bowa8fH1F7OOu7R6F5RSaF5Q87V2km9KgY9kSVRnYd5KmLindE3vRt5UPBqiiz6spPKMoofOkb1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Stored in the first tail page's flags, this flag replaces the destructor. That removes the last of the destructors, so remove all references to folio_dtor and compound_dtor. Signed-off-by: Matthew Wilcox (Oracle) --- Documentation/admin-guide/kdump/vmcoreinfo.rst | 4 ++-- include/linux/mm.h | 14 -------------- include/linux/mm_types.h | 2 -- include/linux/page-flags.h | 6 ++++++ kernel/crash_core.c | 1 - mm/huge_memory.c | 4 ++-- mm/internal.h | 1 - mm/page_alloc.c | 7 +------ 8 files changed, 11 insertions(+), 28 deletions(-) diff --git a/Documentation/admin-guide/kdump/vmcoreinfo.rst b/Documentation/admin-guide/kdump/vmcoreinfo.rst index baa1c355741d..3bd38ac0e7de 100644 --- a/Documentation/admin-guide/kdump/vmcoreinfo.rst +++ b/Documentation/admin-guide/kdump/vmcoreinfo.rst @@ -141,8 +141,8 @@ nodemask_t The size of a nodemask_t type. Used to compute the number of online nodes. -(page, flags|_refcount|mapping|lru|_mapcount|private|compound_dtor|compound_order|compound_head) -------------------------------------------------------------------------------------------------- +(page, flags|_refcount|mapping|lru|_mapcount|private|compound_order|compound_head) +---------------------------------------------------------------------------------- User-space tools compute their values based on the offset of these variables. The variables are used when excluding unnecessary pages. diff --git a/include/linux/mm.h b/include/linux/mm.h index c8c8b1fd64d3..cf0ae8c51d7f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1267,20 +1267,6 @@ void folio_copy(struct folio *dst, struct folio *src); unsigned long nr_free_buffer_pages(void); -/* Compound pages may have a special destructor */ -enum compound_dtor_id { - COMPOUND_PAGE_DTOR, - TRANSHUGE_PAGE_DTOR, - NR_COMPOUND_DTORS -}; - -static inline void folio_set_compound_dtor(struct folio *folio, - enum compound_dtor_id compound_dtor) -{ - VM_BUG_ON_FOLIO(compound_dtor >= NR_COMPOUND_DTORS, folio); - folio->_folio_dtor = compound_dtor; -} - void destroy_large_folio(struct folio *folio); /* Returns the number of bytes in this potentially compound page. */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index da538ff68953..d45a2b8041e0 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -282,7 +282,6 @@ static inline struct page *encoded_page_ptr(struct encoded_page *page) * @_refcount: Do not access this member directly. Use folio_ref_count() * to find how many references there are to this folio. * @memcg_data: Memory Control Group data. - * @_folio_dtor: Which destructor to use for this folio. * @_folio_order: Do not use directly, call folio_order(). * @_entire_mapcount: Do not use directly, call folio_entire_mapcount(). * @_nr_pages_mapped: Do not use directly, call folio_mapcount(). @@ -336,7 +335,6 @@ struct folio { unsigned long _flags_1; unsigned long _head_1; /* public: */ - unsigned char _folio_dtor; unsigned char _folio_order; atomic_t _entire_mapcount; atomic_t _nr_pages_mapped; diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 01d98695e79f..aabf50dc71a3 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -183,6 +183,9 @@ enum pageflags { /* Is a hugetlb page. Stored in first tail page. */ PG_hugetlb = PG_writeback, + /* Has a deferred list (may be empty). First tail page. */ + PG_deferred_list = PG_reclaim, + /* non-lru isolated movable page */ PG_isolated = PG_reclaim, @@ -809,6 +812,9 @@ static inline void ClearPageCompound(struct page *page) BUG_ON(!PageHead(page)); ClearPageHead(page); } +PAGEFLAG(DeferredList, deferred_list, PF_SECOND) +#else +TESTPAGEFLAG_FALSE(DeferredList, deferred_list) #endif #define PG_head_mask ((1UL << PG_head)) diff --git a/kernel/crash_core.c b/kernel/crash_core.c index dd5f87047d06..934dd86e19f5 100644 --- a/kernel/crash_core.c +++ b/kernel/crash_core.c @@ -455,7 +455,6 @@ static int __init crash_save_vmcoreinfo_init(void) VMCOREINFO_OFFSET(page, lru); VMCOREINFO_OFFSET(page, _mapcount); VMCOREINFO_OFFSET(page, private); - VMCOREINFO_OFFSET(folio, _folio_dtor); VMCOREINFO_OFFSET(folio, _folio_order); VMCOREINFO_OFFSET(page, compound_head); VMCOREINFO_OFFSET(pglist_data, node_zones); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 99e36ad540c4..3b5db99eb7ac 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -583,7 +583,7 @@ void prep_transhuge_page(struct page *page) VM_BUG_ON_FOLIO(folio_order(folio) < 2, folio); INIT_LIST_HEAD(&folio->_deferred_list); - folio_set_compound_dtor(folio, TRANSHUGE_PAGE_DTOR); + folio_set_deferred_list(folio); } static inline bool is_transparent_hugepage(struct page *page) @@ -595,7 +595,7 @@ static inline bool is_transparent_hugepage(struct page *page) folio = page_folio(page); return is_huge_zero_page(&folio->page) || - folio->_folio_dtor == TRANSHUGE_PAGE_DTOR; + folio_test_deferred_list(folio); } static unsigned long __thp_get_unmapped_area(struct file *filp, diff --git a/mm/internal.h b/mm/internal.h index 5a03bc4782a2..e3d11119b04e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -417,7 +417,6 @@ static inline void prep_compound_head(struct page *page, unsigned int order) { struct folio *folio = (struct folio *)page; - folio_set_compound_dtor(folio, COMPOUND_PAGE_DTOR); folio_set_order(folio, order); atomic_set(&folio->_entire_mapcount, -1); atomic_set(&folio->_nr_pages_mapped, 0); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 21af71aea6eb..9fe9209605a5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -582,9 +582,6 @@ static inline void free_the_page(struct page *page, unsigned int order) * The remaining PAGE_SIZE pages are called "tail pages". PageTail() is encoded * in bit 0 of page->compound_head. The rest of bits is pointer to head page. * - * The first tail page's ->compound_dtor describes how to destroy the - * compound page. - * * The first tail page's ->compound_order holds the order of allocation. * This usage means that zero-order pages may not be compound. */ @@ -603,14 +600,12 @@ void prep_compound_page(struct page *page, unsigned int order) void destroy_large_folio(struct folio *folio) { - enum compound_dtor_id dtor = folio->_folio_dtor; - if (folio_test_hugetlb(folio)) { free_huge_page(folio); return; } - if (folio_test_transhuge(folio) && dtor == TRANSHUGE_PAGE_DTOR) + if (folio_test_deferred_list(folio)) free_transhuge_folio(folio); mem_cgroup_uncharge(folio); free_the_page(&folio->page, folio_order(folio)); From patchwork Tue Aug 15 03:26:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13353480 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D5F3C04A6A for ; Tue, 15 Aug 2023 03:27:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A7F6190001B; Mon, 14 Aug 2023 23:27:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A30A190000B; Mon, 14 Aug 2023 23:27:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8815F90001B; Mon, 14 Aug 2023 23:27:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7B69490000B for ; Mon, 14 Aug 2023 23:27:04 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4DABC407D1 for ; Tue, 15 Aug 2023 03:27:04 +0000 (UTC) X-FDA: 81124902768.01.799D17E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id 8F6ABC0005 for ; Tue, 15 Aug 2023 03:27:02 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=JGbswiDJ; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692070022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=f59aeb0mmdHQJI1C4RwzY3Ldg6y1fWJH+3t+QmXVWA0=; b=MEZmv8sG7ivuILtRJlJ+7gGaOJ5YWF9chSv2olwFAWX2aWa2zfuAxtjb9m+fw63db2st9Z VwSwXhu5ppGnxrU2YHNA9zYQmSijpoauHsbVnVV1TspdUnG0Zg2CqmbBhTIn1AYRwwjUid yrwL2LdZ98vlXY1oahzhoGewF3JGOhQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=JGbswiDJ; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692070022; a=rsa-sha256; cv=none; b=sDWpBPWuLxnihtKI1KOiwplsZmQlO/Bz39riILmEUMH4v9nV0FPAMz9tO4pQ1PFjGZaVYs Bd6kGLmb/gy3glzwJ7U2i3CAZEBRFYTqIqc2CbfPpmOHX+lG/iWTw/S5hCGGO/YwgMxUX8 x+Rp4ckiEnylSkvi6UMnMyLdw7xCQ9M= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=f59aeb0mmdHQJI1C4RwzY3Ldg6y1fWJH+3t+QmXVWA0=; b=JGbswiDJ34dDq4gaBq7kG58sIS ZFsx9XleJZknvuEtpNq42FfDPlnHuX6TJKg+qKrB9bPr27v53sWti/JJY3r4PKJ0bZg6XW2eRIQ+Q Mdz3lE5/xT1YzPMl2rGLC45LWCh7CMhhqc4XE/Z0tkyAwARxFW4BlJJetWxK1o7XDA96tY9wgNqlt fJC3fFpbLEm4XMYOsXG3w5i+3DeaIX45zQbQvlUVXb0qInhNW50xMQkZz72aFiy+C1WcUi+rJ9wvN uWB4pyQ8mCy6Y5iofpd7VKM0IPNOwIYAaszCRZFEp3heABqH4/zWVq/nij67Ybt462ftX48Hf5zIX RlDlHxrw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVkhm-005qaf-Rr; Tue, 15 Aug 2023 03:26:50 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Jens Axboe , io-uring@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 8/9] mm: Rearrange page flags Date: Tue, 15 Aug 2023 04:26:44 +0100 Message-Id: <20230815032645.1393700-9-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230815032645.1393700-1-willy@infradead.org> References: <20230815032645.1393700-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8F6ABC0005 X-Rspam-User: X-Stat-Signature: gsdaohmbgk5eh4xsser37fzwbe3877b7 X-Rspamd-Server: rspam01 X-HE-Tag: 1692070022-779358 X-HE-Meta: U2FsdGVkX1+oA4X98F0H7ScYIBTL2aNviOypQNHUTV4vkIDTGIE8h5RPQHN7bNUQC/ZpbPHnm5WRkG/4DoIFfIa3KNusF0Ud5XwL7JvY+5lrwgti7FpVKFoAtObTN+2J5FZM85ceP9TPa3MrR/ud8wvafrBlsyLUMkd5Qw8fi1cQSbEVWwq+Q3MQ+zXB3wExYDgBpJpAlTeXHqtA57eWs/KSoW60WquyBG1Y3pj27KohVSoia7zHKWAy3aRkgoHmPfXhZaKz/fX6bJE8oGQybVEsNk7mHe8ynA4sCkm1QdLlnO4QfbQpAierjp7toYcys9+pQSG1pvBCF/xa1KA9tBEc0G76HiGg3MwyBOYyzAmV99/nDkTSEQOlTBPa8629yaBga+Djv0dltZyABKfbwW6tuuZTJlnj8F/11bBcwvsFA4HJ2T9TlT4rLkHXIpJ2z1iXP+LXDER62/1Sy4tArt5mvQlpfaQOZ4ilk7oyWF3BOxJzAkNfpBzBXnBPCYbWWawfFuUWr8MwbRziczvrOZRur4fSWGeWGa1ENIDiHoE1BA9YZ+4Uf+csc+rf7EahXctl8TbB5/KMw8J22mT3ZdpTse/gS2gG3kPcbvjzPCD1Dpd5pOOtiFHAo7O4gh9/+poT+8R+yLwdr/7nyYMcm5BmZPHfVLLCHr4oUUUptDhe2A9jfRMLKTmrA+5AlK//te7RDDkpioLryga2E5mBGS7zMoYMBx1Yi70/OHB6YTqdmEo4lvvknpO2227gOJYXb7zD3KN1HJ6f0SGbl6crA3afH77tQUicVMU4gaPWTnMkMjvJdfNbK5xGEPeG4EXco2hI/sJ1YKJItg5ogBUaqj0++BpOktXYyNMRlEYb7r+AhiLtSiTM9ug/WmXtifm1rQiJet+ieIS2tpz3QxpCSpwJh0zKBSmgSEB8RLWRnQm0l1jzoQhBLgFplNvO7v2kgPAKRtvAH6ZFZJoTt5E RxKC+xK9 FFNpQdCcwSBX/ZE+phrALVp/1CATLdO4VPxu5i1TLd/6Ulf76KxKPK05M3dkZ0oBqAGsQ1smzNJ9PnetzU1hdW7NpBl2aOS6fcjIjNwpYzgsiqLrk6NSmkYrdhe2bqnVIqU5w0F+JTUqnCieVSv4Q+dnMssvMZren9LIwn4aiRBWpRImZw2oAwGkaow== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move PG_writeback into bottom byte so that it can use PG_waiters in a later patch. Move PG_head into bottom byte as well to match with where 'order' is moving next. PG_active and PG_workingset move into the second byte to make room for them. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/page-flags.h | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index aabf50dc71a3..6a0dd94b2460 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -99,13 +99,15 @@ */ enum pageflags { PG_locked, /* Page is locked. Don't touch. */ + PG_writeback, /* Page is under writeback */ PG_referenced, PG_uptodate, PG_dirty, PG_lru, + PG_head, /* Must be in bit 6 */ + PG_waiters, /* Page has waiters, check its waitqueue. Must be bit #7 and in the same byte as "PG_locked" */ PG_active, PG_workingset, - PG_waiters, /* Page has waiters, check its waitqueue. Must be bit #7 and in the same byte as "PG_locked" */ PG_error, PG_slab, PG_owner_priv_1, /* Owner use. If pagecache, fs may use*/ @@ -113,8 +115,6 @@ enum pageflags { PG_reserved, PG_private, /* If pagecache, has fs-private data */ PG_private_2, /* If pagecache, has fs aux data */ - PG_writeback, /* Page is under writeback */ - PG_head, /* A head page */ PG_mappedtodisk, /* Has blocks allocated on-disk */ PG_reclaim, /* To be reclaimed asap */ PG_swapbacked, /* Page is backed by RAM/swap */ @@ -171,21 +171,19 @@ enum pageflags { /* Remapped by swiotlb-xen. */ PG_xen_remapped = PG_owner_priv_1, -#ifdef CONFIG_MEMORY_FAILURE /* - * Compound pages. Stored in first tail page's flags. - * Indicates that at least one subpage is hwpoisoned in the - * THP. + * Flags only valid for compound pages. Stored in first tail page's + * flags word. */ - PG_has_hwpoisoned = PG_error, -#endif - - /* Is a hugetlb page. Stored in first tail page. */ - PG_hugetlb = PG_writeback, - /* Has a deferred list (may be empty). First tail page. */ + /* At least one page is hwpoisoned in the folio. */ + PG_has_hwpoisoned = PG_error, + /* Belongs to hugetlb */ + PG_hugetlb = PG_active, + /* Has a deferred list (does not indicate whether it is active) */ PG_deferred_list = PG_reclaim, + /* non-lru isolated movable page */ PG_isolated = PG_reclaim, From patchwork Tue Aug 15 03:26:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13353484 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F34D2EB64DD for ; Tue, 15 Aug 2023 03:27:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D03090001F; Mon, 14 Aug 2023 23:27:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8809090000B; Mon, 14 Aug 2023 23:27:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7709990001F; Mon, 14 Aug 2023 23:27:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6AEC290000B for ; Mon, 14 Aug 2023 23:27:16 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 475A5A089A for ; Tue, 15 Aug 2023 03:27:16 +0000 (UTC) X-FDA: 81124903272.28.8A2F901 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP id 996F61A0016 for ; Tue, 15 Aug 2023 03:27:14 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=r8z4nRTX; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692070034; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3R6E9lJA1fDVHLkeYLU6dejaQkhuWTMTotkeI2cZgmY=; b=cgCExV8nNn9jVT3r7tMTGeI7t7qZnb4bQjt7btuwuZxWmp8R5BKpZlVf6hZtN49JeurwG9 bTkGDJm0C+trxKPZZ4rT3FozvRFwQ2ch9ZYYE5rvDou9Q3klKGzh/8PagZpYjsURuIV1te wpTzyB7uGnra6zzsYgzHayLQyM2R61s= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=r8z4nRTX; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692070034; a=rsa-sha256; cv=none; b=IQugWaWNi7n/13yC4y31bawOtlgAidur67bOUYlemUmksFmJBQDtlJJvirj3Eg7Yh9xXFN 1h4BnTWpbqU37aHPSauNH8pe9uTNcsgFevWBWo0NUeu+J8Q9TgUsEhwKv1iZOQuODK+R7V smoSFINIm0B2PTsttF8KhvNtDVjYbPc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=3R6E9lJA1fDVHLkeYLU6dejaQkhuWTMTotkeI2cZgmY=; b=r8z4nRTXrUlGt13XLtw/J29M57 5iYsOyQKNe8VJqfMxhGgwmnIe/R287HhpT++UwKivgqqDadScL+cz1sFbsrfBDQx1g3458ci4PhgF AyqC1KGX1VAcEIXgdJddQfARtBt/P3ebW/WGT3S0ssSHOfR1GKjklwZbK3EtyoQLS+kR0WEOzfCWF EEGSEJHJkpURJptXSmoen5k0UFL3E5ZM63L/CZDu1Q7cuogfE2xBXKgx/t89+2l2tkmPEjnl6abqq xwcxNnoVELc6pqnj+swY5/8Tyr64hTvra/bMZy/pb6i6ZklQVPzlMXgJfbUarjRUjB98cD/EFFfm9 NDrl8r2g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVkhm-005qah-Vy; Tue, 15 Aug 2023 03:26:51 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Jens Axboe , io-uring@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 9/9] mm: Free up a word in the first tail page Date: Tue, 15 Aug 2023 04:26:45 +0100 Message-Id: <20230815032645.1393700-10-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230815032645.1393700-1-willy@infradead.org> References: <20230815032645.1393700-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 996F61A0016 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: z7j5m77y68bpqmwfi98c9fjbur8gh33k X-HE-Tag: 1692070034-303528 X-HE-Meta: U2FsdGVkX18p0+JfHALOXJpteSnVZ/fGbjDf/2jfF2bivIxfUMrDhy404M8vEt5HLT7do/1ZkPeA45t3g7zcxA8+wj5qOCT2G4pCkfRlFp9wZQV/oA1epq1W/lek9OmT4/ibdUbtbMXz+ixBOKc08Hi39mImKe2lGqPQQy/oBSRuSXo5G43kRJyuiCE7vLygXAPl39sqIbPXoUol/7GLScooeRzaTw8hvNttorxz16rizbbxG09b1xsVQt04P6+M8CwqkYoaIPLlsYhSBLe1q4Y8UGnNWxM9+eK4nJqHvrrYfOV+Ho1uI7cIPxsOkbXtdHGyUJOqHK69qlj5yugt4pQ9oEIQJDXJfpgl+vqybulAprmnGC9lPZJTVIXReW3yz2EW1M2zEZ8QRBeoYk+JoiefJdDRXp5yGYmjmj7C4L0TphHTWTuXS1f6NzpcuqC9u+6SdihFDid0aCjSCUCxuU6LH4n6R4jK82cXMvV7rcTvGNtnKOy56eCNDjElTyz3ItCICE9X1HxnpW9agqhtlLfgFmjxE6uzGx5B24TC9G41muPkAQc7eXeg1OKmk1AGo+0BOzaU7Wmg0Q6n6sn8CWLyJuRJKWQX7lzZAum7WbG0Q1qJWbEgmun+ZFXjIeyeNMBQywflfwRUxrcDIiQPjSuWRVsXeHF5wXySEOSAPuLmR5dtwD4BbPDjMypiTeFfOJuNO05ei4ZjTzVM27Cl+C5JgU27NzdnLjO5drgsOyTYSSIkpduW5dXc/WCJeF1uibcjPoJ7vMu3MPJzd78410uhgLzuV3FFebOyje+aov3B/zX0u6H72+eZsi6mStX1OXneNDUniyBgN6le6pS2prYJZanVg2bER5I+HYnvdUDly7hF0P/t4MwaOOUFsX4HFaYoI6c7VwPozJkyutrSbnXsb7iJt6vl1nNnlp7PX3e4I/pRrHFJxPeaOwtkuLsqsOz/4Pfwpw30VdpBQe7 jT8E6qwZ Gesy6Zp3DGHWK4wvhIXfvPD8tPQcAchHLHmG8k6qX7MQRLGFqzHPzDj8y9g9lUTnSrpasm4sa6JMn+rGyq941ZNkngzHLKuED+o3zeHJxBtIa3UL3HvBAjqAp6riENf3IVQZ42xXaDWk7pPNufs+Q+l5nWfxW9ovZ5cpYN3LSbOpFm6SGbJfOVm0rREIurAqpu6UpP3d5AJ1SlIeyilxeElMpZd5TsJM8i5YX10RXUfdqblbj15tap9D4AA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Store the folio order in the low byte of the flags word in the first tail page. This frees up the word that was being used to store the order and dtor bytes previously. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 10 +++++----- include/linux/mm_types.h | 3 +-- kernel/crash_core.c | 1 - mm/internal.h | 2 +- mm/page_alloc.c | 4 +++- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index cf0ae8c51d7f..85568e2b2556 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1028,7 +1028,7 @@ struct inode; * compound_order() can be called without holding a reference, which means * that niceties like page_folio() don't work. These callers should be * prepared to handle wild return values. For example, PG_head may be - * set before _folio_order is initialised, or this may be a tail page. + * set before the order is initialised, or this may be a tail page. * See compaction.c for some good examples. */ static inline unsigned int compound_order(struct page *page) @@ -1037,7 +1037,7 @@ static inline unsigned int compound_order(struct page *page) if (!test_bit(PG_head, &folio->flags)) return 0; - return folio->_folio_order; + return folio->_flags_1 & 0xff; } /** @@ -1053,7 +1053,7 @@ static inline unsigned int folio_order(struct folio *folio) { if (!folio_test_large(folio)) return 0; - return folio->_folio_order; + return folio->_flags_1 & 0xff; } #include @@ -2025,7 +2025,7 @@ static inline long folio_nr_pages(struct folio *folio) #ifdef CONFIG_64BIT return folio->_folio_nr_pages; #else - return 1L << folio->_folio_order; + return 1L << (folio->_flags_1 & 0xff); #endif } @@ -2043,7 +2043,7 @@ static inline unsigned long compound_nr(struct page *page) #ifdef CONFIG_64BIT return folio->_folio_nr_pages; #else - return 1L << folio->_folio_order; + return 1L << (folio->_flags_1 & 0xff); #endif } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index d45a2b8041e0..659c7b84726c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -282,7 +282,6 @@ static inline struct page *encoded_page_ptr(struct encoded_page *page) * @_refcount: Do not access this member directly. Use folio_ref_count() * to find how many references there are to this folio. * @memcg_data: Memory Control Group data. - * @_folio_order: Do not use directly, call folio_order(). * @_entire_mapcount: Do not use directly, call folio_entire_mapcount(). * @_nr_pages_mapped: Do not use directly, call folio_mapcount(). * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). @@ -334,8 +333,8 @@ struct folio { struct { unsigned long _flags_1; unsigned long _head_1; + unsigned long _folio_avail; /* public: */ - unsigned char _folio_order; atomic_t _entire_mapcount; atomic_t _nr_pages_mapped; atomic_t _pincount; diff --git a/kernel/crash_core.c b/kernel/crash_core.c index 934dd86e19f5..693445e1f7f6 100644 --- a/kernel/crash_core.c +++ b/kernel/crash_core.c @@ -455,7 +455,6 @@ static int __init crash_save_vmcoreinfo_init(void) VMCOREINFO_OFFSET(page, lru); VMCOREINFO_OFFSET(page, _mapcount); VMCOREINFO_OFFSET(page, private); - VMCOREINFO_OFFSET(folio, _folio_order); VMCOREINFO_OFFSET(page, compound_head); VMCOREINFO_OFFSET(pglist_data, node_zones); VMCOREINFO_OFFSET(pglist_data, nr_zones); diff --git a/mm/internal.h b/mm/internal.h index e3d11119b04e..c415260c1f06 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -407,7 +407,7 @@ static inline void folio_set_order(struct folio *folio, unsigned int order) if (WARN_ON_ONCE(!order || !folio_test_large(folio))) return; - folio->_folio_order = order; + folio->_flags_1 = (folio->_flags_1 & ~0xffUL) | order; #ifdef CONFIG_64BIT folio->_folio_nr_pages = 1U << order; #endif diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9fe9209605a5..0e0e0d18a81b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1115,8 +1115,10 @@ static __always_inline bool free_pages_prepare(struct page *page, VM_BUG_ON_PAGE(compound && compound_order(page) != order, page); - if (compound) + if (compound) { ClearPageHasHWPoisoned(page); + page[1].flags &= ~0xffUL; + } for (i = 1; i < (1 << order); i++) { if (compound) bad += free_tail_page_prepare(page, page + i);