From patchwork Tue Aug 15 03:26:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13353484 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F34D2EB64DD for ; Tue, 15 Aug 2023 03:27:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D03090001F; Mon, 14 Aug 2023 23:27:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8809090000B; Mon, 14 Aug 2023 23:27:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7709990001F; Mon, 14 Aug 2023 23:27:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6AEC290000B for ; Mon, 14 Aug 2023 23:27:16 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 475A5A089A for ; Tue, 15 Aug 2023 03:27:16 +0000 (UTC) X-FDA: 81124903272.28.8A2F901 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP id 996F61A0016 for ; Tue, 15 Aug 2023 03:27:14 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=r8z4nRTX; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692070034; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3R6E9lJA1fDVHLkeYLU6dejaQkhuWTMTotkeI2cZgmY=; b=cgCExV8nNn9jVT3r7tMTGeI7t7qZnb4bQjt7btuwuZxWmp8R5BKpZlVf6hZtN49JeurwG9 bTkGDJm0C+trxKPZZ4rT3FozvRFwQ2ch9ZYYE5rvDou9Q3klKGzh/8PagZpYjsURuIV1te wpTzyB7uGnra6zzsYgzHayLQyM2R61s= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=r8z4nRTX; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692070034; a=rsa-sha256; cv=none; b=IQugWaWNi7n/13yC4y31bawOtlgAidur67bOUYlemUmksFmJBQDtlJJvirj3Eg7Yh9xXFN 1h4BnTWpbqU37aHPSauNH8pe9uTNcsgFevWBWo0NUeu+J8Q9TgUsEhwKv1iZOQuODK+R7V smoSFINIm0B2PTsttF8KhvNtDVjYbPc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=3R6E9lJA1fDVHLkeYLU6dejaQkhuWTMTotkeI2cZgmY=; b=r8z4nRTXrUlGt13XLtw/J29M57 5iYsOyQKNe8VJqfMxhGgwmnIe/R287HhpT++UwKivgqqDadScL+cz1sFbsrfBDQx1g3458ci4PhgF AyqC1KGX1VAcEIXgdJddQfARtBt/P3ebW/WGT3S0ssSHOfR1GKjklwZbK3EtyoQLS+kR0WEOzfCWF EEGSEJHJkpURJptXSmoen5k0UFL3E5ZM63L/CZDu1Q7cuogfE2xBXKgx/t89+2l2tkmPEjnl6abqq xwcxNnoVELc6pqnj+swY5/8Tyr64hTvra/bMZy/pb6i6ZklQVPzlMXgJfbUarjRUjB98cD/EFFfm9 NDrl8r2g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVkhm-005qah-Vy; Tue, 15 Aug 2023 03:26:51 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Jens Axboe , io-uring@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 9/9] mm: Free up a word in the first tail page Date: Tue, 15 Aug 2023 04:26:45 +0100 Message-Id: <20230815032645.1393700-10-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230815032645.1393700-1-willy@infradead.org> References: <20230815032645.1393700-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 996F61A0016 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: z7j5m77y68bpqmwfi98c9fjbur8gh33k X-HE-Tag: 1692070034-303528 X-HE-Meta: U2FsdGVkX18p0+JfHALOXJpteSnVZ/fGbjDf/2jfF2bivIxfUMrDhy404M8vEt5HLT7do/1ZkPeA45t3g7zcxA8+wj5qOCT2G4pCkfRlFp9wZQV/oA1epq1W/lek9OmT4/ibdUbtbMXz+ixBOKc08Hi39mImKe2lGqPQQy/oBSRuSXo5G43kRJyuiCE7vLygXAPl39sqIbPXoUol/7GLScooeRzaTw8hvNttorxz16rizbbxG09b1xsVQt04P6+M8CwqkYoaIPLlsYhSBLe1q4Y8UGnNWxM9+eK4nJqHvrrYfOV+Ho1uI7cIPxsOkbXtdHGyUJOqHK69qlj5yugt4pQ9oEIQJDXJfpgl+vqybulAprmnGC9lPZJTVIXReW3yz2EW1M2zEZ8QRBeoYk+JoiefJdDRXp5yGYmjmj7C4L0TphHTWTuXS1f6NzpcuqC9u+6SdihFDid0aCjSCUCxuU6LH4n6R4jK82cXMvV7rcTvGNtnKOy56eCNDjElTyz3ItCICE9X1HxnpW9agqhtlLfgFmjxE6uzGx5B24TC9G41muPkAQc7eXeg1OKmk1AGo+0BOzaU7Wmg0Q6n6sn8CWLyJuRJKWQX7lzZAum7WbG0Q1qJWbEgmun+ZFXjIeyeNMBQywflfwRUxrcDIiQPjSuWRVsXeHF5wXySEOSAPuLmR5dtwD4BbPDjMypiTeFfOJuNO05ei4ZjTzVM27Cl+C5JgU27NzdnLjO5drgsOyTYSSIkpduW5dXc/WCJeF1uibcjPoJ7vMu3MPJzd78410uhgLzuV3FFebOyje+aov3B/zX0u6H72+eZsi6mStX1OXneNDUniyBgN6le6pS2prYJZanVg2bER5I+HYnvdUDly7hF0P/t4MwaOOUFsX4HFaYoI6c7VwPozJkyutrSbnXsb7iJt6vl1nNnlp7PX3e4I/pRrHFJxPeaOwtkuLsqsOz/4Pfwpw30VdpBQe7 jT8E6qwZ Gesy6Zp3DGHWK4wvhIXfvPD8tPQcAchHLHmG8k6qX7MQRLGFqzHPzDj8y9g9lUTnSrpasm4sa6JMn+rGyq941ZNkngzHLKuED+o3zeHJxBtIa3UL3HvBAjqAp6riENf3IVQZ42xXaDWk7pPNufs+Q+l5nWfxW9ovZ5cpYN3LSbOpFm6SGbJfOVm0rREIurAqpu6UpP3d5AJ1SlIeyilxeElMpZd5TsJM8i5YX10RXUfdqblbj15tap9D4AA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Store the folio order in the low byte of the flags word in the first tail page. This frees up the word that was being used to store the order and dtor bytes previously. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 10 +++++----- include/linux/mm_types.h | 3 +-- kernel/crash_core.c | 1 - mm/internal.h | 2 +- mm/page_alloc.c | 4 +++- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index cf0ae8c51d7f..85568e2b2556 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1028,7 +1028,7 @@ struct inode; * compound_order() can be called without holding a reference, which means * that niceties like page_folio() don't work. These callers should be * prepared to handle wild return values. For example, PG_head may be - * set before _folio_order is initialised, or this may be a tail page. + * set before the order is initialised, or this may be a tail page. * See compaction.c for some good examples. */ static inline unsigned int compound_order(struct page *page) @@ -1037,7 +1037,7 @@ static inline unsigned int compound_order(struct page *page) if (!test_bit(PG_head, &folio->flags)) return 0; - return folio->_folio_order; + return folio->_flags_1 & 0xff; } /** @@ -1053,7 +1053,7 @@ static inline unsigned int folio_order(struct folio *folio) { if (!folio_test_large(folio)) return 0; - return folio->_folio_order; + return folio->_flags_1 & 0xff; } #include @@ -2025,7 +2025,7 @@ static inline long folio_nr_pages(struct folio *folio) #ifdef CONFIG_64BIT return folio->_folio_nr_pages; #else - return 1L << folio->_folio_order; + return 1L << (folio->_flags_1 & 0xff); #endif } @@ -2043,7 +2043,7 @@ static inline unsigned long compound_nr(struct page *page) #ifdef CONFIG_64BIT return folio->_folio_nr_pages; #else - return 1L << folio->_folio_order; + return 1L << (folio->_flags_1 & 0xff); #endif } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index d45a2b8041e0..659c7b84726c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -282,7 +282,6 @@ static inline struct page *encoded_page_ptr(struct encoded_page *page) * @_refcount: Do not access this member directly. Use folio_ref_count() * to find how many references there are to this folio. * @memcg_data: Memory Control Group data. - * @_folio_order: Do not use directly, call folio_order(). * @_entire_mapcount: Do not use directly, call folio_entire_mapcount(). * @_nr_pages_mapped: Do not use directly, call folio_mapcount(). * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). @@ -334,8 +333,8 @@ struct folio { struct { unsigned long _flags_1; unsigned long _head_1; + unsigned long _folio_avail; /* public: */ - unsigned char _folio_order; atomic_t _entire_mapcount; atomic_t _nr_pages_mapped; atomic_t _pincount; diff --git a/kernel/crash_core.c b/kernel/crash_core.c index 934dd86e19f5..693445e1f7f6 100644 --- a/kernel/crash_core.c +++ b/kernel/crash_core.c @@ -455,7 +455,6 @@ static int __init crash_save_vmcoreinfo_init(void) VMCOREINFO_OFFSET(page, lru); VMCOREINFO_OFFSET(page, _mapcount); VMCOREINFO_OFFSET(page, private); - VMCOREINFO_OFFSET(folio, _folio_order); VMCOREINFO_OFFSET(page, compound_head); VMCOREINFO_OFFSET(pglist_data, node_zones); VMCOREINFO_OFFSET(pglist_data, nr_zones); diff --git a/mm/internal.h b/mm/internal.h index e3d11119b04e..c415260c1f06 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -407,7 +407,7 @@ static inline void folio_set_order(struct folio *folio, unsigned int order) if (WARN_ON_ONCE(!order || !folio_test_large(folio))) return; - folio->_folio_order = order; + folio->_flags_1 = (folio->_flags_1 & ~0xffUL) | order; #ifdef CONFIG_64BIT folio->_folio_nr_pages = 1U << order; #endif diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9fe9209605a5..0e0e0d18a81b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1115,8 +1115,10 @@ static __always_inline bool free_pages_prepare(struct page *page, VM_BUG_ON_PAGE(compound && compound_order(page) != order, page); - if (compound) + if (compound) { ClearPageHasHWPoisoned(page); + page[1].flags &= ~0xffUL; + } for (i = 1; i < (1 << order); i++) { if (compound) bad += free_tail_page_prepare(page, page + i);