From patchwork Mon Aug 14 18:44:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13353178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 325E2C04A94 for ; Mon, 14 Aug 2023 18:44:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B1668E0001; Mon, 14 Aug 2023 14:44:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 939A9900003; Mon, 14 Aug 2023 14:44:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 78CFD8E0005; Mon, 14 Aug 2023 14:44:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6A1648E0001 for ; Mon, 14 Aug 2023 14:44:22 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B672F140C37 for ; Mon, 14 Aug 2023 18:44:21 +0000 (UTC) X-FDA: 81123585522.27.2EDFE1C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 91F5C140027 for ; Mon, 14 Aug 2023 18:44:19 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LWIZN6aa; spf=pass (imf09.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692038659; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=K80vu1sJx9DsGDhSbLEtYPGEOnTcZDHaiO/EYxwWGMs=; b=G+wBGno/vemfjqSRGzHuO60FN9yKhdpiebeYz1scu9zU2fQxSQH6Fmw3t9dDvyAPz6Siij qiGZ8QYpGxA9QWtFdggINJBRe8sXOTXBvFmku7lJeL3Fa+tocN+lVKBzaxofaZBLrgsioi epdFwTlPRP9alaouhZozVuVrR6gl2/k= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LWIZN6aa; spf=pass (imf09.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692038659; a=rsa-sha256; cv=none; b=UKOPYZ33uqGN7KHtOxLXCKyKK01XqHy6b3Ry6T2qoO+SQ/6ldIuyAqs5SpUULsF3ZQZJcj zyRXaV2qiTjUMPbE0XD4tVqHoAiPyMtKBGwtCzidCqwkvOXR8qlpjdL3NWo6v9YyHYqbYL PydzE5qdXw0Brxq5zJR6J1tvP3Xw4ho= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1692038659; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K80vu1sJx9DsGDhSbLEtYPGEOnTcZDHaiO/EYxwWGMs=; b=LWIZN6aa3JqGirFL6e89Esdfk9wcCSfbu2topzZX1b7bSTGXY0JKYJv7/7CKf4895hGYF9 bZhfjZsiUPxD5rfuPZFsFk3h934OBYXErHVT5MiSGXnDVQd10+xS/UyVMn1hOuZSufTTqA p9syRkjGkBz08QWZiKbtmd9JGWFwf/8= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-662-NmHe97SoP0ilgB1R_dmB_w-1; Mon, 14 Aug 2023 14:44:15 -0400 X-MC-Unique: NmHe97SoP0ilgB1R_dmB_w-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-645778904dfso8657126d6.1 for ; Mon, 14 Aug 2023 11:44:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692038655; x=1692643455; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K80vu1sJx9DsGDhSbLEtYPGEOnTcZDHaiO/EYxwWGMs=; b=hoz0W8qcv6lftaIarvfacqAo+9VPiYTEWTKFXvF6LeWGpngmKqFM8UIngCo0FlY+tS I5Cuk6DpMfs2y+6TXbEXLKTU4LPxk8nFwNAtvZZQEUGX8dOmQkgJiYtXk5mujP6Jj632 Alz3iD5TbHolkpCg7jIJRTeuneYOfUB1UopNPtqhZSUAUmdZKJ5fPwApaRwkyaF3a0pS Um8McX+OFZRkFNkT/gJjG4OPNoXew/rJg1ERlnh8B2vCYwA3/DWm2kgVmZ3H0xJ+b0HU BiLybKFTBCwGJ7WGvTzh6j5I0C/cl/P1G01pClpNFnYXDsGEgQaIXLHdfWXKHesyEvhq yYRw== X-Gm-Message-State: AOJu0YzXYLgrbmzdvfk+iF1Ez/N9cYpMLmlvyMfyEwkF8iVDn2WRIhNJ 9cCiImUmuA4uGkG0Qww1i16Pao4ibJqlnXFUmxp8Z2zY0u7iSJJ4FPUkrGRkg+6D0Esnn6P7El2 m4MLfi5oZEbugcCu/LPKPGrxgJDUUqA9A8nF1C0zK3aia74mbhhl2dIkb869d7e96WzSm X-Received: by 2002:a05:6214:226b:b0:626:2305:6073 with SMTP id gs11-20020a056214226b00b0062623056073mr14561436qvb.4.1692038655250; Mon, 14 Aug 2023 11:44:15 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGWDv/pTkCgGz4dnpVLXX0rV2wACMdw+Qcy+eJSwM3xg9sOHmswZi98yBi7GjAnG22hvSrGEA== X-Received: by 2002:a05:6214:226b:b0:626:2305:6073 with SMTP id gs11-20020a056214226b00b0062623056073mr14561409qvb.4.1692038654848; Mon, 14 Aug 2023 11:44:14 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id a7-20020a0cb347000000b00630c0ed6339sm3566561qvf.64.2023.08.14.11.44.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Aug 2023 11:44:14 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , David Hildenbrand , Matthew Wilcox , Andrew Morton , peterx@redhat.com, Yu Zhao , Ryan Roberts , Yang Shi , Hugh Dickins , "Kirill A . Shutemov" Subject: [PATCH RFC v2 1/3] mm: Add TAIL_MAPPING_REUSED_MAX Date: Mon, 14 Aug 2023 14:44:09 -0400 Message-ID: <20230814184411.330496-2-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230814184411.330496-1-peterx@redhat.com> References: <20230814184411.330496-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 91F5C140027 X-Rspam-User: X-Stat-Signature: 7ogo1pi5dm7xsxprxfc4d5emcur7dpcy X-Rspamd-Server: rspam01 X-HE-Tag: 1692038659-315777 X-HE-Meta: U2FsdGVkX181KgtHhKH3CoCH8uuwLtqSg1IECwcG4kUblHp7hFnaXkIIW4+B40DnLrho7lekra5LaHTkH3cn8u2R9ItJw93JvCYZ5zh58kajmfx+p2zOEdw7R99tM6VIW9Ddx8XAQif/bH+F3Iv0Gai34QWMJshuHZrqwwTwHxGqEuCOAkDW8dM4T72OVcMp74FBIW7RjV8ddyOBC/pmWyZhW/ObhuFAphS8RfTGc++36e2CvyacV7C7Y+uVqJsAOqKMP6tJj1nG6s0OTa++HJ0YhcUR6Rsf9J5PeQNV9Xbbj4JYW6dhjgOBzpC24k2phaP2CzGC3+e1iL5vpSI3aTtxsi29GzNZ8X9t6YEy2QKY8G5xbVJuyop/9HzOZ9TK+0LXy0BdzrEa7Lyrmwnk4YPKzHFCUNeOJo5M9P4C0eT+3qdxcNVBBVptbYPqHujCF1gKbQB1hLQTeDTIl5IqSK4JdGWB1ujzbFQUOYPG8LcGAxI9LJT+losFZNmo9UvvsDqIuopuq4Xj/Zlj0xbIvO8KdXuxGCYKwFgncgE/6BDgvQRX7j+HQAJtpH2zhUHIKHNyewjDczj03zMSzOrA4me75ZVX1nbS5wdHF92r3RY/jieV1ZyJc/Qp//LYGVxgBHH4PhIlRvsAtB2c3G586IHSIO2G63vax6b6SP5w08tx5yM1E+4xsKNGaDVGrk+7H4zj4/xPSQ5Muj0w2HBXWJqHRaXlLpPnB/0QkS6GLUbGfW7MuRppPrxWRBNv0rDwdGD575HyoCc2vojX2tVGZyv+ruI498r4h77cA0/ELq5gxgDe0YUQxRryCD1rJMNs5Lpzxs2Yg/bBgaznkpRMafFFI6/pS1Me6Ppwl0iVzBjSJpBCb6gziiJS169cufgiyzEvdtTnObZ+pnSB7qLD0xHPYjarX3TAx1L7HyIvX5oevZKkJ5EZw1UQ3hf9CgISR0D8wzzHeU6VpnoCOXb 9od6tRDJ NUv//lQxOb7wi8g5HPb2nDREmuUPzXgPJqUDz3OsPF++zc+zvhQZiqM7/nbacV1RO9T1N6iKVIXeh+wXXxzIgciJuDy0/LUZVkph7vAoJGu1JOd0KOQ7zRfO3h3UoqFj4j9FjbuGB8JlTQ4kSbW7tjTw8tpPyncZf8QXywgc0NrruMOxc0wtFKqEu8a8cNWGGII6DPdxIv1c+WRgI6s2tpySIfHTtPJmQrW+65U6fl7XitMqW0Oso6jLBN8BbH+K6dqs3ToMPzGREo7q0LXJoCsX5bcWJ75xlmZx41fCjvTfBWfyCjKpetX2nMrZhtj4TIypv6yELAHD9Y6p9DBCwc13PEi94tQQlbdrzVoMMZlW7TsD/m5UefCJtI+PZt8x+FBh2UiHO5MBDRLZBN87XGfJEBA3ycP5K486YvuAcjp9vHK5sGATJTDNxjg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Tail pages have a sanity check on ->mapping fields, not all of them but only upon index>2. It's because we reused ->mapping fields of the tail pages index=0,1 for other things. Define a macro for "max index of tail pages that got ->mapping field reused" on top of folio definition, because when we grow folio tail pages we'd want to boost this too together. Signed-off-by: Peter Xu --- include/linux/mm_types.h | 9 +++++++++ mm/huge_memory.c | 6 +++--- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 291c05cacd48..3f2b0d46f5d6 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -248,6 +248,15 @@ static inline struct page *encoded_page_ptr(struct encoded_page *page) return (struct page *)(~ENCODE_PAGE_BITS & (unsigned long)page); } +/* + * This macro defines the maximum tail pages (of a folio) that can have the + * page->mapping field reused (offset 12 for 32bits, or 24 for 64bits). + * + * When the tail page's mapping field reused, it'll be exempted from + * ->mapping poisoning and checks. Also see the macro TAIL_MAPPING. + */ +#define TAIL_MAPPING_REUSED_MAX (2) + /** * struct folio - Represents a contiguous set of bytes. * @flags: Identical to the page flags. diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0b709d2c46c6..72f244e16dcb 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2444,9 +2444,9 @@ static void __split_huge_page_tail(struct page *head, int tail, (1L << PG_dirty) | LRU_GEN_MASK | LRU_REFS_MASK)); - /* ->mapping in first and second tail page is replaced by other uses */ - VM_BUG_ON_PAGE(tail > 2 && page_tail->mapping != TAIL_MAPPING, - page_tail); + /* ->mapping in <=TAIL_MAPPING_REUSED_MAX tail pages are reused */ + VM_BUG_ON_PAGE(tail > TAIL_MAPPING_REUSED_MAX && + page_tail->mapping != TAIL_MAPPING, page_tail); page_tail->mapping = head->mapping; page_tail->index = head->index + tail; From patchwork Mon Aug 14 18:44:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13353180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 063CAC001DB for ; Mon, 14 Aug 2023 18:44:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BAE10900005; Mon, 14 Aug 2023 14:44:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B0FB3900003; Mon, 14 Aug 2023 14:44:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9616D900005; Mon, 14 Aug 2023 14:44:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 77FA6900003 for ; Mon, 14 Aug 2023 14:44:23 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 425D4A0C6B for ; Mon, 14 Aug 2023 18:44:23 +0000 (UTC) X-FDA: 81123585606.10.365E2D3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id DDE0C14000A for ; Mon, 14 Aug 2023 18:44:20 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=e7XN054Q; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692038660; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FNQlVHJQ4pIimjZoX+ixCZ4sXgWXT3PPMnETr4dtlP8=; b=2dWpXeuYaFAgkRWWsUdGrezfAuNh0P1mSpdNjMv2urPAE1j2e+mbMhmQZ4vpfFta/3WM1m 9Q7IREJZSv0R9Rsb0i7xgjJgMiQuWq4FSymRY3IZmdl4Y/c9IA4n+sMOGXHFu4DqJYQ0Uo JEwpzyX2zSMWdsjz7+BOmyBQbLNwzDA= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=e7XN054Q; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692038660; a=rsa-sha256; cv=none; b=asrWvEUl7fRUaKHVpxpksJaowTwB2iEtQ56bls6bR2Pk86Et2l48xU+FxpJ0KyZjQxY4Vn QCoH4F9SLI2eR6N5THpnMW7O6te02LDnW1NSTzpbyAsAz5iSmkQRqj4jjymRUIxR4m7loP v4fkpRwA0C+iBC/GOaFs3ktdD3N3KZ0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1692038660; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FNQlVHJQ4pIimjZoX+ixCZ4sXgWXT3PPMnETr4dtlP8=; b=e7XN054QmRLQr/cKA+MUKQLb/QKlF/HQdNlvqgaWf/0LaKP4XrPRyWZTARCl2ujXdliwJQ lDLEGI9yQxSqblIe3ev+POVggLfTBnip1ZZ+KUTyEd4kzbOigprd33uRbBLQmumn/9GsDz akcNCVAsMrkD7i2RkjuuYe0lLtWHuUY= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-654--aboGh27NwuGXFGSi5dvNg-1; Mon, 14 Aug 2023 14:44:17 -0400 X-MC-Unique: -aboGh27NwuGXFGSi5dvNg-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-63c9463c116so13418416d6.0 for ; Mon, 14 Aug 2023 11:44:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692038656; x=1692643456; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FNQlVHJQ4pIimjZoX+ixCZ4sXgWXT3PPMnETr4dtlP8=; b=T2d2nwg4lefjJD4YNxuTElZ4xOjEYpROqB3ECld77uXeaG5TTp8w8DxbHq/N9e5UjC 7e3WCEU8/f5u0n4/6FXpDJylA8VA4Vl3beq3jsuKz9QrCc965tW32PvEthF/D7IPPSNV /6eYUYQ2mvXmUpRhSsNrKIzBmoJ7KOkUqxITr92OlRGBe2v4fOm68GX98lkG6zswTYkC 7yWrwE0hY9ybjQ56sjxfD/IR7MGF7sDw8joo26onL7txYTBuirKH9aij2F4v6mTc5aCX 0ocEnP4+pIDIZ7W50FegdwDsiMa0lLj3M3/WYCMG3aCZaw+/RdX2WvsMN+s6K0XrkkNn ktng== X-Gm-Message-State: AOJu0YyuSlP2z/vJV+K6jvS9FbOOopyUc/CR57sRChp+GXazlZoUYdV+ CWXqgt2Q5rOBLxetlLVxgqlKV6z4o+mpM0iR3PthoN2RmAfm5sXdVkQXKzXpXYpvwLeNaeFICgw OWqowYLDygAhDVCCTYz2ngO8WRZsXn6Somnp65w0wNJ0B2YDPflBuofGEWUCKJViajBRP X-Received: by 2002:a05:6214:1d09:b0:635:d9d0:cccf with SMTP id e9-20020a0562141d0900b00635d9d0cccfmr14721975qvd.4.1692038656457; Mon, 14 Aug 2023 11:44:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHg5lNOTyDZrEVxNjNHMJ72ccEGFwymh3Zpf7FNl3V4lRHNSY8M7kb1dQ3rnyfSzoGymgcKPg== X-Received: by 2002:a05:6214:1d09:b0:635:d9d0:cccf with SMTP id e9-20020a0562141d0900b00635d9d0cccfmr14721945qvd.4.1692038656034; Mon, 14 Aug 2023 11:44:16 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id a7-20020a0cb347000000b00630c0ed6339sm3566561qvf.64.2023.08.14.11.44.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Aug 2023 11:44:15 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , David Hildenbrand , Matthew Wilcox , Andrew Morton , peterx@redhat.com, Yu Zhao , Ryan Roberts , Yang Shi , Hugh Dickins , "Kirill A . Shutemov" Subject: [PATCH RFC v2 2/3] mm: Reorg and declare free spaces in struct folio tails Date: Mon, 14 Aug 2023 14:44:10 -0400 Message-ID: <20230814184411.330496-3-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230814184411.330496-1-peterx@redhat.com> References: <20230814184411.330496-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Stat-Signature: xmt4jdmazh7un19334t8ttichsn37hfq X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: DDE0C14000A X-HE-Tag: 1692038660-866198 X-HE-Meta: U2FsdGVkX1/v9c4WIcq1/c/1IlkIupumNpcZw1YeQ1+fkeZCUJzuWg+kxpudaK2V9QZMskFfS3dFhwONKCYVrOyRer63J81zWNX7P//OOlbhhJH3f/4mQED+FPYf6VQeHfNkk01Jh+g9mdraxlqhtv83GFNN8Q5CPXKo+mn1TmY7GyKRAK591087UeGxWvsqz1rPB4iJtzlAhqqklvhAwndKPz7IcQbyryrkGFIJMvoblJ/xx2I/fEye6f2A4rQvlEZe4jNnqDdlONG/KfKc+HpqdXbAlN42EDI9Ij/g2laMo3HKPMun1DTZq5tmBvfyMHYU31IlhXIBTXOdXOQjHQN2vFNkQqmlvZSUZP4lDZhoiMtzNRXCoSE4Q1OTgjF2cvdfdd3ZKNgWT1385vBOaoLwdB4mNHbJ5rgXYC2FcNatIHub30Zi4qQjstOuIbKFhy3c8AAhcDJhWLhg6iw08Cm6wuhFOO6Opd0tGPOC6Kd3apkMtgOAeQW6zm929M77fsP6BIsLheT/9EK6R77CR/+Ai4UuktIM6yc9u93NsLdS66N6nhNyw4+aoWF4CAG8ig9c+eOBiovIVX7R2kZ9tVfEIW9kXdJScC04Chpq9SSRI0DX+C5P+m7sl2wXUFoMeCL/DAUDONW/wJAfA0urKf9cVR37hdJEf1x01QGmv9YiS0KW1AKfLvFhyA9e6G60be2aHP4dU9TUC/B9S6O+tbew4AMQhPHG4X+JAhlRluYdXDTxhzl8e61qirRKFzfdevXe/mLI9hLTBdlGDSjvcXSTprN7NpMgcZQnyYqeIfUlOmW3oa6WYTv35qHbgJeP4vjMvWUtqHxo0Vi44tTOzPAyak48WccX6RO6QrkknB257lYSkKeJnNqssb5cgukSiqddnYk7IOOZi64lumnlro3ihe2zYL8BWSQLUUZyflwUHkVuLd+IbncQz1r0vIA04/pUuHK7x0dPGPhnHLF BvEy028p MDCTAXrokFXkqNHwUL8e/eVN88zYeDlFb3Vz8nAXZgsiN2bSSQAB5aUJ0Iec2Q28bBIys5BtLJpHGGgIZxR1CWmvUBTy2jbx++nH9zKrMOxqy4X/DGRwaUUKCL7JactvvVhZ/NEI/gWMBVOqGf8RZzovJD/8ZFd7I3HZIPPyvjJlKAfwL3T6dfUyUqcXuUTWIVSA2hl2ZXOw6/7W/9IH0whDLjxEltHjDoaMtmcT/WsaoJffTKDAmBmTLOdtIFJ6W0qZWRbbepOGC9q0TkvUuHM5lz6rjkNCmrZxx3EuT7t94CgUoHXt3xSRgqmQNRJA2wbyqYs27Zu7aswEXtZbZKTlsqa+zbyKU9ufZRDvOAvgUWt7P8Mbam5+cj5O4M8Qtrgfgy+MeoXUjL7s= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It's not 100% clear on what are the free spaces in the folio tail pages. Currently we defined fields for only tail pages 1-2 but they're not really fully occupied. Add the fields to show what is free, and also reorg them a bit to make 32/64 bits alignment easy. Here _free_1_0 should be a constant hole (of 2 bytes) on any system, make them explicit so people know they can be reused at any time. _free_1_1 is special and need some attention: this will shift tail page 1's fields starting from _entire_mapcount to be 4 bytes later. I don't expect this change much on real performance - if it will it might be good to have _entire_mapcount and _nr_pages_mapped to be put on the same 8B alignment, assuming that _pincount should be rarer to be used in real life. But in all cases the movement shouldn't change much on x86 or anything that has 64B cachelines. This is the major reason why I had this change separate from the upcoming documentation update patch - it may need some attention, and when unwanted things happen (I don't expect) we quickly know what's wrong. _free_1_2 / _free_2_1 just calls out extra free spaces elsewhere and shouldn't affect a thing just like _free_1_0. Signed-off-by: Peter Xu --- include/linux/mm_types.h | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 3f2b0d46f5d6..829f5adfded1 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -329,11 +329,21 @@ struct folio { /* public: */ unsigned char _folio_dtor; unsigned char _folio_order; + /* private: 2 bytes can be reused later */ + unsigned char _free_1_0[2]; +#ifdef CONFIG_64BIT + /* 4 bytes can be reused later (64 bits only) */ + unsigned char _free_1_1[4]; +#endif + /* public: */ atomic_t _entire_mapcount; atomic_t _nr_pages_mapped; atomic_t _pincount; #ifdef CONFIG_64BIT unsigned int _folio_nr_pages; + /* private: 4 bytes can be reused later (64 bits only) */ + unsigned char _free_1_2[4]; + /* public: */ #endif /* private: the union with struct page is transitional */ }; @@ -355,6 +365,8 @@ struct folio { unsigned long _head_2a; /* public: */ struct list_head _deferred_list; + /* private: 8 more free bytes for either 32/64 bits */ + unsigned char _free_2_1[8]; /* private: the union with struct page is transitional */ }; struct page __page_2; From patchwork Mon Aug 14 18:44:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13353179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 447C5EB64DD for ; Mon, 14 Aug 2023 18:44:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23807900004; Mon, 14 Aug 2023 14:44:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 14BC2900003; Mon, 14 Aug 2023 14:44:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB800900004; Mon, 14 Aug 2023 14:44:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DAB1B900003 for ; Mon, 14 Aug 2023 14:44:22 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8019E1A0C96 for ; Mon, 14 Aug 2023 18:44:22 +0000 (UTC) X-FDA: 81123585564.20.8981D0D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 6C6162001F for ; Mon, 14 Aug 2023 18:44:20 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HV5JE647; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692038660; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PXZG/qw2ayWAI5P15GI+5RH7awNhex4k3sDbvA4UjWg=; b=TluH4epoQGaG5TOJ+dKQS4JT7wwSO8ezH4XKQBQtqFFScWSVzJqpXVrqjn7a6g+LJMcsGP LSLfRPfUJIFv6RIw5NtJBCogDwi1F5mSME4BSgPJIfx7VwzlcUeFmOfWvVDGiAPJ2TwdEL bUZZlTJ2Q1QlHRIDL29378wNLCEYaZQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692038660; a=rsa-sha256; cv=none; b=fHn9QsHfx0JFIpsovkkcwEwwkqxDlEV5QezvsxYFh07sBsPp9jUzJYUkP2FWzzFrNtzqql iDUi65+5QdHXpgF7KToLdmzZJ/t284dCAL4RvQaEewxp9E6IfJp+teY+SYDh71c/VFTe7P PvotVSs37fQniizGUOIewohenCqgIVU= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HV5JE647; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1692038659; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PXZG/qw2ayWAI5P15GI+5RH7awNhex4k3sDbvA4UjWg=; b=HV5JE6478oB5UMhfQj/gIez8FUQOPzXtwyBNeOtCSa2m5F218JKadgh8+/eM7QB7NVQ6kZ nAMo5cS7Jz1OI3U7NByWOHGUZ8a+kHVp6ruGGQFYKc99wqNs9xLY/CQg0r0PqtaCNAWyHa xVrD75BB2phShzsqqiRwJYYhP/uDrEw= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-133-e6DJ6oGhN12wqrBPrHPoiw-1; Mon, 14 Aug 2023 14:44:18 -0400 X-MC-Unique: e6DJ6oGhN12wqrBPrHPoiw-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-63d2b88325bso13402066d6.1 for ; Mon, 14 Aug 2023 11:44:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692038657; x=1692643457; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PXZG/qw2ayWAI5P15GI+5RH7awNhex4k3sDbvA4UjWg=; b=ekA+ISMWDU+4MpjKAwoxTU9eLcFeNe+DJ95F9k/mkVlINHuNpD8Dn6SudHWW9+cIZM SSXfzFxYZh1jilRDEb/xU6I01BEQntmT+SXO/n3RQiVckqq3hjgPMqdACdg02ZwGFj4p h2vsL+r+LeT5MbYB0SvPy6V57Sm9yCThZ+15xpO+LvdZrT2uARwROE14rR6tftwSJeLO TkrIWnHVC+mEfXOWoDzk/mEs1AJ1EPc72k32DZELR9jTBcsoj7j9IMRvbrf9vpEpDK8N X0cW5mkLi9eDFrE3a67LFozQ5Hmx9btYeHu3ZJXow4e2hlgb7KzY9Wq0mqaSdN3EPzQA aT0Q== X-Gm-Message-State: AOJu0Yy11yKx7eBg6vQUsT/yr0B39jr3HuxB9j89qOzofS9tHuT2Yfck 2uQPKXjtOyNhvzUjw0Wv54whNa2y3dvsPJBD943KjfPvH7Vh5dhLL/6A3SHhdGqmGhqpW96AcEA AdqklakGu7m9eRgQX3p+4cahR44dbKTnl/KNjJg59xZchmt5VdThb52/JOjQ9o0938iYH X-Received: by 2002:a05:6214:1d09:b0:635:d9d0:cccf with SMTP id e9-20020a0562141d0900b00635d9d0cccfmr14722037qvd.4.1692038657582; Mon, 14 Aug 2023 11:44:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGNyKVLMZ66Hr3NQvsgtXcZ9D0YXzjmGtFWrY2Aeh8Bf9t6sbpvwCpEe2tOq//nQtrBk7xUeg== X-Received: by 2002:a05:6214:1d09:b0:635:d9d0:cccf with SMTP id e9-20020a0562141d0900b00635d9d0cccfmr14722010qvd.4.1692038657287; Mon, 14 Aug 2023 11:44:17 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id a7-20020a0cb347000000b00630c0ed6339sm3566561qvf.64.2023.08.14.11.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Aug 2023 11:44:16 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , David Hildenbrand , Matthew Wilcox , Andrew Morton , peterx@redhat.com, Yu Zhao , Ryan Roberts , Yang Shi , Hugh Dickins , "Kirill A . Shutemov" Subject: [PATCH RFC v2 3/3] mm: Proper document tail pages fields for folio Date: Mon, 14 Aug 2023 14:44:11 -0400 Message-ID: <20230814184411.330496-4-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230814184411.330496-1-peterx@redhat.com> References: <20230814184411.330496-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 6C6162001F X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: roze3opoggagywqfioprq177gwcm88nz X-HE-Tag: 1692038660-398122 X-HE-Meta: U2FsdGVkX18W16QMTm6m4aLvp2v3c+nEA7VN7Yd/hhfG+kt8/6BOmY9umtZgn3bKJDnD8Qipy/9QET5fXjYfOpPnK1AHxfv1ffQqxON5NoFHCNv7DqA6O4kYVvpzRgspPfXPI//xMcLaCXBuEa6otkC9tUlx12eZ7236ZB1mfPfUz/LXXM9hcmot5MZjPoHhmRT9q6DkUmVJJbEUIWohVRWw6GRsreI/SWNxdJwFr7HkdQwjcNf29tbtZ5PjhJ2a4f0gYgBXwaKH+58ZTJ0qhRBnYxlSdtsg0seIJ5rB8EkMqvvohLDiTNmqkruoMEcaqecjiZl7947tKMiftIBeKhvY7hLl/9iV2MFjMIJOe+EoDQNneVQo1Z+PcPVOeJpklcQRDg5U+4X2XgugZ6Wb2i1pYi5jQvGHF9uMAXFd1amgTn5CeU7OF8DYcbQ3CXds/BmZ/24uriZ5i4uOx0A4iu+oUl+IsSRz7apxVx227Ak32RJWKBfUzmrwvXT5TFIsX9dQBK4Kvp9ZbmsYVUD38BqZ7no0mvMsnvYn6jxlyzMVcORCDkLCfYbCMEOTthmDPCO/k3Dr7kbs+e/FZJZ30N8M2WeXVmdcz04UOfhQcfgFQeRRfhcKsX0GyzhkrWBgQ4drSvLZdUsqm65QYoSiHL8yz8lQTMkH4/qSJYY6k/r5HMipeUMstuHTsj3l2wOO3J4bezuuQiqnKm14U0wPpZA0Y6rg3sqZwHqM0FrImv8Z44CJRU2prIJpaTujOq2aM/XIbm8lWdHG9LlD15rf++dk38jdspDUnyEEObKNlzELMUXv+6ahk3EDwj8upn+0mVZLrDwFEymx8mr9owPfmjomwn4pcFwWGoCJ4QslHA2N9MzAMZIMqPn3+4uQGdxzW8NMw1tBh1lYHnBL0ieEmLqZaMWPcAvA97ypoiZZhty1c2J1QX/9N5mS5uxPRw3Q2xgpC8InhxC962seUV6 ffxkwYPk sOPHIbrtdY2g5mVhb9vfvqWs5ZRnq6hLMegR1/nEcDnYEdoLcrU24M8lU63yhU3DlcSyTpV9uKZBAArQkifPpUPwfrbdj+NT7z7I+2xd66ZFOycv6o7xb5Vvm6W1Z2LrzgUEO1bJC5jUfMg5j/HfPg8ig67Kn3x7GOe+XqUmo5ztSRiJfOnUjerSROcSX6/zCrgxEjf4hNeh3X7TG3U+IfMfF0CV/23DrVVE4w5Dm0+OtT7gHpRHBsp6oHnR4ZsUobNqRU2+EyB38VOllNak9nBJqesxaCTWNU7W8NilQVAx6TXhcxwURcXWNlMEctmtvaKKelJNnXruaidwvn+nUX0c0c9tuxtC3Ni58U210hNfsTZ2GTzHO/7nQlIBHLh7GP5CAxrx6umj+Ylo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Tail page struct reuse is over-comlicated. Not only because we have implicit uses of tail page fields (mapcounts, or private for thp swap support, etc., that we _may_ still use in the page structs, but not obvious the relationship between that and the folio definitions), but also because we have 32/64 bits layouts for struct page so it's unclear what we can use and what we cannot when trying to find a new spot in folio struct. We also have tricks like page->mapping, where we can reuse only the tail page 1/2 but nothing more than tail page 2. It is all mostly hidden, until someone starts to read into a VM_BUG_ON_PAGE() of __split_huge_page_tail(). It's also unclear on how many fields we can reuse for a tail page. The real answer is (after help from Matthew): we have 7 WORDs guaranteed on 64 bits and 8 WORDs on 32 bits. Nothing more than that is guaranteed to even exist. Let's document it clearly on what we can use and what we can't when extending folio on reusing tail page fields, with 100% explanations on each of them. Hopefully after the doc update it will make it easier when: (1) Any reader to know exactly what field is where and for what, the relationships between folio tail pages and struct page definitions, (2) Any potential new fields to be added to a large folio, so we're clear which field one can still reuse. This is assuming WORD is defined as sizeof(void *) on any archs, just like the other comment in struct page we already have. Signed-off-by: Peter Xu --- include/linux/mm_types.h | 41 ++++++++++++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 6 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 829f5adfded1..9c744f70ae84 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -322,11 +322,40 @@ struct folio { }; struct page page; }; + /* + * Some of the tail page fields may not be reused by the folio + * object because they're already been used by the page struct. On + * 32bits there're at least 8 WORDs while on 64 bits there're at + * least 7 WORDs: + * + * |--------+-------------+-------------------| + * | index | 32 bits | 64 bits | + * |--------+-------------+-------------------| + * | 0 | flags | flags | + * | 1 | head | head | + * | 2 | FREE | FREE | + * | 3 | FREE [1] | FREE [1] | + * | 4 | FREE | FREE | + * | 5 | FREE | private [2] | + * | 6 | mapcnt | mapcnt+refcnt [3] | + * | 7 | refcnt [3] | | + * |--------+-------------+-------------------| + * + * [1] "mapping" field. It is free to use but needs to be with + * some caution due to poisoning, see TAIL_MAPPING_REUSED_MAX. + * + * [2] "private" field, used when THP_SWAP is on (but disabled on + * 32 bits, so this index is FREE on 32bit or hugetlb folios). + * May need to be fixed finally. + * + * [3] "refcount" field must be zero for all tail pages. See e.g. + * has_unmovable_pages() on page_ref_count() check and comment. + */ union { struct { unsigned long _flags_1; unsigned long _head_1; - /* public: */ + /* public: WORD 2 */ unsigned char _folio_dtor; unsigned char _folio_order; /* private: 2 bytes can be reused later */ @@ -335,7 +364,7 @@ struct folio { /* 4 bytes can be reused later (64 bits only) */ unsigned char _free_1_1[4]; #endif - /* public: */ + /* public: WORD 3 */ atomic_t _entire_mapcount; atomic_t _nr_pages_mapped; atomic_t _pincount; @@ -350,20 +379,20 @@ struct folio { struct page __page_1; }; union { - struct { + struct { /* hugetlb folios */ unsigned long _flags_2; unsigned long _head_2; - /* public: */ + /* public: WORD 2 */ void *_hugetlb_subpool; void *_hugetlb_cgroup; void *_hugetlb_cgroup_rsvd; void *_hugetlb_hwpoison; /* private: the union with struct page is transitional */ }; - struct { + struct { /* non-hugetlb folios */ unsigned long _flags_2a; unsigned long _head_2a; - /* public: */ + /* public: WORD 2-3 */ struct list_head _deferred_list; /* private: 8 more free bytes for either 32/64 bits */ unsigned char _free_2_1[8];