From patchwork Mon Feb 24 21:59:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luiz Capitulino X-Patchwork-Id: 13989004 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 611DDC021B8 for ; Mon, 24 Feb 2025 21:59:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EF477280010; Mon, 24 Feb 2025 16:59:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EA34828000D; Mon, 24 Feb 2025 16:59:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D1FAD280010; Mon, 24 Feb 2025 16:59:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id AD45428000D for ; Mon, 24 Feb 2025 16:59:28 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 55C0B4A21E for ; Mon, 24 Feb 2025 21:59:28 +0000 (UTC) X-FDA: 83156205216.30.951C0C8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 918841C0006 for ; Mon, 24 Feb 2025 21:59:26 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hr5RFEaa; spf=pass (imf18.hostedemail.com: domain of luizcap@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740434366; a=rsa-sha256; cv=none; b=K7DuM8WB2rlleqzOUqsDTJKVIKaXHiJEOwy/XM77EbDOoyFN5f1fdA1+enSbxivxOgISvi bWJ3Nt5Uv6kbsIs5cR7WxbrfLepV4KH4XSHQLHgg2IX9cX5sojTlf6Mq4gInARryaFtibL 6gW6WW12qQLv1Ili95P2RhQbeZ1s9hw= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hr5RFEaa; spf=pass (imf18.hostedemail.com: domain of luizcap@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740434366; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FtYeaQb9yjWDAqIxpFJ3I+Fkvf55cMTzB8yThSJBOoQ=; b=fotZ7p60LL/BJa9iK46fe9QR8VrWimHDAUm4faacuGdA7q4IFnaHLaeNV/YUeufWUuy/4a ih/5xbDBNXAarVxsem+U3wcMOAI6j4PQbxF8nkrFzIW+sbcVX8ZcANRqe+068UazvJlXFo n3qJ6f/LJzngwR7wjR6sfMN018eRQmk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740434365; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FtYeaQb9yjWDAqIxpFJ3I+Fkvf55cMTzB8yThSJBOoQ=; b=hr5RFEaaGhA5LF6hApxCLr88IzXql8nPA6fG+CcGKcrjGI1qf+qpuk0X1sFCm4QwV0fi9F CXoGO6KSVgbWE9FHOr4CqnrEnq9eV/5AF6IuUSWHSxzf9j0t007j2180S8AJaNN71hgIyH 1tgZIcCG2ec5ahsS8QmJedBqrDzESM8= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-462-JUYoEci3NZm2AUhf6yksEw-1; Mon, 24 Feb 2025 16:59:22 -0500 X-MC-Unique: JUYoEci3NZm2AUhf6yksEw-1 X-Mimecast-MFC-AGG-ID: JUYoEci3NZm2AUhf6yksEw_1740434361 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C20701800373; Mon, 24 Feb 2025 21:59:20 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.22.65.50]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 6746D1800359; Mon, 24 Feb 2025 21:59:18 +0000 (UTC) From: Luiz Capitulino To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, david@redhat.com, yuzhao@google.com, pasha.tatashin@soleen.com Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, muchun.song@linux.dev, luizcap@redhat.com Subject: [PATCH v2 1/4] mm: page_ext: make lookup_page_ext() public Date: Mon, 24 Feb 2025 16:59:05 -0500 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 X-Rspamd-Queue-Id: 918841C0006 X-Stat-Signature: xwkoody59nogmztoe5gdqza7b3rpo819 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1740434366-497410 X-HE-Meta: U2FsdGVkX1+6Cn044k45F6QrftOMDyiqrS9wyc5DWybB42m5F30bHXHQYha9IvuB7+yYNYtc+qqnt45ducgouSMqdHOUsU+xWgIqs+ajnasHOfjfpBkMrBR6gZDJ7gb58+ARUr+ch5UCk5tYH7HTH2Npm+X7QkO9j/50+os+dPO/ZdJmQCY+jpT2RLg8HeF0kc3+37mWZbfyrnt6C9KKfzqy81TPSlPIBbsmuOPOf+IGh5bExSMF6bncwjLGH5iaBDVAQDc99XW3U3BAOOrPIBFBSNuwBQ0S2/+3oQetW4op60Cj8tyGKj48/YMeKNmK1iyKkVqM8MwFvkqGoJXzXuC8zWmO0wLapdn3i2iU5QwQBwv41mz7ONuEq1Aoyf4IqglDFo8KCrEMgGmXBaRIRK7C8k7WJBvtwmnVtsCpyZ4UXfgWyTnLVMra36bhKo3i6ZhXEkPANMuqmqctDMz7+lnbOIhpdkRdErLaHyggYvS64LSwQ6WQeAyfLeSRILM9bJOxC7hU98v0hS6fP6BB4ZyZGm2pgVrR8Uci3pPHOenYtQQUCqMUfOtyVac39ubiLb1JDQiIkWzeaXKycBm02Fz5epyBV8qg3r6mTmGZy6mlV3XaVQIJrgnSDd+IHWUvS+eoo0eNrimEmkOtvCDvuVCG5IjmEoitdr7J+ME4ytPVUKC+Ttvx8kba2fEihez8Q/1zhwkCtBpkO2dH5tqsseUinLe/E6NmQmf1IvvijtNHCWasmlQfCAgrd/Wgu6tbnoRQQLHLNRcqeaFwfWIwB6KNQnRbbGDyeX8kwIsigA+LgSE3zsAM/vbjfrozC/A/eAG//OZ1m2sWhEI55JX540DYkDEPqlly/wVWfw6uOu+Eaxy0+71zKcKz+LshU15bRL0xOhGSPwomaQhf95gn8WbbtWAqIdo1mYd5g+sfR/wjAkUftPXoOp2mmbhQKIyT54pnibp6YU6yrDxB4/l ++kDIIQ5 wd3C3czYUjOeF2XedYpr3sEVC6GbtLAMUoEw9SK60GWoP+GZZSI2F2ZMXYS+kngwcAFqXJupN9MzNi3L1R7W9qcjrucgC8MBS5gYbgV7nPEcEzGMEguV/XkG1FERQdhG1dMDQEPfrmVLfAF/UMbHMIZlC0VBx8871BAl0VXf/MU/UN+EmeL1jWvtZ0YXKiGbFUXNUBVXGNIa/a6OY+P++LqGWNK1uDdpAcW7s+a5bXtddW5V78Zx7PAHNrRbZFDRvnXXlMATFXktM9ftJuBa6E8hQ4SzJ2eO3jASUOn1gQuC15KY9WA+i19slR1g2GmMQD+OyD0Xrnsco4uIR9l8wUcLI2rqad0cZSeEkUvHfhBnLBF7Xy0Gl2YKL+g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The next commit will use it. Signed-off-by: Luiz Capitulino --- include/linux/page_ext.h | 1 + mm/page_ext.c | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index e4b48a0dda244..d6fb891c51d1d 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -79,6 +79,7 @@ static inline void page_ext_init(void) extern struct page_ext *page_ext_get(const struct page *page); extern void page_ext_put(struct page_ext *page_ext); +extern struct page_ext *lookup_page_ext(const struct page *page); static inline void *page_ext_data(struct page_ext *page_ext, struct page_ext_operations *ops) diff --git a/mm/page_ext.c b/mm/page_ext.c index 641d93f6af4c1..23ad30597c05c 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -165,7 +165,7 @@ void __meminit pgdat_page_ext_init(struct pglist_data *pgdat) pgdat->node_page_ext = NULL; } -static struct page_ext *lookup_page_ext(const struct page *page) +struct page_ext *lookup_page_ext(const struct page *page) { unsigned long pfn = page_to_pfn(page); unsigned long index; @@ -245,7 +245,7 @@ static bool page_ext_invalid(struct page_ext *page_ext) return !page_ext || (((unsigned long)page_ext & PAGE_EXT_INVALID) == PAGE_EXT_INVALID); } -static struct page_ext *lookup_page_ext(const struct page *page) +struct page_ext *lookup_page_ext(const struct page *page) { unsigned long pfn = page_to_pfn(page); struct mem_section *section = __pfn_to_section(pfn); From patchwork Mon Feb 24 21:59:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luiz Capitulino X-Patchwork-Id: 13989005 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37428C021B6 for ; Mon, 24 Feb 2025 21:59:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C0E56280011; Mon, 24 Feb 2025 16:59:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B971928000D; Mon, 24 Feb 2025 16:59:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A102B280011; Mon, 24 Feb 2025 16:59:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7CA0A28000D for ; Mon, 24 Feb 2025 16:59:31 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 319BBC0204 for ; Mon, 24 Feb 2025 21:59:31 +0000 (UTC) X-FDA: 83156205342.06.C29CAE1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 67F7380006 for ; Mon, 24 Feb 2025 21:59:29 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Q9yhUbfW; spf=pass (imf02.hostedemail.com: domain of luizcap@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740434369; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jIuVuZi/rT8xS75r5ZQPMK2tcpRyNkRr1DXyoirL1iM=; b=IdhooHcUFpHfdIr5S+DtZIYVA2m5hvcdvWQz2yOl3KwNPQPGBG47XRkB/Fnb3ccp4wBS3g FgxS1qrXjfD5fUEn9ndYB6wIjc1U/sWPIRAif67QdxCkK7BL6P0xpWjl1PlzZ/1289uBM6 JOXdVKhDLE0gBFms7oKjEf4nGlWI0P4= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Q9yhUbfW; spf=pass (imf02.hostedemail.com: domain of luizcap@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740434369; a=rsa-sha256; cv=none; b=VXgtnPvv+pQqnJ0emUN5FlkVj+/XszJFGVgl8XUQXjYa9G7W3lnbBfPFju6VJMn5zaYHgp ndYF/uF+UsqXmA8EMR3P7EftcBiKYDFrI7TeydyMsxZgj22AoCMCrsm633fq3RK5PRpLId BrwLorGzxKPv94NzixR00UsF3D/wFkE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740434368; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jIuVuZi/rT8xS75r5ZQPMK2tcpRyNkRr1DXyoirL1iM=; b=Q9yhUbfWszy8ceFqFKdzqccLMhXkMYtPlVI7zwHyecQpxZRjFKftUT2A29LUvZKBzep2ZJ nu6kMiLoAj1lIhfRmo/LaCE5rIrX3GgfV+oVSOk1XpILJ7MZE6jnYqPUZhW1O1+dKd7DQy RhkAqGSeRzPbcqBpkrz10xB93ttwQ48= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-323-eAx3mHagOqq5Asig4w3e0Q-1; Mon, 24 Feb 2025 16:59:24 -0500 X-MC-Unique: eAx3mHagOqq5Asig4w3e0Q-1 X-Mimecast-MFC-AGG-ID: eAx3mHagOqq5Asig4w3e0Q_1740434363 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 3982F1800872; Mon, 24 Feb 2025 21:59:23 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.22.65.50]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 18C811800368; Mon, 24 Feb 2025 21:59:20 +0000 (UTC) From: Luiz Capitulino To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, david@redhat.com, yuzhao@google.com, pasha.tatashin@soleen.com Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, muchun.song@linux.dev, luizcap@redhat.com Subject: [PATCH v2 2/4] mm: page_ext: add an iteration API for page extensions Date: Mon, 24 Feb 2025 16:59:06 -0500 Message-ID: <08aea5d87f5419f4c7033c81d97645f940f87f7e.1740434344.git.luizcap@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 67F7380006 X-Stat-Signature: xxb43c61hycyhcf8b6jmngd1mtt3nbzx X-Rspam-User: X-HE-Tag: 1740434369-774920 X-HE-Meta: U2FsdGVkX19R1NzB/VsPNWoYGCA0nMd0y1IDxsTzC3UrsJKmh63LDBMGrP4kLmcdunsi+UIF324Y5VZRm7uWH89h9z4Z4viiPTsbSQsrOBx7Am8QIrwRgoH8p+xDctawiuIKkS3crt/piTfjqVDxb6p3YKqiiTYa8sKTllmQ08lZt2YCpHGrJ6Gve+UoZf/Z8dZJ7bIMAGaucU2jQkvKNCsuEv5SsoGlZUD+uJWs0Ss3QXUUdXpqcmIQG4DKWowtRbNQ3pH2r0FGbjYPPXj5OehkAIVjhHNRGl5WEpvkPm1K+vAhhB10HA+RcdGrLuBR6PdJ76+ZuSR5gspLhD5da4i5PRADD4rw1brX5DSJ0mBPI5KLGE2ESpemn+nhg2JppC4JyzA9yiBW+uEFWcyG0BvFIm+tGBf9qdskAsKUHRnDEzhFrOoUpxRKSiHFUZjaHQO17RkL5s5sCpM5D71u9OrAbveM1F8+eZquK2ltU4NEOdfoL/xPUN/6f1Bg0qbjNccwA1eTtonHpXf57zn5gGAbt/X5LVTgoD8pRbpU0dqPYu/r4mJ6gBtw/h6h6/HTFmkQaOslEbySkFkKyI/xYLcKReONrBItGDv/pd79zLEpNyhjbtFoLPXE5brhUZws/Jw50yfaRXLbXqsU6iSbINOk1OHay8fN3Sx7YEht9n6hHuymgt3XshSEMN/dSgPeRwc8AST8qhisLaCpM49JcNS8g16wumFUc4ZX5GYYVA1a44LrraihwbS1tauLk2xDwaOBlBHmLVGjhExY2PK2aHfX5MSpAx9l8YCMX33bZ0bN8ySuFpAuweLAXUk++2TSKPwOlZr602oa5M/3k/Y95cVQs0wqCRHe/+w9NAtrV8XAYy99nCX38nO/DDhrsi1rLNTvCbgx9FbfLjLMPL6cwxZT9oIMbSsNXWNdDYtdKdZSVEKG0IWeJDly0FcDNPZwOpmi+U8biHaQxZo90AB 5APVhBRC 4jq9HnnETri5duT4LZB2XU0BOQbf094yrD/UCRNYWRfxqjrwzccRIomttm7yl0h45ZghfKnokMvHMXBizbAYkzrCOLtYWB1AocF4DAwTPabyeU+SvaBK5xnh4OEBV2wC1nw3p7rrqkYuAz7OkO7fqNWLD0qonixP7v3c17sB3u0Q85NpyPj4GNG596gq6ANduydsPiBWkhGgzwe1E59dk/DDbn2BocRpjH+IWXAuwbHZklJvOPFngFBc1qieliDoJZcIbHCw5boQBdiyWS4a+0bl3VrF8KBOc5Q90SajZWKBIVLBbyqDdzKtcG2VxuVzn0csVQ6uO4LHVaSARmBDI2mikGofJ02BxdAjz9kXhWGbV+qKGIXjP1MIWdw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The page extension implementation assumes that all page extensions of a given page order are stored in the same memory section. The function page_ext_next() relies on this assumption by adding an offset to the current object to return the next adjacent page extension. This behavior works as expected for flatmem but fails for sparsemem when using 1G pages. The commit cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") exposes this issue, making it possible for a crash when using page_owner or page_table_check page extensions. The problem is that for 1G pages, the page extensions may span memory section boundaries and be stored in different memory sections. This issue was not visible before commit cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") because alloc_contig_pages() never passed more than MAX_PAGE_ORDER to post_alloc_hook(). However, the series introducing mentioned commit changed this behavior allowing the full 1G page order to be passed. Reproducer: 1. Build the kernel with CONFIG_SPARSEMEM=y and table extensions support 2. Pass 'default_hugepagesz=1 page_owner=on' in the kernel command-line 3. Reserve one 1G page at run-time, this should crash (backtrace below) To address this issue, this commit introduces a new API for iterating through page extensions. The main iteration macro is for_each_page_ext() and it must be called with the RCU read lock taken. Here's an usage example: """ struct page_ext_iter iter; struct page_ext *page_ext; ... rcu_read_lock(); for_each_page_ext(page, 1 << order, page_ext, iter) { struct my_page_ext *obj = get_my_page_ext_obj(page_ext); ... } rcu_read_unlock(); """ The loop construct uses page_ext_iter_next() which checks to see if we have crossed sections in the iteration. In this case, page_ext_iter_next() retrieves the next page_ext object from another section. Thanks to David Hildenbrand for helping identify the root cause and providing suggestions on how to fix and optmize the solution (final implementation and bugs are all mine through). Lastly, here's the backtrace, without kasan you can get random crashes: [ 76.052526] BUG: KASAN: slab-out-of-bounds in __update_page_owner_handle+0x238/0x298 [ 76.060283] Write of size 4 at addr ffff07ff96240038 by task tee/3598 [ 76.066714] [ 76.068203] CPU: 88 UID: 0 PID: 3598 Comm: tee Kdump: loaded Not tainted 6.13.0-rep1 #3 [ 76.076202] Hardware name: WIWYNN Mt.Jade Server System B81.030Z1.0007/Mt.Jade Motherboard, BIOS 2.10.20220810 (SCP: 2.10.20220810) 2022/08/10 [ 76.088972] Call trace: [ 76.091411] show_stack+0x20/0x38 (C) [ 76.095073] dump_stack_lvl+0x80/0xf8 [ 76.098733] print_address_description.constprop.0+0x88/0x398 [ 76.104476] print_report+0xa8/0x278 [ 76.108041] kasan_report+0xa8/0xf8 [ 76.111520] __asan_report_store4_noabort+0x20/0x30 [ 76.116391] __update_page_owner_handle+0x238/0x298 [ 76.121259] __set_page_owner+0xdc/0x140 [ 76.125173] post_alloc_hook+0x190/0x1d8 [ 76.129090] alloc_contig_range_noprof+0x54c/0x890 [ 76.133874] alloc_contig_pages_noprof+0x35c/0x4a8 [ 76.138656] alloc_gigantic_folio.isra.0+0x2c0/0x368 [ 76.143616] only_alloc_fresh_hugetlb_folio.isra.0+0x24/0x150 [ 76.149353] alloc_pool_huge_folio+0x11c/0x1f8 [ 76.153787] set_max_huge_pages+0x364/0xca8 [ 76.157961] __nr_hugepages_store_common+0xb0/0x1a0 [ 76.162829] nr_hugepages_store+0x108/0x118 [ 76.167003] kobj_attr_store+0x3c/0x70 [ 76.170745] sysfs_kf_write+0xfc/0x188 [ 76.174492] kernfs_fop_write_iter+0x274/0x3e0 [ 76.178927] vfs_write+0x64c/0x8e0 [ 76.182323] ksys_write+0xf8/0x1f0 [ 76.185716] __arm64_sys_write+0x74/0xb0 [ 76.189630] invoke_syscall.constprop.0+0xd8/0x1e0 [ 76.194412] do_el0_svc+0x164/0x1e0 [ 76.197891] el0_svc+0x40/0xe0 [ 76.200939] el0t_64_sync_handler+0x144/0x168 [ 76.205287] el0t_64_sync+0x1ac/0x1b0 Fixes: cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") Signed-off-by: Luiz Capitulino --- include/linux/page_ext.h | 92 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index d6fb891c51d1d..33a118b31a222 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -2,7 +2,9 @@ #ifndef __LINUX_PAGE_EXT_H #define __LINUX_PAGE_EXT_H +#include #include +#include #include struct pglist_data; @@ -69,12 +71,26 @@ extern void page_ext_init(void); static inline void page_ext_init_flatmem_late(void) { } + +static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn) +{ + /* + * page_ext is allocated per memory section. Once we cross a + * memory section, we have to fetch the new pointer. + */ + return next_pfn % PAGES_PER_SECTION; +} #else extern void page_ext_init_flatmem(void); extern void page_ext_init_flatmem_late(void); static inline void page_ext_init(void) { } + +static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn) +{ + return true; +} #endif extern struct page_ext *page_ext_get(const struct page *page); @@ -94,6 +110,82 @@ static inline struct page_ext *page_ext_next(struct page_ext *curr) return next; } +struct page_ext_iter { + unsigned long index; + unsigned long start_pfn; + struct page_ext *page_ext; +}; + +/** + * page_ext_iter_begin() - Prepare for iterating through page extensions. + * @iter: page extension iterator. + * @page: The page we're interested in. + * + * Must be called with RCU read lock taken. + * + * Return: NULL if no page_ext exists for this page. + */ +static inline struct page_ext *page_ext_iter_begin(struct page_ext_iter *iter, struct page *page) +{ + iter->index = 0; + iter->start_pfn = page_to_pfn(page); + iter->page_ext = lookup_page_ext(page); + + return iter->page_ext; +} + +/** + * page_ext_iter_next() - Get next page extension + * @iter: page extension iterator. + * + * Must be called with RCU read lock taken. + * + * Return: NULL if no next page_ext exists. + */ +static inline struct page_ext *page_ext_iter_next(struct page_ext_iter *iter) +{ + unsigned long pfn; + + if (WARN_ON_ONCE(!iter->page_ext)) + return NULL; + + iter->index++; + pfn = iter->start_pfn + iter->index; + + if (page_ext_iter_next_fast_possible(pfn)) + iter->page_ext = page_ext_next(iter->page_ext); + else + iter->page_ext = lookup_page_ext(pfn_to_page(pfn)); + + return iter->page_ext; +} + +/** + * page_ext_iter_get() - Get current page extension + * @iter: page extension iterator. + * + * Return: NULL if no page_ext exists for this iterator. + */ +static inline struct page_ext *page_ext_iter_get(const struct page_ext_iter *iter) +{ + return iter->page_ext; +} + +/** + * for_each_page_ext(): iterate through page_ext objects. + * @__page: the page we're interested in + * @__pgcount: how many pages to iterate through + * @__page_ext: struct page_ext pointer where the current page_ext + * object is returned + * @__iter: struct page_ext_iter object (defined in the stack) + * + * IMPORTANT: must be called with RCU read lock taken. + */ +#define for_each_page_ext(__page, __pgcount, __page_ext, __iter) \ + for (__page_ext = page_ext_iter_begin(&__iter, __page); \ + __page_ext && __iter.index < __pgcount; \ + __page_ext = page_ext_iter_next(&__iter)) + #else /* !CONFIG_PAGE_EXTENSION */ struct page_ext; From patchwork Mon Feb 24 21:59:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luiz Capitulino X-Patchwork-Id: 13989006 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3700AC021A4 for ; Mon, 24 Feb 2025 21:59:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92AF1280013; Mon, 24 Feb 2025 16:59:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 83D7B28000D; Mon, 24 Feb 2025 16:59:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CB3F280013; Mon, 24 Feb 2025 16:59:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3AF6928000D for ; Mon, 24 Feb 2025 16:59:33 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id DEFBE1A0240 for ; Mon, 24 Feb 2025 21:59:32 +0000 (UTC) X-FDA: 83156205384.23.43879D5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 2CA6B4000C for ; Mon, 24 Feb 2025 21:59:31 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WjEEr0LO; spf=pass (imf27.hostedemail.com: domain of luizcap@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740434371; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1ZnmS/k+dHYbcrS2T4gnKYBLuSvALIoKj3mKYh0gqWc=; b=65k9j2cpvrSqTsuuSH+xfySAWSCF9+BdgZoMe95M7t+fE9QMIFusTlk5iuePfzwfR1oo1M qm28B4EIRaOdDZ3cqFRNayO3MaIaMOAbCabxOS4jrujhnpyHv7HCqM2CnqHe1fRJr061l9 ht2iBK6yDERkN9Af3KZ4LQSCP2zX3kM= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WjEEr0LO; spf=pass (imf27.hostedemail.com: domain of luizcap@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740434371; a=rsa-sha256; cv=none; b=FNu/tOzuqlDWE6GEaYE3nvxAplNxWkjSJC0VXELpLDhVaKtJHahYQzHcVuGGCyv52ev8Tw R+7P9diTb2nNccvRFX0LGVj3cYtPCgKU6Ocz8cSA4Qd+K6B1A4ZLn6qtvqaqMbgOkTFnrZ kWA7W6vfKkdve/VOw6FUXBX7vvz8Ym4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740434370; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1ZnmS/k+dHYbcrS2T4gnKYBLuSvALIoKj3mKYh0gqWc=; b=WjEEr0LOLIoqFQ/gakrj4kJKmgUHn/HMXKmNZCmiOWNm8uezWir/EKgKZN81bJmhAOrdMZ oYMTYJFw26gPApMqffnozXPv+0/NM/RJxgBgokcSHZdcq8QcPlNzTpDYXZyEzL8hDg29IT coO57BVSd4FLqHBjFUZab+vx3uHrTxM= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-93-d9MzX_ROPZSFbvEmazSggQ-1; Mon, 24 Feb 2025 16:59:26 -0500 X-MC-Unique: d9MzX_ROPZSFbvEmazSggQ-1 X-Mimecast-MFC-AGG-ID: d9MzX_ROPZSFbvEmazSggQ_1740434365 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 76ACE18EB2C3; Mon, 24 Feb 2025 21:59:25 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.22.65.50]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 7E206180035F; Mon, 24 Feb 2025 21:59:23 +0000 (UTC) From: Luiz Capitulino To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, david@redhat.com, yuzhao@google.com, pasha.tatashin@soleen.com Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, muchun.song@linux.dev, luizcap@redhat.com Subject: [PATCH v2 3/4] mm: page_table_check: use new iteration API Date: Mon, 24 Feb 2025 16:59:07 -0500 Message-ID: <30d246e83e73073451dc3d5fe189b733afbefd07.1740434344.git.luizcap@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 2CA6B4000C X-Stat-Signature: 5zjkpc6b5txuyeoq83fennb9qkr3u5j3 X-HE-Tag: 1740434371-649694 X-HE-Meta: U2FsdGVkX18GwqNtVfzx9IFuX/TlCfWPsGQsc8BC8RdaAJgJvU+cYBXLIOAZ/uvl/wq21BzFMtVa/FCLTBvW/Uh5UCBIL8nYQtF57AujCoNyXKhafvLiq1ZDVaNHpso537DfiVwzwTmNus9dv3P+XNoPsQwEoRlHg+FjWafsAEDa4LRZtYmIGM8062e1AqbPtqoVzkFM1im1xQoNAbHwDZejbXveAMrXnfUJQOg7FtfjsCeMlvD8TOYIPFByvFPnuT5JM7QG0JlZVDzSzE714SZ477MAouIKC7bP2xHv+qmdNnYCD1T7bccq9tzD5wR+wQyOhQRFzXKhG5vp5/PBUS0nKIe7IACoSvkDR8WznK2SK30wTXhA1eUo3rkF6xuh5QhFANSijfv08xqrOQgap8ejs3g1AOQ7QncgAtTWkgBzenwjRQUHQcWeMfGqi8hMRaPbnDuDMoWX9Asd8gs+a8D0kJm3kn0vnUrGmP2KWKOns5mrlzeWivbHYZ6lntfbJNIqlXhpV4D9n2oSKqNDePQxyWSuaSDVfzYU+aPHFUJnwGyvm4P4p93HI8ZGxpszdCOuVKmYIl+rJLtOqhklpL4Eu6xwe7KIKqtPFBRYVMGyKkkq+rNP/S+SzJURiW/jA8hG4Z38PqEKrfc4feSoUUH1/5qcQrk9Vo5Ksuy+ty+pLKCdM7cVfep8F9+rRtYwcksCH6wzG8phLShAdpqrIliuo/Z+PQryRlr3pcUDbsgK7JvM5LLLwzmO0uHBHJJs8ZWvjAi8thjZUdFOuuB7HltcbcZ3ueRbulYpe4jODYLzWHPzW7kdIyeuUcKVJXAWqXbOQr4+f8R8oCdtE5loWeXlhDtTjMnHvuk2TW2rdcsBgnfQIrCBZhGwiaSmWwJbdldDdqx7dSXRGjWhN3aeUNXyp0ODula8pqYodX2QyFh7jSdV64CTa0/tlQ9FmKmcfRXWTIt2a090vXQi/qO pKl/EFMP ALhlqqq9AyKnciklvqGiv4JQ3XS2HM8Sa7xEyWMZnDc3lIbkEZC1SUexKhTDhe5eYbSpG7wFhyElgmhquKC7+GtWFS6HqbEZIXws0ofRnl+VL+Pws0/reikBOKqSKwn2jZpdYu5FBA9C86Kpv39M92wSv7XsIBKNiUGQ6TuJJzOMlE5FhiHJc786kx/0jn20Pkc28VJ6c2thkeqhyVeSoptagUQHworcy25Z/jKaoXhOCVVQP3wRodRXB9AeMOJTuuP61kdnu2oC/5mDjynoCc+RjRWr3d7u738JzIQsv57bC0yMokmeOcItowEVSGUTTCjUww8hg927Bb1fZuNYLq6Z7iloA5Ko3YZDVb3KFMjZIxGGITA2V04gBxg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The page_ext_next() function assumes that page extension objects for a page order allocation always reside in the same memory section, which may not be true and could lead to crashes. Use the new page_ext iteration API instead. Fixes: cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") Signed-off-by: Luiz Capitulino --- mm/page_table_check.c | 39 ++++++++++++--------------------------- 1 file changed, 12 insertions(+), 27 deletions(-) diff --git a/mm/page_table_check.c b/mm/page_table_check.c index 509c6ef8de400..e11bebf23e36f 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -62,24 +62,20 @@ static struct page_table_check *get_page_table_check(struct page_ext *page_ext) */ static void page_table_check_clear(unsigned long pfn, unsigned long pgcnt) { + struct page_ext_iter iter; struct page_ext *page_ext; struct page *page; - unsigned long i; bool anon; if (!pfn_valid(pfn)) return; page = pfn_to_page(pfn); - page_ext = page_ext_get(page); - - if (!page_ext) - return; - BUG_ON(PageSlab(page)); anon = PageAnon(page); - for (i = 0; i < pgcnt; i++) { + rcu_read_lock(); + for_each_page_ext(page, pgcnt, page_ext, iter) { struct page_table_check *ptc = get_page_table_check(page_ext); if (anon) { @@ -89,9 +85,8 @@ static void page_table_check_clear(unsigned long pfn, unsigned long pgcnt) BUG_ON(atomic_read(&ptc->anon_map_count)); BUG_ON(atomic_dec_return(&ptc->file_map_count) < 0); } - page_ext = page_ext_next(page_ext); } - page_ext_put(page_ext); + rcu_read_unlock(); } /* @@ -102,24 +97,20 @@ static void page_table_check_clear(unsigned long pfn, unsigned long pgcnt) static void page_table_check_set(unsigned long pfn, unsigned long pgcnt, bool rw) { + struct page_ext_iter iter; struct page_ext *page_ext; struct page *page; - unsigned long i; bool anon; if (!pfn_valid(pfn)) return; page = pfn_to_page(pfn); - page_ext = page_ext_get(page); - - if (!page_ext) - return; - BUG_ON(PageSlab(page)); anon = PageAnon(page); - for (i = 0; i < pgcnt; i++) { + rcu_read_lock(); + for_each_page_ext(page, pgcnt, page_ext, iter) { struct page_table_check *ptc = get_page_table_check(page_ext); if (anon) { @@ -129,9 +120,8 @@ static void page_table_check_set(unsigned long pfn, unsigned long pgcnt, BUG_ON(atomic_read(&ptc->anon_map_count)); BUG_ON(atomic_inc_return(&ptc->file_map_count) < 0); } - page_ext = page_ext_next(page_ext); } - page_ext_put(page_ext); + rcu_read_unlock(); } /* @@ -140,24 +130,19 @@ static void page_table_check_set(unsigned long pfn, unsigned long pgcnt, */ void __page_table_check_zero(struct page *page, unsigned int order) { + struct page_ext_iter iter; struct page_ext *page_ext; - unsigned long i; BUG_ON(PageSlab(page)); - page_ext = page_ext_get(page); - - if (!page_ext) - return; - - for (i = 0; i < (1ul << order); i++) { + rcu_read_lock(); + for_each_page_ext(page, 1 << order, page_ext, iter) { struct page_table_check *ptc = get_page_table_check(page_ext); BUG_ON(atomic_read(&ptc->anon_map_count)); BUG_ON(atomic_read(&ptc->file_map_count)); - page_ext = page_ext_next(page_ext); } - page_ext_put(page_ext); + rcu_read_unlock(); } void __page_table_check_pte_clear(struct mm_struct *mm, pte_t pte) From patchwork Mon Feb 24 21:59:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luiz Capitulino X-Patchwork-Id: 13989007 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D9DBC021A4 for ; Mon, 24 Feb 2025 21:59:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E5D8D280014; Mon, 24 Feb 2025 16:59:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E0C2828000D; Mon, 24 Feb 2025 16:59:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C5C08280014; Mon, 24 Feb 2025 16:59:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A443628000D for ; Mon, 24 Feb 2025 16:59:36 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 63584C021E for ; Mon, 24 Feb 2025 21:59:36 +0000 (UTC) X-FDA: 83156205552.06.EDD21C7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id A3CE720003 for ; Mon, 24 Feb 2025 21:59:34 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MlIThKYl; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf03.hostedemail.com: domain of luizcap@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740434374; a=rsa-sha256; cv=none; b=Vy2hZSfypZphadtOukv4WTMNp+3+rYz3/YoDeZLPsu/lfF64HV5UBEjCtmcjt2Q/s3vOg+ WhojGP8+eQahz5I0PukXzkmoR4Gxc4YmTZ9lzj2csHuzzTn/76WXubZmE2ieghK/nWsnUQ JoGUP8Mv7+DoukYhZg+zpnl2GZ7DKdg= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MlIThKYl; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf03.hostedemail.com: domain of luizcap@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=luizcap@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740434374; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6w7eedefm4DE2XJ3Yn/BDysgoHUEY0apu1z5eddMX8Y=; b=lsulmTKYVP2MDS8PpU83dA3B5ETKdmzPU2uO4V3vGdHuemcjNodA7Nwix0yDu4uy+OKJDk E6mmM8Gxjo2HO6lxUBLzUfjcN2AHmeEgURHCdU1N50JMeeOJMwkXSyl8ICwMF1YrrRwPqM y+PrcSRBLjTpAzo/RIirSG7UEhQkoPM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740434374; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6w7eedefm4DE2XJ3Yn/BDysgoHUEY0apu1z5eddMX8Y=; b=MlIThKYlqRQXAUODBzZWDgWm+tnsJ3U9iqxIgqpqaO3l0mEQ2/HoNy/uWv0r22bqK25FcS ilWQyLKSzCCI916rr0XeWM4iWsW+HfuayG//3i1pmO2yByc8ij2q6bYhTtLTm/p2uK8B2s 0S5SlvpIE0IAgiy2qNeqr4I5VZZT4o0= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-350-JQXE_8nIOY-3y1mHgNHfUw-1; Mon, 24 Feb 2025 16:59:29 -0500 X-MC-Unique: JQXE_8nIOY-3y1mHgNHfUw-1 X-Mimecast-MFC-AGG-ID: JQXE_8nIOY-3y1mHgNHfUw_1740434368 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id AC91419560B9; Mon, 24 Feb 2025 21:59:27 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.22.65.50]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C092D180035F; Mon, 24 Feb 2025 21:59:25 +0000 (UTC) From: Luiz Capitulino To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, david@redhat.com, yuzhao@google.com, pasha.tatashin@soleen.com Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, muchun.song@linux.dev, luizcap@redhat.com Subject: [PATCH v2 4/4] mm: page_owner: use new iteration API Date: Mon, 24 Feb 2025 16:59:08 -0500 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 X-Rspam-User: X-Rspamd-Queue-Id: A3CE720003 X-Rspamd-Server: rspam12 X-Stat-Signature: chkcjau4y3sie5s6fh56pyd4g53gosuk X-HE-Tag: 1740434374-441928 X-HE-Meta: U2FsdGVkX194LDUiQfl4EwbNB8fxkY3WQZT7ifGdTbk2nP5aDv0pwGjXIcBiQLTY9lxrgoriFqKlViZB+z9sn1W0zHh8YnPvZCPLDR6Efzj/udXgTpT3r/lzaLubHKJIqTsLn8AsEkPMInnNhz2OMc40otiyIToe7ak2wL8ifhi+fYRxHqjALXVDR517WzVjwUzR31zJATp6AUKEebKTKQDT4v0zIVYZ47B91ufwz0WbiGvBw2Jr3DKNYG3z8yR+3KqBGxcefmi/lWsoiMPePe2xhOujaykYSCCl64LBtf8PpDkPx2otqw5pFDa+T6gd5eGTmUuR3gaWY3UDb5rCxOMkUbsDVGPtljx/dCiR2CCyKcFFrjfCA2JKF3oEWmNFCscVBwKc3tj3DoYNwHbnOoct2dM+BKqgT40e8M6btxT8V4UAOBeO2kTzPMuFvQrbu+if17VRiwu545jIaXzM92JO7Wjog1qoGuyAL+SLOvwaQTIUTRCvkIOki1ubAYZA9y6cB75LJ0fzlqqWlSunrnj9Q0LcZupfGenERUUXS5r3qtpE1qksbbv9ik1/MJ2/lA1m1yYTocWfJnPbe6j3paaiS88MJV6V9y3EnkRbOtiotC/tkByP4XMCl1M/9m5qtQdG4wIbkPZS44XP0Zor8EhSrtmx/7qf18VT1IEbJyEWbO9NM7bgwjoFUjlW3XSRx0tiforqH1Ty6+RUwX650XUA7l5bBc8RLJnMCl0vx7N5LxFa242K/WSAdYJyzy6ZI/+zoKuq85kFR9SN5FHr4nOEYTjUVeAfU9RtJTLAKQ0Xpyd3K/yIH4Y2P1LHZiuGWl9dsM6wx9Sfuu2v1iyMeNXZSX83TUUMVjU2xywqLvMBzPoNqp8rXEuVoa3uttPJncBwH5VydCjePjcimLvm8JyFstekKNIoNRgF36Eg6ebHfT4Vd1B1i5nohRurOZAz+CsTVlSqm/JJLzjmutZ XeXI3zsQ TJgICsIuYxvgshBrSOY5KpwLU/dz/HP+sX7efc22aRrf7b19cYkPsO5wPievmpfXUu789sCkiJJJGzz2fSoPsB33tkK1T4L73tD3vKT8UpI1hRWDzoyAzWFsEduWRNHzryzFFnLrBnwARdRqWiz3tHIBJ/KbZPVZpFigktipATAf+qD3g694yNLSdZnN+alWyHNU7ixWhb7vkQDIdwEr/Npq7r97EIAnF9+QCenoyUjzWAWgvnvOAxTitFiEAb20tFpX27eFFavZ+QZroGcL1GRAvlRHp8PTOVL+7ezlhxRcVRIPjYXKAZ7sG8fGCTdryCM1SlViyPd4PVpqDLAlPUT1llhr68CL67TwJ9RgBWaT+jHAx8nl7M8X4AA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The page_ext_next() function assumes that page extension objects for a page order allocation always reside in the same memory section, which may not be true and could lead to crashes. Use the new page_ext iteration API instead. Fixes: cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") Signed-off-by: Luiz Capitulino --- mm/page_owner.c | 61 +++++++++++++++++++++++-------------------------- 1 file changed, 29 insertions(+), 32 deletions(-) diff --git a/mm/page_owner.c b/mm/page_owner.c index 2d6360eaccbb6..c9d2c688eb981 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -229,17 +229,19 @@ static void dec_stack_record_count(depot_stack_handle_t handle, handle); } -static inline void __update_page_owner_handle(struct page_ext *page_ext, +static inline void __update_page_owner_handle(struct page *page, depot_stack_handle_t handle, unsigned short order, gfp_t gfp_mask, short last_migrate_reason, u64 ts_nsec, pid_t pid, pid_t tgid, char *comm) { - int i; + struct page_ext_iter iter; + struct page_ext *page_ext; struct page_owner *page_owner; - for (i = 0; i < (1 << order); i++) { + rcu_read_lock(); + for_each_page_ext(page, 1 << order, page_ext, iter) { page_owner = get_page_owner(page_ext); page_owner->handle = handle; page_owner->order = order; @@ -252,20 +254,22 @@ static inline void __update_page_owner_handle(struct page_ext *page_ext, sizeof(page_owner->comm)); __set_bit(PAGE_EXT_OWNER, &page_ext->flags); __set_bit(PAGE_EXT_OWNER_ALLOCATED, &page_ext->flags); - page_ext = page_ext_next(page_ext); } + rcu_read_unlock(); } -static inline void __update_page_owner_free_handle(struct page_ext *page_ext, +static inline void __update_page_owner_free_handle(struct page *page, depot_stack_handle_t handle, unsigned short order, pid_t pid, pid_t tgid, u64 free_ts_nsec) { - int i; + struct page_ext_iter iter; + struct page_ext *page_ext; struct page_owner *page_owner; - for (i = 0; i < (1 << order); i++) { + rcu_read_lock(); + for_each_page_ext(page, 1 << order, page_ext, iter) { page_owner = get_page_owner(page_ext); /* Only __reset_page_owner() wants to clear the bit */ if (handle) { @@ -275,8 +279,8 @@ static inline void __update_page_owner_free_handle(struct page_ext *page_ext, page_owner->free_ts_nsec = free_ts_nsec; page_owner->free_pid = current->pid; page_owner->free_tgid = current->tgid; - page_ext = page_ext_next(page_ext); } + rcu_read_unlock(); } void __reset_page_owner(struct page *page, unsigned short order) @@ -293,11 +297,11 @@ void __reset_page_owner(struct page *page, unsigned short order) page_owner = get_page_owner(page_ext); alloc_handle = page_owner->handle; + page_ext_put(page_ext); handle = save_stack(GFP_NOWAIT | __GFP_NOWARN); - __update_page_owner_free_handle(page_ext, handle, order, current->pid, + __update_page_owner_free_handle(page, handle, order, current->pid, current->tgid, free_ts_nsec); - page_ext_put(page_ext); if (alloc_handle != early_handle) /* @@ -313,19 +317,13 @@ void __reset_page_owner(struct page *page, unsigned short order) noinline void __set_page_owner(struct page *page, unsigned short order, gfp_t gfp_mask) { - struct page_ext *page_ext; u64 ts_nsec = local_clock(); depot_stack_handle_t handle; handle = save_stack(gfp_mask); - - page_ext = page_ext_get(page); - if (unlikely(!page_ext)) - return; - __update_page_owner_handle(page_ext, handle, order, gfp_mask, -1, + __update_page_owner_handle(page, handle, order, gfp_mask, -1, ts_nsec, current->pid, current->tgid, current->comm); - page_ext_put(page_ext); inc_stack_record_count(handle, gfp_mask, 1 << order); } @@ -344,26 +342,24 @@ void __set_page_owner_migrate_reason(struct page *page, int reason) void __split_page_owner(struct page *page, int old_order, int new_order) { - int i; - struct page_ext *page_ext = page_ext_get(page); + struct page_ext_iter iter; + struct page_ext *page_ext; struct page_owner *page_owner; - if (unlikely(!page_ext)) - return; - - for (i = 0; i < (1 << old_order); i++) { + rcu_read_lock(); + for_each_page_ext(page, 1 << old_order, page_ext, iter) { page_owner = get_page_owner(page_ext); page_owner->order = new_order; - page_ext = page_ext_next(page_ext); } - page_ext_put(page_ext); + rcu_read_unlock(); } void __folio_copy_owner(struct folio *newfolio, struct folio *old) { - int i; struct page_ext *old_ext; struct page_ext *new_ext; + struct page_ext *page_ext; + struct page_ext_iter iter; struct page_owner *old_page_owner; struct page_owner *new_page_owner; depot_stack_handle_t migrate_handle; @@ -381,7 +377,7 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old) old_page_owner = get_page_owner(old_ext); new_page_owner = get_page_owner(new_ext); migrate_handle = new_page_owner->handle; - __update_page_owner_handle(new_ext, old_page_owner->handle, + __update_page_owner_handle(&newfolio->page, old_page_owner->handle, old_page_owner->order, old_page_owner->gfp_mask, old_page_owner->last_migrate_reason, old_page_owner->ts_nsec, old_page_owner->pid, @@ -391,7 +387,7 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old) * will be freed after migration. Keep them until then as they may be * useful. */ - __update_page_owner_free_handle(new_ext, 0, old_page_owner->order, + __update_page_owner_free_handle(&newfolio->page, 0, old_page_owner->order, old_page_owner->free_pid, old_page_owner->free_tgid, old_page_owner->free_ts_nsec); @@ -400,11 +396,12 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old) * for the new one and the old folio otherwise there will be an imbalance * when subtracting those pages from the stack. */ - for (i = 0; i < (1 << new_page_owner->order); i++) { + rcu_read_lock(); + for_each_page_ext(&old->page, 1 << new_page_owner->order, page_ext, iter) { + old_page_owner = get_page_owner(page_ext); old_page_owner->handle = migrate_handle; - old_ext = page_ext_next(old_ext); - old_page_owner = get_page_owner(old_ext); } + rcu_read_unlock(); page_ext_put(new_ext); page_ext_put(old_ext); @@ -813,7 +810,7 @@ static void init_pages_in_zone(pg_data_t *pgdat, struct zone *zone) goto ext_put_continue; /* Found early allocated page */ - __update_page_owner_handle(page_ext, early_handle, 0, 0, + __update_page_owner_handle(page, early_handle, 0, 0, -1, local_clock(), current->pid, current->tgid, current->comm); count++;