From patchwork Mon Mar 3 16:29:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B1DFC282CD for ; Mon, 3 Mar 2025 16:30:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CBBF280004; Mon, 3 Mar 2025 11:30:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 27BD3280003; Mon, 3 Mar 2025 11:30:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F6CC280004; Mon, 3 Mar 2025 11:30:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DCDDA280003 for ; Mon, 3 Mar 2025 11:30:41 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 811B6C0C60 for ; Mon, 3 Mar 2025 16:30:41 +0000 (UTC) X-FDA: 83180778282.04.84F5812 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 9EE81140006 for ; Mon, 3 Mar 2025 16:30:37 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hQEXlivw; spf=pass (imf26.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019437; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8IqquS7ke+4+XOv5L24m6qSx7K0PzdfhIUdS/cnqyVA=; b=la8DKA/yxxocr9+nbTO/SS9gFc8CdxRXpKjnBCV55I7RetWiDZ0CATh9ELX+PzdxNXMBE1 mtjZAQDwQWfe3n0mK58tU9fMSpPglv3BPX6B7RNyiJBI7WH/9KnpXvffLGGnTcS0SXTglK puv85t2myxAQtR1o+x9Autq8w5C1X2E= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hQEXlivw; spf=pass (imf26.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019437; a=rsa-sha256; cv=none; b=33TIFGdMbxcRezK48jh+/IY0QXoc0QJU6hmqM6WDxqHjHoq8rE9Zi5iFp2LV/dvHbLIxlf FWQFeXEw1ctDBIMSp4TKl11qQ/+2snHqWX1KJLm6Fmecw+3zIa2TCuGvKQ8vJvFR/65p95 yUo8Yl6ZrKFAYEyEyg25lFPjnLIiDKk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019436; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8IqquS7ke+4+XOv5L24m6qSx7K0PzdfhIUdS/cnqyVA=; b=hQEXlivwrabwBWCdRxsHyAOreN1pKOxB356ByIKhmm9zIFnqLqo3x5kuQxl7qcykoHLA1A ywmvpvs7ZFUteo1+mmBxb2YxTLgAVVf1HHm8SZ+BghdlY46HdVSAzSTw4+KJ+m+YhAIbpf OxWKoYH/XXrthCo5rYHd65am4Zd2vmc= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-241-yQwcOEcPNeyvDsu2A3HKqw-1; Mon, 03 Mar 2025 11:30:20 -0500 X-MC-Unique: yQwcOEcPNeyvDsu2A3HKqw-1 X-Mimecast-MFC-AGG-ID: yQwcOEcPNeyvDsu2A3HKqw_1741019419 Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-4388eee7073so24304415e9.0 for ; Mon, 03 Mar 2025 08:30:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019419; x=1741624219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8IqquS7ke+4+XOv5L24m6qSx7K0PzdfhIUdS/cnqyVA=; b=peJbjlG8GWRK1aKAGxB19Wi+FZEeUgwUyh/qjOPUAlcSmQkaTXw8mYYhlL3PqLtQJD VS1IYa9F1qtXj3ya4lP2yOcKhq5DDoyIjnnPsjNqfYNzdXWTdRjhNJt0iLJ5hjX7JEUp Zhf14qtc6DU9h6IkNsu2sWVi7WqKQBFo7pm5dZ2V/DsIUX5PsOBw04Wp0+c2G27l4XYf PgS//ObuGcN01Z5DECSiMVevMbJ1mwfKfK5eRLi72XI/FVC7eh7TWKxH+WChDNppN5XA v2yL71/6uHcFNs0OzihKTWx08UJTtwVOvND0jIl5YAyRxuegG/N4m1fu8T2ibJypPdTy 07KA== X-Forwarded-Encrypted: i=1; AJvYcCWbYYeoyhK4gPL95KCPCOzpdfl3AVPhUVUSC8qwP9PGk4BHNYNH5RwfLgx/qfcm8hmbJ/KSytHL5A==@kvack.org X-Gm-Message-State: AOJu0Yy1U5aSf2FYN14YU+HO9v++KN+szKfvqj4FCeMPqa8pvhd1shmo XatKWXo9vtopCFz6c2jFtMMkYDiECCeX38H6jur/S37u8SRDlwEn8D/PeKA8EgSAfA+jBL1s4yQ rKDL190mSUClA6r9Nt3086FGKhOGDAIX7UWjw7dKY9R9Z4w7O X-Gm-Gg: ASbGncvXzgdv495dwH9GCiBUkCPda0w+TwWmSmfn0My2398kpPNcyVYHHCzJT7sX8VP DiWAEZoero1Zx3hUCsqAP9T9dwaMmF36Rz4xs+ZWUab+3cVzGPAcvpO8/3biru0u3cZr887+4r6 17s1BjGghOLYkctRZZrdzbS/Jw4tQ+jsakQYee1Dw1/FDPM1le3xL3nDqvUMIUgLu3T1dneu/KK G8OoXQNuGKU0nmO/+kx3lfMNsJRwZAlsYMdxBhPwy6M3tXBSWAl57FZpGHDb+WIy+OFrvp1HJxg Z5E9hh7pTrIZ8vYROLdxDMtTT+0yv6f9TL2T3vJTy6dz3BwRvyaGAbV4bEshA0lUDHBIpqkcckW y X-Received: by 2002:a05:6000:156e:b0:390:f9d0:5e7 with SMTP id ffacd0b85a97d-390f9d00782mr6547296f8f.13.1741019419259; Mon, 03 Mar 2025 08:30:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IGXzBd/+KVXhEWpKLy7bok4As8+LKdGdnCV2QCwLiXkX4QDGRFFrxDGOYHa7Hz+LJLKECZ5dQ== X-Received: by 2002:a05:6000:156e:b0:390:f9d0:5e7 with SMTP id ffacd0b85a97d-390f9d00782mr6547256f8f.13.1741019418889; Mon, 03 Mar 2025 08:30:18 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-390e47a6a5esm15196816f8f.35.2025.03.03.08.30.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:18 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Lance Yang , "Kirill A. Shutemov" Subject: [PATCH v3 01/20] mm: factor out large folio handling from folio_order() into folio_large_order() Date: Mon, 3 Mar 2025 17:29:54 +0100 Message-ID: <20250303163014.1128035-2-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 7xmVavp82JIY2usdnaTa-gR7UvLIA9jUJ8jTeKFloXg_1741019419 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Queue-Id: 9EE81140006 X-Rspamd-Server: rspam09 X-Stat-Signature: y1ob3tyc9s9ts63g4d64x5y6qjchr499 X-HE-Tag: 1741019437-994823 X-HE-Meta: U2FsdGVkX18vc1oSIi4fvl9Pr5r5heGMZFoeJ9JmAu85Xo5rixg2qQloVVbfmAJ6SseD5y1RvAFNsUvC0bxR0p6faurztgBy/cr6DePSuUXjAyZ+xGGxzkUVnWyWCFWSky9e+F3wiOa/8ogsnhMrw7lTA1Z0MqPCG89RLvUrE+c6QC54g5b2/oMw7/d3+ItiJ9SAXhp+kOKq/wEB+qADUoWlQGNwm/a0xTyOjfBBDWpoQD3U/bbG0rSXWEvzaFlUxQ+svkEdblQsT+EkU4EIMYtr7ZcqeCukroXte2TT+26mNkRBi91Nt+lwxIqzuxzHT+mi9IMUoKbj2uNaSoJoNLxex5aIBErVoq2JlbzsDSCPEv7Rhg/hrYknBvh5C81qIeSULyFcP4IKIZMhSyAzzViwBRAtqZfo0XC2OzyYpk22DtliRriTvfek49/QrUX//ncKRrrkXstWAd0crJJfXstPI9C9IKDRP0t2MkhciTfoYlmyWp7B0jOFXNGFBWakoXPw4PowAP9hi3FThy02xknEMJk18baMsT3jKhL8RrG696yt7zcGP3XAdZjEwYhusXugIrYo6f2GODyuenrGS3KhYW2ICwqPD08CLIRc4ANcMB0QgHKt1CVc6Ks2HVRlUft5vxn/dnT90bNW09ClR/K8DSjSaC0YGb8UQhhbZd7ybBkGYxR5UYlDoAMI8JWwnwFG2+Q8zPmwdfc9wRnzb2ad3lD4h9JowxlJH+Ob2F+hMt9OQpGaflK1JQIfiGyeXUrrc6aqxXpK00eMfVIdIDt6WN3+4skswEeZSqUH3bUGfrInNHo8g1qbK0XHxDgq6+tYGfM2ecch1bq+dWPYuHBE7tyA6O+k8comadMPXzhwPrQB1HgSiPcPsk9bSGREUWvHFrya4iL3eG1KfzcC/R6/0vlKI9UUn+ipsxgcnrKSfHf0231RDu3G19Dq7ejvRPfni44LUdBh9/ZYJQf 6ORaB/gH DuFZKjuCgEPcXwLgbbLMypGwZbs+YWpRBdOv8FwsFFYewVcpkV/UFH+gDWKKUSgP1l9Xpyc5zDKKK9OZxEZpQM0J7xi+F/ojbE/0NuQFOTJoEBGrWdvUgNgwnGLsiZ5CCfbz4ey/Ng7AcUFGtmvWYbYiZu75ACRYN++1U7iUcOo6nWT1F6Jtpk/jX/JGU15JWUqYLsplQcLvR3B6SB+ROqaM8LFEkLZ38XLl/rI5rEaW2tNTbG2oia57y3yWVYFBazk6DSBGsl8kyGIe/Zb3rXgoGh4wo8nuMEtJBq+IUbMLHBZh6qH1A6Va+qI7EqmpV26WqpPUHGoIbNFqRwsLvpwBvnyQ/7J8JNCZ9LnjNhnndgbr00OIhTH2KsH2SBo0FsMGt3S3GyNj64+baj02u2stuUmp297sU+BfPgZt0V/Na4P2p4NKH5ky9jok4dITTegEe8texwrTWhS77MfEPwB27EthmG3di44jdTLgNb4U90CLxEj+2+Mpd8IZd/BeWGtXO6vHNE1vYS+VcsxRt/xF0y7CNraV60J1MPrsYlN+xXkn6da8/Syqvoz2LwZX8Q6iKle66iiV3qmk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's factor it out into a simple helper function. This helper will also come in handy when working with code where we know that our folio is large. Maybe in the future we'll have the order readily available for small and large folios; in that case, folio_large_order() would simply translate to folio_order(). Reviewed-by: Lance Yang Reviewed-by: Kirill A. Shutemov Signed-off-by: David Hildenbrand --- include/linux/mm.h | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7b21b48627b05..b2903bc705997 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1194,6 +1194,11 @@ struct inode; extern void prep_compound_page(struct page *page, unsigned int order); +static inline unsigned int folio_large_order(const struct folio *folio) +{ + return folio->_flags_1 & 0xff; +} + /* * compound_order() can be called without holding a reference, which means * that niceties like page_folio() don't work. These callers should be @@ -1207,7 +1212,7 @@ static inline unsigned int compound_order(struct page *page) if (!test_bit(PG_head, &folio->flags)) return 0; - return folio->_flags_1 & 0xff; + return folio_large_order(folio); } /** @@ -1223,7 +1228,7 @@ static inline unsigned int folio_order(const struct folio *folio) { if (!folio_test_large(folio)) return 0; - return folio->_flags_1 & 0xff; + return folio_large_order(folio); } #include @@ -2139,7 +2144,7 @@ static inline long folio_nr_pages(const struct folio *folio) #ifdef CONFIG_64BIT return folio->_folio_nr_pages; #else - return 1L << (folio->_flags_1 & 0xff); + return 1L << folio_large_order(folio); #endif } @@ -2164,7 +2169,7 @@ static inline unsigned long compound_nr(struct page *page) #ifdef CONFIG_64BIT return folio->_folio_nr_pages; #else - return 1L << (folio->_flags_1 & 0xff); + return 1L << folio_large_order(folio); #endif } From patchwork Mon Mar 3 16:29:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1302EC282CD for ; Mon, 3 Mar 2025 16:30:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7386D6B007B; Mon, 3 Mar 2025 11:30:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E8DB6B0082; Mon, 3 Mar 2025 11:30:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 589AF6B0083; Mon, 3 Mar 2025 11:30:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3BBA06B007B for ; Mon, 3 Mar 2025 11:30:28 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C1CB51C8DD1 for ; Mon, 3 Mar 2025 16:30:27 +0000 (UTC) X-FDA: 83180777694.20.E45E064 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 2778E100017 for ; Mon, 3 Mar 2025 16:30:24 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="b/ACEsK6"; spf=pass (imf14.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019425; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YXwoThNtp+x7X/3NlVobY8ZdtRlIGCn0KXkmBvKPXvw=; b=vyLHMmv5Tjz15UEHp0nvQ/11gV2x4xNAQIAYDQ0cQuTP0srIl2XCOgvdg48yLCMj4RZPVw vi3UEBTQF2IvSaIJe1Utal78zD3ID2M77pwLWF3zAnSyCMF/QUqJRYhK2gGYT29BlDhm2m tFQHVcqmPV3GiPcL2UKrezliPsNAHmA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019425; a=rsa-sha256; cv=none; b=k3KponCme9gyGsk0IjmYqij9b82MoqLxHDkcBf7TbdMDVOSyKpApWA7YZH7d2OlDf0mKiW Qf0/9BQ+AwI1H+heh3u62lR4kI8F9KgoSmAv+1ACgNrTFT37dzfWvjIuuncjELOTbmaDGJ nFEhUMiBUNtK2+7mBphcuYAed7s4hUw= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="b/ACEsK6"; spf=pass (imf14.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019424; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YXwoThNtp+x7X/3NlVobY8ZdtRlIGCn0KXkmBvKPXvw=; b=b/ACEsK6XQwU6Ble9cXkXYJCviWnjx2/9Hjexqp73pJHySJEpM74DXH2KJAiF0g5HwRPJz +FnaprOJ6h36sJE8mXBnxat9cAjuEkbzUjMd6mo+L1NpBkCBUXiHhTTExipBbcC/HKRnzY 4ndZwwoN14dXoxmLRQczvInQ7foD4Dg= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-668-d4sY4GhpPOWWxaQhOONkKA-1; Mon, 03 Mar 2025 11:30:22 -0500 X-MC-Unique: d4sY4GhpPOWWxaQhOONkKA-1 X-Mimecast-MFC-AGG-ID: d4sY4GhpPOWWxaQhOONkKA_1741019422 Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-4399c5baac3so33940455e9.2 for ; Mon, 03 Mar 2025 08:30:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019421; x=1741624221; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YXwoThNtp+x7X/3NlVobY8ZdtRlIGCn0KXkmBvKPXvw=; b=JttGIPcqlGzRh6fbAZ3gs/ip23k69+U123pOqfAwHst7xv2QMmC1LDGtj+XX0SZnDm 1gAzfV4deVtRChsqH/MrF0yZdn+d1bA8qpOmEMViWW5F1Jkpl70rDLHBJxZFIlSNtvaI cZeMw+cNAKnqcDDpHBhaFi6GCpbVWLYMJTeYKahEtXOIiJs1dcf5qXGw7e7d6nzcTjvc c3IB+LBi8qSThbhxavWIJ+ig2bfhtbju0B/kIFqy+u/dN0HX48G6HposHmn5NFh+9w7P 48NaOsD4EkVcX+K+lRIbkL487MvtYFwouqsmXtXWdnZfJK0SlsZ7PwO6I7/BEiXNjw+d v1MA== X-Forwarded-Encrypted: i=1; AJvYcCUm6HxAfXTWpvEpe4Jw3xxUfJ3O7krVgFD40KmCPsInZMuSXS0kJfPvagLKdLdG2eLcNbxUq+u2AA==@kvack.org X-Gm-Message-State: AOJu0Yy7CHbLgmjonCF6/gLgHEIHSDM+6irgtv4jC4OI68eaf96Pul/q DD52NtFjQutQlh9nhzYn7letSMtMD1+xtomuGHz6vQUX2ZQTf6MGZ4f+gSn9TmckWaTO+pP+noK BPz7R9g7Q6ycf0NaTPkuE/oEzDPcbRDEkj3J0ugOk7miyjJtD X-Gm-Gg: ASbGncsFhy8wBdLdc5x13oH+mwMt/5cavJT7rGb7NOaYw2YBTZ73zORNTtxOTTXCr3m Z4bLMkthI2TtzXoFCm7HwbFczp2ZBeyhVxykt3gHNVWNBjgZvqhnOns0gjQWNcAQZGNGd7VRfpe x6B+AvbC9k9EJjXrQHFcrhPMuFgdT6N/18ZSuZW1pP2k1DoPwkxfa0tzHIYPaEfeDxq2mAGVC1q RgIXuxvxdb6uPvsWhA+6iNMSR3MC18fS35lWN73fzdEESfM9+fVvi4/+rqOyib9IkV/9OPzzdpk 9EAeY3azlr4p3ZuaW3nE5GQRgQvW9kxPde5ZeRkqlh3aJtIlZ98dn4ocgrbTRXcMmDPDpOYIe5S r X-Received: by 2002:a05:600c:5102:b0:43b:c0fa:f9eb with SMTP id 5b1f17b1804b1-43bc0faff85mr39510415e9.17.1741019421464; Mon, 03 Mar 2025 08:30:21 -0800 (PST) X-Google-Smtp-Source: AGHT+IHuptMXo5vgbefj6sa5a5v3hHufhVKK5lbtZ5OlCKuge8xCX7GzdoorXNmVUyzEzt6+EvCK+w== X-Received: by 2002:a05:600c:5102:b0:43b:c0fa:f9eb with SMTP id 5b1f17b1804b1-43bc0faff85mr39510065e9.17.1741019421019; Mon, 03 Mar 2025 08:30:21 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43aba58720esm198718765e9.40.2025.03.03.08.30.19 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:20 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , "Kirill A. Shutemov" Subject: [PATCH v3 02/20] mm: factor out large folio handling from folio_nr_pages() into folio_large_nr_pages() Date: Mon, 3 Mar 2025 17:29:55 +0100 Message-ID: <20250303163014.1128035-3-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: g2LzIvOz3lR2y6FIXVag4z1xX1yuYx_9BZH8ix85MQE_1741019422 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Stat-Signature: nzdzmfpj3x11rph7erymrzx1bhpgh7eg X-Rspamd-Queue-Id: 2778E100017 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1741019424-55444 X-HE-Meta: U2FsdGVkX1//ovgyqUUSXGGkXS8ffIwK7YqsYdhUDRwrniXVEJSIooub4b3Re2PpFOqApuWIde04WCEwRnUkYTJsVD9S6wacWYQLp5YptFO63FCeTVKhJUg6z1D8QtulryeLU18NCfa1Lo7fmLPy1tya/Ocne7ke+I11PnqAWiVr1Jfgqbvf3cR9pRF2CHTr1gJoYF8PmttpdkYp1VvWmuNmi5eH9ESuJGbRV2HLgihAfPdNLuijT/hxKfFeLgq0MYnR5Gm1lY0I+EF2FySw3tmadEMz1rkivg5irsQ7gXIGYJ8ycJi71c+Zi7ElGuyKOWyHVw+kY2X1PVzzMTt4RPSYJqwCu9sCVeY8jxvdURxk8M3Yd6KoL86ntArcTdHG0/oXkO0FWoXVebfdC4kLTfzIZuRhDqfG+xJmgtzPkzWVFgVzCI6H29pQ35k6xCv1cHCvvbzMb4MCH444uwrjjm1OKQSog+ICgOSOrHT+EmE50TqrcXwrc2LTgIo0PkQ4Qpt46Qh3GsYzGKrkbIZlf3rsBP4urcQVQ4ZZwesuhQMVszd+Nwr1JpvdPDgaoouDH+ve+tENGWafoia+ciDWiaL4jXV/RAbXqg3Ioi3E4r+4Yf//6TF8tfr21iupIqMByNv3DtN84B+60/+I/DLHtPS2C6CRgCk8c8EiW+x5hMsj5BNiTHlHvnVSEiz7YD3cK7atqwLobiAmfpjTjx6oaeVMbek6UVx1FTVpzoqu1D+zPhlRLsoVnSdcABmCqteYtn0c1TjimmTqDEcBABZs2GO+C3pKwd0mD/XUXPwzeIy94ABkFiDQtTDX0EFCUzfX+jy3RxMcBr9J0mJX1GFLato4tWTfjUYiznTEMIqswaqAkv19GvGht3oJst9YM9mEDUb3H/3YEy0vTMlWqaB1BycDnOmMvCaejEjiASCeF9w2KtXGECmhiHfBqYauhGXY9WHJUektPE8DLb1TXKK jjklBuk6 1xcWNdLU9LIERdbBymen8an6sTF3DRquRFZ7RqRL6Hw7rj//DVbXd02176gFi+a6KPM3618kqbe13hdX/0Os8cAFUjfXdQfvaje6knUilu+wcVQEoGTjNMYtyp6xvZGm475N8p8SlJOqv+TxPKxEwQrSmkqEFj+9Z5HMj8TD8zN+TE0G+pTNULC9YeGLqVFxvqHs1/zahF9AkUmORFM0WuHaNQnVecujkKD9ePne1DBolARLMPAuJvg4W+UCpF6g7UAY52cHLLho5oYlo/oIclLsMbChiZDt8elk3dIGrBgPL0bi5K+XY2CXSjKq/c2ehhMEUMeAghotF/9vw5ss/Mra/cyMVzwmLFMHfaFRWAQmd31MeTY6NZFVDP+19+syxzRHLhi3CNQmQlM5Ilf86p5vPGTCQwjbPZOnOUBAXeytAPL01qKyJy73K/S0O8KGi9RR7/UW7mr/MFxU4ysDYwrhkRGjW7BsnRhN+TG23msuZRsDbt8oO1VqZcl9KjKxxVT3JgQyxS+drJu9Nk0QPV2ePg0ZeBw1xdP+wqszdlSVVUkY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's factor it out into a simple helper function. This helper will also come in handy when working with code where we know that our folio is large. While at it, let's consistently return a "long" value from all these similar functions. Note that we cannot use "unsigned int" (even though _folio_nr_pages is of that type), because it would break some callers that do stuff like "-folio_nr_pages()". Both "int" or "unsigned long" would work as well. Maybe in the future we'll have the nr_pages readily available for all large folios, maybe even for small folios, or maybe for none. Reviewed-by: Kirill A. Shutemov Signed-off-by: David Hildenbrand --- include/linux/mm.h | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index b2903bc705997..a743321dc1a5d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1199,6 +1199,18 @@ static inline unsigned int folio_large_order(const struct folio *folio) return folio->_flags_1 & 0xff; } +#ifdef CONFIG_64BIT +static inline long folio_large_nr_pages(const struct folio *folio) +{ + return folio->_folio_nr_pages; +} +#else +static inline long folio_large_nr_pages(const struct folio *folio) +{ + return 1L << folio_large_order(folio); +} +#endif + /* * compound_order() can be called without holding a reference, which means * that niceties like page_folio() don't work. These callers should be @@ -2141,11 +2153,7 @@ static inline long folio_nr_pages(const struct folio *folio) { if (!folio_test_large(folio)) return 1; -#ifdef CONFIG_64BIT - return folio->_folio_nr_pages; -#else - return 1L << folio_large_order(folio); -#endif + return folio_large_nr_pages(folio); } /* Only hugetlbfs can allocate folios larger than MAX_ORDER */ @@ -2160,24 +2168,20 @@ static inline long folio_nr_pages(const struct folio *folio) * page. compound_nr() can be called on a tail page, and is defined to * return 1 in that case. */ -static inline unsigned long compound_nr(struct page *page) +static inline long compound_nr(struct page *page) { struct folio *folio = (struct folio *)page; if (!test_bit(PG_head, &folio->flags)) return 1; -#ifdef CONFIG_64BIT - return folio->_folio_nr_pages; -#else - return 1L << folio_large_order(folio); -#endif + return folio_large_nr_pages(folio); } /** * thp_nr_pages - The number of regular pages in this huge page. * @page: The head page of a huge page. */ -static inline int thp_nr_pages(struct page *page) +static inline long thp_nr_pages(struct page *page) { return folio_nr_pages((struct folio *)page); } From patchwork Mon Mar 3 16:29:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22A56C282CD for ; Mon, 3 Mar 2025 16:32:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A9BB4280016; Mon, 3 Mar 2025 11:32:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A4FBC280015; Mon, 3 Mar 2025 11:32:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8737B280016; Mon, 3 Mar 2025 11:32:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6739F280015 for ; Mon, 3 Mar 2025 11:32:06 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id ACC9180C5B for ; Mon, 3 Mar 2025 16:32:05 +0000 (UTC) X-FDA: 83180781810.26.2966C66 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf05.hostedemail.com (Postfix) with ESMTP id E060110000F for ; Mon, 3 Mar 2025 16:30:37 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PbpiXwcx; spf=pass (imf05.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019438; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0VohxUkhEcl68WNDuA0eZYZJxa5JX4ZCeZ0UqfdwFdI=; b=f+eyqNf6p34f9lzGLdbNhTwCMJeZYkXIadMDBSNSXdqH2/RS7J67FFtNuSGDoW/Yp63rj6 6OXBMJ1MiTc1azSykWD6inf3bW3sR+lUt4gd8vA3JX/ZmSrT6ijuiC+15i0JJAF4vB2qyN BvR0p2XjQ+owhEAtwM6UP1z9cS4h/8I= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PbpiXwcx; spf=pass (imf05.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019438; a=rsa-sha256; cv=none; b=yyujMnAppSP/7IvfuoRfVzWy2tleFrt9tIRCoxXet+4WleH0trAHHDX60y3HtLjvyNucuz enKiwPSqeODe+COgkpqlmo2y2O6ZLAtEMmqLc2G7cdscvakmjvXmw6TEhWfSvb2y57Txrl FjfmN4QpJ7dP4mI72EntJDY9rqcfs+w= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019437; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0VohxUkhEcl68WNDuA0eZYZJxa5JX4ZCeZ0UqfdwFdI=; b=PbpiXwcxv6cKk6gx3p57BbKEk+ZUwMR42sRFSC7bnahyr83ODThUFOdP/xwL7jtWuloM+t l8LodgSEpip/FBHHMp7kZDsOEbrcvCXph1pY0eFAQKczvuYVNk4Z/ZioejAjcipxc2vvcC FN7B48CaG/mWAXyA1OAQoF9B9qBqvEM= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-571-NuJxteS0NouTchkgPMZFSQ-1; Mon, 03 Mar 2025 11:30:25 -0500 X-MC-Unique: NuJxteS0NouTchkgPMZFSQ-1 X-Mimecast-MFC-AGG-ID: NuJxteS0NouTchkgPMZFSQ_1741019424 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-43ba50406fcso27671415e9.3 for ; Mon, 03 Mar 2025 08:30:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019424; x=1741624224; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0VohxUkhEcl68WNDuA0eZYZJxa5JX4ZCeZ0UqfdwFdI=; b=C/SooBsqA1RCSqhL5v0GHQJqOrM76py9wPZasGSmXWuN8vTgm6pKm6kIvtoZjk9qhb anBcgd2fHgZ0k0CDmKRS34EM5cjYAE93CdInLQXo/KoHnuo/QFqrL2hpshufCbGc1sYR 7y/aJ5G7CaQikgnH+nBY91XFbVXpaIrHCXcjqNXFNgsKDUOQoLgwIKbQWXXcXXXj4niQ +Cn7m1n5Od47Bb9qyHUPKcKR3II8jqoph1uoybUC7RP5dTgeShJzE4CDJPZ/rAx5fnDh 9uumEuPE1Q+NOLlzWoEh3kb8LmWs8X+cqJQvP3RQfCXP74kVz2J9h1hNelqZDCNo2qdt e47g== X-Forwarded-Encrypted: i=1; AJvYcCXpw1OPRP7PzkHFTMgVk3fjOn0cFiNPSI8c3G6Ey57jcf/g3zTsWjJORop5BIY2ENOhppHX0NRTAw==@kvack.org X-Gm-Message-State: AOJu0YzEA2xNZu9QQYS19Inxxx9ZXaztN27E4a1R5gUMcUs+oc4Cf454 KgtHJBzqdYdVpN6bOVbpxsq4PSdPWMKB8j1JV2gVcAQjAzku1FuhjpqxGunV8YZ4k5V5i/ymOVj htZrGPH0POpwO2d0g8cj2l0hFMHjPKLTCehTmZERV/4zBXWg9 X-Gm-Gg: ASbGncsAjYwF2wXDODnG390WJhoT0QXaUf5LddHD0NhyQ5iMUSuh0M8kbkAAmA95NUb 29CEFIe0/WhLDItO/edHJVLQd/BLcGc0bTTdNWcCet0rI5DJ04E/Mk0Y9wTrE73kKCWiBTAAAnQ coemdcOrqRkrLSHcuS86rqV/PET/os74pACP93tcx+leRi3z8YGN9iO7QJb7uxoxqYZiZFICxs4 WAK8uD9SeAsNG90+vdTRK/5/cFFQuBdAc7Sgr368PWSIqCbui1cNaduxkKcEYne5WMqOuWYhxvO /2FPT9vHjjApi4DRugunudTGaY8IwCbGE2zZ3Qhkrsx2KKmMxk/N38DA6NMSXZnJYGIrVA/fZRI E X-Received: by 2002:a05:600c:3596:b0:439:9494:318 with SMTP id 5b1f17b1804b1-43ba66e252dmr97507635e9.1.1741019423941; Mon, 03 Mar 2025 08:30:23 -0800 (PST) X-Google-Smtp-Source: AGHT+IEA60b0YZJ7pQqMV6rPSAHKqjeBG5mk2kZJ3T/m9wf5+mg4ZPniiyD7qwD32/lnlnL6IXv1QQ== X-Received: by 2002:a05:600c:3596:b0:439:9494:318 with SMTP id 5b1f17b1804b1-43ba66e252dmr97507315e9.1.1741019423515; Mon, 03 Mar 2025 08:30:23 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-390e485ddedsm14691606f8f.89.2025.03.03.08.30.21 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:22 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , "Kirill A. Shutemov" Subject: [PATCH v3 03/20] mm: let _folio_nr_pages overlay memcg_data in first tail page Date: Mon, 3 Mar 2025 17:29:56 +0100 Message-ID: <20250303163014.1128035-4-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: L7Sufx5R5i73fzdl3MGeoey_kiFAY9gADaJgZD0s7PY_1741019424 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Stat-Signature: mrjhcj17gzqh7uwkqnzntiutbe389ir7 X-Rspamd-Queue-Id: E060110000F X-Rspamd-Server: rspam06 X-Rspam-User: X-HE-Tag: 1741019437-16229 X-HE-Meta: U2FsdGVkX1+0eKIc4q84txz4PnMeyKBP+KhmjvPX16vNgs25liEZn4NwFaNPIGXyqph9DKv7+LuYbVB3mPfw8PY3XCs4UT2U/dNa5TcQI/8Q3sWr96oJEwQk/lWQhMmMU1spWvy0o5idf4dcOBQWC5cNNvoD97h32gKapwxFJOVy4P7kTXHnki1G0dSjpu34jgSUumpBB2ajgNTBxwlhyNMiMzRq/mwuIovEhVCAOsmLiu0iz9Gmi9CXEhQxJ5QtuAugehnLSOKsezVbjwEmsnZgX9wkI9O5sjSwqXk/dQgcAck2z8yAm8SEoD8o4cO20VgMf/k0M3xR1E54c0vjItXb86FdKb9S0g4yUYXdlZaoCUUHyO08gEe/4z45tTfqxfQLRmHHsEcAbs2WOQzVr6DPpeujpeOlDWBs5OFug/Qd1gGsY5Vvj1yT2X/k51iNWWg6qVozuoahXPi46qIUC+cLoEDWUH+hfQY4ixcIt7ig6YCtgtVwr1J/urnqoSSKEacM9fwsjvQkAlNhwSeFZOwzspIeLI+m7+tIfZv3NoMyH8vGxuUObzrI4SET3o7ij1SO0+uhHgF1Fkwtd0NBzVdvMNBF8eYxb0QQ/MvGTgiBUQGB+OXaM+/LpLF7dLXFboJnawOiLYP08ADgfYxegvWfihwjnSoe+ow9qWdt3QZZOi4wBiojXfeGajXol65W8inGIGvyBK620j8BBnA5FfRtb4IOHod3Sssty7Tj/QNeMFtdUeVlFjSNInvkTtW7aFxAVbNtW8VLir0g9tQQuKLve10F3Y3uaGDTkKhPP2iT7JZPq1uIC2hTJtelD6uRyFpRUOlwmHokpsgRaNDLuyJomD5BYbczUcETREEtrkKCSlgVRCKsMjoR+up3MUFd1S8ScOSst/zcM44tLxBV/gLlzROnVAOvcsm+7ZlcENm4f/cO1buE/SiiC/2/vNNImCDTcl14rbvZ62T6uMk 0bC/XCwZ UA9OFKjJ3MhlmXCc9Z6Dsl2kC6xcZ/jGoKEANhcmHseS9lts4HpKBme/b/PHvGLc2o0vWxpxNUAuA3Df9Vkc8Cz7MubnXSL4THU1ntFwJJ2evH00yKFLgJJ2mzvM2wDAQCAqIBOTZhakfWTOKyOQ3jsTGZTpzauH3+Ukm8Yj5B/Op299dKrcJR+wKHpKqeUFPEV0Y1mRRVsiy6VkvqQTZmnsSoDUSolzO3IKUAmMUKm1xpSf7Bu1DgO3fU7+/4TnoGgbhx6j+JWoZBZBERdYBzISiVJOoFuPbppthH2CwoXKRQdIKpdGoypICrBNm8SNZtRfSVaw0oaGWub5vOAcmYuLJFYAqbAwOMv4h0zhsw9fXJjMbABxP1nw3sIpFD0I8qq5mD+85PpL/yCs/3iaxqZzIEwSXIvV97eLAFiSQj7Znbm7XdwpINRzK2EZdpf8Ha3ArDY4XKVUtlt5DSisXbXEW0zj9zep+T1UcctP3rgHpUPFD/kYxfsddZfGPoSOI4NMZUkVgXB/1lGVMjXWl6keZFu8aF89NFJTMeaMecwy3Dfo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's free up some more of the "unconditionally available on 64BIT" space in order-1 folios by letting _folio_nr_pages overlay memcg_data in the first tail page (second folio page). Consequently, we have the optimization now whenever we have CONFIG_MEMCG, independent of 64BIT. We have to make sure that page->memcg on tail pages does not return "surprises". page_memcg_check() already properly refuses PageTail(). Let's do that earlier in print_page_owner_memcg() to avoid printing wrong "Slab cache page" information. No other code should touch that field on tail pages of compound pages. Reset the "_nr_pages" to 0 when splitting folios, or when freeing them back to the buddy (to avoid false page->memcg_data "bad page" reports). Note that in __split_huge_page(), folio_nr_pages() would stop working already as soon as we start messing with the subpages. Most kernel configs should have at least CONFIG_MEMCG enabled, even if disabled at runtime. 64byte "struct memmap" is what we usually have on 64BIT. While at it, rename "_folio_nr_pages" to "_nr_pages". Hopefully memdescs / dynamically allocating "strut folio" in the future will further clean this up, e.g., making _nr_pages available in all configs and maybe even in small folios. Doing that should be fairly easy on top of this change. Reviewed-by: Kirill A. Shutemov Signed-off-by: David Hildenbrand --- include/linux/mm.h | 4 ++-- include/linux/mm_types.h | 30 ++++++++++++++++++++++-------- mm/huge_memory.c | 16 +++++++++++++--- mm/internal.h | 4 ++-- mm/page_alloc.c | 6 +++++- mm/page_owner.c | 2 +- 6 files changed, 45 insertions(+), 17 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a743321dc1a5d..694704217df8a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1199,10 +1199,10 @@ static inline unsigned int folio_large_order(const struct folio *folio) return folio->_flags_1 & 0xff; } -#ifdef CONFIG_64BIT +#ifdef NR_PAGES_IN_LARGE_FOLIO static inline long folio_large_nr_pages(const struct folio *folio) { - return folio->_folio_nr_pages; + return folio->_nr_pages; } #else static inline long folio_large_nr_pages(const struct folio *folio) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 689b2a7461892..e81be20bbabc6 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -287,6 +287,11 @@ typedef struct { unsigned long val; } swp_entry_t; +#if defined(CONFIG_MEMCG) || defined(CONFIG_SLAB_OBJ_EXT) +/* We have some extra room after the refcount in tail pages. */ +#define NR_PAGES_IN_LARGE_FOLIO +#endif + /** * struct folio - Represents a contiguous set of bytes. * @flags: Identical to the page flags. @@ -312,7 +317,7 @@ typedef struct { * @_large_mapcount: Do not use directly, call folio_mapcount(). * @_nr_pages_mapped: Do not use outside of rmap and debug code. * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). - * @_folio_nr_pages: Do not use directly, call folio_nr_pages(). + * @_nr_pages: Do not use directly, call folio_nr_pages(). * @_hugetlb_subpool: Do not use directly, use accessor in hugetlb.h. * @_hugetlb_cgroup: Do not use directly, use accessor in hugetlb_cgroup.h. * @_hugetlb_cgroup_rsvd: Do not use directly, use accessor in hugetlb_cgroup.h. @@ -377,13 +382,20 @@ struct folio { unsigned long _flags_1; unsigned long _head_1; /* public: */ - atomic_t _large_mapcount; - atomic_t _entire_mapcount; - atomic_t _nr_pages_mapped; - atomic_t _pincount; -#ifdef CONFIG_64BIT - unsigned int _folio_nr_pages; -#endif + union { + struct { + atomic_t _large_mapcount; + atomic_t _entire_mapcount; + atomic_t _nr_pages_mapped; + atomic_t _pincount; + }; + unsigned long _usable_1[4]; + }; + atomic_t _mapcount_1; + atomic_t _refcount_1; +#ifdef NR_PAGES_IN_LARGE_FOLIO + unsigned int _nr_pages; +#endif /* NR_PAGES_IN_LARGE_FOLIO */ /* private: the union with struct page is transitional */ }; struct page __page_1; @@ -435,6 +447,8 @@ FOLIO_MATCH(_last_cpupid, _last_cpupid); offsetof(struct page, pg) + sizeof(struct page)) FOLIO_MATCH(flags, _flags_1); FOLIO_MATCH(compound_head, _head_1); +FOLIO_MATCH(_mapcount, _mapcount_1); +FOLIO_MATCH(_refcount, _refcount_1); #undef FOLIO_MATCH #define FOLIO_MATCH(pg, fl) \ static_assert(offsetof(struct folio, fl) == \ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6ac6d468af0d4..07d43ca6db1c6 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3307,10 +3307,11 @@ bool can_split_folio(struct folio *folio, int caller_pins, int *pextra_pins) * It splits @folio into @new_order folios and copies the @folio metadata to * all the resulting folios. */ -static void __split_folio_to_order(struct folio *folio, int new_order) +static void __split_folio_to_order(struct folio *folio, int old_order, + int new_order) { - long nr_pages = folio_nr_pages(folio); long new_nr_pages = 1 << new_order; + long nr_pages = 1 << old_order; long index; /* @@ -3528,12 +3529,21 @@ static int __split_unmapped_folio(struct folio *folio, int new_order, } } + /* + * Reset any memcg data overlay in the tail pages. + * folio_nr_pages() is unreliable until prep_compound_page() + * was called again. + */ +#ifdef NR_PAGES_IN_LARGE_FOLIO + folio->_nr_pages = 0; +#endif + /* complete memcg works before add pages to LRU */ split_page_memcg(&folio->page, old_order, split_order); split_page_owner(&folio->page, old_order, split_order); pgalloc_tag_split(folio, old_order, split_order); - __split_folio_to_order(folio, split_order); + __split_folio_to_order(folio, old_order, split_order); after_split: /* diff --git a/mm/internal.h b/mm/internal.h index bb9f3624cf952..bcda1f604038f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -682,8 +682,8 @@ static inline void folio_set_order(struct folio *folio, unsigned int order) return; folio->_flags_1 = (folio->_flags_1 & ~0xffUL) | order; -#ifdef CONFIG_64BIT - folio->_folio_nr_pages = 1U << order; +#ifdef NR_PAGES_IN_LARGE_FOLIO + folio->_nr_pages = 1U << order; #endif } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index dd7e280a61c69..ae0f2a2e87369 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1178,8 +1178,12 @@ __always_inline bool free_pages_prepare(struct page *page, if (unlikely(order)) { int i; - if (compound) + if (compound) { page[1].flags &= ~PAGE_FLAGS_SECOND; +#ifdef NR_PAGES_IN_LARGE_FOLIO + folio->_nr_pages = 0; +#endif + } for (i = 1; i < (1 << order); i++) { if (compound) bad += free_tail_page_prepare(page, page + i); diff --git a/mm/page_owner.c b/mm/page_owner.c index 2d6360eaccbb6..a409e2561a8fd 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -507,7 +507,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret, rcu_read_lock(); memcg_data = READ_ONCE(page->memcg_data); - if (!memcg_data) + if (!memcg_data || PageTail(page)) goto out_unlock; if (memcg_data & MEMCG_DATA_OBJEXTS) From patchwork Mon Mar 3 16:29:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B53EC282D1 for ; Mon, 3 Mar 2025 16:30:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A9573280002; Mon, 3 Mar 2025 11:30:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A44E76B0085; Mon, 3 Mar 2025 11:30:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BF94280002; Mon, 3 Mar 2025 11:30:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 651CA6B0083 for ; Mon, 3 Mar 2025 11:30:33 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 26077B6421 for ; Mon, 3 Mar 2025 16:30:33 +0000 (UTC) X-FDA: 83180777946.04.2C7BBDA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 23373180026 for ; Mon, 3 Mar 2025 16:30:29 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MOLaBRvc; spf=pass (imf16.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019430; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XXsaZOoTqmrj90mJbBg0yoAKUyLdOWV9hsabR6NX2f0=; b=CSYNZJyKTH3E5NUQpVn3hBnZvt5+UGssItBJryFQ6/nDXKN+x787V3+UbRGc29B2UXSKmD vBsY9nHcTgVH1xE35LJnWIcnridVpT3MFujXpnHzldv/r0GndR0cjD+sIljRGUOrETAE4P sGgYuUasZEaawhPjGfXNO/5c+IEHC8k= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MOLaBRvc; spf=pass (imf16.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019430; a=rsa-sha256; cv=none; b=XXOR93Ilxd9vGX1OLRjJgik2EDShb1knENkGbBTRKJuRyNK/taTcek64juVrWAgItxuZXT exPpydMpGl5wh9i0+8YFdud6HY2bVIAZPrM0InWsCfyAsIBZY/DXxwLQP32CDyweUhhF8Q A6CHv9BU0VdIzm0UI49q+fnEtiNG9C0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019429; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XXsaZOoTqmrj90mJbBg0yoAKUyLdOWV9hsabR6NX2f0=; b=MOLaBRvc1xAcPzsA7FnKy9qyl7O7OqrwuClmVNiLO39sgNTIxQNU0fKBOehWbIKaWTQq3/ Wf+LsUFP3qBbv4Rw8SngnBlZrBh9PNtx/10YDLrJvbjA3/lzb/jM864Y+oPIU0L1tvD57N Q1UY4KeMK8U7RFWhFN1f2HTPvrJ8TxY= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-134-VlWHn8jtPeCrSqd6LD1Wiw-1; Mon, 03 Mar 2025 11:30:27 -0500 X-MC-Unique: VlWHn8jtPeCrSqd6LD1Wiw-1 X-Mimecast-MFC-AGG-ID: VlWHn8jtPeCrSqd6LD1Wiw_1741019426 Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-43ba50406fcso27671685e9.3 for ; Mon, 03 Mar 2025 08:30:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019426; x=1741624226; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XXsaZOoTqmrj90mJbBg0yoAKUyLdOWV9hsabR6NX2f0=; b=AmJxeUkFOyXTFcntv2qfWGFQYYEErCIRvXP+9WXxF+rFlv9NUXZA100SvIg8KQVCHV cv+WkUQjlcetJIdBZrF2RMsRgr0OdG3L4NdRR4kxbIMZ3YDjJB+uuVZrKT93KI/l+GmY TP7rekmRRtXm7Chr5+q2H9R4yAoB9DDJYqxIq6EgtkKamFDyXjBRZJm3iK4hcgTfave7 nCj+UKewmnJR0Jv39yw1m1tNJStLJwK7ML2cYn78PV+zjm8fHDGeU0L1D5TUZJ/Yn8gF +SqVbwYRY6MLTdi+3oDQZeU/p2AQJuGPvnziPXh56iapO3Dl4tnOITzyVadjauf8Iyms T6ZA== X-Forwarded-Encrypted: i=1; AJvYcCUGqGp5fEsEG4wvzYsURY9SgipwUm7TuYPkt87bL1W2PHytyRRwcwmgYryC5hCRs9LFy3r/7kiahw==@kvack.org X-Gm-Message-State: AOJu0YxmIVGxlAyzxD4S7ZnMIZlQVWe6xygqsrqG3lgms7mI4Dn57kkI u/7LqAfV/7gWSuehvrtvpJUHJtgAD2c4O6MsqhNIRW6Wu+BLoQBDGhrBKK9hl5JHmltA7Pqz7l/ VA06myHc2X0MxxTAQxDukd19CXezuw7tC/KdV4L66x/WDhRjs X-Gm-Gg: ASbGncsXrfFdh10It2ds/sIyJefSnYS+vPowpxzUQySEyWgPLz9gpegW3LYJe/4K+fx 1wMNdvJR+pvRAyQZgp8jZ1rq4AYVPfTJa/ucjxxeUB5skub/wXf45bAUmAWHJb5D7GDZDbtlRrK nK+UvYF0Yyk+9hlTGY3UB+Jk9jLuEvPiSKwNYxSqZnuh1Ajal7LaO8loCOpI7SqRWwhQAjRgdUG yoFnLkR+IwC03fmILjqye21+h3heZndN8nqoHzVhD92kwRIjyFEXGIIH9wkpOjzfMd05jejWkNb 3MNvZ0JV0UgaP/K6M4iNSYf8E1SR6WpdmnxMBc1GZkFGmp6tMR3qyTboj03djOuUb9Gcyhv4Oo7 j X-Received: by 2002:a05:6000:1565:b0:391:9b2:f49a with SMTP id ffacd0b85a97d-39109b2f905mr4268668f8f.55.1741019425852; Mon, 03 Mar 2025 08:30:25 -0800 (PST) X-Google-Smtp-Source: AGHT+IHyH4J1EAqaHR0tUbY9OuS6FmG3s0lngE2eSOblLtE+C7Z78Ry//BFuo4023zOsYhDT+DJ8lw== X-Received: by 2002:a05:6000:1565:b0:391:9b2:f49a with SMTP id ffacd0b85a97d-39109b2f905mr4268630f8f.55.1741019425450; Mon, 03 Mar 2025 08:30:25 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-390e47a7b88sm14820945f8f.40.2025.03.03.08.30.24 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:25 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 04/20] mm: move hugetlb specific things in folio to page[3] Date: Mon, 3 Mar 2025 17:29:57 +0100 Message-ID: <20250303163014.1128035-5-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: gXUGd8ivnOvWAoMRel7VnXfekuvZyOJ7gC2mh-A2114_1741019426 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 23373180026 X-Stat-Signature: 5fn3bauu5sq8g163wq4bcpyabzrog8k8 X-HE-Tag: 1741019429-748821 X-HE-Meta: U2FsdGVkX19DhZ/I2qyMxdDOYJEN2rrdPmxyoGAOsUYhtXeWrRbO5hMVoOGY378qsgR5+kRl27UK8omwZgIoSeOrYwK4TRjDUp9th8oRJ1QTfacpI7CpXA8hqOoJDBXU/+1O9z74DpytcY/wGMLkRlvSx3NX+7vwBHm4ugUgVQMtI2uTJz4ZXXKmKoEEZtjUhG9UWi3gVKXMsTvopaavPlaKnQSezfWsXOBQcI+JEfGQF3n2HW+iV8ECX83cW6Qrs85YFQbyBFzPNkd6fMmE4WDZ26NkWXOJgQBt2Mkl0h5l5NPousGJhbASljGU3xlzQ0InIu4wKyXRT+uizGChHd8dRMBMRXAriRhplq2NxMWxymXsjm3nHDi6Isy5+iEzgV6yriLnyzeCD404lw66LooC/7XTJdGhUOM8FrXMpHie7ZE3psrnW3O0ROwHys4ha8GBnlZtZldthzMCVeY3gMxJ5kQrcFgLpwHpQqUutT1kwbF9yws5faZkHG1GpDi9D2kjuZ+UZzhpddkz8YsBELzOc5DeFRB7kWeNOt6zXhKmvEewn7DjQGQ1j/yo4VObYxpjB/TswzGDAnbGGgghNOT8rO0hb5/lzw8UUN8B5iL/BmeUiK+qNyFZUzTxlIXRGObl5SB0ycpBsBXpYdTL1OcygOjuTBiS5ILL5rZ+337TYASsI+XdfFxKb8GaxHgETc8k9B2wcN3tAPFZvxK64pbYuo/9bqjTw6B2L2BdOG8pNrkqsd3EicQcDbF7ZcP2zHKnNTaiIQqVtlflFxluHKmvpKk4nBvE09NFWDmoX8iif80ILP85gR0k4+5IEHOzqSOj75nCz2pSaTsFTVeF2O6g5hb2q4NJyYaa5H9mQ5n3Effk7jIyZ9EsDEgqRUlw+GvZ7zKruyVWwL/jDegN06kF3WgeEytwkGOs8KVnHQw89nNTHqx9WqAOoUE/i7Adtj6cVNwugFCyGiGUcKe TRqKLiwX BDR8FVsVqdFn8/tsEnNfZWJwxM3v3zgVSc6fyF3jy4IPB9/u6a/MB4wAbTnbTUes3wqxGv6YPC5vRERFEIghLEPm074ir8ACe+B2K7GpAQEhzzKhIv6VjPfv0GsLluV7O41mpu6hPXHXeVMnxzCRI8gkWsQ7M0CGFBuCu3sHKS2JoaOpj3IqBb7Z8nPFKyCf6YE4+ltLTpCFRLnivFNW8QKC1UbvpOW+WO/icN8PAAofo/HBDRYervwMauT4jksoT89EtVAMPWcwIEeCEbiyP05eaRMZV2J+yON5SYZLbb02HmPDIvVJGYEGdM6rFr2H1+o4gaGr7v0Zs2fiJ5LgI6QKlEJCzEj9e5oWYhqSZXh5/qokU5poWV/1Xbq7akqjDFivjmKhRzQXn1nAbB56MoIHNOciqEuG+qFAM+dJZsrCGua7REnOOOa65TQMET+jOnEq85c6jtkPGIVnwSuFhb1y5yDXRprrvy/RY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's just move the hugetlb specific stuff to a separate page, and stop letting it overlay other fields for now. This frees up some space in page[2], which we will use on 32bit to free up some space in page[1]. While we could move these things to page[3] instead, it's cleaner to just move the hugetlb specific things out of the way and pack the core-folio stuff as tight as possible. ... and we can minimize the work required in dump_folio. We can now avoid re-initializing &folio->_deferred_list in hugetlb code. Hopefully dynamically allocating "strut folio" in the future will further clean this up. Signed-off-by: David Hildenbrand --- include/linux/mm_types.h | 27 +++++++++++++++++---------- mm/hugetlb.c | 1 - mm/page_alloc.c | 5 +++++ 3 files changed, 22 insertions(+), 11 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index e81be20bbabc6..1d9c68c551d42 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -405,20 +405,23 @@ struct folio { unsigned long _flags_2; unsigned long _head_2; /* public: */ - void *_hugetlb_subpool; - void *_hugetlb_cgroup; - void *_hugetlb_cgroup_rsvd; - void *_hugetlb_hwpoison; + struct list_head _deferred_list; /* private: the union with struct page is transitional */ }; + struct page __page_2; + }; + union { struct { - unsigned long _flags_2a; - unsigned long _head_2a; + unsigned long _flags_3; + unsigned long _head_3; /* public: */ - struct list_head _deferred_list; + void *_hugetlb_subpool; + void *_hugetlb_cgroup; + void *_hugetlb_cgroup_rsvd; + void *_hugetlb_hwpoison; /* private: the union with struct page is transitional */ }; - struct page __page_2; + struct page __page_3; }; }; @@ -455,8 +458,12 @@ FOLIO_MATCH(_refcount, _refcount_1); offsetof(struct page, pg) + 2 * sizeof(struct page)) FOLIO_MATCH(flags, _flags_2); FOLIO_MATCH(compound_head, _head_2); -FOLIO_MATCH(flags, _flags_2a); -FOLIO_MATCH(compound_head, _head_2a); +#undef FOLIO_MATCH +#define FOLIO_MATCH(pg, fl) \ + static_assert(offsetof(struct folio, fl) == \ + offsetof(struct page, pg) + 3 * sizeof(struct page)) +FOLIO_MATCH(flags, _flags_3); +FOLIO_MATCH(compound_head, _head_3); #undef FOLIO_MATCH /** diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 70a012af4a8d2..c15723c8d5e7f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1634,7 +1634,6 @@ static void __update_and_free_hugetlb_folio(struct hstate *h, folio_ref_unfreeze(folio, 1); - INIT_LIST_HEAD(&folio->_deferred_list); hugetlb_free_folio(folio); } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ae0f2a2e87369..2fc03cb13e49d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -975,6 +975,11 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) goto out; } break; + case 3: + /* the third tail page: hugetlb specifics overlap ->mappings */ + if (IS_ENABLED(CONFIG_HUGETLB_PAGE)) + break; + fallthrough; default: if (page->mapping != TAIL_MAPPING) { bad_page(page, "corrupted mapping in tail page"); From patchwork Mon Mar 3 16:29:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34696C282D2 for ; Mon, 3 Mar 2025 16:30:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BCD2B280005; Mon, 3 Mar 2025 11:30:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B7E2A280003; Mon, 3 Mar 2025 11:30:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A7B5280005; Mon, 3 Mar 2025 11:30:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7DFE7280003 for ; Mon, 3 Mar 2025 11:30:44 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D679FC0C4D for ; Mon, 3 Mar 2025 16:30:43 +0000 (UTC) X-FDA: 83180778366.23.8E78BCA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 836AA4001A for ; Mon, 3 Mar 2025 16:30:41 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gDKY7Ir5; spf=pass (imf07.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019441; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2ZRGaM3lOA1apQzTwMmA5Ztu3DsqGrxsLDUuGH3sRgQ=; b=glz5HjgYf+p7y8j715S6BBZpYdkgDMXm7ujoDwYPEM0ZTj5+qFD1d2Gp6F/BjzDlLK0O45 Wskodme88/N+TY+FyaZi1s8mzVbBSuodBTNjNLCg9NSUK/sKQmnl5O/Wbcn4GSbLq3LkW5 TfUmcN2mc57jhoT4BVMLCaxwX6L8MDM= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gDKY7Ir5; spf=pass (imf07.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019441; a=rsa-sha256; cv=none; b=YoDXyXK5QkKzlkTlwQ8GbQ/EmLHjpf3rW8U8OoICCARqai8n6KMQCHMxk5PX+w5ntpxk90 PuSLIaPvoLY45I2p3SGurXrnIBLS4z3PKjkpBrN3VyPkhuSlfhAbiPc5amuS0O8pHopVJ6 pWlrJoIehJCtJV0C3xVvB/M/RNTipeQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019440; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2ZRGaM3lOA1apQzTwMmA5Ztu3DsqGrxsLDUuGH3sRgQ=; b=gDKY7Ir5ZJ0jUZbwtk63BIBEYZBY8egfhG72VNwUbI6+rfl4u6BRLGNDc6PlTYJhD6/ZK/ /m86v/Xxb/jKJLvLIJkASdyrTeqLiY86upoL7wYL7XYGpN1ITAofl2tPRho8FyrCZkDaug S4fI9Nle+/k99TwxAJH7vBewR49Lihw= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-548-t5uJd7M9OSWuPAVv85wHFQ-1; Mon, 03 Mar 2025 11:30:29 -0500 X-MC-Unique: t5uJd7M9OSWuPAVv85wHFQ-1 X-Mimecast-MFC-AGG-ID: t5uJd7M9OSWuPAVv85wHFQ_1741019428 Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-438e180821aso24934225e9.1 for ; Mon, 03 Mar 2025 08:30:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019428; x=1741624228; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2ZRGaM3lOA1apQzTwMmA5Ztu3DsqGrxsLDUuGH3sRgQ=; b=CTCHXUsWRN8gXWnoXEVPmlF0KYB3G5lG1cgeqNjVDEVvvgCSJmznxgclu0fTyj7Jkq XSVc9Zmd36PhFbioLtUKwNi1k7cSxGLOp86Ntk4CDE8+IsN0ZplfNo+oXTzRllzFz/vT kxEOQ0aNl6/huGC8K0yd/VW44c0Wu1kZ9tI/F+DP0DPlM06uF8M9ndha8hZLoWAi34hY TKOfZzboRcE1Sv1yprmc6lQvqgT9xhbvuoG7McT0036UI1riJqYmH/gM2wKGp3izMhbM 2sA6Nb2pU88NhKkze2fGa0XAzzYboR0kNk+SgexS9WI1fRkWlknxwv8DS968Xrq4hl/y kZdA== X-Forwarded-Encrypted: i=1; AJvYcCWDWTFz65eu12c3ILL/O5sMXzX2FdADJLn6odE2Uhh2jTw7WB2dK/qVIX2xzaNaXaZrkOFLeL0OBw==@kvack.org X-Gm-Message-State: AOJu0Yxbu/q+nOW0ncVXe22olQXdw8u8W7DykfEPHqbSkVbugnc6Pwr4 5+OqLCovr/5tiYLRUc9p8gdKnokEOKbz5vSXjZdUmy7EUCr9TUj8r8QHR7pdFJjLQciHLMGkW6x fN5NIELp8dkOCtXmGRzZHMP+kJr2uaEgRhKdZ8F+7dwo5U2Tq X-Gm-Gg: ASbGncvh2SUsdP28epor59be8JYx9Xr5ykKXy3vBOVd4Xz7KMlKhuVnOvp5hzdir2K2 TADvHHhYAaweoJz2s62MfSwkWo+3MRJoSVC8TFHInJLEqauOIxxDTapVMdS4NaHMtbSvSuK0Gqw Ae1BKakMvS9/CP4Rp4WyVrDnL4mzolB11aHHZUtsZSYSfVuI9s0YtyyBxTldG5SB+Ozfpe0CMfn 3OYEMqK7L1LygNWdXJp6Ydg0dN3qi3C+JgvUO5Rd0q7heubp9Cw4UEhaWobtdPS3FFN9HmgTeOs wEOjSnRlGfRHvp0PGtvSgiBS7u9W27FSxjusxZRu/67g5E+MA/9e7shPcdBdiQT5H6vwtnaTFbN P X-Received: by 2002:a05:600c:a0b:b0:439:8c80:6af2 with SMTP id 5b1f17b1804b1-43ba675dc1emr98607995e9.21.1741019428078; Mon, 03 Mar 2025 08:30:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IF6dZzZR0lDpQGPMEmueLq9IDDBNY6mKLfVqXKdpTe4yUOjFN5iEduxrBBHG4nXUacNUeptAQ== X-Received: by 2002:a05:600c:a0b:b0:439:8c80:6af2 with SMTP id 5b1f17b1804b1-43ba675dc1emr98607715e9.21.1741019427652; Mon, 03 Mar 2025 08:30:27 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43bc52369f2sm28653165e9.38.2025.03.03.08.30.26 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:27 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 05/20] mm: move _pincount in folio to page[2] on 32bit Date: Mon, 3 Mar 2025 17:29:58 +0100 Message-ID: <20250303163014.1128035-6-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: QWwgrjuq_TIol69j700-52Fxckih9iR_cLpeE5YDFvc_1741019428 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Stat-Signature: kr87ikywcz35eu3rt6h44fa5rdqtzpad X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 836AA4001A X-Rspam-User: X-HE-Tag: 1741019441-76874 X-HE-Meta: U2FsdGVkX19xgOmbI75WWTAdNB26ydq2R1dZpX9EZgbm6x3JdIgT8gEvLLrlbdfBmrNPp1CtH5nHhejxreumqM5N2MJa4ZHvSIBSwVxtTRCDjAhEbx1P/ZbnBQk2pToFiKVGvnJ5e0I1ruF9hC4SwearD7y+3TAX7gf2ocE8+7ZMKTBNkmrPBW1tWd7BF7vvY1CL1coZRRVUhP8qirUae3L42gfr/gCHqh49JSqQttZ3OtcJCASk6uC/oMF+hOuoM+YsCZlPldANhLXHTFG0ZBVzCDcoonx4zj9gAQA7uN9GuSXGYn72W0pCD+SgzRAcnuenfc6dur3pc6JfUfWDeXd1s1CVt/lORykx7TccGIKn/AJQ/RpPmIMsYwr3HnKk3qB3dlTtJrkVvRSkzpY702a2K04mazEJXXlYPXb/b4CzfDRnot6XvEt4qs2UJ35uBnN6+Jwr2aNMgt2Su9i6kx7YaiB3NzZG19zoteR6gfw8sva9iSPoSdHXMJ4jS9zCKvuWWDnaEP+tV04Htvo0rvuwa/erX4umOhuFpHq0qVhy2dlyoDm7Deci8C8ezLepYpE9K9SRH4zMgBdBuRNgNCycFWgRvaUz/+o7Hf7q/HAABvXWQvE0NhgwulCU+kGQT0vzOp7HpmezmSosXWkQB5y52c76LQk2eZFiY1uWAhKDEKk1kDVm9yOWp20y8lROCMc/33JUONeSIj4hRn2s8/aY2u7Cez6Fkl0XYHAmhAe6RFF7u2zdUD6Quz5Gi0ZX0o+5a/ZJlUihn4LFWw0PTYY/WjTaidZ+BzhFYxfzL9dV7jnFEqCUOpzqFi7R6QaBAVW+YQUuOZiEpoWKMgdAG4LXwY/qlC+Z04QeGpfXy7iim0F0SzYhiN99Jzqecj8gmv7G+rpudX+sXYR7bMs3J2gtiT5ufu4VVDSUn4QZvQiNY3thEWenSmddEqvKoDWMBFwaBSKWN6Nplk2DhRS 2F6lJe4d vBhVyUNFiII1DzYMIuEliBy8cs8ffr+8QmpDrUXSSvlQ42qz/Z0qFiC3ucv941irF4rlcUUs1maoA5uSGWFuasRpgfya0oTj9u96jy7G3sNNRXcPTnaHWQ4WwfHYZKknLJwPO/xzPgmlKnYwQfLSBGv7xQx8sOAAhoeifFGub/jC+NsUBosdS/EYZzMHr7t8FlQfcxoOs3vg++CrhyrG8FbMWJtXDEQF/RRGXSsyOqbvljKHLBFd7YUuYPPNVcVRQSB5faCdAz6YvXdoPOXRj5x+m+Mb5RBqR6CtzYZ5v1NfsIqprcbfKKus589i6jBgWwgTQSi0mdEHOhtw5Uti4PlOaVHtbEzmxA+v49BOcHnLjfNHU5Sgr/nittxn91t0rgIeSNJ1XaUfIih4bUL82MS0uqjU8jxDfKwoDK45iwQqTyba/AIpMSopairWsAMcky4QmMxTrxomKmhGPnY5KZcuXwMic9knmjD40xQFpEmQ3DNLWh9+U/8zp8R7C0Hj4JHANBSOQ3AfQVA4dU5XuXBghOBhSiTrJmrJESIRSkKq1b3r04JPhRamk/W2JMMtCOSxf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's free up some space on 32bit in page[1] by moving the _pincount to page[2]. For order-1 folios (never anon folios!) on 32bit, we will now also use the GUP_PIN_COUNTING_BIAS approach. A fully-mapped order-1 folio requires 2 references. With GUP_PIN_COUNTING_BIAS being 1024, we'd detect such folios as "maybe pinned" with 512 full mappings, instead of 1024 for order-0. As anon folios are out of the picture (which are the most relevant users of checking for pinnings on *mapped* pages) and we are talking about 32bit, this is not expected to cause any trouble. In __dump_page(), copy one additional folio page if we detect a folio with an order > 1, so we can dump the pincount on order > 1 folios reliably. Note that THPs on 32bit are not particularly common (and we don't care too much about performance), but we want to keep it working reliably, because likely we want to use large folios there as well in the future, independent of PMD leaf support. Once we dynamically allocate "struct folio", fortunately the 32bit specifics will likely go away again; even small folios could then have a pincount and folio_has_pincount() would essentially always return "true". Signed-off-by: David Hildenbrand --- include/linux/mm.h | 11 +++++++++-- include/linux/mm_types.h | 5 +++++ mm/debug.c | 10 +++++++++- mm/gup.c | 8 ++++---- mm/internal.h | 3 ++- mm/page_alloc.c | 14 +++++++++++--- 6 files changed, 40 insertions(+), 11 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 694704217df8a..c1414491c0de2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2004,6 +2004,13 @@ static inline struct folio *pfn_folio(unsigned long pfn) return page_folio(pfn_to_page(pfn)); } +static inline bool folio_has_pincount(const struct folio *folio) +{ + if (IS_ENABLED(CONFIG_64BIT)) + return folio_test_large(folio); + return folio_order(folio) > 1; +} + /** * folio_maybe_dma_pinned - Report if a folio may be pinned for DMA. * @folio: The folio. @@ -2020,7 +2027,7 @@ static inline struct folio *pfn_folio(unsigned long pfn) * get that many refcounts, and b) all the callers of this routine are * expected to be able to deal gracefully with a false positive. * - * For large folios, the result will be exactly correct. That's because + * For most large folios, the result will be exactly correct. That's because * we have more tracking data available: the _pincount field is used * instead of the GUP_PIN_COUNTING_BIAS scheme. * @@ -2031,7 +2038,7 @@ static inline struct folio *pfn_folio(unsigned long pfn) */ static inline bool folio_maybe_dma_pinned(struct folio *folio) { - if (folio_test_large(folio)) + if (folio_has_pincount(folio)) return atomic_read(&folio->_pincount) > 0; /* diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 1d9c68c551d42..31f466d8485bc 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -387,7 +387,9 @@ struct folio { atomic_t _large_mapcount; atomic_t _entire_mapcount; atomic_t _nr_pages_mapped; +#ifdef CONFIG_64BIT atomic_t _pincount; +#endif /* CONFIG_64BIT */ }; unsigned long _usable_1[4]; }; @@ -406,6 +408,9 @@ struct folio { unsigned long _head_2; /* public: */ struct list_head _deferred_list; +#ifndef CONFIG_64BIT + atomic_t _pincount; +#endif /* !CONFIG_64BIT */ /* private: the union with struct page is transitional */ }; struct page __page_2; diff --git a/mm/debug.c b/mm/debug.c index 2d1bd67d957bc..83ef3bd0ccd32 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -79,12 +79,17 @@ static void __dump_folio(struct folio *folio, struct page *page, folio_ref_count(folio), mapcount, mapping, folio->index + idx, pfn); if (folio_test_large(folio)) { + int pincount = 0; + + if (folio_has_pincount(folio)) + pincount = atomic_read(&folio->_pincount); + pr_warn("head: order:%u mapcount:%d entire_mapcount:%d nr_pages_mapped:%d pincount:%d\n", folio_order(folio), folio_mapcount(folio), folio_entire_mapcount(folio), folio_nr_pages_mapped(folio), - atomic_read(&folio->_pincount)); + pincount); } #ifdef CONFIG_MEMCG @@ -146,6 +151,9 @@ static void __dump_page(const struct page *page) if (idx < MAX_FOLIO_NR_PAGES) { memcpy(&folio, foliop, 2 * sizeof(struct page)); nr_pages = folio_nr_pages(&folio); + if (nr_pages > 1) + memcpy(&folio.__page_2, &foliop->__page_2, + sizeof(struct page)); foliop = &folio; } diff --git a/mm/gup.c b/mm/gup.c index e5040657870ea..2944fe8cf3174 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -109,7 +109,7 @@ static void gup_put_folio(struct folio *folio, int refs, unsigned int flags) if (is_zero_folio(folio)) return; node_stat_mod_folio(folio, NR_FOLL_PIN_RELEASED, refs); - if (folio_test_large(folio)) + if (folio_has_pincount(folio)) atomic_sub(refs, &folio->_pincount); else refs *= GUP_PIN_COUNTING_BIAS; @@ -164,7 +164,7 @@ int __must_check try_grab_folio(struct folio *folio, int refs, * Increment the normal page refcount field at least once, * so that the page really is pinned. */ - if (folio_test_large(folio)) { + if (folio_has_pincount(folio)) { folio_ref_add(folio, refs); atomic_add(refs, &folio->_pincount); } else { @@ -223,7 +223,7 @@ void folio_add_pin(struct folio *folio) * page refcount field at least once, so that the page really is * pinned. */ - if (folio_test_large(folio)) { + if (folio_has_pincount(folio)) { WARN_ON_ONCE(atomic_read(&folio->_pincount) < 1); folio_ref_inc(folio); atomic_inc(&folio->_pincount); @@ -575,7 +575,7 @@ static struct folio *try_grab_folio_fast(struct page *page, int refs, * is pinned. That's why the refcount from the earlier * try_get_folio() is left intact. */ - if (folio_test_large(folio)) + if (folio_has_pincount(folio)) atomic_add(refs, &folio->_pincount); else folio_ref_add(folio, diff --git a/mm/internal.h b/mm/internal.h index bcda1f604038f..378464246f259 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -721,7 +721,8 @@ static inline void prep_compound_head(struct page *page, unsigned int order) atomic_set(&folio->_large_mapcount, -1); atomic_set(&folio->_entire_mapcount, -1); atomic_set(&folio->_nr_pages_mapped, 0); - atomic_set(&folio->_pincount, 0); + if (IS_ENABLED(CONFIG_64BIT) || order > 1) + atomic_set(&folio->_pincount, 0); if (order > 1) INIT_LIST_HEAD(&folio->_deferred_list); } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 2fc03cb13e49d..594a552c735cd 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -963,9 +963,11 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) bad_page(page, "nonzero nr_pages_mapped"); goto out; } - if (unlikely(atomic_read(&folio->_pincount))) { - bad_page(page, "nonzero pincount"); - goto out; + if (IS_ENABLED(CONFIG_64BIT)) { + if (unlikely(atomic_read(&folio->_pincount))) { + bad_page(page, "nonzero pincount"); + goto out; + } } break; case 2: @@ -974,6 +976,12 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) bad_page(page, "on deferred list"); goto out; } + if (!IS_ENABLED(CONFIG_64BIT)) { + if (unlikely(atomic_read(&folio->_pincount))) { + bad_page(page, "nonzero pincount"); + goto out; + } + } break; case 3: /* the third tail page: hugetlb specifics overlap ->mappings */ From patchwork Mon Mar 3 16:29:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88B4BC282D1 for ; Mon, 3 Mar 2025 16:30:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1947C280006; Mon, 3 Mar 2025 11:30:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1460A280003; Mon, 3 Mar 2025 11:30:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F01AB280006; Mon, 3 Mar 2025 11:30:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8BEFA280003 for ; Mon, 3 Mar 2025 11:30:47 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D79E1120C34 for ; Mon, 3 Mar 2025 16:30:46 +0000 (UTC) X-FDA: 83180778492.19.4F3C31C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 9A091C002B for ; Mon, 3 Mar 2025 16:30:44 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=f6o4DVPf; spf=pass (imf28.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019444; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5CNQID9OyUlmGzlaLMJyJN9WCGZqSjZdKvQcCGQkCwE=; b=D9UAHU4MFR0hSuk+i3P6XqRldvFG3X7jZIr1+hJa/yNzyWKuQBfrLOhhX58GQsb0gfh4Wn C+ZaWSbR0tKUnUTqpL55eN+znXFarbPc0WmxIW1n6wkm9eHBRivqG5CILFwZuavR4q0Qpa E44HGMAljYwFSaajrcPuJleTdS4IUwI= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=f6o4DVPf; spf=pass (imf28.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019444; a=rsa-sha256; cv=none; b=kBYyvbKyso1Jd9ds0w/PP6aGfZI+9yIFu6DVekOf5zwfMhwZFOfeQzZXJ+d9pF37Ck2M5V F74OsHPNsshSHNobdQ0llDsZblKu6cRA0L+l5LuX+PKLFDNOCVlUVRJ6v//OLNz9p1VmHN VzM3/a3t4WFbgSNe1r4OtOROlMRwHcs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019443; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5CNQID9OyUlmGzlaLMJyJN9WCGZqSjZdKvQcCGQkCwE=; b=f6o4DVPft8VhQ8iuHgeN76kz28ChcBpdoGlbyIvoL/J+sU3QBdB0SM2ue6x/+3tVwQeTDy lm8TdcXgrmH60c8k/qXNt4hY2DgYX92cfZyi5cB53WpWwPFuiXE7s9vcDZf8v8N+CK1pii K//Som96Yy5zUQ851UKL3Sj3PkPaWLI= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-609-Sn3AMhINPdKZfu0e7L31lg-1; Mon, 03 Mar 2025 11:30:32 -0500 X-MC-Unique: Sn3AMhINPdKZfu0e7L31lg-1 X-Mimecast-MFC-AGG-ID: Sn3AMhINPdKZfu0e7L31lg_1741019430 Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-4393e89e910so26048655e9.0 for ; Mon, 03 Mar 2025 08:30:32 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019430; x=1741624230; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5CNQID9OyUlmGzlaLMJyJN9WCGZqSjZdKvQcCGQkCwE=; b=Rvexw4+I5IsB3Zdb/G/YJ6LlYrxbTjbOZe9qJS73gC5a8+gdyK6+RfxOgTA1m84tUG 5UL1wgF5KGVY4GR1sqwI+isrvL75WkNCoZ9+2mIoqe/5+ki35mhLN6qDWqGKnHHC7A7W x2YlxKBnt0Cdd9Rz2FLDNG78wyEwzkpKwusdERn8ZEbs+CSNG2R2FyrMN17N1AIoMbVA c6nVk1eTNfWSDw+Ahu6OVVdwA3GbZ+lyuHY9itmF8L4fyPfH6BPdnHklvqgqGGZ6yat/ EQ9u8957ok4yoPi6+guAj4rsa8kjeuPCpQ8OMDUZffvy6GOltj+yPhDzLvbmYqhCnN0n WreA== X-Forwarded-Encrypted: i=1; AJvYcCWsygNIUZZsT3MSlh+F1Dq+sEMBHl+kOs4lTx58sEkYgUotQxrq5cQIj5ucigpWzcJOLoJOjr1aBA==@kvack.org X-Gm-Message-State: AOJu0YxoiOYySt7vEg17MZDbmwk5NA5fC3Jj/ZHfaAbuFhBkJW2LhTaw WtiOkq8M7ystyAd19Y6HfPSl1RAIrokE9dzUmCMavoc16vAHqe0Yn87l8zA1U3gz52kSuO0YjuL Dr10GYIIE7i41TMmu4Nc7adxt+ouVhn10IYoMb0lqlzC/e6TB X-Gm-Gg: ASbGncv1hukfAsrkFGqBjcbl/DGaJQvlr9gbANfm+0FomVjSDQB+k0/qqwlOKnmta8c Eo4S2Om0mMzLHffZxblw+9Qz9TkI1aJUKqpM5IAWCn7qp/jRHmEzVZAylZYoU0oQ13yepr7niTx 8RVfDSTRlU+YD0xxk/96yNGg5a+3yXc0q9uGeYA8gVwOZxRSmpU4PcRvulCwKx3x2r1VzQ/+0CN aAUGZqupVP2VQ6nAePtlEf1Z8lNn7YqkzcJIYBVWZZtUF6WIbpEnm/rX93fPaAzxdsUoWF+Rwu5 lyihRb4yCtxMkf8ojdeQORABLoaaiYIc2eYtmjV+xXhyFVAOx6s1tPtNIw4Db09+OEHElz8JRI/ n X-Received: by 2002:a5d:6da3:0:b0:38d:e48b:1766 with SMTP id ffacd0b85a97d-390ec7c67c4mr11859523f8f.6.1741019430372; Mon, 03 Mar 2025 08:30:30 -0800 (PST) X-Google-Smtp-Source: AGHT+IHRquSNZRyjpwgx3Bpqn4SKjwIglVAovJ799aL6PA2lqrvYWWUmjvtNcaSqxs0FHQnHdYBh0w== X-Received: by 2002:a5d:6da3:0:b0:38d:e48b:1766 with SMTP id ffacd0b85a97d-390ec7c67c4mr11859482f8f.6.1741019430024; Mon, 03 Mar 2025 08:30:30 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43bc4478f70sm23988695e9.0.2025.03.03.08.30.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:29 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 06/20] mm: move _entire_mapcount in folio to page[2] on 32bit Date: Mon, 3 Mar 2025 17:29:59 +0100 Message-ID: <20250303163014.1128035-7-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: WXwzAUG6XIrRW-32m-wQt0U-IIAI92N8ab_6mANhp7U_1741019430 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Stat-Signature: 6zgwxqpy6hfoqa689o3119nyu7xkjmsd X-Rspamd-Queue-Id: 9A091C002B X-Rspamd-Server: rspam06 X-Rspam-User: X-HE-Tag: 1741019444-795831 X-HE-Meta: U2FsdGVkX19xEItDmAtPAypHDZa2exaEF5VEd326SsvLE8RYaulanQ1kxOfXfRDRgwuitcOEQor6c2XQ/N935pPH1D/trOg6raVppZ9VRXW1/JZvedvBiCRwG0EX3Tm13utFm/ZKDuF2fY5c4mAbWwJJgi24xzOCbobd/WXcCMTkIynvNcWWgmHy+7iU+urO9u6zpy2iNWdwdT5X6kOpugpKaQQ9iMG+RKtgH+G8UaY1XUavwkXXNYz3K6xAPj3gKSyEY03xMeiYvLdvvP1REte2WuLi4ZjAkvtORc1BZUivhUlIPvgG7/gbeUUkmB3cIP/LvsjuQbctrYoR5WzX+l+WdR5qqjpNShBkzbu/j1IHK5nXqN2tZachHbLpe24XonGSH11Gi4ChSxCCKbmPmPzisRtE4iFXKoE+JMKoXBo5BEsJn++VWbcl38jvPxoUqfapuSPAIhuEhJd+w2cmtwfLylbzykLDcBFBmHv4YwLZGBxm9QJIScvoavYFYM0Ja3EriDoC8xH00tB5e7FiUWdWHv9XMxID7qCLYjYI7jsS9QZahk/J/d0xc+HGRuF6fa0h8uixBT9nHGw93rLSWffZKQ6hJo2lhOoi3JvBI22wFIgehmyKqoqdPvQs6dQuq9SnwssMXqPjK5mba63x12cwuBzTkAduRjItxRLC99l5Nlvk7P8P+epk0tj/qEONBILc2PdCV7ULfTAToGFUQHO1t9P8lJlAIcyM9ljYXEf9IRbfq3c3uXJ/OzuPnGIAQWXAqp3KSe5W/2EqRGoWYXxCA2XHCVyWkd3IgCOCADxVsBY3IFHTPg8FgYA6IB1+forQEOWb0OuD40RHysKoOilJuTDN74dljuyxNhRscncijaaE2XXyS8pkRMZpRm0/szCiXQ8v7Nr9d5hXr2qXdaSTOXQkvVSlciZVsOMcZEia4pEAbsnFeFqVfKYHnnlMaV2FdaWZZ+NNuO1LrJI +/yu86cD rRYfjNuMFrHbl0KVI+Lv+n1DdQ2hSibkzJA+XIz2PKgPgf46MplEFBriObdzElmruoPKWntNzrVu3X220HTEHyz1UJiWB8uO4aLwriv+UI/+XaU11N7q0Wi7nIqlQ4S8b1/zOBO1lPTHhjef/hdTj4eVvTdOo76mM9C95uD/dZG0Zdz724YyY3v+RSNMtOw4lMHvw/rPJIQkTOQTLtMXowkjCX8jb+Ealy/rrWZ7UN+wlBmuSgV1CVt+SRX9C+KJU0min+o7Ck9tj1D7bku6rL8SSELvc0YfQG/M+bLbHTya9S3qOrT5FQDLAi4goCIC5PSVoCSt2lKFqILxhqa0ETND20GuQa3hMdmxpHl9W186bIO11IjDT8Ox6o+trzRIwU9HpdAqUft84ZHqyRajLEvpn/D6vpij6TD9kss0v/JgqjFrlM6wybm0mdMX7Pj6j6i5tHlVDkgR0a3PU41Q2yYCRWKDsBUiSeQ/3cy+Js4JCsT0wNzHy3QkIWTFoNcIEWVEfpWWqymPoQjzzPuDN5+iP9Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's free up some space on 32bit in page[1] by moving the _pincount to page[2]. Ordinary folios only use the entire mapcount with PMD mappings, so order-1 folios don't apply. Similarly, hugetlb folios are always larger than order-1, turning the entire mapcount essentially unused for all order-1 folios. Moving it to order-1 folios will not change anything. On 32bit, simply check in folio_entire_mapcount() whether we have an order-1 folio, and return 0 in that case. Note that THPs on 32bit are not particularly common (and we don't care too much about performance), but we want to keep it working reliably, because likely we want to use large folios there as well in the future, independent of PMD leaf support. Once we dynamically allocate "struct folio", the 32bit specifics will go away again; even small folios could then have a pincount. Signed-off-by: David Hildenbrand --- include/linux/mm.h | 2 ++ include/linux/mm_types.h | 3 ++- mm/internal.h | 5 +++-- mm/page_alloc.c | 12 ++++++++---- 4 files changed, 15 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index c1414491c0de2..53dd4f99fdabc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1333,6 +1333,8 @@ static inline int is_vmalloc_or_module_addr(const void *x) static inline int folio_entire_mapcount(const struct folio *folio) { VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); + if (!IS_ENABLED(CONFIG_64BIT) && unlikely(folio_large_order(folio) == 1)) + return 0; return atomic_read(&folio->_entire_mapcount) + 1; } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 31f466d8485bc..c83dd2f1ee25e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -385,9 +385,9 @@ struct folio { union { struct { atomic_t _large_mapcount; - atomic_t _entire_mapcount; atomic_t _nr_pages_mapped; #ifdef CONFIG_64BIT + atomic_t _entire_mapcount; atomic_t _pincount; #endif /* CONFIG_64BIT */ }; @@ -409,6 +409,7 @@ struct folio { /* public: */ struct list_head _deferred_list; #ifndef CONFIG_64BIT + atomic_t _entire_mapcount; atomic_t _pincount; #endif /* !CONFIG_64BIT */ /* private: the union with struct page is transitional */ diff --git a/mm/internal.h b/mm/internal.h index 378464246f259..9860e65ffc945 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -719,10 +719,11 @@ static inline void prep_compound_head(struct page *page, unsigned int order) folio_set_order(folio, order); atomic_set(&folio->_large_mapcount, -1); - atomic_set(&folio->_entire_mapcount, -1); atomic_set(&folio->_nr_pages_mapped, 0); - if (IS_ENABLED(CONFIG_64BIT) || order > 1) + if (IS_ENABLED(CONFIG_64BIT) || order > 1) { atomic_set(&folio->_pincount, 0); + atomic_set(&folio->_entire_mapcount, -1); + } if (order > 1) INIT_LIST_HEAD(&folio->_deferred_list); } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 594a552c735cd..b0739baf7b07f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -951,10 +951,6 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) switch (page - head_page) { case 1: /* the first tail page: these may be in place of ->mapping */ - if (unlikely(folio_entire_mapcount(folio))) { - bad_page(page, "nonzero entire_mapcount"); - goto out; - } if (unlikely(folio_large_mapcount(folio))) { bad_page(page, "nonzero large_mapcount"); goto out; @@ -964,6 +960,10 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) goto out; } if (IS_ENABLED(CONFIG_64BIT)) { + if (unlikely(atomic_read(&folio->_entire_mapcount) + 1)) { + bad_page(page, "nonzero entire_mapcount"); + goto out; + } if (unlikely(atomic_read(&folio->_pincount))) { bad_page(page, "nonzero pincount"); goto out; @@ -977,6 +977,10 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) goto out; } if (!IS_ENABLED(CONFIG_64BIT)) { + if (unlikely(atomic_read(&folio->_entire_mapcount) + 1)) { + bad_page(page, "nonzero entire_mapcount"); + goto out; + } if (unlikely(atomic_read(&folio->_pincount))) { bad_page(page, "nonzero pincount"); goto out; From patchwork Mon Mar 3 16:30:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CD27C282CD for ; Mon, 3 Mar 2025 16:30:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 58DD5280009; Mon, 3 Mar 2025 11:30:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 53D6F280003; Mon, 3 Mar 2025 11:30:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 38DD7280009; Mon, 3 Mar 2025 11:30:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 19216280003 for ; Mon, 3 Mar 2025 11:30:55 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id BB2ADA4963 for ; Mon, 3 Mar 2025 16:30:54 +0000 (UTC) X-FDA: 83180778828.30.08E341F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 7EB371A0026 for ; Mon, 3 Mar 2025 16:30:52 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VOULsAUS; spf=pass (imf19.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019452; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6obLj5/+7cSJHkmjj4/LIEhJkcwhtMxgxzLmveILbME=; b=FMShOteff4G9YOY/OjFLhIv/Wn8BiUPlALsneUt6UTn4waAE3TNmPGArwKrjirCR3pSedj A4Ufxcl1GtzwjrNclZYGIisMj3M3sjqT/K8Wp+wo5n3Rc3dMNVIzFIyCNMBUmhlEj7+7qQ OfVliua/kUT/hGFd5AIWlBNbDUVtouM= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VOULsAUS; spf=pass (imf19.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019452; a=rsa-sha256; cv=none; b=wCtsBmtwF0sYbTHBZjRN/V6nA92U00NUi3+xlMkh7TVeK2OSbZZlJtspkr3tiW1nSA5k4Z ywVaeAi0sYLx9QOBLNl9KoZUDDqugArf4ppCBcsLc1/jYnFXgYMd0qClRZROJcC92lgAIg hC5A1VcGCnqJdH3P+gZVI2u8VGq2TCE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019451; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6obLj5/+7cSJHkmjj4/LIEhJkcwhtMxgxzLmveILbME=; b=VOULsAUSHXop6cq7IZQrZhOgluh5jvgKrIPIK4/hAZqdK+qG44KFH80/zNUBTFwP/aTZu/ fsVsQbGAOETHG0S3Ii5O7INw25gmMGdiIfTfNylu6gEsTbquiQSB2HX/mX4Cdh0lukZ2AT qsIcUg/+MlcJP8H7/eX/hPUUIOcoUK8= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-111-FbkYBVvsOKG4GeWc29Ptfw-1; Mon, 03 Mar 2025 11:30:33 -0500 X-MC-Unique: FbkYBVvsOKG4GeWc29Ptfw-1 X-Mimecast-MFC-AGG-ID: FbkYBVvsOKG4GeWc29Ptfw_1741019433 Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-39104223bb5so609668f8f.2 for ; Mon, 03 Mar 2025 08:30:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019432; x=1741624232; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6obLj5/+7cSJHkmjj4/LIEhJkcwhtMxgxzLmveILbME=; b=ZO9hu/ZKRHl804sFPTlau0w2taYRC2fZjWsxlUdjKkw+b+W7EPrp04eW1p/+d0l3E4 GleZi1KRXU8cMyL3TNbDPi71Y0gBMwqczArc0x75Lld+4IhtL/Drx8KjMCJM0/jpyvFr btgJhXmN1JqjrcBs/QDIYM1ro61Gwh2UaJaw5IxRtfMAyEEy/LB85DMeWej456cvMBn0 aBGvUD4TxbIPI9leZeD1bTDNhV9wJat2gFEFPpKJePm1M7XUBwUYyQfSzBGRPH+8gEIA hjXk/arzK5d9ovptWnRMo2jxdZhDHSR+yRoWqQzTBnOUCKIaTSy3SVn5GiMjKdqsmSAn Ohjg== X-Forwarded-Encrypted: i=1; AJvYcCWiwtsxjNdEVl8tZj3bvFO61EAcaOIH7aZwdV3dwwEv2pSKQKRnmp6IZbfsivev8IFVtr+rrBZL8A==@kvack.org X-Gm-Message-State: AOJu0Yx5GcTR46nrKhKSSAF468DIamIZ+9eDMCjCaxdLGtJj1NqjNIjM 2t69vAlz+jNNtGKPo6fU8aEFJRLd/ZRW3NyiDdayvfiNSPR7ikqNz8yEKU79w09n/WNkZKCsPiy BmHVVz9IsbYmz5LTxmoTZLg+YI70892IriMr1uW/D0HCYIkUk X-Gm-Gg: ASbGncstfANPwR1lhRUQMjKPFFEM0Ne5R4A8o70hldG9mHVC4a7Qdy3dwevE9ujDpVN b1A8I13Y0aa17jmkjD/jtUcuElFNB9roNAfm8X8ciV9Af8+RzMFjRnG5OnRVmjiyQWXrJudgpdM r5kVc8vtRqHDiMIx9Ds0G3b5lTNvesaEjoyq9AuGgixL8XSevLOrTHQBRY418y5LfBulyM7yrvC hziy//bemJCIya63y7byrdrJXkFNourP29HAbQcyeslWkdoScaBRYvGwylRgLBDGDtGPFfEDnPL 5hEKoavVilsgWMndPjvE9bD/zqWh5U51LSShHNq9ztIFaQ0PjtCrMjC3Uxhj6ATrEh2R5ymmjT1 0 X-Received: by 2002:a05:6000:21c6:b0:390:fd23:c145 with SMTP id ffacd0b85a97d-390fd23c2c2mr4404031f8f.36.1741019432578; Mon, 03 Mar 2025 08:30:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IG8V3sUJRCUTqrZLWnaeVZz5cWv6qO720rAawUL8eI33aPl04j18jrwhTf4sp/MF8gqzp2DbA== X-Received: by 2002:a05:6000:21c6:b0:390:fd23:c145 with SMTP id ffacd0b85a97d-390fd23c2c2mr4403996f8f.36.1741019432210; Mon, 03 Mar 2025 08:30:32 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43bbfece041sm38658125e9.1.2025.03.03.08.30.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:31 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 07/20] mm/rmap: pass dst_vma to folio_dup_file_rmap_pte() and friends Date: Mon, 3 Mar 2025 17:30:00 +0100 Message-ID: <20250303163014.1128035-8-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: xvT2JL1BSX2mogQmer-hwl-IHdxmXhMxXakwepVH6I0_1741019433 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Stat-Signature: au98ytyzcuxiecqhsrbr4cpxrfa115c3 X-Rspamd-Queue-Id: 7EB371A0026 X-Rspamd-Server: rspam07 X-HE-Tag: 1741019452-681489 X-HE-Meta: U2FsdGVkX181Cy+HkLOF0vwV0ZAjMv/rnzLeYncSDn4Gsg+idb4Uw3kepAioWiaIMSszPF55dsHkyuBeFJ6a5Wmn7XFYDrv4SzeQ03a8Lv0l8GMx0vvZbn1eJI6vl/TeUnqJS4xXRadYD9F+o1j+7tmXhqd4n8d/iipJuYOxJXU2geAe1+Cxahhj9MO0JREAvKVrn5sSJpbzO4Dm0OAvfVkKRVYU2EeLggxXP8ptM3UTfdRwgs0L1BEv1LjSUsbxErOcqCehUI9GxZES666W96oUe9QuMDbl+oIlQVGmkkUcCas0oHJNbwhsDdYZ0HIBDtFWgZTxN/WukuXX9q43s1gvA5/wTueDUF5zcjlcoxytrOjt3Lfm9kAZEkrQ4gtCZJirOwC9uYZH6RWQcmzg0x6tX3U6YNOQO9PSo+9CsYIIiTXUazLk0z4NNOJ3qFGRX44X63IoK4pNOkrLxTUO9O9sb/5XlyVjJgCtTaUOszeX3QHgQR1RU8W4maVL4X7vkXS6NS//JXcVqezDI4x00GZ3Nv7K6OMT7EwxS7ywQv4j29/1fkfE/VBUrz3jePxU3xodEurNk3bJseRlcz/BaLWm/zfERo18wx/zhUYAmlMhgzx0tVuI5wx3/HTFOkU4jxmCY7DMf0Hgb7LispG50/DSafCw+G9AuikmiQfM1fyRVpl++IGNdrJ/xk7Me/WDZ+hesuxOADzv8L2dBHIPR0L3Coc6ni2PKzimVYiX1SRA5ZUXJOPJTc5ydVapfL1yP0WFyauOBR6o+US6UOxSXBl+QDWVckG1teVGtuGcGXO34KTb8klufvoW7obZMmkActA2PIWoQtK00NT9vseDmE9z/JJPk0ZHJb4DmWCXx4TIHTh+C/xVMRgJnC8vLZn2Hd3bxTWCkLfh8QAnesaciReBvwE79n7QAIJehp7uhwDwf5MOEo+zfMbgoyWDP/mgov0JA7nRuwoiGnemibC hnVciMVt OgP9x59LI6Gn8wYb1y2VP0Mt0fEbp4jCXrRwCyjLeUsltADu9RG2HblY22sAiFSKIy+5k3w7gpSiWZerM10rfQagYdPgo0rowof8WLdCem74eutqSM4v54x+TMxVgu+6nrrh6w4gEePcYN4iyk0HjkjaSYCLCxrkzQrdE4ETLCPOilj6TfyX7xtmq8jdg5V/KcjCrjqZJ80RySt0t+u5HmcZ5hV5wTw7aKAiMiWJRLRf8Bd8tjmjr5CJ4Xw8DNDUQiKi/W33vKDNsJQO9W6zFr+U34n+gwMOzWPNYUDR75QrUtp4+rechh0eM5pAYCYafqpX9L88GDJVq+sYNATui0QdnTQLR+Z3FU9e0CEI2zmNgWg5zzZ4Pj/IpAHQ/ZYK2PrcT2uKzdZyuA3ByN9uOEpTM+38Pn/pFg2eS4T6I4SQwVc7Zc9N6VpuQVFI5BF5Z2HAlASyATn/ZcJHpfKNpWcIUOKH6BspHQVP8YISTpoKHlkccBPAMaw76jYCOAv8B/ygbvmkznT+v2Xc/KZmP1aTvtg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We'll need access to the destination MM when modifying the large mapcount of a non-hugetlb large folios next. So pass in the destination VMA. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 42 +++++++++++++++++++++++++----------------- mm/huge_memory.c | 2 +- mm/memory.c | 10 +++++----- 3 files changed, 31 insertions(+), 23 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 6abf7960077aa..e795610bade80 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -335,7 +335,8 @@ static inline void hugetlb_remove_rmap(struct folio *folio) } static __always_inline void __folio_dup_file_rmap(struct folio *folio, - struct page *page, int nr_pages, enum rmap_level level) + struct page *page, int nr_pages, struct vm_area_struct *dst_vma, + enum rmap_level level) { const int orig_nr_pages = nr_pages; @@ -366,45 +367,47 @@ static __always_inline void __folio_dup_file_rmap(struct folio *folio, * @folio: The folio to duplicate the mappings of * @page: The first page to duplicate the mappings of * @nr_pages: The number of pages of which the mapping will be duplicated + * @dst_vma: The destination vm area * * The page range of the folio is defined by [page, page + nr_pages) * * The caller needs to hold the page table lock. */ static inline void folio_dup_file_rmap_ptes(struct folio *folio, - struct page *page, int nr_pages) + struct page *page, int nr_pages, struct vm_area_struct *dst_vma) { - __folio_dup_file_rmap(folio, page, nr_pages, RMAP_LEVEL_PTE); + __folio_dup_file_rmap(folio, page, nr_pages, dst_vma, RMAP_LEVEL_PTE); } static __always_inline void folio_dup_file_rmap_pte(struct folio *folio, - struct page *page) + struct page *page, struct vm_area_struct *dst_vma) { - __folio_dup_file_rmap(folio, page, 1, RMAP_LEVEL_PTE); + __folio_dup_file_rmap(folio, page, 1, dst_vma, RMAP_LEVEL_PTE); } /** * folio_dup_file_rmap_pmd - duplicate a PMD mapping of a page range of a folio * @folio: The folio to duplicate the mapping of * @page: The first page to duplicate the mapping of + * @dst_vma: The destination vm area * * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) * * The caller needs to hold the page table lock. */ static inline void folio_dup_file_rmap_pmd(struct folio *folio, - struct page *page) + struct page *page, struct vm_area_struct *dst_vma) { #ifdef CONFIG_TRANSPARENT_HUGEPAGE - __folio_dup_file_rmap(folio, page, HPAGE_PMD_NR, RMAP_LEVEL_PTE); + __folio_dup_file_rmap(folio, page, HPAGE_PMD_NR, dst_vma, RMAP_LEVEL_PTE); #else WARN_ON_ONCE(true); #endif } static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, - struct page *page, int nr_pages, struct vm_area_struct *src_vma, - enum rmap_level level) + struct page *page, int nr_pages, struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma, enum rmap_level level) { const int orig_nr_pages = nr_pages; bool maybe_pinned; @@ -470,6 +473,7 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, * @folio: The folio to duplicate the mappings of * @page: The first page to duplicate the mappings of * @nr_pages: The number of pages of which the mapping will be duplicated + * @dst_vma: The destination vm area * @src_vma: The vm area from which the mappings are duplicated * * The page range of the folio is defined by [page, page + nr_pages) @@ -488,16 +492,18 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, * Returns 0 if duplicating the mappings succeeded. Returns -EBUSY otherwise. */ static inline int folio_try_dup_anon_rmap_ptes(struct folio *folio, - struct page *page, int nr_pages, struct vm_area_struct *src_vma) + struct page *page, int nr_pages, struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma) { - return __folio_try_dup_anon_rmap(folio, page, nr_pages, src_vma, - RMAP_LEVEL_PTE); + return __folio_try_dup_anon_rmap(folio, page, nr_pages, dst_vma, + src_vma, RMAP_LEVEL_PTE); } static __always_inline int folio_try_dup_anon_rmap_pte(struct folio *folio, - struct page *page, struct vm_area_struct *src_vma) + struct page *page, struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma) { - return __folio_try_dup_anon_rmap(folio, page, 1, src_vma, + return __folio_try_dup_anon_rmap(folio, page, 1, dst_vma, src_vma, RMAP_LEVEL_PTE); } @@ -506,6 +512,7 @@ static __always_inline int folio_try_dup_anon_rmap_pte(struct folio *folio, * of a folio * @folio: The folio to duplicate the mapping of * @page: The first page to duplicate the mapping of + * @dst_vma: The destination vm area * @src_vma: The vm area from which the mapping is duplicated * * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) @@ -524,11 +531,12 @@ static __always_inline int folio_try_dup_anon_rmap_pte(struct folio *folio, * Returns 0 if duplicating the mapping succeeded. Returns -EBUSY otherwise. */ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, - struct page *page, struct vm_area_struct *src_vma) + struct page *page, struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma) { #ifdef CONFIG_TRANSPARENT_HUGEPAGE - return __folio_try_dup_anon_rmap(folio, page, HPAGE_PMD_NR, src_vma, - RMAP_LEVEL_PMD); + return __folio_try_dup_anon_rmap(folio, page, HPAGE_PMD_NR, dst_vma, + src_vma, RMAP_LEVEL_PMD); #else WARN_ON_ONCE(true); return -EBUSY; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 07d43ca6db1c6..8e8b07e8b12fe 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1782,7 +1782,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, src_folio = page_folio(src_page); folio_get(src_folio); - if (unlikely(folio_try_dup_anon_rmap_pmd(src_folio, src_page, src_vma))) { + if (unlikely(folio_try_dup_anon_rmap_pmd(src_folio, src_page, dst_vma, src_vma))) { /* Page maybe pinned: split and retry the fault on PTEs. */ folio_put(src_folio); pte_free(dst_mm, pgtable); diff --git a/mm/memory.c b/mm/memory.c index 1efc393e32b6d..73b783c7d7d51 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -866,7 +866,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, folio_get(folio); rss[mm_counter(folio)]++; /* Cannot fail as these pages cannot get pinned. */ - folio_try_dup_anon_rmap_pte(folio, page, src_vma); + folio_try_dup_anon_rmap_pte(folio, page, dst_vma, src_vma); /* * We do not preserve soft-dirty information, because so @@ -1020,14 +1020,14 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma folio_ref_add(folio, nr); if (folio_test_anon(folio)) { if (unlikely(folio_try_dup_anon_rmap_ptes(folio, page, - nr, src_vma))) { + nr, dst_vma, src_vma))) { folio_ref_sub(folio, nr); return -EAGAIN; } rss[MM_ANONPAGES] += nr; VM_WARN_ON_FOLIO(PageAnonExclusive(page), folio); } else { - folio_dup_file_rmap_ptes(folio, page, nr); + folio_dup_file_rmap_ptes(folio, page, nr, dst_vma); rss[mm_counter_file(folio)] += nr; } if (any_writable) @@ -1045,7 +1045,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma * guarantee the pinned page won't be randomly replaced in the * future. */ - if (unlikely(folio_try_dup_anon_rmap_pte(folio, page, src_vma))) { + if (unlikely(folio_try_dup_anon_rmap_pte(folio, page, dst_vma, src_vma))) { /* Page may be pinned, we have to copy. */ folio_put(folio); err = copy_present_page(dst_vma, src_vma, dst_pte, src_pte, @@ -1055,7 +1055,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma rss[MM_ANONPAGES]++; VM_WARN_ON_FOLIO(PageAnonExclusive(page), folio); } else { - folio_dup_file_rmap_pte(folio, page); + folio_dup_file_rmap_pte(folio, page, dst_vma); rss[mm_counter_file(folio)]++; } From patchwork Mon Mar 3 16:30:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CE45C282CD for ; Mon, 3 Mar 2025 16:30:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 08245280007; Mon, 3 Mar 2025 11:30:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F26CC280003; Mon, 3 Mar 2025 11:30:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7916280007; Mon, 3 Mar 2025 11:30:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B52EF280003 for ; Mon, 3 Mar 2025 11:30:51 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7151F160CBF for ; Mon, 3 Mar 2025 16:30:51 +0000 (UTC) X-FDA: 83180778702.09.10EAC0E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id BC3CB40021 for ; Mon, 3 Mar 2025 16:30:48 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KQjaIWRK; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf27.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019448; a=rsa-sha256; cv=none; b=HUwjSud3uP3wO3ZYgYDpnopiN64SOPrcyvWAwg6CdYIHD48e8sL9vALRw99wwl0iBDSAmT WzS5K0oIQcgn0TdOlp6x/evzrxc9uVcENkppVcNBr5fbl32czqugIaOpd929wSPHsvP8nu Yhv6V1hmaj6HhSW1HOHQ/heKMm167nI= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KQjaIWRK; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf27.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019448; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lWnUBbJbNal4OtQP6KylSC1HMhJkWgV+LTGJMK3tR3E=; b=OD2eTj7ylF/gR6794BgKnaYh5FHZ8p1wkh9v10GlEkBpZmkAX6db42SRpD2nUrBedgYdXD qHO+WWeUib7gC+NZ+UMGtzyY4qzMNOyHrOwAbTs68sJPfOhBGXAYp7vEBcQY6D4/v6v1Ja N8cDm4Pg6hgrp1jcN0ugBKnr9jX5i7U= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019447; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lWnUBbJbNal4OtQP6KylSC1HMhJkWgV+LTGJMK3tR3E=; b=KQjaIWRK8ARRkYCGNd/Y4b/zCiXRa3wg08ygCDTaTuAK7elwIutK4fqd2WqKo8QK0zKeQq ILin87UhuuKWU042zdkJvVgA9IzHYd33tPDrnUSbFK5vPsAnrJFexVy85WfV8oUkWNLbF4 JwRYzMOb911PxCeUx+y4o/nYjFR2L9w= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-602-Z3K23fSXNpmsphGr8MmEjw-1; Mon, 03 Mar 2025 11:30:36 -0500 X-MC-Unique: Z3K23fSXNpmsphGr8MmEjw-1 X-Mimecast-MFC-AGG-ID: Z3K23fSXNpmsphGr8MmEjw_1741019435 Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-438da39bb69so33337005e9.0 for ; Mon, 03 Mar 2025 08:30:36 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019435; x=1741624235; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lWnUBbJbNal4OtQP6KylSC1HMhJkWgV+LTGJMK3tR3E=; b=hwPDvd3zrjk3qEGuib+r7xKCCkK3Fpjm3vmIWrvocmIA3MTe8/ibcevFGB8EyA/lET Q8Fi90vmi6/7ANSGuMqnnvonDi5XbTpsrfu0H1jax4WfIJNb0k2MGF117WtL7a1F29+z AIR+h525oKrmcBoLS7gKXb/wA3A2AwX+2yaygqiY6OghLJWWhfg8/y6OMZlPVXqXQ/1L ZViV3EwJx+yRawxy25kxIAXn3ibjZd8wLHR0zSi5y2a+f4kaPx0RdErAtiTnJxd1uT+d Un6x9l6sql8DO1DDShCHZkCwc/PBf2HPnk1HIR2DuBSIkUCa00pTxdZaHa3F3/xRVO50 zZQQ== X-Forwarded-Encrypted: i=1; AJvYcCULxFJMdXPadDASarw7wObdhoJN8ClPnj2s6N5Vuikw7o5hwwpH7Xg6hok0fby0I9U5QmJ/yeKH/w==@kvack.org X-Gm-Message-State: AOJu0YwDMriWhe3glwFYvTmmyvL+0Ehx5N7E4Jb87n4XEJ7jQN0AFZLq /dfhlgkxsf2X0AukvqQwDuMqgEACHW8+yYn2lGrIshD7HUStCteHn3TRCfcKqyjYq9uyhGQy6Vw +WTXR12QIzOkJa/JwY3s4/W6WDD8l6TBtyVIrnqS6wwyiZRjD X-Gm-Gg: ASbGnctgboo3irwHA2MYWoLT4iw7EoAnDN3PNjYVH6e5cz6WxmUYwibStQbAqcsY3Ga 9A/rySnxcU7s68ydaUXXLkijWWV+E6tmHA5Juifysc46azab5hnwsk/6UD2g2MpAFdu8dU/h++l 9UrbUP84uVRTNkGpu+T1QqEnF5FZsSnlm4LzX1uu1Y15sylmgiHUL0R0QRGSGhNhkrwMvrROl42 gk345If4tZ5DEsYWYf7fMC9HZNmHjLKbhwcL1DOz3DEYz7TjsIy1MUWmHYuOyu7nlHygffEnzMg SjU1yDjfDu/AqHlbhl/cK5oM9BOlu0+5FI60sSFyhwmEeDjx01BPRa7RLe2P0WNg0fh67Q0h1kE K X-Received: by 2002:a05:600c:19c7:b0:439:6b57:c68 with SMTP id 5b1f17b1804b1-43ba6710a51mr128366305e9.17.1741019435400; Mon, 03 Mar 2025 08:30:35 -0800 (PST) X-Google-Smtp-Source: AGHT+IEHH2uqcGG6+HqLvILt026VDKtCx0hMTXydneDmvGOTb2RHDZET1RLo/ABtW5TIvoZEd6mWJw== X-Received: by 2002:a05:600c:19c7:b0:439:6b57:c68 with SMTP id 5b1f17b1804b1-43ba6710a51mr128365725e9.17.1741019434989; Mon, 03 Mar 2025 08:30:34 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43bc63877desm18077035e9.1.2025.03.03.08.30.33 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:33 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 08/20] mm/rmap: pass vma to __folio_add_rmap() Date: Mon, 3 Mar 2025 17:30:01 +0100 Message-ID: <20250303163014.1128035-9-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 1VEBggcC4ZsCxm9pIdpHMHQEqkST9o10MJNuf38HlKM_1741019435 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: BC3CB40021 X-Stat-Signature: aqirnru5rjrdp3hqwrrf6wqabgc5agkd X-Rspam-User: X-HE-Tag: 1741019448-320870 X-HE-Meta: U2FsdGVkX19ovnSR3hGEiFqXGcCxxeB5943d+3Y/1rvpeTrA+JbI73EZhtS5deyZbPerkGgHpbDrrw1TMFaKUf+Tk5F1bGW1N+6zU54M1igFaaZXoIXBvjJEeovsXgF14J6vwu/bkgOGkhmztxeDjKNQI4tDVS1xCMF77blddYjIVLSSh75DKmDXxA7kuXMSZqNth22NfwNZ8KkrFqNySThYKMpOSuyN4UB4cJdznFU8uJqsfFkiPQO+8/2IZpWQjqDN/4sPvaH9bz8/txefYLLIzOZ0g9/7y/bIy5ScFPuEcM5/0CBis+z/fKirrPibnJtkJvuVJoI/yhhN2tK2QoysMV0Ck7T/8/Ythall8Spx2C9tErLtXJO2ufcshbFdQO5cQxwLblZwj/u9/93UQuCk/5fC9gr86x2kPUCPbaLJQINeWe414Ulh0LtOXHWt1fgAxHBNTnT7vCqfgWEGz72LzUmojYSDzZ0+5/Nv/+tyTtZ86FgwOxQhbBIkTaK/S956rsO7d3ud0XfmAvWeZ7z2mYCcBwhX2QfwmlgeKJsy9UEl1SjqlYjX2Uecd/IJCopUF5+fWkdmp5u/5ZjUdN22vbdTXV3XMx+F9E7ic0NMA9+3talsvYuJUia3ubMzhABsrZ6XEqxp7aHdqaphKK6k6S8O5ttiuydJd7d3pBXjSzbHp3uLdGo4o7j6DMVKZfsAbTqKQEm93BkH16FReg1qkbsQSKQfWa2IBe7j+xh4y5FKM4S+jTH405hEVOT6wZuqJU+uBdQETJBXtSPmOxX72lkp2T3m1RcP92LAJBbOzLEpWKesfGcftNFE0PJ6lKqhxebJ+Gf8z1B8Qkow0pbxs8vBafjOMf/JzFxsfyevODQR9XLmh2M9XQf7D7QXFsbJChYROoZ+Ug98F0t5tpy2GJYFaqUq2Tv5bdxwsLjLFJ9dqrl4fTgpJGH429ZKsOWoKxHHcW/aqZ7eTmO DXtzgfiI sN70fq8TYD4i0wD6xtrpY0v7KLZSkjeujwdKN8KJ5KQVH+tOJf5nzKe+s7lQtGMl4QV6Vd2IPaFL/CDMeZBrIL3YTrcXWln1YKU0T8pH3G33NA3tAvamBzVLRlQZBwupNdL4MP2lHVXpbKLOzfROGnjuVrufWkgw02ALSpKkHMAeceQb2J+PbxqPFOlG6hhDBqakueReWFZNyaszfrZHrAm5T4AE8nKg6/e69ELoktMXf31EzmSq1R1DM3kf9Tml9TymQhmWfyYBNEKPJBXVwqM0MEunifHY9m86vTl7roXCAyMLyOlybDIzyyLZ+X3X/YZ7Xkt5jqRo108cWxFzDRdo00py1rgc8RmMrISZFoj+59hxtL/65zreqerK2br5VJgtWdpv6WSlMHiDFjUBOMCkhMz8iEUnOo7dddeIHiJmqTFSZ5nKE87cX7J+xr4FQGbrti5dLMu34Xjou5vrVsT3NF2AgBX/mCRbBeLCtCqNdjgCOtqEzZteryuCuZzjqNSWLaHmEOg3VxytCwDE/Nftm4Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We'll need access to the destination MM when modifying the mapcount large folios next. So pass in the VMA. Signed-off-by: David Hildenbrand --- mm/rmap.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index bcec8677f68df..8a7d023b02e0c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1242,8 +1242,8 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, } static __always_inline unsigned int __folio_add_rmap(struct folio *folio, - struct page *page, int nr_pages, enum rmap_level level, - int *nr_pmdmapped) + struct page *page, int nr_pages, struct vm_area_struct *vma, + enum rmap_level level, int *nr_pmdmapped) { atomic_t *mapped = &folio->_nr_pages_mapped; const int orig_nr_pages = nr_pages; @@ -1411,7 +1411,7 @@ static __always_inline void __folio_add_anon_rmap(struct folio *folio, VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); - nr = __folio_add_rmap(folio, page, nr_pages, level, &nr_pmdmapped); + nr = __folio_add_rmap(folio, page, nr_pages, vma, level, &nr_pmdmapped); if (likely(!folio_test_ksm(folio))) __page_check_anon_rmap(folio, page, vma, address); @@ -1582,7 +1582,7 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio, VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); - nr = __folio_add_rmap(folio, page, nr_pages, level, &nr_pmdmapped); + nr = __folio_add_rmap(folio, page, nr_pages, vma, level, &nr_pmdmapped); __folio_mod_stat(folio, nr, nr_pmdmapped); /* See comments in folio_add_anon_rmap_*() */ From patchwork Mon Mar 3 16:30:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 743B1C282CD for ; Mon, 3 Mar 2025 16:31:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B5A14280012; Mon, 3 Mar 2025 11:31:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AE56D28000C; Mon, 3 Mar 2025 11:31:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93361280012; Mon, 3 Mar 2025 11:31:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1BDD928000C for ; Mon, 3 Mar 2025 11:31:04 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4970D140C81 for ; Mon, 3 Mar 2025 16:31:04 +0000 (UTC) X-FDA: 83180779248.19.ED5C0CD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id C79E040024 for ; Mon, 3 Mar 2025 16:30:56 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gnrzZ0oI; spf=pass (imf04.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019456; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Cb8Ah8lJoOkHJddlEV7mqidKdKB3PYdRohUjXTfR2AA=; b=LJ0ixxz+hHf7n0FREmiLplNLVMfeh7W+ZMzBoW05AxmwoLEGbhNgUh1jAbliQerc+lEkQS x7PwjMaE6GHebdGYnw8mMEKfY84U2vVJSXv1hpMwzps3OeynLrqu5abQgGE68+YRqLzpGA xOjN7gyscJGY83bKSBBoARzXZ5U9JVo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019456; a=rsa-sha256; cv=none; b=vWKxu1SYhtjG3siG2TPz/Uo8v2Pq+fW8CiGxeeYCdVVSi8EYKe6q8vetk6A1PbTBfH9hux mywV5Devt5vqADULV0eQ3TqlQpHZQqOjQafQ4ZF6f5kvAg15uvp5melNTBGhB7FQtUZKGG 1gJxYF/QAoBGnyshs58b6xA1sJV09Mc= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gnrzZ0oI; spf=pass (imf04.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019455; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cb8Ah8lJoOkHJddlEV7mqidKdKB3PYdRohUjXTfR2AA=; b=gnrzZ0oIN+OGKFm6I/4/6bEbFWNgiOpIwwO4+BWi7m3AClYeVkHyzBYUkNV3FOlfDrrtUT 5lJawTtEP1fb22m0XgDlU8d25Q1uqYsTYKLalfe9hOURu4bTjlqkV1d8vVpdFwMlcExw0n LnVtBwfXp094zffeEBH683UQ+BvunVE= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-511-i86xujf7Nk-aPcT3bNyBNw-1; Mon, 03 Mar 2025 11:30:39 -0500 X-MC-Unique: i86xujf7Nk-aPcT3bNyBNw-1 X-Mimecast-MFC-AGG-ID: i86xujf7Nk-aPcT3bNyBNw_1741019438 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-438e180821aso24935245e9.1 for ; Mon, 03 Mar 2025 08:30:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019438; x=1741624238; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Cb8Ah8lJoOkHJddlEV7mqidKdKB3PYdRohUjXTfR2AA=; b=JWsFHcnpIsQpPOpPCFiKo3BZUleVIsnGC5uenqV/EWWXR9mOFjkbWlKiPrRWxGquFo Zpy47Dyrv8osBKgTp7WDNJZlC5w5Xn3JObi9fde6DRlkkIIcBYBctS1pQu3wQXW2WvPZ 4J7DFPze6dAXub9rM/R8qwurA/G8ynS+vR+r/IqUfwzIY4wQNXqeJjUqVub1PI/8w5vy ONYOin1Vl+eW8ao+xBCxVDoO4K8G0rIuCCFFASoTim+PUuQK8CZ1Z8Hhz09Obncwx3Id X2ZudnvNeNXWrdFr5gJSBxhsnkPW1muyCLZ4srxHNDB1vs1dXt/I5Sdpibeer9FV1z0k i+2A== X-Forwarded-Encrypted: i=1; AJvYcCWZiuZfNBSRG1NGo/wprcSU9n10LqCAyMsSzKxMqkVfoaSoIJn45nx6PYY6N/4p1cjJVnap/L9eGA==@kvack.org X-Gm-Message-State: AOJu0YzcmEtzyuwW0qHBidBCEBgUWbFyZJwmIsjZRn9jRe+6OJUiN/gP gpZzXNtFbdJ/rcmoNnwLvl2+D9MDdvytE4rJhQZ1hl3yqohEoCYlIyy6NFfj3Y7XUo3FUPujVKR jv6hZ3l2mFC4fjU+eDs3DnJP6cvgMGZfcUpfvtN1pMa86Nysm X-Gm-Gg: ASbGncvlVID5eZtDzNM3/FQUHUPGK/70Rph0bbhhTwoYEZo03oziyJ7Cae9gATrlxH6 +6SUF5A+9bdejkItkM2aEOdKEzpx2RKoAnF0lcAo9btdoOyHBB/KWz6istQ7/hc0Jgl3e9Vx5qH JJiEa0pUMCR+nxGRZg/YyohjygTT9lzod2FxikbiAOOarZWtHh0D7/9rZ67+ljYOzZACSvTEVFK IBu6u3E/vpEYWmTF2v70JLEfA6GF8JfMGJLgRTKpuhSnddM+6tzwZxdmyBdai80WHcgIMyab7FU ROoq5uvL6CdVBJOFA/h23fSWbEi95r2zIMTXQm7CA/VMK2fUH7+Wkb+ZOrFqtxiIJiFT3pGDSKS F X-Received: by 2002:a05:600c:4f0c:b0:439:91dd:cf9c with SMTP id 5b1f17b1804b1-43ba66e1f3amr118907985e9.10.1741019438198; Mon, 03 Mar 2025 08:30:38 -0800 (PST) X-Google-Smtp-Source: AGHT+IFdpz0gqig8RUNxdVw3Ebkq3f5eeFacb1rX/1/2my2Nw7/Vx42fOquiEWgwrBfEGJjD+4uhoA== X-Received: by 2002:a05:600c:4f0c:b0:439:91dd:cf9c with SMTP id 5b1f17b1804b1-43ba66e1f3amr118907545e9.10.1741019437826; Mon, 03 Mar 2025 08:30:37 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43bc06eba55sm45297035e9.21.2025.03.03.08.30.35 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:37 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 09/20] mm/rmap: abstract large mapcount operations for large folios (!hugetlb) Date: Mon, 3 Mar 2025 17:30:02 +0100 Message-ID: <20250303163014.1128035-10-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: nPWVAo68XpqXPWC1pJ_sFFeyJwGbaaDRncBKzBH91AY_1741019438 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Stat-Signature: pkocr831zi7oubongyymajjxag6f38pi X-Rspamd-Queue-Id: C79E040024 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1741019456-945916 X-HE-Meta: U2FsdGVkX19EdmvtiuMG+XA6VQiqNU83gBhLRtN8TUBsKufEVE74uIPcKwaXyjBt9Z4lmE1VBC2MFraV2mHgVC6II57CxYmegpYzMw6NrwmMVUG/a+3vEVTW4vtwkJBsRNA6CAF/owkVYRZbXaCZQea1lv1Icc5ZYtmcEIUb64n3uR4wrc6ZanWJKC7lQgWmtwx3O9gKc0naezp5aEt3pyUzSCZU2uqEXSJuLZ1a/yB2qNd1VoRinMtC+p3gaIbfJTEyTWfhPr4665/7ceEH7+m7aNPG2LqHvRXLy3VMm/KgZLukYHq8Y9533UZv16Q6tTkM9wuWxv4Q78g612P89spo/DFv8VX1NQyS86YmJFvlPJ1SYMmFEEhOf9r5YE817RIO3KArjgRaGdIDac1clh1O8Qb2slwf56w6qpcViEniw/OzEzJMguRK5d7ePGI/WnvQHzo0Js+8z7W6IouzHIl7wT1YZdj5dLp/SXCVKTEEHoHOlKx1mF/uiImE8w9YS7l8+1Ga/hKnKL5VFwuiPXFcXPTrGLbiXIhC3zS+COWZ9cEdaxgGfL95yfTC5wslZMnyA3PKDmrUNBA21u7l9qWb8Br98igs/F+g5SAmbRa4TBCC5KJMhWNhuVjyR7jlKWEasw9TlqImCcXVAG0VTnaVLPDvAtqHnUNw9+OEc+2PDb2B6EHiRsTrAJlgJS+biXC8mjMB9BUlQ3b97qznykb/DtN7w0ZAMokeE/9pV5ytt029XUKkQ6eaQd17e8zJZKNarHxwgu4wCL6SWKpfsThruSFEc3t3IxyMsQnjgVX0oEjp1+bUT0Fru5hHOX04jAn5Ea8yC8oIuGgaDHlGdsJErFeoEnRveoMBfxk5L2AJJ72+/6hyREzfQ0EkDIPhU3VKWpNzGs/bXTU+B5exbOQ+cAKWRqSa35GFnER29sNj6gZvyKPMMfySbPlWvGviO8yrNL3xkpDL/dQn7YX s8VG8sAY WzBfhaHCmWzx6Cl/MBrP1uSyZqSqhMg7DhUQCovf0LtUYXeIzyfbZRmDUwtmhK0mrE2I2/qsIqxAuWY8dWGNxfHYj/fTQt419W9B9DWitTfcSnBXlMWzScunZkBmCde27NgjSiRIXo/n9Qvjl6c7Mx6EKFma9QZud4s0+YZ9Ct2xuoyN5BWRPaxoh8joDmjjbqD07kJevkufQOQPIC8ruruPgphT8AxiArbAsxOUXsAnmO8GNeF5dfpZsICFaWTz0h+/NCqPoG+gZeStWlrpbxoPmaLExVhvEiDAxL54KuvM1Apro12+UEcYp5op3nXmBGlzdHXqo3ARXe1/hgoV41wl/vjAtgHTsl6PsRRrSjEzFtkLnKiokjF+C/S7fYSjxww8AmyGobSgxkwtecmHYPnBvSghD6qrWdInLi+CRjibdgOpBir5q9ZaFYjNcZn2shgIGfmFGY1mmhvCSBuP5+rdbUHTMqSSf66Qrpza2nV2voE5Zrv7n5+d8vvcOo+NV+5VUSdD63/6Hfm4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's abstract the operations so we can extend these operations easily. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 32 ++++++++++++++++++++++++++++---- mm/rmap.c | 14 ++++++-------- 2 files changed, 34 insertions(+), 12 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index e795610bade80..d1e888cc97a58 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -173,6 +173,30 @@ static inline void anon_vma_merge(struct vm_area_struct *vma, struct anon_vma *folio_get_anon_vma(const struct folio *folio); +static inline void folio_set_large_mapcount(struct folio *folio, int mapcount, + struct vm_area_struct *vma) +{ + /* Note: mapcounts start at -1. */ + atomic_set(&folio->_large_mapcount, mapcount - 1); +} + +static inline void folio_add_large_mapcount(struct folio *folio, + int diff, struct vm_area_struct *vma) +{ + atomic_add(diff, &folio->_large_mapcount); +} + +static inline void folio_sub_large_mapcount(struct folio *folio, + int diff, struct vm_area_struct *vma) +{ + atomic_sub(diff, &folio->_large_mapcount); +} + +#define folio_inc_large_mapcount(folio, vma) \ + folio_add_large_mapcount(folio, 1, vma) +#define folio_dec_large_mapcount(folio, vma) \ + folio_sub_large_mapcount(folio, 1, vma) + /* RMAP flags, currently only relevant for some anon rmap operations. */ typedef int __bitwise rmap_t; @@ -352,12 +376,12 @@ static __always_inline void __folio_dup_file_rmap(struct folio *folio, do { atomic_inc(&page->_mapcount); } while (page++, --nr_pages > 0); - atomic_add(orig_nr_pages, &folio->_large_mapcount); + folio_add_large_mapcount(folio, orig_nr_pages, dst_vma); break; case RMAP_LEVEL_PMD: case RMAP_LEVEL_PUD: atomic_inc(&folio->_entire_mapcount); - atomic_inc(&folio->_large_mapcount); + folio_inc_large_mapcount(folio, dst_vma); break; } } @@ -451,7 +475,7 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, ClearPageAnonExclusive(page); atomic_inc(&page->_mapcount); } while (page++, --nr_pages > 0); - atomic_add(orig_nr_pages, &folio->_large_mapcount); + folio_add_large_mapcount(folio, orig_nr_pages, dst_vma); break; case RMAP_LEVEL_PMD: case RMAP_LEVEL_PUD: @@ -461,7 +485,7 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, ClearPageAnonExclusive(page); } atomic_inc(&folio->_entire_mapcount); - atomic_inc(&folio->_large_mapcount); + folio_inc_large_mapcount(folio, dst_vma); break; } return 0; diff --git a/mm/rmap.c b/mm/rmap.c index 8a7d023b02e0c..08846b7eced60 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1266,7 +1266,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, atomic_add_return_relaxed(first, mapped) < ENTIRELY_MAPPED) nr = first; - atomic_add(orig_nr_pages, &folio->_large_mapcount); + folio_add_large_mapcount(folio, orig_nr_pages, vma); break; case RMAP_LEVEL_PMD: case RMAP_LEVEL_PUD: @@ -1290,7 +1290,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, nr = 0; } } - atomic_inc(&folio->_large_mapcount); + folio_inc_large_mapcount(folio, vma); break; } return nr; @@ -1556,14 +1556,12 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, SetPageAnonExclusive(page); } - /* increment count (starts at -1) */ - atomic_set(&folio->_large_mapcount, nr - 1); + folio_set_large_mapcount(folio, nr, vma); atomic_set(&folio->_nr_pages_mapped, nr); } else { /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); - /* increment count (starts at -1) */ - atomic_set(&folio->_large_mapcount, 0); + folio_set_large_mapcount(folio, 1, vma); atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED); if (exclusive) SetPageAnonExclusive(&folio->page); @@ -1665,7 +1663,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, break; } - atomic_sub(nr_pages, &folio->_large_mapcount); + folio_sub_large_mapcount(folio, nr_pages, vma); do { last += atomic_add_negative(-1, &page->_mapcount); } while (page++, --nr_pages > 0); @@ -1678,7 +1676,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, break; case RMAP_LEVEL_PMD: case RMAP_LEVEL_PUD: - atomic_dec(&folio->_large_mapcount); + folio_dec_large_mapcount(folio, vma); last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { nr = atomic_sub_return_relaxed(ENTIRELY_MAPPED, mapped); From patchwork Mon Mar 3 16:30:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB837C282CD for ; Mon, 3 Mar 2025 16:31:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C489628000A; Mon, 3 Mar 2025 11:30:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BD8D428000B; Mon, 3 Mar 2025 11:30:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1E7428000A; Mon, 3 Mar 2025 11:30:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 84A59280003 for ; Mon, 3 Mar 2025 11:30:59 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2A0B5A4960 for ; Mon, 3 Mar 2025 16:30:59 +0000 (UTC) X-FDA: 83180779038.26.C4AEFDD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 652CC12001C for ; Mon, 3 Mar 2025 16:30:56 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XSZM7zAo; spf=pass (imf29.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019456; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VusPlBbKEOGpL4wS1BqoWY6CYb2f4HQ+/wIDh8p+vAo=; b=yMkYgzxTzm4+1WN9ojWZ0wUvOYB1bL10bna7QN6dF3YlYMfaMZXe6DRw6uh11ihhF7+ya7 OtogA1Fq5chE4bQqJhlMf2c5870dLJ0OBzvoOeL5wgIFCkrk+vIyusqKMm+p5ALQUAvXHU nyVU0N9xYTeQiNEHqnWb0q8zZZsxxrU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019456; a=rsa-sha256; cv=none; b=CIZ1lVgh+9lI3CDPSVdOhdBgUz1OtPWEEwviBTw6n7Qe+1L5kAGPtJGR3GLgxcfeUUHuwI 0RaT7R5pXs5eyMsjyBrLVSa0BAG6SGwPCnjyV3oHhKpyLGqXW3Sub2zpdLM98JC2D1VXFC 0Z3CyeaBLzsCqWSMfsb7XHWzIWK/rog= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XSZM7zAo; spf=pass (imf29.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019455; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VusPlBbKEOGpL4wS1BqoWY6CYb2f4HQ+/wIDh8p+vAo=; b=XSZM7zAojUVklKQobu24Roi8FXCvTmnmvssa7yGdJ4e0Vq8BnVpMqM0Ufe4yPGo93qX3oX huwYfbzZq9VycY1yFq/5oVKQjfJFTP/8YUUX8VRU220c8FeQgEG9B+kiVFk67n/1DErPzS tBj9rJjmFRiRgexl0FBfI84eqUvjC7Q= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-642-JXO8oKebOxGKcpxyjxJJww-1; Mon, 03 Mar 2025 11:30:43 -0500 X-MC-Unique: JXO8oKebOxGKcpxyjxJJww-1 X-Mimecast-MFC-AGG-ID: JXO8oKebOxGKcpxyjxJJww_1741019441 Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-4394040fea1so22860935e9.0 for ; Mon, 03 Mar 2025 08:30:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019441; x=1741624241; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VusPlBbKEOGpL4wS1BqoWY6CYb2f4HQ+/wIDh8p+vAo=; b=AxxItoQf4O2bth1G79Le1EPivqRXbrIkvUqSLxzZ1Gf7F0LvcOrL7/jCDWkvNemWrz bNCgn8bHQA2s0vYU8NIzLH02P6ZENdRREAH8O98g13OIC6BOE+2iHtCXZFUDE/NAODMe fpYzoKnMOd3x1Nhaqce3VsgP/OvSSl0Ds7kOlKoHvL/BKcKWM/MBSbSz9AEIABXXnGgK yD9H1OG4l2HR3N3N9ct4pj1H67TAV6q3SY1N3tzVBFmJpasVH+vPMaadpfCRWVxM0Jc7 3tBFaGsETqqRu0uvKmXJNovTJyV37Tbr2eLWEFvv9qxZ7cR70ZMJA6QpULfKgSVcQ0UW HO+Q== X-Forwarded-Encrypted: i=1; AJvYcCV7D1k/LAheymbLmcKesVkHY7c5FRJd41Sgx78LWuRS1CyXo2Dkcq+nH1TOwvZefpb3xq8OOokYgQ==@kvack.org X-Gm-Message-State: AOJu0YyMBcF2RmDpyt5MDOBTpoN9japlzRw1mal+IbHRbfmheHs4Gqvm wTtKfvRk/6GzftbYnUAgcgDncvG19XJgu24UCxNtNGRPAA0sIXvmSSxKqqvpw2xdcn3CfCC8ggf DVrdaQGOerA+V1gZN8aL0QaMjRtYmAvcEVTFg3Q/6uvw83tR6 X-Gm-Gg: ASbGnctpdVBRmILQQh8qHWDjzw7PJgIEJyJLjgfsEfDY/lRmAOWSEqTWHO/v3or8VVX AkG4IUHp4aWZSUpR3Ip/8xUc7DTmqi/9eBli+mLe/tlumBvtM4CK9ECYygbFueYoJFcVjdptNw7 +G20YGeMtuq13ADLxqxZhNkI8Z99N5CmJ4ogJq+pY2mXM7hgG1SrEaaQUSgWAt3zx1ZsiRANsMi ceiwezXJmf7whxzkta5qUDG+zLsm27dvmdUFu0CRyYbg5GeNnw9K/Y+YKSvm9myekaG3eXmdtmi Xn1ti0JC0UBnsp2+4gZtNW3fQXD4gJDimySyN5KtDOvXPcONorXQTJlco+Y4wOjKZoqc27OYLJK y X-Received: by 2002:a05:600c:26c6:b0:43b:bb72:1dce with SMTP id 5b1f17b1804b1-43bbb721f8fmr44417465e9.5.1741019441213; Mon, 03 Mar 2025 08:30:41 -0800 (PST) X-Google-Smtp-Source: AGHT+IF6XV9dtRCnC9UYnKtk79gh146334CB5ZefECKJp7wi6xxTybGeqstnFZTU3vfWQysm8+TPqw== X-Received: by 2002:a05:600c:26c6:b0:43b:bb72:1dce with SMTP id 5b1f17b1804b1-43bbb721f8fmr44417035e9.5.1741019440886; Mon, 03 Mar 2025 08:30:40 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43bc1b5db02sm42954325e9.19.2025.03.03.08.30.38 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:40 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 10/20] bit_spinlock: __always_inline (un)lock functions Date: Mon, 3 Mar 2025 17:30:03 +0100 Message-ID: <20250303163014.1128035-11-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: WfXq_G-NjpfCPZUoj38iuCHt0Sr_fLRiLdFxruyTCyc_1741019441 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Stat-Signature: wozgmhac4reu8p857d37wbsp5ws6cgbw X-Rspamd-Queue-Id: 652CC12001C X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1741019456-965292 X-HE-Meta: U2FsdGVkX19M+jfzkQbWei+BrWxJ6xPWzq6ManR67ngldllUtGO+yclnpvOyjg2urmgzWq/ERCjayoZJC8HyPFIaWOXSYsw7E37D9g+bb17k8iTGXHDEzSzgzOCnvpWOctoZI1hI7CtvlgG9xbYC2ZoWdnpy7l2gavq7LkaqOaPhbbkN1jnLOXcb8ZDuOJGnDam5B89rVzeETcAgVaUPAVDpDIh0tMHUfoYf/zfiyvKVhDYpjj4r/7ZeUlJ7etksjfEQ0bKiRt/0wtfxMAhSaqWi8+X413TY3jhnfvpMkozjn+52+wWcUw3b82RIh3f2QnnGUYOHzzcKiPfu/fifZSZkvthfpZ1tifiDkVRaSnvoCOi39nel+wcjma7FS+W25CWvlWrJ53EAxvv3gv7QGOm+M4UDYDDooLTurkgMIeiUvSLIkNinYXskp5Y4VbgGPQHWkBsbj4zVGRAZg41kvSn6mshPbXVIzllRUadt9ZEY08cGflWBs4x2LlpIMe8QL/TVgfJI6XRC96zIeWoh9TPFvP0HzEmJJdagG/tGwaeEEuYdw1cMcA3wsZ+EqhHFkcBbiPtL4bFAAMo/JvoQdKxz4izNgzqArIbHVvZAIMcQBsApJOfsZIbPfO02GrYYrOzpvqUwg+h89r1paqFk1jWm2XMsXei2G99II8fH427BCRw+LLiZIPo7AC692c7ml+jTIc4FbW99CW6h6bCSsfQ8rJaUBbOBs+ZnU/BX9b0jO6Ykmsu2cxeFhoRoBe2R48/7HdTtOP9AtEDg2gTsKn2Xi2xXCnzqW8NIcg1tJQjjT39eedeB63rMbHUuT+f3oQMgi+84ukzC26Yvs290rOPlQ6F6OW2nAmRZAui/FbYHuPbyMymo6tHDoM0pBnKpn3foAXNETvkmakQlrSze9QDBH0gEKCuY1vGZckoGdQKjekmKoR9NyamgBapVWMpZJJ3e0Lj2tbpHf8rPRn9 df2OELFe 2moKll14EvbK9jivXOz22933uBFVsuMDLOjokBdoB6Cj5gD85CbIOJuyxBgJUePTimNIKeCG8Wu9tXItOSi9HE2TqSO7yQFFR76bjywiJgdbVwqLGQaYtdcLsg87oy1z1M0OGl0Fm0ceE4lAI6vpQ7XSCtdeFmNEpsTbqi3U6O+0U6LW5suPYFm1JTsi1+GVf+H8/kCvzxBMnaFicd1IWghyodHauc8FdBuMr2qvOTi6sTl3VPGPw+2IgNjiEQvI+djw8P7vw9BPVhGt0v7U/XEKkMgYUs7Xf321S0CYp94kFeCvok01Tjf7Wk6lMIbncnnT7mN8VXZhBxRbDNUfHQi+gNWWQ9UKsQBgDuYoJi+3T1kJ1JQGgW+8ZVhdMa8Gw//hqQZ23x9vn9ETIG9pmxEaZLUQvnVuO3QgnPCKNIykORi44s4+Sl0uzWdtV8A+T+/+4cyqjw/MQQjJM7I7/L+piEtEYENOGfOPguExRCibyorXFTs0rBE/nQeIru6ySijBPNXbXyJpLONQYGqEbL35mxg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The compiler might decide that it is a smart idea to not inline bit_spin_lock(), primarily when a couple of functions in the same file end up calling it. Especially when used in RMAP map/unmap code next, the compiler sometimes decides to not inline, which is then observable in some micro-benchmarks. Let's simply flag all lock/unlock functions as __always_inline; arch_test_and_set_bit_lock() and friends are already tagged like that (but not test_and_set_bit_lock() for some reason). If ever a problem, we could split it into a fast and a slow path, and only force the fast path to be inlined. But there is nothing particularly "big" here. Signed-off-by: David Hildenbrand --- include/linux/bit_spinlock.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/bit_spinlock.h b/include/linux/bit_spinlock.h index bbc4730a6505c..c0989b5b0407f 100644 --- a/include/linux/bit_spinlock.h +++ b/include/linux/bit_spinlock.h @@ -13,7 +13,7 @@ * Don't use this unless you really need to: spin_lock() and spin_unlock() * are significantly faster. */ -static inline void bit_spin_lock(int bitnum, unsigned long *addr) +static __always_inline void bit_spin_lock(int bitnum, unsigned long *addr) { /* * Assuming the lock is uncontended, this never enters @@ -38,7 +38,7 @@ static inline void bit_spin_lock(int bitnum, unsigned long *addr) /* * Return true if it was acquired */ -static inline int bit_spin_trylock(int bitnum, unsigned long *addr) +static __always_inline int bit_spin_trylock(int bitnum, unsigned long *addr) { preempt_disable(); #if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK) @@ -54,7 +54,7 @@ static inline int bit_spin_trylock(int bitnum, unsigned long *addr) /* * bit-based spin_unlock() */ -static inline void bit_spin_unlock(int bitnum, unsigned long *addr) +static __always_inline void bit_spin_unlock(int bitnum, unsigned long *addr) { #ifdef CONFIG_DEBUG_SPINLOCK BUG_ON(!test_bit(bitnum, addr)); @@ -71,7 +71,7 @@ static inline void bit_spin_unlock(int bitnum, unsigned long *addr) * non-atomic version, which can be used eg. if the bit lock itself is * protecting the rest of the flags in the word. */ -static inline void __bit_spin_unlock(int bitnum, unsigned long *addr) +static __always_inline void __bit_spin_unlock(int bitnum, unsigned long *addr) { #ifdef CONFIG_DEBUG_SPINLOCK BUG_ON(!test_bit(bitnum, addr)); From patchwork Mon Mar 3 16:30:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EE30C282CD for ; Mon, 3 Mar 2025 16:31:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E174B280003; Mon, 3 Mar 2025 11:31:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DA03E28000C; Mon, 3 Mar 2025 11:31:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF215280003; Mon, 3 Mar 2025 11:31:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6236728000C for ; Mon, 3 Mar 2025 11:31:02 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8AC121C887E for ; Mon, 3 Mar 2025 16:31:01 +0000 (UTC) X-FDA: 83180779122.12.42194DC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 8BC2F140032 for ; Mon, 3 Mar 2025 16:30:58 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DgQt16XG; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf09.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019458; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TqnqolcXMjdbgOoHxDkquZtgq0jreZJuwmbRkc4obqU=; b=REJhy+HSgJvQFbhc7f2OXPRlbewNCMjr1gmgkKDA0u532LazDY8mOpDoQUwrH04xmQ9aJy WO56oPuew3opFf/fB75+014t1DZdPbjUqNASXK0MHBqSJMBy68FKbevis5K+kyQhd00+R5 b38QDnVjVl5HcUZvKiiVgNPc0ZqF6oo= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DgQt16XG; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf09.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019458; a=rsa-sha256; cv=none; b=AaXO7ddPX0Wje8ht0tf+vGhXJh1FJOhhjpXn4WOgftTdR1BwOqzoHwUO4csJ1VOw1jb/zn DSDYdl0WBUY3GunFAVNIEZAQLl6ejcXEyU7q8VG8StofJKmx7qe5i4Whn3VZ5VLADK948s xsztWTa6DKWseIUYQH79H+ZFS+d25WM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019457; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TqnqolcXMjdbgOoHxDkquZtgq0jreZJuwmbRkc4obqU=; b=DgQt16XG5pb48JKEAu8P382hlc+KvTjNcyr0nkKj5Xd4CiM8I8eggfqxzED/k1UE8c7rQH JfMkbiIIVSLEgQMsyCgtkP+aeEq1uI/vquuOizhnt+6qJ40gCGdTnXWun9Hm5MMdlY49Lm VPwG/OJeo9miSodOqMKju7MDHKGvX2c= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-567-tlwA82M3OJS0M_YqWLVHFA-1; Mon, 03 Mar 2025 11:30:44 -0500 X-MC-Unique: tlwA82M3OJS0M_YqWLVHFA-1 X-Mimecast-MFC-AGG-ID: tlwA82M3OJS0M_YqWLVHFA_1741019444 Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-390ddebcbd1so2897381f8f.2 for ; Mon, 03 Mar 2025 08:30:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019443; x=1741624243; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TqnqolcXMjdbgOoHxDkquZtgq0jreZJuwmbRkc4obqU=; b=H0/iGJAm7dBeP3+YhQ8+EZRfmkNdn1SPwVF39vbACJ96wL55PI60XlryNftOWp0rMx e/utJ3UvViqJIUqu+h2TTiHkvL6zAInaLcggssmzChOaveoLSMsUsNYgysQNOnFt1lJC 5jPYsoqdr57EJ6J/lPf1s9bkf4ztNNnspIYbuyUVR0a5BFnCGYbzU4uM6F0EOrGFDUwx HSEE4sVtZUqOx5PCSCHyuq5OjHVA+u64cGfqTrGD5wreWgn5bEogRXUmslHq9rlvl0q2 pWhN4LXBybI1xpPEK/9fUQEDc6pJBHaAmq+FgMzUubm2pbvo8it5u/FNR0YMCgJV88RN fWHg== X-Forwarded-Encrypted: i=1; AJvYcCVQkf1Jt4e0BC1hTktxWax2gvdA8vqbI9KXGvY1wIhhWFA85zzmPnFJvxuj1cPvLq6XCOE6iy3PCQ==@kvack.org X-Gm-Message-State: AOJu0YwF5YpKx5NM5HKJTCsMPKbQ7iPSBVPqUbg/oOOX2EJ/3q7058rh OLxFEZJJHoZQ5QfmNFHM8oTqNX521smYe7+nH5mQsfkJjntEvODw3yeXuA9GN+oicrnFuRSFEjA epCadIj8S+8cI2jeP72FcLi8mQNYfJOn+5Qa5qcuSFXjAC3D8 X-Gm-Gg: ASbGnctlGMkSd72dTvlKsfh5yAZ2l8xNCmBQtnuzJJ6DsK0laDADEpNJ27n+R8yNBV4 OmEl9Bt7Ym20n9TbwgR92e/9CFR4JVRxhkegyHWv9HKdOKhmd/QXcaUIxBgTGZD1fFe73USsr6r GtKFE6LlSXfa2AQs4rmcV7L4eiMgaRp2wYgaSsAEcspLPabAPVMXnBkn1YhDWvgHeheFPZ3+wuA mB+wj2AyQfbb2zMYLYuQ4BJ3IvKVTCvAOPlGRQp2getSDVGLFkrvUbCZBOh+auzOpUoZuD7NeJc zHt+0nwfhUMo5Bico5PffRRRnKltX+rpCM+GmEjTHrsTCfntXZG08iQyBJk3g+TJMlA479H6l/D L X-Received: by 2002:a05:6000:1fa3:b0:390:f400:2083 with SMTP id ffacd0b85a97d-390f400227cmr11383303f8f.0.1741019443578; Mon, 03 Mar 2025 08:30:43 -0800 (PST) X-Google-Smtp-Source: AGHT+IGnhjPORrAR9y+RvgL1OaYsuXRTyHneFQKcSCvW+9sKqMuJZykkJsayBSNel9nRS5Z27sSgaA== X-Received: by 2002:a05:6000:1fa3:b0:390:f400:2083 with SMTP id ffacd0b85a97d-390f400227cmr11383264f8f.0.1741019443130; Mon, 03 Mar 2025 08:30:43 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43bc032d049sm46650855e9.5.2025.03.03.08.30.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:42 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , "Kirill A. Shutemov" Subject: [PATCH v3 11/20] mm/rmap: use folio_large_nr_pages() in add/remove functions Date: Mon, 3 Mar 2025 17:30:04 +0100 Message-ID: <20250303163014.1128035-12-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: aWzdm0OIBqDcJhhx9T_U3dmskpRaUHzHpoOX64ulgKg_1741019444 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 8BC2F140032 X-Rspam-User: X-Stat-Signature: r6man7acnc884r8cxz6yr5qzczcqpaqi X-HE-Tag: 1741019458-180594 X-HE-Meta: U2FsdGVkX18jBlZhJRvugZTkmn9hJfa568z/nQA8NgllMHhpjA4j+ynh1Op1Be6Xp+xOWq29p5Rh5Uc4Nwbv5r1SWjhSS4JcnNu7exrdSWX4OEhILLwnWmJ2omOFW9ZZcRWsO5iJinR2xJmMBsTrRDMjCXSsxag4cS3lj+h5x9ebDM10jI1qSrozAjEGP5P3tATD5ughQP085a0LNvMtlvcxipGQ1m82T0kKEJZfkxl1hpI3cFQkxV5JNHaPSVaoyRIpo2m6BMhQhtcXxaLCaBoBXlYSnZVm8LO+oZ1HVG1JChmeC53r7+fCif6yzvJcbvhBbpSkb9zb1bMJliwrah2kog6R7sRXygrLnZGlODrIASKQFSrUXeLHTxPBvwOtIsEhvMPF8VzED5UcxV8a+1uqkmnMpS3kLd6tIJAPrKY9rxBjIm5ZK2YYqmJYE4EOXy+Eu8Z6jCtQfXNFjFPKb1InLQE6Km4A65H5PxKeEWimIrhHFlApVb42enJ2Ih0FCI1xJJ28nPUoxoo9Vk07s+eCCXM03HcqGVqi2pPakAS50/HCUqGKSNg2BSI2oCma4O1Me9IwsdaqR5A8x6WGAzRqV2J4co+NCgsoDUnffEub5rdZLQEaQ6Sa21HidZZnXHeEAzIef+OgQjDk6pEP8CHjyvHu2mm/rLRxzO0GD0bXJLN8Ci4RuN5zR+LXCodPu+opLJNzuhj0QhxGv1JwLIbW0wWyqeJD3Gyka+dNdv9lqMAmJXldkgUH7vzQqyoy/TyULe6jCmoSNrOgDZtuTFKo+9kgXPtKyd4k7wQJiKX+eU1RitHot2gQcqeYha+OBmrUYLrnUttEZlAvccXIqDy7msHjh7FSm1goZdvxgvCCvBvlDHCfQRuO0cw7GGmQ+5ZWchORow4s8fjlCbDZcm0iiegMyCg372f6INMwH+OrlffOxmpHs+4Iis0TESaFlV3nOS/+vz51ldU3I9o B2l9zXs5 fELjVu0maWwv/g2QywiGuwf740a81qEQcFp43xKQX1HYGAe4UQT3jc+xhEZ9CEKKCpl9eaF9qJMyefQ+7gN/jy/r9xEQoM8OsHooS25MHXf5GbkJrRNUYoE8GxeiS9sv+RXGWDPb9p4QKR3SyYfGCfW9EdhFAEsUxUnKF6o5EGdrI6npnm8F9BPwXar4V1Ol08Ln34JBMfekxzLS+wXtGI1rzt9u32cIl2lOE4XwTKrou4xrxDueNSNV3Tt5dWQSZs0WhLOd7tTwDwL46NQdjle1nZYxQOqRkg9Kc9Hwc05JDT8phHNuf6cd6aImPggL42T+amFthGCWk//jrp/PbZIVLRtao8KsvYwUaUY1vhhPMs1BaUFBLLtjdHL7z+roVSKtVdYyxM378neyY1eJgfnL2vbOHnti+CY2JKuno+CqtRWNb9GoXNuZ8C14MYfqqgySvkiFc2NZ3yq2TIVZYQwZurGxtdhnkFx25R1UnM8LSOqv6clQK+SfcY96vVVFZSq91UegFVmiN2ygs5aBJpBM2Qf4E46CLqk0pev8X/qVnUb4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's just use the "large" variant in code where we are sure that we have a large folio in our hands: this way we are sure that we don't perform any unnecessary "large" checks. While at it, convert the VM_BUG_ON_VMA to a VM_WARN_ON_ONCE. Maybe in the future there will not be a difference in that regard between large and small folios; in that case, unifying the handling again will be easy. E.g., folio_large_nr_pages() will simply translate to folio_nr_pages() until we replace all instances. Reviewed-by: Kirill A. Shutemov Signed-off-by: David Hildenbrand --- mm/rmap.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 08846b7eced60..c9922928616ee 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1274,7 +1274,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, if (first) { nr = atomic_add_return_relaxed(ENTIRELY_MAPPED, mapped); if (likely(nr < ENTIRELY_MAPPED + ENTIRELY_MAPPED)) { - nr_pages = folio_nr_pages(folio); + nr_pages = folio_large_nr_pages(folio); /* * We only track PMD mappings of PMD-sized * folios separately. @@ -1522,14 +1522,11 @@ void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page, void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address, rmap_t flags) { - const int nr = folio_nr_pages(folio); const bool exclusive = flags & RMAP_EXCLUSIVE; - int nr_pmdmapped = 0; + int nr = 1, nr_pmdmapped = 0; VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!exclusive && !folio_test_locked(folio), folio); - VM_BUG_ON_VMA(address < vma->vm_start || - address + (nr << PAGE_SHIFT) > vma->vm_end, vma); /* * VM_DROPPABLE mappings don't swap; instead they're just dropped when @@ -1547,6 +1544,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, } else if (!folio_test_pmd_mappable(folio)) { int i; + nr = folio_large_nr_pages(folio); for (i = 0; i < nr; i++) { struct page *page = folio_page(folio, i); @@ -1559,6 +1557,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, folio_set_large_mapcount(folio, nr, vma); atomic_set(&folio->_nr_pages_mapped, nr); } else { + nr = folio_large_nr_pages(folio); /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); folio_set_large_mapcount(folio, 1, vma); @@ -1568,6 +1567,9 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, nr_pmdmapped = nr; } + VM_WARN_ON_ONCE(address < vma->vm_start || + address + (nr << PAGE_SHIFT) > vma->vm_end); + __folio_mod_stat(folio, nr, nr_pmdmapped); mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON, 1); } @@ -1681,7 +1683,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, if (last) { nr = atomic_sub_return_relaxed(ENTIRELY_MAPPED, mapped); if (likely(nr < ENTIRELY_MAPPED)) { - nr_pages = folio_nr_pages(folio); + nr_pages = folio_large_nr_pages(folio); if (level == RMAP_LEVEL_PMD) nr_pmdmapped = nr_pages; nr = nr_pages - (nr & FOLIO_PAGES_MAPPED); From patchwork Mon Mar 3 16:30:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52141C282CD for ; Mon, 3 Mar 2025 16:30:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C072280008; Mon, 3 Mar 2025 11:30:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 870AD280003; Mon, 3 Mar 2025 11:30:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 67628280008; Mon, 3 Mar 2025 11:30:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3D4C2280003 for ; Mon, 3 Mar 2025 11:30:54 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4275A120BDA for ; Mon, 3 Mar 2025 16:30:53 +0000 (UTC) X-FDA: 83180778786.18.C98D951 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 91F25100028 for ; Mon, 3 Mar 2025 16:30:49 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="ZES8wTG/"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019449; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ght3iO8/lkNOzwmnKtw9vXpGviqp6AJbx3c5inxt3jw=; b=zpqhWtaD6430G3TlxMmLoDzsGVe41BJOCZsFOWueVqN98i6ML+U9s5J34RUgYNTzbva/cR YYOBoil+oIpd9ZDHl2bN4EN6LMymetf3VRESsD+cXK5a7IyKCvjxXgZVROonCA35FpD6Ge 6y+kNVDwO9le3Zai16zOnwMPYcL7Tkg= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="ZES8wTG/"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019449; a=rsa-sha256; cv=none; b=aWPN1tuC2UnZpxvfDDPbFtRJNvwHD0G5aDBE2jpWaZVUzae+lcjgspAyP8cZ40E6iEXx5E Pi8kNf9vpqF52r1Rjz2OOxH79uEeoA1I8b6wqmpeJM3/n3zLOARz90iJdhVz2ZvSdTZ7wg bISbyzktj8WWv1/Hn3bpjtkEydVU3X8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019448; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ght3iO8/lkNOzwmnKtw9vXpGviqp6AJbx3c5inxt3jw=; b=ZES8wTG/+GKn9Zokcs/JcWFY/aEAVUVYE8fBffhvhDCy9QIVkARhXGPjUU6RVTskF/0Siy LsuCvF8X33dR6G3E2uJMVAkHI8Q4Wx1siG/KcX9ezP/Nz5fohbZixvvDFDOqnDludvuxcK fhRvQ0wEfMamyFNDrKyg5tH/htxMhXU= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-694-787AjgGCMLiwIsmQVdA5Hw-1; Mon, 03 Mar 2025 11:30:47 -0500 X-MC-Unique: 787AjgGCMLiwIsmQVdA5Hw-1 X-Mimecast-MFC-AGG-ID: 787AjgGCMLiwIsmQVdA5Hw_1741019446 Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-390d73c20b6so3097786f8f.2 for ; Mon, 03 Mar 2025 08:30:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019446; x=1741624246; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ght3iO8/lkNOzwmnKtw9vXpGviqp6AJbx3c5inxt3jw=; b=be3FiI4pejiRUVm437puxUvkyCI8daJxg5V1dYE1yxV7NSdWkihz2l/t1iFh38i+Ae ucSy2FcseYQ+pSg1l8u4UGaO6ueuooX7rCvlb5u1+1TR2yPbL9BinRsjeqvsE+fwx23d N6zao92rtJm3ACB6BOsBXxpR+RmZN74LgeXdR6FgSzEhRHFpBsWhQu75Z8EiRxVGoQgF ZCjR7HDVOqsJep3QcsGI0j2+KxWQnvldn0DzZNj5f5T7oUIywgcMg0tJaeIwOCu9g2WG ojjEI8PzlaaLrYppzIvspYCGCkPzh4TdCOyWb8OfWQ8NjAPVQ08yo6yF6IXnrCnEe4lj ErrA== X-Forwarded-Encrypted: i=1; AJvYcCWxOSkQ16zq6fYqRF1j4TmaI4YB7/B+KzzJ0z9VyZqKFq+TZmmBc9jaDK5fSfiIl1eECflsm8P76Q==@kvack.org X-Gm-Message-State: AOJu0YwFktPC4O3jQnF0dm4aqBDIIcjkoryP1bX4bbWh4320HArGV/LV F+NFf6jdLiZCAsRa8cetIj7rOp1jEo9dFk2N6FEQqfzbbIIuG/uRzoDMvt8wE7fQjDo0EV8d5L5 h1db9gTBaw715XSCV6CedGaYoBOm/mDZRqYjFYZ7uJs/rYQFa X-Gm-Gg: ASbGncvLn+Mj3z49AlTnaa4nVbofuavkTj3kkQWAqRpSrEGEHVYOTOFoYEAYdBr4qsS HByu1O2oIG8/9CY2ml9Z5AHdptsXOU44QZs2v5OPRWWqsPSjnpZzBc0vh/dJn5e9R/XUp78hwjS nYZcnDIgz4xQzvvYkyEeAKOA7Kc7Db/Z+rIgwxgvtBPIqHOZJEKVKivAvj2S4/gPCdyafnQPOkq uSOdiT/jH/eeGBBcufid1iOw2TAEeFxDXcp/rbB/7AehA3kEGhAuxAUmm0Kp9z+oGazpW6FaxwT zQgnrJkE9Qa/TZCCVRqAmr3Fz5rCAz0sHwM+Kv4frrRP9U5u4T7w6uo1ZhITLn78gLz0KGi6/HM C X-Received: by 2002:a5d:6d0e:0:b0:390:ff25:79c8 with SMTP id ffacd0b85a97d-390ff257cccmr5482781f8f.20.1741019445763; Mon, 03 Mar 2025 08:30:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IFmOzbyVhjQHnOiNa5KjVt0NLNOpFtZnZeHlUY2sgZnsiw2i9l5pVj5IO3A736y2GBbWD/9gw== X-Received: by 2002:a5d:6d0e:0:b0:390:ff25:79c8 with SMTP id ffacd0b85a97d-390ff257cccmr5482733f8f.20.1741019445180; Mon, 03 Mar 2025 08:30:45 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43aba5870e7sm199192765e9.35.2025.03.03.08.30.43 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:44 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 12/20] mm/rmap: basic MM owner tracking for large folios (!hugetlb) Date: Mon, 3 Mar 2025 17:30:05 +0100 Message-ID: <20250303163014.1128035-13-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: mnGFxCn-ECtkeSLfwWtpcJKnrO_59fCpU5ybbMxzi8k_1741019446 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 91F25100028 X-Rspam-User: X-Stat-Signature: kyw79i61pxwbgg8szf54cgnjkfk8jwyi X-HE-Tag: 1741019449-611389 X-HE-Meta: U2FsdGVkX1+vdyFX3NJ3LMxbSFHdLt49mJEGVNF5pARrWXjo6wKFywO6OH13V8BJxOiVoC63Cb8k6/axu746TCtgnYfSGyn9fbnx5h7gO24Ws0ZZM03jzkzFebZ6kSOasCJDZ73a2GhErMr7ADnYoQqVhKRKTK4H5y2W+unhblL7//aOYI0FN046R9t4V4ZvpNEno8eIw2fkU8H9n4o6DDV45IApDLk3YatGXl5C8JBmHBJlTRg4x6NNKhB7Ij214gItQyTtxXyenZKR0nLOXvzuRjDYN2y7ykDaZoTZ7ynwdavYgXADlSaFuANvKxl0vJ1XAB2U49aB+7JOwXlbTHxZyJo3dDQyPX2wVtksibThs0YVJeFakxTBONJ8VGTO/ytcMhz1n8lD2THzqqgrawUYJV0l7B2VCpgLrvS3E9HZxUH+2EFeEaxkdXlskfyYmt6al8cb5O5oSIqPrZSfiOhhy/5/+a75c1yQnfVeX5AlUjenWh6rjB/6KDA8iH5Psr0HccJEocqF+lpnopP188g+mjlszn+A07LU0MBkr40Sxl96onYkMxamLU/5rRxoGy2YJ9+p7uSIZJhwx+sOCGCpgMkOb0vHAQfMKbbq18z+/KVPlKZx3qxZ6TB82s0DhUgQ0Zibx3w/ksIHlxPxvBHD1ULh/dDJoFyLr4tMrJsVqD/5JYwyhuptoxNi94+BSffhSIi2smOuncHT/Yo2IrgK9fSCAvXcQySoYTlMh0URKtJGraCpw2ASTQyMXVKMpRE7RT7AJTqRLQvqNUbbDy4L+Ags2H+etH4D8/Jlm+uDbOOwlYRBsX4MGr6bzzP7Xn34RiQNEPOEQVQ7zXCnnkih5hStOR9xiUalgUz9GHd/5Ziq4r8Ipa7Do2CAt3eigtLD/tRtOMxIKs+FHUW+Cjf+F44yI9FpiEisgVgrVc/HBz01E8ax/91nbUDSdd5VjfdkxGbj0G3zCJZv2rK ld5xZaHY uH1tclZbPJsX3r4VkSdZa1wZ+45K8iJz/o+MrJvIh3wDkaA9JAEJB4L95yG3J1XwJ29eL2L/G4IwVhZvUXtC+KvoCnDxJ8S98jHr+G8S89tWeCbIBkvEZOfAf1HgE0oirupy5OQCAlHlRr4mY+/wHDH7oLpUjT+B5vcLY8GD74BT/msN3NqKE2YV7eAJYESdb/tbJKVppfnMYKbYqMCTwb9dSk2rnS3649KFWXB8Jbzad4YmC7h3j8YblSaqTeyE+1iqn0tKbccTrRTtyYO0gTfb4FtCoQHm9XBjAqGr4dnNO5A6Fq2UaEwKtnnDoagclUbCm7pcaM5AelxxXCuwiin7Uunw5P9lRAn2wJjrXtzacizVU8/snb4WyOCqoR/pRkepSjdsp+cGXDWfACDud/OV66jQD0G4rd8Eh6WO1AXsSoCdSPqQr80szXXUnW7XXaeg+0VSoHbWVTNGBpu/Nr9L7Z43ZyI8bGTZ1CreKpJEkgBnxmAS7N4vqhxzmJxrRd9kLttu5+asw1T4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For small folios, we traditionally use the mapcount to decide whether it was "certainly mapped exclusively" by a single MM (mapcount == 1) or whether it "maybe mapped shared" by multiple MMs (mapcount > 1). For PMD-sized folios that were PMD-mapped, we were able to use a similar mechanism (single PMD mapping), but for PTE-mapped folios and in the future folios that span multiple PMDs, this does not work. So we need a different mechanism to handle large folios. Let's add a new mechanism to detect whether a large folio is "certainly mapped exclusively", or whether it is "maybe mapped shared". We'll use this information next to optimize CoW reuse for PTE-mapped anonymous THP, and to convert folio_likely_mapped_shared() to folio_maybe_mapped_shared(), independent of per-page mapcounts. For each large folio, we'll have two slots, whereby a slot stores: (1) an MM id: unique id assigned to each MM (2) a per-MM mapcount If a slot is unoccupied, it can be taken by the next MM that maps folio page. In addition, we'll remember the current state -- "mapped exclusively" vs. "maybe mapped shared" -- and use a bit spinlock to sync on updates and to reduce the total number of atomic accesses on updates. In the future, it might be possible to squeeze a proper spinlock into "struct folio". For now, keep it simple, as we require the whole thing with THP only, that is incompatible with RT. As we have to squeeze this information into the "struct folio" of even folios of order-1 (2 pages), and we generally want to reduce the required metadata, we'll assign each MM a unique ID that can fit into an int. In total, we can squeeze everything into 4x int (2x long) on 64bit. 32bit support is a bit challenging, because we only have 2x long == 2x int in order-1 folios. But we can make it work for now, because we neither expect many MMs nor very large folios on 32bit. We will reliably detect folios as "mapped exclusively" vs. "mapped shared" as long as only two MMs map pages of a folio at one point in time -- for example with fork() and short-lived child processes, or with apps that hand over state from one instance to another. As soon as three MMs are involved at the same time, we might detect "maybe mapped shared" although the folio is "mapped exclusively". Example 1: (1) App1 faults in a (shmem/file-backed) folio page -> Tracked as MM0 (2) App2 faults in a folio page -> Tracked as MM1 (4) App1 unmaps all folio pages -> We will detect "mapped exclusively". Example 2: (1) App1 faults in a (shmem/file-backed) folio page -> Tracked as MM0 (2) App2 faults in a folio page -> Tracked as MM1 (3) App3 faults in a folio page -> No slot available, tracked as "unknown" (4) App1 and App2 unmap all folio pages -> We will detect "maybe mapped shared". Make use of __always_inline to keep possible performance degradation when (un)mapping large folios to a minimum. Note: by squeezing the two flags into the "unsigned long" that stores the MM ids, we can use non-atomic __bit_spin_unlock() and non-atomic setting/clearing of the "maybe mapped shared" bit, effectively not adding any new atomics on the hot path when updating the large mapcount + new metadata, which further helps reduce the runtime overhead in micro-benchmarks. Signed-off-by: David Hildenbrand --- Documentation/mm/transhuge.rst | 8 ++ include/linux/mm_types.h | 49 ++++++++++ include/linux/page-flags.h | 4 + include/linux/rmap.h | 165 +++++++++++++++++++++++++++++++++ kernel/fork.c | 36 +++++++ mm/Kconfig | 4 + mm/internal.h | 5 + mm/page_alloc.c | 10 ++ 8 files changed, 281 insertions(+) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index a2cd8800d5279..baa17d718a762 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -120,11 +120,19 @@ pages: and also increment/decrement folio->_nr_pages_mapped by ENTIRELY_MAPPED when _entire_mapcount goes from -1 to 0 or 0 to -1. + We also maintain the two slots for tracking MM owners (MM ID and + corresponding mapcount), and the current status ("maybe mapped shared" vs. + "mapped exclusively"). + - map/unmap of individual pages with PTE entry increment/decrement page->_mapcount, increment/decrement folio->_large_mapcount and also increment/decrement folio->_nr_pages_mapped when page->_mapcount goes from -1 to 0 or 0 to -1 as this counts the number of pages mapped by PTE. + We also maintain the two slots for tracking MM owners (MM ID and + corresponding mapcount), and the current status ("maybe mapped shared" vs. + "mapped exclusively"). + split_huge_page internally has to distribute the refcounts in the head page to the tail pages before clearing all PG_head/tail bits from the page structures. It can be done easily for refcounts taken by page table diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index c83dd2f1ee25e..2d657ac8e9b0c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -292,6 +292,44 @@ typedef struct { #define NR_PAGES_IN_LARGE_FOLIO #endif +/* + * On 32bit, we can cut the required metadata in half, because: + * (a) PID_MAX_LIMIT implicitly limits the number of MMs we could ever have, + * so we can limit MM IDs to 15 bit (32767). + * (b) We don't expect folios where even a single complete PTE mapping by + * one MM would exceed 15 bits (order-15). + */ +#ifdef CONFIG_64BIT +typedef int mm_id_mapcount_t; +#define MM_ID_MAPCOUNT_MAX INT_MAX +typedef unsigned int mm_id_t; +#else /* !CONFIG_64BIT */ +typedef short mm_id_mapcount_t; +#define MM_ID_MAPCOUNT_MAX SHRT_MAX +typedef unsigned short mm_id_t; +#endif /* CONFIG_64BIT */ + +/* We implicitly use the dummy ID for init-mm etc. where we never rmap pages. */ +#define MM_ID_DUMMY 0 +#define MM_ID_MIN (MM_ID_DUMMY + 1) + +/* + * We leave the highest bit of each MM id unused, so we can store a flag + * in the highest bit of each folio->_mm_id[]. + */ +#define MM_ID_BITS ((sizeof(mm_id_t) * BITS_PER_BYTE) - 1) +#define MM_ID_MASK ((1U << MM_ID_BITS) - 1) +#define MM_ID_MAX MM_ID_MASK + +/* + * In order to use bit_spin_lock(), which requires an unsigned long, we + * operate on folio->_mm_ids when working on flags. + */ +#define FOLIO_MM_IDS_LOCK_BITNUM MM_ID_BITS +#define FOLIO_MM_IDS_LOCK_BIT BIT(FOLIO_MM_IDS_LOCK_BITNUM) +#define FOLIO_MM_IDS_SHARED_BITNUM (2 * MM_ID_BITS + 1) +#define FOLIO_MM_IDS_SHARED_BIT BIT(FOLIO_MM_IDS_SHARED_BITNUM) + /** * struct folio - Represents a contiguous set of bytes. * @flags: Identical to the page flags. @@ -318,6 +356,9 @@ typedef struct { * @_nr_pages_mapped: Do not use outside of rmap and debug code. * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). * @_nr_pages: Do not use directly, call folio_nr_pages(). + * @_mm_id: Do not use outside of rmap code. + * @_mm_ids: Do not use outside of rmap code. + * @_mm_id_mapcount: Do not use outside of rmap code. * @_hugetlb_subpool: Do not use directly, use accessor in hugetlb.h. * @_hugetlb_cgroup: Do not use directly, use accessor in hugetlb_cgroup.h. * @_hugetlb_cgroup_rsvd: Do not use directly, use accessor in hugetlb_cgroup.h. @@ -390,6 +431,11 @@ struct folio { atomic_t _entire_mapcount; atomic_t _pincount; #endif /* CONFIG_64BIT */ + mm_id_mapcount_t _mm_id_mapcount[2]; + union { + mm_id_t _mm_id[2]; + unsigned long _mm_ids; + }; }; unsigned long _usable_1[4]; }; @@ -1111,6 +1157,9 @@ struct mm_struct { #endif } lru_gen; #endif /* CONFIG_LRU_GEN_WALKS_MMU */ +#ifdef CONFIG_MM_ID + mm_id_t mm_id; +#endif /* CONFIG_MM_ID */ } __randomize_layout; /* diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 30fe3eb62b90c..01716710066df 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -1222,6 +1222,10 @@ static inline int folio_has_private(const struct folio *folio) return !!(folio->flags & PAGE_FLAGS_PRIVATE); } +static inline bool folio_test_large_maybe_mapped_shared(const struct folio *folio) +{ + return test_bit(FOLIO_MM_IDS_SHARED_BITNUM, &folio->_mm_ids); +} #undef PF_ANY #undef PF_HEAD #undef PF_NO_TAIL diff --git a/include/linux/rmap.h b/include/linux/rmap.h index d1e888cc97a58..c131b0efff0fa 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -13,6 +13,7 @@ #include #include #include +#include /* * The anon_vma heads a list of private "related" vmas, to scan if @@ -173,6 +174,169 @@ static inline void anon_vma_merge(struct vm_area_struct *vma, struct anon_vma *folio_get_anon_vma(const struct folio *folio); +#ifdef CONFIG_MM_ID +static __always_inline void folio_lock_large_mapcount(struct folio *folio) +{ + bit_spin_lock(FOLIO_MM_IDS_LOCK_BITNUM, &folio->_mm_ids); +} + +static __always_inline void folio_unlock_large_mapcount(struct folio *folio) +{ + __bit_spin_unlock(FOLIO_MM_IDS_LOCK_BITNUM, &folio->_mm_ids); +} + +static inline unsigned int folio_mm_id(const struct folio *folio, int idx) +{ + VM_WARN_ON_ONCE(idx != 0 && idx != 1); + return folio->_mm_id[idx] & MM_ID_MASK; +} + +static inline void folio_set_mm_id(struct folio *folio, int idx, mm_id_t id) +{ + VM_WARN_ON_ONCE(idx != 0 && idx != 1); + folio->_mm_id[idx] &= ~MM_ID_MASK; + folio->_mm_id[idx] |= id; +} + +static inline void __folio_large_mapcount_sanity_checks(const struct folio *folio, + int diff, mm_id_t mm_id) +{ + VM_WARN_ON_ONCE(!folio_test_large(folio) || folio_test_hugetlb(folio)); + VM_WARN_ON_ONCE(diff <= 0); + VM_WARN_ON_ONCE(mm_id < MM_ID_MIN || mm_id > MM_ID_MAX); + + /* + * Make sure we can detect at least one complete PTE mapping of the + * folio in a single MM as "exclusively mapped". This is primarily + * a check on 32bit, where we currently reduce the size of the per-MM + * mapcount to a short. + */ + VM_WARN_ON_ONCE(diff > folio_large_nr_pages(folio)); + VM_WARN_ON_ONCE(folio_large_nr_pages(folio) - 1 > MM_ID_MAPCOUNT_MAX); + + VM_WARN_ON_ONCE(folio_mm_id(folio, 0) == MM_ID_DUMMY && + folio->_mm_id_mapcount[0] != -1); + VM_WARN_ON_ONCE(folio_mm_id(folio, 0) != MM_ID_DUMMY && + folio->_mm_id_mapcount[0] < 0); + VM_WARN_ON_ONCE(folio_mm_id(folio, 1) == MM_ID_DUMMY && + folio->_mm_id_mapcount[1] != -1); + VM_WARN_ON_ONCE(folio_mm_id(folio, 1) != MM_ID_DUMMY && + folio->_mm_id_mapcount[1] < 0); + VM_WARN_ON_ONCE(!folio_mapped(folio) && + folio_test_large_maybe_mapped_shared(folio)); +} + +static __always_inline void folio_set_large_mapcount(struct folio *folio, + int mapcount, struct vm_area_struct *vma) +{ + __folio_large_mapcount_sanity_checks(folio, mapcount, vma->vm_mm->mm_id); + + VM_WARN_ON_ONCE(folio_mm_id(folio, 0) != MM_ID_DUMMY); + VM_WARN_ON_ONCE(folio_mm_id(folio, 1) != MM_ID_DUMMY); + + /* Note: mapcounts start at -1. */ + atomic_set(&folio->_large_mapcount, mapcount - 1); + folio->_mm_id_mapcount[0] = mapcount - 1; + folio_set_mm_id(folio, 0, vma->vm_mm->mm_id); +} + +static __always_inline void folio_add_large_mapcount(struct folio *folio, + int diff, struct vm_area_struct *vma) +{ + const mm_id_t mm_id = vma->vm_mm->mm_id; + int new_mapcount_val; + + folio_lock_large_mapcount(folio); + __folio_large_mapcount_sanity_checks(folio, diff, mm_id); + + new_mapcount_val = atomic_read(&folio->_large_mapcount) + diff; + atomic_set(&folio->_large_mapcount, new_mapcount_val); + + /* + * If a folio is mapped more than once into an MM on 32bit, we + * can in theory overflow the per-MM mapcount (although only for + * fairly large folios), turning it negative. In that case, just + * free up the slot and mark the folio "mapped shared", otherwise + * we might be in trouble when unmapping pages later. + */ + if (folio_mm_id(folio, 0) == mm_id) { + folio->_mm_id_mapcount[0] += diff; + if (!IS_ENABLED(CONFIG_64BIT) && unlikely(folio->_mm_id_mapcount[0] < 0)) { + folio->_mm_id_mapcount[0] = -1; + folio_set_mm_id(folio, 0, MM_ID_DUMMY); + folio->_mm_ids |= FOLIO_MM_IDS_SHARED_BIT; + } + } else if (folio_mm_id(folio, 1) == mm_id) { + folio->_mm_id_mapcount[1] += diff; + if (!IS_ENABLED(CONFIG_64BIT) && unlikely(folio->_mm_id_mapcount[1] < 0)) { + folio->_mm_id_mapcount[1] = -1; + folio_set_mm_id(folio, 1, MM_ID_DUMMY); + folio->_mm_ids |= FOLIO_MM_IDS_SHARED_BIT; + } + } else if (folio_mm_id(folio, 0) == MM_ID_DUMMY) { + folio_set_mm_id(folio, 0, mm_id); + folio->_mm_id_mapcount[0] = diff - 1; + /* We might have other mappings already. */ + if (new_mapcount_val != diff - 1) + folio->_mm_ids |= FOLIO_MM_IDS_SHARED_BIT; + } else if (folio_mm_id(folio, 1) == MM_ID_DUMMY) { + folio_set_mm_id(folio, 1, mm_id); + folio->_mm_id_mapcount[1] = diff - 1; + /* Slot 0 certainly has mappings as well. */ + folio->_mm_ids |= FOLIO_MM_IDS_SHARED_BIT; + } + folio_unlock_large_mapcount(folio); +} + +static __always_inline void folio_sub_large_mapcount(struct folio *folio, + int diff, struct vm_area_struct *vma) +{ + const mm_id_t mm_id = vma->vm_mm->mm_id; + int new_mapcount_val; + + folio_lock_large_mapcount(folio); + __folio_large_mapcount_sanity_checks(folio, diff, mm_id); + + new_mapcount_val = atomic_read(&folio->_large_mapcount) - diff; + atomic_set(&folio->_large_mapcount, new_mapcount_val); + + /* + * There are valid corner cases where we might underflow a per-MM + * mapcount (some mappings added when no slot was free, some mappings + * added once a slot was free), so we always set it to -1 once we go + * negative. + */ + if (folio_mm_id(folio, 0) == mm_id) { + folio->_mm_id_mapcount[0] -= diff; + if (folio->_mm_id_mapcount[0] >= 0) + goto out; + folio->_mm_id_mapcount[0] = -1; + folio_set_mm_id(folio, 0, MM_ID_DUMMY); + } else if (folio_mm_id(folio, 1) == mm_id) { + folio->_mm_id_mapcount[1] -= diff; + if (folio->_mm_id_mapcount[1] >= 0) + goto out; + folio->_mm_id_mapcount[1] = -1; + folio_set_mm_id(folio, 1, MM_ID_DUMMY); + } + + /* + * If one MM slot owns all mappings, the folio is mapped exclusively. + * Note that if the folio is now unmapped (new_mapcount_val == -1), both + * slots must be free (mapcount == -1), and we'll also mark it as + * exclusive. + */ + if (folio->_mm_id_mapcount[0] == new_mapcount_val || + folio->_mm_id_mapcount[1] == new_mapcount_val) + folio->_mm_ids &= ~FOLIO_MM_IDS_SHARED_BIT; +out: + folio_unlock_large_mapcount(folio); +} +#else /* !CONFIG_MM_ID */ +/* + * See __folio_rmap_sanity_checks(), we might map large folios even without + * CONFIG_TRANSPARENT_HUGEPAGE. We'll keep that working for now. + */ static inline void folio_set_large_mapcount(struct folio *folio, int mapcount, struct vm_area_struct *vma) { @@ -191,6 +355,7 @@ static inline void folio_sub_large_mapcount(struct folio *folio, { atomic_sub(diff, &folio->_large_mapcount); } +#endif /* CONFIG_MM_ID */ #define folio_inc_large_mapcount(folio, vma) \ folio_add_large_mapcount(folio, 1, vma) diff --git a/kernel/fork.c b/kernel/fork.c index 364b2d4fd3efa..f9cf0f056eb6f 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -802,6 +802,36 @@ static int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) #define mm_free_pgd(mm) #endif /* CONFIG_MMU */ +#ifdef CONFIG_MM_ID +static DEFINE_IDA(mm_ida); + +static inline int mm_alloc_id(struct mm_struct *mm) +{ + int ret; + + ret = ida_alloc_range(&mm_ida, MM_ID_MIN, MM_ID_MAX, GFP_KERNEL); + if (ret < 0) + return ret; + mm->mm_id = ret; + return 0; +} + +static inline void mm_free_id(struct mm_struct *mm) +{ + const mm_id_t id = mm->mm_id; + + mm->mm_id = MM_ID_DUMMY; + if (id == MM_ID_DUMMY) + return; + if (WARN_ON_ONCE(id < MM_ID_MIN || id > MM_ID_MAX)) + return; + ida_free(&mm_ida, id); +} +#else /* !CONFIG_MM_ID */ +static inline int mm_alloc_id(struct mm_struct *mm) { return 0; } +static inline void mm_free_id(struct mm_struct *mm) {} +#endif /* CONFIG_MM_ID */ + static void check_mm(struct mm_struct *mm) { int i; @@ -905,6 +935,7 @@ void __mmdrop(struct mm_struct *mm) WARN_ON_ONCE(mm == current->active_mm); mm_free_pgd(mm); + mm_free_id(mm); destroy_context(mm); mmu_notifier_subscriptions_destroy(mm); check_mm(mm); @@ -1289,6 +1320,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, if (mm_alloc_pgd(mm)) goto fail_nopgd; + if (mm_alloc_id(mm)) + goto fail_noid; + if (init_new_context(p, mm)) goto fail_nocontext; @@ -1308,6 +1342,8 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, fail_cid: destroy_context(mm); fail_nocontext: + mm_free_id(mm); +fail_noid: mm_free_pgd(mm); fail_nopgd: free_mm(mm); diff --git a/mm/Kconfig b/mm/Kconfig index fba9757e58147..4034a0441f650 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -809,11 +809,15 @@ config ARCH_WANT_GENERAL_HUGETLB config ARCH_WANTS_THP_SWAP def_bool n +config MM_ID + def_bool n + menuconfig TRANSPARENT_HUGEPAGE bool "Transparent Hugepage Support" depends on HAVE_ARCH_TRANSPARENT_HUGEPAGE && !PREEMPT_RT select COMPACTION select XARRAY_MULTI + select MM_ID help Transparent Hugepages allows the kernel to use huge pages and huge tlb transparently to the applications whenever possible. diff --git a/mm/internal.h b/mm/internal.h index 9860e65ffc945..e33a1fc5ed667 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -720,6 +720,11 @@ static inline void prep_compound_head(struct page *page, unsigned int order) folio_set_order(folio, order); atomic_set(&folio->_large_mapcount, -1); atomic_set(&folio->_nr_pages_mapped, 0); + if (IS_ENABLED(CONFIG_MM_ID)) { + folio->_mm_ids = 0; + folio->_mm_id_mapcount[0] = -1; + folio->_mm_id_mapcount[1] = -1; + } if (IS_ENABLED(CONFIG_64BIT) || order > 1) { atomic_set(&folio->_pincount, 0); atomic_set(&folio->_entire_mapcount, -1); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b0739baf7b07f..e3b8bfdd0b756 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -959,6 +959,16 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) bad_page(page, "nonzero nr_pages_mapped"); goto out; } + if (IS_ENABLED(CONFIG_MM_ID)) { + if (unlikely(folio->_mm_id_mapcount[0] != -1)) { + bad_page(page, "nonzero mm mapcount 0"); + goto out; + } + if (unlikely(folio->_mm_id_mapcount[1] != -1)) { + bad_page(page, "nonzero mm mapcount 1"); + goto out; + } + } if (IS_ENABLED(CONFIG_64BIT)) { if (unlikely(atomic_read(&folio->_entire_mapcount) + 1)) { bad_page(page, "nonzero entire_mapcount"); From patchwork Mon Mar 3 16:30:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0592C282CD for ; Mon, 3 Mar 2025 16:31:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2315328000F; Mon, 3 Mar 2025 11:31:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C33828000C; Mon, 3 Mar 2025 11:31:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F28B728000F; Mon, 3 Mar 2025 11:31:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CD1C828000C for ; Mon, 3 Mar 2025 11:31:03 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 94DDCA494C for ; Mon, 3 Mar 2025 16:31:03 +0000 (UTC) X-FDA: 83180779206.11.6032082 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id DA4C14001A for ; Mon, 3 Mar 2025 16:30:56 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jFTTM+kq; spf=pass (imf11.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019456; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=L4FvEMzbdPGmsETJ2MBrv3sRxLuQ35ROZ1b2dyZFsHo=; b=BAgva+jeTtCLMkhTmWMQG2QM78K3rICfI9GRP4PYcd9Vb9j6R1t2b8At5Jt/MhKek1UO5t i9Y3tcquSkhvfi8X/RYtOgdl66Jxf/jXhy8/RG82m4WhfTfRyDmOiLUx5tzIhg4uGX4KsC 50QgOm51CP5vj0YRcQ5B306TenOkBfU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019456; a=rsa-sha256; cv=none; b=fklHv7QlC52UgmLr+27bUBtlpwQAKoMb3qa9GoLYuTI+Hf8cVNG2S7f8k1ZQ/wxZ8ECsS6 RB9wKp4PX/KilBg0BWwlu+FNn/Ma/gH516Yx4ru2WJZTP/vXI4KmBOrS10SNiOarByHm1R saqM9edvdyrDqT69jw5WDLFu5scMHUM= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jFTTM+kq; spf=pass (imf11.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019455; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L4FvEMzbdPGmsETJ2MBrv3sRxLuQ35ROZ1b2dyZFsHo=; b=jFTTM+kqDPgoMTdn+rFwQqy0/QULIs31F+iqVN/3CmifnlT/nAZdOrXvLOxTviVCKlu6tA HjaWLTnAVVo2hoamaj70OKoQJUVURhSNDbqCJYmzRJRzt9lruLemmXZ/2wGs84KUB5mKv7 VMtqDHEuVhtK9xo5J0+dgo1HpA5jLG4= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-45-nLLjbb4UMOq5qjE1yWsLBw-1; Mon, 03 Mar 2025 11:30:48 -0500 X-MC-Unique: nLLjbb4UMOq5qjE1yWsLBw-1 X-Mimecast-MFC-AGG-ID: nLLjbb4UMOq5qjE1yWsLBw_1741019448 Received: by mail-wr1-f72.google.com with SMTP id ffacd0b85a97d-390f365274dso1237035f8f.0 for ; Mon, 03 Mar 2025 08:30:48 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019447; x=1741624247; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=L4FvEMzbdPGmsETJ2MBrv3sRxLuQ35ROZ1b2dyZFsHo=; b=GMOLVnVArCW1UyC3OE8AxkfiCH1mbuDzk57ZRf2I35lFYz8Lta/2gbFJ/hcu7cl+zM RIDITw2DZVt/xG86BbpxleuvkNLNqOsGM46oJDZnDKA0pkbVlV83eFgzmBzYKU/iK7TR /uDqNdHntidh81Kdlw8/E/xkAJrCPN70J/GuoeeCArRvKz+AFzsBSS/srNCpBggJCEoa /EgOGwPru/qz8J4ppi5tvj21OMZrIZ66VYY7uUGttAf5KBcI/sBItEHjlUoSvLv3/bhq ASzsnLuF+Fa2rthuyEZ13bFf2ALqBxtlGKDnXnCnyGet4A0nH4+nNvXf9tbwVK5N8yYZ kmaQ== X-Forwarded-Encrypted: i=1; AJvYcCUur4KUQPr+0cf1Qznp56IV711/V84+m8CCIlsQy3b5Xy1qKZZxLwtxoptghLCfDzt3wzLntvoq4w==@kvack.org X-Gm-Message-State: AOJu0Yy1eqF4umE5HChVCJqG0kQpwo6EriSzRZWLE0Fr/bHgPQq5Ux1F UweOUcNI3iDU/bH1twWUR/k4jC47JPuNIC74tJ1FdlGc7tdD4XffMzoa6EF0qUnjfsCL1ypQ381 x0EVl0H8ukHh9VMUMraSONw75FSVbVDZD+Mt3trWxopAqx86m X-Gm-Gg: ASbGnctkOpAB5Mf+4xm0vKttFppIVOTk73yxG9+IZuYJ71hMFjB4oeSvy6TR4BIzJGW 3xBQPY8lCW+9Pl3JtcUMCEeU85HaU8385zGHV1cqVYLtcCRV1SekMt6WluZDLHWZOGnOeaaqFgq E6qqIICLfdqSGtVDnYCoDmHFUstwfT5s6b5tEWdFf4wggMY2uob3PYfhyD/s+zITwqpNphWk6fE dRHg5EN1hjyQOOTaCgLGc+jKdDfbvrkz6MBcsEWY81fDDLOZeZo/Dvnzne5uj68QhAN1ZovCzVR 95j/7sIbrSg4PYB8Y+YEgKXKSC/knr9Nf/VgRUoX3XzVFnFaZV0dPLUGNUvSVoxpLW6Qb86NPSR I X-Received: by 2002:a05:6000:1563:b0:390:e9e0:5cc6 with SMTP id ffacd0b85a97d-390e9e062d6mr13290396f8f.1.1741019447605; Mon, 03 Mar 2025 08:30:47 -0800 (PST) X-Google-Smtp-Source: AGHT+IFZ9qLz9Zew7W+f+EXmCL5po72v+Q03kK3/3MSwVL/z8h10XiTsQrbDI7Khsstadj0a/aIAmA== X-Received: by 2002:a05:6000:1563:b0:390:e9e0:5cc6 with SMTP id ffacd0b85a97d-390e9e062d6mr13290341f8f.1.1741019447204; Mon, 03 Mar 2025 08:30:47 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-390e4795d1asm14736287f8f.4.2025.03.03.08.30.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:46 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 13/20] mm: Copy-on-Write (COW) reuse support for PTE-mapped THP Date: Mon, 3 Mar 2025 17:30:06 +0100 Message-ID: <20250303163014.1128035-14-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: ZllzbOIhlLf7toB4usFruD2fE8tBnYhMYCdtFrkfxik_1741019448 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam02 X-Stat-Signature: rwguskgxqsbcdhn7ugbz91shkinereim X-Rspamd-Queue-Id: DA4C14001A X-Rspam-User: X-HE-Tag: 1741019456-859075 X-HE-Meta: U2FsdGVkX18vSRmsONTz8HsZ1w52M3qZSo6RNE1XXJl4krqEu5dzs6PBaXh63pHuX0TnHLFuPgzlWIn8mJM0QdLYEkzsnhWWOFSk3t9yBWRjryu1onGluLVzWXPTyO1ez0ECu4/n/cRe5dpg8KDm2d4ENMZXUVKQDwBcfIg3Yc1VeC9faznm5MNzu/LI6mWGj08xesOKExFspys3dFomxi2504zqvYI5xwADJ8UNbQdQH4Q7k15I6cAn8rSE8MG83ad3Um8uk5SHnduGWp7T6a2oT/eZhiNC+2+gcjuaaU5DHEXIyWy+cILRbFNuEBbayM7va+mkqvd63uGU/Zb0PT1Xt5vKEpinXkysTZK48dcMU0tJt61GM+5Q9DRvm5WoeyRa9KP7lUDdRT38ZhJN8FpQVadfskJwYuMBHMdGv6M2bOUQKGrTQ7odZg9cHPJ+SLLl5L58R9rgcZDhGBQ9EooppRmolE5jEIe3zHWbnpBTSA8MVUeYlvrYU/a1yz5SoMwgkyqllL3CmYRBJYdyp38kKS6hIEtIczWNGlUahSC9IFyazTvGACH2QW8AdJzq927inQdOazv6eQrWc2Tps+Ttvkh0tzgrjX4jrCoR/7RkrOIVY9XxC7FK3U4JywWKeEbRcTeYPdc9k7N5t32y9oT2/NqzCKuV4iya9Li5PWhn7uOeDnKyslXcEbxs3ZYQ+YzI8BjJBsCIbYGXjruAbcmPVjK1N2+rUBVqXC14ChHqGzUMjkMo4OXTUl+0kVuNGu10PFcHUa+z71HDRYz8tsKD2mMdZp1VxN1gRQBIMug4C9feVJbavdPwuUlflUjoRYC0cpSC6E1Lyhx/GOSQ77cmZ9+bMyt+wxEnlAOF+9UbHZw1BPbIb09YOmf8s+Eo4ALXZbKd/2HvyS7gT4R0k2mV44NpgAqC40D8hq0coK92oWn9UpL4dTFB7+W9tBFhp7MrSWOW5BfvEm9vk1o urRmg20I eoQSoinjARsQ5pGkeTsbcQK3gMTPzdAeiOx6HhUe8RvrZ3/s+D6HqfGSo9JeKJndRMLrbIYjB4JYlW9FcaP8vUFSNoRU7j/rAnjprgVPFsOEXLkcD9EpZEtgngtGRKDO4G4vF+2lTFsQZ7OlXF0t7l3jdo1Mw8L28jC/bcjKyU518xjuBgtXI+2hbT2PkSFKt9l0mU2xpKLnFjp7Sc0qOTzeyn0cfRWTMWGRzIxwFAQRWoYvXY/93Lp6S20Ey8siDtqUoX7Pn990kJHw0s7K2VKxYyLDVsz6ZOz4ioqNE9ITpHSJsmezAtr6mkatFmZw6vKumJmgqoVbVw6EK2QMd+Yv/zEDcGeu4vicj97GnFdJlmS2hTXXt/QTW3adbPdOFmRke12oDWysahNiPcXVuZ4PFGdM1S+2jZkqPlmeSkKcBB8Igftv8jBVOsAtE9BG4KQXeYLfJecqn057K6Xq92uwZCicClWQkbspe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, we never end up reusing PTE-mapped THPs after fork. This wasn't really a problem with PMD-sized THPs, because they would have to be PTE-mapped first, but it's getting a problem with smaller THP sizes that are effectively always PTE-mapped. With our new "mapped exclusively" vs "maybe mapped shared" logic for large folios, implementing CoW reuse for PTE-mapped THPs is straight forward: if exclusively mapped, make sure that all references are from these (our) mappings. Add some helpful comments to explain the details. CONFIG_TRANSPARENT_HUGEPAGE selects CONFIG_MM_ID. If we spot an anon large folio without CONFIG_TRANSPARENT_HUGEPAGE in that code, something is seriously messed up. There are plenty of things we can optimize in the future: For example, we could remember that the folio is fully exclusive so we could speedup the next fault further. Also, we could try "faulting around", turning surrounding PTEs that map the same folio writable. But especially the latter might increase COW latency, so it would need further investigation. Signed-off-by: David Hildenbrand --- mm/memory.c | 83 +++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 75 insertions(+), 8 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 73b783c7d7d51..bb245a8fe04bc 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3729,19 +3729,86 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) return ret; } -static bool wp_can_reuse_anon_folio(struct folio *folio, - struct vm_area_struct *vma) +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static bool __wp_can_reuse_large_anon_folio(struct folio *folio, + struct vm_area_struct *vma) { + bool exclusive = false; + + /* Let's just free up a large folio if only a single page is mapped. */ + if (folio_large_mapcount(folio) <= 1) + return false; + /* - * We could currently only reuse a subpage of a large folio if no - * other subpages of the large folios are still mapped. However, - * let's just consistently not reuse subpages even if we could - * reuse in that scenario, and give back a large folio a bit - * sooner. + * The assumption for anonymous folios is that each page can only get + * mapped once into each MM. The only exception are KSM folios, which + * are always small. + * + * Each taken mapcount must be paired with exactly one taken reference, + * whereby the refcount must be incremented before the mapcount when + * mapping a page, and the refcount must be decremented after the + * mapcount when unmapping a page. + * + * If all folio references are from mappings, and all mappings are in + * the page tables of this MM, then this folio is exclusive to this MM. */ - if (folio_test_large(folio)) + if (folio_test_large_maybe_mapped_shared(folio)) + return false; + + VM_WARN_ON_ONCE(folio_test_ksm(folio)); + VM_WARN_ON_ONCE(folio_mapcount(folio) > folio_nr_pages(folio)); + VM_WARN_ON_ONCE(folio_entire_mapcount(folio)); + + if (unlikely(folio_test_swapcache(folio))) { + /* + * Note: freeing up the swapcache will fail if some PTEs are + * still swap entries. + */ + if (!folio_trylock(folio)) + return false; + folio_free_swap(folio); + folio_unlock(folio); + } + + if (folio_large_mapcount(folio) != folio_ref_count(folio)) return false; + /* Stabilize the mapcount vs. refcount and recheck. */ + folio_lock_large_mapcount(folio); + VM_WARN_ON_ONCE(folio_large_mapcount(folio) < folio_ref_count(folio)); + + if (folio_test_large_maybe_mapped_shared(folio)) + goto unlock; + if (folio_large_mapcount(folio) != folio_ref_count(folio)) + goto unlock; + + VM_WARN_ON_ONCE(folio_mm_id(folio, 0) != vma->vm_mm->mm_id && + folio_mm_id(folio, 1) != vma->vm_mm->mm_id); + + /* + * Do we need the folio lock? Likely not. If there would have been + * references from page migration/swapout, we would have detected + * an additional folio reference and never ended up here. + */ + exclusive = true; +unlock: + folio_unlock_large_mapcount(folio); + return exclusive; +} +#else /* !CONFIG_TRANSPARENT_HUGEPAGE */ +static bool __wp_can_reuse_large_anon_folio(struct folio *folio, + struct vm_area_struct *vma) +{ + BUILD_BUG(); +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + +static bool wp_can_reuse_anon_folio(struct folio *folio, + struct vm_area_struct *vma) +{ + if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && folio_test_large(folio)) + return __wp_can_reuse_large_anon_folio(folio, vma); + /* * We have to verify under folio lock: these early checks are * just an optimization to avoid locking the folio and freeing From patchwork Mon Mar 3 16:30:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BA81C282CD for ; Mon, 3 Mar 2025 16:31:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7AFE728000D; Mon, 3 Mar 2025 11:31:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 76042280003; Mon, 3 Mar 2025 11:31:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4068628000D; Mon, 3 Mar 2025 11:31:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0293528000C for ; Mon, 3 Mar 2025 11:31:01 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BC690160CE2 for ; Mon, 3 Mar 2025 16:31:01 +0000 (UTC) X-FDA: 83180779122.15.D6FFC24 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 9ADA780008 for ; Mon, 3 Mar 2025 16:30:58 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=c8YFcinq; spf=pass (imf30.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019458; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JxmqLp4raSjE/u9Ir1kxxBRe4WYvModjbml+BLkpapQ=; b=ASjrqeRxoxeZR3Lqip0ZUsA1p0O4dBFtM9kzbwuH2Amw8SHZMZjCh8b7JEMGZJXwKOgjIH 0gDaufTHxllFRizz/98WTDv4rywpuu0kzCnlnkYjsdG5SlCeG0Yzuo8hpm3UKMDg8bQbK3 nXQ0NUNIwjgkiqIXzJonbMUfxf7GSjA= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=c8YFcinq; spf=pass (imf30.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019458; a=rsa-sha256; cv=none; b=LZFDtCuHK9h5Ml6MJOeK4L7eYToTx9XQVqUD7QSrJxbJaRhM5LGquHYB2WXFQqlpw419ZI uEH7OlhHdM9MPNC0OXjaiR/1WC0e1x1VNwI2xzNNyKfwg6MpOQesary5lhyUgqYctzka9c SWo3Ll9/1KOu6URjifUUYY9S30gDwAs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019457; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JxmqLp4raSjE/u9Ir1kxxBRe4WYvModjbml+BLkpapQ=; b=c8YFcinq5JTGhtuHql07H6cZgUelAXG0CEz76Y9KOvvKV6n9am4JRaJlbUNSNOyuaVS8I2 su7YXKoz8l9C90/MejZuv3dJC8z2UPJf7x9Te3SnscCefhb3Id2PAZkEtLffEknxpMH2xv vQI6EGlwE2FvHsNehrCeii7HX4aFitc= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-332-69QMp396M428x0PAbM6ZxA-1; Mon, 03 Mar 2025 11:30:51 -0500 X-MC-Unique: 69QMp396M428x0PAbM6ZxA-1 X-Mimecast-MFC-AGG-ID: 69QMp396M428x0PAbM6ZxA_1741019450 Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-43bbfc1681eso6628935e9.0 for ; Mon, 03 Mar 2025 08:30:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019450; x=1741624250; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JxmqLp4raSjE/u9Ir1kxxBRe4WYvModjbml+BLkpapQ=; b=ScAfWGcgxWBDoNfdYMzi85YGVRj0+CsOdoQwAOcAh/HDwzXPQGf885oY47w2HZ951H HoQXyVfzVtk5mLpcAMVlR/VXxrt77aoJZWuKeVshiTQuSlEsZ2jsDI5kiYHNu2OeP8SW pK5/K7e2YbQAgqSsHcYiKxr7tx6bKXatlz/AfbEpqntGSUwFXIj+We6BRqNuMVPzL1wm zQhJriIqQbIYRhKmQ0UBRk/ZQGdDVtgGy2aZMqPNQ18RVkMQzCxcki3itcMoHQdqtR2L MFpGtGm1CWzKg6bEbWB4arc/Rel/rUAyPQwpyEktny5g2n7ruptpryphPqU1uii2fIwN AcKw== X-Forwarded-Encrypted: i=1; AJvYcCUybGEGlpLqZ+txTlOxXOggFEpXoqRBQUs0tSq3L95Dfm+uvl7xN1pmBX5JeVglco+gBOZpNaWdgw==@kvack.org X-Gm-Message-State: AOJu0YwilnnRgcCQS3IHUtnKqlgNyEs46N89SA03jVtbcXmLN+d0u+Oh Eq7nJSBjBhelejwfsnyJHroK42xkLlXKf7Gvxizbnh0r8URaS3emr0yERUBZRot1gACHXwrLm8I FXFEtcEVcOW81Ibut/9K0u/KT69EbQ3TqIp40GdxqKtEwGRri X-Gm-Gg: ASbGncssmovLYe2bUS5lar8tRMQeVvcOA0g5E7bAaGW3/SyZk7OpQrja/cPfhm1eRo8 QDBFz7r5HxD+iJjSYJfY9z2QifOVshWJ/O36v8q1oPWV2B4BnjO7pqYPR4+4ZHA+UmnXk+d6ojq 8tEobnhROVr5vEQUaarDmO6CmJWmfYtf3pwEbmkzgjrkwtn9A9faNtvn6IBpSP6E9MgJFoWj1D7 04/tRcXf2LZvzEtKwmeDPC1c3KFTXvim1f714fv4+5kw30Flqj03K8AKWHpo9Qg3GwDdflAoJaD v74+92jCnOTDJZnuD+OdLpAKMJiNaFlV+WW7wZ4+Z4MwTXUwO8iEzD9Yd4x5cdZXNLYSD8AeBP6 / X-Received: by 2002:a05:600c:3584:b0:43b:c541:51d3 with SMTP id 5b1f17b1804b1-43bc541542fmr24517775e9.6.1741019449772; Mon, 03 Mar 2025 08:30:49 -0800 (PST) X-Google-Smtp-Source: AGHT+IGVn/HGhyotsr/snfJTLhgk+vvrfF/OlvUq6vlbvbEOG4XZJVEXeDZKaxpl01mM+yROzC4hsQ== X-Received: by 2002:a05:600c:3584:b0:43b:c541:51d3 with SMTP id 5b1f17b1804b1-43bc541542fmr24517095e9.6.1741019449189; Mon, 03 Mar 2025 08:30:49 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43bc2828afesm38792575e9.9.2025.03.03.08.30.47 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:48 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 14/20] mm: convert folio_likely_mapped_shared() to folio_maybe_mapped_shared() Date: Mon, 3 Mar 2025 17:30:07 +0100 Message-ID: <20250303163014.1128035-15-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 6wk_gYjIh8ui_Yk1ngan9mlf-YFHGwBtzhkOE6V3DQQ_1741019450 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Queue-Id: 9ADA780008 X-Rspamd-Server: rspam09 X-Stat-Signature: d5nw3hj55roi1b1omtpciagb1opog47f X-HE-Tag: 1741019458-485245 X-HE-Meta: U2FsdGVkX18dnoOjDDzKoRYBT2c3npHrwHw00pygwehnmDy+HtX+cd/eDfvXnrHBWiaC9OKuCcd2gG+F7dLCGmafWyrgEC3uXjpvabrokPCiJbU6CAnY1I6SjhqaNqoVSG+vJ2nQSP04G3wPbC0WO4Dq8vFXe7GpVFvVdddMouDEjQ6hie9DjakIpHKYHmvfIALCXDkhKc6jyDw+APxxam2ckEfWYZtmbmBsch2BhdSFgH3lEgmjAWCEYNK0IrhzcSlx+DjuFXeZSZw7h86IdRCpO4psr/jfLfaTsI9TL9tdkBzUI53SQ0P25+KfcvqEz5SFJjHRLjVwksmrE2LE5CMNSjfmaUzrbWS7xGy6dqb+sXA5jD08JHlT7aBxif4kPlk00UfOwjbK+mh4UoZLgpgWv0v6HCrDUEiAVmANwWL8xChV1FSOLp5pYpxvfJvEnrav4S81/QcEcjpn8fGBGBBA5LuFA6ntNbHQHimd0NiPZTWKgcXwi2FLyNKjGJVGH4o25MrLtLbbnDEyjb8lSviZw3wVhByz+eZjJzPNLHbioepFcw8soibq8v+x7fICO4PyO8GqFQ/B9uKhgC0BwelDSBkGANPxVS6qPX/2X51SNvbf3Mmk/mo7Kp6mnMHFjNM/OYZmqyYAV7/yWfeqkZ9nOJzsBq/JyoxhKfIh4GP6FdUnaPGXTJwtnE/ddKZN5vuSXqe5a6z5eCVgW9E7S6m904F2hUPnYI36dkHP6gU/Q4ANxAWHJzbwnAz3Lgf8qWvNjPmcHwaIJe8P68+5w8gWH8wjvYmUD62Hso8labcwiWKucVVy9Irkb5yvbH/gcRrpd00/TQMebyuf6JiFthZHoml6rXWqNSo5XuRWv6SiPE1Yb9brl9VijiMOkIBB7mfSAUMcol9RNpiiA5PC9LemuW3tPdyXyU1SYlt7fFmMwXmBqnzBK7rusEgcADOtPs+s/I2veSYy9UPabEY cDoOmeXY LzwPZ1nvCRvWYOwTMfDTPCvp+zK3QjqtTUmHgHzpD8EjIx4TNVsh11+I1KrE/oKvfMrY5nvHUwWOlZ+NlAc2eDDC2AehGLq5GuyiSDGfCD9RoxJtc1b/3/EMMhtDUTD7oQIbhyAn/YM/y8AV9XzP1gy5WGAxvghPBWfv9Zd8ebFQ0z6iHGQn7pWkTTJ/PRjpqgWcB3V9aiNdRRjy8QiAXsAY+4Z8pxBRT28eS9eJgldECIOttbeDa1WSjNIFxZXUa7xpiz+2AQXD2GKSYTstgSOsf/Rw15bZEC5RmOo1RycXPET1I2CFHebxhtPcyUnVOPM6FuSKDVJl/hxbkOx4InPNGxRob/og0shbWklTZrhvel0yv5OZDuY8NvKtww3LJHQyawTbr2vTIKPiODdqs+Mk5jbuI/jVGTHhMccBUs6/ncq6/9FSFiFr5EWi0ah4fPRGXrBTdHj+biQQ+JR1ndXQELXpwtjF0fYOkShqXhUhYMu/vqOiIG1+RrtsINnYhxy4mWtPm/Q+ylHhqDC4q3gWvaQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's reuse our new MM ownership tracking infrastructure for large folios to make folio_likely_mapped_shared() never return false negatives -- never indicating "not mapped shared" although the folio *is* mapped shared. With that, we can rename it to folio_maybe_mapped_shared() and get rid of the dependency on the mapcount of the first folio page. The semantics are now arguably clearer: no mixture of "false negatives" and "false positives", only the remaining possibility for "false positives". Thoroughly document the new semantics. We might now detect that a large folio is "maybe mapped shared" although it *no longer* is -- but once was. Now, if more than two MMs mapped a folio at the same time, and the MM mapping the folio exclusively at the end is not one tracked in the two folio MM slots, we will detect the folio as "maybe mapped shared". For anonymous folios, usually (except weird corner cases) all PTEs that target a "maybe mapped shared" folio are R/O. As soon as a child process would write to them (iow, actively use them), we would CoW and effectively replace these PTEs. Most cases (below) are not expected to really matter with large anonymous folios for this reason. Most importantly, there will be no change at all for: * small folios * hugetlb folios * PMD-mapped PMD-sized THPs (single mapping) This change has the potential to affect existing callers of folio_likely_mapped_shared() -> folio_maybe_mapped_shared(): (1) fs/proc/task_mmu.c: no change (hugetlb) (2) khugepaged counts PTEs that target shared folios towards max_ptes_shared (default: HPAGE_PMD_NR / 2), meaning we could skip a collapse where we would have previously collapsed. This only applies to anonymous folios and is not expected to matter in practice. Worth noting that this change sorts out case (A) documented in commit 1bafe96e89f0 ("mm/khugepaged: replace page_mapcount() check by folio_likely_mapped_shared()") by removing the possibility for "false negatives". (3) MADV_COLD / MADV_PAGEOUT / MADV_FREE will not try splitting PTE-mapped THPs that are considered shared but not fully covered by the requested range, consequently not processing them. PMD-mapped PMD-sized THP are not affected, or when all PTEs are covered. These functions are usually only called on anon/file folios that are exclusively mapped most of the time (no other file mappings or no fork()), so the "false negatives" are not expected to matter in practice. (4) mbind() / migrate_pages() / move_pages() will refuse to migrate shared folios unless MPOL_MF_MOVE_ALL is effective (requires CAP_SYS_NICE). We will now reject some folios that could be migrated. Similar to (3), especially with MPOL_MF_MOVE_ALL, so this is not expected to matter in practice. Note that cpuset_migrate_mm_workfn() calls do_migrate_pages() with MPOL_MF_MOVE_ALL. (5) NUMA hinting mm/migrate.c:migrate_misplaced_folio_prepare() will skip file folios that are probably shared libraries (-> "mapped shared" and executable). This check would have detected it as a shared library at some point (at least 3 MMs mapping it), so detecting it afterwards does not sound wrong (still a shared library). Not expected to matter. mm/memory.c:numa_migrate_check() will indicate TNF_SHARED in MAP_SHARED file mappings when encountering a shared folio. Similar reasoning, not expected to matter. mm/mprotect.c:change_pte_range() will skip folios detected as shared in CoW mappings. Similarly, this is not expected to matter in practice, but if it would ever be a problem we could relax that check a bit (e.g., basing it on the average page-mapcount in a folio), because it was only an optimization when many (e.g., 288) processes were mapping the same folios -- see commit 859d4adc3415 ("mm: numa: do not trap faults on shared data section pages.") (6) mm/rmap.c:folio_referenced_one() will skip exclusive swapbacked folios in dying processes. Applies to anonymous folios only. Without "false negatives", we'll now skip all actually shared ones. Skipping ones that are actually exclusive won't really matter, it's a pure optimization, and is not expected to matter in practice. In theory, one can detect the problematic scenario: folio_mapcount() > 0 and no folio MM slot is occupied ("state unknown"). One could reset the MM slots while doing an rmap walk, which migration / folio split already do when setting everything up. Further, when batching PTEs we might naturally learn about a owner (e.g., folio_mapcount() == nr_ptes) and could update the owner. However, we'll defer that until the scenarios where it would really matter are clear. Signed-off-by: David Hildenbrand --- fs/proc/task_mmu.c | 4 ++-- include/linux/mm.h | 43 ++++++++++++++++++++++--------------------- mm/huge_memory.c | 2 +- mm/khugepaged.c | 8 +++----- mm/madvise.c | 6 +++--- mm/memory.c | 2 +- mm/mempolicy.c | 8 ++++---- mm/migrate.c | 7 +++---- mm/mprotect.c | 2 +- mm/rmap.c | 2 +- 10 files changed, 41 insertions(+), 43 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index c17615e21a5d6..1162f0e72df2e 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1023,7 +1023,7 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, if (folio) { /* We treat non-present entries as "maybe shared". */ - if (!present || folio_likely_mapped_shared(folio) || + if (!present || folio_maybe_mapped_shared(folio) || hugetlb_pmd_shared(pte)) mss->shared_hugetlb += huge_page_size(hstate_vma(vma)); else @@ -1882,7 +1882,7 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask, if (!folio_test_anon(folio)) flags |= PM_FILE; - if (!folio_likely_mapped_shared(folio) && + if (!folio_maybe_mapped_shared(folio) && !hugetlb_pmd_shared(ptep)) flags |= PM_MMAP_EXCLUSIVE; diff --git a/include/linux/mm.h b/include/linux/mm.h index 53dd4f99fdabc..a4f2c56fcf524 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2245,23 +2245,18 @@ static inline size_t folio_size(const struct folio *folio) } /** - * folio_likely_mapped_shared - Estimate if the folio is mapped into the page - * tables of more than one MM + * folio_maybe_mapped_shared - Whether the folio is mapped into the page + * tables of more than one MM * @folio: The folio. * - * This function checks if the folio is currently mapped into more than one - * MM ("mapped shared"), or if the folio is only mapped into a single MM - * ("mapped exclusively"). + * This function checks if the folio maybe currently mapped into more than one + * MM ("maybe mapped shared"), or if the folio is certainly mapped into a single + * MM ("mapped exclusively"). * * For KSM folios, this function also returns "mapped shared" when a folio is * mapped multiple times into the same MM, because the individual page mappings * are independent. * - * As precise information is not easily available for all folios, this function - * estimates the number of MMs ("sharers") that are currently mapping a folio - * using the number of times the first page of the folio is currently mapped - * into page tables. - * * For small anonymous folios and anonymous hugetlb folios, the return * value will be exactly correct: non-KSM folios can only be mapped at most once * into an MM, and they cannot be partially mapped. KSM folios are @@ -2269,8 +2264,8 @@ static inline size_t folio_size(const struct folio *folio) * * For other folios, the result can be fuzzy: * #. For partially-mappable large folios (THP), the return value can wrongly - * indicate "mapped exclusively" (false negative) when the folio is - * only partially mapped into at least one MM. + * indicate "mapped shared" (false positive) if a folio was mapped by + * more than two MMs at one point in time. * #. For pagecache folios (including hugetlb), the return value can wrongly * indicate "mapped shared" (false positive) when two VMAs in the same MM * cover the same file range. @@ -2287,7 +2282,7 @@ static inline size_t folio_size(const struct folio *folio) * * Return: Whether the folio is estimated to be mapped into more than one MM. */ -static inline bool folio_likely_mapped_shared(struct folio *folio) +static inline bool folio_maybe_mapped_shared(struct folio *folio) { int mapcount = folio_mapcount(folio); @@ -2295,16 +2290,22 @@ static inline bool folio_likely_mapped_shared(struct folio *folio) if (!folio_test_large(folio) || unlikely(folio_test_hugetlb(folio))) return mapcount > 1; - /* A single mapping implies "mapped exclusively". */ - if (mapcount <= 1) - return false; - - /* If any page is mapped more than once we treat it "mapped shared". */ - if (folio_entire_mapcount(folio) || mapcount > folio_nr_pages(folio)) + /* + * vm_insert_page() without CONFIG_TRANSPARENT_HUGEPAGE ... + * simply assume "mapped shared", nobody should really care + * about this for arbitrary kernel allocations. + */ + if (!IS_ENABLED(CONFIG_MM_ID)) return true; - /* Let's guess based on the first subpage. */ - return atomic_read(&folio->_mapcount) > 0; + /* + * A single mapping implies "mapped exclusively", even if the + * folio flag says something different: it's easier to handle this + * case here instead of on the RMAP hot path. + */ + if (mapcount <= 1) + return false; + return folio_test_large_maybe_mapped_shared(folio); } #ifndef HAVE_ARCH_MAKE_FOLIO_ACCESSIBLE diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 8e8b07e8b12fe..826bfe907017f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2155,7 +2155,7 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, * If other processes are mapping this folio, we couldn't discard * the folio unless they all do MADV_FREE so let's skip the folio. */ - if (folio_likely_mapped_shared(folio)) + if (folio_maybe_mapped_shared(folio)) goto out; if (!folio_trylock(folio)) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 5f0be134141e8..cc945c6ab3bdb 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -607,7 +607,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); /* See hpage_collapse_scan_pmd(). */ - if (folio_likely_mapped_shared(folio)) { + if (folio_maybe_mapped_shared(folio)) { ++shared; if (cc->is_khugepaged && shared > khugepaged_max_ptes_shared) { @@ -1359,11 +1359,9 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, /* * We treat a single page as shared if any part of the THP - * is shared. "False negatives" from - * folio_likely_mapped_shared() are not expected to matter - * much in practice. + * is shared. */ - if (folio_likely_mapped_shared(folio)) { + if (folio_maybe_mapped_shared(folio)) { ++shared; if (cc->is_khugepaged && shared > khugepaged_max_ptes_shared) { diff --git a/mm/madvise.c b/mm/madvise.c index e01e93e179a8a..388dc289b5d12 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -387,7 +387,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, folio = pmd_folio(orig_pmd); /* Do not interfere with other mappings of this folio */ - if (folio_likely_mapped_shared(folio)) + if (folio_maybe_mapped_shared(folio)) goto huge_unlock; if (pageout_anon_only_filter && !folio_test_anon(folio)) @@ -486,7 +486,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, if (nr < folio_nr_pages(folio)) { int err; - if (folio_likely_mapped_shared(folio)) + if (folio_maybe_mapped_shared(folio)) continue; if (pageout_anon_only_filter && !folio_test_anon(folio)) continue; @@ -721,7 +721,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, if (nr < folio_nr_pages(folio)) { int err; - if (folio_likely_mapped_shared(folio)) + if (folio_maybe_mapped_shared(folio)) continue; if (!folio_trylock(folio)) continue; diff --git a/mm/memory.c b/mm/memory.c index bb245a8fe04bc..a838c8c44bfdc 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5700,7 +5700,7 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, * Flag if the folio is shared between multiple address spaces. This * is later used when determining whether to group tasks together */ - if (folio_likely_mapped_shared(folio) && (vma->vm_flags & VM_SHARED)) + if (folio_maybe_mapped_shared(folio) && (vma->vm_flags & VM_SHARED)) *flags |= TNF_SHARED; /* * For memory tiering mode, cpupid of slow memory page is used diff --git a/mm/mempolicy.c b/mm/mempolicy.c index bbaadbeeb2919..530e71fe91476 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -642,11 +642,11 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask, * Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio. * Choosing not to migrate a shared folio is not counted as a failure. * - * See folio_likely_mapped_shared() on possible imprecision when we + * See folio_maybe_mapped_shared() on possible imprecision when we * cannot easily detect if a folio is shared. */ if ((flags & MPOL_MF_MOVE_ALL) || - (!folio_likely_mapped_shared(folio) && !hugetlb_pmd_shared(pte))) + (!folio_maybe_mapped_shared(folio) && !hugetlb_pmd_shared(pte))) if (!folio_isolate_hugetlb(folio, qp->pagelist)) qp->nr_failed++; unlock: @@ -1033,10 +1033,10 @@ static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist, * Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio. * Choosing not to migrate a shared folio is not counted as a failure. * - * See folio_likely_mapped_shared() on possible imprecision when we + * See folio_maybe_mapped_shared() on possible imprecision when we * cannot easily detect if a folio is shared. */ - if ((flags & MPOL_MF_MOVE_ALL) || !folio_likely_mapped_shared(folio)) { + if ((flags & MPOL_MF_MOVE_ALL) || !folio_maybe_mapped_shared(folio)) { if (folio_isolate_lru(folio)) { list_add_tail(&folio->lru, foliolist); node_stat_mod_folio(folio, diff --git a/mm/migrate.c b/mm/migrate.c index 365c6daa8d1b1..fb4afd31baf0c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2228,7 +2228,7 @@ static int __add_folio_for_migration(struct folio *folio, int node, if (folio_nid(folio) == node) return 0; - if (folio_likely_mapped_shared(folio) && !migrate_all) + if (folio_maybe_mapped_shared(folio) && !migrate_all) return -EACCES; if (folio_test_hugetlb(folio)) { @@ -2653,11 +2653,10 @@ int migrate_misplaced_folio_prepare(struct folio *folio, * processes with execute permissions as they are probably * shared libraries. * - * See folio_likely_mapped_shared() on possible imprecision + * See folio_maybe_mapped_shared() on possible imprecision * when we cannot easily detect if a folio is shared. */ - if ((vma->vm_flags & VM_EXEC) && - folio_likely_mapped_shared(folio)) + if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio)) return -EACCES; /* diff --git a/mm/mprotect.c b/mm/mprotect.c index 1444878f7aeb2..62c1f79457412 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -133,7 +133,7 @@ static long change_pte_range(struct mmu_gather *tlb, /* Also skip shared copy-on-write pages */ if (is_cow_mapping(vma->vm_flags) && (folio_maybe_dma_pinned(folio) || - folio_likely_mapped_shared(folio))) + folio_maybe_mapped_shared(folio))) continue; /* diff --git a/mm/rmap.c b/mm/rmap.c index c9922928616ee..8de415157bc8d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -889,7 +889,7 @@ static bool folio_referenced_one(struct folio *folio, if ((!atomic_read(&vma->vm_mm->mm_users) || check_stable_address_space(vma->vm_mm)) && folio_test_anon(folio) && folio_test_swapbacked(folio) && - !folio_likely_mapped_shared(folio)) { + !folio_maybe_mapped_shared(folio)) { pra->referenced = -1; page_vma_mapped_walk_done(&pvmw); return false; From patchwork Mon Mar 3 16:30:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 873BAC282CD for ; Mon, 3 Mar 2025 16:31:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 16EC028000B; Mon, 3 Mar 2025 11:31:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0A787280003; Mon, 3 Mar 2025 11:31:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC50228000C; Mon, 3 Mar 2025 11:30:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AE0DC280003 for ; Mon, 3 Mar 2025 11:30:59 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 91D08140C86 for ; Mon, 3 Mar 2025 16:30:58 +0000 (UTC) X-FDA: 83180778996.03.BD3A727 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 0526D18001C for ; Mon, 3 Mar 2025 16:30:54 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=E2Z+QIc6; spf=pass (imf06.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019455; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=azxC6HZJ9nBU2F+f5yWV4gQ6J7sT6YtNWxk4WN8FkrY=; b=5Tzdk2Qk67vklvhi2UW4QTAjzh7zs6CgCkF5qjmEWFn2SgTXb3gg2+fdRqgYMpEaaogmkF ilioALoVEgbzGNubDSJY91oE/7LAI8k5Vga7pa6U0PbCM0KL1PJzIAkoK+7xBY35+eNC3c NrDBfNsWV/e9hf1NHCklkfDCN9gxiVg= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=E2Z+QIc6; spf=pass (imf06.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019455; a=rsa-sha256; cv=none; b=RPrl+RuSZT9fgjK1+f5Bj34bEtLDccutllP78A1xj9vYnV7mTykcnOh43DUDWi490bhCGx vCdO1So06vuL2F91fS4tee+8yI5Ccv0XuzO7+GXXO+imAwRrK4MaYyU+yOJ3UmHPsA28Ap 8BxjojV+HBVG5OOhVWS9td/WdIhyOnE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019454; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=azxC6HZJ9nBU2F+f5yWV4gQ6J7sT6YtNWxk4WN8FkrY=; b=E2Z+QIc6iE7/F70LZfyWEMCMNsNVEneVMZtyoPc54+wrHwiaJsznKddaAYubWNkOH1pjmy i57Fs9IQd/afDG9L3976LzrwWM8XEswfM9EY6rTHHv6PtQoghxPym1z3daA2qHZufmMOJ3 0yKrpGFy6q38KoVX+ydeM08fv1+zJUI= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-318-1Bl9HJ0mMye2GzYBWasnqw-1; Mon, 03 Mar 2025 11:30:53 -0500 X-MC-Unique: 1Bl9HJ0mMye2GzYBWasnqw-1 X-Mimecast-MFC-AGG-ID: 1Bl9HJ0mMye2GzYBWasnqw_1741019452 Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-4399d2a1331so22759165e9.1 for ; Mon, 03 Mar 2025 08:30:52 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019452; x=1741624252; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=azxC6HZJ9nBU2F+f5yWV4gQ6J7sT6YtNWxk4WN8FkrY=; b=DwQ7jPKxcOqG1bbr94Aicz4IYkHBBQh5JZMAhqtFf3F4HZWpVKAh6T/BoPhx7i+vP2 xoSZMfCuS2mVDBNQccTb1rFafIqUqXmszIj0I7BhzRU2BEFlK0FRt6kQ7b1dMwg8cm3L lrwDU+SmxEyI5S1Sc+aI0SFqPz3crjNFCp2ygq3zQxQbCKz0cNfWV42bzaVTbwhpws9g QvD+a9ROtaEArvQzravRyuE95QDuT7r6O9g6otJoZDMea8i+n8rVOCXKu1SAytJ2IKGG cO8+RR/AJDDfbMgfe6DxPpdS4UFhIogOo0GEWod4gb2tBsrE5y/I2i8Y+/iTB3VxkhIc 3mFw== X-Forwarded-Encrypted: i=1; AJvYcCVRjsZEB7iLnM7QZj1c+4p3vSTL1ywRL/r3QXVsQHk9u8zfbcqNjSBzJUTgUqvpV0yYO2JKjo4VqA==@kvack.org X-Gm-Message-State: AOJu0Yx+QJmgmXIuk2JoGO7q+J20QfdmmG3+0mhbqH+YnFslbjN6tTCi MFkMO5vnHj0LM/dnriX4jiDWD/+fL4bnJLD2XiAx62DqUcIliO1Tx04tZaqFu37zw3CVwf+BMvK Ec98443HNwjNtpJ3RqU+E5F+Womkgr51N2QHNuBAGfUEfS/NO X-Gm-Gg: ASbGncsELKtAehilL5Vac/C29i8yHKj35iZl0gaIKVLcFNJCVlQFfTjb5Ry4koX9V6K 8bbG9Gm7atuuuiRyF64csg7wLT+c85lLDGHLJtJdUjSZfICGSIefiHZPPhiFfs7itp6SK/X6mq2 U8EOxp2seOUtrehHziX0O3FvT1Cv31fJTD/1Wv7+mV9SRYg84m3wKoabxDh5lk8qwW/xCGqoEay JW4+yiPPQWzh8LfA5kEhd6RKWDUaxBDRvotAJFr+3J+0/IEfA+bViRrC7JcLYfaX9rH3MIanHfT OFkDWAOzYvZzcH7ZmgIY61QQfgQaILJdFrdoyLtzdOB7h/PrOrY2YIdCKVt4LrELCLm1xZ7vSAy 9 X-Received: by 2002:a05:600c:5585:b0:43b:bedc:fcf6 with SMTP id 5b1f17b1804b1-43bbedcfe47mr44358365e9.1.1741019451744; Mon, 03 Mar 2025 08:30:51 -0800 (PST) X-Google-Smtp-Source: AGHT+IFzQb6PWhSpsnKcZbPTy2ai+36EkVbTPhZ8phSQIWx5xoPi2ki+HFSKpKKxfbHb4k7pKztumg== X-Received: by 2002:a05:600c:5585:b0:43b:bedc:fcf6 with SMTP id 5b1f17b1804b1-43bbedcfe47mr44358005e9.1.1741019451351; Mon, 03 Mar 2025 08:30:51 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43b736f74e8sm172202265e9.7.2025.03.03.08.30.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:50 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 15/20] mm: CONFIG_NO_PAGE_MAPCOUNT to prepare for not maintain per-page mapcounts in large folios Date: Mon, 3 Mar 2025 17:30:08 +0100 Message-ID: <20250303163014.1128035-16-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: B_mwKFnDdfk0qgUYwXt90A1LlVfcRc3ut5sFcF89cz4_1741019452 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Stat-Signature: qrp3wfsh7h6r7dkpryrsjadgutcr5k9w X-Rspamd-Queue-Id: 0526D18001C X-Rspamd-Server: rspam07 X-HE-Tag: 1741019454-672734 X-HE-Meta: U2FsdGVkX18UxX2zINWBOy6/lq/5Qw6GyPIKQOaUvQYO6DG2p14QHt2ewHyRp8QbcuHPOzIOvSQYKClL98+7u5PlJuhoRGkuVfZxO2vHW77q+SjV0cFp3baBZx2wovGOdcllOaPKG+vrRB1rsggpg2mf3ybZqR5gakcLqc5HpAr7ygn3bh1bwgpwZ1jXNdV8VC+PZnMlwdvJ4Q+7BRp8d0J97vAvil2NPrR5Fpsevsz6p7t5j50smoSnQxnNMfofTQyeGdRCh4R3GlUZE5ZFvOUki0waTremnrGMr/1qj8mSPJ9ehuMdaZIiSGQ5hL1tDhkU2BpemzaAn3uaW2WB+C0hlJ1L8fj0I0VB3JVC8+1oR3hZOOZbpJwGA+Rowd3UiunTK6Eqq4RhiHF60UmKpZa89RCZcJRYA8trgUHBV/GGGi7fIIfY4WCo0u/uYFC9PZFFStrmsbvTA0o1cUVgpCuDQ1Sxb7bMzHWjvhSMbF0G1RX1mqUoqQqFq6Ih7IJS73ovYXoGIFm+AGSZHGGqeTHDSk8/eudrVx3ChYeoP/d1gsxlrdMpcB0Spl0ptAT1e1O367Xv0Wv6T/Kj9n4PjsttCQ1V+8n8w7N96ACN7/t9OVnv1Z/svQJOBW4/XElgycDKrpoTtdr3JrPIUoskN7HVuhn/ufY6xNjMlZU0hI9wzFtPaGwA1IsJLf6CnjcAI+dRqnM0hCICDMrjlPjk9YPSPoJLHm3Cfukg7mMden+qylXIWg0evkBJu3h89EC5oiqmEiiB3JRpPBvQAQ4yuz/F6rG1JmpIZL0NfJTeCRGpvc1/l8VNxelvx5YPksZmmbsIA0g2SFLJAPZC0CTrMfq2603C5FMNolz1G1fLgiYpmjlPxsvheJeXO4n5Mg+9KfP7Sqpevt9MZXHj5oTjBfBP3oSKZeC98EzauFNSt/xOODBOU+2PoT8kOYVPpbGXr+QC1xTOI4T4/W4KKM1 Ot0Y2Gki oYXJ1WjZ4LzufY5UlTSkionrxWpXva42YN+Zf218pm87kGH5pASpQdwWo8xhvDJquPzspamNi23a9BYknP5Ff+WrA0PlOg2afiXmePTVdqmpBK6uTymY/MVpzGLU5/JJWBx9caDWTMkFsqE4KimfK9L9WXdYZ2UGPlEodc06t1xfklEoHIwn34kJpvPLOhFBQPl7m3jiplQUZqyu8xdEUz26mRTaCIZDQOxFtzV2D5vDWHAQpLy9Svw5JDxYSqz0+JryTIEa3d2OFkt1i1UVnPKN4H2jNAArydRhZ+oqSRpOxsVrggdubqRKTMFf2qgF3QrarQIeojDXMuUuJcf3deDwkuvFoj3l45nWRwGLOxDqkIk2pvyIM8DazIeT+P1LsDcYAwR+3zIkQNfgbYXwd7OZe8xpqcX/TKb2N/Ri5Rqek1vFY62jPoBZN9yGvtqtuqMTWDOmZ5Ne4b2qOzDLKW4y1SgYQpxdVwlVA482n29fhU9DYfoq/mKEDfkyYCiFEWdaVUG1cyhGwenbeV6v/IowjVw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We're close to the finishing line: let's introduce a new CONFIG_NO_PAGE_MAPCOUNT config option where we will incrementally remove any dependencies on per-page mapcounts in large folios. Once that's done, we'll stop maintaining the per-page mapcounts with this config option enabled. CONFIG_NO_PAGE_MAPCOUNT will be EXPERIMENTAL for now, as we'll have to learn about some of the real world impact of some of the implications. As writing "!CONFIG_NO_PAGE_MAPCOUNT" is really nasty, let's introduce a helper config option "CONFIG_PAGE_MAPCOUNT" that expresses the negation. Signed-off-by: David Hildenbrand --- mm/Kconfig | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 4034a0441f650..e4bdcf11d1b86 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -881,8 +881,25 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles. +config NO_PAGE_MAPCOUNT + bool "No per-page mapcount (EXPERIMENTAL)" + help + Do not maintain per-page mapcounts for pages part of larger + allocations, such as transparent huge pages. + + When this config option is enabled, some interfaces that relied on + this information will rely on less-precise per-allocation information + instead: for example, using the average per-page mapcount in such + a large allocation instead of the per-page mapcount. + + EXPERIMENTAL because the impact of some changes is still unclear. + endif # TRANSPARENT_HUGEPAGE +# simple helper to make the code a bit easier to read +config PAGE_MAPCOUNT + def_bool !NO_PAGE_MAPCOUNT + # # The architecture supports pgtable leaves that is larger than PAGE_SIZE # From patchwork Mon Mar 3 16:30:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32C81C282CD for ; Mon, 3 Mar 2025 16:31:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40D6428000E; Mon, 3 Mar 2025 11:31:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 34765280003; Mon, 3 Mar 2025 11:31:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1996428000D; Mon, 3 Mar 2025 11:31:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EE3AC280003 for ; Mon, 3 Mar 2025 11:31:01 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7B624120C5F for ; Mon, 3 Mar 2025 16:31:01 +0000 (UTC) X-FDA: 83180779122.21.9B13AA6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 726F8140023 for ; Mon, 3 Mar 2025 16:30:58 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AYQ3OBYV; spf=pass (imf26.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019458; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dM3d7FaE0s/3W03pV+cOCeas1Rcyk2uf0DwTQW3h1Ys=; b=WBpU31nvMtiBQdkyhpEX0ThnbPU8zwtIPFDmAx9STxW1rfXz1O41PbELNqg6Ye6cS0VHur ug4YxMgLRyE0dXdJCnVq2EQXdudC9t4EZvUmKaAdhdJX/nNuU7BDJWBetf43QOJy7qhsv7 KnYVPguhRX/tN31VcmwcS95M2Pv7hwg= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AYQ3OBYV; spf=pass (imf26.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019458; a=rsa-sha256; cv=none; b=Q3Mdo9P+7IOSGzMtdQO3W1lHJUnnCoFjFOQu7y+1Hmz1X/XM5dyidu5DJpc3lN9UPeunTT jv+vndsj82k/ON+CPqxIjDenb7dOz4MrkpFYqAmnvDcJy8AobL8psbpPfxsYgZy7Dv+8no Q8wXRX+Hc1eEICaksvio++NTdX9TCJI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019457; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dM3d7FaE0s/3W03pV+cOCeas1Rcyk2uf0DwTQW3h1Ys=; b=AYQ3OBYVp3u4GgUydN25A1a75DADqy0p5DM3eo0rwFcmjo2Bj1XbLplpEFCIBaiZfTl9No ASYp8p0qzqYloQFBPPiQjP5K9iGGw95iTD65BGTKTfH5P3zzbHzuRt+QiSFdocNjO0xoDt T9QsA/Yz0sHlUqczrxKDbNpTgwzkSgg= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-139-OGf2ypZIMvKIt_8wgT5yPw-1; Mon, 03 Mar 2025 11:30:55 -0500 X-MC-Unique: OGf2ypZIMvKIt_8wgT5yPw-1 X-Mimecast-MFC-AGG-ID: OGf2ypZIMvKIt_8wgT5yPw_1741019454 Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-390f7db84faso1754494f8f.2 for ; Mon, 03 Mar 2025 08:30:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019454; x=1741624254; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dM3d7FaE0s/3W03pV+cOCeas1Rcyk2uf0DwTQW3h1Ys=; b=k2EIx5FuBfqkGdWDbk9v3AxlatUySq7iZVf4muC2rKnWX1yx+y8GcfKoVd9JtoDb3D Uk09nkWwcUxQY0IRwdq4xFO4UAqAOdxgB5muKNv+zQkWpYdQwjhKYyuPHx8dH7j0fZK2 OwsWRJU0WKNXoWDHxYtZ7DCARrRi3HuVmnJSnEaRTXl0PxkKhKFQr+1SRPxwT1ttrpCg iqJuF5bdaopppN88lHY52BUKerSuQdRT5S17Ji6xvBjbW3R/KnHBPseHp9zzbf7Cmsyy NSAQWqvQso6BxCVpMUW0XztDwFI2CTTqg1Q5+BWKFkAGykuZjV0arzeoPJSE/tjBYmO7 cjUw== X-Forwarded-Encrypted: i=1; AJvYcCVc3rM3I5CfbL8wX4S+A0SIlUgJLAib960l0vkO7ZSKEyyGe8+P/BrfFbC4SPA6DvBnsVAvRD4pNQ==@kvack.org X-Gm-Message-State: AOJu0YzQxsGBhmfeQqVd/qC4g3bAYl1jY/3BFtowULKsZl372UUUHoMQ ZptLUmVswXKlADrTE6nlQc1Nz7weV7PrJiOqEng3CuN8iXYpCchu0hk4COsDvHQpef3CgwxdUFp f9Jb8QjRVkCJWK/wnyLxv17g6ff1XT5zKqowrOkpJYyvg49V5X11LCVB+x+k= X-Gm-Gg: ASbGncsbeYbcUxQn1JlGq7uTx+/W3ZfM7dZCxQ4hKn45eenhNIN7e9dwOAqsfClR9Th M73ZLSyKnzoHI5a8CCtmlEVOUy6ek/17n1oQEtdDLvoQOSmeG9YEieuHCKCvsmdXVGZmvxp5kvJ zs7n6yQYUYGdx9V1EiDQUDPVmwWByxCxdLnssYaaj++umDVwmXpjNuQnMrGIVQf92FcyTLVXgKr z59C5whlEnCaEZvXNiTct3MYvaIkbFPMe3nX0JRD90xhEgYBVNRKDyLnphTqd1T5NyR0EfmoBAa vvsxtbRJYbW0D0H7YNMmdlQdLgqQMXg67xnGirYkGNbGBc0bgqCKCTI1NsYaM7SqS3BaiSJrfx2 c X-Received: by 2002:a05:6000:4c6:b0:38d:c44a:a64b with SMTP id ffacd0b85a97d-390eca2589amr11095192f8f.48.1741019453856; Mon, 03 Mar 2025 08:30:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IFDds/9qj55eCL2sJuyoEBZpLXmN9fljVQo3ZoynXY2Ft4dglS/KTUjLVmC7la2kWf5hOqFtQ== X-Received: by 2002:a05:6000:4c6:b0:38d:c44a:a64b with SMTP id ffacd0b85a97d-390eca2589amr11095139f8f.48.1741019453402; Mon, 03 Mar 2025 08:30:53 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43bbbff18b3sm56885465e9.24.2025.03.03.08.30.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:53 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 16/20] fs/proc/page: remove per-page mapcount dependency for /proc/kpagecount (CONFIG_NO_PAGE_MAPCOUNT) Date: Mon, 3 Mar 2025 17:30:09 +0100 Message-ID: <20250303163014.1128035-17-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 7J8scf22mgA_CAbzyWUQ6D8mtbYpWZeQmD6TOJDmNms_1741019454 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Stat-Signature: 7uga4idtqoz4medoo4bokdws8cgibq5s X-Rspamd-Queue-Id: 726F8140023 X-Rspamd-Server: rspam07 X-HE-Tag: 1741019458-572382 X-HE-Meta: U2FsdGVkX18Z0qp41hfVwMmZF07FM12fdiMkJHlYxFIdNaWUr3hSWk5HkM2pwpcFefwC9fMnTgxbCAPS6R4xv7LT9pCXbAmd168zp6p9p+J5xh0G0lcP8vGsgdD+/vQmXzqYvucwuEcSdjyulzLUqzBAMXvbkm7t8xoV0CiLXIjft+iw66vT8ExUlVqUOGNViRh1OE+tBccZSoRShGXa0v77fwR+JelH8Cgn1TBUBA8P+u4xIsRNc14RpAaaLDHCdf53HOIjE1vA8HL0QD5lKYmpSa30+jGHuAHAUtLrJNyJz1iKGB8WEQ7gS/tI2C98y6k/TQy6GXA5NS1634Ua2ueDAkdq5s+vW86DhHmWCvtYee33aVl8F8gCCo8S5JcyUfh90N1m3clV9ypVvJF4hG1jB1L2mKilB6bIWn1F3qu78lc/PXEq9TI0KlVIwHHh8LiUNcJ7qMU+YVHflqKHPfXYC7Oql7u64K5c5FWcnEkFvYqIq8hmbyNWg4DV6Lg3uFvaeXlh/KuARLPmPX3z+tLjfcUN+bFm6mxiBBMWsT4vNf99tSIpsNP4I8Lf73uJKmX5VwNMn671slup9rJg9uxrgaVxf3lD6bjm34Dyzag7II0WuxW/hV6vcFOyfDXb85C8BhZWgZWIAAzhDbDvbiBphymYGZD6h56uwpcJqhKYNerNsMRRkowsXn10V2jsqdJt7ztXz+9hnG9KAA8B3vF29JNdRP14npKbNFjZR+Qvp7y9riWzkMwHCNqNJp4KiLq5GUTGYfCUWtBnPQuD2u0IyCZF6N5F7IDJRFhswl9ExEB3JRLlUcAZzQGgJrC7TowdoTTlizikTCamHx6RKjx9300iLW3uPgdx4ReO2xw4qfzOKDf7Ts31vXiPxyZGQ57CWcOFB+/oLMIQWkvjoE0kJvVWnFzCf+zM3vTpHd02BnxATF5vexGo4I+pJ411DHKB+8NaIfzK1JJYFxB FCQ1BYLK 0DjE5zJwOjc13H/GvQ4/9M+Ya5g4X++32+6ELyYXRs4TvNbz1rsnJJ+oDClS9abTpd5MNPXVcSvIyhzMT69KCGH5WAmAnfFv9WzsbW8tXDzWfH8bsjRbRPWpCQqqBcXoBML5aL8bdoSibZrY0R0FxbGeeuBwr7Y7IHDH5up1wPYfsC13uR3OyYGcqgQWX/yGLz7ncyQAVXkyaStfP2DWE8+WQ2lcMRNSGQKCKMRafQkK/gKhA1ilJh4i/GgkNARaxFlpQlKFkqEQewudMTX8SZ1n55EkyYYkN5bJkIjEwBipAMcVoLzqnOyrNc3N7SeENSsbYHbO/Me40731tnC0Fse4KXcDUAWkicJ1sMVgt9FwwXGOq1jQHpT0WeL1vts74JW+PU+5eKOSW1QWi4qQBnxjak8S3SqpqymsCzi01jq5kM9tSteIzOnE6kxPfL7aXyv4sQshK2qLeaTY3wNEL6owbjVdEkSE6EUMSGqd9Oz3kw4oRB/FcnSAJl8PccgWIANIqd0q6gReIqJVCUBnvVmQlMg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT. For large folios, we'll return the per-page average mapcount within the folio, whereby we round to the closest integer when calculating the average: however, we'll always return at least 1 if the folio is mapped. So assuming a folio with 512 pages, the average would be: * 0 if not pages are mapped * 1 if there are 1 .. 767 per-page mappings * 2 if there are 767 .. 1279 per-page mappings ... For hugetlb folios and for large folios that are fully mapped into all address spaces, there is no change. We'll make use of this helper in other context next. As an alternative, we could simply return 0 for non-hugetlb large folios, or disable this legacy interface with CONFIG_NO_PAGE_MAPCOUNT. But the information exposed by this interface can still be valuable, and frequently we deal with fully-mapped large folios where the average corresponds to the actual page mapcount. So we'll leave it like this for now and document the new behavior. Note: this interface is likely not very relevant for performance. If ever required, we could try doing a rather expensive rmap walk to collect precisely how often this folio page is mapped. Signed-off-by: David Hildenbrand --- Documentation/admin-guide/mm/pagemap.rst | 7 ++++- fs/proc/internal.h | 35 ++++++++++++++++++++++++ fs/proc/page.c | 11 ++++++-- 3 files changed, 49 insertions(+), 4 deletions(-) diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst index a297e824f9900..d6647daca9122 100644 --- a/Documentation/admin-guide/mm/pagemap.rst +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -43,7 +43,12 @@ There are four components to pagemap: skip over unmapped regions. * ``/proc/kpagecount``. This file contains a 64-bit count of the number of - times each page is mapped, indexed by PFN. + times each page is mapped, indexed by PFN. Some kernel configurations do + not track the precise number of times a page part of a larger allocation + (e.g., THP) is mapped. In these configurations, the average number of + mappings per page in this larger allocation is returned instead. However, + if any page of the large allocation is mapped, the returned value will + be at least 1. The page-types tool in the tools/mm directory can be used to query the number of times a page is mapped. diff --git a/fs/proc/internal.h b/fs/proc/internal.h index 1695509370b88..96ea58e843114 100644 --- a/fs/proc/internal.h +++ b/fs/proc/internal.h @@ -174,6 +174,41 @@ static inline int folio_precise_page_mapcount(struct folio *folio, return mapcount; } +/** + * folio_average_page_mapcount() - Average number of mappings per page in this + * folio + * @folio: The folio. + * + * The average number of user page table entries that reference each page in + * this folio as tracked via the RMAP: either referenced directly (PTE) or + * as part of a larger area that covers this page (e.g., PMD). + * + * The average is calculated by rounding to the nearest integer; however, + * to avoid duplicated code in current callers, the average is at least + * 1 if any page of the folio is mapped. + * + * Returns: The average number of mappings per page in this folio. + */ +static inline int folio_average_page_mapcount(struct folio *folio) +{ + int mapcount, entire_mapcount, avg; + + if (!folio_test_large(folio)) + return atomic_read(&folio->_mapcount) + 1; + + mapcount = folio_large_mapcount(folio); + if (unlikely(mapcount <= 0)) + return 0; + entire_mapcount = folio_entire_mapcount(folio); + if (mapcount <= entire_mapcount) + return entire_mapcount; + mapcount -= entire_mapcount; + + /* Round to closest integer ... */ + avg = ((unsigned int)mapcount + folio_large_nr_pages(folio) / 2) >> folio_large_order(folio); + /* ... but return at least 1. */ + return max_t(int, avg + entire_mapcount, 1); +} /* * array.c */ diff --git a/fs/proc/page.c b/fs/proc/page.c index a55f5acefa974..23fc771100ae5 100644 --- a/fs/proc/page.c +++ b/fs/proc/page.c @@ -67,9 +67,14 @@ static ssize_t kpagecount_read(struct file *file, char __user *buf, * memmaps that were actually initialized. */ page = pfn_to_online_page(pfn); - if (page) - mapcount = folio_precise_page_mapcount(page_folio(page), - page); + if (page) { + struct folio *folio = page_folio(page); + + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + mapcount = folio_precise_page_mapcount(folio, page); + else + mapcount = folio_average_page_mapcount(folio); + } if (put_user(mapcount, out)) { ret = -EFAULT; From patchwork Mon Mar 3 16:30:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02952C282CD for ; Mon, 3 Mar 2025 16:31:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D47228000C; Mon, 3 Mar 2025 11:31:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 087A7280013; Mon, 3 Mar 2025 11:31:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DEED428000C; Mon, 3 Mar 2025 11:31:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7AAE3280011 for ; Mon, 3 Mar 2025 11:31:05 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 91AFA80C81 for ; Mon, 3 Mar 2025 16:31:04 +0000 (UTC) X-FDA: 83180779248.14.8A2C24A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf16.hostedemail.com (Postfix) with ESMTP id E81A518002D for ; Mon, 3 Mar 2025 16:31:01 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=C98sW9BJ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019462; a=rsa-sha256; cv=none; b=u3wvlXW87dxtapzh2W0T+MjwlksdPJWSEBhTeEOrAFSHamu1FOywSpTb43DgTLLnjGMggX w9sgGa2FEW6c8Gerr78NEZE3DHf+7ot3qseXL8ZclRfhP4rhPBCPXoP9biMTIwjtCvWFWe d31ZxAASaqnsw22ngEbwvk4qYAxKFzs= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=C98sW9BJ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019462; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6Ha8UZDq7Lu61upXZpfKkXckZhfouMk5cB7+X6dnx8Y=; b=WboEiokgHLaaeXihBKD3wSa6cWyjm55yzHT1VdYbPq8xt47Bx3qLzphfCnPvJ/unUp7xWv Rsa9rSzhDrhnXdDML334ZGTYVk2ITbAIUZjSI1aEmRmkUYjlKh/juRic/D38iIXi9MRgoa hE1MSt8Iw7yWyRh7GjTYFcFGuMLe4F4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019461; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6Ha8UZDq7Lu61upXZpfKkXckZhfouMk5cB7+X6dnx8Y=; b=C98sW9BJyUqqx72Yo/iILcqzlDS5Zk74Q5B98Ele51F1Z5PPnj6KgHbDR9pVMSS5NBauch iBcpPIO7Go5em4kSqtq7ljeaGKMB24h5ghCPegt/hb9pkPrl58ndQ7v5c1mFevZbxuc476 IeHqa2bL1ySNa4peO4WVhQs9ZwK2iVI= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-614-zCUIZIi0Noe-Ncw2Y0qpeA-1; Mon, 03 Mar 2025 11:31:00 -0500 X-MC-Unique: zCUIZIi0Noe-Ncw2Y0qpeA-1 X-Mimecast-MFC-AGG-ID: zCUIZIi0Noe-Ncw2Y0qpeA_1741019456 Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-390f9d00750so1344834f8f.3 for ; Mon, 03 Mar 2025 08:30:56 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019456; x=1741624256; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6Ha8UZDq7Lu61upXZpfKkXckZhfouMk5cB7+X6dnx8Y=; b=TQC/qkRrb0JVA9guQs2k+MknniqjIiCbh1k0kMZnH0aRZThfKmuE81BEpqZWja6j+4 Wx7sdKqUdLsl21EkNHbQFyxxuv2TmM08ilivi87K2r6uW2UXc0kt9xIBrkfeIamglnoY KfAqfU6ICEp5PkikTGXPklnPTkte3RRPp24DlRDqwVLFpCeSZM3/Q1lYwc3xCrQYKfkn ljgPkX9uhp1/uIH/P/6y37iStsDJ7i90rlaqS7RbXu7TDMz2TrFRYfMdSAREB9f3TZ/p 1ZZccdF0Mxr14tj9WBGLmp2UcEjZC7xp6sqyMIdZ9YxoR/d2gSdS0yWVhzzMk9+GSivD kiyw== X-Forwarded-Encrypted: i=1; AJvYcCXHbhgJ8FN/wAZUYPAYKe1PLuWr/IoFSubdccVi9oxhSBT/sBvB6RG1l8H5ITlGiYcsgBlbL1JGdQ==@kvack.org X-Gm-Message-State: AOJu0YyQkv+Zo/z45OxXFBMgaKvBs4zLA2sN9wL6W9EhqtIHWkgka0K7 7ZEGcyBSYCczc016ACQy7Q4cZNR9rZlE2iZMhaVNqPufLdJAqkYMnS6QNl/1ByVIbm/aA1o4mgg Gnj9lp6o8Pp8vFCE7c9ax6y5SLxAh3C9fz83ngq6IbFarby71 X-Gm-Gg: ASbGncs2cCq3wiRIr/RQIjWCmQd9yFmEdtYr7BVuSZ0kQjt7PGo70jJEH0kTV0B/veN es5LEjP1azKmhWKd9qncJPEjx3zeDq1nd25nk/Up01Tx3BbfZT1uRRD6MzEAzzYZmu/c9Ce9643 nAf5BAQ+uffPgxAoiC9H+1YrxPHSsfrxjhbeC5dgPSPV+ykfkPH6x8in8zx8ZTEsnxB5XVh/jl3 NnYa6Y55MklcqdAg7ILqOH+BYu6Hg3DRXWj36xfF/rejCtSTIAWmzwI9+6MlqVTrGQIEY7pnlmr tuwVeCDJkDf/BMOdZ1BAzIaTsXsTsN662BDKw8yTObxfBR5cmAfeoyvR0d5jXba71nQ21ZJyzom I X-Received: by 2002:a5d:64cf:0:b0:390:df6c:591f with SMTP id ffacd0b85a97d-390ec7cd2ddmr9512470f8f.17.1741019455923; Mon, 03 Mar 2025 08:30:55 -0800 (PST) X-Google-Smtp-Source: AGHT+IEU1cXWjWugfmabLrmr0XJAbs+GQI7107y37ij+zO9LNUtV9tqHS/ZXKWHCRmLlaLTb16eXbQ== X-Received: by 2002:a5d:64cf:0:b0:390:df6c:591f with SMTP id ffacd0b85a97d-390ec7cd2ddmr9512425f8f.17.1741019455528; Mon, 03 Mar 2025 08:30:55 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-390e485ddd1sm15052695f8f.94.2025.03.03.08.30.54 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:55 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 17/20] fs/proc/task_mmu: remove per-page mapcount dependency for PM_MMAP_EXCLUSIVE (CONFIG_NO_PAGE_MAPCOUNT) Date: Mon, 3 Mar 2025 17:30:10 +0100 Message-ID: <20250303163014.1128035-18-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: sAUTudQflFNshL2OALMn9amfHgDSZLYHOC9-dEKUnoI_1741019456 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: E81A518002D X-Stat-Signature: 69sugo5skdtxmayi4kgtgtu56mzd4o79 X-Rspam-User: X-HE-Tag: 1741019461-234001 X-HE-Meta: U2FsdGVkX1+pVmHWWs4+3BavVS/z35ncTYEfGBmD8vGa17dFRQvr9KiWFft7bIC7Y37eeG7poeb1PKVKTfNJhhqYVq92mDTX7DH4SxTB1xDabxHyKpLVMaovOAKUjzXuDHZIfd12yf35D4zcDhG37rn2kfF0itHm8RcBYVQY0LutgXGkVVEj6kMvHsw2vTvV3DaT/5u9iGcuSa2/n75LS/zU/NRRPnHjVAb87ivEu2BzNq9mIftfvvZ/QP7Ssgr2DDD6Ptgm7Pn9ZJDdyGF2F6TXJEJDzZHllkaIKpLRfuVrtnWI+n23MigGss/qHjdUzJyb9+hgKR2yU68dWTh6PlETHwx60Y7X7EWt240x8TiIid9rl36WbKfNvh0Srt4qvBPxak1udJqbWhuNe7WwhN7n5wiGLpab+/WSZnjxvg9SfSxsi/uITQ/CDMAR8gcFHd5hvKnT49l3/9c9CJYRuMGOp6bwjyYyUCwhGys8iMXAe862ZAmGhI0KY6JpFzJJzjrWrpXODGz5yTVNwK0xG2m+vJ9FbqlYcICCaU45q45w8m8T/f8vxvoQSyqQ4lo8RBWdkUhPoM8dLokyoGYa0FSO1/QZaNBt3lo40le4CSyo/2GrH6mt9ALXGQXExZW5Nctstqta6iuEQ+4KrJoO1RrmHUkxAUKeQCmahouiAQJvdEqk9prv68cHecHl44xlHyeT4Yp0GB8qpdPy5rKPQq4JMDxoahct8KowRNLepleGbo9zYHJ908Fphm/S6ruLwhnxbWWlrvBxH6jkE9HKRePYstb3MC2LLjk7lsEkBqhwkkTqbs3vavO/2CyHdSs9YshjxOHAGZbZVSvNFEfv7AvnoPUhEUlH/yG7WNIeHjvYYYS3BP4b97nKmXZYZr6rgbZy62QvYY9A8RGwWHg+vJEKtax8BNsyJno0CAPIo+LVAg0EoIH1AQ805IolIEiIKSiXfvzBhDYK7h1//Zj sMnK2+0Z /098zzLubq9lL/I7SLX8Bt6vZQgAJP0MHV82yQKn5R9ru/e/GonV85RX2b6DtwegFMmqsja5DrqqR15Q1RCPRgrJoqK9gPMCxyCysegyV99eM691VUx2oeDcKsaUSKZ8ottfmuoYXAg4cqznl28OZS/f+QqOmKm99Tybr/D9hr+IvZF9NU6kx5N0VHdqhgJThpkYPYMBi2dbq3gIRBuVBi7x3bol5GcnM+IpB7/7lo1w6F0DdSgBtySRKC6cbQ6zV6n/BFcR4oOGcNHIjhVU6xqd98sQ1esFS+gEdQkZqwPWpP0vssYEPIS600+t0ycihoq+Vu4fV45UQT2a8kEInsPPNavq1nsCz2/T5zeMuprgw0vWH8QaVXf3vkymVR92DERBmClYaEaD/v9Pui/FS57evktjCapR+sfo0c74vqWnN0/ED6E8HcBWBl6bJ6e8qyS8269O/AFKuSN+qrbba/dMqTPgRJdDqm+MO3vD+UAqFx5ceB//5BciDJHS7p05tv/hFova3/TKN91h96WwdCmH2CA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT. PM_MMAP_EXCLUSIVE will now be set if folio_likely_mapped_shared() is true -- when the folio is considered "mapped shared", including when it once was "mapped shared" but no longer is, as documented. This might result in and under-indication of "exclusively mapped", which is considered better than over-indicating it: under-estimating the USS (Unique Set Size) is better than over-estimating it. As an alternative, we could simply remove that flag with CONFIG_NO_PAGE_MAPCOUNT completely, but there might be value to it. So, let's keep it like that and document the behavior. Signed-off-by: David Hildenbrand --- Documentation/admin-guide/mm/pagemap.rst | 11 +++++++++++ fs/proc/task_mmu.c | 11 +++++++++-- 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst index d6647daca9122..afce291649dd6 100644 --- a/Documentation/admin-guide/mm/pagemap.rst +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -38,6 +38,17 @@ There are four components to pagemap: precisely which pages are mapped (or in swap) and comparing mapped pages between processes. + Traditionally, bit 56 indicates that a page is mapped exactly once and bit + 56 is clear when a page is mapped multiple times, even when mapped in the + same process multiple times. In some kernel configurations, the semantics + for pages part of a larger allocation (e.g., THP) can differ: bit 56 is set + if all pages part of the corresponding large allocation are *certainly* + mapped in the same process, even if the page is mapped multiple times in that + process. Bit 56 is clear when any page page of the larger allocation + is *maybe* mapped in a different process. In some cases, a large allocation + might be treated as "maybe mapped by multiple processes" even though this + is no longer the case. + Efficient users of this interface will use ``/proc/pid/maps`` to determine which areas of memory are actually mapped and llseek to skip over unmapped regions. diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 1162f0e72df2e..f937c2df7b3f4 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1652,6 +1652,13 @@ static int add_to_pagemap(pagemap_entry_t *pme, struct pagemapread *pm) return 0; } +static bool __folio_page_mapped_exclusively(struct folio *folio, struct page *page) +{ + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + return folio_precise_page_mapcount(folio, page) == 1; + return !folio_maybe_mapped_shared(folio); +} + static int pagemap_pte_hole(unsigned long start, unsigned long end, __always_unused int depth, struct mm_walk *walk) { @@ -1742,7 +1749,7 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, if (!folio_test_anon(folio)) flags |= PM_FILE; if ((flags & PM_PRESENT) && - folio_precise_page_mapcount(folio, page) == 1) + __folio_page_mapped_exclusively(folio, page)) flags |= PM_MMAP_EXCLUSIVE; } if (vma->vm_flags & VM_SOFTDIRTY) @@ -1817,7 +1824,7 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, pagemap_entry_t pme; if (folio && (flags & PM_PRESENT) && - folio_precise_page_mapcount(folio, page + idx) == 1) + __folio_page_mapped_exclusively(folio, page)) cur_flags |= PM_MMAP_EXCLUSIVE; pme = make_pme(frame, cur_flags); From patchwork Mon Mar 3 16:30:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999204 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 066CBC282CD for ; Mon, 3 Mar 2025 16:31:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF4BC280010; Mon, 3 Mar 2025 11:31:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DA0ED28000C; Mon, 3 Mar 2025 11:31:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C18CE280010; Mon, 3 Mar 2025 11:31:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A37E028000C for ; Mon, 3 Mar 2025 11:31:04 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5F263C0C80 for ; Mon, 3 Mar 2025 16:31:04 +0000 (UTC) X-FDA: 83180779248.05.89D3824 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 1A52A120012 for ; Mon, 3 Mar 2025 16:31:01 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BrJ0s2W0; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019462; a=rsa-sha256; cv=none; b=EYVww4wJOB3C2ihXE3mx4KB6lgU7RXdt+Rtnq/BBVFJmeV6yasMQD9gntaFWx8Mf3zqFAq vOhDA+lCf+U7wlrNGnwYeABDrCbPUOxzUN6plqaRicos32bwpI2Zc1rYy8fTi/zYFR44mS RFQc+bA4lrqnFzx4Bgay1SN1cWGnQjA= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BrJ0s2W0; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019462; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sbAdVb8IlT1mHMxqkPSZKSMArZKBklKlP7kdJIBKPLI=; b=E8PqSmr2Gh0i0wXAE7Atve7oRwdkC9x7HLZP2r8z6neuQmeqwXQBqtyFjG80G/m6oCHSOI M6e/P2Gm5dPKBBEt660l2OvtG56lBKiZ5LZxDuFOUX2txkadtZ7oMdD0uP/0TRxGCRA4yC lqI4CVO6nJah7yun87mQu+elvBe2gJQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019461; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sbAdVb8IlT1mHMxqkPSZKSMArZKBklKlP7kdJIBKPLI=; b=BrJ0s2W0ubjky4tyJawNP3hOp/ADnmWcfwBmWS33n5iIpSGMymDWOrvY0F5VYXdEZv4XB0 +q9BH0kweV7525O2gvMt1w1xNUmjmvg8M8Pp8GaLQssFlk+yk5aNpcP7A9rO12x9mo9p+g iBhBgJ3H4c59YseL/pLg4lAPuKXl3Dk= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-427-y-SlsvfYMhmuBRzc84elwg-1; Mon, 03 Mar 2025 11:30:59 -0500 X-MC-Unique: y-SlsvfYMhmuBRzc84elwg-1 X-Mimecast-MFC-AGG-ID: y-SlsvfYMhmuBRzc84elwg_1741019458 Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-390de58dc09so3113114f8f.0 for ; Mon, 03 Mar 2025 08:30:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019458; x=1741624258; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sbAdVb8IlT1mHMxqkPSZKSMArZKBklKlP7kdJIBKPLI=; b=m/AB/NN6w/cpR1h1so5rI7mfzZtFQLIy0o6reJ+P4ygpsqSi3OsP9QJJ0+uQc+SfXG P+z/EmuPrubfmbpwhoNyYoRa3WIQS/uEI9Yv18bgTzjjYTX+MfUCa6xkv8qbqsuPTT7s 69+e2JcpZYXpveI+3KScJcBjDMW9RoxbH2dRwcjd/pYi0m25MrnYsWmqUDr3CAYkhQI0 PB2t/k+miCzeM2re6ads7HAA/D8gGrp7hCLzgtnfP/kRFLqeURhTZJf3OaZDWqWPcPE1 R9iKMsoRZA2wNf1PSS40x5a2PPXVpUTPl6Sh7pPqvOO77/ki9I7CTU5zXYhARhUJw17M lwAQ== X-Forwarded-Encrypted: i=1; AJvYcCWject0IC9wbaJNHTDRIuNbOwPbnR8IRE2uJA251kAr+4F5XnCduroskfLry+LcMjkrQTD3HMI/PQ==@kvack.org X-Gm-Message-State: AOJu0YxIHCKxNEzihfcIyVHL/tXSGpHs3EoA5AXD/h5HYiZl7Ji0jh6d jaI2oWqr1ehfwqH1bFTrXs4bmGaYEBjqFEiI7+zTQeAZneLdNVh/skOcbx5kBHm/FPijVjUTWAx 6+HYKkV+2ZuiiyGed/1/SlPeF8WphcOqMSqSyyOLlyT3QQ5tF X-Gm-Gg: ASbGnctFyMJ3dQq4N98iW4/NDHSeKNn7KZHI1rI2nqbvdPQ4l8d9B7pxQqReGi0tPWk +sUiWk6psmqEMreEPR3aNS3UHbmuZsl+zdjmHGNB9l9jbqKCtqxIjPwL9QgXGzQdWoNtg8c0rfP AO5kgsbrywsWefG3uY+f7pI41aEaKDRIQxNzRZSBa7wpBxFXdbgI2/XjzS6Oaef6khhDd3bwQe/ Ptq8fwIMyYksX8jinUGiilBe3dvCOjYARM0FyeAdCCS83v1SbPU7ghnZLovIZQaWMUCPduiuk1Q 8wXV0SC5BIRRxuFIJXIe1j9Kq8UQhkxE9QxKxDfjM5UXJ3GWK81upPT3CW+Jn0OXJh3p12MpWU/ q X-Received: by 2002:a05:6000:156d:b0:38d:e3db:9058 with SMTP id ffacd0b85a97d-390ec7cb945mr10994389f8f.12.1741019457892; Mon, 03 Mar 2025 08:30:57 -0800 (PST) X-Google-Smtp-Source: AGHT+IEvRr8UGYMtbKKLfF4vYk+epXuFbk91XT0kZhy1XfmXmyE27quXMsv2mgA7hFUbYP+d7m+S+Q== X-Received: by 2002:a05:6000:156d:b0:38d:e3db:9058 with SMTP id ffacd0b85a97d-390ec7cb945mr10994345f8f.12.1741019457510; Mon, 03 Mar 2025 08:30:57 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-390e47a6aabsm14858335f8f.26.2025.03.03.08.30.56 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:57 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 18/20] fs/proc/task_mmu: remove per-page mapcount dependency for "mapmax" (CONFIG_NO_PAGE_MAPCOUNT) Date: Mon, 3 Mar 2025 17:30:11 +0100 Message-ID: <20250303163014.1128035-19-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: jOBz1ooOPal-CwUVn7697lhnLT7jkZnJ2_qHEYnCOfw_1741019458 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 1A52A120012 X-Stat-Signature: qwxmqh9jndsmmtrhjg1sc3z8oh18cpzk X-HE-Tag: 1741019461-221500 X-HE-Meta: U2FsdGVkX18rr0olybgU71rOHz1tTRGQbuppGCSrw5TCKR5i+PzYYwgdtNKBiWC88rwtiRxZqDEdWju5lEp9tZxH0Qq8++ErENGXK6bt5bj4L0EqthENmiI0piHfKXf00ibS2I1jBhlzzQhKvK1yJ269s6wqKcnoKCmgsfLO4dZsWz0+qIYbdjl69ZyqVFyhTz/N+07LmSc/lEGvDejI7B9ECunPIqVUk6S0fDzJ2z1JxGRxmf7+wt3xnGaJKct9kHTUj9NQk0DiNHlInaR/I/Xe1sfKWrbe016vGDEv6gBQYCetbG2mVTts7oSxr7+ANX4x3SBTP8iPEJtZU8ilazUagDm2fNrPAvALPDfGx1z832TJFNdk8NwKJ9pmQkdhgdvgA9aTKpPvB2XoKHLxrKM7UZIfQ4sgP3lx3E0XqNM090FYcobvcyF7si3X/es8tePg0YT8Pgt2y1LRihVEO+kaMWIojlzYE3w5UiTAA+KI2JraJG9R5eBgV/qtJdvVcT73VoW/Qz7JaUzkQBW85bH0HG+8h9yimn50XbmPKSnwTAcp6Js3hEsUW5LHYh+6uqQaPF3b8sFD73bRASWjjPbVdlbuspwiFC7//I32Yqa7hfCMG/NXYuhYqZRc88sR2D0CPjjE+HNHrrR8SXl/zOhGCZ/SIRNngZw9GJFfzEVSDD0QtY2AEHMbPj5aClCEREf24rHWUruNV7BN3vBvXlQKEzJZO/hei1SNVy4UxBDFm1we78IKzJIaPFhf4wySuMdH6McbMkuujRppy1zGOk4VPQVBAH5vrVWk0aB1Yz/PLKGMlEra10jIeOqt6h95Z9U3pWbSPuL2c3LB8Sv6bQZPtj+1ZFPbyZZ8k74n+Yn0rjatyB9y32olYD2Aqta+lhODC8z1+Kj6vySDsRzK9OtoTjY0YSRfj2FIgMBLkdOM7c+O89M7yVlUNFyjS3ErMNG9HoTTNP0ucZ+CTvF ZNnq2B6I qxICgLxevmcH6mX56q3h7TX/x/KLUGECCq5mPuHvSFcObLIo+n0x4I7qR5vlF0WG7w24HfA69KDRC/H9Xi5cCAAE5U6jsQExlzXLmelURChU6OPZZeewtiUq6174WLa4+wqFdTGp/r8jZbxiJVxn+HVmjrNn/KGQLk8KVj4XX7p5ckydJDAo3QtGbbtB5RfSxWCRKZ/hKmxi1tyDLzgZL3aCR2bKn0Bwzl8CRJOxxMTe8zP3E9bLt1ig7Il/ny5TKMJWlCza+tQqiIsnDXBKRG1nnZYTWnUslbUyzwQgww8t79ymwMv7RDMCUmO5bIzaspRK11cdfcDtPNGjqnkft60ampH8mUai/L8aWpgUIGfoh8RDGfMaVl8pb3aZzAjaV+qHsUWmENpnSBQyOf61kupZh/7csdW7i451xmh7gvlXQfURPQcD89z/PGfnXX3DX/WIwEYQv+b8QI125SLwQ7GJsoDTx2Q18QMTjnwLggpWbsHuoNZnzq8iMD3p7N22a2HxGlRJ4DQww2ZUKoqJipVAR8g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT. For calculating "mapmax", we now use the average per-page mapcount in a large folio instead of the per-page mapcount. For hugetlb folios and folios that are not partially mapped into MMs, there is no change. Likely, this change will not matter much in practice, and an alternative might be to simple remove this stat with CONFIG_NO_PAGE_MAPCOUNT. However, there might be value to it, so let's keep it like that and document the behavior. Signed-off-by: David Hildenbrand --- Documentation/filesystems/proc.rst | 5 +++++ fs/proc/task_mmu.c | 7 ++++++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 09f0aed5a08ba..1aa190017f796 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -686,6 +686,11 @@ Where: node locality page counters (N0 == node0, N1 == node1, ...) and the kernel page size, in KB, that is backing the mapping up. +Note that some kernel configurations do not track the precise number of times +a page part of a larger allocation (e.g., THP) is mapped. In these +configurations, "mapmax" might corresponds to the average number of mappings +per page in such a larger allocation instead. + 1.2 Kernel data --------------- diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index f937c2df7b3f4..5043376ebd476 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -2866,7 +2866,12 @@ static void gather_stats(struct page *page, struct numa_maps *md, int pte_dirty, unsigned long nr_pages) { struct folio *folio = page_folio(page); - int count = folio_precise_page_mapcount(folio, page); + int count; + + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + count = folio_precise_page_mapcount(folio, page); + else + count = folio_average_page_mapcount(folio); md->pages += nr_pages; if (pte_dirty || folio_test_dirty(folio)) From patchwork Mon Mar 3 16:30:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4687BC282D1 for ; Mon, 3 Mar 2025 16:31:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 53E3C280013; Mon, 3 Mar 2025 11:31:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4ED9D280011; Mon, 3 Mar 2025 11:31:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C92F280014; Mon, 3 Mar 2025 11:31:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E659B280011 for ; Mon, 3 Mar 2025 11:31:05 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9691B140C6C for ; Mon, 3 Mar 2025 16:31:05 +0000 (UTC) X-FDA: 83180779290.22.4EEA91D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 3AEB1120024 for ; Mon, 3 Mar 2025 16:31:03 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QS585Uf0; spf=pass (imf29.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019463; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=d6h5undxasqJ0N3FiNJqqgjSewvjkbEXc2mo8LWFs/U=; b=s4/u+HT7sygzOY/8txnoY3cmxWdYcJYs/1nKYD4UhFRkHkMSUhnmsEkgGfQvgq8LGSKVYb tQBtuNsYiQ9AA/CJRTHyFMRFrYjRR6cj2X+pgLwbi85wDF2rI4VWyMZcjjeSM/FSH+WmCU 4QcqHomsJ4uI2bD5O9MOje49cvfURzo= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QS585Uf0; spf=pass (imf29.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019463; a=rsa-sha256; cv=none; b=BU+iiETIR4PQJvCMm2N5p+evmeOvX5VstJvprvjnpcZngyVeg6XFL6mfsmXmVjhPUMyEmZ imGskfpxtn9HjkY0CA/JBQDZ8ceEA5VoGyt+g/basgam2zwKaxRNLFZbQpXwFba3PfA65A YezaqghJlEP6afRVYHU0Yh/59XOIo8U= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019462; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d6h5undxasqJ0N3FiNJqqgjSewvjkbEXc2mo8LWFs/U=; b=QS585Uf08NR7zMvmFFh6v2E2mzT3pEGhN1mCdC8xIaqao6tRWHiAuE0thmZt4cFIbuvvco N/fLkql36gkDuPCeL9BvF4iXjU3G2TJCRQqQyLbXXKKt7A5hGz2IK3FO3bJZ1Svd4fbW6w 5aekMX+AoU8bYpUNUR+hVGC+wZPIryY= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-152--eXCdIDlMZmJ1k_wxQXWQw-1; Mon, 03 Mar 2025 11:31:01 -0500 X-MC-Unique: -eXCdIDlMZmJ1k_wxQXWQw-1 X-Mimecast-MFC-AGG-ID: -eXCdIDlMZmJ1k_wxQXWQw_1741019460 Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-438e180821aso24939065e9.1 for ; Mon, 03 Mar 2025 08:31:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019460; x=1741624260; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d6h5undxasqJ0N3FiNJqqgjSewvjkbEXc2mo8LWFs/U=; b=dDtq5AI7Exb9rNpv0VD7MaJ1s8fidoYTsUDve8hcodw7Iuzfe3v8T/mlileZKzr481 0GIHV7H2zlqdLsk2oWg+fbkJHK2ogZ2n7MXNKp8rCW8nReUwZrXoCLfIINkySK6qj9fG hr2Iq6laS1cvjbnatnyETDPvdTqbanpSvKljfdx3imFYkXJEEDEXULwWZl7PXQ7V5FBK 63fhBNkeI5ib8/CvZC9QUAiiLkT/Wky0rBYWn3I5jBdA6u0WVd/dv46wgoNv3NP3jbOz AIKOzoprmtaQQboxR1dCUrjV9tOtIUsdYJ9P8VLq+U/saaT7+AzIio3BpmVpYNoJXtzD 5ptw== X-Forwarded-Encrypted: i=1; AJvYcCWqUVqmWoggZTHMWQXIdc/1p2QAgtnr8NwvkIqi9qLIluc8pR1asIyCvvb8og7HLGjP2YB1CIc53A==@kvack.org X-Gm-Message-State: AOJu0Yxpyys/mq0BIj9JLFo2so6BlkNk8KYp/SWv8MV6X/r1lXfQaT7p fTBhh2Q4VMYeaqgN8Nw7LZnvEBrjOanoporBt3lYBFQhuERX/h12d6xW+sNOIgswsXIjt3Lgum8 CuEEFRApSi47ljgOK8bmj+EBspU7hQrhApyPhr7+YXEhVixkp X-Gm-Gg: ASbGncsNjorxTxZ7/y/aFfCcsRljS0AzUbnXvqKWmr+q4rD75kFD+OJMKZcGw/MeNKq XX2TjMR+l65Q56QH/ZhkV7UhVWnJ+ana9PPEKOSZ7OMYALtrrRv+lfu7YW6ZgMq1638UEVpEs/0 becCiA0Pzr3ltNEpFXie0Wv1xRS4914TFSajUKwcaPgvp3kSbQ1Ju97KYyl7HGVKPfFZWD2orUJ FCMs72NYZrOCgJpuXa5FGKAYXYR6nRtpblMEvGDGW13j628qqugCqujG3HDfyBDq34s4eZb+p1X O/fBrDwfl4mHQIwa4biX0J7KeI0RZrAKmXzfA9NcW7Wux+N4BDtkujTnJgJHzi4w4cZIKmfMS4J q X-Received: by 2002:a05:600c:1387:b0:439:a138:20 with SMTP id 5b1f17b1804b1-43ba675d737mr104551605e9.20.1741019460133; Mon, 03 Mar 2025 08:31:00 -0800 (PST) X-Google-Smtp-Source: AGHT+IF077jzHsKl488EfYl1BqdHJTJ6b3kDjSKQKMSwBjouvZD2J29Qu4rXpeiVOA7Tvc80mUiuyg== X-Received: by 2002:a05:600c:1387:b0:439:a138:20 with SMTP id 5b1f17b1804b1-43ba675d737mr104551065e9.20.1741019459654; Mon, 03 Mar 2025 08:30:59 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-43bca3df658sm4293845e9.9.2025.03.03.08.30.58 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:30:59 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 19/20] fs/proc/task_mmu: remove per-page mapcount dependency for smaps/smaps_rollup (CONFIG_NO_PAGE_MAPCOUNT) Date: Mon, 3 Mar 2025 17:30:12 +0100 Message-ID: <20250303163014.1128035-20-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: MzQ7fmQSk0H--G9sRAEOV1C8Xcmo6hf8FigIwD_J2ds_1741019460 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Queue-Id: 3AEB1120024 X-Rspamd-Server: rspam09 X-Stat-Signature: sns1c9uexa3y978cjkjhggixd6iwwnnm X-HE-Tag: 1741019463-946058 X-HE-Meta: U2FsdGVkX1/yQXw9A5MvSGWXHy/M6xe3PXSjDb5J9l9QkrpFwe4RWUXXTioze9iiQQCoNtpLdlEW7aBvNIFrzJHCkzocbc8kVMx/+EXR1pMkBE3lubI8MrpeRqPfRQhKl8jS96+PIsZU2rAISSeUsFEYuFUKoaqr25VhV0SecttqpSr7iFfwRQJ3g0ysek99rR1jTbPFhEZ9TZji25QgQGjcfy0DPo5zHZ/Qj6W7+jF3avG49OjTIF05B+sFtFuopIpb0x+kGb1E/y871hPUnFqHjrnxsabfsokLNiy6LDJe0JSAf/44z877wMeg744mIxvdcKfFd6oFGsoEAu2I25KVlffKeOEdvbNFPib0G6kmCvHhgW/hHP07szj/B0jrSylHUDPdiEzOTxKwVoanr/mRimvgqn+1cVGzrzFR0lTaVLDtGBGyeg6HPKzHMOXMDa+7pH2Stmm609mdQ5TQjkdVB3aOm91xN0Lhg6BvY0ifJ1zox0EM11OaALPP6tx9Lr6VHZ9lBK17Oc60NNWOh2pUkm1k3yUVNfwIPLfiI2l2c0TQJ8YT8JYtJ6EUbgM6eyUJAT6CqfomBORbZLoc0OEM79Xwsr9up6a0fybYIUFBkqXM9IShBBYenhXDoBCf6Z39KTqWwHPqzqkt7+O9e5oLEsNtLFwd4bvMNVFqLfoL7HvU3XY2c3EKV4pNHZB8ejI8H5LPgjknA4KYfoEtzwd6/fUA3LkX5pOCWbsRWB+ttMExMsHX2m7806KfHwisTExl24oocmQAxRDEkiJGiZyErgEoxwxxLt72vCItqfs+QmVxadp0A6jiQ9dUzayrd9zuSD/k4EQ7YIks7v3Mg9fqmx4i403ktvNkIzsBkbzNBECvCdhOnd4hn0dc8KjPnxuu2rNfQxTYSBQhLgBbSfXValpp5RSzzn37ZkPdEdH29So5hF7nClohF+f03I7dCxtQpQAPDAfoBxWfHkl tPBVQDcF wbKDmZH9UflwIBa9tL1mEanSa3SnIBugqDNIAEtAXALDsz0ajUWRypZAdorpzpe6TVjmIA62U2FdR/jfDMBaO/uclNIALX5AkdYxy4glAdcO20mQ9xPqi4CXAuuvy1BQwtAtiAsvlSVafHOChQhIL0XH7MiV2t3IJ50hZXUq3YG9OOK8EAI0YfHHJckaw6b6P9FT7oP3oe8/DgKETZlkBs3LvNvPEW34x5SByFs8IqMMPU7fppuk13TzyIwEalDWO5+sWPsE46/6CrldEilLuJ8YeUlNVutM/nYQcRsHE+Km4+oB4u8oKnonTABZPljIWDARgrQTqeEmcHo+eXEj3WksJrWAPdVZYa9VkNf1UFgKPRE2h/AD6KPHEGjHXHKvLaWqMM5mLHh0X3EJ7ZSDobYOp9mZ8YCc2iRVwzLIYgse12nPzNQXj0qrQsvzPbJ5FBQ1/w0Ktx5Z52XHt2onz3LtTTDvgg8bt9kWdAnV2klRDUPI/J4tqrE5d2i4vusQJGB0XiIM2G6qd/Tk20O1SFhEnbQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT. When computing the output for smaps / smaps_rollups, in particular when calculating the USS (Unique Set Size) and the PSS (Proportional Set Size), we still rely on per-page mapcounts. To determine private vs. shared, we'll use folio_likely_mapped_shared(), similar to how we handle PM_MMAP_EXCLUSIVE. Similarly, we might now under-estimate the USS and count pages towards "shared" that are actually "private" ("exclusively mapped"). When calculating the PSS, we'll now also use the average per-page mapcount for large folios: this can result in both, an over-estimation and an under-estimation of the PSS. The difference is not expected to matter much in practice, but we'll have to learn as we go. We can now provide folio_precise_page_mapcount() only with CONFIG_PAGE_MAPCOUNT, and remove one of the last users of per-page mapcounts when CONFIG_NO_PAGE_MAPCOUNT is enabled. Document the new behavior. Signed-off-by: David Hildenbrand --- Documentation/filesystems/proc.rst | 22 +++++++++++++++++++--- fs/proc/internal.h | 8 ++++++++ fs/proc/task_mmu.c | 17 +++++++++++++++-- 3 files changed, 42 insertions(+), 5 deletions(-) diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 1aa190017f796..c9e62e8e0685e 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -502,9 +502,25 @@ process, its PSS will be 1500. "Pss_Dirty" is the portion of PSS which consists of dirty pages. ("Pss_Clean" is not included, but it can be calculated by subtracting "Pss_Dirty" from "Pss".) -Note that even a page which is part of a MAP_SHARED mapping, but has only -a single pte mapped, i.e. is currently used by only one process, is accounted -as private and not as shared. +Traditionally, a page is accounted as "private" if it is mapped exactly once, +and a page is accounted as "shared" when mapped multiple times, even when +mapped in the same process multiple times. Note that this accounting is +independent of MAP_SHARED. + +In some kernel configurations, the semantics of pages part of a larger +allocation (e.g., THP) can differ: a page is accounted as "private" if all +pages part of the corresponding large allocation are *certainly* mapped in the +same process, even if the page is mapped multiple times in that process. A +page is accounted as "shared" if any page page of the larger allocation +is *maybe* mapped in a different process. In some cases, a large allocation +might be treated as "maybe mapped by multiple processes" even though this +is no longer the case. + +Some kernel configurations do not track the precise number of times a page part +of a larger allocation is mapped. In this case, when calculating the PSS, the +average number of mappings per page in this larger allocation might be used +as an approximation for the number of mappings of a page. The PSS calculation +will be imprecise in this case. "Referenced" indicates the amount of memory currently marked as referenced or accessed. diff --git a/fs/proc/internal.h b/fs/proc/internal.h index 96ea58e843114..8c921bc8652d9 100644 --- a/fs/proc/internal.h +++ b/fs/proc/internal.h @@ -143,6 +143,7 @@ unsigned name_to_int(const struct qstr *qstr); /* Worst case buffer size needed for holding an integer. */ #define PROC_NUMBUF 13 +#ifdef CONFIG_PAGE_MAPCOUNT /** * folio_precise_page_mapcount() - Number of mappings of this folio page. * @folio: The folio. @@ -173,6 +174,13 @@ static inline int folio_precise_page_mapcount(struct folio *folio, return mapcount; } +#else /* !CONFIG_PAGE_MAPCOUNT */ +static inline int folio_precise_page_mapcount(struct folio *folio, + struct page *page) +{ + BUILD_BUG(); +} +#endif /* CONFIG_PAGE_MAPCOUNT */ /** * folio_average_page_mapcount() - Average number of mappings per page in this diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 5043376ebd476..061f16b767118 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -707,6 +707,8 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page, struct folio *folio = page_folio(page); int i, nr = compound ? compound_nr(page) : 1; unsigned long size = nr * PAGE_SIZE; + bool exclusive; + int mapcount; /* * First accumulate quantities that depend only on |size| and the type @@ -747,18 +749,29 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page, dirty, locked, present); return; } + + if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) { + mapcount = folio_average_page_mapcount(folio); + exclusive = !folio_maybe_mapped_shared(folio); + } + /* * We obtain a snapshot of the mapcount. Without holding the folio lock * this snapshot can be slightly wrong as we cannot always read the * mapcount atomically. */ for (i = 0; i < nr; i++, page++) { - int mapcount = folio_precise_page_mapcount(folio, page); unsigned long pss = PAGE_SIZE << PSS_SHIFT; + + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) { + mapcount = folio_precise_page_mapcount(folio, page); + exclusive = mapcount < 2; + } + if (mapcount >= 2) pss /= mapcount; smaps_page_accumulate(mss, folio, PAGE_SIZE, pss, - dirty, locked, mapcount < 2); + dirty, locked, exclusive); } } From patchwork Mon Mar 3 16:30:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13999208 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 925EAC282CD for ; Mon, 3 Mar 2025 16:31:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 872F0280014; Mon, 3 Mar 2025 11:31:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FA9F280011; Mon, 3 Mar 2025 11:31:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6267C280014; Mon, 3 Mar 2025 11:31:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3AF2B280011 for ; Mon, 3 Mar 2025 11:31:09 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 98E361C8787 for ; Mon, 3 Mar 2025 16:31:08 +0000 (UTC) X-FDA: 83180779416.20.967E37E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf20.hostedemail.com (Postfix) with ESMTP id D1E621C0032 for ; Mon, 3 Mar 2025 16:31:05 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=d7ufStnZ; spf=pass (imf20.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741019466; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YlS6X8Uew29SCA4ywjbEdY/YLTwcEHUkyCcmw2xO2pk=; b=NGQLeu0A9PvMdgvibzLQiHO2bVjF3HPkz2ajjw9XyUKmZu+L9BEsrz1zbvryxsQBUf/rrq 1doTLT/R/57fmvaFqsZET46mbwZ4+VMhftkZ/6KKAqzk2sm86YuuY8UDvJFTZt5jxRN/77 e4Aeyj4B92XHlHcXf6qda9V9XOsTXY4= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=d7ufStnZ; spf=pass (imf20.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741019466; a=rsa-sha256; cv=none; b=XhoD2LavHCT0vD7yfsLv0gEOuBjOw6UCjRrxQ4mNJz6l1uGnzlTUdblQsiKEB1r/4XskFC DTQBqG0gw7hcvQcbntjEJ6Az5dUdDsJLFcLRSgkvv2uQwNW09YJKp/OyNaa7Mp/ZqDecNU 8NCKI9J4zb1nKuHPGzttC55mULwaA9A= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741019465; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YlS6X8Uew29SCA4ywjbEdY/YLTwcEHUkyCcmw2xO2pk=; b=d7ufStnZQzqFDk/jHoTIk0+n//9JrmsQvHJQlIiUZEywN/Xlmhy8giXMJEsNeXQyKtGDQ9 5QpEI6B2yPJMfXyCGba9wtXewSA6gSFL0EObDeXeB/ga3OrbWBRY644qI05IrpO/jMZLgO E67PLIdEKDFtDK0G8fYoZQ832ze5L0g= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-201-yNWFPOkTPbaCV_MFHTaLaw-1; Mon, 03 Mar 2025 11:31:03 -0500 X-MC-Unique: yNWFPOkTPbaCV_MFHTaLaw-1 X-Mimecast-MFC-AGG-ID: yNWFPOkTPbaCV_MFHTaLaw_1741019463 Received: by mail-wr1-f72.google.com with SMTP id ffacd0b85a97d-390e5996058so1837860f8f.0 for ; Mon, 03 Mar 2025 08:31:03 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741019463; x=1741624263; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YlS6X8Uew29SCA4ywjbEdY/YLTwcEHUkyCcmw2xO2pk=; b=mis8XsTiCeVzuSr7EpijI+GSLc3S/0vYLPxyZXwRc0DNFWyC88ezdu3dNO9XjPVlw+ QDkxq9q9A10L3SUEZwIwR3JXg5fN+lt975dbO+LwNnYkTrZtRRHUVFfD58xD1vTJ9Eo6 HjzRNOiya9SUWxKX595JbAsRcSAA5rtD5k3yGKyD4DXbImWdQzUEPdZHOWtdEPlzTPyk 7hLeo/qn5ngHOoXcc46VM5P1DxBeTA7dMuxh+zOk6sY8B3tFvuw/GZSxkAHlmeeejecf 80cUgi9UTLmudioZzT9o0jd0O3WHHFud9B9POg7JvcF4dXKC6EfwcZj+gKCZdpP72anT UfgA== X-Forwarded-Encrypted: i=1; AJvYcCXALTjCD42SlQvHJoAIV0xePKN3cycQoCsGhGzbRLr73PBD/XhsTlocuoteDx8s79UFGXgOzeKT4A==@kvack.org X-Gm-Message-State: AOJu0YwXM57rqAd3SwbjUaUWzlEpm2TAh9AESHCo++ycX7UeFpdtwTid 1usJN9JihpQv86OVisXlfr3V3rEM5MxZTF4kzwDJlWq0eZSB4nLD7nFFfhydqVPNV722rQlfgcn OReZTDckK9T4c/ECM0zKRJoOe0DhXLU7wov1QqWCLjGQAEpDVNSx5tU7ek1k= X-Gm-Gg: ASbGncsKRPBb2zbkS/+xCwBsIV3T0kYNJLl+dHdsVPhYQqDjWnnR2iSawR+FvyTcLy4 Cc1eAiEifHNHYQ4emZewVrcX7+KluWbHCW9B054inEvsxtOWRDtv5bpmfG/cZr0acC1iBaHjnKm ZxObhx9Ga0kE9qdtX7TMrKLoXc2XXc4LB/71JidzPzTGRfMvfj0jdQFUXYDgSOQYra39H9f3/4P hK2Bq0bc1ZI3IGw9WI51QGs4wroBiWNwoBvuteyY7VdV5QX9tRE3Ikdd08PP7LJ5hLh5oDHu3b8 XklzvFIxaX25pRCn7jlC2qdRb9kgj0/jp9pHwD3HBmZ9Pe/sd84eNVOF+gsiirlrhNmMryTacye M X-Received: by 2002:a05:6000:2102:b0:390:d777:6505 with SMTP id ffacd0b85a97d-390ec7cebb6mr8484117f8f.19.1741019462309; Mon, 03 Mar 2025 08:31:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IG4Wm1Nrswdi59UkfQX4E0Mp6V76OD2ctdPYdeeFY/isIJhyYJQe8UFw7Y7ah5xkjWGZGxEPQ== X-Received: by 2002:a05:6000:2102:b0:390:d777:6505 with SMTP id ffacd0b85a97d-390ec7cebb6mr8484079f8f.19.1741019461762; Mon, 03 Mar 2025 08:31:01 -0800 (PST) Received: from localhost (p200300cbc7349600af274326a2162bfb.dip0.t-ipconnect.de. [2003:cb:c734:9600:af27:4326:a216:2bfb]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-390e485df22sm15178083f8f.97.2025.03.03.08.31.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Mar 2025 08:31:01 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v3 20/20] mm: stop maintaining the per-page mapcount of large folios (CONFIG_NO_PAGE_MAPCOUNT) Date: Mon, 3 Mar 2025 17:30:13 +0100 Message-ID: <20250303163014.1128035-21-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303163014.1128035-1-david@redhat.com> References: <20250303163014.1128035-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: J0-6_bb0ZlLODE2Gg6RvRRdYst4PxWfAKqq0uAsb2F4_1741019463 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Queue-Id: D1E621C0032 X-Rspamd-Server: rspam09 X-Stat-Signature: ri4wpycb3wopjsk653utj6xd8dbtkrs9 X-HE-Tag: 1741019465-720325 X-HE-Meta: U2FsdGVkX1+ImMZD2DSkD13LktEN+amhSxf5LOs51UCBsCISZ0zOoLgTFRmsPGpf4lRT1C58L5U1rtEZ+UM6dq26C7SMQpqM35mv8gNQB8fy2Gk5lRr6G6m95N2KpbS6jGRRrDIVb+DFc3bwgyx2aiQ8NRbaJEmkK2D4HkPfxBS9lQ4oyiRz5lhocQQDKuWm8nv+HW1tnvD+imdpv4qHKzf1OFTFVVArOiI+sAkEJrcFTeAv7d8QETqfA9A8UqrOn4Rg7dkIJHzqdesWVIfqeNTUcKlvLwv6cpno+JRGfYgMErmpg3qo7BRn4PIYQYJxKGEeEFvplqQTLs5v50HubHTGupLGHoGgAUUAQ67wXIgkqvIE5Jc5ElDGBW9H0lEZ+w+yhPyx3eeBKUcRE7/MM2QBQcIfo8tfcju/21gVOEHZhoyo0Aa64EpHunEZZx2s0JbbIzj9JIUkJEA1W3a+p2Rywz9cAWrpPRUlMVTkTTiCD+nnDXxbXJulZmAQZuzuHnC9gZqZcUoxe55KJfS0T+Q4Y1rJkzfhWcc4GK5FN926DH3ZIV5v8+dk5rO45hw3btX7EZM54j92LQLTk2iy2viDRMK0QIslzWIfR8oh9dZJ1fSUNjPzZNj0aL2IQQ3DNyi64fcbm7X3lXy6vi/BZKMF4O/PpOqgnhCWCCLfAYYQWKNzVifjgdjq5oMXNg/1zsmQ7zwrsuBzr+cdAm8bYZ9mC9+LDNc4IJ0mJqA3H7phBwRTHPOiekgVxoRbVVY4777Ksh1/kMSDeoX+1/CpirxkOSvuOjOQZoG8I4f/dFhXx72cMWbCexTEdY+TQhC+IxhuyFvvma2QEKs310GL0NEmgncsA2R6BC+vqBk87wXZ+vTAWr8hLiz1X//2ZR4NwLHX07hko+FzSbArecrM4FqxXeTPZZpOyqfyZXm1tsOMSfcVKT+aNK3aef+lMdoqSD6Rg2tKSEw1Imzudpj Pu4N0XhN CqWoCVHJP+/Z8LwxmNscIYfQS9pJKCOZ3ADAK3I6ODDTQJyAOX1YDianlnIAoJ8mll9lS2LdxvjD+fahJGDZByZLaVNzm02o4Ayszml0t2o/j8d5FnXz35jMdjAnKXKAIO9Re1voKQoauQqpzScPKxAmT57qsZFMD4oTdnK+o7GewCUw/vnlmROZh/Rj5MN4qX9tajWwmPhPORqrkykFtqhTfnf94IFv843JMGsAMNAOjbgA/mVELW13hnmDObNz4iF2o8suwFufRnLlMFbR7sH4x/mOSnjsB/MSNKv/gfE5oww7pYYviYoaPJIfnSGGJ6E5PAbi6Qvl8+hqQqTnNWECv/CzlEUBKublCOQnjTDv9MX2ZpEwYcBDfolYeSQ2xp20JP8pkrehUkuLnpyu8zGlTQH92oS+GOgurb2/LPciojMbODOtNS0jB3qWo/8A5ePw0ovfv6/PQpr9R9b9E9o3M0+93XiWHyV0aQahxXvlUVhk3zzLrxLWR0vNSles542djROREG0yZNT+mIo0TZerwfA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Everything is in place to stop using the per-page mapcounts in large folios: the mapcount of tail pages will always be logically 0 (-1 value), just like it currently is for hugetlb folios already, and the page mapcount of the head page is either 0 (-1 value) or contains a page type (e.g., hugetlb). Maintaining _nr_pages_mapped without per-page mapcounts is impossible, so that one also has to go with CONFIG_NO_PAGE_MAPCOUNT. There are two remaining implications: (1) Per-node, per-cgroup and per-lruvec stats of "NR_ANON_MAPPED" ("mapped anonymous memory") and "NR_FILE_MAPPED" ("mapped file memory"): As soon as any page of the folio is mapped -- folio_mapped() -- we now account the complete folio as mapped. Once the last page is unmapped -- !folio_mapped() -- we account the complete folio as unmapped. This implies that ... * "AnonPages" and "Mapped" in /proc/meminfo and /sys/devices/system/node/*/meminfo * cgroup v2: "anon" and "file_mapped" in "memory.stat" and "memory.numa_stat" * cgroup v1: "rss" and "mapped_file" in "memory.stat" and "memory.numa_stat ... can now appear higher than before. But note that these folios do consume that memory, simply not all pages are actually currently mapped. It's worth nothing that other accounting in the kernel (esp. cgroup charging on allocation) is not affected by this change. [why oh why is "anon" called "rss" in cgroup v1] (2) Detecting partial mappings Detecting whether anon THPs are partially mapped gets a bit more unreliable. As long as a single MM maps such a large folio ("exclusively mapped"), we can reliably detect it. Especially before fork() / after a short-lived child process quit, we will detect partial mappings reliably, which is the common case. In essence, if the average per-page mapcount in an anon THP is < 1, we know for sure that we have a partial mapping. However, as soon as multiple MMs are involved, we might miss detecting partial mappings: this might be relevant with long-lived child processes. If we have a fully-mapped anon folio before fork(), once our child processes and our parent all unmap (zap/COW) the same pages (but not the complete folio), we might not detect the partial mapping. However, once the child processes quit we would detect the partial mapping. How relevant this case is in practice remains to be seen. Swapout/migration will likely mitigate this. In the future, RMAP walkers could check for that for that case (e.g., when collecting access bits during reclaim) and simply flag them for deferred-splitting. Signed-off-by: David Hildenbrand --- .../admin-guide/cgroup-v1/memory.rst | 4 + Documentation/admin-guide/cgroup-v2.rst | 10 ++- Documentation/filesystems/proc.rst | 10 ++- Documentation/mm/transhuge.rst | 31 +++++-- include/linux/rmap.h | 35 ++++++-- mm/internal.h | 5 +- mm/page_alloc.c | 3 +- mm/rmap.c | 80 +++++++++++++++++-- 8 files changed, 150 insertions(+), 28 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v1/memory.rst b/Documentation/admin-guide/cgroup-v1/memory.rst index 286d16fc22ebb..53cf081b22e81 100644 --- a/Documentation/admin-guide/cgroup-v1/memory.rst +++ b/Documentation/admin-guide/cgroup-v1/memory.rst @@ -609,6 +609,10 @@ memory.stat file includes following statistics: 'rss + mapped_file" will give you resident set size of cgroup. + Note that some kernel configurations might account complete larger + allocations (e.g., THP) towards 'rss' and 'mapped_file', even if + only some, but not all that memory is mapped. + (Note: file and shmem may be shared among other cgroups. In that case, mapped_file is accounted only when the memory cgroup is owner of page cache.) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 175e9435ad5c1..53ada5c2620a7 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -1448,7 +1448,10 @@ The following nested keys are defined. anon Amount of memory used in anonymous mappings such as - brk(), sbrk(), and mmap(MAP_ANONYMOUS) + brk(), sbrk(), and mmap(MAP_ANONYMOUS). Note that + some kernel configurations might account complete larger + allocations (e.g., THP) if only some, but not all the + memory of such an allocation is mapped anymore. file Amount of memory used to cache filesystem data, @@ -1491,7 +1494,10 @@ The following nested keys are defined. Amount of application memory swapped out to zswap. file_mapped - Amount of cached filesystem data mapped with mmap() + Amount of cached filesystem data mapped with mmap(). Note + that some kernel configurations might account complete + larger allocations (e.g., THP) if only some, but not + not all the memory of such an allocation is mapped. file_dirty Amount of cached filesystem data that was modified but diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index c9e62e8e0685e..3c37b248fc4f1 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -1153,9 +1153,15 @@ Dirty Writeback Memory which is actively being written back to the disk AnonPages - Non-file backed pages mapped into userspace page tables + Non-file backed pages mapped into userspace page tables. Note that + some kernel configurations might consider all pages part of a + larger allocation (e.g., THP) as "mapped", as soon as a single + page is mapped. Mapped - files which have been mmapped, such as libraries + files which have been mmapped, such as libraries. Note that some + kernel configurations might consider all pages part of a larger + allocation (e.g., THP) as "mapped", as soon as a single page is + mapped. Shmem Total memory used by shared memory (shmem) and tmpfs KReclaimable diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index baa17d718a762..0e7f8e4cd2e33 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -116,23 +116,28 @@ pages: succeeds on tail pages. - map/unmap of a PMD entry for the whole THP increment/decrement - folio->_entire_mapcount, increment/decrement folio->_large_mapcount - and also increment/decrement folio->_nr_pages_mapped by ENTIRELY_MAPPED - when _entire_mapcount goes from -1 to 0 or 0 to -1. + folio->_entire_mapcount and folio->_large_mapcount. We also maintain the two slots for tracking MM owners (MM ID and corresponding mapcount), and the current status ("maybe mapped shared" vs. "mapped exclusively"). + With CONFIG_PAGE_MAPCOUNT, we also increment/decrement + folio->_nr_pages_mapped by ENTIRELY_MAPPED when _entire_mapcount goes + from -1 to 0 or 0 to -1. + - map/unmap of individual pages with PTE entry increment/decrement - page->_mapcount, increment/decrement folio->_large_mapcount and also - increment/decrement folio->_nr_pages_mapped when page->_mapcount goes - from -1 to 0 or 0 to -1 as this counts the number of pages mapped by PTE. + folio->_large_mapcount. We also maintain the two slots for tracking MM owners (MM ID and corresponding mapcount), and the current status ("maybe mapped shared" vs. "mapped exclusively"). + With CONFIG_PAGE_MAPCOUNT, we also increment/decrement + page->_mapcount and increment/decrement folio->_nr_pages_mapped when + page->_mapcount goes from -1 to 0 or 0 to -1 as this counts the number + of pages mapped by PTE. + split_huge_page internally has to distribute the refcounts in the head page to the tail pages before clearing all PG_head/tail bits from the page structures. It can be done easily for refcounts taken by page table @@ -159,8 +164,8 @@ clear where references should go after split: it will stay on the head page. Note that split_huge_pmd() doesn't have any limitations on refcounting: pmd can be split at any point and never fails. -Partial unmap and deferred_split_folio() -======================================== +Partial unmap and deferred_split_folio() (anon THP only) +======================================================== Unmapping part of THP (with munmap() or other way) is not going to free memory immediately. Instead, we detect that a subpage of THP is not in use @@ -175,3 +180,13 @@ a THP crosses a VMA boundary. The function deferred_split_folio() is used to queue a folio for splitting. The splitting itself will happen when we get memory pressure via shrinker interface. + +With CONFIG_PAGE_MAPCOUNT, we reliably detect partial mappings based on +folio->_nr_pages_mapped. + +With CONFIG_NO_PAGE_MAPCOUNT, we detect partial mappings based on the +average per-page mapcount in a THP: if the average is < 1, an anon THP is +certainly partially mapped. As long as only a single process maps a THP, +this detection is reliable. With long-running child processes, there can +be scenarios where partial mappings can currently not be detected, and +might need asynchronous detection during memory reclaim in the future. diff --git a/include/linux/rmap.h b/include/linux/rmap.h index c131b0efff0fa..6b82b618846ee 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -240,7 +240,7 @@ static __always_inline void folio_set_large_mapcount(struct folio *folio, folio_set_mm_id(folio, 0, vma->vm_mm->mm_id); } -static __always_inline void folio_add_large_mapcount(struct folio *folio, +static __always_inline int folio_add_return_large_mapcount(struct folio *folio, int diff, struct vm_area_struct *vma) { const mm_id_t mm_id = vma->vm_mm->mm_id; @@ -286,9 +286,11 @@ static __always_inline void folio_add_large_mapcount(struct folio *folio, folio->_mm_ids |= FOLIO_MM_IDS_SHARED_BIT; } folio_unlock_large_mapcount(folio); + return new_mapcount_val + 1; } +#define folio_add_large_mapcount folio_add_return_large_mapcount -static __always_inline void folio_sub_large_mapcount(struct folio *folio, +static __always_inline int folio_sub_return_large_mapcount(struct folio *folio, int diff, struct vm_area_struct *vma) { const mm_id_t mm_id = vma->vm_mm->mm_id; @@ -331,7 +333,9 @@ static __always_inline void folio_sub_large_mapcount(struct folio *folio, folio->_mm_ids &= ~FOLIO_MM_IDS_SHARED_BIT; out: folio_unlock_large_mapcount(folio); + return new_mapcount_val + 1; } +#define folio_sub_large_mapcount folio_sub_return_large_mapcount #else /* !CONFIG_MM_ID */ /* * See __folio_rmap_sanity_checks(), we might map large folios even without @@ -350,17 +354,33 @@ static inline void folio_add_large_mapcount(struct folio *folio, atomic_add(diff, &folio->_large_mapcount); } +static inline int folio_add_return_large_mapcount(struct folio *folio, + int diff, struct vm_area_struct *vma) +{ + BUILD_BUG(); +} + static inline void folio_sub_large_mapcount(struct folio *folio, int diff, struct vm_area_struct *vma) { atomic_sub(diff, &folio->_large_mapcount); } + +static inline int folio_sub_return_large_mapcount(struct folio *folio, + int diff, struct vm_area_struct *vma) +{ + BUILD_BUG(); +} #endif /* CONFIG_MM_ID */ #define folio_inc_large_mapcount(folio, vma) \ folio_add_large_mapcount(folio, 1, vma) +#define folio_inc_return_large_mapcount(folio, vma) \ + folio_add_return_large_mapcount(folio, 1, vma) #define folio_dec_large_mapcount(folio, vma) \ folio_sub_large_mapcount(folio, 1, vma) +#define folio_dec_return_large_mapcount(folio, vma) \ + folio_sub_return_large_mapcount(folio, 1, vma) /* RMAP flags, currently only relevant for some anon rmap operations. */ typedef int __bitwise rmap_t; @@ -538,9 +558,11 @@ static __always_inline void __folio_dup_file_rmap(struct folio *folio, break; } - do { - atomic_inc(&page->_mapcount); - } while (page++, --nr_pages > 0); + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) { + do { + atomic_inc(&page->_mapcount); + } while (page++, --nr_pages > 0); + } folio_add_large_mapcount(folio, orig_nr_pages, dst_vma); break; case RMAP_LEVEL_PMD: @@ -638,7 +660,8 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, do { if (PageAnonExclusive(page)) ClearPageAnonExclusive(page); - atomic_inc(&page->_mapcount); + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + atomic_inc(&page->_mapcount); } while (page++, --nr_pages > 0); folio_add_large_mapcount(folio, orig_nr_pages, dst_vma); break; diff --git a/mm/internal.h b/mm/internal.h index e33a1fc5ed667..bbedb49f18230 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -84,6 +84,8 @@ void page_writeback_init(void); */ static inline int folio_nr_pages_mapped(const struct folio *folio) { + if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) + return -1; return atomic_read(&folio->_nr_pages_mapped) & FOLIO_PAGES_MAPPED; } @@ -719,7 +721,8 @@ static inline void prep_compound_head(struct page *page, unsigned int order) folio_set_order(folio, order); atomic_set(&folio->_large_mapcount, -1); - atomic_set(&folio->_nr_pages_mapped, 0); + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + atomic_set(&folio->_nr_pages_mapped, 0); if (IS_ENABLED(CONFIG_MM_ID)) { folio->_mm_ids = 0; folio->_mm_id_mapcount[0] = -1; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e3b8bfdd0b756..bd65ff649c115 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -955,7 +955,8 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) bad_page(page, "nonzero large_mapcount"); goto out; } - if (unlikely(atomic_read(&folio->_nr_pages_mapped))) { + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT) && + unlikely(atomic_read(&folio->_nr_pages_mapped))) { bad_page(page, "nonzero nr_pages_mapped"); goto out; } diff --git a/mm/rmap.c b/mm/rmap.c index 8de415157bc8d..67bb273dfb80d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1258,6 +1258,16 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, break; } + if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) { + nr = folio_add_return_large_mapcount(folio, orig_nr_pages, vma); + if (nr == orig_nr_pages) + /* Was completely unmapped. */ + nr = folio_large_nr_pages(folio); + else + nr = 0; + break; + } + do { first += atomic_inc_and_test(&page->_mapcount); } while (page++, --nr_pages > 0); @@ -1271,6 +1281,18 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, case RMAP_LEVEL_PMD: case RMAP_LEVEL_PUD: first = atomic_inc_and_test(&folio->_entire_mapcount); + if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) { + if (level == RMAP_LEVEL_PMD && first) + *nr_pmdmapped = folio_large_nr_pages(folio); + nr = folio_inc_return_large_mapcount(folio, vma); + if (nr == 1) + /* Was completely unmapped. */ + nr = folio_large_nr_pages(folio); + else + nr = 0; + break; + } + if (first) { nr = atomic_add_return_relaxed(ENTIRELY_MAPPED, mapped); if (likely(nr < ENTIRELY_MAPPED + ENTIRELY_MAPPED)) { @@ -1436,13 +1458,23 @@ static __always_inline void __folio_add_anon_rmap(struct folio *folio, break; } } + + VM_WARN_ON_FOLIO(!folio_test_large(folio) && PageAnonExclusive(page) && + atomic_read(&folio->_mapcount) > 0, folio); for (i = 0; i < nr_pages; i++) { struct page *cur_page = page + i; - /* While PTE-mapping a THP we have a PMD and a PTE mapping. */ - VM_WARN_ON_FOLIO((atomic_read(&cur_page->_mapcount) > 0 || - (folio_test_large(folio) && - folio_entire_mapcount(folio) > 1)) && + VM_WARN_ON_FOLIO(folio_test_large(folio) && + folio_entire_mapcount(folio) > 1 && + PageAnonExclusive(cur_page), folio); + if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) + continue; + + /* + * While PTE-mapping a THP we have a PMD and a PTE + * mapping. + */ + VM_WARN_ON_FOLIO(atomic_read(&cur_page->_mapcount) > 0 && PageAnonExclusive(cur_page), folio); } @@ -1548,20 +1580,23 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, for (i = 0; i < nr; i++) { struct page *page = folio_page(folio, i); - /* increment count (starts at -1) */ - atomic_set(&page->_mapcount, 0); + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + /* increment count (starts at -1) */ + atomic_set(&page->_mapcount, 0); if (exclusive) SetPageAnonExclusive(page); } folio_set_large_mapcount(folio, nr, vma); - atomic_set(&folio->_nr_pages_mapped, nr); + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + atomic_set(&folio->_nr_pages_mapped, nr); } else { nr = folio_large_nr_pages(folio); /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); folio_set_large_mapcount(folio, 1, vma); - atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED); + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED); if (exclusive) SetPageAnonExclusive(&folio->page); nr_pmdmapped = nr; @@ -1665,6 +1700,19 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, break; } + if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) { + nr = folio_sub_return_large_mapcount(folio, nr_pages, vma); + if (!nr) { + /* Now completely unmapped. */ + nr = folio_nr_pages(folio); + } else { + partially_mapped = nr < folio_large_nr_pages(folio) && + !folio_entire_mapcount(folio); + nr = 0; + } + break; + } + folio_sub_large_mapcount(folio, nr_pages, vma); do { last += atomic_add_negative(-1, &page->_mapcount); @@ -1678,6 +1726,22 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, break; case RMAP_LEVEL_PMD: case RMAP_LEVEL_PUD: + if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) { + last = atomic_add_negative(-1, &folio->_entire_mapcount); + if (level == RMAP_LEVEL_PMD && last) + nr_pmdmapped = folio_large_nr_pages(folio); + nr = folio_dec_return_large_mapcount(folio, vma); + if (!nr) { + /* Now completely unmapped. */ + nr = folio_large_nr_pages(folio); + } else { + partially_mapped = last && + nr < folio_large_nr_pages(folio); + nr = 0; + } + break; + } + folio_dec_large_mapcount(folio, vma); last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) {