From patchwork Mon Feb 24 16:55:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988494 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63364C021BC for ; Mon, 24 Feb 2025 16:56:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5A656B0082; Mon, 24 Feb 2025 11:56:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D0ACA6B0083; Mon, 24 Feb 2025 11:56:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAC586B0088; Mon, 24 Feb 2025 11:56:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A00C06B0082 for ; Mon, 24 Feb 2025 11:56:15 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 34A2F161992 for ; Mon, 24 Feb 2025 16:56:15 +0000 (UTC) X-FDA: 83155441110.07.03E0D4C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id E0E9E1C000B for ; Mon, 24 Feb 2025 16:56:12 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hL5bbURV; spf=pass (imf18.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416173; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8Yy4w/XZlsv8uxY4mojETBma8CfgELpSYNnd4DKCaPQ=; b=EexDPEWcCVD+sIPmEzwPQ0FDjM7V69o+x5r7mGmDMyVMoGM+PCMLFGWNXPdIGAgzdEy4oP cvqQG1yRCqeKMjeqNa5VNCFklf5ZetbfQdLl6K1Y2QfPszFbt918DtNULIrin/87DISjpt xKZSeni42h5qy74hJh5RQ80dQSZEgtM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416173; a=rsa-sha256; cv=none; b=4heAVOJ7b5yissCvFi5qxwDBGDLvynqsxGBbiEupFUYki40uqzQGh5ZdcIk1TfksV4ruI9 VDSTfjRma1hK2CMQhRPf7LK/DB32JSxuPYLngHATpvOReCs4DfqpK8IxB5wwIt+LI9/voq OMCnCthF1QEJqaYp4Y7XYZBiidFf+4Q= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hL5bbURV; spf=pass (imf18.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416172; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8Yy4w/XZlsv8uxY4mojETBma8CfgELpSYNnd4DKCaPQ=; b=hL5bbURV2MM22caGFxaMEY4cYA0B5dnJXePp3ENJu0/FQm4xso12wFU7JFKhC056dv1Cgx pIWdMDJUk6hqZ8ABq53D+aHtxrB5fk9vrM6tyz86ngiaEG+XgZYl+ZrHpR0lWU8WGnGWXL KrBkDIWSAtE1BCIpfzN1g8AzOPNEms4= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-364-3NM1LBHpPrmtMkU0e3pu2w-1; Mon, 24 Feb 2025 11:56:09 -0500 X-MC-Unique: 3NM1LBHpPrmtMkU0e3pu2w-1 X-Mimecast-MFC-AGG-ID: 3NM1LBHpPrmtMkU0e3pu2w_1740416168 Received: by mail-wr1-f72.google.com with SMTP id ffacd0b85a97d-38f27bddeeeso4482748f8f.3 for ; Mon, 24 Feb 2025 08:56:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416168; x=1741020968; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8Yy4w/XZlsv8uxY4mojETBma8CfgELpSYNnd4DKCaPQ=; b=XxL6XXQmnKg4aVwkEbGDFVByYBQ+1ATJ1j1SyAmXRsLi1JdBnnlNgK8W7323D7gfty lyv6dlJJ5RS+gCmpwvqK5aBlyOUp/7zlp8Ihk4dvSoY20HTcaT5/CQtj+4DPAk1U5tVR Lkc0w5RMN5fWJ3JIPxb+7ONvNzfvQrTyYa3lldNXM7wE9BaaE++wrEXwE5zKNGv6WPkI It9ic6/5aojwjWMm8Mlkq1SU8fxTV33ZsshPabc30xIBBlWvONV/Os6nmOJvOMo8dXhR S2N7kljUfdinl7tAASMY4muTOM0h3pekioM/2UVevIOviqvJb2H809ugMxWvJE1NDIol xFmg== X-Forwarded-Encrypted: i=1; AJvYcCWkIUGBtKbH25pqRvGGzkcnB2h5Yp+HTZyyNPjONeObJBR1e1q4mCfOVTkx3Y73HOfNXvobXCcL1A==@kvack.org X-Gm-Message-State: AOJu0YzPT1mb5LbJ0k9Xqd4Iy7KsD0d8y4r3x+ueGiNndO4c+lvAOKhz FZb5yeKRIGRjWpyHvmLjoxnWsH1KS5dPfPPuu/3CMdEYPBP+FaZ48m9EuyExAXgpw+ujwvPmdRL LBYvznqTT2r8t/uTogE1sEZt4gPFkpY322rrOgFT8Wd2Z3o2d X-Gm-Gg: ASbGncvlLh+psYEnPkjwbpO4CRSyQHqzivgOWsXo30vYgAmPxyO4QDqjv5+qSpBVPpW dUz/Ruv7tmQZ7GuXbhJWNPiVXN+S5CBFIciPUlSeWNL96H3sWYJVU61JfiO/kgd199OebDl1rE/ Gri6n29KLkyvtauRm/Jj+7P2iAx/M8fXMmkxEBiv4OVB9TjyO889YPqXyg8kBvu7+EK+ANB19Pl ZFpxOdDyI7ck+3VZHDhnDmSJ6z+L/BNQSak8Fj09kB+Bcqh/NCGszdlQXZQTudLnPqDuWKWWRQ6 2uDb6M0i59Flp6dFgpQ5par9FPLdeowvpxfipENd/A== X-Received: by 2002:a05:6000:1ac7:b0:38f:2a99:b377 with SMTP id ffacd0b85a97d-38f6f0c4523mr13709702f8f.53.1740416167618; Mon, 24 Feb 2025 08:56:07 -0800 (PST) X-Google-Smtp-Source: AGHT+IHrYsCtuTnI9EW4wn4ZrbygGJqAwQI73Wf+OvRO5jakiYr702W8tJwU3BfJXILKECpWzIC3lQ== X-Received: by 2002:a05:6000:1ac7:b0:38f:2a99:b377 with SMTP id ffacd0b85a97d-38f6f0c4523mr13709656f8f.53.1740416167205; Mon, 24 Feb 2025 08:56:07 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38f259d5e92sm32611493f8f.66.2025.02.24.08.56.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:06 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Lance Yang , "Kirill A. Shutemov" Subject: [PATCH v2 01/20] mm: factor out large folio handling from folio_order() into folio_large_order() Date: Mon, 24 Feb 2025 17:55:43 +0100 Message-ID: <20250224165603.1434404-2-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: dU14NwpvoRzUHBe_ZddvrAQoWNidNGcr_CH5lp9MhOA_1740416168 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Queue-Id: E0E9E1C000B X-Rspamd-Server: rspam07 X-Stat-Signature: gg3f1sdh1x7uewqtopura98iieu8iugt X-HE-Tag: 1740416172-18334 X-HE-Meta: U2FsdGVkX1/z8AGG20431wEGBK0pywg6rPivZgfke9Bu1YQVWAFAQK/bQ5O4LErikI96dFKZZSQKWvsOqrJNkKr7dHjS8Bqnv5XLJRQLU81IWkecgK0aXZU9HBRM1EO1LGOyTgabKJhyDXjIEyaVshJQ2qjHsWijtyG5HjSIrPzdS7p38SY23ktUxhrrDjqw0akRd4tAOvBnz7RuCvkKUXH4HWO31HA6Qe9Rl8H+klOMKQL7KuxJVHP2gDhzsC3VsyOVcQ43wQXChGl5vKsY5BVRvBfswwgSAI6ZDriDERsvwBnJFqw6TVdKXdYw3lpEe5z1gfIR+3FQQIaHx3AAmMO6XbxbiC+UoZQITX7CXVroe4FE/fjRZ6w0FNnkQ2WcOzbQyzdCQcvR0CjM5zf5Q0AsrmRlr0hE8YE5y461Ml0+stWSbODVcnWn2ptZJHVMee7b83NriUaQcohXWLa3uonzfVRU3XBnvCv2JZF0esSFSUzc9Xxt3osAKILugq0pYXSxgurqEwCDK9ujxzpYDwM+p6sDxER8zzyLbXzTMyZ/gjlVI0us9G/vjADomjrDPR/jBVae8gR/bBLgEYUuKe2kyvKJviC7DkIFRTh5NmloIs5Ek+3gSFJdk3D+qGslx2d1gFQcFrRZMBZfANwweSQQdxJQRPEGVgbLQQNMC044YCxR1dGBxF1qZTEFTqNJKJEpfAMYsrlbp2PvMixdm+MJoy1mgUZLShijIDcZfsdYFgMTYymxjdR6iPm3mqWTEYBJnvml1gbOS67Wc12kfxDGrOwl7ciJGwtcrhs4/oeWG0ja/hIZG2m598QWLw0y8OYB7moJEvsgKFBD81UhQ98AUkAz/qal/hjpwHwdSonnFkwwhrXFmGG3CAAgJgbhhQlFYGPRkQAVFuWdmXn2cAacCBv00c9GHzAG1gP90YJY9FQjI/kmrwpRxiIKdcYZFgRi1SvNWL8wkBx6nIw uIy3VLX+ kSkiNHCkZxGsy8wn5weIDM8cYSjpugWI0/KQYHcIJHbQleqDh+RVsL9gZmrNiWDf4/jDM7PT+WWkVDvbmQUdnoqT6b8zRiTVsUdhQqybdNa6fsYtGlRgGPmwkxz0lHCI9j12W/JQOyw4gEhhhtkIrY6+0P69Jgxj81Xo7GDBg8ZxQRbKHcUApsg3Pr5kjQzDWA8O9kgmc3/J+0mELb3epp+KsI3c6zJAZekct5ybyjr9Rg5J3WMXicEimScHm9uSep31FOuPhKLVDJVEi5Hx6I7w8q8EMu54LGV9ezRWgunfc+X8mNz9oW7ZHxPVNxoT/Q60oa6AAlezUh5kZqlBucSSRoxxijFpdL8wmKFocT2Iq2VwVkj5z/2l46zhUoOidFWV5jwSeIWuWrE9rP9h4OgLxK0Xf543G4mAfwf5pzuh2D+JO6X2XwTjzLyWpJ5CCP3XQb9LV6A8mSF1y2bpcgAPtny/1N/vFeXDkBUuilHU3ghPHNxWb5h7l+mu+XM6VoR1Ooi6+Phpg/+qh7V746gxivMJqAvwJVRGwiFPJfnymgmeRky/zqdtb/Fr+1sPqmticRZVMIfbkAMS4w4/fkWJedw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's factor it out into a simple helper function. This helper will also come in handy when working with code where we know that our folio is large. Maybe in the future we'll have the order readily available for small and large folios; in that case, folio_large_order() would simply translate to folio_order(). Reviewed-by: Lance Yang Reviewed-by: Kirill A. Shutemov Signed-off-by: David Hildenbrand --- include/linux/mm.h | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d82feabbe44f8..d00214e22a174 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1194,6 +1194,11 @@ struct inode; extern void prep_compound_page(struct page *page, unsigned int order); +static inline unsigned int folio_large_order(const struct folio *folio) +{ + return folio->_flags_1 & 0xff; +} + /* * compound_order() can be called without holding a reference, which means * that niceties like page_folio() don't work. These callers should be @@ -1207,7 +1212,7 @@ static inline unsigned int compound_order(struct page *page) if (!test_bit(PG_head, &folio->flags)) return 0; - return folio->_flags_1 & 0xff; + return folio_large_order(folio); } /** @@ -1223,7 +1228,7 @@ static inline unsigned int folio_order(const struct folio *folio) { if (!folio_test_large(folio)) return 0; - return folio->_flags_1 & 0xff; + return folio_large_order(folio); } #include @@ -2139,7 +2144,7 @@ static inline long folio_nr_pages(const struct folio *folio) #ifdef CONFIG_64BIT return folio->_folio_nr_pages; #else - return 1L << (folio->_flags_1 & 0xff); + return 1L << folio_large_order(folio); #endif } @@ -2164,7 +2169,7 @@ static inline unsigned long compound_nr(struct page *page) #ifdef CONFIG_64BIT return folio->_folio_nr_pages; #else - return 1L << (folio->_flags_1 & 0xff); + return 1L << folio_large_order(folio); #endif } From patchwork Mon Feb 24 16:55:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DF13C021A4 for ; Mon, 24 Feb 2025 16:56:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9177C6B0088; Mon, 24 Feb 2025 11:56:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C4F06B0089; Mon, 24 Feb 2025 11:56:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65380280001; Mon, 24 Feb 2025 11:56:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4614C6B0088 for ; Mon, 24 Feb 2025 11:56:17 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E60CCA3CE6 for ; Mon, 24 Feb 2025 16:56:16 +0000 (UTC) X-FDA: 83155441152.02.E098CF9 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 6D206140008 for ; Mon, 24 Feb 2025 16:56:14 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QaB8nhGw; spf=pass (imf26.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416174; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uW/SnbGZfnRd/ZrL3KWYJ9GZqWtDH7XgQk5JORWTALU=; b=qBpxeMKVYjr/bQ8BtQeeXpISkzR+c0y1pL28dQqsd57UmTKU2SFYJTbjAoqloEX0+19sgs a5+jAj44xeeked/KvJ55nLuRb9Nqd+CIf/AKBmRWjwKThziJMaeJnURfLZ+91d/4faLsRI zNA1dnOQtTpguM2zKdsG7FLtV6DKnZ0= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QaB8nhGw; spf=pass (imf26.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416174; a=rsa-sha256; cv=none; b=5ggatU+ZLTj+E6kmlcQg+i1eGoo75ClS1XG+kPhXET69cHjGKzioXJt5bMCT46k52Qt9oJ ozAAM+bRm1A+uoJLlLmSZDnC1Daq2rppXplS1WmNONIVS1HbjYqxW9rVgsaNalkQYzhPd0 XppKlC+cYUdxV+wKM9dv2x7PA/z+OUc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416173; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uW/SnbGZfnRd/ZrL3KWYJ9GZqWtDH7XgQk5JORWTALU=; b=QaB8nhGwzCv91IpV1xDNyyMsDfFYhJE67m//m0sIICe0gyjv3U4o9XJtK8rayIL8Coxjmi /+B/pJMEjHL14I6JaI1HhJ/Xe0kQLHd7NtXPYPl9pkXX/oMwq7ajziW9S9Fsj/uHzGIc0j We4w6GSQPpaiWa+f6VZtx501BbKr/Zk= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-208-RaO7pPhSM9KhxYXAfaNMRA-1; Mon, 24 Feb 2025 11:56:11 -0500 X-MC-Unique: RaO7pPhSM9KhxYXAfaNMRA-1 X-Mimecast-MFC-AGG-ID: RaO7pPhSM9KhxYXAfaNMRA_1740416170 Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-43947979ce8so19945465e9.0 for ; Mon, 24 Feb 2025 08:56:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416170; x=1741020970; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uW/SnbGZfnRd/ZrL3KWYJ9GZqWtDH7XgQk5JORWTALU=; b=k8Tnl8ykYMGF5UAzDcX4E5m0Zdka8Ola7wn5fx4r+6MDNFmdGj/ADBjOaHX57eNDzq aRHU80tjQ5g6OwBecgMVoEkvOlo/FPP23AWKxw4mypUWMcxHBan9k6b00QZawYocXO62 /0HTwR6LdKdonBHIuvri5Fa4yHRUX/0KVul82yOleKG2OO+RTuQ7HE03gDHb+d9VjQqI Q/cnoany3NWxXSrZvzKDziubPBsfbAeEU/u+HTMycZTtl0f9a+hrDU/CpUlOCDILcC/O EFo6VphNWJx9R4CchvCehZz73egZGxbRi6+GcJvfFu9bYsbziLO32Ape+kP3FsR4OmkN aeQA== X-Forwarded-Encrypted: i=1; AJvYcCUtVMNIS5E20KrgUoJlL5XvDfC2KEK6kUrdL/xdUdy1/VSbK4q21Py1js6UqhVxHiNz09dilgK6aw==@kvack.org X-Gm-Message-State: AOJu0YxgqTDKNwKD8Y38IO3EnGGonBJf2GETDV3IJ/13za5Qrdiax671 IDVwEqbApZagcvzzhiRgPEHzV4KIXUdp4Q2nd1y/0GDp3Vg+k0T2bBnvF88g2C8jz7sG/p5917G 5/pGz1PzeHL06DKB6Jo8Urpxq4vo0Twz5CjAJ/QnX1k7tsrdn X-Gm-Gg: ASbGnct2zo3YsmUJVA2f+2FQ76TNnvmd1gobprZ/3xpKNY7BkIzby7fEb8rARNnhX1B iDakxh1d5FdspKS0PaHMMK1xmWBUhj8xkRWAI1zMJ65trKPbFz5q1/W6GRBoalh51yI/B5wce1Z a9YK7Wa+5HH7u4rE3rdIFDNL4eefvKA1hthWli5NrDd/zFIWihzEpBQr0NP8I//DvN6CiThyr3J 17BLjgwJaN9mx1wZy7yXB5lKynDg+khlZVdzAHAETWHco5PjWZL0wmIbfVdQhKUw3dLc8k3P4rC AgODPqU6EASwhmLxpmIEiCg0PbAIUMLaqK8ICgWfBQ== X-Received: by 2002:a05:600c:34c2:b0:439:9828:c447 with SMTP id 5b1f17b1804b1-439aecf1426mr121409385e9.17.1740416169806; Mon, 24 Feb 2025 08:56:09 -0800 (PST) X-Google-Smtp-Source: AGHT+IFMOxvVak9cDaBV2o8+q98fYh3JQOIg7zVYXNoNjPQAK1ZeuqEa4rOOZBLfaDjBIQaJ00l68A== X-Received: by 2002:a05:600c:34c2:b0:439:9828:c447 with SMTP id 5b1f17b1804b1-439aecf1426mr121408975e9.17.1740416169419; Mon, 24 Feb 2025 08:56:09 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-439b02f3eaesm111237045e9.24.2025.02.24.08.56.08 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:08 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , "Kirill A. Shutemov" Subject: [PATCH v2 02/20] mm: factor out large folio handling from folio_nr_pages() into folio_large_nr_pages() Date: Mon, 24 Feb 2025 17:55:44 +0100 Message-ID: <20250224165603.1434404-3-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: i4_e4gEhOEnfI9WxjGqrB5EwoIFLOi8-YeO13i0p-O4_1740416170 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 6D206140008 X-Stat-Signature: 3sgxxyuyn188sktcyyx8yjh617ze19ey X-HE-Tag: 1740416174-31242 X-HE-Meta: U2FsdGVkX1+vPmir3/8tmnUtsiuT3KiQHTUJVnhvkvQOoCZTGgUsyEwQsoBipbp564t1Fqzev7VQNNuDqQ5e9BY+0k/GdP/V8JKVztP4yV952kEzF6gWr+iQ42tfhkxJ5buTRQVB2akZhfYWTJjq4n5JJmK3ZLd9URv/5QIlX1i36xl4WlqcGtUS0YEZB6xY83fHCSRev4Vz0qZiUbCbH3Nf/QYWCfYASjZZc+HiMGnrRppKOmBNaae16CKMd3o/d+CIBxG3LNLfKxzy7EbZmjLUR5oPf3fAR7mjmt/PIaQV8/UsnJDnN9rLItaAjqUyJ/59uaAvPJH2kAxJifHs9qitZHmMtzoB8vYRNqOeDw+Zc/ygpGOP1febc0OQbpOAZTqIbnQmdXcumjPJQpP3I2axPppuKW6LyjRrJaU9nOpcNGJx6j8k9xuIsQ4bdedSRByMafrxAtOt1EZZ0aiZSdVup4wQ90K4tvffqmkNOMHpPle7g3mlbHFuDAiaqCBZOAL9gn8sBAHf+atwl2NNAPBu247LO1/Q7lZU2YvFuJAFrF1Rxyb0AcccLOVHa9fM9ip+rb1v2lleUBsIv5jsMDujtHHddc7XXn3hWqAGLeDN064bQbOOfbz3qLI0gOxj/9YfeyAWM5g2PfsfUiGpu7Q7hgcKFcym/KlCaiyD69ONEi2sIMmrY6Ik/tD/qj3Z0RoczWtLljjZj+yEOqxQhi0hDm4zTvylAyzIuk+WeDzr2z26BhG8E3I6yQf3967sc7G2QtKdJo9vgGWKFV3nimMuNuFDKmajTrMnROWEMnHn8TzPUsKxnghn3vhOr5AmW4XXuSCi6Le0LblTKTEgOuVjCmac0Z5/miBq8rjB9F0hGoylfHP7AzDUMm/W4f5NjDGHNlApNJ0bJiSj16KWtacfzYzwmJhm98rgxmDwSq+OjOGPNdsyrNUyNkj2FtS2A7La1GHgZMPgrG5JJTE uFQagtqz 58lkXkV6zqdg2C56O3H6I88y2Ym/EUKyrmY0JZCbseXvoL6TBxo/wAwp7EeHVfGo1KeOJe5QhDis7B23MNjjrgAxUjqRi3RaEaiBDBBw8iJE5r2l4px1AibPp9skACq3r+FPQHQSY7QjuqMxs5EQwJHchnf4OqdqO88SqF8Nu9s34kZJPGZ9T4zEXlzVWA3wnK8T28Z1JdfZI9LJTs/wn4VwMzWk/F66c/PAG+zbq/+fQ3V2e9htzKIazNtHRWeVcAZLoZ2YsaQKSqHjJDxP59xp9/kYnRGNlo9aOjPMXTvn/H5Y5Zng7TZaL9KC+w8GFnoTZ01KCoUyTik34z72HoEI4FOpRgdN9N4ZFohJYvFW4/yMKFFLR7z1XEsbY2KeHWgxI7kjVt59N/Znrsx35XK0N9O6iedxfCX3OC3bw1z7e5ReAG6HoJjZEOS9TgJl/WzE5MBclJuiktGd/EueWjLhk3v5tyrio8OHuDnhpI9CMllehoCtdmekUVuNKftrSWltBn/48Ukzx1QVMkbRoy+Z5UUEkozDjQB3m3s0+rL/qWafPP63a+ZYP20WSxeg3njcz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's factor it out into a simple helper function. This helper will also come in handy when working with code where we know that our folio is large. While at it, let's consistently return a "long" value from all these similar functions. Note that we cannot use "unsigned int" (even though _folio_nr_pages is of that type), because it would break some callers that do stuff like "-folio_nr_pages()". Both "int" or "unsigned long" would work as well. Maybe in the future we'll have the nr_pages readily available for all large folios, maybe even for small folios, or maybe for none. Reviewed-by: Kirill A. Shutemov Signed-off-by: David Hildenbrand --- include/linux/mm.h | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d00214e22a174..7c5a8fd29cfcd 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1199,6 +1199,18 @@ static inline unsigned int folio_large_order(const struct folio *folio) return folio->_flags_1 & 0xff; } +#ifdef CONFIG_64BIT +static inline long folio_large_nr_pages(const struct folio *folio) +{ + return folio->_folio_nr_pages; +} +#else +static inline long folio_large_nr_pages(const struct folio *folio) +{ + return 1L << folio_large_order(folio); +} +#endif + /* * compound_order() can be called without holding a reference, which means * that niceties like page_folio() don't work. These callers should be @@ -2141,11 +2153,7 @@ static inline long folio_nr_pages(const struct folio *folio) { if (!folio_test_large(folio)) return 1; -#ifdef CONFIG_64BIT - return folio->_folio_nr_pages; -#else - return 1L << folio_large_order(folio); -#endif + return folio_large_nr_pages(folio); } /* Only hugetlbfs can allocate folios larger than MAX_ORDER */ @@ -2160,24 +2168,20 @@ static inline long folio_nr_pages(const struct folio *folio) * page. compound_nr() can be called on a tail page, and is defined to * return 1 in that case. */ -static inline unsigned long compound_nr(struct page *page) +static inline long compound_nr(struct page *page) { struct folio *folio = (struct folio *)page; if (!test_bit(PG_head, &folio->flags)) return 1; -#ifdef CONFIG_64BIT - return folio->_folio_nr_pages; -#else - return 1L << folio_large_order(folio); -#endif + return folio_large_nr_pages(folio); } /** * thp_nr_pages - The number of regular pages in this huge page. * @page: The head page of a huge page. */ -static inline int thp_nr_pages(struct page *page) +static inline long thp_nr_pages(struct page *page) { return folio_nr_pages((struct folio *)page); } From patchwork Mon Feb 24 16:55:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988496 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB3D5C021BC for ; Mon, 24 Feb 2025 16:56:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5E1556B008A; Mon, 24 Feb 2025 11:56:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 590D66B008C; Mon, 24 Feb 2025 11:56:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36F30280001; Mon, 24 Feb 2025 11:56:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EF0B46B008A for ; Mon, 24 Feb 2025 11:56:19 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 762261218FB for ; Mon, 24 Feb 2025 16:56:19 +0000 (UTC) X-FDA: 83155441278.30.C6C2613 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 2BEC14000D for ; Mon, 24 Feb 2025 16:56:17 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cDgaioi0; spf=pass (imf17.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416177; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8Z/i7SxHkRJm72w2k3iTqc1vo3DynQo4UiIchyZFPhI=; b=sARGekFYDYDwAqlDu1NVY/jaGQa5LjDc6Slp7Mv+w7fUGVAX45FXf3zhJV6U5lgJIKEAIp QUaG9x2rzIMbV0Qe104PAMur6xJceeAxtBXnPQOTJRhPDqei3Xc1JG043wkr/hCMd7CCYi WMYLwY3SLA4vKRHyFR+WAu1eQuBypAQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416177; a=rsa-sha256; cv=none; b=rZauYwwjk+hvEhAvaEikRZ1SLR9tWPnh+G3Fd1NrT4ZDNWyjXEa/jo3kk12r2X7LTfERIm 6C4orxom/6XAcKSd8qcRSrTjC2aWOv+HcZlUguMBCvOz/ZiwkhvB3J0GiV/pT6Z6JZunxY HMkttDwvyU7cmC0/dzhS/kWVZmYjQU4= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cDgaioi0; spf=pass (imf17.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416176; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8Z/i7SxHkRJm72w2k3iTqc1vo3DynQo4UiIchyZFPhI=; b=cDgaioi0ZaOpqqYQ5RQ4Y6O2YTU2E1+G6Lo6koYElhFS6qevD3oDicLebdDu/mH6Yvx7QF wJfyKY6vkx6Z4DY0F7f0Qg7uQWmW6EVkSeCzliJuIa5NlP38sIbYnkgrM2kDABLkS2UzFe 7XrMF0tDHeCapndoUkCCS5XY5LqoWs0= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-531-7zETdBKUOE6f1ka6MfGZ9Q-1; Mon, 24 Feb 2025 11:56:13 -0500 X-MC-Unique: 7zETdBKUOE6f1ka6MfGZ9Q-1 X-Mimecast-MFC-AGG-ID: 7zETdBKUOE6f1ka6MfGZ9Q_1740416172 Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-4399d2a1331so22851115e9.1 for ; Mon, 24 Feb 2025 08:56:12 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416172; x=1741020972; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8Z/i7SxHkRJm72w2k3iTqc1vo3DynQo4UiIchyZFPhI=; b=HPwdtELH8/MfNKuT08wwpehebZtrxUzxVAJ/oGsOFlNOeUouDcIeQnFlONRXsS0xc5 zIWv/0bRNbW11iHPZPdkrKvrwsLsLWjGU6+R+KmQfTHfzC4IhMynaWdXLzvVNwfvhKh+ PCTh+aXrHlEwB1PZ5gPFmObGXft/10K2WemdR5cBXhMgsGR0MaUkRuwekTku3UWjQmOY hCgqUPTVEgmgRgG5Ipuc6CoxaRI6iX9UBu5XbnuKE8V0u9HMy5x2iM6xbsYYC8PWwUap T/iH5MOpIrWOmv6RW5aIB6j7HWzsU6D7/LWQKATUxArVTyjENRFZHZ/u37iYgvCHM0OO LRaw== X-Forwarded-Encrypted: i=1; AJvYcCVlx4l34Vqo//vsHJvKiGUC2p7eaYDuAOUjmseyWF2M0YxG3vjwLlefFQBm2MK+GG+UHCWSX1/jcg==@kvack.org X-Gm-Message-State: AOJu0Yyso8jqaYspHBQ7PXK6yvaD9sZIYGXkMu/eej/CTVT5elXn+Csg 7rtzYFCrfke+EttuGwIJ87QRpEUu3hhfzl+tSTGqrEn1fqRBxKobMBvsv6s8LIjFLDituovRVti JEozPCmxNJbbOXB2PaC+fp0bpbFoU8/0yIJygSLkYNm8MeYhH X-Gm-Gg: ASbGnctmx5n417/G9tzwdllZh5AnDPQEBxwImeeECRGZH8vWvDGcjW86S5X38n/4eWZ Wibq/Nf5N/P45LJBf3L8Zhpr2KPQDXj3z/qmyK+0PnW+8PKqBpFn6+PmPSWm5mRPNpFZcBQz7TM XW30aho29HyCzg6AXHmeH0Cee5mAe1a2f354xVlS4JwXihedfgdZDZ2s5qX+nQFx9zoYTV2naV2 I1OOCgBKIVGxaE5rLOMLx1zdHZu4M+64fE8aHaNFtgkegBVPtIFuJaMWLJa4iXoQnC9ugA5TLkD ZK20oz/6X2crc/pBWwIY1Msrd03hCqbs2acYs83g0A== X-Received: by 2002:a05:600c:1c85:b0:439:968b:6669 with SMTP id 5b1f17b1804b1-439ae2d254bmr112029005e9.1.1740416171872; Mon, 24 Feb 2025 08:56:11 -0800 (PST) X-Google-Smtp-Source: AGHT+IGPiVGKd002kgHq/C+YhiMRT6ZEt5tkRc6eny1qWEWThiWAlrLvHc+EEKxBr1Q0DjH+goSmcg== X-Received: by 2002:a05:600c:1c85:b0:439:968b:6669 with SMTP id 5b1f17b1804b1-439ae2d254bmr112028725e9.1.1740416171409; Mon, 24 Feb 2025 08:56:11 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-439b0371b7bsm112170735e9.33.2025.02.24.08.56.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:10 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , "Kirill A. Shutemov" Subject: [PATCH v2 03/20] mm: let _folio_nr_pages overlay memcg_data in first tail page Date: Mon, 24 Feb 2025 17:55:45 +0100 Message-ID: <20250224165603.1434404-4-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: gojpNXwnuMKRVlv2luA32i6RL28uWH4lT6-nKOC6Z0o_1740416172 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Queue-Id: 2BEC14000D X-Rspamd-Server: rspam07 X-Stat-Signature: 9cn86z4qbew59ckqagtj3ruaf4jiwns4 X-HE-Tag: 1740416177-186968 X-HE-Meta: U2FsdGVkX18ZZBwvXRZXKcg+uqpgc4oARZTdCEBWbtBA1bfETnsmYBH7c9Dnf4g6lHCbOu1XkU7n+aOBav41+CpGbCBAdW6lfqUDeV7hBphzCyFEg2qZsiDgf40vhb0hJy+6ahCbmas4xQW7UHUfBq/jLqaZfLtB8xQDBUzkMocyexhQsnzUQRXJ4gRbUDiCZG1a2ew6ceJyJy4YQ7fA9Tn1ni0YzC3SljxQzPzmmuyM0e2BaB4kslXjFgc1VA4LK0bJPZ8V0mFGQ4kzNx1y1Hk0Fyzj22X+RBqBogt6pybgAZBhwk/+NOQRIoTQ8J8YJCZDEIMseaNmkde4KGmrlD42QEdRS6iOWyImUl1SOp8lTJMu4u4tRR4Xc8OECnRucK1bbP+3C6t02zi+3oAfSYW0TfOZTegJ42pzjnaizM6v2IrUWEx33lDh3RjL+xn/bqYV81JK1fzOy9JyIiyqwpuI6MaO4XySYJQSxCaxQGRg/leOXRxcj89lMHF2/U0AVygAXNyl42jlcRVEiAf6xfGes5ZMq9abmr3GqXYSCTe5cs5VlCrLUXB/7LVYGg0Z/jDFFbFIKaGPgBsTtcybzWyQpq/ZmjHITTUdPSYdx6zDGqNOND5M//UKFLa5xx3U2OVYIP4hicFYZE136uzuKJdU6WGtittCVdlz1sZ8ayv5OrXYG9VUVUqhjqmvHqOld1YJjkRCRwZNlD1fAz+P3LSKgNoz0FClup3tdEm2oBT76TLX7dnMS8FkJ+jiUGavidg3OPx5Gk11mzTqcfz2jXwRl2lFeZkHdPAYpaxsYy6JkyXL7xvBLlFfJOjKQJ/atpzmkwTpoSakizjHtSsk9EXJ29NqyxA27vmQ0bijLvhtSWBmpZtdHh8IIJYymRwAJXfIYd1KdoHvwUUWE1ej+lKbH7UunaBjF9VzlqMWanfNDKpw8geY3n4qostyqpRjnlfj+Jk5uf3BMk/9qYO 9bYoEN13 uV1z6HS0J4kSTYxQVMhimK+lWlWi5FMfsCToit2Zk8CwIG6eMat/OJT8BQ0WWLZJpQ3gaIreOGbt+4y3QHwymZH57U8r1Nt7JvTozH2A6Gr4XOMDYVXvuYjryzlCRiWWok+40E7W36ScmGq+XYm7ndNU3xkg/oGXpOlCIOCe8h+atlX7cXShQC9NBhNNEGLswp1Nf6e9dI5vqoJ/hlqJRxT77Qza9r7ylem7/gRPpBAdbopULY7m2PujofSOjtYofzTQ/brSq8LaXvuU/ZchbfIhKH3qEDzWNvN9l+J00NZF+HjplPjPuMeE9sZASm+wY7kt4mlhbcMZpJd+/IbGTii2X03N3c6bXB4dkQi60Y/V3L9ta1B9tfd+HDkkimUjmJhUKCQbkHeTpDcaLTyF0z6WRTZG4J8ZiWJHvqd9h4ug5zAS1NljnOzzQuftz6G1p4y1RSk5LUeFNvSJGz7Hp+fFDrqSocOhlQMwDZUY5s4Z6iWMBTryy+BH+Xcq0FOiYzvs+efwOto9HxWEjU8SCFWGP6G8wzI7uQrkG2BcEOnFvABuvClCCF5XPwAjIVBzM8v/H X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's free up some more of the "unconditionally available on 64BIT" space in order-1 folios by letting _folio_nr_pages overlay memcg_data in the first tail page (second folio page). Consequently, we have the optimization now whenever we have CONFIG_MEMCG, independent of 64BIT. We have to make sure that page->memcg on tail pages does not return "surprises". page_memcg_check() already properly refuses PageTail(). Let's do that earlier in print_page_owner_memcg() to avoid printing wrong "Slab cache page" information. No other code should touch that field on tail pages of compound pages. Reset the "_nr_pages" to 0 when splitting folios, or when freeing them back to the buddy (to avoid false page->memcg_data "bad page" reports). Note that in __split_huge_page(), folio_nr_pages() would stop working already as soon as we start messing with the subpages. Most kernel configs should have at least CONFIG_MEMCG enabled, even if disabled at runtime. 64byte "struct memmap" is what we usually have on 64BIT. While at it, rename "_folio_nr_pages" to "_nr_pages". Hopefully memdescs / dynamically allocating "strut folio" in the future will further clean this up, e.g., making _nr_pages available in all configs and maybe even in small folios. Doing that should be fairly easy on top of this change. Reviewed-by: Kirill A. Shutemov Signed-off-by: David Hildenbrand --- include/linux/mm.h | 4 ++-- include/linux/mm_types.h | 30 ++++++++++++++++++++++-------- mm/huge_memory.c | 16 +++++++++++++--- mm/internal.h | 4 ++-- mm/page_alloc.c | 6 +++++- mm/page_owner.c | 2 +- 6 files changed, 45 insertions(+), 17 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7c5a8fd29cfcd..f6b6373a864dd 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1199,10 +1199,10 @@ static inline unsigned int folio_large_order(const struct folio *folio) return folio->_flags_1 & 0xff; } -#ifdef CONFIG_64BIT +#ifdef NR_PAGES_IN_LARGE_FOLIO static inline long folio_large_nr_pages(const struct folio *folio) { - return folio->_folio_nr_pages; + return folio->_nr_pages; } #else static inline long folio_large_nr_pages(const struct folio *folio) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 689b2a7461892..e81be20bbabc6 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -287,6 +287,11 @@ typedef struct { unsigned long val; } swp_entry_t; +#if defined(CONFIG_MEMCG) || defined(CONFIG_SLAB_OBJ_EXT) +/* We have some extra room after the refcount in tail pages. */ +#define NR_PAGES_IN_LARGE_FOLIO +#endif + /** * struct folio - Represents a contiguous set of bytes. * @flags: Identical to the page flags. @@ -312,7 +317,7 @@ typedef struct { * @_large_mapcount: Do not use directly, call folio_mapcount(). * @_nr_pages_mapped: Do not use outside of rmap and debug code. * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). - * @_folio_nr_pages: Do not use directly, call folio_nr_pages(). + * @_nr_pages: Do not use directly, call folio_nr_pages(). * @_hugetlb_subpool: Do not use directly, use accessor in hugetlb.h. * @_hugetlb_cgroup: Do not use directly, use accessor in hugetlb_cgroup.h. * @_hugetlb_cgroup_rsvd: Do not use directly, use accessor in hugetlb_cgroup.h. @@ -377,13 +382,20 @@ struct folio { unsigned long _flags_1; unsigned long _head_1; /* public: */ - atomic_t _large_mapcount; - atomic_t _entire_mapcount; - atomic_t _nr_pages_mapped; - atomic_t _pincount; -#ifdef CONFIG_64BIT - unsigned int _folio_nr_pages; -#endif + union { + struct { + atomic_t _large_mapcount; + atomic_t _entire_mapcount; + atomic_t _nr_pages_mapped; + atomic_t _pincount; + }; + unsigned long _usable_1[4]; + }; + atomic_t _mapcount_1; + atomic_t _refcount_1; +#ifdef NR_PAGES_IN_LARGE_FOLIO + unsigned int _nr_pages; +#endif /* NR_PAGES_IN_LARGE_FOLIO */ /* private: the union with struct page is transitional */ }; struct page __page_1; @@ -435,6 +447,8 @@ FOLIO_MATCH(_last_cpupid, _last_cpupid); offsetof(struct page, pg) + sizeof(struct page)) FOLIO_MATCH(flags, _flags_1); FOLIO_MATCH(compound_head, _head_1); +FOLIO_MATCH(_mapcount, _mapcount_1); +FOLIO_MATCH(_refcount, _refcount_1); #undef FOLIO_MATCH #define FOLIO_MATCH(pg, fl) \ static_assert(offsetof(struct folio, fl) == \ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1e45064046a0f..13152313e77ce 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3307,9 +3307,10 @@ bool can_split_folio(struct folio *folio, int caller_pins, int *pextra_pins) * It splits @folio into @new_order folios and copies the @folio metadata to * all the resulting folios. */ -static void __split_folio_to_order(struct folio *folio, int new_order) +static void __split_folio_to_order(struct folio *folio, int old_order, + int new_order) { - long nr_pages = folio_nr_pages(folio); + long nr_pages = 1 << old_order; long new_nr_pages = 1 << new_order; long index; @@ -3529,12 +3530,21 @@ static int __split_unmapped_folio(struct folio *folio, int new_order, } } + /* + * Reset any memcg data overlay in the tail pages. + * folio_nr_pages() is unreliable until prep_compound_page() + * was called again. + */ +#ifdef NR_PAGES_IN_LARGE_FOLIO + folio->_nr_pages = 0; +#endif + /* complete memcg works before add pages to LRU */ split_page_memcg(&folio->page, old_order, split_order); split_page_owner(&folio->page, old_order, split_order); pgalloc_tag_split(folio, old_order, split_order); - __split_folio_to_order(folio, split_order); + __split_folio_to_order(folio, old_order, split_order); after_split: /* diff --git a/mm/internal.h b/mm/internal.h index b07550db2bfd1..7f6d5def00fa0 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -684,8 +684,8 @@ static inline void folio_set_order(struct folio *folio, unsigned int order) return; folio->_flags_1 = (folio->_flags_1 & ~0xffUL) | order; -#ifdef CONFIG_64BIT - folio->_folio_nr_pages = 1U << order; +#ifdef NR_PAGES_IN_LARGE_FOLIO + folio->_nr_pages = 1U << order; #endif } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 16dfcf7ade74a..bdfc954dab9aa 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1174,8 +1174,12 @@ __always_inline bool free_pages_prepare(struct page *page, if (unlikely(order)) { int i; - if (compound) + if (compound) { page[1].flags &= ~PAGE_FLAGS_SECOND; +#ifdef NR_PAGES_IN_LARGE_FOLIO + folio->_nr_pages = 0; +#endif + } for (i = 1; i < (1 << order); i++) { if (compound) bad += free_tail_page_prepare(page, page + i); diff --git a/mm/page_owner.c b/mm/page_owner.c index 2d6360eaccbb6..a409e2561a8fd 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -507,7 +507,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret, rcu_read_lock(); memcg_data = READ_ONCE(page->memcg_data); - if (!memcg_data) + if (!memcg_data || PageTail(page)) goto out_unlock; if (memcg_data & MEMCG_DATA_OBJEXTS) From patchwork Mon Feb 24 16:55:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF9E0C021A4 for ; Mon, 24 Feb 2025 16:56:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 397556B008C; Mon, 24 Feb 2025 11:56:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2FA7F6B0092; Mon, 24 Feb 2025 11:56:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0DA4A6B0093; Mon, 24 Feb 2025 11:56:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DE3AC6B008C for ; Mon, 24 Feb 2025 11:56:20 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A9BB616199A for ; Mon, 24 Feb 2025 16:56:20 +0000 (UTC) X-FDA: 83155441320.13.120D982 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 37A6C180015 for ; Mon, 24 Feb 2025 16:56:17 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GvUAQUbR; spf=pass (imf16.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416177; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2J0EAZe5fI1EPFno0gWSdFeb6AH1J2rAdhMuBREbwsI=; b=A/bD0Jw5iUaFbVleeRZDDiL3xjtevCYaGDIYLz8ZUoJSNUn/Nov4e2kVPnJMJ/2gQkp865 DxisSGfGTXvzNpRuqev7e+bi1KZZCWqZGu+eydmI37hhjv4/m2N2ctG5k00jHuwRaL4DA7 S4ByQkLxPfPTQrczBYWGOFbr2Oc/Rpo= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GvUAQUbR; spf=pass (imf16.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416177; a=rsa-sha256; cv=none; b=WYvZ73z66/5M7sxcQQ+GUAEKg3s9Zo9QzwII8UM/3t0eqDid9p+NLIlCdprWcZ10bSiroq LZzQCRL98J6+/hG/Iz/bKjZVUNZOjSPE3kddXfsHp5T8cyOU+MyUP4nuLOh/olxJLytjpS P87Z/9BdqalGFHSIctANF3kadV7EQgM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416176; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2J0EAZe5fI1EPFno0gWSdFeb6AH1J2rAdhMuBREbwsI=; b=GvUAQUbRRQ8LxtskxgLAV5J6UGdXjml2GtTc/dC6XpfSKztktiZoU2nMWXMKU96JKY5WH2 +QNR35H3w9b7T+MzOQWKZR0AKsBXWzM4nqYfVMz51QPY7ou/O9RNtM/78Nn9jcbbAfuvBq mPN5FDi+hanbDhapXwH8GkhtoO/tj2k= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-593-1L6u5aBYOlq2ALk6OzFoYQ-1; Mon, 24 Feb 2025 11:56:15 -0500 X-MC-Unique: 1L6u5aBYOlq2ALk6OzFoYQ-1 X-Mimecast-MFC-AGG-ID: 1L6u5aBYOlq2ALk6OzFoYQ_1740416174 Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-38f455a8fcaso3573126f8f.2 for ; Mon, 24 Feb 2025 08:56:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416174; x=1741020974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2J0EAZe5fI1EPFno0gWSdFeb6AH1J2rAdhMuBREbwsI=; b=jFSKDkXOl7eUIMCSF32jvOL31Qu8FSEoUhZMUrsdJ+iXaoj4x9AJwo00ggO6InqOq6 EPjfmNEnabKOaAQgMACoi59Gi0HoI62qfPB1k3S49YCDS9H0IbfAObk8yGnrhudt9s4V QQCcB/hE7wSAEAM4a/tMVpwAgwk0y1J9wRw217U3S5IboX/eqttJWdZZO1Ow1mC7ODcA PSnhEYS5S+JLx5tYf2lMsAUUecOYQ7X1MfmtZan9X3PT58/Y9pdCgcoJbff+1i4U7LkK XH3/X3QBh6cv3qwc41eFtS7q3OITiy755j7FWSOOQTEoMFb1aB2S+DPIR7jdnZMZtOnu R8eA== X-Forwarded-Encrypted: i=1; AJvYcCW8aiFZxjj62TNg45tUWCmjrxvOJbvwtNP2sUe5bc9Kb5TTzV9UPHWNcrw+ANDgOXH76CyM21TzDg==@kvack.org X-Gm-Message-State: AOJu0YzKR6nQztd9/UH/NuNwxsRgHyGwnl73RPPaGxLHm9ap7zfYEb46 wg5WjjVks0NENj8cjRAPddjbj6aBkzAvWocwvQzL0PO4fzdoQDq1qvLlh3KdlmbuErNQkq1lknd YiXY7bTbwsiocmyD72gQy/PpVzNsxppdd/rvn2ux27Mh3+4T4 X-Gm-Gg: ASbGncsUNqjBUywaaokrlT8Ntkada4nTXRZeRt0jaJ6NkB4nmINV63jy71/gaJXe+Hu F9WQqwj2iTvg0GVEzhHzyFJY8c8fxnUQ9lqJk7LGw9Yhk4qqxEUflBv/imkhyoAtWNktxGasO7/ VLuN6HyKKf8Iw1tiPRXb494ht7Vpe36LS5IX0jkbCMMnQROfW7eRPKEa2hi7qW2/0SzGW8LkNSG L+GsZIpeAxvlMCGVjL+QM3Ps3Ds8ixqGiHi1mAMXPJQApcJkgi3/GHTB7/A33cbaTddi0vp7vW4 N8x1VSkURwbmxzlMz/0QfnuGZDouaFBE5PlbhU89HQ== X-Received: by 2002:a05:6000:178c:b0:38f:32ac:7e55 with SMTP id ffacd0b85a97d-38f6f0bd36dmr13581419f8f.48.1740416173936; Mon, 24 Feb 2025 08:56:13 -0800 (PST) X-Google-Smtp-Source: AGHT+IHEK4+EskiCQe6b4Tt/1yywMxIN3N8u+em5znQyoHp6oO/UQ3gXRmnxMS/Oum/lKFSKXiOoHw== X-Received: by 2002:a05:6000:178c:b0:38f:32ac:7e55 with SMTP id ffacd0b85a97d-38f6f0bd36dmr13581391f8f.48.1740416173570; Mon, 24 Feb 2025 08:56:13 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38f258ddbe0sm32403003f8f.39.2025.02.24.08.56.12 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:13 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 04/20] mm: move hugetlb specific things in folio to page[3] Date: Mon, 24 Feb 2025 17:55:46 +0100 Message-ID: <20250224165603.1434404-5-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Q_7W7wf0YhstYVe3ix8oYRSErUHT9yv30f8yNNFaw1w_1740416174 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Queue-Id: 37A6C180015 X-Stat-Signature: oxrfhy3a73s6cx61goz1n6fu87qmcnfk X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1740416177-461529 X-HE-Meta: U2FsdGVkX199lWyFDc2Ag+o39YOMDyzKO3POaZFo01eypqHB26ccSUea51/5WYZ0mnG3h1hQoNc5fh7IBZvwIL1hDxC/9/b0VThtuXYpsCrgKhUWXfeh45tpL3n2VZzlxU+EOi+r56FVj7GvABEDT/rjsPK67KH/oQGTiH0IZIBQJD55GS91jEkhaqQ7KAdP5Uj1lhybchOQLkRnLQMAaDGEVBwN5RBeUw6w7m5ANjAy4oErEoLRg2GholAwqpbnpN8ju4f9OBxb4F+AeGfp95jOky+3NDqgMa5fn2xltJfkTMQojnY0wI9A8Z/4NUSUVomJQATkcZdS8H0UOI34e1Pu2dk7ls3FMamX6ZTmNJG3d177udIPOh1sXmobCaSpKFOnuj6niXnnu1yEFS4FA8vS3pbOVf9yWX0+NzHkhOfiHqynPS9YHC5H3uvIZF0HbcElcFEdw1RvhOHippuD8Q6Ocl7GfgcF4jckORaWrTdcAP7ROsuUZaD1zU8LA4nnwQHC3R2hPDVSX9fvDtgwLQALwO8ft3J8jOZpvPS+N0VWTk/zSOVpr+EcsgP+8cjwJ3NTngM+eQ3Zp6Jv+9nglLYt/SzT6KcMtgdvYwPzsjn15p4HGcutBhmOMdbHyyIMCLIBDkSupklhdIVdNHEiSWhFoCjTe+vvYaTu+EHa+xlzR0V3air/CRLu+BBMnNQfFXqhxJUrFMhXUCvIJT210Vu1vOvAOMlZagDinYNrP2GSdUHuqikEEAwjRyZeiVf4Ev0k7FQyqzrPDA4Hlg5S+qPHqb0M4Cyx1NyG4G4yO7Uy7ysptK/9AswyztDH5o4AuxRWxxXt2gWLt4b/EZGtYzDvGGCS7+4kn4FMrSxY9NqyfiRJ/o65fRTO8gTm9I708HiV/e1llKRm+C0REngbeNvn0FJ3zTUMgFWtxk+KqTOOScYR9hpIVtQDfG54OcVZnqSvssMrZXP9l5rBtJu 99VViGM1 mCyIAPYLiLJEVrxiRWWuUSsq4kXcrt5L1R538kb6vmsrHufqUaPID+yGwKiSM7cBQu7xQRtISGDqbIA3li/68jQmTgQQPk9pC9fRM9VFhlhYPslQgXCe6KAIR3AAuXdgITki2sHemAUNqgIIBa3yJj/oLml00u6fwiiN38Z+HsJErrdHn4NoHGmvedPL7d9EGKGcScRUR4ueVG6foHxslGRbBWdP1q8dFlLKiG4H2bLASMhCFjmEfKQUmyqEK7AuujADxIn35egzGS6KcTc5zxx7lP+DbKi1GOTBRLBIm2YB4VoWSn/1cshsIlp6pUfanAMxI/SFcxEDDzzP0DMB8RAxnSbCJIh3y0kebo9qd6xhdLxCjo8KLSc3+HuuTGGnD1p+Y/HdcXGTnzjtUHE++wCRiOwYh9VGQxgn91LiZ2031BMkFGAgDOpwjKiiy0Ga2j9cU1qbMS6Xra9WQ7GWMjoUAVExqWMUCwrYOxJbMl/XuB39xmzt/HtF5Fw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's just move the hugetlb specific stuff to a separate page, and stop letting it overlay other fields for now. This frees up some space in page[2], which we will use on 32bit to free up some space in page[1]. While we could move these things to page[3] instead, it's cleaner to just move the hugetlb specific things out of the way and pack the core-folio stuff as tight as possible. ... and we can minimize the work required in dump_folio. We can now avoid re-initializing &folio->_deferred_list in hugetlb code. Hopefully dynamically allocating "strut folio" in the future will further clean this up. Signed-off-by: David Hildenbrand --- include/linux/mm_types.h | 27 +++++++++++++++++---------- mm/hugetlb.c | 1 - mm/page_alloc.c | 5 +++++ 3 files changed, 22 insertions(+), 11 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index e81be20bbabc6..1d9c68c551d42 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -405,20 +405,23 @@ struct folio { unsigned long _flags_2; unsigned long _head_2; /* public: */ - void *_hugetlb_subpool; - void *_hugetlb_cgroup; - void *_hugetlb_cgroup_rsvd; - void *_hugetlb_hwpoison; + struct list_head _deferred_list; /* private: the union with struct page is transitional */ }; + struct page __page_2; + }; + union { struct { - unsigned long _flags_2a; - unsigned long _head_2a; + unsigned long _flags_3; + unsigned long _head_3; /* public: */ - struct list_head _deferred_list; + void *_hugetlb_subpool; + void *_hugetlb_cgroup; + void *_hugetlb_cgroup_rsvd; + void *_hugetlb_hwpoison; /* private: the union with struct page is transitional */ }; - struct page __page_2; + struct page __page_3; }; }; @@ -455,8 +458,12 @@ FOLIO_MATCH(_refcount, _refcount_1); offsetof(struct page, pg) + 2 * sizeof(struct page)) FOLIO_MATCH(flags, _flags_2); FOLIO_MATCH(compound_head, _head_2); -FOLIO_MATCH(flags, _flags_2a); -FOLIO_MATCH(compound_head, _head_2a); +#undef FOLIO_MATCH +#define FOLIO_MATCH(pg, fl) \ + static_assert(offsetof(struct folio, fl) == \ + offsetof(struct page, pg) + 3 * sizeof(struct page)) +FOLIO_MATCH(flags, _flags_3); +FOLIO_MATCH(compound_head, _head_3); #undef FOLIO_MATCH /** diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 9faa1034704ff..2ad5c292568ab 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1646,7 +1646,6 @@ static void __update_and_free_hugetlb_folio(struct hstate *h, folio_ref_unfreeze(folio, 1); - INIT_LIST_HEAD(&folio->_deferred_list); hugetlb_free_folio(folio); } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index bdfc954dab9aa..05a2a9492cdb0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -971,6 +971,11 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) goto out; } break; + case 3: + /* the third tail page: hugetlb specifics overlap ->mappings */ + if (IS_ENABLED(CONFIG_HUGETLB_PAGE)) + break; + fallthrough; default: if (page->mapping != TAIL_MAPPING) { bad_page(page, "corrupted mapping in tail page"); From patchwork Mon Feb 24 16:55:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988498 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6100AC021A4 for ; Mon, 24 Feb 2025 16:56:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8F0766B0093; Mon, 24 Feb 2025 11:56:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 877A46B0095; Mon, 24 Feb 2025 11:56:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6CA216B0096; Mon, 24 Feb 2025 11:56:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4BB3A6B0093 for ; Mon, 24 Feb 2025 11:56:22 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0389DC097B for ; Mon, 24 Feb 2025 16:56:21 +0000 (UTC) X-FDA: 83155441404.09.4B9D9F7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf29.hostedemail.com (Postfix) with ESMTP id BF7F8120012 for ; Mon, 24 Feb 2025 16:56:19 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gmzURY23; spf=pass (imf29.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416179; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v9Kk0sDLIOZEo9TGeBWuftLWUgUisx0QoodxZudqT8w=; b=atOhKYH4D5u2IpJ8uFiJ7S6wzqkCIqwvZjP3Si9NC3lTKdAV5+k9LOtETvHzXv52a98nRb BJaJlVlnzOcOix/aBDD6LxgHQk8cJMw5sqzUvyBziUZFrM6zvi4LBp9es/OG08TTU616lI WatvGkjWzt9W6KFUGJrLnd0JLZjxh1g= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gmzURY23; spf=pass (imf29.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416179; a=rsa-sha256; cv=none; b=kAqZ2mmp0rIkXzw+eKx/KkBqIMRkeH7/X1fc8N/as7gsUFYswA7u64whCuZbBPMcGAMF7J ZnVqhFdl5Xh4Yn+tvKJDoM5PP6dndnjI8WTylJARR7ROKdErECTjJ2sFNLYxqXx977YeWM GOzE6RIcOynH564uVgj8GTGFCdVzH5Y= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416179; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v9Kk0sDLIOZEo9TGeBWuftLWUgUisx0QoodxZudqT8w=; b=gmzURY23HMitCNJcpq5XOaXuPYE+zEo6bUmbGTUszPCwe3Q4ZZ5G5pCZbT72RJJtrg/ZhT YU3N8e+odeHfKo3gtdDd47OViXmkSg3eWGTQpFF7eTgJMeJD7mulZEVckwwXL1M1nl3cyC Oz3hdYfl8BcF+ctOvFeo9HKIjKKHc6M= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-463-xGRp438EOx65OZxnqyRDow-1; Mon, 24 Feb 2025 11:56:17 -0500 X-MC-Unique: xGRp438EOx65OZxnqyRDow-1 X-Mimecast-MFC-AGG-ID: xGRp438EOx65OZxnqyRDow_1740416176 Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-4388eee7073so24508825e9.0 for ; Mon, 24 Feb 2025 08:56:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416176; x=1741020976; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v9Kk0sDLIOZEo9TGeBWuftLWUgUisx0QoodxZudqT8w=; b=rxGi3tNPTBP5I2P6gjtVJZXhsFvdsh3IBCKubJHpL1nYb9V/25fD8/qrQ2hToGkNxc j326n1S35mDruL3CZJP52r0+zd2btFUbHy16gBxirCjp0DfUKci0a4NkoeIgQMmdXR3C tXaG51Ng4Fj2mh8oxz2Dqq1yJR9NhsU8Vjnulu9OBT6qfgaSzaXokoBN0ea3y6jLZ35Y lijL61/b4s2kmqKuThQhbGpM1Tn+z8EaUIpRH5Fp5fu4DkDi8EEABsqHrpitFZtTtgmz 7yCEm6Uigg7rmbr4oYjvzTm0B34HtsaXxSpeewp/iAHbPEjtcedgBh5XS7yTqhpl6Crb uaow== X-Forwarded-Encrypted: i=1; AJvYcCUbHJOugdsdsKkDNfms4xFXOiZjP2Np0JySVsFfdRHFr4hyhzQaEMvmUJdNUG5A2jJ1E6DUPevdjg==@kvack.org X-Gm-Message-State: AOJu0Yw2k2pLJDgLKVeiXarA41xI/qZ0pf6yPyDymTTzgc8YBkLfxGIc cx1WwDKr4q4/K7kgkDaqHhWU9mS3Jz6k1lDLLGQyGIbSG4rCtQPEcccjvBSqIVPsSF1Q6aArcUd srrcZpVjTc4a30+G+rYl7g1UOKgLMMuYdH8yL2EpIOZ7Ghp9e X-Gm-Gg: ASbGncsSKtxbl1fF3YI8392uGoWM7b3mA34CtOlAk0iEwZeUZurlpPsq0NivAjpG5f1 1IgqVjH2jqqtj9h56PVFHErdvDIIKnG6BbxiLUiwajAyRlBC6x/kuPIGCoQyjCCfTpPWHdFT4DF 9BF8eDZF9IMZX1QZwTynCIXSMWYtziWT4GxGA/oCknahCeY7RD23+qNRy4QmQTYGsiaefEbZv47 OnzEdhZVsvWW4nkCmdc8L4q8ryg6s8PdwzAG8Sbqb9iW55wcSWFTF5vZTXOZQL0USacJM2Prp53 GYU6SjTYYdasG6WYr981QQE2tZxf3D2yuu1YU7XtlA== X-Received: by 2002:a05:600c:5949:b0:439:8345:17a6 with SMTP id 5b1f17b1804b1-439a30d38e9mr130585495e9.12.1740416176144; Mon, 24 Feb 2025 08:56:16 -0800 (PST) X-Google-Smtp-Source: AGHT+IGy0+PSAOTGdTFyYGF1ZzYS11kOEi6LhvuqHcevrUwSSRUG4LJeLXU4Rq4oj/c975sX5HkQ4A== X-Received: by 2002:a05:600c:5949:b0:439:8345:17a6 with SMTP id 5b1f17b1804b1-439a30d38e9mr130585275e9.12.1740416175731; Mon, 24 Feb 2025 08:56:15 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-439b02ce404sm113163905e9.7.2025.02.24.08.56.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:15 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 05/20] mm: move _pincount in folio to page[2] on 32bit Date: Mon, 24 Feb 2025 17:55:47 +0100 Message-ID: <20250224165603.1434404-6-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: j8FX-8JVjs0GPzZVRBJkoBNQvT7R2AzhcvDuBFq1ccg_1740416176 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Queue-Id: BF7F8120012 X-Stat-Signature: g8dawwfes99epn43u99o6mgk56e6ibu6 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1740416179-122205 X-HE-Meta: U2FsdGVkX1/m+nWOlCIThvYJ77kd4I8aaQXiJJfqgtFzyMeamKJHZC4hodiAtJu250R34Qw/UAfiltYDU32Irh6ktsfDDUtCJtDhx2L6MeVSdNEKWTnvjv5dEBoCwY1qXVQBk/0zQP9Q5ZnjRCDZsSCN7G+XgIru8SvPv2EO4Nr3c+OTVcDE/ZsO2b9Zi373e/wRim8wh+Id8+D/Nw/1E0Amnnf2vEUYZ7Ok266mN/F+TKYL96QS13l4HuqmsUT5rfLhp7GyMTKrdFWn0TmJrqFP2QG2quiEB6/C3Jf1/uwWUIBqCGXZM6d7sILKxD0DOS7EDbc1oL52Vd17p5pMH5SO8EJgOm+XEZ0hMzDIGXyJ3SqKoA9PQVD3dzg0Cwt02n+pISmUpDwKmHUqDsJMZocUdadzMf1QjAp332UCuKx7IWHGE8ntmxZKmp+ISBNkUkgeVMFiJ2lHcTcmMt3f1EDPT7Qhw5LCabaHm4HdimYg5F+p0+8zp+uMJHxwgJ5Dtv/3wsFemq0P24uOR41Wxmo7wnY1Yl/6zt/Wg2gqkSHrpvm4cjv1qa+OjJwdHFQoJbJIQpiTnMMnMumUgPHW3teVkMBi1RJHQl4XF7imS70MxR60RlKXHjOAMVGnlw+G5cKpTKnbsAxdW0XvmCOmuhnpc2AAB887FqiTXWkRflWFOUzxeKpTSONgr5cLM2eyLdXyigEVJmei/0p3nQXIDtT7Kzh1JzUp8+O2J7xluq3ZxmVIy3baJF15VupZH5EPGH0+Z6yRXaGRhXBWtG6Kk69jZJ0vwp8l+R8nu3P4qQ+YxEsSMhQKqL/G/1JD4cqX3HvOVIrAxdOZpfVGFP5o1JMCeRgchZh/3opa4zmrYy3BWRyo3Rxt+7K4w4ob5VnAe3i2ZIPxPafiL3T4Z4D7BmjwwdmAvypRnbYxiMtS0fPZuvcuz34QoyLtuP8PrdAXK7OUxXOSDAgGeHyZza3 uwZUBlgi urD3qQwCGUP2nJcHJYhW9oTxJBuuRDCBR43jwtJDrlorAPbH+AETXjJnpuoyJcAVFmiCD2dFSyZEcZv/igDaLPatMLMCb7xOk6dJ5eWMhx38IUfprWBfoKBUJUiT5wWamUYKy1Osmm3a8jm9JIX+rIy1y1nRBq4gC/u+ht2Soy5eQXGpybqonQtomBgn78zvGjWExM5QTfFpCCjxp3PwOqikY90c2CxWxyFaywVtOmSVX2dL1u6CNFmMO26u8DBX4p1MkZDN5Q/yVeG0Ut9f1ruCLsNEivhW/Rm76bAgG/B23gaanjT0gjOd1lg+n3tnHDDQf0h/XU5kOu9dTAhHMMnAWekEEy2ln4GCsDXRf0jdzfGUhKB6EQ0q1Qf+M3mrSJGTtLHxXfXFdsodaFuqHgn5MNa5RzDjeVtxVs92bykZgJEkzGo3xgKabbJYTpbNW22jr93pL6R2bpntZY2oa3UhVNeRvy+FrAgDZ6e/DyFL/IuSSNydMjeMhsk8wAaSDXkVGRLinTgo6xUOAyOFfv6AShV0VFruYaZvvMh55TpXCQslrgSxUPjXZwe+sF7m5fPETKf/vjGHKbGs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's free up some space on 32bit in page[1] by moving the _pincount to page[2]. For order-1 folios (never anon folios!) on 32bit, we will now also use the GUP_PIN_COUNTING_BIAS approach. A fully-mapped order-1 folio requires 2 references. With GUP_PIN_COUNTING_BIAS being 1024, we'd detect such folios as "maybe pinned" with 512 full mappings, instead of 1024 for order-0. As anon folios are out of the picture (which are the most relevant users of checking for pinnings on *mapped* pages) and we are talking about 32bit, this is not expected to cause any trouble. In __dump_page(), copy one additional folio page if we detect a folio with an order > 1, so we can dump the pincount on order > 1 folios reliably. Note that THPs on 32bit are not particularly common (and we don't care too much about performance), but we want to keep it working reliably, because likely we want to use large folios there as well in the future, independent of PMD leaf support. Once we dynamically allocate "struct folio", fortunately the 32bit specifics will likely go away again; even small folios could then have a pincount and folio_has_pincount() would essentially always return "true". Signed-off-by: David Hildenbrand --- include/linux/mm.h | 11 +++++++++-- include/linux/mm_types.h | 5 +++++ mm/debug.c | 10 +++++++++- mm/gup.c | 8 ++++---- mm/internal.h | 3 ++- mm/page_alloc.c | 14 +++++++++++--- 6 files changed, 40 insertions(+), 11 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index f6b6373a864dd..1a4ee028a851e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2004,6 +2004,13 @@ static inline struct folio *pfn_folio(unsigned long pfn) return page_folio(pfn_to_page(pfn)); } +static inline bool folio_has_pincount(const struct folio *folio) +{ + if (IS_ENABLED(CONFIG_64BIT)) + return folio_test_large(folio); + return folio_order(folio) > 1; +} + /** * folio_maybe_dma_pinned - Report if a folio may be pinned for DMA. * @folio: The folio. @@ -2020,7 +2027,7 @@ static inline struct folio *pfn_folio(unsigned long pfn) * get that many refcounts, and b) all the callers of this routine are * expected to be able to deal gracefully with a false positive. * - * For large folios, the result will be exactly correct. That's because + * For most large folios, the result will be exactly correct. That's because * we have more tracking data available: the _pincount field is used * instead of the GUP_PIN_COUNTING_BIAS scheme. * @@ -2031,7 +2038,7 @@ static inline struct folio *pfn_folio(unsigned long pfn) */ static inline bool folio_maybe_dma_pinned(struct folio *folio) { - if (folio_test_large(folio)) + if (folio_has_pincount(folio)) return atomic_read(&folio->_pincount) > 0; /* diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 1d9c68c551d42..31f466d8485bc 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -387,7 +387,9 @@ struct folio { atomic_t _large_mapcount; atomic_t _entire_mapcount; atomic_t _nr_pages_mapped; +#ifdef CONFIG_64BIT atomic_t _pincount; +#endif /* CONFIG_64BIT */ }; unsigned long _usable_1[4]; }; @@ -406,6 +408,9 @@ struct folio { unsigned long _head_2; /* public: */ struct list_head _deferred_list; +#ifndef CONFIG_64BIT + atomic_t _pincount; +#endif /* !CONFIG_64BIT */ /* private: the union with struct page is transitional */ }; struct page __page_2; diff --git a/mm/debug.c b/mm/debug.c index 2d1bd67d957bc..83ef3bd0ccd32 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -79,12 +79,17 @@ static void __dump_folio(struct folio *folio, struct page *page, folio_ref_count(folio), mapcount, mapping, folio->index + idx, pfn); if (folio_test_large(folio)) { + int pincount = 0; + + if (folio_has_pincount(folio)) + pincount = atomic_read(&folio->_pincount); + pr_warn("head: order:%u mapcount:%d entire_mapcount:%d nr_pages_mapped:%d pincount:%d\n", folio_order(folio), folio_mapcount(folio), folio_entire_mapcount(folio), folio_nr_pages_mapped(folio), - atomic_read(&folio->_pincount)); + pincount); } #ifdef CONFIG_MEMCG @@ -146,6 +151,9 @@ static void __dump_page(const struct page *page) if (idx < MAX_FOLIO_NR_PAGES) { memcpy(&folio, foliop, 2 * sizeof(struct page)); nr_pages = folio_nr_pages(&folio); + if (nr_pages > 1) + memcpy(&folio.__page_2, &foliop->__page_2, + sizeof(struct page)); foliop = &folio; } diff --git a/mm/gup.c b/mm/gup.c index e5040657870ea..2944fe8cf3174 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -109,7 +109,7 @@ static void gup_put_folio(struct folio *folio, int refs, unsigned int flags) if (is_zero_folio(folio)) return; node_stat_mod_folio(folio, NR_FOLL_PIN_RELEASED, refs); - if (folio_test_large(folio)) + if (folio_has_pincount(folio)) atomic_sub(refs, &folio->_pincount); else refs *= GUP_PIN_COUNTING_BIAS; @@ -164,7 +164,7 @@ int __must_check try_grab_folio(struct folio *folio, int refs, * Increment the normal page refcount field at least once, * so that the page really is pinned. */ - if (folio_test_large(folio)) { + if (folio_has_pincount(folio)) { folio_ref_add(folio, refs); atomic_add(refs, &folio->_pincount); } else { @@ -223,7 +223,7 @@ void folio_add_pin(struct folio *folio) * page refcount field at least once, so that the page really is * pinned. */ - if (folio_test_large(folio)) { + if (folio_has_pincount(folio)) { WARN_ON_ONCE(atomic_read(&folio->_pincount) < 1); folio_ref_inc(folio); atomic_inc(&folio->_pincount); @@ -575,7 +575,7 @@ static struct folio *try_grab_folio_fast(struct page *page, int refs, * is pinned. That's why the refcount from the earlier * try_get_folio() is left intact. */ - if (folio_test_large(folio)) + if (folio_has_pincount(folio)) atomic_add(refs, &folio->_pincount); else folio_ref_add(folio, diff --git a/mm/internal.h b/mm/internal.h index 7f6d5def00fa0..d33db24c8b17b 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -723,7 +723,8 @@ static inline void prep_compound_head(struct page *page, unsigned int order) atomic_set(&folio->_large_mapcount, -1); atomic_set(&folio->_entire_mapcount, -1); atomic_set(&folio->_nr_pages_mapped, 0); - atomic_set(&folio->_pincount, 0); + if (IS_ENABLED(CONFIG_64BIT) || order > 1) + atomic_set(&folio->_pincount, 0); if (order > 1) INIT_LIST_HEAD(&folio->_deferred_list); } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 05a2a9492cdb0..3dff99cc54161 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -959,9 +959,11 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) bad_page(page, "nonzero nr_pages_mapped"); goto out; } - if (unlikely(atomic_read(&folio->_pincount))) { - bad_page(page, "nonzero pincount"); - goto out; + if (IS_ENABLED(CONFIG_64BIT)) { + if (unlikely(atomic_read(&folio->_pincount))) { + bad_page(page, "nonzero pincount"); + goto out; + } } break; case 2: @@ -970,6 +972,12 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) bad_page(page, "on deferred list"); goto out; } + if (!IS_ENABLED(CONFIG_64BIT)) { + if (unlikely(atomic_read(&folio->_pincount))) { + bad_page(page, "nonzero pincount"); + goto out; + } + } break; case 3: /* the third tail page: hugetlb specifics overlap ->mappings */ From patchwork Mon Feb 24 16:55:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988499 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 426D2C021A4 for ; Mon, 24 Feb 2025 16:56:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E7196B0096; Mon, 24 Feb 2025 11:56:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 375E76B0098; Mon, 24 Feb 2025 11:56:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14AEE280001; Mon, 24 Feb 2025 11:56:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C94F86B0096 for ; Mon, 24 Feb 2025 11:56:24 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 35B55C183C for ; Mon, 24 Feb 2025 16:56:24 +0000 (UTC) X-FDA: 83155441488.16.4600D63 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 7D918140020 for ; Mon, 24 Feb 2025 16:56:21 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YyW74i2v; spf=pass (imf23.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416181; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JowchFKNsu8Ay0dQlyPFoFvu2rxlXy5D20/ejxoZdUQ=; b=sisWeQZZ08GAuopx/IUiLXfJzORAzzXNv9Qw9EXOhlnAIWWi+00oTGU50JZqNLicEmG5a3 kQdhLaxadI4vbXfr+4Tl9K0qMvXzPrq3PNYTAiHMSDgdTj+g3ZJ6DzpLk4p4qXgNeC8fCm 2WqVSwmeYo4EQqhjFzSW8qwW0g7zPsg= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YyW74i2v; spf=pass (imf23.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416181; a=rsa-sha256; cv=none; b=x7nP9xwJ2a6dtQCmr/UTeAjIt5XTnXeGw5c+XoEHdlXFFYYMb/0ShSZiMcu+qNef1ZFjEd YCbNfgWQCwQxACF7DsJQaLMjwTPTzy40URJ7bOOuGsWWWVTYWnIvWETx+d6n1Q0aDctA02 SdjC1ZzCRNb9ue4oL3bEAn4lLNcRT18= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416180; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JowchFKNsu8Ay0dQlyPFoFvu2rxlXy5D20/ejxoZdUQ=; b=YyW74i2vQogoWbjmYljrA7rBG43uw/jzeoHagACtzlsch3oQ7EjHx3IqAShoxfAGkwq0vQ pcQIznnOu52NdaOuBrVy6eysdgzltLefZKAQkB3EVM8b6Sj9FlxTko1C7ge0ekCxxh1CIH SMVBOU/3SRt4kpXphdDhNHU+jEob3GU= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-261-ttA0LvF0PoSA5mKDrjWMkQ-1; Mon, 24 Feb 2025 11:56:19 -0500 X-MC-Unique: ttA0LvF0PoSA5mKDrjWMkQ-1 X-Mimecast-MFC-AGG-ID: ttA0LvF0PoSA5mKDrjWMkQ_1740416178 Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-38f4bf0d5faso2472195f8f.0 for ; Mon, 24 Feb 2025 08:56:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416178; x=1741020978; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JowchFKNsu8Ay0dQlyPFoFvu2rxlXy5D20/ejxoZdUQ=; b=E4BjraOZ04k85FxRQUDiWOw+nB9o+pMUrsBIrkZ/8+5U6bZJE8mZCvjiq1m9mf/HCH IMfXy1Ky4kOev7obr/70uuief9/Xq/D0Dv403kh1e/Ye8l5ws94zUZ4LKEfvlMdVJyPp E4XvsgF/Fqj9HFu/rNovy4OQajZnKNPIrgz14bw3t5PfuqgGg7gTtzTs+l4mxax6qCSJ szijQZo2iUZ0ZD0EV4pzylmB+w3UhoCXpW5XKdXGKbRrr35IyUJCNPgBFnZdNDUG62Kp iSQK9AuC8KWjv7HbtgoWjR5p/wdWw4lfTD89AKcIHDGeVe1DTqRxl4NfLYfdAyVLTY0H Igvw== X-Forwarded-Encrypted: i=1; AJvYcCVF2RTFXQMcqcWfRWVEUfgfkUPYT4xXGSmBa1Xett/NZy7PCsKo52b7ItG+wtLpcKv0iWpZ8PriDg==@kvack.org X-Gm-Message-State: AOJu0YxNVz+lJYRNotN33JbPA/fSZJuuktSM7y528/6xriKrit8RE/8m QQkCYGQH2KPwIMt9tK6dKmWwNIGB9AllQR/HDjh5OZ101bfYl+bFt7f4iKLMkm6FgWT+hgUnjNu zfRNnuz0o/PVo+XMRN2TPAMAfHFXusPpFe1rhSPuibu20FjzO X-Gm-Gg: ASbGncsBAhJf1lvj1lwGOB1IpixkIihdtJyB5piuwZ/M596LhKEppwRSTtK+KpioZ3M fsfI1s3ye/4Anpdo2aVZaCdLV+EUFLvtTz6/sj/2HGj63n/rJUwElIzfaLb3n5CURltOH4JRDGN exprSZoxGXsuOFJMgDu6uqGIleDEFt8tECsAb8jwGWsRLT16SlJTiUPPQM6Uy1ja3Ixu3Fh8RNG 4fwaWCen7gw+Aths/s1btt5qx6vYO2w8Vq7lR/TskSvwESlnovjiOmfawbXAHcKWO4RI1DW6GIV tJZBvh0XyijV/w+Cf0iv/UfN3IfNugfTxM+qQhFSNg== X-Received: by 2002:a5d:47a3:0:b0:38d:d8fb:e90f with SMTP id ffacd0b85a97d-38f6e975ca7mr11045687f8f.24.1740416178105; Mon, 24 Feb 2025 08:56:18 -0800 (PST) X-Google-Smtp-Source: AGHT+IEppyEZ/CiRop1NWtbK9zceuiQ/u5I/zpQhUebwrOc1yf1KZs6E/F1s5c1KapHNpyvpSRQcvA== X-Received: by 2002:a5d:47a3:0:b0:38d:d8fb:e90f with SMTP id ffacd0b85a97d-38f6e975ca7mr11045649f8f.24.1740416177739; Mon, 24 Feb 2025 08:56:17 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38f258f5fb6sm31629683f8f.44.2025.02.24.08.56.16 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:17 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 06/20] mm: move _entire_mapcount in folio to page[2] on 32bit Date: Mon, 24 Feb 2025 17:55:48 +0100 Message-ID: <20250224165603.1434404-7-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: PIV85dd0NfopDVD1xma1_UZoAlb4vD5-YNGnF5QODGE_1740416178 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7D918140020 X-Stat-Signature: 4znno5p65n8i3im6aojeo5m7ww1emrcu X-Rspam-User: X-HE-Tag: 1740416181-362999 X-HE-Meta: U2FsdGVkX19SBm/TS7fH6BVYW/wF89L1Q4aDn7TtKZr5wZzKxNPSq8IO/zuYf9vuR4tUcvwe/88EVXN2PCdMEoHilH1ctvIBk1M8UNK4QP1cEVG0nyHZti8VOCyM13UBK0EDC6nlYdE0tqy67tQObyNVzBcnQoJFSAw1ZaH0K411Mv0aK+WSH+Q/00s+CCxs01vLZITlkyZ4r9i3X/C46BhfMMKiH8nmKcutd1hezM5rhr1Nyu4Jlko0+mabQsDFCgauZqbUlfiKcEhpm5xBBBdP+1SFSrGP+WPl7wVRQks6MY+98yhFqYXSMgJQzZyxKRdyWaSEkXhK7f0RyB7mcSQz8Y08SY5RIKfblcqPdIX4ndkGVeQwLSspE+4rM/tZfrLgancGzPOnyTTb4rDhZzVV/A+eipdX69x4beP/f/+Vjlc6Fcy1DtXn0kGF2RaQf3tQOGCKv8Ipl0Y6GkB6AbUXOSo3vrWNOqB5CGIU2Ua+No/wVGW622UiMvkb+76RR4PsXRJFw9dmIRniRbMI3073273BXVN5ACWQr5cavJ8mf9wSWgSeyJAi+PieHdBpDAHGS8IJQt/ZIs13YE5Sa5hQ2sFECWq1VWAyujJR9eZ169Bf7h1BoNSYf6T+NNDWLLZZdpC5TV3ysUn2lxuj77I4E2taOplNKsXlPn3sNz5+5UDkFjtsxij9GrP6NK1H0T0MOJtj1LeptY0zccQ3Sjx9cJAPdtsKNnQBsWjuFmxl8CYAz+u9FD8O2C0/mNQ1sWt1RP3muermtGUXtN/mjWXKPyjpoXIt0GB3xj75xuPdIfyBMGQsYdvtefdRraWWl+YMpKsv9rQe/PMyE8nneni5ZtoEj7U3nLQkCvMRATXUadK4VgpCbCBzY4JTyKTL/8EiO60ShEYJfoExtvjBDBIXqJnctC/oV6f2IC24ohZUVg24dIZFcQGLx1LOeBLgI7N0PflvPB+5IZ21/dH KaoYS45S w4TnV2AP2ekXcGWR8DOoD98IpTN3g73N0D46LgEsaDqD0F5r9LsjqEAATMw+0wo2ikZaludJNs0rZkg/nNqcHsk828oHsdA83whV2p8s3f8MC0dG2OQTueAhAMasasWku6C27ESNJIvbK+6we+sgw6A8PnHpjqlfneVGRIQlNEn/+XgTAWeailtwS762u6zSvHMB37fqY4O4SG0llTy2+lcJXR87rSyzclrCZj+7YzMYL2bB3lmFRqcBOeMwiq3b+wd2DqQr4W35dclE9/VltUd8fVEL3NVdnKKAx8yvMi88WnDUXZMQuSbcfNgGSlvVxrn2k5pyeVllZ2cwJPNgThWigj6X6USzsIryFLdsQv+/5oSIlWMkkrxp04OLU8jf3mAiniTJE6SFUFymtkfSm+IRnGkykYTJxumgMskwlIY9hgL1byfDE8FLADQDquET/MxbxUUOmdL0xfSnphq7Rzlff83wj9QDVDyVe0E3IVmmyrXQyUuXqtBG1M0uFJvlh+U9iFBz8vzDkny8Og3D2mAECi/sdfRvcc/qZgzhWs35i0zE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's free up some space on 32bit in page[1] by moving the _pincount to page[2]. Ordinary folios only use the entire mapcount with PMD mappings, so order-1 folios don't apply. Similarly, hugetlb folios are always larger than order-1, turning the entire mapcount essentially unused for all order-1 folios. Moving it to order-1 folios will not change anything. On 32bit, simply check in folio_entire_mapcount() whether we have an order-1 folio, and return 0 in that case. Note that THPs on 32bit are not particularly common (and we don't care too much about performance), but we want to keep it working reliably, because likely we want to use large folios there as well in the future, independent of PMD leaf support. Once we dynamically allocate "struct folio", the 32bit specifics will go away again; even small folios could then have a pincount. Signed-off-by: David Hildenbrand --- include/linux/mm.h | 2 ++ include/linux/mm_types.h | 3 ++- mm/internal.h | 5 +++-- mm/page_alloc.c | 12 ++++++++---- 4 files changed, 15 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1a4ee028a851e..9c1290588a11e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1333,6 +1333,8 @@ static inline int is_vmalloc_or_module_addr(const void *x) static inline int folio_entire_mapcount(const struct folio *folio) { VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); + if (!IS_ENABLED(CONFIG_64BIT) && unlikely(folio_large_order(folio) == 1)) + return 0; return atomic_read(&folio->_entire_mapcount) + 1; } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 31f466d8485bc..c83dd2f1ee25e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -385,9 +385,9 @@ struct folio { union { struct { atomic_t _large_mapcount; - atomic_t _entire_mapcount; atomic_t _nr_pages_mapped; #ifdef CONFIG_64BIT + atomic_t _entire_mapcount; atomic_t _pincount; #endif /* CONFIG_64BIT */ }; @@ -409,6 +409,7 @@ struct folio { /* public: */ struct list_head _deferred_list; #ifndef CONFIG_64BIT + atomic_t _entire_mapcount; atomic_t _pincount; #endif /* !CONFIG_64BIT */ /* private: the union with struct page is transitional */ diff --git a/mm/internal.h b/mm/internal.h index d33db24c8b17b..ffdc91b19322e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -721,10 +721,11 @@ static inline void prep_compound_head(struct page *page, unsigned int order) folio_set_order(folio, order); atomic_set(&folio->_large_mapcount, -1); - atomic_set(&folio->_entire_mapcount, -1); atomic_set(&folio->_nr_pages_mapped, 0); - if (IS_ENABLED(CONFIG_64BIT) || order > 1) + if (IS_ENABLED(CONFIG_64BIT) || order > 1) { atomic_set(&folio->_pincount, 0); + atomic_set(&folio->_entire_mapcount, -1); + } if (order > 1) INIT_LIST_HEAD(&folio->_deferred_list); } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3dff99cc54161..7036530bd1bca 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -947,10 +947,6 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) switch (page - head_page) { case 1: /* the first tail page: these may be in place of ->mapping */ - if (unlikely(folio_entire_mapcount(folio))) { - bad_page(page, "nonzero entire_mapcount"); - goto out; - } if (unlikely(folio_large_mapcount(folio))) { bad_page(page, "nonzero large_mapcount"); goto out; @@ -960,6 +956,10 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) goto out; } if (IS_ENABLED(CONFIG_64BIT)) { + if (unlikely(atomic_read(&folio->_entire_mapcount) + 1)) { + bad_page(page, "nonzero entire_mapcount"); + goto out; + } if (unlikely(atomic_read(&folio->_pincount))) { bad_page(page, "nonzero pincount"); goto out; @@ -973,6 +973,10 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) goto out; } if (!IS_ENABLED(CONFIG_64BIT)) { + if (unlikely(atomic_read(&folio->_entire_mapcount) + 1)) { + bad_page(page, "nonzero entire_mapcount"); + goto out; + } if (unlikely(atomic_read(&folio->_pincount))) { bad_page(page, "nonzero pincount"); goto out; From patchwork Mon Feb 24 16:55:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988500 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2577EC021A4 for ; Mon, 24 Feb 2025 16:56:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1283A6B0099; Mon, 24 Feb 2025 11:56:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D5AA6B009A; Mon, 24 Feb 2025 11:56:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DCF3B6B009B; Mon, 24 Feb 2025 11:56:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BB57E6B0099 for ; Mon, 24 Feb 2025 11:56:27 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7C10B1C8EE7 for ; Mon, 24 Feb 2025 16:56:27 +0000 (UTC) X-FDA: 83155441614.07.DBBD945 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 6AE861C000B for ; Mon, 24 Feb 2025 16:56:23 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TTs+TEzu; spf=pass (imf18.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416183; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=evcOi+TBEtbvV15MXszgYeJTQO+N+ZLjDDY2W9tEmL0=; b=V0pzbbtrTmMwwFgIL8qX4bjk16MUhXEvZ8Xm0ybgDz6VDfjhE1myvZbeY4IwR/OVFIwS3a gtm9JINHTmfgdRGCdEBqCKuiisRgIlG2zmP1Uz/6vaL8Cyy6ub9fPpoetkKKG/QdDs6CdO 6k13jZILajKvVO6T+jtZVM3vIXwKTSY= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TTs+TEzu; spf=pass (imf18.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416183; a=rsa-sha256; cv=none; b=ikydH+qDuZKpNKMc1bF+V8pgva/7GC1Ct+/uGB7XX+QQqvk3mCSdEGHxsKaISGonnsZs3y 45AhCcAwHVphcn7wOO7VdcKhO23d/r0aF5y2CUOE9mB/6ZMd1GoJcKRyUV6J0hqS6WdM85 2GgKVig79wfwummuLGyGSvO7GvRcOKA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416182; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=evcOi+TBEtbvV15MXszgYeJTQO+N+ZLjDDY2W9tEmL0=; b=TTs+TEzucdHJxmBQQfKqN3jIL2eIET4m3/Yz8+NBm+U4UTDcZmSX1SPUOay1HfPuCCKUL1 falqoU7XEMhDSao2+T1aHWley9qnhWm4n3P72HkLwEDA1O08pZuupfCEvRTu2fNpvMlD5w 8Jd+F2kMhTkT0K93Yy4YnYu81YEWfK4= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-48-CHp1k-JeOuuqdQbgvVrOpA-1; Mon, 24 Feb 2025 11:56:21 -0500 X-MC-Unique: CHp1k-JeOuuqdQbgvVrOpA-1 X-Mimecast-MFC-AGG-ID: CHp1k-JeOuuqdQbgvVrOpA_1740416180 Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-38f27acb979so4601720f8f.0 for ; Mon, 24 Feb 2025 08:56:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416180; x=1741020980; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=evcOi+TBEtbvV15MXszgYeJTQO+N+ZLjDDY2W9tEmL0=; b=G45Efj7WnKc7evBVRDY9/aGY7LGjoBFbbhvH5tYPzyub88lyMatDEhVJ4tX92lpmvn SN96qv6l+O6C4kHF5WiZkvI+SlxU2C+c1kIZkqoxGlOBBZSK96IAazJUOwuDMBaZYPFZ Mwnj2wTsp2bFTN7vsVa+cqSXPrPSfhu4uSvGDwXL0/Yq1yH2IbS9iwbw9FNbIuShOUNP /h4puyzam3euZc1TaLY7N76Z+NpiIHgTTdrMwpZysWZhMcM4rASdwyu9Q+CJOcsSmbEo l8z/qysYrGtyoSuc/GbpvnC0PWLmLhoL+MXCcx+X558zkigizdtcU58OWRhzzc/QL6AO CMIw== X-Forwarded-Encrypted: i=1; AJvYcCWctZIVswfBvmlnnoN0jd6XiuN3AA8r4JyrZ/yoprvNJxNYYHk5p+dvJDVjqB8lSJBq8TjSECyzSw==@kvack.org X-Gm-Message-State: AOJu0YxzcVhiLZonnkTqh8h38YuLbhLAY5Rf7O28EXj8s1voSsIwSb5V GWBz4zjpHHtEhBQhjmIapwDJrc/aXvm1mzVzrvs+oy2gSIjTaMGH9kPNHUMz2JSMTlINA/q/Ml8 jJmX3vQL2ymeBgvBqxszWvLXqG5VG4ve2RgSMAkP3+eXO06q1 X-Gm-Gg: ASbGncsHl/lnI9q48TzRj5j9pKaW83BkM+12hgNJiyzGfgSessUVlGO1z+kwQDj01tu x/e1/7PJO0A1amaAdqTZhinrV1OQ9Rfmjc6JuRSRIbtCbRPz2P47dzqKinCjk4olOVeSO4iZMX+ iAwWgK9Y78HwTPIesWzQH9SUTNpRHaVlsWYI4/GvAvXkpnVPP+BsJnB/Ys+KflRUkqACcqT6/iu nTTBi9KZozf25YUe7C3GGMLYxdUb51r++cfsO6ywA22Lg2SMqYEx2kkxDxW7+tF/SimIB2cxO+O IaS1wTG2QEUY7yNeMdiCdx4Tx2Xb13p7ML7BM+beMQ== X-Received: by 2002:a05:6000:18a2:b0:38f:50bd:ad11 with SMTP id ffacd0b85a97d-38f707840afmr10692069f8f.5.1740416180203; Mon, 24 Feb 2025 08:56:20 -0800 (PST) X-Google-Smtp-Source: AGHT+IHJ0CTZOs7oip6JQ6V5mLo1AyAa7BC2bCEfy4T3EJKtPxndTNVUXGmx+vtnsEhrmmF+DjMIVg== X-Received: by 2002:a05:6000:18a2:b0:38f:50bd:ad11 with SMTP id ffacd0b85a97d-38f707840afmr10692042f8f.5.1740416179763; Mon, 24 Feb 2025 08:56:19 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-439b02e489esm111463335e9.21.2025.02.24.08.56.18 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:19 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 07/20] mm/rmap: pass dst_vma to folio_dup_file_rmap_pte() and friends Date: Mon, 24 Feb 2025 17:55:49 +0100 Message-ID: <20250224165603.1434404-8-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: pp9QWuLkXezdC1zd0cus60JEivZP77XVA8Hwfm3WUzE_1740416180 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 6AE861C000B X-Stat-Signature: qspezmxsz9xr47hqpbu6zeu8gu5pnobu X-HE-Tag: 1740416183-72418 X-HE-Meta: U2FsdGVkX1+7/XH0UIuaA/KR2kqIDgIP/2GsYzopjLMrvulRwGr+kzpq5y5drmQfEP0i3YLB/mrngCXYRWV6rKlsuoAdlRshlbOjFdkCUTP35BQ+Qiizajc6Ogr1a5P6MoaQFLXC5dsL8JeqrnW2tFs2OguMwFkRBReHPm3tmcLXzG+8H2XJ48/ulzRoogwVqfMCgm/hKF2pGfPq00agrS/hKU9eG5tk06wihe1CRUSbj0EODllxhsAibcOW3mM4snwxFnP/cRa036zkBMkAFj+KPJ//bm0rTqR6Y3VKLKaLOyQGJnTC7giFHXE+KJRwXFfuTd0rjZYj2eZMHmd/QqWfzd3FOBIo0kJ5urtJH8yOEzP3H3k+LPAUj4ziCRStR07LnAa1leveWUVq5DdobOV19ze09s98wcRPJfXaVsvDDp1EythlM2k6VKo6oasWrbuVlqgsJRGk2DXZq59/dMIjZAkL/tlhTYa78TdlUXkAbVRqJohUCgK2klSG/a9vx/5Hi/BnTn8Vo2MdV0t4NI5JKy7uXBuUfv0X90B7iW+/fW0ySrrIHIyxf3gTP1nn1H26r7Ch0gPYuHPLTQvEAc7aUPgr+s6Sf378r1lIJfGrpw1RvLW7g1T71RiuCt0ncK2sNZ7Ih4MFubcz6pkfCu7yJZbIcTnpekYHBguMc7yjnc09Yf6RpCGTQbmwnGukAGrqQbvWnUfLRzDBy+/TFq5GwkeV0jdFGdXLgF0TylchvBekH7ypkLydC1zFAH8/thqCQtzMNtvkTrwgX6emuV9cML/IODx7pErDeyFxFyWBIZogrpmXFE4MpITedILlke/y6oC9qZ53UxyNiTBkUzebblsUOhi8natYm2TfSPwhcaoPYK/oUMlRdJ8YnWb5Qg7b32YMJ5zW65aGNNqeIL9lGz20PFmGVsutCBDw4H1+jZeLmqsv4Nv2wHCAqaOyroK29nwYlM/YJ8ZGb3Y 4Kxegzyi NTOXHfhjVMxc5czZtjSggVh7lE2oNZISsb6Jk+p24Sr5FP02gNpqR7hZZy4hPOQ8QObQrYX7/VHufxVxydqrAzA76aROLE1s95NpLeyr1hzSjft1mc0aZrzMky4PP1te9FKDnyDIczxOHruOFambAQUCwlWqoAUCNvT9aDneKm/exSBy7soeTaKKLIID7ooD856kThlIToKpe072y2oA74sXEssjh2wJOEjjjgPQgIft+IKyvT96/p+q8SucqT0iCft2MLvPQDxxx/zfh/oHPL0hOCxQ3wnwmbxfeWRZSArcaYx8oL38Rn2DmMemC3vqF/E0PjcXd4ZLvhJux16k3bLddOT+gaABho4foVYNtqo48jbphIOtaW4Q8EYWuLdyABHh6cY3HZn2y0Ro255eGhx5QPz3Qe3ew59guh6lcbumW06H30JnSu0Jw7GqDJrY0KdUa5zyxNp6NLKtu1ufVFXM2akeu97tiRvZb7QeqXFA6hPbYuPXgNmXNNHWjitYBbCtHVzOW/w+ZMRPKQGchCUy6iPTcKf8HFEdtTFXkhs0WXVA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We'll need access to the destination MM when modifying the large mapcount of a non-hugetlb large folios next. So pass in the destination VMA. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 42 +++++++++++++++++++++++++----------------- mm/huge_memory.c | 2 +- mm/memory.c | 10 +++++----- 3 files changed, 31 insertions(+), 23 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 6abf7960077aa..e795610bade80 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -335,7 +335,8 @@ static inline void hugetlb_remove_rmap(struct folio *folio) } static __always_inline void __folio_dup_file_rmap(struct folio *folio, - struct page *page, int nr_pages, enum rmap_level level) + struct page *page, int nr_pages, struct vm_area_struct *dst_vma, + enum rmap_level level) { const int orig_nr_pages = nr_pages; @@ -366,45 +367,47 @@ static __always_inline void __folio_dup_file_rmap(struct folio *folio, * @folio: The folio to duplicate the mappings of * @page: The first page to duplicate the mappings of * @nr_pages: The number of pages of which the mapping will be duplicated + * @dst_vma: The destination vm area * * The page range of the folio is defined by [page, page + nr_pages) * * The caller needs to hold the page table lock. */ static inline void folio_dup_file_rmap_ptes(struct folio *folio, - struct page *page, int nr_pages) + struct page *page, int nr_pages, struct vm_area_struct *dst_vma) { - __folio_dup_file_rmap(folio, page, nr_pages, RMAP_LEVEL_PTE); + __folio_dup_file_rmap(folio, page, nr_pages, dst_vma, RMAP_LEVEL_PTE); } static __always_inline void folio_dup_file_rmap_pte(struct folio *folio, - struct page *page) + struct page *page, struct vm_area_struct *dst_vma) { - __folio_dup_file_rmap(folio, page, 1, RMAP_LEVEL_PTE); + __folio_dup_file_rmap(folio, page, 1, dst_vma, RMAP_LEVEL_PTE); } /** * folio_dup_file_rmap_pmd - duplicate a PMD mapping of a page range of a folio * @folio: The folio to duplicate the mapping of * @page: The first page to duplicate the mapping of + * @dst_vma: The destination vm area * * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) * * The caller needs to hold the page table lock. */ static inline void folio_dup_file_rmap_pmd(struct folio *folio, - struct page *page) + struct page *page, struct vm_area_struct *dst_vma) { #ifdef CONFIG_TRANSPARENT_HUGEPAGE - __folio_dup_file_rmap(folio, page, HPAGE_PMD_NR, RMAP_LEVEL_PTE); + __folio_dup_file_rmap(folio, page, HPAGE_PMD_NR, dst_vma, RMAP_LEVEL_PTE); #else WARN_ON_ONCE(true); #endif } static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, - struct page *page, int nr_pages, struct vm_area_struct *src_vma, - enum rmap_level level) + struct page *page, int nr_pages, struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma, enum rmap_level level) { const int orig_nr_pages = nr_pages; bool maybe_pinned; @@ -470,6 +473,7 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, * @folio: The folio to duplicate the mappings of * @page: The first page to duplicate the mappings of * @nr_pages: The number of pages of which the mapping will be duplicated + * @dst_vma: The destination vm area * @src_vma: The vm area from which the mappings are duplicated * * The page range of the folio is defined by [page, page + nr_pages) @@ -488,16 +492,18 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, * Returns 0 if duplicating the mappings succeeded. Returns -EBUSY otherwise. */ static inline int folio_try_dup_anon_rmap_ptes(struct folio *folio, - struct page *page, int nr_pages, struct vm_area_struct *src_vma) + struct page *page, int nr_pages, struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma) { - return __folio_try_dup_anon_rmap(folio, page, nr_pages, src_vma, - RMAP_LEVEL_PTE); + return __folio_try_dup_anon_rmap(folio, page, nr_pages, dst_vma, + src_vma, RMAP_LEVEL_PTE); } static __always_inline int folio_try_dup_anon_rmap_pte(struct folio *folio, - struct page *page, struct vm_area_struct *src_vma) + struct page *page, struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma) { - return __folio_try_dup_anon_rmap(folio, page, 1, src_vma, + return __folio_try_dup_anon_rmap(folio, page, 1, dst_vma, src_vma, RMAP_LEVEL_PTE); } @@ -506,6 +512,7 @@ static __always_inline int folio_try_dup_anon_rmap_pte(struct folio *folio, * of a folio * @folio: The folio to duplicate the mapping of * @page: The first page to duplicate the mapping of + * @dst_vma: The destination vm area * @src_vma: The vm area from which the mapping is duplicated * * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) @@ -524,11 +531,12 @@ static __always_inline int folio_try_dup_anon_rmap_pte(struct folio *folio, * Returns 0 if duplicating the mapping succeeded. Returns -EBUSY otherwise. */ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, - struct page *page, struct vm_area_struct *src_vma) + struct page *page, struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma) { #ifdef CONFIG_TRANSPARENT_HUGEPAGE - return __folio_try_dup_anon_rmap(folio, page, HPAGE_PMD_NR, src_vma, - RMAP_LEVEL_PMD); + return __folio_try_dup_anon_rmap(folio, page, HPAGE_PMD_NR, dst_vma, + src_vma, RMAP_LEVEL_PMD); #else WARN_ON_ONCE(true); return -EBUSY; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 13152313e77ce..a3264d88d4b49 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1782,7 +1782,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, src_folio = page_folio(src_page); folio_get(src_folio); - if (unlikely(folio_try_dup_anon_rmap_pmd(src_folio, src_page, src_vma))) { + if (unlikely(folio_try_dup_anon_rmap_pmd(src_folio, src_page, dst_vma, src_vma))) { /* Page maybe pinned: split and retry the fault on PTEs. */ folio_put(src_folio); pte_free(dst_mm, pgtable); diff --git a/mm/memory.c b/mm/memory.c index cacf6d53bdf32..ef74c4f009f84 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -845,7 +845,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, folio_get(folio); rss[mm_counter(folio)]++; /* Cannot fail as these pages cannot get pinned. */ - folio_try_dup_anon_rmap_pte(folio, page, src_vma); + folio_try_dup_anon_rmap_pte(folio, page, dst_vma, src_vma); /* * We do not preserve soft-dirty information, because so @@ -999,14 +999,14 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma folio_ref_add(folio, nr); if (folio_test_anon(folio)) { if (unlikely(folio_try_dup_anon_rmap_ptes(folio, page, - nr, src_vma))) { + nr, dst_vma, src_vma))) { folio_ref_sub(folio, nr); return -EAGAIN; } rss[MM_ANONPAGES] += nr; VM_WARN_ON_FOLIO(PageAnonExclusive(page), folio); } else { - folio_dup_file_rmap_ptes(folio, page, nr); + folio_dup_file_rmap_ptes(folio, page, nr, dst_vma); rss[mm_counter_file(folio)] += nr; } if (any_writable) @@ -1024,7 +1024,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma * guarantee the pinned page won't be randomly replaced in the * future. */ - if (unlikely(folio_try_dup_anon_rmap_pte(folio, page, src_vma))) { + if (unlikely(folio_try_dup_anon_rmap_pte(folio, page, dst_vma, src_vma))) { /* Page may be pinned, we have to copy. */ folio_put(folio); err = copy_present_page(dst_vma, src_vma, dst_pte, src_pte, @@ -1034,7 +1034,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma rss[MM_ANONPAGES]++; VM_WARN_ON_FOLIO(PageAnonExclusive(page), folio); } else { - folio_dup_file_rmap_pte(folio, page); + folio_dup_file_rmap_pte(folio, page, dst_vma); rss[mm_counter_file(folio)]++; } From patchwork Mon Feb 24 16:55:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988501 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE280C021BB for ; Mon, 24 Feb 2025 16:56:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8A0106B009B; Mon, 24 Feb 2025 11:56:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 851796B009C; Mon, 24 Feb 2025 11:56:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 67C00280001; Mon, 24 Feb 2025 11:56:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 483356B009B for ; Mon, 24 Feb 2025 11:56:30 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 05E3FB6B1F for ; Mon, 24 Feb 2025 16:56:30 +0000 (UTC) X-FDA: 83155441740.30.D5CDECB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 931AC160006 for ; Mon, 24 Feb 2025 16:56:27 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=i8Y+nR2e; spf=pass (imf08.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416187; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lWnUBbJbNal4OtQP6KylSC1HMhJkWgV+LTGJMK3tR3E=; b=eI+N4eeenmgTlwJJACvyzpgg3NRNrb2Y9l65i/DqwW5qudTBnmHDSqA9lgDjob7XTXiHVW riUvBVWBbJjqs8DO0ZVFyjEfqbDOIKE2oT3p9QbDs4+r1hkPRmE6GeR4yGy+arQF+yup+e Vl0OK4wA3DDZrTddcrv1V8HIDpVWWws= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=i8Y+nR2e; spf=pass (imf08.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416187; a=rsa-sha256; cv=none; b=kRxPs3y1IDN1qESToBHBNw2WnleiOeYpIbrOGlaFghPpvsiqkzrW1uVngFHQ1CHVGzIFP0 1dbdtNhswUoZ+b5JxXzpFIykBTNdbzZyWIES8W1NBA+WEd/7jr6mnTnEnHLTqoPj14fsYc OsfZ3R7KfBlDXQxV7Y+1ukS0IIeKBPE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416186; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lWnUBbJbNal4OtQP6KylSC1HMhJkWgV+LTGJMK3tR3E=; b=i8Y+nR2eJ2mPPq7sietGi3wP6/JfjsMDa2lNTEOWVMiE8cQrQW5ndNJspJRb2vNO7DwPoa Z+LPGSHduHoSI84zxamPQpePtY53vJXeZlB1miDZIJ9SA8AvC2Cj0zjkJZZw/mH/I0+6Ks as1U3E6JNUh++9P32PEuQbiIafNpCjQ= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-115-lvovgq3RPoe-SUfUNcyE8g-1; Mon, 24 Feb 2025 11:56:23 -0500 X-MC-Unique: lvovgq3RPoe-SUfUNcyE8g-1 X-Mimecast-MFC-AGG-ID: lvovgq3RPoe-SUfUNcyE8g_1740416182 Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-38f2f78aee1so1894145f8f.0 for ; Mon, 24 Feb 2025 08:56:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416182; x=1741020982; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lWnUBbJbNal4OtQP6KylSC1HMhJkWgV+LTGJMK3tR3E=; b=XcJ7Unzpj9DCmCbPjufcIQfFJf/1VhBk4inKSdMdQ1U4aCUWdocPivwXt6HkK1VLtX vlDCFUXJpAvhxbq3JtoZrG6yZ7A2aaccXl1tqzp/jra+LU/nSezCm1wv6Ghts0SSq6cF 5fBrtyStUQsn2ZhRUfHD+Fg6+xzWWdbqkXz1TS8P9jmEhGY739pZmZh+/ecOZmgjxNpp xaNFtZHsT+sp+ihA9d5vhdVhjTEW3SIycvrwPQE2hzEd5eHjEoWBD8brmMbyCwOL5uz2 0OlMWHIcQ7IJXyktaAqIStAFHkX3HLbsR4VzS0QO574r38UiD3Qp7FtteyJ0QUBVh1UY HM+Q== X-Forwarded-Encrypted: i=1; AJvYcCVRXhdeZJaKHACJ/Cs0L9DNDlNQeoQGsYpL7nrw/Kanzlc+UihlQdPrISwrstrT2p0G3FHr2eY6Rw==@kvack.org X-Gm-Message-State: AOJu0YzHZn7hmqD5TknugB9q+syyL5bz4wQD8ZK/87mskzP7sCWWgPhK jm4Ill1qma7c7V/4bPwOssarfq01yKIhe8gpQKwoXfHBstU6sul2GHSLe3dMK9dQb0hqTnNPbt2 bl/QyO7JlHdzqYkuv1W3rx38YpicUzgy9bx6WiSJXqXqAPpzk X-Gm-Gg: ASbGncsTBA0f+GtimT+173eebj/eM0qQ4v7PIgpeOy497LY7W15kCPk0avNEqsFHC2O IvkjonU0M+gI5MNexkte/WAC7VVC2y+IySnoiw5LMIQXU8v5Sp5wLfth6ZgxpEDpZD8czhhNKuP kp+DGt8hC1KsLKf3CKnM4ldKFK1U96IDoWGRzKZcr2dTt3/aW3UzfwIFo55bEtHpeoSNYq5d4Ac XZ5U9Ya5QSHrgX0GVl0ZWiTnqBNhuF1ZzFMkr1yDwigA2c436WPEHLwZ7jh8HokwELiSLmwRxit blYYQ4iNGeSyWmT6bnVguHtekwTWbyOiRfpvYtUZNA== X-Received: by 2002:a5d:6d82:0:b0:38a:4184:14ec with SMTP id ffacd0b85a97d-38f6f3c507amr11123605f8f.1.1740416182297; Mon, 24 Feb 2025 08:56:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IHG35xVnH9LhIY4oSSBr3n8xfuxml7Goul51SSXyqtQOXZX4W9oDib+9PjJY2pvFXqp8djOQA== X-Received: by 2002:a5d:6d82:0:b0:38a:4184:14ec with SMTP id ffacd0b85a97d-38f6f3c507amr11123573f8f.1.1740416181836; Mon, 24 Feb 2025 08:56:21 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38f259f8115sm32604557f8f.92.2025.02.24.08.56.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:21 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 08/20] mm/rmap: pass vma to __folio_add_rmap() Date: Mon, 24 Feb 2025 17:55:50 +0100 Message-ID: <20250224165603.1434404-9-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: aPIl4PLy86fZ5KxXxKknr98Ol2xdOroQpQCobW_jq2I_1740416182 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 931AC160006 X-Stat-Signature: 7mg9zza58nfnssnd9zky6to9t9o71c7x X-Rspam-User: X-HE-Tag: 1740416187-310895 X-HE-Meta: U2FsdGVkX1/j3Q2yhbfhI4puFCN5d7jF/F5CxPlUb/TVPsQJHAaE+pRDUMlECJ3bdZbUjDlwzLVXMyxpi8RHIEPjN8pONbR+x0VtNwrZGGgi39/HssgcXJKsjlFH30O+3NF8ylHMgd8Meknz89N4j43utZTi87B83r10BxsctRz3ADoeawaxz3hN2JBNme9PqYn2Zs9qUfZMY5rY4P1ueR0ubCe1N0WFf6KMylVUssu+xGDmVt2HwUWbgbQ6WGdDtNQcTljo+ku3JgG7CprAaATHgsdCN3oGaIcy3xNjUmzVJxEWd2xa8RhE92EBuqFZvAhwC4lvGvgMD5z4vauuIEP/6prsLs2KKDk24poR336bxDJ/UK9uBugw2hSPICpM6G1pAJbS9UkzHVzoXvsjqYX9Sm83Y8C4Bxunjk2JHNzpgOg9tM+6nn1PayPEqYB0HdtJdAHqpVFmzktQUfHlUqlW4F/L6/0vS9pxP7W8xqB4jJ4X+AJlMX/7EFICDZTrB6qGBCkHk+fdAFkNkT9p6CsPOh0U7S1WQI8yXjPHw5MmaVWXdQUvLKsJgE9tMgaG987aL6HPkItYWIL7FtUbDevQyP7/Ov14l3ByNgci58JQZuAHpIQIIcN7y2ahxzgdz5vcvkXbp+i29KIWncD79QWeZFgDVSW1OGpl7OZBxT3caLbPJJ9xvZuiOMrsB7gASgmRkUSX3X40/BnpnXyn+YN0cvLdXfYwZC8cxriUMCF/YOuk4mMP0jfUnQno01eIqHoGuUlyUfJOXdXsVXMv/DJyKtjERsWiIUFMawI1I3pq72rmYHHBNb0+mY3Hd3oXF8xmcYxbb7xIcFl2LxxY0NSnlPx+2//FheVJ1Ggvsbl0ZBx+ZR4hKkoOFLgsvnyLyUa/EA3suATsGvRCmMyOLkiHCUZJWlnAR8n6Rid1gZo81u1fjewnBdX5VZ2zGpmnSwxS7qf0WsFRmj5Wzyl 6SP1xDfU Sq+GvYc+vVb2UhMuysN2Fd7fhQb+5PkHXhd6TD4MoBvTWE1Ah9WqNNEY55iNjwilKSkwFh4+af2LCZu1PBYfkLb7ldu72wV4BdGbGvzxgnuaCD+NHN3+Sc7enNOeryPxWJxm6IlBkBU3uxDxPesy8yA1AZP55+BoKkjxr9alImRNxtpkZYNcwWmkwjkJ+Shitf42hRzmQp9oOT7u3DV9N6v8ZaE0kP0Qes86kIeCy5uF8H0dRvGK7eLDxTSTAVnKvi9BVw84rjc+FmtOgDKnH7wWFxeE8fq+VSCPlvQeFWEzWNa2ENRJHCQKOa0B2sCiSt4pbjiLPyywDLRokpLbpba24+WLOmwW3zO+kL6Bql9ca9zgUDivnGUkWYiHKlxf3Ux+9L2a4N0NOCNgSJAB08K7nV3M03U0uneg0RldETT5Yygmed/O8meyROwzIVstwwh4fNwVz2ZynjM6079nr7R3fmjDpM3h0r8wyXdsYaVh7HzHUR3A9Wbp3MmugnN520N4oQqAi9aHoEMXSUduuH9pxpOk4M3Z+DRdt+bAcM1FTmsA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We'll need access to the destination MM when modifying the mapcount large folios next. So pass in the VMA. Signed-off-by: David Hildenbrand --- mm/rmap.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index bcec8677f68df..8a7d023b02e0c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1242,8 +1242,8 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, } static __always_inline unsigned int __folio_add_rmap(struct folio *folio, - struct page *page, int nr_pages, enum rmap_level level, - int *nr_pmdmapped) + struct page *page, int nr_pages, struct vm_area_struct *vma, + enum rmap_level level, int *nr_pmdmapped) { atomic_t *mapped = &folio->_nr_pages_mapped; const int orig_nr_pages = nr_pages; @@ -1411,7 +1411,7 @@ static __always_inline void __folio_add_anon_rmap(struct folio *folio, VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); - nr = __folio_add_rmap(folio, page, nr_pages, level, &nr_pmdmapped); + nr = __folio_add_rmap(folio, page, nr_pages, vma, level, &nr_pmdmapped); if (likely(!folio_test_ksm(folio))) __page_check_anon_rmap(folio, page, vma, address); @@ -1582,7 +1582,7 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio, VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); - nr = __folio_add_rmap(folio, page, nr_pages, level, &nr_pmdmapped); + nr = __folio_add_rmap(folio, page, nr_pages, vma, level, &nr_pmdmapped); __folio_mod_stat(folio, nr, nr_pmdmapped); /* See comments in folio_add_anon_rmap_*() */ From patchwork Mon Feb 24 16:55:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988502 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68349C021A4 for ; Mon, 24 Feb 2025 16:56:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D8086B009D; Mon, 24 Feb 2025 11:56:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 08A296B009E; Mon, 24 Feb 2025 11:56:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF70F6B009F; Mon, 24 Feb 2025 11:56:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BFEE56B009D for ; Mon, 24 Feb 2025 11:56:32 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 570F6120BDF for ; Mon, 24 Feb 2025 16:56:32 +0000 (UTC) X-FDA: 83155441824.24.B8A57EE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf16.hostedemail.com (Postfix) with ESMTP id C3984180010 for ; Mon, 24 Feb 2025 16:56:29 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Yq2Vnekq; spf=pass (imf16.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416189; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Cb8Ah8lJoOkHJddlEV7mqidKdKB3PYdRohUjXTfR2AA=; b=fu0mauNoBn+ftIorkVip/JH7KVlxJnI0qLXRzP9FquYPFsAloG3OJ2aVZJ1CmfCCvwwj6L UbzAeO4/VVt0co8CEEUQ44Et//CfEabjmyBQ2p1AOytTgbAyPKnPiqLAbtGoaQT/yQOzpM bAIBYraIYwJcILEBLe6sDnD55EE3pPM= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Yq2Vnekq; spf=pass (imf16.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416189; a=rsa-sha256; cv=none; b=TAZHe47U/rn/cfMIuCDTYQmLdtjxI0moZI+8zR2rSVpVPRwSk9eq+yJqdu4dl+g9uLnFpH /wVTdOrWdtQQnJKCcJnQ+44AOc8OZ6PJnGIP85MnOdpOp1LWhRUg1FWIqqMo54WeAISAkw zU5vaseDLyYescmoOacULqgFR3tc3ms= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416189; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cb8Ah8lJoOkHJddlEV7mqidKdKB3PYdRohUjXTfR2AA=; b=Yq2VnekqTsEPrE3qB51D3ielMktnGD9InoR6XWb0CXtiN6/tNv071DHpYQk9mwkZQJYBzx 1QsYZKM1AWpc/EgGu1mwTahindzy6gqzZceZ4T8E6qh2KiAvvRt2BZvv1wCsPHVvTDN6w2 98EXwNEHvXOfJCaHPttwxFW0AhHqvmQ= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-494-6RjJDzu_N4auYsuHO4TDyQ-1; Mon, 24 Feb 2025 11:56:25 -0500 X-MC-Unique: 6RjJDzu_N4auYsuHO4TDyQ-1 X-Mimecast-MFC-AGG-ID: 6RjJDzu_N4auYsuHO4TDyQ_1740416184 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-43935e09897so32828945e9.1 for ; Mon, 24 Feb 2025 08:56:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416184; x=1741020984; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Cb8Ah8lJoOkHJddlEV7mqidKdKB3PYdRohUjXTfR2AA=; b=ibG8RphXvu1t0LvWZ86esg5JMR9uYKwzmFxH8lqw3D2prBeF7JwXLKaaMSRUiwbkeN sEFOR0wdqhlPzhwitJvoZ1wyFEWkOOpUHQmvxncLcuBwh4pB2RKWUB0TpQmB8kpxFkJA frSX2M80xFs5vEXYFTWBopHUhTj+7+6xiAb357UOd0OlQsUESqKcgpOtMPskpsEL9+Ry IbTlpXbuj2zMTxI6D/2WTahkM3BDObeZWSR91J9m+0AOWwKLDnYByl+RxsWmMfpvCqzY rTOU9oWc6YecAuHI44KEbSt3ZgtG/+LLDLR8wAQdDLsT/pnm1829sA3S59yuAlUCwOHl Qpdw== X-Forwarded-Encrypted: i=1; AJvYcCXs3MJV4Xy+qEsq2v2dQjiYwk/tFVMGZMXJtqIPPtm9rvJRDbcolYl5pXasj+XnDeInG2343Uo9IA==@kvack.org X-Gm-Message-State: AOJu0YysvymlIIq4iyHcf0lO07knppiDolgCv3hVZjnFbRj2ImiacIX1 jjIp/jqJeqECwowJ0HhdmbL3ZxYtvTkuzpoS0K2KtarpsKnS2eVXXQ5bqAVkSZ7ETkb9yhbm4do FGpOVrohigNLaVrCZ8Mq56tjTEiJel77JUI28U1xFVGI3z+7K X-Gm-Gg: ASbGncvLFRC6zxrcnVMo9At97XZiaRfhIuS/XOEM1AkjNBsCNXzsCnp15Z7bDCzjZww dtGdQxQpB/R35cXPIjs9IbYT00AUgV2CdsMb0ulU/WO7CzOCSbV2/rL/kl+O5NcHiDBeVQ+0msK 5u+YM+mS04g5IG4OAcqfp4NfcQVXhMIFuUrjKr3Odv7BC5RPyjgEFRonbWyFf68OL5hOfJrLmrw hFT+L6kmog0D/sLY0KoLxKhsd9fekoxsQCK1Y6/fYfOnKAxbQrloWS1wfXEckYO++mErSSpnapT wlPdeecf5A1TRa/AocJKXEdHW4XzAn0vYasNqK1DVA== X-Received: by 2002:a05:600c:4e8c:b0:439:6712:643d with SMTP id 5b1f17b1804b1-439ae1e6c15mr101103425e9.9.1740416184317; Mon, 24 Feb 2025 08:56:24 -0800 (PST) X-Google-Smtp-Source: AGHT+IELVR15SfmZbX6a5/LWsD6f0rg6Y8hiw7+oMACeve7JZ9CA6HteUq/ZR57fRSUerpUCoh+bqg== X-Received: by 2002:a05:600c:4e8c:b0:439:6712:643d with SMTP id 5b1f17b1804b1-439ae1e6c15mr101103135e9.9.1740416183869; Mon, 24 Feb 2025 08:56:23 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-439b031b613sm111082365e9.33.2025.02.24.08.56.22 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:23 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 09/20] mm/rmap: abstract large mapcount operations for large folios (!hugetlb) Date: Mon, 24 Feb 2025 17:55:51 +0100 Message-ID: <20250224165603.1434404-10-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: EOoeUjN2GlXgciCR1eKB65Uq5TMHZDNWruANHZJUZmE_1740416184 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Stat-Signature: mguop8wm6iwaw9r7ykkdujeqgxjfiu6i X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: C3984180010 X-HE-Tag: 1740416189-578440 X-HE-Meta: U2FsdGVkX1+ZrUn0S/jtcwu9iP5azUH6NaYmzd4MzIKE/ZpZZx4+1gRiWDypiu/G157XC1az8lxiP4mp/r12/L0NvgpVwksOSR7rkmQgRLBtQuVQEiLWaUguXMZ6l/hXygkIsIxEGHYuJGyyldtvrrVvp9Bva9Dxl3hGxPiXfFBkBeLFsdWpjF2r6i/M/vpJixQ4LD6VRlGiO0Aqp/6mDnTm2YPwUypBmiunItyd/L8uJaeNE3UdjeO2QOhWSrIyeP2ertEWP4JWuLXl+dBjbaiO5I0eUHrZwI2bPQqLiGD/6dA1DqL1gEcGBbdfDseUsAgQj2Vb6Jz+xYzNl2aQ/eDUUPg7TINnNgpc2awX3xIP1YqUhbyzwyO+KQjaupr7iyiKJUGNLKheK1fcX/8m2qdwpmf5LxyqO+ICodI7UdKkLX5w0eo+sD8fS/03JmhGS/J+2R70H1bCy54cBvixx1FS2xcvyLyANd/g11Mb7GTvBcacO1ivMxV9pX6mjNWLcD8vv1l9/BpEAP/IJVUw7o/e4GMjVhgAOP/sYfqbp0IxjXRUyVSW5mDIWemLTeqIKn/JVX3+uu4t1BMG+ngk4Q76prnFYTsQ+PnZtUdilW6kgz/Bk2n6wta+iokvVdOWBx/n4aQQyrHd2VdTBPYm2/KGhDETaVe75vDJNmVtepda4Vgntgh8lMj3vAojoRs+F3BVrfoBxbg49nammwG2z/6z17+25aA0JsTEopyeWxqSnipy7xwml4d/DPMcwmebUkbNCHCtzBYDU4oMtRftFqITXB8EGbcwP7HPptsCoVzcSWcJ+iQjQ4ethlwy0xSe1tCl2gikcW3q45NCD5gAKo2z4UU96FenaNHrPCV4epwB4G22PSK11H7WJ6JQbHg5q3qFwhhTkD/+N9Zyx211ucbT9/HCwh46acMrnywsEVaUDqQ+LDGWoCDSX+8jR6Xz1cXTnQbO76Yf68WJBV+ rLWa1VuB NXyECD/wqexicXoXWZwT2cndxh04tqpjDtm/JySd6lu7VFU6WFqp21CEgvxV/qUSuesSJT5K9z0ld0Xi1U6xk/Ek7BIVYJVBdA4bYmYV9FTLYFETUKQyJshZyHM2BOuBrQ8FBnPCqe+lsbmtxUuPBi58Eg0EiUtNfU03wk8WCgYfBKp+jsw8i2IEBWXaJoNEfqe4PBPWJUZkWlwy8mXHWwUiCmRPekt7iPYOaunU0YxY/onfiBajUfwph02OKN02D0UEN2CSyKczxlvk2SBGaAJIexFJFw3Oyxa9SULY0PI/PZozwb6IdjWaTXkfBIxUKP5+xX2av59l7BsZagi3xOSoBNPy3Jm/ys+a3mUCpZ0Pg9kWZxQNDXhfSmL3kA+bC+jkebirgT27MFf2XvIVqhUdQBKDjnwik4K4Sf5kKL0ditYKIXeetV2x08ElTe7vnXL7spJzLlDb4xhfjtU+n3uqM6ojVMED5gItG5DDvjEJ5CfRwBFXf4qotcRiBRNvJNPwAjlsC82JaUyQn+Hstk/V6vzzAkc9Ndm8n X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's abstract the operations so we can extend these operations easily. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 32 ++++++++++++++++++++++++++++---- mm/rmap.c | 14 ++++++-------- 2 files changed, 34 insertions(+), 12 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index e795610bade80..d1e888cc97a58 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -173,6 +173,30 @@ static inline void anon_vma_merge(struct vm_area_struct *vma, struct anon_vma *folio_get_anon_vma(const struct folio *folio); +static inline void folio_set_large_mapcount(struct folio *folio, int mapcount, + struct vm_area_struct *vma) +{ + /* Note: mapcounts start at -1. */ + atomic_set(&folio->_large_mapcount, mapcount - 1); +} + +static inline void folio_add_large_mapcount(struct folio *folio, + int diff, struct vm_area_struct *vma) +{ + atomic_add(diff, &folio->_large_mapcount); +} + +static inline void folio_sub_large_mapcount(struct folio *folio, + int diff, struct vm_area_struct *vma) +{ + atomic_sub(diff, &folio->_large_mapcount); +} + +#define folio_inc_large_mapcount(folio, vma) \ + folio_add_large_mapcount(folio, 1, vma) +#define folio_dec_large_mapcount(folio, vma) \ + folio_sub_large_mapcount(folio, 1, vma) + /* RMAP flags, currently only relevant for some anon rmap operations. */ typedef int __bitwise rmap_t; @@ -352,12 +376,12 @@ static __always_inline void __folio_dup_file_rmap(struct folio *folio, do { atomic_inc(&page->_mapcount); } while (page++, --nr_pages > 0); - atomic_add(orig_nr_pages, &folio->_large_mapcount); + folio_add_large_mapcount(folio, orig_nr_pages, dst_vma); break; case RMAP_LEVEL_PMD: case RMAP_LEVEL_PUD: atomic_inc(&folio->_entire_mapcount); - atomic_inc(&folio->_large_mapcount); + folio_inc_large_mapcount(folio, dst_vma); break; } } @@ -451,7 +475,7 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, ClearPageAnonExclusive(page); atomic_inc(&page->_mapcount); } while (page++, --nr_pages > 0); - atomic_add(orig_nr_pages, &folio->_large_mapcount); + folio_add_large_mapcount(folio, orig_nr_pages, dst_vma); break; case RMAP_LEVEL_PMD: case RMAP_LEVEL_PUD: @@ -461,7 +485,7 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, ClearPageAnonExclusive(page); } atomic_inc(&folio->_entire_mapcount); - atomic_inc(&folio->_large_mapcount); + folio_inc_large_mapcount(folio, dst_vma); break; } return 0; diff --git a/mm/rmap.c b/mm/rmap.c index 8a7d023b02e0c..08846b7eced60 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1266,7 +1266,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, atomic_add_return_relaxed(first, mapped) < ENTIRELY_MAPPED) nr = first; - atomic_add(orig_nr_pages, &folio->_large_mapcount); + folio_add_large_mapcount(folio, orig_nr_pages, vma); break; case RMAP_LEVEL_PMD: case RMAP_LEVEL_PUD: @@ -1290,7 +1290,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, nr = 0; } } - atomic_inc(&folio->_large_mapcount); + folio_inc_large_mapcount(folio, vma); break; } return nr; @@ -1556,14 +1556,12 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, SetPageAnonExclusive(page); } - /* increment count (starts at -1) */ - atomic_set(&folio->_large_mapcount, nr - 1); + folio_set_large_mapcount(folio, nr, vma); atomic_set(&folio->_nr_pages_mapped, nr); } else { /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); - /* increment count (starts at -1) */ - atomic_set(&folio->_large_mapcount, 0); + folio_set_large_mapcount(folio, 1, vma); atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED); if (exclusive) SetPageAnonExclusive(&folio->page); @@ -1665,7 +1663,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, break; } - atomic_sub(nr_pages, &folio->_large_mapcount); + folio_sub_large_mapcount(folio, nr_pages, vma); do { last += atomic_add_negative(-1, &page->_mapcount); } while (page++, --nr_pages > 0); @@ -1678,7 +1676,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, break; case RMAP_LEVEL_PMD: case RMAP_LEVEL_PUD: - atomic_dec(&folio->_large_mapcount); + folio_dec_large_mapcount(folio, vma); last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { nr = atomic_sub_return_relaxed(ENTIRELY_MAPPED, mapped); From patchwork Mon Feb 24 16:55:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA503C021BB for ; Mon, 24 Feb 2025 16:56:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D8396B009E; Mon, 24 Feb 2025 11:56:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 088DB6B009F; Mon, 24 Feb 2025 11:56:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D0C626B00A0; Mon, 24 Feb 2025 11:56:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7DE9F6B009E for ; Mon, 24 Feb 2025 11:56:34 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DB55BA3D01 for ; Mon, 24 Feb 2025 16:56:33 +0000 (UTC) X-FDA: 83155441866.09.E0C8ABB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 84B5D180015 for ; Mon, 24 Feb 2025 16:56:31 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SagYhZ08; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416191; a=rsa-sha256; cv=none; b=hcMbKCxphWAQXck+KXOYP1wUaPOl/3pAPOblPY+Swv255Ap8umJEzOUklAhS6dhh3h64mh QnocCl/t4+/jsee8IX55nq6yKly4nw3fMbowOzh9btzGJYgEKOPvzIHynu8IeQPDxMeHkk Gd5HlP6+22NfqVM9dpxcV4mM45YfhAQ= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SagYhZ08; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416191; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VusPlBbKEOGpL4wS1BqoWY6CYb2f4HQ+/wIDh8p+vAo=; b=S0/WeYAKr0MGK5oXzpgRtwtrv75eYjr1eJdb0orogu27W196AFxPLDlGe4D2RdtzxRa3Hz Y9UL4Bg6dWRjOwbZf+BjoXXIN88L4ThuwDXOovCitCsW1R5V5mTfw/7hUbt6Ir43B7a4qa CX/HY3pPBqlqUTS8Jj0YrcqRUEQNmJk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416190; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VusPlBbKEOGpL4wS1BqoWY6CYb2f4HQ+/wIDh8p+vAo=; b=SagYhZ08ddx6wZoexyDBawUCryJLb4mDtmuzhxkHb8FWShggtBTks8JsUeTak2aqsJgU/M l0+lTQLoZQ0F2YHObpEmsN2YkTJFWyLOD3utDqnElzM0UviJYfn+FMZvNR7MLf8sb16fUX G+oa7cO90ubtiNVd881jvDHbwp0JNR8= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-407-eug87DjtMBGweG6JxBsf-A-1; Mon, 24 Feb 2025 11:56:27 -0500 X-MC-Unique: eug87DjtMBGweG6JxBsf-A-1 X-Mimecast-MFC-AGG-ID: eug87DjtMBGweG6JxBsf-A_1740416186 Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-43998ec3733so23637215e9.2 for ; Mon, 24 Feb 2025 08:56:27 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416186; x=1741020986; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VusPlBbKEOGpL4wS1BqoWY6CYb2f4HQ+/wIDh8p+vAo=; b=DV162+LLNAxgjNZ/r/XDLvJ0P/0zdtfcsI1H1zI4H2NcINHrW9OLjgXC79nCDYyNmp X/WqAYL9TjA+TVOQQ+EGHnbl5HFI1HVyVcc8qXS4a98aBXxSULt5G7JVk1NMHPaWPvGU jLoqqJc8PduVc/1Wtf4bQv5kbnO4CjUEf8vFxXxxZdhz2NnAy5Y6/EZ3Jn7DI4Q3mCtM J1WrXFeG/2wjbxNm8AZLEMIRYCWhvmKTcAxDNACIwwhllP8brphgzAbCWpgkjNdb4Jer wHr1YKNe66B6sZp9ZqImMyvh/ucMkfrICdGpRiDATaBxkTgCaQIwsHr0nTjuHeNAerOC 92Rw== X-Forwarded-Encrypted: i=1; AJvYcCWOeEtUPl3HImAN0XGoTl3OXVWxpmZA4NwWhMbyz1Y/aW+8piDJURDg4DnB5wzkGaBUJdp9lt6ZTw==@kvack.org X-Gm-Message-State: AOJu0YwfoXAp5y6Rxv/clxnOrKjj/AzodIarGiwiP964L6ZOL4NEJPMT kEdxKiubVYaQirnuFV9K892M7MrlsJEgQNYvgrbJEJzXm1qP7TMPsx5hq2YPSw0hUeaDbM9h3Vq +IgWPfwLCeLkKUMH92spgxWvycWYXUz7FBzU+Pp/ZAbbm+5HR X-Gm-Gg: ASbGncsBnY8NMXSD/YZm3FyCx566IvXsU7QOEYDS/jk7VY480nwOkS8MRuRY9qj20aF GISinHLXYKJShVSe1awTMgJ/1OdFiMEA3cGpwKXyj0ynGZPHeANmWf28VhYg0RNmjhbN80kCCAV iC+r4C7fym1bV/TAudW0DUFB7y7T22H7EQZ60sZ60NQAYLLY9iZR4xKk10Bze7wzv+3l65TVYt6 Lbwu5FZ5IS3TiZCLMaZ81yRKhiwAr9d9S71YwckJ2jbyd7/ked4TZCqKpQFBYewxsK1gflS2Mwg 2bQ817CGHJtqn/wtCd9Q6A4UEYwDfe1yfGGwz+F+8Q== X-Received: by 2002:a05:600c:4f43:b0:439:9828:c425 with SMTP id 5b1f17b1804b1-43ab0f2db61mr336635e9.7.1740416186361; Mon, 24 Feb 2025 08:56:26 -0800 (PST) X-Google-Smtp-Source: AGHT+IHWfaIC9jLsYbcLXi3Fdeg1qCi0/sJNUC8bLA+3qaJ5mWtkIjQvv3YtNjkQQj/btQBGxQxPlw== X-Received: by 2002:a05:600c:4f43:b0:439:9828:c425 with SMTP id 5b1f17b1804b1-43ab0f2db61mr336445e9.7.1740416185910; Mon, 24 Feb 2025 08:56:25 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-439b030b347sm111391345e9.26.2025.02.24.08.56.24 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:25 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 10/20] bit_spinlock: __always_inline (un)lock functions Date: Mon, 24 Feb 2025 17:55:52 +0100 Message-ID: <20250224165603.1434404-11-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: MHEXvPYY4LhHN7bbbnYdzlT0FsRdsjtqzta8EH-eMOo_1740416186 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 84B5D180015 X-Stat-Signature: i9n8ddp81ww66uy5dyqpjxtwjsj7e554 X-Rspam-User: X-HE-Tag: 1740416191-86869 X-HE-Meta: U2FsdGVkX185GY/Ks6+e6f6KlZ6g57Mc1gZ3HVyjYjTncE/QsA1V9CPooHrvQBbtYrd7pn6svgbT4GlsvUppwZzIFEg5TwrlEAZT7GiE+HH49oexCnUhxKEg9LkXI7Xrcz6l92ybrD6HEH8XVmAf8HoRxW48jMH6i2GF6YdjSKQpDO5t71VQA5IPtRkGzdtRJQM2/goU9R6S3RnTToSd9UtcftHjrb5S9K56jXpYaJ7GWs0bBBj28rQf6oH8zsmppDa5h8hoXdkTb+3Totr/F3kFqHVXHhFQp9mOCNHtrR+VyY/+TPBAyneeiwOe/dIk3ETcXIgINdzNnUnRLXp3lW6l6Sh27SPrCiU1KREMzRYiiP26R4bGjSzfHRb8Iwae77TzR2j/EBB5CPF/oKlDIzUq+DcrykpMz29yYo8ZG9Euq1wtjQCAos1zbG4g5a09elTvmaYUtN/JzMzS75wo8TJ71vXVyBUrUqZ4gfLbwPu84zrSmEGDj3J6Ojaas2IwxMs9M+MPgIBOpX3XT17hDLEebY9ms4bZz7hqjF53ddb2QMzyWwR5aol3kDKYOfjoZ5eNjZMEnC22B2whs4ifosfiQ9jvKkfd1ujd0OJs5dBXtaw6B9U0kHysXVkTnm7FFFTgEy2XwK6eglRWfpG9h3lAYTgOccC2mL+E2CRZ80S6b9ezt+kym2CFGXGiLe7gifFc0GGvz9UB8tfYsIS0QlS9H0Szx1l9uDWAkWMUTtziWQyZWKqSL6T1EGHXOSDFKO4EUr4KyIiZUZ9Z1dnbHb0S/6ANNwRZuwIjKKxBBPe52XOtDaQYXXopbWe/hwikhk5LC1xvTeDvfyDyN/hqViP3/HSOJKIPTPf3lNoX1MepPah9709Ut7ofq/4heHvOkXLPlHPRgCGKDhvK77lVkyfxChv5SwG5t4LWAAAq+cxrb8029tZMzTFndEPU5273v/zilvSeEqd3I6nvnBG VGmICPFz ehZdK/+IEcowke7UnTdnANH20yyxQkVweWungs2RcPYhgn9nYIn3L5BOCjZnL7MmOlCC/DwFGRRyD5ILCby9w5CznJj9OsHgwYucsgOvMKRMic5s7+unsHk3A45HUytGWWG6o/Ueo6y1tlTXhrY5ud41fD6RKFd8EFUmkyecjfkHVqekQNuc681Do11IQjw2rIFxyy0ec3grBFAZbRF34ZsFlJHom6wRWj3fbzKZFMONuo6IqqqwQieBZRHm0kSeHiwCj9b0/jpTgvVjuQ6yHe5CCS9zZl1BqrOdMruu5W7bRE4RXgHia/ruIZsiQ1iK9p7pq7F5DZPsbpNZ3SoetixTkwdfWFssJ6F7tHYOO8fUiIvWxRW0T2B+DJlhcMbPPZ+dvv6E2MTfmIjSikHrcOajF/rTDTaWS3GrB8oCbfulJSbp1RB1M1D11KDAU926VCD80c6JEYlJGVfIsMusGaA0WmyTzb9n7IpIEMx6dmql9u0buebn5GaKvvO83l7l+12njgSJYInkaEXRgdu6kAV6wZ6fDlumYdpLdZ+7GuV6m1LM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The compiler might decide that it is a smart idea to not inline bit_spin_lock(), primarily when a couple of functions in the same file end up calling it. Especially when used in RMAP map/unmap code next, the compiler sometimes decides to not inline, which is then observable in some micro-benchmarks. Let's simply flag all lock/unlock functions as __always_inline; arch_test_and_set_bit_lock() and friends are already tagged like that (but not test_and_set_bit_lock() for some reason). If ever a problem, we could split it into a fast and a slow path, and only force the fast path to be inlined. But there is nothing particularly "big" here. Signed-off-by: David Hildenbrand --- include/linux/bit_spinlock.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/bit_spinlock.h b/include/linux/bit_spinlock.h index bbc4730a6505c..c0989b5b0407f 100644 --- a/include/linux/bit_spinlock.h +++ b/include/linux/bit_spinlock.h @@ -13,7 +13,7 @@ * Don't use this unless you really need to: spin_lock() and spin_unlock() * are significantly faster. */ -static inline void bit_spin_lock(int bitnum, unsigned long *addr) +static __always_inline void bit_spin_lock(int bitnum, unsigned long *addr) { /* * Assuming the lock is uncontended, this never enters @@ -38,7 +38,7 @@ static inline void bit_spin_lock(int bitnum, unsigned long *addr) /* * Return true if it was acquired */ -static inline int bit_spin_trylock(int bitnum, unsigned long *addr) +static __always_inline int bit_spin_trylock(int bitnum, unsigned long *addr) { preempt_disable(); #if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK) @@ -54,7 +54,7 @@ static inline int bit_spin_trylock(int bitnum, unsigned long *addr) /* * bit-based spin_unlock() */ -static inline void bit_spin_unlock(int bitnum, unsigned long *addr) +static __always_inline void bit_spin_unlock(int bitnum, unsigned long *addr) { #ifdef CONFIG_DEBUG_SPINLOCK BUG_ON(!test_bit(bitnum, addr)); @@ -71,7 +71,7 @@ static inline void bit_spin_unlock(int bitnum, unsigned long *addr) * non-atomic version, which can be used eg. if the bit lock itself is * protecting the rest of the flags in the word. */ -static inline void __bit_spin_unlock(int bitnum, unsigned long *addr) +static __always_inline void __bit_spin_unlock(int bitnum, unsigned long *addr) { #ifdef CONFIG_DEBUG_SPINLOCK BUG_ON(!test_bit(bitnum, addr)); From patchwork Mon Feb 24 16:55:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988504 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8F2DC021BB for ; Mon, 24 Feb 2025 16:56:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A00ED6B00A0; Mon, 24 Feb 2025 11:56:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9AFB16B00A1; Mon, 24 Feb 2025 11:56:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7DAEC6B00A2; Mon, 24 Feb 2025 11:56:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5BCAD6B00A0 for ; Mon, 24 Feb 2025 11:56:36 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A7E4EB6E57 for ; Mon, 24 Feb 2025 16:56:35 +0000 (UTC) X-FDA: 83155441950.01.1035186 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 70D1340010 for ; Mon, 24 Feb 2025 16:56:33 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Y+psJHEr; spf=pass (imf11.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416193; a=rsa-sha256; cv=none; b=juDaAExN7SMe/SjxxybaySCNuqkmqisfDASl9HZ3gdaunbIQ2Z0RfVIln9rdmjQLF+GM82 wvicSKlt2XBjlwLFdoJj8QIItJ6QmStDdvTxieGhPaGqhz8+Pr/Jt2Z31wRJYYhds59En6 K4OASzqGhd2ZiXYL+9e/QVn5ro/PML8= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Y+psJHEr; spf=pass (imf11.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416193; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TqnqolcXMjdbgOoHxDkquZtgq0jreZJuwmbRkc4obqU=; b=Kz/CLDpJzBRuVTGxU+n3DFa2fPKJ34/3K7JYhqZyMCiqYB75BXgUhpwXH5K7b99O+8XOJf paQ+KH6wEEuX7dhEWvaUbYwH5TrqwyUmfVojIwFPLb9r5pHBXQOnYRzswfMGQJzY7JT+4A zU/IFiR0ENF4UWZxXGRHL3lapqQTjP0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416192; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TqnqolcXMjdbgOoHxDkquZtgq0jreZJuwmbRkc4obqU=; b=Y+psJHErEitDzn5R9xGERMXsH3nnTw+2eHmP81esVSwdj4v/y2TQofUw0crHEVu2p3klO5 BjlQJejvbY/bf76M5+IOAcAXUN+IDUPtLsXKFKiTc6IR7d4Et45oocR/KIylBRs6OEe8EC 57Qv/IfiKOaBpF/Ad0oCSq6xRbiQbfg= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-505-gieVFSQ_PxGX-heaTvTaAQ-1; Mon, 24 Feb 2025 11:56:29 -0500 X-MC-Unique: gieVFSQ_PxGX-heaTvTaAQ-1 X-Mimecast-MFC-AGG-ID: gieVFSQ_PxGX-heaTvTaAQ_1740416188 Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-43988b9ecfbso22627235e9.0 for ; Mon, 24 Feb 2025 08:56:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416188; x=1741020988; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TqnqolcXMjdbgOoHxDkquZtgq0jreZJuwmbRkc4obqU=; b=SaaEGCV/DA9du53+x2KC1b5F8utZB8tDAC/somZ8LGQH3fsUsY2Km0rdkQRKS6CKag sTqx9+K0V0OcmnLWEMVkpAFIQWrzeahRdUJ6nOs3jIz/bbVcBtT8aDdPPqkJJm1LPBiB Kw57qcq2AODvJOPfc2OwoTaXnSZ5Dz2sgWxk3cqttRhMktboOESNPv4Ywkd8y5OR1yHh F5paSy1JemC/CIyS62S/pppt6t54rb5X8wvmf//iCjFZq7LzIjs5YjJPKpDws0ledWWo RRNger3iGjFxJs5pg1xO+30m6rOQEYsEOWrbjvRNlUQ5CP1Q4p4b9pAT19MUYJsgCuf0 aptg== X-Forwarded-Encrypted: i=1; AJvYcCW5gIZq/zfx0Udqx6INARJAajKuLi1EAJYrdOeBmCCymTMTKxANbNdZ10pzo6YMO9mB+0h+7USNhg==@kvack.org X-Gm-Message-State: AOJu0YxK0YRwyL6LUNSRnvjzhyAXKze6wre2Vi2dcoBbMdjZl/b9vH1+ r2ngwPWkusAihT4zDaMgl12853qNuUqdxtPDx7roO4GqxElSZAnhIfmQipvEpd4PRhogKW2j9Yw KKww8YLIW+LaUut44XO9jRW9vSDucu5JZfYyTIO62KtRK2Ahxso7RHPnaYBE= X-Gm-Gg: ASbGnct7PB8V5/fu5c7nlKdHABW73FHoEKc33h0XKkWkdyDU1KEdW7xwHIGLswwJnuK QL5K5Wpz2DBY2OHBeKAuceplA3M5kDbFw/APVtwKe66xZI7nFYiR6Ma+3CPvW88nEWjf30GzD/c HY6GTWSM8gBuSJBFNSanl8+WqR1UpvWkKPDxSB+TyBtZD7E56BZP4KZgAbP98s7vdINRZSk8//v kBURaRQM5qDvdfKbQ1Gps04gjaOhVwMncu2b+nlGPfvLZlhXQ9Jw3bmmsaDTj74KNgWH4m5qp0Z pJTe7eJAPqmDSuUCnj7TQOb8AbwjhcyiOKqLAEsB4w== X-Received: by 2002:adf:e50d:0:b0:38f:2f9f:5249 with SMTP id ffacd0b85a97d-38f6e95b1acmr9525759f8f.19.1740416188298; Mon, 24 Feb 2025 08:56:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IEFpYW1I6Rdgq9wh83UDsQAXL1XjDcZxKn17wlosPtiSthbm9EkXmkPxjrqhOMZ4wG7mJ1LrA== X-Received: by 2002:adf:e50d:0:b0:38f:2f9f:5249 with SMTP id ffacd0b85a97d-38f6e95b1acmr9525729f8f.19.1740416187961; Mon, 24 Feb 2025 08:56:27 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38f259f8121sm31621063f8f.88.2025.02.24.08.56.26 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:27 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , "Kirill A. Shutemov" Subject: [PATCH v2 11/20] mm/rmap: use folio_large_nr_pages() in add/remove functions Date: Mon, 24 Feb 2025 17:55:53 +0100 Message-ID: <20250224165603.1434404-12-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: wEysZjVMvYgjTtUaamFUrw9TE0Di6LlV8RqzTOG2NUk_1740416188 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Stat-Signature: nyau6hey7d4iaow8o35qnausgq1q7fei X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 70D1340010 X-Rspam-User: X-HE-Tag: 1740416193-23223 X-HE-Meta: U2FsdGVkX19nJRF1YEa17IPoQz7TXXip/HVT6KUwyfKP2w6dZM3SMz5P7IfAlDItzrIbdEP++EOrbHB+Xov8yeMxpSWhwrIXHqCaVDj9kIEl/bXqgjjS2ygiO5oXqtudf8z5T/DhZ0z3vrYJIX0Etne2Kxrno0Dj08UrMzK+jnLODurgUhfh05uZQNaN+wI4ukdrYkGKuRMhcW28ej74xO3vPrarzetKXy7/PUovp9O2swbRt9A99eqfDoIGgyNxQUJPUC6At0vwxSJ+FpnM9SgAGj/pBHAFN7Bn5ir5lJXAdn1Arn51tcSntJWmhEC3TPDlJ9f0zzbMpw7oogI8JF2eC2JhspOJmxCxlfusdrgJR1PCHk8w52dNwI/7X7r0rDWzavQFRJgw1ByuaEMNGDkVTdoLTpKBwrsID+QJTa0mu5gEH9rq/fvTpL6C/zTVXlNU4IgSwyDumqtKzc5Q/3h6Pae97UtvZOUCqSOFV+9QCFKLeRJLPL6CVBwubX3dyBcSGLMkUwVcTTNuVcH5ghG1YT9RcIOHWZFHsd34fBAIrUaKkBntI3ABP6Kt1bonlJNbIeAzsh2t5bUqAj+l7GsGp36kF2MJDrxw2x8FrNa5y5qP5veB81jGeDk2O4+LhAeIKejhLaXAlPwyFc7HtUho/it/9bwQlK5BB2XuxC10NQxtUAQYIb7rvPVF48nxWvK9p1R0jm5l25oC0EfWW8VuoS63CjuJKZbHba2334Afh/eFJwrp6r7ToUQOWWporw6TY/V9zGtjdkg6MwgXu1IkLETdEhuoJTJZ2WN25VFwnbNTB45qPGnpsvlgMvc15lfey7SPz/V/eGiBoVhnL2uq7T0CpV+CuYrKfH33/lIl66mXNBQJPiPYi8Jdv4vgMy44naA5KDRm6lbfCq2D/Avjp+Wi22GTrU7+cW4lSFOVH51X7e44ML4Kxn3EepKj3eY9xXGSma/Z9e1W73E LCa2Jrj2 kRiXLK3kgM9j2es/CrplZfyTvOJ5XAZxlGkjEWFQMqFIoxVz12oZxvgGHNECUpEZrCxxYfsf1KsFNFOBsf+0jrx+byw4xyZyFsueK08ytv2jfJoqH0gbEPouIK50pUTg35UhhGiaE9MOj7cku4jNnD5bfm39bxO5azhgbFRJoybf42goWHt0NRKHZGd1sn1lDwYr+oQCDvNgUfbgcanPTOmpGmPRRg6sFK++Yy3UZ/gEzCs7lbUnFt+PJMLlelx/4fgsjlhH5quR3K4TzSPKp31yld4pLc0NGBthZwANXnjV7TK3KsD1Ifb/5/j/b5r5iU8Am7mN9ljJBbbd2Z2eZc9hCweNmA2kQIxjuMpAiOziifZRJm63X94/PQp9LZ2cG4zH8jokX9D9bz6fAyqVv6Ygr6YWPkf45p57uRgqRBTfXumdS+hyryALxkwiNko7BFw8dsIIH/kxlUOAJnQlqGpSZRetsg1dnt0wx4NSJzqvzWkawsOavxrHHBRtlfiewczd1un7x8F8z7lE2cHJsyhdcod6IbxuGgNverc60KqPUCule+dW5D+THgvxq45xN7wep X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's just use the "large" variant in code where we are sure that we have a large folio in our hands: this way we are sure that we don't perform any unnecessary "large" checks. While at it, convert the VM_BUG_ON_VMA to a VM_WARN_ON_ONCE. Maybe in the future there will not be a difference in that regard between large and small folios; in that case, unifying the handling again will be easy. E.g., folio_large_nr_pages() will simply translate to folio_nr_pages() until we replace all instances. Reviewed-by: Kirill A. Shutemov Signed-off-by: David Hildenbrand --- mm/rmap.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 08846b7eced60..c9922928616ee 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1274,7 +1274,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, if (first) { nr = atomic_add_return_relaxed(ENTIRELY_MAPPED, mapped); if (likely(nr < ENTIRELY_MAPPED + ENTIRELY_MAPPED)) { - nr_pages = folio_nr_pages(folio); + nr_pages = folio_large_nr_pages(folio); /* * We only track PMD mappings of PMD-sized * folios separately. @@ -1522,14 +1522,11 @@ void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page, void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address, rmap_t flags) { - const int nr = folio_nr_pages(folio); const bool exclusive = flags & RMAP_EXCLUSIVE; - int nr_pmdmapped = 0; + int nr = 1, nr_pmdmapped = 0; VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!exclusive && !folio_test_locked(folio), folio); - VM_BUG_ON_VMA(address < vma->vm_start || - address + (nr << PAGE_SHIFT) > vma->vm_end, vma); /* * VM_DROPPABLE mappings don't swap; instead they're just dropped when @@ -1547,6 +1544,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, } else if (!folio_test_pmd_mappable(folio)) { int i; + nr = folio_large_nr_pages(folio); for (i = 0; i < nr; i++) { struct page *page = folio_page(folio, i); @@ -1559,6 +1557,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, folio_set_large_mapcount(folio, nr, vma); atomic_set(&folio->_nr_pages_mapped, nr); } else { + nr = folio_large_nr_pages(folio); /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); folio_set_large_mapcount(folio, 1, vma); @@ -1568,6 +1567,9 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, nr_pmdmapped = nr; } + VM_WARN_ON_ONCE(address < vma->vm_start || + address + (nr << PAGE_SHIFT) > vma->vm_end); + __folio_mod_stat(folio, nr, nr_pmdmapped); mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON, 1); } @@ -1681,7 +1683,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, if (last) { nr = atomic_sub_return_relaxed(ENTIRELY_MAPPED, mapped); if (likely(nr < ENTIRELY_MAPPED)) { - nr_pages = folio_nr_pages(folio); + nr_pages = folio_large_nr_pages(folio); if (level == RMAP_LEVEL_PMD) nr_pmdmapped = nr_pages; nr = nr_pages - (nr & FOLIO_PAGES_MAPPED); From patchwork Mon Feb 24 16:55:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988506 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 140E5C021BC for ; Mon, 24 Feb 2025 16:56:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F735280003; Mon, 24 Feb 2025 11:56:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A823280002; Mon, 24 Feb 2025 11:56:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 283BD280003; Mon, 24 Feb 2025 11:56:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 00426280002 for ; Mon, 24 Feb 2025 11:56:38 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8A54580DC9 for ; Mon, 24 Feb 2025 16:56:38 +0000 (UTC) X-FDA: 83155442076.26.B01D107 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 1D19E18001F for ; Mon, 24 Feb 2025 16:56:35 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AxOXfKgw; spf=pass (imf16.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416196; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4nktJgh7Km5OllUbbPT14AR851zLdnVlyxLYdX13mgM=; b=oT7cnsnrJareqHzdCciw+v6VhKcoJEWvo0lSXCkYmJ6KMPApAfVGqhgitEyq/5XzT1z3lB RhlXwj213+NqMGoc6M5oANmnqM26T5Dr8dy5L85xV0zmDCZ0aZP8JttTMmYzc2H4jS8f97 g75K1sSRk0yMZhIUUkts+TK45uU5wxI= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AxOXfKgw; spf=pass (imf16.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416196; a=rsa-sha256; cv=none; b=mFLcInZ1rsvp/J4NIedJler9QQJlr4wpjP5z7fGOf1q+L2E3qjgH/GsbLig4k40SFkMVPZ PfgcVqJC1IzXdYOL+PEM0teZMrDWoeh4THY6pZt8g8WlNsieO+ACB9XjQIiWN5S4M9213Z jAJyIng9vitEMmgZZHPqtXfE8TKdzbo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416195; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4nktJgh7Km5OllUbbPT14AR851zLdnVlyxLYdX13mgM=; b=AxOXfKgwJGGIcMn4oCWDi+cBmb8D1UL6zVEjWyfHSbz2I13IrJ79Lj4Dsavs0G5PsvGiFl ukeyNmOvBCr9mLEPAQ8SSs6e4KPsG7ewEWjK4/i1xZv+Vfksz2neCEACoHWOQy1jZ8cfsv 5Se5BIZyRo34bjQW3Gm8Z1wQgpUuCXw= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-224-JICqNNekMOOBHsIcMWHguw-1; Mon, 24 Feb 2025 11:56:32 -0500 X-MC-Unique: JICqNNekMOOBHsIcMWHguw-1 X-Mimecast-MFC-AGG-ID: JICqNNekMOOBHsIcMWHguw_1740416191 Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-38f51a3a833so1827816f8f.2 for ; Mon, 24 Feb 2025 08:56:31 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416191; x=1741020991; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4nktJgh7Km5OllUbbPT14AR851zLdnVlyxLYdX13mgM=; b=TvgjY8zJxXeyj0Mdr22+E2F/cRvWKMs7XefIH8kSAFK2kF/+VyFKO/r5YpjjWiEqQj KoBxtEN32CQyVzmqzAYfQ/U2yVs2e0+yA/MvnOCPVm6Rv0fzPkmwvBYX2T3hoX1K/Mtx hOx7SlwSq0AL16eNF02m6hJ/X5ovt7OJHRz7hHGCnTtR97RbHY+6ITg/g9GM5iitQB7V PY2FRuOUa7Srs56nL2wgr2NjfEIa4/a2J4WMhYTExXzQHlW3SXeq07rH7R7nfU3Djeqk 7o+A02TopAQuIacRHojeOEvSsXhfV5RO9s0P8lK+UuLTeX2llqEjCdKaid48b/ceiPiT ie8A== X-Forwarded-Encrypted: i=1; AJvYcCVrsXJd7J/JGRKxjR+fqVu5ZIxDYUsZQOrBOxIl6t5SNZGwRT8LIrERCqR6siJkQVn9zXxmgcwejA==@kvack.org X-Gm-Message-State: AOJu0Yxq+W96icz+V/nUQdAYBBwYpQeFh8xfJLKMsvmeCqFSbvntNa+S tunW+nLXpEYffHqU8s7BpQcSPBUSZBAeo4w3A48+tcN5ZhPsCZSWEFZZE7Z1CZsDlJF/OmzXU3M FjKPgTg78irqyndm3iaIUk3u1JqZQWH5MIiqK4cDQ59qf4LAd X-Gm-Gg: ASbGncs94YsTTyXBIfX8C8i/l8+oicWHH4IZQmOk5fL6bxhAxWZajtTIoqSDpF8HUT5 kaZ/UWiYXCX4qXcthI+Hg/gULXTYJDHCw+Ljz0YyPh0HVQqoZN/WE5zeOh63QKKCSUY9PQsmIXu T/3EVNXKAc6OeGKgj7fHA/RtKv45krTQI6bolnZEg4cSGxnZDFQmAM9E39T/TC0S95z8Jo6f0of Zucu26Qcc4+4RPgxMJ7LQfX+v/TSTXubRAVUzwd4JCPvcoHv17OttzfdvOQd2VfKlovquzHJC2B tdwHtFkLW67qu+EQqA41HC6hJfptoBV9cUa/cJ5KXg== X-Received: by 2002:a05:6000:188d:b0:38d:df83:7142 with SMTP id ffacd0b85a97d-38f7078c0a9mr10960289f8f.22.1740416190668; Mon, 24 Feb 2025 08:56:30 -0800 (PST) X-Google-Smtp-Source: AGHT+IHolbkh2QWz49qXbuMSOIH32yACqqeS6botLOgN7rWbjv1cvtvSFiQIUCwh9g31Rr/N7UNb4A== X-Received: by 2002:a05:6000:188d:b0:38d:df83:7142 with SMTP id ffacd0b85a97d-38f7078c0a9mr10960239f8f.22.1740416190066; Mon, 24 Feb 2025 08:56:30 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38f258fc7ecsm31759439f8f.49.2025.02.24.08.56.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:29 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 12/20] mm/rmap: basic MM owner tracking for large folios (!hugetlb) Date: Mon, 24 Feb 2025 17:55:54 +0100 Message-ID: <20250224165603.1434404-13-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: fMI8KZ8FtNwLUSTla4xOZ22KZBSwO6N7qcadcUhO_Yg_1740416191 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Stat-Signature: 19is71r971wxfkw8ybbsawmnn6xwyp9b X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 1D19E18001F X-HE-Tag: 1740416195-614784 X-HE-Meta: U2FsdGVkX1+Zrf2c9SMyDkjyTiXWpCKwCCX1xixZD1TZ+zRxrv79wHMKtz16sL+yqsHyjG5kLcMN9I7Y24EODMyGr+IrSGJfHe0XiegbsCYjcvQ+LPoZoxnOkTIyRXhSjf4RS4RgJNIk03Qo5dLOKVGoDnRtBmqsYLqdj3Zeew95+hyjNhZt1b1nz1D7yfAAuYsAYSos47FLB3dqU3iMdmNW/2/+GrJXLHarne8sbF/7HSoBJ22//BPJdn6eQdQERstAAVontsYBbG9kjp5Q6BiDeDnqxRItEezz1SA/m7/mf7N4bvAHLW4pdudeNMovPXtN5wHkZ2dadtGROMRMp76r1JeDIFO5mp+AmOWCwpicfm6XmntemUQJKr25cnMPgScxNhNCheH6lpDUjxawu+O8T3uOV8LiRMv5k5Lsd/oi77WfprQQ4hWRRmV3NWxMmZnd0JYXPs6tu8q/vswDVGK+rzwiquxV+8VlnHzFKqm6xgOEQr75z+O0Wxc3HLQ2fpmHExqGJNHQn/eGppZtwIX1HMiutvNut50hKRk1n+YIwE5CUVsYBJI2cC9vdFtkJarKVoecbqcPFvEc5X+OgZIyfEnSv+oMHjiMjepq0OyErl3s0xOzTomhg+LM4xV15xiJWEhA9irBbiz51wxHM/wbyVev5lCY4LVoJ01pA/+zoQy2ocD71yWLf63aCF/OXFtjq91ifRDyv1nrbeTU4y3XAPGtQXkFtJpRj39zG5AoSff7rikG4Bj06dr+PlkNZO1L5cxcx9PLsKpkTGp1tYknMn+XdHEEfD5PzNgDoqUUzcPNo/N4Nj5VwYBaicvs771Zd7i8QVOEvNk2IRCzYqNmKFiAY2mxRM5o2fG2sWZWstpdSGzcT0Xltb4PtnITf7v+rycUavNs2uno8etHQ7gEUYPyyMlHHcZvBtly2T7dJ6rjUrxO5/W+jWs1WUZHosy4awEEBqtBKx5mFM/ syWoaHnl uJrubKT5fjSYgbxobCeg6qvuEk5ufQEybQyOUw/evPy3ajq1prdE2ucMV9PSbtSlPAzTck5dYHIlh/40Xrut+DOLXQdp55JdZb3HcSUsiuLCSylPQp1IrPqEwmKLfc5TIX+dROKSyVUIspnKBg2oFyTANF+ZSn/yuyoxpF+JpJ0eeTWzTAeLu3nebOVJABcwC8T4s0Qpn7ZFass7tMfJqpGu8SvEGKWu3bjBSfvDf/keDwWbG5dP91k1I1LW4T2XoKCzMjMtlOi8G/qFBlAILygWo0W9YdykmD+vwHhahgZwMg+BCT3mQqcVyodqe3CUba6SADNqeSSeP2MXHB6IHttbxOykWtxxgt7ZfnVopDFwGQaRpNAIwC7o+XB8LItzURq2bYkcwUbnu7HSMGrYA7LhUMx/vQnTcx3AnBK8wjN2K+8SgszeVn3Q4nq0b434rfB+XXXCIdxyLRjqP21gUVvNw8OIgKxkW3M0C6UH/VDX9B+hDp7Owce2jW0M310gqpF9IPu2wTtAVXMoo8lzbwvAjKokHt7j1DZwI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For small folios, we traditionally use the mapcount to decide whether it was "certainly mapped exclusively" by a single MM (mapcount == 1) or whether it "maybe mapped shared" by multiple MMs (mapcount > 1). For PMD-sized folios that were PMD-mapped, we were able to use a similar mechanism (single PMD mapping), but for PTE-mapped folios and in the future folios that span multiple PMDs, this does not work. So we need a different mechanism to handle large folios. Let's add a new mechanism to detect whether a large folio is "certainly mapped exclusively", or whether it is "maybe mapped shared". We'll use this information next to optimize CoW reuse for PTE-mapped anonymous THP, and to convert folio_likely_mapped_shared() to folio_maybe_mapped_shared(), independent of per-page mapcounts. For each large folio, we'll have two slots, whereby a slot stores: (1) an MM id: unique id assigned to each MM (2) a per-MM mapcount If a slot is unoccupied, it can be taken by the next MM that maps folio page. In addition, we'll remember the current state -- "mapped exclusively" vs. "maybe mapped shared" -- and use a bit spinlock to sync on updates and to reduce the total number of atomic accesses on updates. In the future, it might be possible to squeeze a proper spinlock into "struct folio". For now, keep it simple, as we require the whole thing with THP only, that is incompatible with RT. As we have to squeeze this information into the "struct folio" of even folios of order-1 (2 pages), and we generally want to reduce the required metadata, we'll assign each MM a unique ID that can fit into an int. In total, we can squeeze everything into 4x int (2x long) on 64bit. 32bit support is a bit challenging, because we only have 2x long == 2x int in order-1 folios. But we can make it work for now, because we neither expect many MMs nor very large folios on 32bit. We will reliably detect folios as "mapped exclusively" vs. "mapped shared" as long as only two MMs map pages of a folio at one point in time -- for example with fork() and short-lived child processes, or with apps that hand over state from one instance to another. As soon as three MMs are involved at the same time, we might detect "maybe mapped shared" although the folio is "mapped exclusively". Example 1: (1) App1 faults in a (shmem/file-backed) folio page -> Tracked as MM0 (2) App2 faults in a folio page -> Tracked as MM1 (4) App1 unmaps all folio pages -> We will detect "mapped exclusively". Example 2: (1) App1 faults in a (shmem/file-backed) folio page -> Tracked as MM0 (2) App2 faults in a folio page -> Tracked as MM1 (3) App3 faults in a folio page -> No slot available, tracked as "unknown" (4) App1 and App2 unmap all folio pages -> We will detect "maybe mapped shared". Make use of __always_inline to keep possible performance degradation when (un)mapping large folios to a minimum. Note: by squeezing the two flags into the "unsigned long" that stores the MM ids, we can use non-atomic __bit_spin_unlock() and non-atomic setting/clearing of the "maybe mapped shared" bit, effectively not adding any new atomics on the hot path when updating the large mapcount + new metadata, which further helps reduce the runtime overhead in micro-benchmarks. Signed-off-by: David Hildenbrand --- Documentation/mm/transhuge.rst | 8 ++ include/linux/mm_types.h | 49 ++++++++++ include/linux/page-flags.h | 4 + include/linux/rmap.h | 165 +++++++++++++++++++++++++++++++++ kernel/fork.c | 36 +++++++ mm/Kconfig | 4 + mm/internal.h | 5 + mm/page_alloc.c | 10 ++ 8 files changed, 281 insertions(+) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index a2cd8800d5279..baa17d718a762 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -120,11 +120,19 @@ pages: and also increment/decrement folio->_nr_pages_mapped by ENTIRELY_MAPPED when _entire_mapcount goes from -1 to 0 or 0 to -1. + We also maintain the two slots for tracking MM owners (MM ID and + corresponding mapcount), and the current status ("maybe mapped shared" vs. + "mapped exclusively"). + - map/unmap of individual pages with PTE entry increment/decrement page->_mapcount, increment/decrement folio->_large_mapcount and also increment/decrement folio->_nr_pages_mapped when page->_mapcount goes from -1 to 0 or 0 to -1 as this counts the number of pages mapped by PTE. + We also maintain the two slots for tracking MM owners (MM ID and + corresponding mapcount), and the current status ("maybe mapped shared" vs. + "mapped exclusively"). + split_huge_page internally has to distribute the refcounts in the head page to the tail pages before clearing all PG_head/tail bits from the page structures. It can be done easily for refcounts taken by page table diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index c83dd2f1ee25e..2d657ac8e9b0c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -292,6 +292,44 @@ typedef struct { #define NR_PAGES_IN_LARGE_FOLIO #endif +/* + * On 32bit, we can cut the required metadata in half, because: + * (a) PID_MAX_LIMIT implicitly limits the number of MMs we could ever have, + * so we can limit MM IDs to 15 bit (32767). + * (b) We don't expect folios where even a single complete PTE mapping by + * one MM would exceed 15 bits (order-15). + */ +#ifdef CONFIG_64BIT +typedef int mm_id_mapcount_t; +#define MM_ID_MAPCOUNT_MAX INT_MAX +typedef unsigned int mm_id_t; +#else /* !CONFIG_64BIT */ +typedef short mm_id_mapcount_t; +#define MM_ID_MAPCOUNT_MAX SHRT_MAX +typedef unsigned short mm_id_t; +#endif /* CONFIG_64BIT */ + +/* We implicitly use the dummy ID for init-mm etc. where we never rmap pages. */ +#define MM_ID_DUMMY 0 +#define MM_ID_MIN (MM_ID_DUMMY + 1) + +/* + * We leave the highest bit of each MM id unused, so we can store a flag + * in the highest bit of each folio->_mm_id[]. + */ +#define MM_ID_BITS ((sizeof(mm_id_t) * BITS_PER_BYTE) - 1) +#define MM_ID_MASK ((1U << MM_ID_BITS) - 1) +#define MM_ID_MAX MM_ID_MASK + +/* + * In order to use bit_spin_lock(), which requires an unsigned long, we + * operate on folio->_mm_ids when working on flags. + */ +#define FOLIO_MM_IDS_LOCK_BITNUM MM_ID_BITS +#define FOLIO_MM_IDS_LOCK_BIT BIT(FOLIO_MM_IDS_LOCK_BITNUM) +#define FOLIO_MM_IDS_SHARED_BITNUM (2 * MM_ID_BITS + 1) +#define FOLIO_MM_IDS_SHARED_BIT BIT(FOLIO_MM_IDS_SHARED_BITNUM) + /** * struct folio - Represents a contiguous set of bytes. * @flags: Identical to the page flags. @@ -318,6 +356,9 @@ typedef struct { * @_nr_pages_mapped: Do not use outside of rmap and debug code. * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). * @_nr_pages: Do not use directly, call folio_nr_pages(). + * @_mm_id: Do not use outside of rmap code. + * @_mm_ids: Do not use outside of rmap code. + * @_mm_id_mapcount: Do not use outside of rmap code. * @_hugetlb_subpool: Do not use directly, use accessor in hugetlb.h. * @_hugetlb_cgroup: Do not use directly, use accessor in hugetlb_cgroup.h. * @_hugetlb_cgroup_rsvd: Do not use directly, use accessor in hugetlb_cgroup.h. @@ -390,6 +431,11 @@ struct folio { atomic_t _entire_mapcount; atomic_t _pincount; #endif /* CONFIG_64BIT */ + mm_id_mapcount_t _mm_id_mapcount[2]; + union { + mm_id_t _mm_id[2]; + unsigned long _mm_ids; + }; }; unsigned long _usable_1[4]; }; @@ -1111,6 +1157,9 @@ struct mm_struct { #endif } lru_gen; #endif /* CONFIG_LRU_GEN_WALKS_MMU */ +#ifdef CONFIG_MM_ID + mm_id_t mm_id; +#endif /* CONFIG_MM_ID */ } __randomize_layout; /* diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 30fe3eb62b90c..01716710066df 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -1222,6 +1222,10 @@ static inline int folio_has_private(const struct folio *folio) return !!(folio->flags & PAGE_FLAGS_PRIVATE); } +static inline bool folio_test_large_maybe_mapped_shared(const struct folio *folio) +{ + return test_bit(FOLIO_MM_IDS_SHARED_BITNUM, &folio->_mm_ids); +} #undef PF_ANY #undef PF_HEAD #undef PF_NO_TAIL diff --git a/include/linux/rmap.h b/include/linux/rmap.h index d1e888cc97a58..c131b0efff0fa 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -13,6 +13,7 @@ #include #include #include +#include /* * The anon_vma heads a list of private "related" vmas, to scan if @@ -173,6 +174,169 @@ static inline void anon_vma_merge(struct vm_area_struct *vma, struct anon_vma *folio_get_anon_vma(const struct folio *folio); +#ifdef CONFIG_MM_ID +static __always_inline void folio_lock_large_mapcount(struct folio *folio) +{ + bit_spin_lock(FOLIO_MM_IDS_LOCK_BITNUM, &folio->_mm_ids); +} + +static __always_inline void folio_unlock_large_mapcount(struct folio *folio) +{ + __bit_spin_unlock(FOLIO_MM_IDS_LOCK_BITNUM, &folio->_mm_ids); +} + +static inline unsigned int folio_mm_id(const struct folio *folio, int idx) +{ + VM_WARN_ON_ONCE(idx != 0 && idx != 1); + return folio->_mm_id[idx] & MM_ID_MASK; +} + +static inline void folio_set_mm_id(struct folio *folio, int idx, mm_id_t id) +{ + VM_WARN_ON_ONCE(idx != 0 && idx != 1); + folio->_mm_id[idx] &= ~MM_ID_MASK; + folio->_mm_id[idx] |= id; +} + +static inline void __folio_large_mapcount_sanity_checks(const struct folio *folio, + int diff, mm_id_t mm_id) +{ + VM_WARN_ON_ONCE(!folio_test_large(folio) || folio_test_hugetlb(folio)); + VM_WARN_ON_ONCE(diff <= 0); + VM_WARN_ON_ONCE(mm_id < MM_ID_MIN || mm_id > MM_ID_MAX); + + /* + * Make sure we can detect at least one complete PTE mapping of the + * folio in a single MM as "exclusively mapped". This is primarily + * a check on 32bit, where we currently reduce the size of the per-MM + * mapcount to a short. + */ + VM_WARN_ON_ONCE(diff > folio_large_nr_pages(folio)); + VM_WARN_ON_ONCE(folio_large_nr_pages(folio) - 1 > MM_ID_MAPCOUNT_MAX); + + VM_WARN_ON_ONCE(folio_mm_id(folio, 0) == MM_ID_DUMMY && + folio->_mm_id_mapcount[0] != -1); + VM_WARN_ON_ONCE(folio_mm_id(folio, 0) != MM_ID_DUMMY && + folio->_mm_id_mapcount[0] < 0); + VM_WARN_ON_ONCE(folio_mm_id(folio, 1) == MM_ID_DUMMY && + folio->_mm_id_mapcount[1] != -1); + VM_WARN_ON_ONCE(folio_mm_id(folio, 1) != MM_ID_DUMMY && + folio->_mm_id_mapcount[1] < 0); + VM_WARN_ON_ONCE(!folio_mapped(folio) && + folio_test_large_maybe_mapped_shared(folio)); +} + +static __always_inline void folio_set_large_mapcount(struct folio *folio, + int mapcount, struct vm_area_struct *vma) +{ + __folio_large_mapcount_sanity_checks(folio, mapcount, vma->vm_mm->mm_id); + + VM_WARN_ON_ONCE(folio_mm_id(folio, 0) != MM_ID_DUMMY); + VM_WARN_ON_ONCE(folio_mm_id(folio, 1) != MM_ID_DUMMY); + + /* Note: mapcounts start at -1. */ + atomic_set(&folio->_large_mapcount, mapcount - 1); + folio->_mm_id_mapcount[0] = mapcount - 1; + folio_set_mm_id(folio, 0, vma->vm_mm->mm_id); +} + +static __always_inline void folio_add_large_mapcount(struct folio *folio, + int diff, struct vm_area_struct *vma) +{ + const mm_id_t mm_id = vma->vm_mm->mm_id; + int new_mapcount_val; + + folio_lock_large_mapcount(folio); + __folio_large_mapcount_sanity_checks(folio, diff, mm_id); + + new_mapcount_val = atomic_read(&folio->_large_mapcount) + diff; + atomic_set(&folio->_large_mapcount, new_mapcount_val); + + /* + * If a folio is mapped more than once into an MM on 32bit, we + * can in theory overflow the per-MM mapcount (although only for + * fairly large folios), turning it negative. In that case, just + * free up the slot and mark the folio "mapped shared", otherwise + * we might be in trouble when unmapping pages later. + */ + if (folio_mm_id(folio, 0) == mm_id) { + folio->_mm_id_mapcount[0] += diff; + if (!IS_ENABLED(CONFIG_64BIT) && unlikely(folio->_mm_id_mapcount[0] < 0)) { + folio->_mm_id_mapcount[0] = -1; + folio_set_mm_id(folio, 0, MM_ID_DUMMY); + folio->_mm_ids |= FOLIO_MM_IDS_SHARED_BIT; + } + } else if (folio_mm_id(folio, 1) == mm_id) { + folio->_mm_id_mapcount[1] += diff; + if (!IS_ENABLED(CONFIG_64BIT) && unlikely(folio->_mm_id_mapcount[1] < 0)) { + folio->_mm_id_mapcount[1] = -1; + folio_set_mm_id(folio, 1, MM_ID_DUMMY); + folio->_mm_ids |= FOLIO_MM_IDS_SHARED_BIT; + } + } else if (folio_mm_id(folio, 0) == MM_ID_DUMMY) { + folio_set_mm_id(folio, 0, mm_id); + folio->_mm_id_mapcount[0] = diff - 1; + /* We might have other mappings already. */ + if (new_mapcount_val != diff - 1) + folio->_mm_ids |= FOLIO_MM_IDS_SHARED_BIT; + } else if (folio_mm_id(folio, 1) == MM_ID_DUMMY) { + folio_set_mm_id(folio, 1, mm_id); + folio->_mm_id_mapcount[1] = diff - 1; + /* Slot 0 certainly has mappings as well. */ + folio->_mm_ids |= FOLIO_MM_IDS_SHARED_BIT; + } + folio_unlock_large_mapcount(folio); +} + +static __always_inline void folio_sub_large_mapcount(struct folio *folio, + int diff, struct vm_area_struct *vma) +{ + const mm_id_t mm_id = vma->vm_mm->mm_id; + int new_mapcount_val; + + folio_lock_large_mapcount(folio); + __folio_large_mapcount_sanity_checks(folio, diff, mm_id); + + new_mapcount_val = atomic_read(&folio->_large_mapcount) - diff; + atomic_set(&folio->_large_mapcount, new_mapcount_val); + + /* + * There are valid corner cases where we might underflow a per-MM + * mapcount (some mappings added when no slot was free, some mappings + * added once a slot was free), so we always set it to -1 once we go + * negative. + */ + if (folio_mm_id(folio, 0) == mm_id) { + folio->_mm_id_mapcount[0] -= diff; + if (folio->_mm_id_mapcount[0] >= 0) + goto out; + folio->_mm_id_mapcount[0] = -1; + folio_set_mm_id(folio, 0, MM_ID_DUMMY); + } else if (folio_mm_id(folio, 1) == mm_id) { + folio->_mm_id_mapcount[1] -= diff; + if (folio->_mm_id_mapcount[1] >= 0) + goto out; + folio->_mm_id_mapcount[1] = -1; + folio_set_mm_id(folio, 1, MM_ID_DUMMY); + } + + /* + * If one MM slot owns all mappings, the folio is mapped exclusively. + * Note that if the folio is now unmapped (new_mapcount_val == -1), both + * slots must be free (mapcount == -1), and we'll also mark it as + * exclusive. + */ + if (folio->_mm_id_mapcount[0] == new_mapcount_val || + folio->_mm_id_mapcount[1] == new_mapcount_val) + folio->_mm_ids &= ~FOLIO_MM_IDS_SHARED_BIT; +out: + folio_unlock_large_mapcount(folio); +} +#else /* !CONFIG_MM_ID */ +/* + * See __folio_rmap_sanity_checks(), we might map large folios even without + * CONFIG_TRANSPARENT_HUGEPAGE. We'll keep that working for now. + */ static inline void folio_set_large_mapcount(struct folio *folio, int mapcount, struct vm_area_struct *vma) { @@ -191,6 +355,7 @@ static inline void folio_sub_large_mapcount(struct folio *folio, { atomic_sub(diff, &folio->_large_mapcount); } +#endif /* CONFIG_MM_ID */ #define folio_inc_large_mapcount(folio, vma) \ folio_add_large_mapcount(folio, 1, vma) diff --git a/kernel/fork.c b/kernel/fork.c index 364b2d4fd3efa..f9cf0f056eb6f 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -802,6 +802,36 @@ static int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) #define mm_free_pgd(mm) #endif /* CONFIG_MMU */ +#ifdef CONFIG_MM_ID +static DEFINE_IDA(mm_ida); + +static inline int mm_alloc_id(struct mm_struct *mm) +{ + int ret; + + ret = ida_alloc_range(&mm_ida, MM_ID_MIN, MM_ID_MAX, GFP_KERNEL); + if (ret < 0) + return ret; + mm->mm_id = ret; + return 0; +} + +static inline void mm_free_id(struct mm_struct *mm) +{ + const mm_id_t id = mm->mm_id; + + mm->mm_id = MM_ID_DUMMY; + if (id == MM_ID_DUMMY) + return; + if (WARN_ON_ONCE(id < MM_ID_MIN || id > MM_ID_MAX)) + return; + ida_free(&mm_ida, id); +} +#else /* !CONFIG_MM_ID */ +static inline int mm_alloc_id(struct mm_struct *mm) { return 0; } +static inline void mm_free_id(struct mm_struct *mm) {} +#endif /* CONFIG_MM_ID */ + static void check_mm(struct mm_struct *mm) { int i; @@ -905,6 +935,7 @@ void __mmdrop(struct mm_struct *mm) WARN_ON_ONCE(mm == current->active_mm); mm_free_pgd(mm); + mm_free_id(mm); destroy_context(mm); mmu_notifier_subscriptions_destroy(mm); check_mm(mm); @@ -1289,6 +1320,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, if (mm_alloc_pgd(mm)) goto fail_nopgd; + if (mm_alloc_id(mm)) + goto fail_noid; + if (init_new_context(p, mm)) goto fail_nocontext; @@ -1308,6 +1342,8 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, fail_cid: destroy_context(mm); fail_nocontext: + mm_free_id(mm); +fail_noid: mm_free_pgd(mm); fail_nopgd: free_mm(mm); diff --git a/mm/Kconfig b/mm/Kconfig index 2761098dbc1a2..af72fe92f96b0 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -817,11 +817,15 @@ config ARCH_WANT_GENERAL_HUGETLB config ARCH_WANTS_THP_SWAP def_bool n +config MM_ID + def_bool n + menuconfig TRANSPARENT_HUGEPAGE bool "Transparent Hugepage Support" depends on HAVE_ARCH_TRANSPARENT_HUGEPAGE && !PREEMPT_RT select COMPACTION select XARRAY_MULTI + select MM_ID help Transparent Hugepages allows the kernel to use huge pages and huge tlb transparently to the applications whenever possible. diff --git a/mm/internal.h b/mm/internal.h index ffdc91b19322e..7303ddd9dac04 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -722,6 +722,11 @@ static inline void prep_compound_head(struct page *page, unsigned int order) folio_set_order(folio, order); atomic_set(&folio->_large_mapcount, -1); atomic_set(&folio->_nr_pages_mapped, 0); + if (IS_ENABLED(CONFIG_MM_ID)) { + folio->_mm_ids = 0; + folio->_mm_id_mapcount[0] = -1; + folio->_mm_id_mapcount[1] = -1; + } if (IS_ENABLED(CONFIG_64BIT) || order > 1) { atomic_set(&folio->_pincount, 0); atomic_set(&folio->_entire_mapcount, -1); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 7036530bd1bca..08caa92553998 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -955,6 +955,16 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) bad_page(page, "nonzero nr_pages_mapped"); goto out; } + if (IS_ENABLED(CONFIG_MM_ID)) { + if (unlikely(folio->_mm_id_mapcount[0] != -1)) { + bad_page(page, "nonzero mm mapcount 0"); + goto out; + } + if (unlikely(folio->_mm_id_mapcount[1] != -1)) { + bad_page(page, "nonzero mm mapcount 1"); + goto out; + } + } if (IS_ENABLED(CONFIG_64BIT)) { if (unlikely(atomic_read(&folio->_entire_mapcount) + 1)) { bad_page(page, "nonzero entire_mapcount"); From patchwork Mon Feb 24 16:55:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988505 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5683DC021BE for ; Mon, 24 Feb 2025 16:56:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7DE86280001; Mon, 24 Feb 2025 11:56:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 78B66280002; Mon, 24 Feb 2025 11:56:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E34B280001; Mon, 24 Feb 2025 11:56:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3B43F6B00A2 for ; Mon, 24 Feb 2025 11:56:38 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id ED63A1A18AB for ; Mon, 24 Feb 2025 16:56:37 +0000 (UTC) X-FDA: 83155442034.24.A922845 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 908261C001E for ; Mon, 24 Feb 2025 16:56:35 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TOSoDHGv; spf=pass (imf18.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416195; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pg2vY9vqWz7IHDYregwabj5LRZT1HxPxFKsGucVLMU4=; b=UDBarfazrK9ezY3iKJPKQlDQMOsUoxCiGRiRRg9Lh6bfmA2reuKZ4VORgvVoeYZE91X1CQ 2v1EA967Y3GMaZZ/GOwDjrBdNLwarQIwtTrq0K5PnAoF6yhJym4l6wk/r7dDabApxfp1b+ 5igQFbX0LxgY9KnbbNO1P8DfedWdvwE= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TOSoDHGv; spf=pass (imf18.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416195; a=rsa-sha256; cv=none; b=6G3D8JyeOfNcbyRAAQ2oRTYqxbSXvKgzbqamW9050Mnh6kQHd3Vq7eEqFdJjjoEvrgkNg/ eZ6UrYQGkPPoAu+oB65Ka3mepvrU+nAcrslyeLdX1rD2zteEVsgEU6YQyZkRqHV2JjucSs Cui3TWP4TKsWVsJjkKY7lX8X2vOuv70= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416194; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pg2vY9vqWz7IHDYregwabj5LRZT1HxPxFKsGucVLMU4=; b=TOSoDHGvtqDqGdAik58FKFsJa+3q6/FSRF/QyyxZNw5yZxWAMhXVoznESuMZTwDYhzp+Mg sx/wpBIloP3o42Ad3pJPv4gFKcCc8AVWL8ik1rocHJrzd2wU31FdGnbtOSW6nrMBli3v6P dtl8icNlqQU/lEWZ/SY/VC3fNDyvl74= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-686-6vr8IsOpMFGPOEfGNsPLYg-1; Mon, 24 Feb 2025 11:56:33 -0500 X-MC-Unique: 6vr8IsOpMFGPOEfGNsPLYg-1 X-Mimecast-MFC-AGG-ID: 6vr8IsOpMFGPOEfGNsPLYg_1740416192 Received: by mail-wr1-f72.google.com with SMTP id ffacd0b85a97d-38f4cce15c8so2207823f8f.2 for ; Mon, 24 Feb 2025 08:56:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416192; x=1741020992; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pg2vY9vqWz7IHDYregwabj5LRZT1HxPxFKsGucVLMU4=; b=EJb5Safj/3jxMJDXzk1k4giZ3eetAZnglQgT9dr/PavIK+hrI0B8mupYsc4K1SFkP7 nN/f4Rmvnnm/17QPNjWcgynqesFjgSYp8VKlKJ3C6Mi01f9rfMVc4pAppur/OtiWIDmV gy6GO65MsB+IlKU5pg2om2WJgdJZ1ZkcVRsnWmmMW1NCv5X8RM4LVsZ/nvlux1s+kP2B PUoUitPetl8ju85uxAX21Zqk0VI6oQSR7ZfFdpiQWLIibeTHhrV7MbzYLhFJpp4SI4gY 5oDBIMM2F77dyWLsc8S+wAPTH+D1lBqf1QYltyLdm88LERWT4WKlzbWLtuz+LJ1uexaV 1/fg== X-Forwarded-Encrypted: i=1; AJvYcCXMsjkgtGafhdz0pelkz2YxVdkb3XLOuPI78n1kQBP3wM2ppxbA+DxipG3N4ZwVo+ljfiZ1nILwEg==@kvack.org X-Gm-Message-State: AOJu0YwM99esIJrrXAdi6vLBFS0pF4zcnXPY911Qikdr+h/MLdKGuDDy 0AEREkcn6cswAQqhQ9x7VksgNfoCNrElyg1K5nwL35ixynUblpYCWyBn0HmQFfT4zVbCThyQlch l/M1wy7IKcEm5xDpFwxx6uCEokGc1j7gghUVrkqyzDWZlzMKp X-Gm-Gg: ASbGncuqti4PTJ9YB3mJvPqiieOQ/9XZkZ76xg31q/NU33hn8kfDG9hdO+paOuI2vWt lD6v0c41K/SfViXZbOM0To7EOBftoXifKIuANZTaqAqNNXQRN55JRkoN2XGhGeRNY7no49bzfYq 0ZpPIwT6Zw89LqqJsIMv3kIW4V+dnZh0ny1+ApjFX1nGSviQD/2mwoXASg9BI+iECUhlOeTxG5X xUTi8PsOXIwCbnZwfioXJqYSGZ/hB4jZHJUtshbulhKyg3NMgb/TQDe3TsSFw6Gj2mnsJOQjOr3 8PVac/4K7/oyAFcuJH6GIJgF4t9AmnENZeUDTDWF3Q== X-Received: by 2002:a5d:648a:0:b0:38d:db7b:5d90 with SMTP id ffacd0b85a97d-38f6f0951bbmr11757103f8f.41.1740416192491; Mon, 24 Feb 2025 08:56:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IHaIBsZiKHsVYYcLtCIPoeZB3FHIPU3YJhCLPdlvukEQ+/K5hEIgCDRy8AjGPWVYvGgArScYA== X-Received: by 2002:a5d:648a:0:b0:38d:db7b:5d90 with SMTP id ffacd0b85a97d-38f6f0951bbmr11757061f8f.41.1740416192060; Mon, 24 Feb 2025 08:56:32 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38f258b4163sm32531566f8f.19.2025.02.24.08.56.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:31 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 13/20] mm: Copy-on-Write (COW) reuse support for PTE-mapped THP Date: Mon, 24 Feb 2025 17:55:55 +0100 Message-ID: <20250224165603.1434404-14-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: yhunryq_A2e_j3NNtvf3JXUAK3jik3ndJDTDt69DLO8_1740416192 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 908261C001E X-Stat-Signature: f4w3oqpyonb9a8jew4yxm7xumirhjq78 X-HE-Tag: 1740416195-905444 X-HE-Meta: U2FsdGVkX18COP5vmDZOBBaPZysbK4zHgec9swPzTrNO1uRwmO1u1pKTIpcjRspt77L9EK30xBxReJhgynX8syq2ysrzrLN/idyMpZuVZ2uz2+WBHU1qzdhHcNHiwM/Ntr11qKCkLJ43rifAYmxUnrGoPAPWf1PjgaJav32eHKfq+DlvTQYxwtPwhrj+Hiw2zA19/DKU4FIT3U4ZwphSCO5+P9XjqppYPC1vWqRtLo7skmhiO8BZRI94xOx2zsbEcUZeYPWu5oD3OIoBUaap5UxfgSsmMHTZZteem+o/serVUf3gkXUNneNvc8F03nO6Ta2TWKNC5GgprbSv7FYnfacfsULiDzk1kGHGCXZ+0CiCnCxoHR6VItg7DA/Z65mjvp2IiqVUwComy/M01ViTRMuppJqsh0XuKLedFSsHpUTw4vXdcfHyFhiVAnheZHXgUjzKVbN/H7ACX5Q1bkjBpxEGT13spwh+BcbhBv499wLnFvE75VnWtQVE4qsBFXi+JffqXeMp383ebZE7hcK5KrcQHnT6Y6QRafF6XB9FBxewrTO0ibf06zektqJwmCDvYDvX5DtWkJrI+8Poi8tXVr8TIk0Klhxm2w9Gah5GNpFQI6ENZRSXjl4ic0nklKDLr1MbA7zzR8uumVBJTi67IIVgiv3clHvotyJEV5dNl7Eg32vjBJpwsFDXQQGxJiIfrxHbUgrcvlLqpxSDN1Dzb1vDNMJsR3ojUeo7NM2tqTxel2yQeRg6bj/n9f8JF6WaNxOeqJRQhMDfAbeISiMAut4IOIIjWRkk8jmo7Oh9swxjkWlB82t4AAwnH/vis02EHDEriFN9VVX7GvNJev/YYLIXQRZC9VviqIGZriM2TSWjfwVEQ7RpgXe/BZe+BIDZjMxECawS/cNWXLAd0W7mr8UkNY3M9DyJnCepnw9EjRqt20RB4YQBbzCdrJuPirOtusc9cpYQWA4X7M31n7z qrL2YF3o aeuX86z6pHiimiE+jEic91KCQ+c7Nrd01/BBO/E6lPzdqpT731pLX9YTamLcd9ONPSumj2H8S3eDvuY1NNucmnsCdvnmMdZ+o8rWPpC9/xFRfbj/tNZlXbW99L2AC5kCEKWXT86ldWOdIGAUkeW2wEmXmMfn8iwFkgpGSZoeS4hN/7uaguwVjRnrHnnlJJLv7WUM/PU6yogMseAQESxMT8KVVPMCOexxnigNNDBwP8Pn70/+JJgnfgEunN29mdzmVUbzXEq5+fg/T2I8psdvKwtlOCF8CkfWE1T8ZQK6JL548ItigJpy8IZnBl59qsSZ95Jj3H4e0YD8dUfbRlhu0YNvmHwFLQSsBVI71tZXjsFeotbcmuQBQfnZOwwVYwu9RAR+i1oZpCCy1ZSQTgsyLbw9aHsGYwb9mxdtYl+1BhNCTi3bqZ3QyE8yipZzimBsEVUyaPUjyz8ZTk/ZNdwKI04mCs/AHyAlqVpdtmAMCFW++6DzGpTnLZpvT8g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, we never end up reusing PTE-mapped THPs after fork. This wasn't really a problem with PMD-sized THPs, because they would have to be PTE-mapped first, but it's getting a problem with smaller THP sizes that are effectively always PTE-mapped. With our new "mapped exclusively" vs "maybe mapped shared" logic for large folios, implementing CoW reuse for PTE-mapped THPs is straight forward: if exclusively mapped, make sure that all references are from these (our) mappings. Add some helpful comments to explain the details. CONFIG_TRANSPARENT_HUGEPAGE selects CONFIG_MM_ID. If we spot an anon large folio without CONFIG_TRANSPARENT_HUGEPAGE in that code, something is seriously messed up. There are plenty of things we can optimize in the future: For example, we could remember that the folio is fully exclusive so we could speedup the next fault further. Also, we could try "faulting around", turning surrounding PTEs that map the same folio writable. But especially the latter might increase COW latency, so it would need further investigation. Signed-off-by: David Hildenbrand --- mm/memory.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 76 insertions(+), 8 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index ef74c4f009f84..8dc241961b684 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3706,19 +3706,87 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) return ret; } -static bool wp_can_reuse_anon_folio(struct folio *folio, - struct vm_area_struct *vma) +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static bool __wp_can_reuse_large_anon_folio(struct folio *folio, + struct vm_area_struct *vma) { + bool exclusive = false; + + /* Let's just free up a large folio if only a single page is mapped. */ + if (folio_large_mapcount(folio) <= 1) + return false; + /* - * We could currently only reuse a subpage of a large folio if no - * other subpages of the large folios are still mapped. However, - * let's just consistently not reuse subpages even if we could - * reuse in that scenario, and give back a large folio a bit - * sooner. + * The assumption for anonymous folios is that each page can only get + * mapped once into each MM. The only exception are KSM folios, which + * are always small. + * + * Each taken mapcount must be paired with exactly one taken reference, + * whereby the refcount must be incremented before the mapcount when + * mapping a page, and the refcount must be decremented after the + * mapcount when unmapping a page. + * + * If all folio references are from mappings, and all mappings are in + * the page tables of this MM, then this folio is exclusive to this MM. */ - if (folio_test_large(folio)) + if (folio_test_large_maybe_mapped_shared(folio)) + return false; + + VM_WARN_ON_ONCE(folio_test_ksm(folio)); + VM_WARN_ON_ONCE(folio_mapcount(folio) > folio_nr_pages(folio)); + VM_WARN_ON_ONCE(folio_entire_mapcount(folio)); + + if (unlikely(folio_test_swapcache(folio))) { + /* + * Note: freeing up the swapcache will fail if some PTEs are + * still swap entries. + */ + if (!folio_trylock(folio)) + return false; + folio_free_swap(folio); + folio_unlock(folio); + } + + if (folio_large_mapcount(folio) != folio_ref_count(folio)) return false; + /* Stabilize the mapcount vs. refcount and recheck. */ + folio_lock_large_mapcount(folio); + VM_WARN_ON_ONCE(folio_large_mapcount(folio) < folio_ref_count(folio)); + + if (folio_test_large_maybe_mapped_shared(folio)) + goto unlock; + if (folio_large_mapcount(folio) != folio_ref_count(folio)) + goto unlock; + + VM_WARN_ON_ONCE(folio_mm_id(folio, 0) != vma->vm_mm->mm_id && + folio_mm_id(folio, 1) != vma->vm_mm->mm_id); + + /* + * Do we need the folio lock? Likely not. If there would have been + * references from page migration/swapout, we would have detected + * an additional folio reference and never ended up here. + */ + exclusive = true; +unlock: + folio_unlock_large_mapcount(folio); + return exclusive; +} +#else /* !CONFIG_TRANSPARENT_HUGEPAGE */ +static bool __wp_can_reuse_large_anon_folio(struct folio *folio, + struct vm_area_struct *vma) +{ + BUILD_BUG(); +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + +static bool wp_can_reuse_anon_folio(struct folio *folio, + struct vm_area_struct *vma) +{ + if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && folio_test_large(folio)) + return __wp_can_reuse_large_anon_folio(folio, vma); + VM_WARN_ON_ONCE(folio_test_large(folio)); + /* * We have to verify under folio lock: these early checks are * just an optimization to avoid locking the folio and freeing From patchwork Mon Feb 24 16:55:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E17F2C021BB for ; Mon, 24 Feb 2025 16:56:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC1FA280004; Mon, 24 Feb 2025 11:56:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C5501280005; Mon, 24 Feb 2025 11:56:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2763280004; Mon, 24 Feb 2025 11:56:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7C512280002 for ; Mon, 24 Feb 2025 11:56:43 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 34AC51411D3 for ; Mon, 24 Feb 2025 16:56:43 +0000 (UTC) X-FDA: 83155442286.19.A868983 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf15.hostedemail.com (Postfix) with ESMTP id CB164A000A for ; Mon, 24 Feb 2025 16:56:40 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ShL5ZJgF; spf=pass (imf15.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416200; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SzcTE+N2KyOzlARv+w+SRDPyrebeHo8NXdkT+2IHowQ=; b=tnPhGRNyu5QZmBhW9cSvuWFaTlPQqm1ywqt5Xu+SETOEsbHKKXdpAOwpSEq7DIXExcnEFq /5W5QawcDMDJsb2NdUQ7EL5fx5xysdm7VB3QJjpE1D2Y2Q46ytW28yWf5alJpHkt1tabE+ LzfH3M6o0P5QFXZxlHZF787uqwWl43k= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ShL5ZJgF; spf=pass (imf15.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416200; a=rsa-sha256; cv=none; b=3An+gvEJlwifoqp3S45TaJowXcWYzWeZaqHOxawuIK1rKtsvBrfZUpGNCs16Rlu/esfVZd uKeNCQ1yj70pEqhPfbTuLHccUGSaNpeYeKF2+jAPfT3571cuOBAaJ9w0HEgb3NKT6MifXG eDklETDec9ZuER7ndglk/k7S355z6rQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416199; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SzcTE+N2KyOzlARv+w+SRDPyrebeHo8NXdkT+2IHowQ=; b=ShL5ZJgF3yLA6iW7voWsyhxJRWoqu2YrxLBa/fkR5Lk0c9BBNv1esEcVzoQSy8yOXDfLaL LTtiqT2uDPcykwQYp7ByhirJUTpb1FprRM60E/PHCoNgUysVvN8QnUOO7egRJcm+dSLnlo mLPeiRHVvZYJMTJqXBEzynVFWRqnONw= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-297-Q_x--m_ROT69wpbzXeVwzQ-1; Mon, 24 Feb 2025 11:56:36 -0500 X-MC-Unique: Q_x--m_ROT69wpbzXeVwzQ-1 X-Mimecast-MFC-AGG-ID: Q_x--m_ROT69wpbzXeVwzQ_1740416195 Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-38f36fcf4b3so2856484f8f.1 for ; Mon, 24 Feb 2025 08:56:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416195; x=1741020995; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SzcTE+N2KyOzlARv+w+SRDPyrebeHo8NXdkT+2IHowQ=; b=H2iwWao62TezHoIWmJQogURJGC0r/5pSXOXn2QcnVlfoXHWQCiQP+W3w6BUDECU8+b Zcoy+S+CM1fdR+m9pe1RfwhHCw9n4lRN2kmH9Jigs/1VmOGPh7cSSqd/qeezZXbUNk1G nTsjeQZKR5oq0VE6KJ9+i/IxbQOvt7v57eQp1RK2cZUX9eag8nUOKwwSZqdtjHsdzErR B8b3sPiQFNzMmgV0tbG1+7M21JgyF+B0xHWXadLsnJOf8LUT9Q9sJBkfYAlHmDuCBNQQ aDgh+zG2Ob3jW2kczJ+XHh/7MQlmBRNCX5/JfeUbsrGljzwzRzKVVtx/jP6vdnhqUaLu c7Pg== X-Forwarded-Encrypted: i=1; AJvYcCXhnDbYd44JzfGMRh88EeqsrjWhkbWVawIn0aoSOBrZWP6igDc88IF6GzB/r9XHlxGd6JzKSpWJVw==@kvack.org X-Gm-Message-State: AOJu0Yz2+yOcxXJS3nrkBdQGGdSa5yci6DZRlweM78OaL/wDmrx5CdsR yFoqka5MM8fsmqXk8YYAFfI6SOx2YS33hOY08gfUhQdgbH5iwFkhD7HEHh3VaqxYReaIw3AHrDr VpBJFH7g87TSfSSfeHjXXgHz6mKPsZbJM0sQ0HdN9J9zxtPty X-Gm-Gg: ASbGncsxPp6QfctMp0bXgJIt0WDnMQl4G8XoDD/qw1Rt03RTJTzmdEm0klcNn0JCYeg 3VAFSOzWH1yfgcLHW185AsaltPfZ8/xRGc81JuSfbSgcTK22jK3iSEYv6LzJNMJ0BPbA20ZBvrf lpDqGA5nPgFlOTf30AWwbqN9oSMKilY90PMI8sXLjXuKc7j0whNz2m8JW2kuLcZP4lPbie8TaNc QkZpwh5dIuSQFMUIHEOVozzjIzN+vlXbTNnNp258ycduh/HTHxjSQg511fX9s8tMRZrJdeh+hnj TIVl7dzdRTTeMgp4jOj2FkvRQynemtC6cGtni18bMw== X-Received: by 2002:adf:f20f:0:b0:38f:2b34:5004 with SMTP id ffacd0b85a97d-38f6f0863c9mr12658016f8f.38.1740416194684; Mon, 24 Feb 2025 08:56:34 -0800 (PST) X-Google-Smtp-Source: AGHT+IEXaXs4/LysHX+bj1gIx5L2JWtgUcH0vURwv8svqv3a7UoxYuTerlEfxa7PfRrFkuwf5fEjKw== X-Received: by 2002:adf:f20f:0:b0:38f:2b34:5004 with SMTP id ffacd0b85a97d-38f6f0863c9mr12657972f8f.38.1740416194154; Mon, 24 Feb 2025 08:56:34 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-439b0372124sm110600815e9.39.2025.02.24.08.56.32 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:33 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 14/20] mm: convert folio_likely_mapped_shared() to folio_maybe_mapped_shared() Date: Mon, 24 Feb 2025 17:55:56 +0100 Message-ID: <20250224165603.1434404-15-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Ta_GCi2--b4DvuHlqnUyBhYuAZlB_ATzuAr2x4wVjoI_1740416195 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: CB164A000A X-Stat-Signature: tdxhoxq3typuo3f9mxedxkyd8e3iwxtp X-HE-Tag: 1740416200-445244 X-HE-Meta: U2FsdGVkX1+ljpYTvZ6nmiWi4+N4OYL0fQkKFkdFZeAPrEpEftHw4O821n2pg9Y8cLZOsrrVK36UI+1Kp+VHmWVwqufS0x0LLEcQBJeI6C0Mb4jh/Fke8kcrDtaczz8YztFnJ7uBOSPhAaTLhPU/zqCVLmdLKkFbPQZFNm7i78inn8O3hAF9Ks71HaLNmz5jb2EFqyNbnvyEFlLDyRsxfnHr+MxitvwgwfLJ1uLnzwlrJbAg+/JGCJKANw9XS/yY8eXM1i3Fnsoi9+5sAK3ZfRpWPtMfQHmRy4ZZirIRU9iirQJ3meiwvSQix5SVwVsgiNI6Q0Z5YdLHs4wj+RmK0/WDFmAsNhV8/C+08gPBPs5L2AdNxVp4uhAsuZ+e6uVvaAlhMi4J70c1IVTu89RsxI2zv2zE1fOoBfGUcCBJmVXYIjYXGoRC5HelItigIh/87U9ra/g+1628mHIXiktFzW6dity9aHBkhRDAJCnE1jmT/zYkV6P+xZLzjhG5YffZAad3S7WIW2AhhT2eU1Fbb5bv87Vm7ZvGUpHhvazV7rBRoUp8SIMvtVadz+QjUg2V2JqubPdnyrif96KsxdfAK7WARVcLYZexQYa38/DX9uSyG8mTyFSRyIL+nEHW7XTqKSkJxhGU+VwBtFuE1XG2oP6piygi5kfOo1BU1HJIMztkQaSLcgAz3tJSXCYgEKmATZ6j7/oygcSXdidx6lRzBFpt/uEdwhlvDHD4PMhauJn1okwimuYanWe/+hTqJipsAvdEZ4hMMeXHfG77FpfKBCExnCPXj4/y9X2oIxkXUBU+vQOtzIi1ZGtvo+Mn3hMIvICimUUO8QukCZV9Y9hmks/BzrzRb++UijGz0W888DRyYan1CvXAUqrDDCLUXYZ7A6sAmAXQKntkyddPzH5hCmv2YOWO+zBy9UHNUxWeF4/2XzH3bGjIFmphAtjj8NQE2F43bp9pkzhz93i8Hsv +FXoran4 Y58biv7RnGzRBi+3T2IKnlmAWL9YaA4cXw5pseBWPAmg02RMylgDFCb+ELi9VkTQgbEB1Tvnwz/mV7BoPxM5i+oBEgWCRpJf2Zh/3aq98nTbigEji5adhm51cas12Rr8uQb92YDn2m70T/OSazWPFWu05/ntlEj1FaTzJPzRskw4qAZQuiaF24voK7oL3pwdAzoNmacuJ+UXu2j8pOVR4+SzXww0Oc3sJ+N7Qjwmv/l7UnDMdpC4WZInsmnZvJO0SNw8NAIVfuXLL8egKLZ5R4mUFOm2BvMnEOdJnOVhDBSVtVmZgowGskwYPKetT/1BLiywIj6l9RLpwkj72kgGgcuTXCTSA4Ray5U2zdMI9OWLScFI6n3p/qq+YMjzOMew8bR5aHMivZ+XcJnAY93NIuD0Af9uRG8Am7/QrrxVK3YqD8ItmQBIF3JH0yx6Bz4DotQoJoo55C3rkl5wP0D3RsCpNZcD98gnhaT/szJnFf2wyCQldwOt5eVMMGk26faMBE13vzfqbinD71uG5x1BkWLdgrAoh483ohZtYS4Cd/Hf4TsI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's reuse our new MM ownership tracking infrastructure for large folios to make folio_likely_mapped_shared() never return false negatives -- never indicating "not mapped shared" although the folio *is* mapped shared. With that, we can rename it to folio_maybe_mapped_shared() and get rid of the dependency on the mapcount of the first folio page. The semantics are now arguably clearer: no mixture of "false negatives" and "false positives", only the remaining possibility for "false positives". Thoroughly document the new semantics. We might now detect that a large folio is "maybe mapped shared" although it *no longer* is -- but once was. Now, if more than two MMs mapped a folio at the same time, and the MM mapping the folio exclusively at the end is not one tracked in the two folio MM slots, we will detect the folio as "maybe mapped shared". For anonymous folios, usually (except weird corner cases) all PTEs that target a "maybe mapped shared" folio are R/O. As soon as a child process would write to them (iow, actively use them), we would CoW and effectively replace these PTEs. Most cases (below) are not expected to really matter with large anonymous folios for this reason. Most importantly, there will be no change at all for: * small folios * hugetlb folios * PMD-mapped PMD-sized THPs (single mapping) This change has the potential to affect existing callers of folio_likely_mapped_shared() -> folio_maybe_mapped_shared(): (1) fs/proc/task_mmu.c: no change (hugetlb) (2) khugepaged counts PTEs that target shared folios towards max_ptes_shared (default: HPAGE_PMD_NR / 2), meaning we could skip a collapse where we would have previously collapsed. This only applies to anonymous folios and is not expected to matter in practice. Worth noting that this change sorts out case (A) documented in commit 1bafe96e89f0 ("mm/khugepaged: replace page_mapcount() check by folio_likely_mapped_shared()") by removing the possibility for "false negatives". (3) MADV_COLD / MADV_PAGEOUT / MADV_FREE will not try splitting PTE-mapped THPs that are considered shared but not fully covered by the requested range, consequently not processing them. PMD-mapped PMD-sized THP are not affected, or when all PTEs are covered. These functions are usually only called on anon/file folios that are exclusively mapped most of the time (no other file mappings or no fork()), so the "false negatives" are not expected to matter in practice. (4) mbind() / migrate_pages() / move_pages() will refuse to migrate shared folios unless MPOL_MF_MOVE_ALL is effective (requires CAP_SYS_NICE). We will now reject some folios that could be migrated. Similar to (3), especially with MPOL_MF_MOVE_ALL, so this is not expected to matter in practice. Note that cpuset_migrate_mm_workfn() calls do_migrate_pages() with MPOL_MF_MOVE_ALL. (5) NUMA hinting mm/migrate.c:migrate_misplaced_folio_prepare() will skip file folios that are probably shared libraries (-> "mapped shared" and executable). This check would have detected it as a shared library at some point (at least 3 MMs mapping it), so detecting it afterwards does not sound wrong (still a shared library). Not expected to matter. mm/memory.c:numa_migrate_check() will indicate TNF_SHARED in MAP_SHARED file mappings when encountering a shared folio. Similar reasoning, not expected to matter. mm/mprotect.c:change_pte_range() will skip folios detected as shared in CoW mappings. Similarly, this is not expected to matter in practice, but if it would ever be a problem we could relax that check a bit (e.g., basing it on the average page-mapcount in a folio), because it was only an optimization when many (e.g., 288) processes were mapping the same folios -- see commit 859d4adc3415 ("mm: numa: do not trap faults on shared data section pages.") (6) mm/rmap.c:folio_referenced_one() will skip exclusive swapbacked folios in dying processes. Applies to anonymous folios only. Without "false negatives", we'll now skip all actually shared ones. Skipping ones that are actually exclusive won't really matter, it's a pure optimization, and is not expected to matter in practice. In theory, one can detect the problematic scenario: folio_mapcount() > 0 and no folio MM slot is occupied ("state unknown"). One could reset the MM slots while doing an rmap walk, which migration / folio split already do when setting everything up. Further, when batching PTEs we might naturally learn about a owner (e.g., folio_mapcount() == nr_ptes) and could update the owner. However, we'll defer that until the scenarios where it would really matter are clear. Signed-off-by: David Hildenbrand --- fs/proc/task_mmu.c | 4 ++-- include/linux/mm.h | 43 ++++++++++++++++++++++--------------------- mm/huge_memory.c | 2 +- mm/khugepaged.c | 8 +++----- mm/madvise.c | 6 +++--- mm/memory.c | 2 +- mm/mempolicy.c | 8 ++++---- mm/migrate.c | 7 +++---- mm/mprotect.c | 2 +- mm/rmap.c | 2 +- 10 files changed, 41 insertions(+), 43 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index f02cd362309a0..2bddcea65cbf1 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1023,7 +1023,7 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, if (folio) { /* We treat non-present entries as "maybe shared". */ - if (!present || folio_likely_mapped_shared(folio) || + if (!present || folio_maybe_mapped_shared(folio) || hugetlb_pmd_shared(pte)) mss->shared_hugetlb += huge_page_size(hstate_vma(vma)); else @@ -1879,7 +1879,7 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask, if (!folio_test_anon(folio)) flags |= PM_FILE; - if (!folio_likely_mapped_shared(folio) && + if (!folio_maybe_mapped_shared(folio) && !hugetlb_pmd_shared(ptep)) flags |= PM_MMAP_EXCLUSIVE; diff --git a/include/linux/mm.h b/include/linux/mm.h index 9c1290588a11e..98a67488b5fef 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2245,23 +2245,18 @@ static inline size_t folio_size(const struct folio *folio) } /** - * folio_likely_mapped_shared - Estimate if the folio is mapped into the page - * tables of more than one MM + * folio_maybe_mapped_shared - Whether the folio is mapped into the page + * tables of more than one MM * @folio: The folio. * - * This function checks if the folio is currently mapped into more than one - * MM ("mapped shared"), or if the folio is only mapped into a single MM - * ("mapped exclusively"). + * This function checks if the folio maybe currently mapped into more than one + * MM ("maybe mapped shared"), or if the folio is certainly mapped into a single + * MM ("mapped exclusively"). * * For KSM folios, this function also returns "mapped shared" when a folio is * mapped multiple times into the same MM, because the individual page mappings * are independent. * - * As precise information is not easily available for all folios, this function - * estimates the number of MMs ("sharers") that are currently mapping a folio - * using the number of times the first page of the folio is currently mapped - * into page tables. - * * For small anonymous folios and anonymous hugetlb folios, the return * value will be exactly correct: non-KSM folios can only be mapped at most once * into an MM, and they cannot be partially mapped. KSM folios are @@ -2269,8 +2264,8 @@ static inline size_t folio_size(const struct folio *folio) * * For other folios, the result can be fuzzy: * #. For partially-mappable large folios (THP), the return value can wrongly - * indicate "mapped exclusively" (false negative) when the folio is - * only partially mapped into at least one MM. + * indicate "mapped shared" (false positive) if a folio was mapped by + * more than two MMs at one point in time. * #. For pagecache folios (including hugetlb), the return value can wrongly * indicate "mapped shared" (false positive) when two VMAs in the same MM * cover the same file range. @@ -2287,7 +2282,7 @@ static inline size_t folio_size(const struct folio *folio) * * Return: Whether the folio is estimated to be mapped into more than one MM. */ -static inline bool folio_likely_mapped_shared(struct folio *folio) +static inline bool folio_maybe_mapped_shared(struct folio *folio) { int mapcount = folio_mapcount(folio); @@ -2295,16 +2290,22 @@ static inline bool folio_likely_mapped_shared(struct folio *folio) if (!folio_test_large(folio) || unlikely(folio_test_hugetlb(folio))) return mapcount > 1; - /* A single mapping implies "mapped exclusively". */ - if (mapcount <= 1) - return false; - - /* If any page is mapped more than once we treat it "mapped shared". */ - if (folio_entire_mapcount(folio) || mapcount > folio_nr_pages(folio)) + /* + * vm_insert_page() without CONFIG_TRANSPARENT_HUGEPAGE ... + * simply assume "mapped shared", nobody should really care + * about this for arbitrary kernel allocations. + */ + if (!IS_ENABLED(CONFIG_MM_ID)) return true; - /* Let's guess based on the first subpage. */ - return atomic_read(&folio->_mapcount) > 0; + /* + * A single mapping implies "mapped exclusively", even if the + * folio flag says something different: it's easier to handle this + * case here instead of on the RMAP hot path. + */ + if (mapcount <= 1) + return false; + return folio_test_large_maybe_mapped_shared(folio); } #ifndef HAVE_ARCH_MAKE_FOLIO_ACCESSIBLE diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a3264d88d4b49..d9a7614fe739a 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2155,7 +2155,7 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, * If other processes are mapping this folio, we couldn't discard * the folio unless they all do MADV_FREE so let's skip the folio. */ - if (folio_likely_mapped_shared(folio)) + if (folio_maybe_mapped_shared(folio)) goto out; if (!folio_trylock(folio)) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 5f0be134141e8..cc945c6ab3bdb 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -607,7 +607,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); /* See hpage_collapse_scan_pmd(). */ - if (folio_likely_mapped_shared(folio)) { + if (folio_maybe_mapped_shared(folio)) { ++shared; if (cc->is_khugepaged && shared > khugepaged_max_ptes_shared) { @@ -1359,11 +1359,9 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, /* * We treat a single page as shared if any part of the THP - * is shared. "False negatives" from - * folio_likely_mapped_shared() are not expected to matter - * much in practice. + * is shared. */ - if (folio_likely_mapped_shared(folio)) { + if (folio_maybe_mapped_shared(folio)) { ++shared; if (cc->is_khugepaged && shared > khugepaged_max_ptes_shared) { diff --git a/mm/madvise.c b/mm/madvise.c index e01e93e179a8a..388dc289b5d12 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -387,7 +387,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, folio = pmd_folio(orig_pmd); /* Do not interfere with other mappings of this folio */ - if (folio_likely_mapped_shared(folio)) + if (folio_maybe_mapped_shared(folio)) goto huge_unlock; if (pageout_anon_only_filter && !folio_test_anon(folio)) @@ -486,7 +486,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, if (nr < folio_nr_pages(folio)) { int err; - if (folio_likely_mapped_shared(folio)) + if (folio_maybe_mapped_shared(folio)) continue; if (pageout_anon_only_filter && !folio_test_anon(folio)) continue; @@ -721,7 +721,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, if (nr < folio_nr_pages(folio)) { int err; - if (folio_likely_mapped_shared(folio)) + if (folio_maybe_mapped_shared(folio)) continue; if (!folio_trylock(folio)) continue; diff --git a/mm/memory.c b/mm/memory.c index 8dc241961b684..2a1e7d9722866 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5672,7 +5672,7 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, * Flag if the folio is shared between multiple address spaces. This * is later used when determining whether to group tasks together */ - if (folio_likely_mapped_shared(folio) && (vma->vm_flags & VM_SHARED)) + if (folio_maybe_mapped_shared(folio) && (vma->vm_flags & VM_SHARED)) *flags |= TNF_SHARED; /* * For memory tiering mode, cpupid of slow memory page is used diff --git a/mm/mempolicy.c b/mm/mempolicy.c index bbaadbeeb2919..530e71fe91476 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -642,11 +642,11 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask, * Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio. * Choosing not to migrate a shared folio is not counted as a failure. * - * See folio_likely_mapped_shared() on possible imprecision when we + * See folio_maybe_mapped_shared() on possible imprecision when we * cannot easily detect if a folio is shared. */ if ((flags & MPOL_MF_MOVE_ALL) || - (!folio_likely_mapped_shared(folio) && !hugetlb_pmd_shared(pte))) + (!folio_maybe_mapped_shared(folio) && !hugetlb_pmd_shared(pte))) if (!folio_isolate_hugetlb(folio, qp->pagelist)) qp->nr_failed++; unlock: @@ -1033,10 +1033,10 @@ static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist, * Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio. * Choosing not to migrate a shared folio is not counted as a failure. * - * See folio_likely_mapped_shared() on possible imprecision when we + * See folio_maybe_mapped_shared() on possible imprecision when we * cannot easily detect if a folio is shared. */ - if ((flags & MPOL_MF_MOVE_ALL) || !folio_likely_mapped_shared(folio)) { + if ((flags & MPOL_MF_MOVE_ALL) || !folio_maybe_mapped_shared(folio)) { if (folio_isolate_lru(folio)) { list_add_tail(&folio->lru, foliolist); node_stat_mod_folio(folio, diff --git a/mm/migrate.c b/mm/migrate.c index 365c6daa8d1b1..fb4afd31baf0c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2228,7 +2228,7 @@ static int __add_folio_for_migration(struct folio *folio, int node, if (folio_nid(folio) == node) return 0; - if (folio_likely_mapped_shared(folio) && !migrate_all) + if (folio_maybe_mapped_shared(folio) && !migrate_all) return -EACCES; if (folio_test_hugetlb(folio)) { @@ -2653,11 +2653,10 @@ int migrate_misplaced_folio_prepare(struct folio *folio, * processes with execute permissions as they are probably * shared libraries. * - * See folio_likely_mapped_shared() on possible imprecision + * See folio_maybe_mapped_shared() on possible imprecision * when we cannot easily detect if a folio is shared. */ - if ((vma->vm_flags & VM_EXEC) && - folio_likely_mapped_shared(folio)) + if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio)) return -EACCES; /* diff --git a/mm/mprotect.c b/mm/mprotect.c index 1444878f7aeb2..62c1f79457412 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -133,7 +133,7 @@ static long change_pte_range(struct mmu_gather *tlb, /* Also skip shared copy-on-write pages */ if (is_cow_mapping(vma->vm_flags) && (folio_maybe_dma_pinned(folio) || - folio_likely_mapped_shared(folio))) + folio_maybe_mapped_shared(folio))) continue; /* diff --git a/mm/rmap.c b/mm/rmap.c index c9922928616ee..8de415157bc8d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -889,7 +889,7 @@ static bool folio_referenced_one(struct folio *folio, if ((!atomic_read(&vma->vm_mm->mm_users) || check_stable_address_space(vma->vm_mm)) && folio_test_anon(folio) && folio_test_swapbacked(folio) && - !folio_likely_mapped_shared(folio)) { + !folio_maybe_mapped_shared(folio)) { pra->referenced = -1; page_vma_mapped_walk_done(&pvmw); return false; From patchwork Mon Feb 24 16:55:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988508 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB15FC021BC for ; Mon, 24 Feb 2025 16:56:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1EB6D280005; Mon, 24 Feb 2025 11:56:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 19BB6280002; Mon, 24 Feb 2025 11:56:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E961F280006; Mon, 24 Feb 2025 11:56:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BBF88280002 for ; Mon, 24 Feb 2025 11:56:43 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 811C016199B for ; Mon, 24 Feb 2025 16:56:43 +0000 (UTC) X-FDA: 83155442286.21.A6D6CD4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 01CDA1C0011 for ; Mon, 24 Feb 2025 16:56:40 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=K23DX1+w; spf=pass (imf20.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416201; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hjQ9M+/7gFJUkrRnlhRJAIZPqjntNNCFGRlUUDQiAps=; b=67fm98yHbwIOuIzaVVf0hiILErcmdIgmRYjxpMGcJofL9hQleGvUphUEUEiTMZSKKMnfxt N+a8RLM9tI06msFP4I7RZBzZnFtU7itj3xb97OFmHP6PMtKTtDVyKMFb5cKvqXtMBOXbFF d7iyWDSCgszkHQ36/50oACi/LHqTpE4= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=K23DX1+w; spf=pass (imf20.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416201; a=rsa-sha256; cv=none; b=NSHx9NDXGIhXsIXhPaDtm0lGzqQ0YzRmC9E9k65GuZOHgFMWS25759MKyZE+IqwxIaNuRu su41k/oiSppoiUW/iC7J77FslAtuFnMjriVgrcf6xYasaTrMv6UiOPbsOLlUEl6bzQEQ+E u9vkHGRjP+U19g7iFHxL53zb+FY3vGM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416200; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hjQ9M+/7gFJUkrRnlhRJAIZPqjntNNCFGRlUUDQiAps=; b=K23DX1+wKWeoHVKP/Vt9RrRBmmHILu9upGVmJWzTLOYRbKWIqO0aaUWQMX4khqVjrq499T Rja011ctVvelU5sV5/G8AgOfg/KDYbMJvVDvHkMEgI04LdV5/kCxEBY7db+DyF3Tp2fHAg 7hy5Cp2W8/Zn2R4xWvuS3al29AFdEHE= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-616-KLDAaWfsO2ih0-4hV9Ymww-1; Mon, 24 Feb 2025 11:56:38 -0500 X-MC-Unique: KLDAaWfsO2ih0-4hV9Ymww-1 X-Mimecast-MFC-AGG-ID: KLDAaWfsO2ih0-4hV9Ymww_1740416197 Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-38f255d44acso2001194f8f.0 for ; Mon, 24 Feb 2025 08:56:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416197; x=1741020997; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hjQ9M+/7gFJUkrRnlhRJAIZPqjntNNCFGRlUUDQiAps=; b=Y9nuZoFxmhyadu7pGUvKvYLUt038m2nQTVXot7kbew55ae0dHt0PKnrYAy9GTBCFeQ Zxmihk3eMVT+kyJhoSPs216qHs/ajmCLYJJkPFWrDMHKW2eG3Hgffxf5O9ukvbPer1YF 8upEeuP9emUUYAkXae4k21mSPDg0YkJedPIkt7PvhF3qIJMlNwudamZBZcL5K3DIpCuN JT0uH06O6/7cdPHzOfIXFKynDtKr7AWoRGTgaxjooqFsT/KQ6imu6SflkX15ahbI1Hwt niaqOotTcmdjAamYX+MMu56a8HqofqDcjVTbtdPZtNWEQaR0Hm/y5E57HY9Kqf/+0c9V FqDg== X-Forwarded-Encrypted: i=1; AJvYcCWM/QVMQKUGWtU3hNB/Fyx1//oyEOmgCLA4AQng5RSBZ94S2xOvU6e1runqefOQTAhV1XO0CWD1fg==@kvack.org X-Gm-Message-State: AOJu0YwbI/QqwnE7ENxwucSvzoa9/XXXVR3mvt5MK6f/UpcTuC9G2ZL7 JzWjjHOvIUGd4oC/Zp5SORM+e8EsF2mT7jeLgppi95BAOhqun4E+0j+vCXZyDNTQSe60IaVXRny QG7UAPD9TRl+0Lft1jSBicjM+FXTSpGX62AdcwYWqg5jSaBLo X-Gm-Gg: ASbGncuWlTDueIIJbXUpEJQGqdq0VRxmlh2JkmxDuzq3iwPaTVwEY8FBYmc7C/lHUJG /ORz9Tpk+DjcZrsWWPrXPEY09cIt0+5GZDn1bTdSThZy0CpHcKXE+ayhytAEUvgW7RZ3NP5Gpdg J4KMvQuz7pDfee8D2q+nskaf/OcCY049LiSAsuYWejUsOkPwqFgA+bb6KMia1K1DKCpiadiTjIQ n6tPpgghvrYqvQ8cfo6bFnOatIoM+4Y/Ld7e+LLlSObn40XhAQsxxuueGIgCsm3+kum8DzMHxz0 miGDB4KmuAwWrP7gLvOXohlU/xv0qkaJ9Q3xz/KjGQ== X-Received: by 2002:a05:600c:154b:b0:439:9f19:72ab with SMTP id 5b1f17b1804b1-43aa6d14bf5mr34158735e9.23.1740416197419; Mon, 24 Feb 2025 08:56:37 -0800 (PST) X-Google-Smtp-Source: AGHT+IHJ9sKTQ3EBTX4dQr6LdeGl2B92vuZUpZDHC83nbKU72kdYMMOPYVrf72peoEztlZbl1dnGRA== X-Received: by 2002:a05:600c:154b:b0:439:9f19:72ab with SMTP id 5b1f17b1804b1-43aa6d14bf5mr34158365e9.23.1740416197028; Mon, 24 Feb 2025 08:56:37 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-439b02d519dsm109604025e9.11.2025.02.24.08.56.35 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:35 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 15/20] mm: CONFIG_NO_PAGE_MAPCOUNT to prepare for not maintain per-page mapcounts in large folios Date: Mon, 24 Feb 2025 17:55:57 +0100 Message-ID: <20250224165603.1434404-16-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: CkztnpdganP7C7nT07uLNWb6LZhDgSTzuB2U-pIZKDk_1740416197 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Queue-Id: 01CDA1C0011 X-Stat-Signature: d91uftu4qonw51sierrjm9t41wmbz7rd X-Rspamd-Server: rspam03 X-HE-Tag: 1740416200-466626 X-HE-Meta: U2FsdGVkX1895Gha4LKgzyxoCx+VamQhlY81V4y1aWqWiGWu44JitucaGbR9ZiPf+f9XmsPd9W/ekr1r169ibTuAJWh72r3nPSmMDubYvfEWG9MI7R1OYK3r+JqJrJ+/dvrUKAJhnttu0aHBo/VhUM/i9d3NiGMwMYzPj/Zv+AMDUBbMIU7mz85r7akUoA1tzlK204di5T6R8N0COkBnI1u7vCQ8n3QCE20PMh08osF1ZoDA4fmybclhgP1ABqYrfBXe4MXgUto7NIFcSHXXBXZN0GXddf4efiz//mgGvqWrPsBdqvz94fMGDJWECZUln7MIf/oKDukvIjHEFXHqIhKD5TWQVOnw/vAyokrV0Tcn5nsXqH3ztk3HGAApK5pA7TLD9MK5jTz1DJk2kiFVsqddG/ZWep2nhdzc/jVCLm8o5+q48YN3hmK3clBU+Nhrw+cnsJhrU4W44yAeJy/rg/whqJxK1nlL23NDqs50692/1qwAlkDRDjuoAI7uEEqQifI9YQJ7pdzh4DEOxIcNF8ZyKx8nYCYoko5BdccoTJN1favfMMV35avPC1AoxlTpcsHX2VK382kz3iAoMMU8xaVCQD7tOySGhVy+R+X6W9E9sq9x/00o2p9kL4yM2/pRYmRX0dR7YErvx0/G+OVQQCe+YzLzAjyNdms9+GAmhFhjOXyQ1H8OznBavfycNnRRPSMS3Ob+Wm7jUAGYvdyuo/lExgyXspxAcE/X8ynuh4bdg1QjkBs6aJ2N4wmiBMRqMimixEfgDVifoPjfXomnmKhxI4FtGxWlcBfiNmlGjZB7i9sUa7/kWJBt7LtfBCmpHve6zaOzwavc3BWCPr6VOdVhM2gHyrb/a2reXHArLT8/oGRAZgHqXWQKHus2nJkZM0h/5wR2yhM4Uw3B2LWOy/E/6fybZp7eq0nc5enGjIi+4+4Vh3W+inB6QtBNY+js3IiWrNVXiqv3GkT02y+ AdAVhQIm dbLvjn8hmxt3+ByJeFIw4GHwBUfn+B7BJ5dtvw1z9qxEIH+3yMED1dFeRklbndbF8xLhZvZnAG6DUalB/QZy8ecI8IEiy9hL+6/249chIgRRawVeTtfy9QUTOvaStBtn1p26Tzk2ZQtF0dkBFrNZrD88o9mj8Y79lWa7fVdsyuuFZfIw4KKOKsnKPtfB8yHFrTaa5c5Hb8RkX6KwTYsegExZNLHiq5TtHis1YtTzaFuZGavJZdhsh/LljykzerwxpIYA2ZBOpQtmXXURSvNTGdLXpxLukJ3hJ0nXNJJawkv8MPtgaw62aprIgkNK71HKnu8l/GAr8UQbNx9hdfXVNg51Q8EIsHdMMAQY7eGgQUWb00hiaETsDB6PTGqsjnO65wsG8EqNE4tGhhKRzec0GZHEDQn2kASpAghhx6a2RYB611PkUKwpv1j7y9+OykMtOqxj2PcDc9TPn84ZB+90cDt2Ux8BVEf8LDICeXMT98AYfnQykA5UVrsXRrqDeYzcOSYTQEg+Dvs/+5VPrj3O5QjWGfhX73o2cHBFO+tim+kcnz0o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We're close to the finishing line: let's introduce a new CONFIG_NO_PAGE_MAPCOUNT config option where we will incrementally remove any dependencies on per-page mapcounts in large folios. Once that's done, we'll stop maintaining the per-page mapcounts with this config option enabled. CONFIG_NO_PAGE_MAPCOUNT will be EXPERIMENTAL for now, as we'll have to learn about some of the real world impact of some of the implications. As writing "!CONFIG_NO_PAGE_MAPCOUNT" is really nasty, let's introduce a helper config option "CONFIG_PAGE_MAPCOUNT" that expresses the negation. Signed-off-by: David Hildenbrand --- mm/Kconfig | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index af72fe92f96b0..64bb80a281dd9 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -889,8 +889,26 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles. +config NO_PAGE_MAPCOUNT + bool "No per-page mapcount (EXPERIMENTAL)" + help + Do not maintain per-page mapcounts for pages part of larger + allocations, such as transparent huge pages. + + When this config option is enabled, some interfaces that relied on + this information will rely on less-precise per-allocation information + instead: for example, using the average per-page mapcount in such + a large allocation instead of the per-page mapcount. + + EXPERIMENTAL because the severity of some of the implications first + have to be understood properly. + endif # TRANSPARENT_HUGEPAGE +# simple helper to make the code a bit easier to read +config PAGE_MAPCOUNT + def_bool !NO_PAGE_MAPCOUNT + # # The architecture supports pgtable leaves that is larger than PAGE_SIZE # From patchwork Mon Feb 24 16:55:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988509 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DDCDC021BB for ; Mon, 24 Feb 2025 16:56:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 326D3280006; Mon, 24 Feb 2025 11:56:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AB6F280002; Mon, 24 Feb 2025 11:56:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 061AE280006; Mon, 24 Feb 2025 11:56:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D3E2F280002 for ; Mon, 24 Feb 2025 11:56:45 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9435F1C988F for ; Mon, 24 Feb 2025 16:56:45 +0000 (UTC) X-FDA: 83155442370.28.B320FDF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 4A23718000C for ; Mon, 24 Feb 2025 16:56:43 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SpfcU5I8; spf=pass (imf24.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416203; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I7/ovFwFtUshRpZfEG8TPamCJVjib0AvdCdidGnTX/0=; b=YXA5NNuh3ykKIGJRYvj9pxP/GElub+PIlH6Hrqpu/WvkrIbEGAsR7PcOBuPv9e5Tkc8gwt npEIZO8ta2WLN88DSqHXscPEbQbWyC9/nFPxvQjLYTApUyjWzHG6qzw+iLm/ZtdOwPfHUO EBu4G0FkHpXnhnlx4kxD8rkpDt4hC4M= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SpfcU5I8; spf=pass (imf24.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416203; a=rsa-sha256; cv=none; b=q8q7878dYvNv3g/H9GLul+xV+ILwm2IKdDSbTfh33zAC/yVn+E1LvMm/32wJ+pC5sUtlGj vk1mHDn1lfyWkd9hR3qgb5QQPlGCoph10hw+kM6A5xjglUFt8a4q4mOEj+qMBW1DFVqI0Y 64hrVebryQnwfpEJO6VWcCfOFoCQWHA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416202; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I7/ovFwFtUshRpZfEG8TPamCJVjib0AvdCdidGnTX/0=; b=SpfcU5I88Xk+/K0JQ5hYodIX/U4qcaUpe2MpqlKEq4QGZ9UEDn8+WWUsG1xg0mzrUlHFd+ 8Z5IPRjJbZ2GsWvW2tQZtMU7E0e9Z5tIQos5kaJSv34gLE6Y9OrmmB0+nbffEV5UDThbg7 DwJvt+Uh0apTHtFuD+Ek9wjuo9Wtsfs= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-280-RxofC5BNOW6OeR1Jmv-_AQ-1; Mon, 24 Feb 2025 11:56:41 -0500 X-MC-Unique: RxofC5BNOW6OeR1Jmv-_AQ-1 X-Mimecast-MFC-AGG-ID: RxofC5BNOW6OeR1Jmv-_AQ_1740416200 Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-38f34b3f9f1so3474966f8f.0 for ; Mon, 24 Feb 2025 08:56:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416200; x=1741021000; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=I7/ovFwFtUshRpZfEG8TPamCJVjib0AvdCdidGnTX/0=; b=Z8pVFq6D/kFNv5NLJUmMG51f8NikDkx2E435NPDfShs6IWoetux70QCB4KnaUms6Q9 EPMi1GY5XOCr+fxNvB/h+yvoPxbVoVB87IYYNg9jH7HVLK5SlUOwE2IshQsQNAjCCt35 EMd3D0CBbVt9CoTQ3B7g8CXMkIVUijwbxsqOAlM2+UnKkTGjmb3tJ+53spKyLVUEskCr nlI8tGzNB8a4KIZhiJPLzHKCvvXfwLU+VLBX9TWcDohAR7SEwWWk1w0E3EWL2sdLBzYX BeRa+5ccYn9SuNRSlG3ddZ1ca3rzr2xQ90VD7GlorySjuBYCKjIEFo7ln0Q9CDvrjiBN aAug== X-Forwarded-Encrypted: i=1; AJvYcCUkEciGycSMCYVuQ7lNRYH7Fl8HedBhm2pIQdMubuGm5Wu0HTGW71IbcMrlnRuy9gJsHXceLf1cfQ==@kvack.org X-Gm-Message-State: AOJu0Yw3B4mJBdZ4YtBGJ8Dcoi0pHHi0dAXNX+SCbF3TwZJUPS14EHTT 7xKnd/MyX0Sa3ZDgY2iy/jGbBKEcPij/X0taqgio+TAIji7LyeKJQwWbk4s0jdguBekcRgNTipy NsIgXNGAX6PBWY5PmKyv4AU6wuOUoTFIei1NE++il/S/T64IU X-Gm-Gg: ASbGnctnAYeHvBzWcV9NJYAnAp/XfufTeWYFQ4BXL5Y5RkSiveQoaCMJZGLdUDnjKhu w6zJvFEfv4Bd/IXR2J7XylTZUMY6Fl8qY0nUnxkk0GxIMtcAxrp+BN2/cbOVrlgy6BbdWhASCM+ CF1/myuwcwlyrDtvpIvD3Pnck1dxTDuKEDxCal7oFy+ajSCOi7Zx0+mptKb1Dun4JGAiLpuETkr 7l+P7houcqr+E8oQAIy29/C9C1vwCwrhb5GLRuffCcr0J2mTagrEl8zdIxfb7IQc4LR+805ZZkO 9fMuK7HTW3nNBpqVM0G31RJEwvfaleYF5P45rJfjIA== X-Received: by 2002:a5d:588d:0:b0:38f:330a:acbc with SMTP id ffacd0b85a97d-38f6f0c743bmr12449964f8f.54.1740416200265; Mon, 24 Feb 2025 08:56:40 -0800 (PST) X-Google-Smtp-Source: AGHT+IFfMgOxaCcP3QUPAwnlgJXjUNV21VqnGE//pcrHn7Js38XBMUq8lKiO2xorUlCAe/uWbu52qQ== X-Received: by 2002:a5d:588d:0:b0:38f:330a:acbc with SMTP id ffacd0b85a97d-38f6f0c743bmr12449918f8f.54.1740416199872; Mon, 24 Feb 2025 08:56:39 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38f259f8121sm31621452f8f.88.2025.02.24.08.56.37 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:39 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 16/20] fs/proc/page: remove per-page mapcount dependency for /proc/kpagecount (CONFIG_NO_PAGE_MAPCOUNT) Date: Mon, 24 Feb 2025 17:55:58 +0100 Message-ID: <20250224165603.1434404-17-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: XkoYpkgQsPFFRN0myMT-6JKlzVvfCxVSlWTvf-t3gKQ_1740416200 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 4A23718000C X-Stat-Signature: howbfca7agc89yzqhz7p6kc86r9j34ma X-HE-Tag: 1740416203-161957 X-HE-Meta: U2FsdGVkX19O4toe8bS61MnF/crElU3NtqaGI4+tUeJDE80qh01SzaJSvUS143OEWub/ut3I6j/WC694RJf1UJqHBk7IMFZtGrTvaMnonUCPu1l6jtdJHSHmMAj5H8MZqv/8MOmhfwZwUTmtkMNmso/mNc3Arsf9do2104sKykIdOWFLuxtHb5PLh47Bo3LlxtWbVmxSuwX7wejfYC7QRHT49jwwsCrMj//dzRdzPD3saGyHkcD4p43wuAWj2bDv/GENKsXZRQzgYohTTs1qtsWLA0M7+03qiyCRPcCFqYQ03NMu8ktllf74uRvoyy73C8ufGumj6RJiTETJqlg7kd/63tSqIPFkNumezVB26FXZh8o/9r3KjAk5i/2lUYlMsOUnBNazOOFrlpOQSvN2oniPbcRAbQPoT+bhQcexdFMBng2R9YoCJM9HX8S6U273BnG6Fb7AMnQCA3Gj7WfKG5K70yLetMPmY9Gk4bN6RhUBRvn+/V7rolbSiO8Ve+q4rYiESwDvmWQqecdEutnumh9BfPAFnytgMKWvX6022H28CKgG8mOScptdCUYAG5E/ubkrYSHpXYxzLfU1RKbcENpXcAncQRpuOhl0/19OCRZa/AqG/WvPbtpaCmaUSAlhF3/lryeVfSoHAKvJptO6xatRgxjRQZL3z3OCmIE/g+lJFFglY0snMNZb/bx0+AGJlqJsBzhXzsmcNVo9pJrQYS2JM16OXhLwA7Ffi2PlmXi8e63vdwyBVQIxY9lspMvBZRGbR9Af5Oet6ChCq3X8mkVEC99JGcWIskBnDM0MTEz4iyzeIOiwMzEpo96jKdpbNpruqalJj3ZP0vndiPxN4gkGIN4l8vWh4r6sr3jS9NNzPCxX0rsgidTbsDV+8J7QfaEgZHu2TG4+Tn1RxBikb6PQy88L74ftbz02aL3tufWnsJRhfCIubWwn9Uv7+r316GxI/fep2t6lcFtBMO9 qebxatuQ XIODpUTA/3lmjDczNt0I0n8uzRwts2C0CQ7IBZjpsKakrWLznE13um6L8dUkuElbVAbyPNct6mMApsr9saeUmOAQ2UJ1T7pcyRRG1VG7povI3lOuXaSBsrZeacpGWw+7/qD67WxMUHf2e0acF7y+Y7zfM61U/GFRniPQj3MGeyp2RV4mqCheAc+p5XyN06P/poO33CWUG9ifvcWG+4W0TMhM7OQfQdm+YkjBQ0GCcyg3raMAyNrLQr1Q39d5/mJbzJSL4s3nsmaVN5xFIwRfeZ0JvCDK6VjZHgPeGYcZuMh9waZLNJeiKWroreEYW/4eNrR8jDVd1GVmBdOOn209ICSnL6k1K3xBuP3tjh8NRUorvbxiTXGOe/AdAh7OwuXeVEFu/waURQfJsZZXpUOeoYoW1xRGf0SjiAi6ZWLvkHkTT9Ixa6r5cBysjyGqmdqpwMUIJ27bLDYPz2PwrajNJ3LkGOSjJwuBvUCXWNfBgAn4ncl+JWwJCKt4rkKgfI48ubw9hfIIH1L0W4XX+X08j+bxMFipyx5s4yIDdXemEwirXdrWpRk76xgJPq1hYmO6owcKAr8YBhYNcNAM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT. For large folios, we'll return the per-page average mapcount within the folio, except when the average is 0 but the folio is mapped: then we return 1. For hugetlb folios and for large folios that are fully mapped into all address spaces, there is no change. As an alternative, we could simply return 0 for non-hugetlb large folios, or disable this legacy interface with CONFIG_NO_PAGE_MAPCOUNT. But the information exposed by this interface can still be valuable, and frequently we deal with fully-mapped large folios where the average corresponds to the actual page mapcount. So we'll leave it like this for now and document the new behavior. Note: this interface is likely not very relevant for performance. If ever required, we could try doing a rather expensive rmap walk to collect precisely how often this folio page is mapped. Signed-off-by: David Hildenbrand --- Documentation/admin-guide/mm/pagemap.rst | 7 +++++- fs/proc/internal.h | 31 ++++++++++++++++++++++++ fs/proc/page.c | 19 ++++++++++++--- 3 files changed, 53 insertions(+), 4 deletions(-) diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst index caba0f52dd36c..49590306c61a0 100644 --- a/Documentation/admin-guide/mm/pagemap.rst +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -42,7 +42,12 @@ There are four components to pagemap: skip over unmapped regions. * ``/proc/kpagecount``. This file contains a 64-bit count of the number of - times each page is mapped, indexed by PFN. + times each page is mapped, indexed by PFN. Some kernel configurations do + not track the precise number of times a page part of a larger allocation + (e.g., THP) is mapped. In these configurations, the average number of + mappings per page in this larger allocation is returned instead. However, + if any page of the large allocation is mapped, the returned value will + be at least 1. The page-types tool in the tools/mm directory can be used to query the number of times a page is mapped. diff --git a/fs/proc/internal.h b/fs/proc/internal.h index 1695509370b88..16aa1fd260771 100644 --- a/fs/proc/internal.h +++ b/fs/proc/internal.h @@ -174,6 +174,37 @@ static inline int folio_precise_page_mapcount(struct folio *folio, return mapcount; } +/** + * folio_average_page_mapcount() - Average number of mappings per page in this + * folio + * @folio: The folio. + * + * The average number of present user page table entries that reference each + * page in this folio as tracked via the RMAP: either referenced directly + * (PTE) or as part of a larger area that covers this page (e.g., PMD). + * + * Returns: The average number of mappings per page in this folio. 0 for + * folios that are not mapped to user space or are not tracked via the RMAP + * (e.g., shared zeropage). + */ +static inline int folio_average_page_mapcount(struct folio *folio) +{ + int mapcount, entire_mapcount; + unsigned int adjust; + + if (!folio_test_large(folio)) + return atomic_read(&folio->_mapcount) + 1; + + mapcount = folio_large_mapcount(folio); + entire_mapcount = folio_entire_mapcount(folio); + if (mapcount <= entire_mapcount) + return entire_mapcount; + mapcount -= entire_mapcount; + + adjust = folio_large_nr_pages(folio) / 2; + return ((mapcount + adjust) >> folio_large_order(folio)) + + entire_mapcount; +} /* * array.c */ diff --git a/fs/proc/page.c b/fs/proc/page.c index a55f5acefa974..4d3290cc69667 100644 --- a/fs/proc/page.c +++ b/fs/proc/page.c @@ -67,9 +67,22 @@ static ssize_t kpagecount_read(struct file *file, char __user *buf, * memmaps that were actually initialized. */ page = pfn_to_online_page(pfn); - if (page) - mapcount = folio_precise_page_mapcount(page_folio(page), - page); + if (page) { + struct folio *folio = page_folio(page); + + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) { + mapcount = folio_precise_page_mapcount(folio, page); + } else { + /* + * Indicate the per-page average, but at least "1" for + * mapped folios. + */ + mapcount = folio_average_page_mapcount(folio); + if (!mapcount && folio_test_large(folio) && + folio_mapped(folio)) + mapcount = 1; + } + } if (put_user(mapcount, out)) { ret = -EFAULT; From patchwork Mon Feb 24 16:55:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988510 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63C73C021BC for ; Mon, 24 Feb 2025 16:56:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F6F5280007; Mon, 24 Feb 2025 11:56:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 67B17280002; Mon, 24 Feb 2025 11:56:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4CECF280007; Mon, 24 Feb 2025 11:56:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2E652280002 for ; Mon, 24 Feb 2025 11:56:48 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E6550B6E48 for ; Mon, 24 Feb 2025 16:56:47 +0000 (UTC) X-FDA: 83155442454.22.224FECC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 4F6D720013 for ; Mon, 24 Feb 2025 16:56:45 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=g5xHBKQv; spf=pass (imf13.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416205; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qrzeSAQepC6qjgZK5hS618fC1aWLshwTFA4Bvic/TRg=; b=O/CUvelHdqbN/+42VQobPCT5lBxQzvX0ITHfuNEkqVfqFLAqoKp6z6oTkJMMgeLiS7n0P8 SDoF8ETlgC3WByal9+QRFOC7K/SVASkHDudi1aAH+GWqQL8SSDvgLrNU0+nEfgFEHdzvgZ Zge+ICKDXA6QfMnLuRkIoXIFvFB8v1A= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=g5xHBKQv; spf=pass (imf13.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416205; a=rsa-sha256; cv=none; b=S5W1Q4SBP7BIQA2xjHS7Kq5vNSXegHZBrj1mwO1w8gWJXns1IEHj0eHMcpD+dSOB6iDOO7 Fp4JO9/dOv7MPRlaRSYxi7YrLgd6zdLoUktSpZJdCJqjKAwIDeD6w9l1SRDKZiHu39tRTB RT9sxpAej7BoJyw5DyfQWs8v/xv+GZA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416204; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qrzeSAQepC6qjgZK5hS618fC1aWLshwTFA4Bvic/TRg=; b=g5xHBKQvdkwgISSU+dIfZMZYJR8gFd/AXuF04vcX3LaaKeWq5eE10Ubf2krd5JnyVFL8PQ AQKgDBJ9sDN81vwrAbgT7zR6uJJxBx5Rtp4Dod+oxrl30VCSZDcoMCUldb9oV62MZx9wcB XFE4EbZR1jywW3uPFksa9zLF+KqDB0c= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-461-Bn2W3iKIPsKUDiq-QaLFNw-1; Mon, 24 Feb 2025 11:56:43 -0500 X-MC-Unique: Bn2W3iKIPsKUDiq-QaLFNw-1 X-Mimecast-MFC-AGG-ID: Bn2W3iKIPsKUDiq-QaLFNw_1740416202 Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-4393ed818ccso34921195e9.3 for ; Mon, 24 Feb 2025 08:56:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416202; x=1741021002; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qrzeSAQepC6qjgZK5hS618fC1aWLshwTFA4Bvic/TRg=; b=e97BNKzbeH7uPOrScv3luBhxIruIYyOHdq4SS6G8cEpEmzPcl+Y6veFmjmewFPQliV jGGeYR86+cJHjb+qvAub4mndz5nfpyg3ifOxeL3g041RKsz30+sM+zYxlDSinT64gLZx 6chUxWqSaZkpysocIPm9JTnAIkYIEqq9xjB4P9flawehdJ213sBc+6Dfio+iYf0fwHMc 7bfjhGGqD47x+2tVwdrLdy9xjaU4T4DOfKX9/hEntnZS/bPgms6X7Tae5GkZ2HUp1/GV Mkk8pxxnloEqnBpfTjSmJQLfAlhW8CrImw/kQZB5b9+dO8xKDlXsoP3t600Qi+ZxLg4r hJ8Q== X-Forwarded-Encrypted: i=1; AJvYcCWYSLVfEfnH+rTBqIG2w9r8X58azCDpDCo0eHfLFzTrPBYrFKoYc+3QIVuPKVyXqifGn4OrR4CTUA==@kvack.org X-Gm-Message-State: AOJu0YxTs1QRQVOmQZplIe8V2iRScw6V7qwmDvXJ27LcuYn6ak/K5cNi 0RSihO96ojw9CFTTke7JV1kkxwZkpU/ll+JYv3B6TMvbRf5+6pHb/y0sWQBWxl3DEhos3kIlZ3q gvhsqCJEznrYsZ8RBp0qcZX0jLi4EPYxHoI4/8uCmROgig/Ks X-Gm-Gg: ASbGncuttfvsA8DepptSmHHfs0vpOOhmi18BlbQP8V0ftVQTMBXNeUCE3qQwkMpS1pq Apd6ze6XuSedQPRY30pezzKAm9vFqGIU66UVM1ul9vwP1u9jO3oO4G7l4O711v9ArO0rB+WPu2R ZylJqel5CY12DrMK4fMc25bC2sRC1GJH5I253vr3PZm7xJiwMiEfDcsrhVpqTTcO76KW0H5fPGR 0e5RDbrR0ZBm4rKQiqJo/1pvhl2lrUfoSde4/+EBnd299b99SONqNA96hPjdosfmt6muyQycwFg gLPWSghtwuE8iE8XIQiGEdwm5lAxOn9p4oZo12AUVg== X-Received: by 2002:a5d:588d:0:b0:38f:330a:acbc with SMTP id ffacd0b85a97d-38f6f0c743bmr12450062f8f.54.1740416202166; Mon, 24 Feb 2025 08:56:42 -0800 (PST) X-Google-Smtp-Source: AGHT+IHB6f6br0E5FryNlZozgeR2JsRbAfPcg+xWwWdf/YpxK+ukwffc3aXIkqUDyRFtyhNU8GNoFQ== X-Received: by 2002:a5d:588d:0:b0:38f:330a:acbc with SMTP id ffacd0b85a97d-38f6f0c743bmr12450037f8f.54.1740416201773; Mon, 24 Feb 2025 08:56:41 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38f259f8602sm32962279f8f.94.2025.02.24.08.56.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:41 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 17/20] fs/proc/task_mmu: remove per-page mapcount dependency for PM_MMAP_EXCLUSIVE (CONFIG_NO_PAGE_MAPCOUNT) Date: Mon, 24 Feb 2025 17:55:59 +0100 Message-ID: <20250224165603.1434404-18-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 6Hk5i2fQK52iHkOTzbLOEyKnLZaxoxtRIQXnwgKfj9Q_1740416202 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Stat-Signature: fznm5q3nc914hmstoto4b557kx6k566j X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 4F6D720013 X-HE-Tag: 1740416205-102627 X-HE-Meta: U2FsdGVkX1+t/pXcUydb5TjqrMNtSSkybKbHFFN2nLuApvhIHYrFY5VUKbrij9F0BQDpfdII1WNasihEeOe43cOmUyrTsYHSRY9Q8MLNcxAcwD9FUEK3xJ/fMYfQravsk0i+aLfjhageqgpZ1AC29iD9rYZlM7Sxd6TYXHHbwhjUOu2meVhJvPhwf1SKXP2Ky2L7P+4waAsUTX7lR/1Tn8FTnYpj4e9rcpHt0b8n4uTM/aUs9NjJ+pOeLZECFuGcCHFJ9WF1qEHWlvZsk9fXCU+yD2vfmDmG9TP+6znH9FNDtRnX+bKuqb1y82cy8PQvcX4xaRbajToTFj+gboP+V+skHwhTi0yjWIN/H5Dy8fdgiZxdQPidrwyZFUdVvR3CbGx5KjfX41XQXBPaSpxDRJwZgBEfvJtV8Fo0YKqx0J0gIO1qOGaVJegLYaejocvEbf90tleloozoAW8X1S5UXPxMqgGOt849z5mue4ELprsYeduZ9HiI/uing9SOQWth1yhcn/3TnsKX3DOyvrfbj4wR7Mr/GiHQDqKYgBT9tFuoP6iWZtFzsZ2ptaU59OjxKMEQPRVMi7l6u8Y6efzR9luACOKdwIEb2M/RbTprZekcAUNgHV1N9wmemQT+wx6aQtOaMkpj8QJF/4tCSChsOGDIn0itLa5WP+yiBJccjraD0hKj+o+mA1lNj9L7Mzy7DZNRN/Dd2udrVKKhLuY+lJC5+KTvF9wpVBg4Ojf1lVQCem8u0qwwBOyCTw6eL7nd0nijsv5hh5gp9AlMhT6BQYFyJAQRH8UFX9w447sU9Sifcpr4D38GOBrFB57Og46gcVKqcjGZUfrNyAXG1SyhtdLZhXEXW7tdkB29lJ1Nj1MR9Xl8tfkd0k6QojngpabSLPvS7knmh1CqallasXaYNINZEUJW3b1/Lswq7t4xPih+7rb6PStihm4e9lNRcislALJisH0Juk1mmjnVO6R Oefd6sOx c8IZnOJ5E3KFh3SwIP0Q7ARPAx0jqILXM01aP149Gh/jMVEaH9wx+HWKjAZd4FZp0VWzDZVVg5ZEdKbFQd7wU7vlZiRUEuO0zbR2Vq/BVB6b0i357qu3JkzrLVqQOsFpCoVYJ8WZZkqjRbCXpmmG3iO1+adKp04VYfRL0xK9vjO4uECAEkKvR5na8ZubMG3ggZKWe0L5lrjAb5OI7zkZVfSLl6LpKCDz5/bc+L9A9rEdAgzXx9FqxnYzB90pWoxJW6J967CY76IRwsjJQuusUjvDSNuLQdW3zxLDDsA/y/ffgkZDUBiHYZ4lVQDSkcaVAJPVuQqpV/8O7yWV8D/cwnxMUhaGT5GfLH6OVvLpKcRk1x1ogAP1v4/RZqYChkUdumhyIinBNO7LCDzWFaTbfCzW1y2JOSjNrbbA+TChjq74IJCq0vjijllHzMbGEmeKI/7eGBDX3RchlLyx2sQN7sva/uZe2ZaxH4fX4syp75HxwhuyNqlfIqkrNQW9Tg5OFbNKm8pfBx/52rBiJarbb77NG/7Xl/G9HvcP+pzLcNwtTj7k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT. PM_MMAP_EXCLUSIVE will now be set if folio_likely_mapped_shared() is true -- when the folio is considered "mapped shared", including when it once was "mapped shared" but no longer is, as documented. This might result in and under-indication of "exclusively mapped", which is considered better than over-indicating it: under-estimating the USS (Unique Set Size) is better than over-estimating it. As an alternative, we could simply remove that flag with CONFIG_NO_PAGE_MAPCOUNT completely, but there might be value to it. So, let's keep it like that and document the behavior. Signed-off-by: David Hildenbrand --- Documentation/admin-guide/mm/pagemap.rst | 9 +++++++++ fs/proc/task_mmu.c | 11 +++++++++-- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst index 49590306c61a0..131c86574c39a 100644 --- a/Documentation/admin-guide/mm/pagemap.rst +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -37,6 +37,15 @@ There are four components to pagemap: precisely which pages are mapped (or in swap) and comparing mapped pages between processes. + Note that in some kernel configurations, all pages part of a larger + allocation (e.g., THP) might be considered "mapped shared" if the large + allocation is considered "mapped shared": if not all pages are exclusive to + the same process. Further, some kernel configurations might consider larger + allocations "mapped shared", if they were at one point considered + "mapped shared", even if they would now be considered "exclusively mapped". + Consequently, in these kernel configurations, bit 56 might be set although + the page is actually "exclusively mapped" + Efficient users of this interface will use ``/proc/pid/maps`` to determine which areas of memory are actually mapped and llseek to skip over unmapped regions. diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 2bddcea65cbf1..80839bbf9657f 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1651,6 +1651,13 @@ static int add_to_pagemap(pagemap_entry_t *pme, struct pagemapread *pm) return 0; } +static bool __folio_page_mapped_exclusively(struct folio *folio, struct page *page) +{ + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + return folio_precise_page_mapcount(folio, page) == 1; + return !folio_maybe_mapped_shared(folio); +} + static int pagemap_pte_hole(unsigned long start, unsigned long end, __always_unused int depth, struct mm_walk *walk) { @@ -1739,7 +1746,7 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, if (!folio_test_anon(folio)) flags |= PM_FILE; if ((flags & PM_PRESENT) && - folio_precise_page_mapcount(folio, page) == 1) + __folio_page_mapped_exclusively(folio, page)) flags |= PM_MMAP_EXCLUSIVE; } if (vma->vm_flags & VM_SOFTDIRTY) @@ -1814,7 +1821,7 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, pagemap_entry_t pme; if (folio && (flags & PM_PRESENT) && - folio_precise_page_mapcount(folio, page + idx) == 1) + __folio_page_mapped_exclusively(folio, page)) cur_flags |= PM_MMAP_EXCLUSIVE; pme = make_pme(frame, cur_flags); From patchwork Mon Feb 24 16:56:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09B86C021A4 for ; Mon, 24 Feb 2025 16:57:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 60BBB280008; Mon, 24 Feb 2025 11:56:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5B7F4280002; Mon, 24 Feb 2025 11:56:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E4A1280008; Mon, 24 Feb 2025 11:56:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1EE2A280002 for ; Mon, 24 Feb 2025 11:56:50 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DF793A3D01 for ; Mon, 24 Feb 2025 16:56:49 +0000 (UTC) X-FDA: 83155442538.04.9DB310C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf15.hostedemail.com (Postfix) with ESMTP id A36AEA0004 for ; Mon, 24 Feb 2025 16:56:47 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=R4JRLb1K; spf=pass (imf15.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416207; a=rsa-sha256; cv=none; b=aRDZm78dLjmNalupZjMpQoql3FSLFjxxN9i9ytIg8HaUXOKtXUlrxGZfUloJ8E0810K3mR jTbRuzcZ4wYkLusf8WRuCei93jGqTK4KDXge9XHl9+NQzyC68TjPXH8H/ub8Q61lu+Ahet bgak2w6SiZ6sxux8Y3ylxJA1+U0dqUo= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=R4JRLb1K; spf=pass (imf15.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416207; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4IIiK38txs39RTm2yNSD9uzsg2Epj+W/QUBXdJN6a7s=; b=6yc71zKGDWLnXbQ4k0ia/626T0ZY1NwuZDejZ7148jlZ06B9KW9Re/NtYvvsx027c3/mFn nQlM2fO+FO5yYwC3U0YoiBOvOIUiUIb2F+Gp+4rdGtfaMEh2FhBCaz0pK7rQQnkRtfQ4fq maaHYiVFWXvEV3Dn+2p9og7fLtcrKCc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416206; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4IIiK38txs39RTm2yNSD9uzsg2Epj+W/QUBXdJN6a7s=; b=R4JRLb1K/TgLzhmqV8GP0jB9hPEcYwowAspqZGi1AstAvT2JK3Rr8u4iYsKfVOjmNm5JHa e5ReahdhItHMVw5Wp4bxLC9qrvg8nXntIaGlRs0kRXOXoE7wcOAnyCi9i03EQnB6mapt82 mA177ZhCQVA0V/qZD+ibpsHUjXoJuqU= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-261-5J51aA5YNRCvCaHQbssBTw-1; Mon, 24 Feb 2025 11:56:45 -0500 X-MC-Unique: 5J51aA5YNRCvCaHQbssBTw-1 X-Mimecast-MFC-AGG-ID: 5J51aA5YNRCvCaHQbssBTw_1740416204 Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-38f2f78aee1so1894293f8f.0 for ; Mon, 24 Feb 2025 08:56:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416204; x=1741021004; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4IIiK38txs39RTm2yNSD9uzsg2Epj+W/QUBXdJN6a7s=; b=C0YS5EscBto8iVGgKkT++U6nEZr9elOl0iNq6eyor+fHvCvTTSGgBYuPrFpPAQoT5M AOD3w8ZE/L5RnfhOgb9Q7o9lW1+dpPUCmPlKQzWK6v/gNIDpfhaGsWoie5X8Hig//y24 elZeX8x1XTHuRwpjTW0KPfKTjqHsAvL92sywE1uTjFby6cMUQRBu/a2710dRtdQuV76f ZO2AxLZk/N5MVp/0WCf9rRdgawZybXGL75l0wpDVw+oK70sRV1ApMaRgg9dzZNGrS6F4 EPt8isqdrjqEni6rcff5M0cFLuTujkcC3fh+hXRm578gtp+yzsfI10FBPYpY1rzR2fBQ RQMg== X-Forwarded-Encrypted: i=1; AJvYcCVfLNQwDx2+TB6KHRCrU0yQRHQZQP5ZZCkuS1OSAviUqJR8ZbljtIODzvl5WfcwRRCtxrKwQNiJAA==@kvack.org X-Gm-Message-State: AOJu0YzahuCNNR89xIxbdFpDbh6BTnz0ukxVVltaZRDJS5gVlsO5A3zj 1Dxg+iywHAgVgwUTqY4GYRMUkk+sJRwusaWkm3ym3afSXqA12QG5vrSxzD9wTo7uBHiM7NOKZdR xUeiTtZ2xX4by2zxCxBii1cicMAllXyZcUAhDsgIyy533JQ32 X-Gm-Gg: ASbGncvEZ0cCmDoE6v1DO8R1QGNpsRwySKTtPa8p7wZ82gSnR/XLwPxv9G8qvSj2rtS q2LhdTWcbPSNYPu082m4TuCwhvN5qhbxj2aD7ktHpuwrP8D0PrjcrTDJNuG3KUMBzNspRDXYqA4 IWMk0vFbfYf2mrItm1yoAw4Ir2S+RPA5HV6J2GA5tG+nFHJ8g5AIKS85gOgQFAxqQUO2b/JQd99 eAUM4r6M+klaAtvkF4KuyNwirbqdOJ8Ljij89L6gtI2xVccXFt/C6ZUEsOVp/JR2ClgA0ke7RZu JedybBxCWKnS/Xfht18sdITWXktlJ9hIh+qLOHY36g== X-Received: by 2002:a5d:5f90:0:b0:38d:c6b8:9fe1 with SMTP id ffacd0b85a97d-38f6f51d885mr12699069f8f.24.1740416204114; Mon, 24 Feb 2025 08:56:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IHr48iDkbeT3BD6T31jawE7AoIz4aDQ8fc4EtI1CIkHNhBzvnREnXgHgZXA/JhJC2+PnfKZ2A== X-Received: by 2002:a5d:5f90:0:b0:38d:c6b8:9fe1 with SMTP id ffacd0b85a97d-38f6f51d885mr12699040f8f.24.1740416203760; Mon, 24 Feb 2025 08:56:43 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38f25914171sm32572475f8f.53.2025.02.24.08.56.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:43 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 18/20] fs/proc/task_mmu: remove per-page mapcount dependency for "mapmax" (CONFIG_NO_PAGE_MAPCOUNT) Date: Mon, 24 Feb 2025 17:56:00 +0100 Message-ID: <20250224165603.1434404-19-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: kvYajrwBBevezQ77J8pvGrfAj9dkrO10-CuPEi0425I_1740416204 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Stat-Signature: jgh53agymz4q4dmah849bzdkdgeorggi X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A36AEA0004 X-Rspam-User: X-HE-Tag: 1740416207-344477 X-HE-Meta: U2FsdGVkX19JEgN6GZYX0itMl6BCNYKdSj67+DGdLeCylB7/fR3/+evfNawYYy2EW3JehNZHZLSzYmW9kQk188vxRdIrk3Syr+A3ybabfgc9fbBdWdG8Znl/8MauLn9eBFtj2KCw+chYf6mBg12IYnnyYlES6M0PBlUO6E3qKrBN2ehk7ZyjM6kZeaMsKD1S5zdwOCveHtA5LnFc09JYI1nBV1qkIb0RK3HLexFJtw4SB5ycAth+uzB1nQVPWA4Z1upB0hH6jPLRZAx0G9pItEFRUYOHOCdMoeBGcTPLY+Qvc3Kgk84h7XLcIal01YAGh+asiLYShh1lg5We6zL3r/83rD5R4Z1dT60ghjVWVRTEtz8gp0OLfXRIQewbumph6TTENkSexn43WjEOv3as2pDn7DYFPp2ih5gfPHDhzxKf8pB1X2qEPCtht7diINx6rIF7AFdKUcOe5rip8XMxyKt2UsKNargSg2apNESYbmPsMp0fapPe7DpR/szi8qO6OqKyvXrKSDLTk6I2I3WabFQoYAIX0RGs89xsFQEO8u9BgreKGVMTalV/kNfx2P8W2+q0ihPtY2qMAOS3UnQVVEj1PNswPBfM5ky5V/fdu6bwK/IrH7PyAk68jQuZ+kaj13QPG3fWw+SAMCAvDotFJG5QdcylAwiSXxNZvfcFPH7SMGmJ5zZBQV+rXwfLYdpAt5pZZnAX/bVUWcFs6OEfTL9dh/D5r4Yk3bNVZ5YNRzYT+Ty2ZkcFBFMUx8DabQZFvMKru+UwjoehnjWHmRqLaUC7gDYzxdH6YHNXK12RiKUkKotN5YU44EjCaBgvf1NSl8s9bY+k3jknMjZgTL/anBNQdglrFavnQAY8aeddS3EkAf9IwnodNIjnT+bpP7w5FT211tO+9dI4laumZI9sNEQuc+LkfGO7g68GlvagNnwYyooG9nr8YMeUw4mYaSjftu/uWQK64LoEpHSo6qV 0cP5ubdd AlRk+Ot+4N7YXd5xIs1gDi5DkWgrJSKpsXfh5PoPPkUfJJri6jjGNJrNtSwhJ6g6C/0C26pze3S44wqKl4AO31bnAs1m+y6gtUp2VQ4a1JwNTisn8F9z9jQHSKTbGDMIJpyz+tXEEZPOeYHvSGO8WwTERumPiyFcpU5YJHEtUGZaP35JAl02CoYYaX76C2EQBAxWUwMBsJmP5KCTwDb5oDTFaEbnME8xrS7dBpUhuAhbQay6NRwCcc73h/3XcjZ7Y1DvkWirL1ln5Y3LJn1PBI03rxpWafmldbj6tAvL/Ektg00SMV43OuolffAkLCAzPhjVKx2irJB54dBdUCCS4qT4NP0RJtB1pmceqQkaUjB8IF2ObeNhfwnqlkuhn/COuy0dmEHofjEK4Dj/cSM20z0FiWCX4Lzcqr8Pi7I7HJXNFEmPd4pAt+DAVIljG2vzNejVKQMbMY3XfwIOUIZzQ3++hqGC7DnVKnqgstfCzHk4aSs6CZXeb9P3HbK4MUkDpG33Zbq8gOdmjlO/EjWgN15+Ws/S4qS5LyZCUxAD8nV3eWlI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT. For calculating "mapmax", we now use the average per-page mapcount in a large folio instead of the per-page mapcount. For hugetlb folios and folios that are not partially mapped into MMs, there is no change. Likely, this change will not matter much in practice, and an alternative might be to simple remove this stat with CONFIG_NO_PAGE_MAPCOUNT. However, there might be value to it, so let's keep it like that and document the behavior. Signed-off-by: David Hildenbrand --- Documentation/filesystems/proc.rst | 5 +++++ fs/proc/task_mmu.c | 7 ++++++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 09f0aed5a08ba..1aa190017f796 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -686,6 +686,11 @@ Where: node locality page counters (N0 == node0, N1 == node1, ...) and the kernel page size, in KB, that is backing the mapping up. +Note that some kernel configurations do not track the precise number of times +a page part of a larger allocation (e.g., THP) is mapped. In these +configurations, "mapmax" might corresponds to the average number of mappings +per page in such a larger allocation instead. + 1.2 Kernel data --------------- diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 80839bbf9657f..d7ee842367f0f 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -2862,7 +2862,12 @@ static void gather_stats(struct page *page, struct numa_maps *md, int pte_dirty, unsigned long nr_pages) { struct folio *folio = page_folio(page); - int count = folio_precise_page_mapcount(folio, page); + int count; + + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + count = folio_precise_page_mapcount(folio, page); + else + count = min_t(int, folio_average_page_mapcount(folio), 1); md->pages += nr_pages; if (pte_dirty || folio_test_dirty(folio)) From patchwork Mon Feb 24 16:56:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988512 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71AE1C021BB for ; Mon, 24 Feb 2025 16:57:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C2FF280009; Mon, 24 Feb 2025 11:56:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1733E280002; Mon, 24 Feb 2025 11:56:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB97F280009; Mon, 24 Feb 2025 11:56:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CA795280002 for ; Mon, 24 Feb 2025 11:56:51 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7F7371218EB for ; Mon, 24 Feb 2025 16:56:51 +0000 (UTC) X-FDA: 83155442622.16.2DA4B44 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 26C2E1C0011 for ; Mon, 24 Feb 2025 16:56:48 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PK8AnGaP; spf=pass (imf18.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416209; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YYLglXDyFMXUmuyPM7L4Wt5NI1TaS59XhalfSABwrEE=; b=DoV/4Y4J+Vw2+MUZKAeLF69UOS3Cngr1DLS6lrjUN5HVZ5nrHT0oddYwha1/PeMiKsPIra 4SdsX8k05ZnMgaeZBucp2vCvNrdD0dKp+JmCdh4CXV8py/yFOXkVCPDKG/UGpI+8HQiHM1 lWJC1KcOnIpJgKCl+M87TnTvNfbNVtw= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PK8AnGaP; spf=pass (imf18.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416209; a=rsa-sha256; cv=none; b=2KLuSf87pNAD+RfMFHGY5B0+OG7X7mFd7i+erVtpAfK5YfpG8p/C0KikyHHW1ihqFWmrPK shH8fxzVUjrNM7Q9uaRcZ66h9m2pMpWMe9GmtzvTUV75hWSh8JKta9wThzO3WStD3rslDR 6IMz3RvkPKOtulhvVwPxMmIDychJHX8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416208; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YYLglXDyFMXUmuyPM7L4Wt5NI1TaS59XhalfSABwrEE=; b=PK8AnGaPVfFzTpwr7RnfruTlOZMZfmx860HxQ5MTIbJy/eD72qGAVdzCFM1gAxzsVeOaUi VtptPJeZj78v1rWmc0DpmdofIV7aqA8SgdWICU5U3WCCrCC8HMGiy9PaMiZJYZy4hy1PDx q/jKj4Pje22Jj9bsSyYkQSl9vlbgGM0= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-492-gk4fGlr0P6OPfleVOuYr0A-1; Mon, 24 Feb 2025 11:56:47 -0500 X-MC-Unique: gk4fGlr0P6OPfleVOuYr0A-1 X-Mimecast-MFC-AGG-ID: gk4fGlr0P6OPfleVOuYr0A_1740416206 Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-38f27acb979so4601945f8f.0 for ; Mon, 24 Feb 2025 08:56:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416206; x=1741021006; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YYLglXDyFMXUmuyPM7L4Wt5NI1TaS59XhalfSABwrEE=; b=tkI7KIanUvmfACNvnoQ2MjN5wwJnf3HPcRPKYSRQ/S7MtQHccmhc0XHhpOrK3Fp0kS vxUhD+f+EThul/sAUHbrk+3u+fzfxRlLhKkXNRipRnSm/eZYOX+blpbE+8m63SVNuGWQ JeTupRuKVVhdUDNZXmojpCcXeW8z/qAwx4+0+Rf0ze/fK4kFnZKSPkJewHcbeTQ/1Y91 p/JcvTXS+QhS64pfKipeqq7+pZhfBy1VTXjW92AvOEgKv0/7hmmvkSUDPrcP6sumQlk9 c5AxOcnEgd8HZRJwHV3k3e1BF+56A4zH07TcSccJ6yDN6q9Q9+6b8fv8FtqZjOXeLsgj fnhA== X-Forwarded-Encrypted: i=1; AJvYcCWkeT7p1fDwO2mtOTmHKHAPF+FmeOvLKLJTnbyUU7lfCjjIjs8KpQ4kUm3+JXukCiG58/ks8GC2Gg==@kvack.org X-Gm-Message-State: AOJu0Yx0NKuxT+zkXa+RABpparfnbeNVkRfQIHuwk83qNWJQSgt5jtdu sbH+uAMzQ4AKlm27EGPK5BIsPftPzqezcV8LN3mlVqLi9h4y/P+MKKFvjuuIoGtPA8MTJV8KGdz 385Y0DBsU2HE54Cq6GL1ST65QaIpKqwv7HaFy6PAIJLb5c6pA X-Gm-Gg: ASbGncv8XFY0VzIcx8yb925r3F26jmK1lJMZigqorC4NXpbdSg5T6jYyIw/7nhWOwoz mxCT69YdNnQepuqclnTY6sisupCHuvQfYIidei+VfJaVtxJXc6rhQuFescw2BgB1tdnjdoHJZ1o 5JTKtKoIrsSinCdUZvI0nsQTuZsDEuJjxmrmGEnUgSE+esDqpDvFGHsvyyqc7slO9605+3bO6Jq JD5131D60/QGDypeua22fbfXmu9sdG3NWBApTuIRvu3oS1UBcwFXAajqOedt5BQ3x7mNR6AmkVM fCZVEToVTrt0Bp2qhDHUeUS0u1VVu96Bv6r0GINIsQ== X-Received: by 2002:a5d:6da5:0:b0:38d:cf33:31a1 with SMTP id ffacd0b85a97d-38f707afc79mr12824178f8f.23.1740416206025; Mon, 24 Feb 2025 08:56:46 -0800 (PST) X-Google-Smtp-Source: AGHT+IH7oc6/f/Omy9ecKXhn6XnBr6bqzxs+GSFB5uGM8J4SWQqdkeD7RWsCxWskwEhPaUkzkpZm0Q== X-Received: by 2002:a5d:6da5:0:b0:38d:cf33:31a1 with SMTP id ffacd0b85a97d-38f707afc79mr12824140f8f.23.1740416205579; Mon, 24 Feb 2025 08:56:45 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38f259f7998sm31659273f8f.82.2025.02.24.08.56.44 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:45 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 19/20] fs/proc/task_mmu: remove per-page mapcount dependency for smaps/smaps_rollup (CONFIG_NO_PAGE_MAPCOUNT) Date: Mon, 24 Feb 2025 17:56:01 +0100 Message-ID: <20250224165603.1434404-20-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: xJT_O8Rm-x3gkIMq2RXKKDvcZt8dFI27D3gK7AoxMOo_1740416206 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Queue-Id: 26C2E1C0011 X-Stat-Signature: 7cjbm46ruezna1e8yrdz5w9zpr394h3d X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1740416208-187688 X-HE-Meta: U2FsdGVkX1+witqDHZ7uc3UahdNRXYboXpoDRLjdr8BGNtywybKWSz8hyR69HDy2sIuTCtcG4nlUTRX3XpJiEWCJWNNspU81DhtBtNADW70RJ5cm46k8uUeQ4E6l0xa6u0evzWmKBSt6hY7eVe12+KZzMJdnNXimgF+R3P0rPxybKUpm2WjMv+cuxudBJ9LmLk2JjiplC0GSUSJwfGZGwNb6xoeLSvCaFGucONfrOHRdtJLkwZ4SJ47G9IZZuZz0otJVkcx6vABXU2sReq4tNFczaBY4OpozMX63Q6h1I4lBHmLSsCu8zunszAZZsL6IegcuMEZ2fJA9VF8b6+Jl0G1sMUSjEKJJsqOkFEbobhi+j9oDvfmZpn+ss8S/nNQAwp5JO7Kqm7UZaESLeCPeM9epWIemIsRSDyZ8PxrgZvk+yG5vTp24csHlvsK1aM4wemII6NmPYpaEcMB8dwdFgXGaci38nXspU79+abv7D8Yw8pw8NFixmPuM4sKwHeVbApNTzqnMBTiw/9QzRPQXxGpOAQDsaIWNtN1YehU5zlF50mkt9B3OgIp8at0Sn+PVbtgFC1R6du9B1PMO/VuhkNkYmjUn/u08hrhRIWl+nwEA+drSrmxl0f45xoJykrIzmNhfcmYYvcgnhdeN/3pd155YRPLhUUlDKR9JaXAQ2Gn+HzTMXhhHoQj7W2Czn4ZsTdmoE7OTpz/kNFlD8lwPJtqOqbM479xjG5h3xFS4ssNNDuzIlFEK6MOirY1ROmeNcRV+tnBTY2qP1sZU4CGYhGdmVmSVKOjN1HLKwvp5v1oQCuaLeu+FWv6BahRrqLzpq++XqmrL8WT+G/OXMah75ya/+ySOE0bTDD/O6kRKeNSgEvX8rw63vYGvIQo5YY6UyhESIedYS1Ft17t3Z1ysfo+f9LIJr8a4hJHlVCiGNPYJ8R4cCaRx1pyC/8h94iRp/NqFF5RjmPVp2UCZ6cO biKJ+bU7 XRGXXDL1ziWJJXF0jVUFQLTPLLn4ZulFVl6wZxJspQ417z63ezpfpeUJK2BeSXpLbJnnObtIlBp7xmCJHl8rvzQsgRXr3nad0W8UEBWr0uIeYO4MwuouWxmn45i6sINKD88yTOzY230cviFBH6xzUfwYl0P+8VxaxOHYeTW9fJbR++/1WloJ2iajh9mTwPwCbmGShLrsGi9W8WOCYXDL9qOBRiU3WZ0EegYKgG7+upVJIG9ATRVMaKCxqzNs91oX7Upk9xUpvFcr+C8l68qd0vy93MN4cKAzFos2OHsptPKi1JsAkmSC7bhkWm/dhiRPrwntrJITxuuUcSnAqlj7ERpjw9VAG2iRhUIhAvI6pErer2H8qqHECUJpxFR8lohwKuHVHzb8Boi7/OuSELfD1tVygD+dw093OxiC3V72DmMO+iCWAJ/B1XWUQJLaPvUdXZIq7Chr13f7Ne+PM+zIbBCJKEPwm0J8AH+PdCZMiMmyUSHlEAt9KN+1U4iev0zNuON8LYwLlDauv7Q7L7Cr0nja9Va1xf8A5xOCtPi/AeApLmx8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT. When computing the output for smaps / smaps_rollups, in particular when calculating the USS (Unique Set Size) and the PSS (Proportional Set Size), we still rely on per-page mapcounts. To determine private vs. shared, we'll use folio_likely_mapped_shared(), similar to how we handle PM_MMAP_EXCLUSIVE. Similarly, we might now under-estimate the USS and count pages towards "shared" that are actually "private" ("exclusively mapped"). When calculating the PSS, we'll now also use the average per-page mapcount for large folios: this can result in both, an over-estimation and an under-estimation of the PSS. The difference is not expected to matter much in practice, but we'll have to learn as we go. We can now provide folio_precise_page_mapcount() only with CONFIG_PAGE_MAPCOUNT, and remove one of the last users of per-page mapcounts when CONFIG_NO_PAGE_MAPCOUNT is enabled. Document the new behavior. Signed-off-by: David Hildenbrand --- Documentation/filesystems/proc.rst | 13 +++++++++++++ fs/proc/internal.h | 8 ++++++++ fs/proc/task_mmu.c | 17 +++++++++++++++-- 3 files changed, 36 insertions(+), 2 deletions(-) diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 1aa190017f796..57d55274a1f42 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -506,6 +506,19 @@ Note that even a page which is part of a MAP_SHARED mapping, but has only a single pte mapped, i.e. is currently used by only one process, is accounted as private and not as shared. +Note that in some kernel configurations, all pages part of a larger allocation +(e.g., THP) might be considered "shared" if the large allocation is +considered "shared": if not all pages are exclusive to the same process. +Further, some kernel configurations might consider larger allocations "shared", +if they were at one point considered "shared", even if they would now be +considered "exclusive". + +Some kernel configurations do not track the precise number of times a page part +of a larger allocation is mapped. In this case, when calculating the PSS, the +average number of mappings per page in this larger allocation might be used +as an approximation for the number of mappings of a page. The PSS calculation +will be imprecise in this case. + "Referenced" indicates the amount of memory currently marked as referenced or accessed. diff --git a/fs/proc/internal.h b/fs/proc/internal.h index 16aa1fd260771..70205425a2daa 100644 --- a/fs/proc/internal.h +++ b/fs/proc/internal.h @@ -143,6 +143,7 @@ unsigned name_to_int(const struct qstr *qstr); /* Worst case buffer size needed for holding an integer. */ #define PROC_NUMBUF 13 +#ifdef CONFIG_PAGE_MAPCOUNT /** * folio_precise_page_mapcount() - Number of mappings of this folio page. * @folio: The folio. @@ -173,6 +174,13 @@ static inline int folio_precise_page_mapcount(struct folio *folio, return mapcount; } +#else /* !CONFIG_PAGE_MAPCOUNT */ +static inline int folio_precise_page_mapcount(struct folio *folio, + struct page *page) +{ + BUILD_BUG(); +} +#endif /* CONFIG_PAGE_MAPCOUNT */ /** * folio_average_page_mapcount() - Average number of mappings per page in this diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index d7ee842367f0f..7ca0bc3bf417d 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -707,6 +707,8 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page, struct folio *folio = page_folio(page); int i, nr = compound ? compound_nr(page) : 1; unsigned long size = nr * PAGE_SIZE; + bool exclusive; + int mapcount; /* * First accumulate quantities that depend only on |size| and the type @@ -747,18 +749,29 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page, dirty, locked, present); return; } + + if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) { + mapcount = folio_average_page_mapcount(folio); + exclusive = !folio_maybe_mapped_shared(folio); + } + /* * We obtain a snapshot of the mapcount. Without holding the folio lock * this snapshot can be slightly wrong as we cannot always read the * mapcount atomically. */ for (i = 0; i < nr; i++, page++) { - int mapcount = folio_precise_page_mapcount(folio, page); unsigned long pss = PAGE_SIZE << PSS_SHIFT; + + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) { + mapcount = folio_precise_page_mapcount(folio, page); + exclusive = mapcount < 2; + } + if (mapcount >= 2) pss /= mapcount; smaps_page_accumulate(mss, folio, PAGE_SIZE, pss, - dirty, locked, mapcount < 2); + dirty, locked, exclusive); } } From patchwork Mon Feb 24 16:56:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13988513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3C2FC021BC for ; Mon, 24 Feb 2025 16:57:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9D2C28000B; Mon, 24 Feb 2025 11:56:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C21CC28000A; Mon, 24 Feb 2025 11:56:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B2CD28000B; Mon, 24 Feb 2025 11:56:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 73CB728000A for ; Mon, 24 Feb 2025 11:56:56 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 27489121904 for ; Mon, 24 Feb 2025 16:56:56 +0000 (UTC) X-FDA: 83155442832.13.EE00199 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf24.hostedemail.com (Postfix) with ESMTP id BE3F518000F for ; Mon, 24 Feb 2025 16:56:53 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YH+Lits1; spf=pass (imf24.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740416213; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MgdgCkfQT710L4EQa1Wd5rppib0ue2uPYRNKLBnKmzg=; b=oqot45UNyXoBVfd9FpVyRnAEuqFbCgn4ftARA3rFqNI83tRnd6QdqUuibqWUOVu3484t2f gGhgMRtImUMwMPipJOziqgX96EOyUfoC3ROdV4xGWeBHkqyrb/ju742C2TnJkRQgag+cgq vDUbRJU3m1R2qcIK1Rds7bFkwn3ayYU= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YH+Lits1; spf=pass (imf24.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740416213; a=rsa-sha256; cv=none; b=EFWAf/sa7QuYNFPQKkh8LGWr78G8vnZxHkVCEG908+sZ4XLHRRUsTyPO5QMcz87JZRGHon 4wp99WPHsADsK363Z+F1mq8rn6EpntsCoBRJ1PNu9bbC8zSnsP9GFEcFMsieasGhALoQj4 uRYCBfUwHGuty6nkOvfMKK8f1cK10Lg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740416213; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MgdgCkfQT710L4EQa1Wd5rppib0ue2uPYRNKLBnKmzg=; b=YH+Lits1/2mpRShC/tEPS49+cSGPzwAPZ9FMS+VreeyMXt4PqVp2R+9E3kQb3icXbG8b7t gx2TBNVerTiTqxXph/cft2VfICLmAdNcaM+GCAnENH4v5GNOOha9yPdlKTWKm4uSr7Q4Vb JdGsQK3ycXeGUkoTsuG9huFW0i2lkmE= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-515-Fp0rguKIOOSmLdrji9GMzw-1; Mon, 24 Feb 2025 11:56:49 -0500 X-MC-Unique: Fp0rguKIOOSmLdrji9GMzw-1 X-Mimecast-MFC-AGG-ID: Fp0rguKIOOSmLdrji9GMzw_1740416208 Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-43935e09897so32831445e9.1 for ; Mon, 24 Feb 2025 08:56:49 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740416208; x=1741021008; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MgdgCkfQT710L4EQa1Wd5rppib0ue2uPYRNKLBnKmzg=; b=FcqKZZxpyLngyYJVwMpzZPJ9QpmVNiPuPWmWKTMvN5wgRI1wYXfTrfLqBdPdPFM57t BaKQH5qk3lL6wDcGSunxiyGlAmUBYc4V7moO0yBlrqyHfRbAQnkUDZPzRn5YDn7QscJl 6VWtTvq3O4JKsAle1lO33agEzhOkF02NFVN7GkCzj2q4HYRumY9mNPUQQUwBCCChflEA LHC2EpOc/FekJUEN0YigiHgGUmflj4tiWwATXV2uNs+Fu//L1KyQ0so5sUfo7NWSkIkd Y2KqRPG+WYG0XzOz8t1aDUlRAf02gJkVptP6FUmMqSs+6kuPYByoMr0bQc+p8VqIe+I+ KJtw== X-Forwarded-Encrypted: i=1; AJvYcCX8GKEk8n08GNKRjHMTjcCh9IOP3kyfpQaLVENKoa7YmjZkct5I6L8OLiZS8ebThjERhMxImN++jg==@kvack.org X-Gm-Message-State: AOJu0YyZZptdEtnDzBfFfS+PYYjw/mkX82R2MBxJJBcu44l5q/rE28la YyORWT2LVu+pOxH6kudDBcmi9lljKCqoben30SBv7X1Jx2myjwaEiiKND6zLCEF4+xxLdTq123M zmOecwXRoOx/7gRDxpyR7Yj9uq5BJcYv4j0b40BBXkEjU1v+a X-Gm-Gg: ASbGncsO0LHNRf4jUt4/U3k/hoDMFy0fhktckUAo8ZGWzWf5bXtC6FU3YxuZRg7LNFk rkNATMHJikS4d6MVLN8e2nk+kqCfKCaSKMK1x83PZlBKgzChp2rVvG75BnyQevKDCd3jSoJz2Ui KfcCAKmdNZvZFkiqkrZaTbWpIFiwqfEj33kMYJOhxQy5U4Fo4y45jRBLVWHxo39qhiroY/jVBNP p2OSjotzqRGoTUJEypG89aqh+JxlfBmRgqudOFh867Km3e/f6lnmxF4UBfUGKBTNOGtlOtlQQvs b03ylctN0Z22O9eqCTnSaUjx6SIeQqy7j1SU/ovrZw== X-Received: by 2002:a05:600c:5112:b0:434:a781:f5d5 with SMTP id 5b1f17b1804b1-43aa4ee1f53mr49951475e9.30.1740416208237; Mon, 24 Feb 2025 08:56:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IEkH/GtVfiPSlmNTBjcB5VCGgkWNht+vToPEPB5qduC+IDFH5nKzmUcOLgdLr6vKVUyR6uQ9w== X-Received: by 2002:a05:600c:5112:b0:434:a781:f5d5 with SMTP id 5b1f17b1804b1-43aa4ee1f53mr49951055e9.30.1740416207692; Mon, 24 Feb 2025 08:56:47 -0800 (PST) Received: from localhost (p4ff234b6.dip0.t-ipconnect.de. [79.242.52.182]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-439b02ce41dsm115554595e9.1.2025.02.24.08.56.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Feb 2025 08:56:47 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Muchun Song , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn Subject: [PATCH v2 20/20] mm: stop maintaining the per-page mapcount of large folios (CONFIG_NO_PAGE_MAPCOUNT) Date: Mon, 24 Feb 2025 17:56:02 +0100 Message-ID: <20250224165603.1434404-21-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224165603.1434404-1-david@redhat.com> References: <20250224165603.1434404-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: TtLKjEwuIJBTND-gS56-eRZnuZqObtQp1ZHF3eFmtMU_1740416208 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Queue-Id: BE3F518000F X-Stat-Signature: hpm6an4md4eqphx7dqfuqusnq58zdrk4 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1740416213-895941 X-HE-Meta: U2FsdGVkX1/KSXkBhAmF0h9nGhRHgcmki9R4zWJpCXdeZNoe8pchVEyBhAkw8gcHdQ2LZ6yQ0pjnHyYvPYXEv2+QBr6dzlXDkZ5KH2B8Xb7bGbv6G34QH4hZeooyQrBHza+GWQOB0kJF+5CFi3iWbOi5m7mSAC74r8eJ1ezuU8u9tWcBwQeh6WnVHETLQtBb6UoSLj7zSPWiAKophldavPNFm5W1XYjc35ZBHLRd5MeW/8DMhsL7/xEWPwWhQpynPSL3gXrUedkHJs4I65PPDzzFNcTEvH2rl2KkWGYV2K95KF9KasNtSXAW0IZ+zQ6hK5mQUNNfc1ii4IGdF/wR82H9LXi3lTc+Ei8B0Me4diPUZ3m68vgOFTRPPaaf2G+Aix0CI2FCgNbb+J0lHM5mCFb4S0lkciSmxVqNFsXITm63camCEvA1HaXTRLj+Nmj4f9PybylAUiU9x/dAai91vgb3TRbVSwalhsmJjgR1K/VTo65vUKzv/DwMCTFXiDHsjcVDh7eYKFVO2WB4hQNyktyag4MMgQ2tAi01c1tBHBPGUj99izz/5muqQC9TJsXDmnzk/qf0lYW49IMRJjqP1Rs6UX0y7v+yu9GCqXgISEEzNqUtApiGkAL8d69rjbnwwzcmvunIFKM8YZhJnFCXLNwVHZcjQFqKhLQqz4hOV7QGSCH0fejk7yVsEg5uv2B+yHM1IMOMiy62t1tDidWGBrX5Sv+XfZMcaayz3ZGXevhYW37rvtiLHMPalZtoxtfRU6VCcVQy1yrZ7osGVQouHil80eMPGgw2RhTytSVsDMlE9GZV8IwvU27YuL1ibd07n8G86GUUI7vvNc3MJCtPYYXNipGQqE0LeFLzD1FX0fRUuJQHYhBT7U764yxB4/yXP5dFs8l4sA+/uGG46h/fnWNs+SJteG1TbWzHSvZfKMnLmqTJUDPqCXXS2xyIXkxLM1c6hzzIMIklv5kwRcO Buu6wkkT BH8FXE9i/GDgyEuCx1MapNDgTPCZNa0TzaSqw1bJzQFEqAtS+u0DHjPwN5V++9csVRKcCrXA4vuomg6S/mXmXj8avJOZUU2UIoc19wVbi1x5I1NY6wuTh9bc7xRohxQ/wy5lCwkPDGu+5PDH1+tg+SUFa8NSzQ5QVh0TDlPio2PrVI4orJcY+k9GHmfNPNBeT1A7VKQd67YYbooD4LxSebB2gLDuf+MxDqwbY/ZJvJniuJ71f1hU5poSu3ai8M6QbE/meLXnaPuUCg2CpK6GNEqbdw7sJ2RNw3viAVhFjyNaiT5M22dOjBis86ycbjw3OQjTsm+wEQRc0WPkRgXo7TfbBy9snwk+xUWGwgLmIv9z37BacPRXYPTjQtt9cGPC8/shoKOPdaAsIoyS69fDNsI9rx+bT3Akw2YygjPn0M+JsyQ3wpQPyJmPGTfPWA84pu12PKtg+wMV1tbOMGr0743zRyzf6XNHe23E3Osn6VQL7+1/y5NP15BeEXvJ9g/CpLH02zQvpWEHP2SvwEwRZGvdb7CoNfLs4S9CdcJS4MWApEP0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Everything is in place to stop using the per-page mapcounts in large folios: the mapcount of tail pages will always be logically 0 (-1 value), just like it currently is for hugetlb folios already, and the page mapcount of the head page is either 0 (-1 value) or contains a page type (e.g., hugetlb). Maintaining _nr_pages_mapped without per-page mapcounts is impossible, so that one also has to go with CONFIG_NO_PAGE_MAPCOUNT. There are two remaining implications: (1) Per-node, per-cgroup and per-lruvec stats of "NR_ANON_MAPPED" ("mapped anonymous memory") and "NR_FILE_MAPPED" ("mapped file memory"): As soon as any page of the folio is mapped -- folio_mapped() -- we now account the complete folio as mapped. Once the last page is unmapped -- !folio_mapped() -- we account the complete folio as unmapped. This implies that ... * "AnonPages" and "Mapped" in /proc/meminfo and /sys/devices/system/node/*/meminfo * cgroup v2: "anon" and "file_mapped" in "memory.stat" and "memory.numa_stat" * cgroup v1: "rss" and "mapped_file" in "memory.stat" and "memory.numa_stat ... can now appear higher than before. But note that these folios do consume that memory, simply not all pages are actually currently mapped. It's worth nothing that other accounting in the kernel (esp. cgroup charging on allocation) is not affected by this change. [why oh why is "anon" called "rss" in cgroup v1] (2) Detecting partial mappings Detecting whether anon THPs are partially mapped gets a bit more unreliable. As long as a single MM maps such a large folio ("exclusively mapped"), we can reliably detect it. Especially before fork() / after a short-lived child process quit, we will detect partial mappings reliably, which is the common case. In essence, if the average per-page mapcount in an anon THP is < 1, we know for sure that we have a partial mapping. However, as soon as multiple MMs are involved, we might miss detecting partial mappings: this might be relevant with long-lived child processes. If we have a fully-mapped anon folio before fork(), once our child processes and our parent all unmap (zap/COW) the same pages (but not the complete folio), we might not detect the partial mapping. However, once the child processes quit we would detect the partial mapping. How relevant this case is in practice remains to be seen. Swapout/migration will likely mitigate this. In the future, RMAP walkers could check for that for that case (e.g., when collecting access bits during reclaim) and simply flag them for deferred-splitting. Signed-off-by: David Hildenbrand --- .../admin-guide/cgroup-v1/memory.rst | 4 + Documentation/admin-guide/cgroup-v2.rst | 10 ++- Documentation/filesystems/proc.rst | 10 ++- Documentation/mm/transhuge.rst | 31 +++++-- include/linux/rmap.h | 35 ++++++-- mm/internal.h | 5 +- mm/page_alloc.c | 3 +- mm/rmap.c | 80 +++++++++++++++++-- 8 files changed, 150 insertions(+), 28 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v1/memory.rst b/Documentation/admin-guide/cgroup-v1/memory.rst index 286d16fc22ebb..53cf081b22e81 100644 --- a/Documentation/admin-guide/cgroup-v1/memory.rst +++ b/Documentation/admin-guide/cgroup-v1/memory.rst @@ -609,6 +609,10 @@ memory.stat file includes following statistics: 'rss + mapped_file" will give you resident set size of cgroup. + Note that some kernel configurations might account complete larger + allocations (e.g., THP) towards 'rss' and 'mapped_file', even if + only some, but not all that memory is mapped. + (Note: file and shmem may be shared among other cgroups. In that case, mapped_file is accounted only when the memory cgroup is owner of page cache.) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 175e9435ad5c1..53ada5c2620a7 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -1448,7 +1448,10 @@ The following nested keys are defined. anon Amount of memory used in anonymous mappings such as - brk(), sbrk(), and mmap(MAP_ANONYMOUS) + brk(), sbrk(), and mmap(MAP_ANONYMOUS). Note that + some kernel configurations might account complete larger + allocations (e.g., THP) if only some, but not all the + memory of such an allocation is mapped anymore. file Amount of memory used to cache filesystem data, @@ -1491,7 +1494,10 @@ The following nested keys are defined. Amount of application memory swapped out to zswap. file_mapped - Amount of cached filesystem data mapped with mmap() + Amount of cached filesystem data mapped with mmap(). Note + that some kernel configurations might account complete + larger allocations (e.g., THP) if only some, but not + not all the memory of such an allocation is mapped. file_dirty Amount of cached filesystem data that was modified but diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 57d55274a1f42..c5052acfa0747 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -1150,9 +1150,15 @@ Dirty Writeback Memory which is actively being written back to the disk AnonPages - Non-file backed pages mapped into userspace page tables + Non-file backed pages mapped into userspace page tables. Note that + some kernel configurations might consider all pages part of a + larger allocation (e.g., THP) as "mapped", as soon as a single + page is mapped. Mapped - files which have been mmapped, such as libraries + files which have been mmapped, such as libraries. Note that some + kernel configurations might consider all pages part of a larger + allocation (e.g., THP) as "mapped", as soon as a single page is + mapped. Shmem Total memory used by shared memory (shmem) and tmpfs KReclaimable diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index baa17d718a762..0e7f8e4cd2e33 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -116,23 +116,28 @@ pages: succeeds on tail pages. - map/unmap of a PMD entry for the whole THP increment/decrement - folio->_entire_mapcount, increment/decrement folio->_large_mapcount - and also increment/decrement folio->_nr_pages_mapped by ENTIRELY_MAPPED - when _entire_mapcount goes from -1 to 0 or 0 to -1. + folio->_entire_mapcount and folio->_large_mapcount. We also maintain the two slots for tracking MM owners (MM ID and corresponding mapcount), and the current status ("maybe mapped shared" vs. "mapped exclusively"). + With CONFIG_PAGE_MAPCOUNT, we also increment/decrement + folio->_nr_pages_mapped by ENTIRELY_MAPPED when _entire_mapcount goes + from -1 to 0 or 0 to -1. + - map/unmap of individual pages with PTE entry increment/decrement - page->_mapcount, increment/decrement folio->_large_mapcount and also - increment/decrement folio->_nr_pages_mapped when page->_mapcount goes - from -1 to 0 or 0 to -1 as this counts the number of pages mapped by PTE. + folio->_large_mapcount. We also maintain the two slots for tracking MM owners (MM ID and corresponding mapcount), and the current status ("maybe mapped shared" vs. "mapped exclusively"). + With CONFIG_PAGE_MAPCOUNT, we also increment/decrement + page->_mapcount and increment/decrement folio->_nr_pages_mapped when + page->_mapcount goes from -1 to 0 or 0 to -1 as this counts the number + of pages mapped by PTE. + split_huge_page internally has to distribute the refcounts in the head page to the tail pages before clearing all PG_head/tail bits from the page structures. It can be done easily for refcounts taken by page table @@ -159,8 +164,8 @@ clear where references should go after split: it will stay on the head page. Note that split_huge_pmd() doesn't have any limitations on refcounting: pmd can be split at any point and never fails. -Partial unmap and deferred_split_folio() -======================================== +Partial unmap and deferred_split_folio() (anon THP only) +======================================================== Unmapping part of THP (with munmap() or other way) is not going to free memory immediately. Instead, we detect that a subpage of THP is not in use @@ -175,3 +180,13 @@ a THP crosses a VMA boundary. The function deferred_split_folio() is used to queue a folio for splitting. The splitting itself will happen when we get memory pressure via shrinker interface. + +With CONFIG_PAGE_MAPCOUNT, we reliably detect partial mappings based on +folio->_nr_pages_mapped. + +With CONFIG_NO_PAGE_MAPCOUNT, we detect partial mappings based on the +average per-page mapcount in a THP: if the average is < 1, an anon THP is +certainly partially mapped. As long as only a single process maps a THP, +this detection is reliable. With long-running child processes, there can +be scenarios where partial mappings can currently not be detected, and +might need asynchronous detection during memory reclaim in the future. diff --git a/include/linux/rmap.h b/include/linux/rmap.h index c131b0efff0fa..6b82b618846ee 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -240,7 +240,7 @@ static __always_inline void folio_set_large_mapcount(struct folio *folio, folio_set_mm_id(folio, 0, vma->vm_mm->mm_id); } -static __always_inline void folio_add_large_mapcount(struct folio *folio, +static __always_inline int folio_add_return_large_mapcount(struct folio *folio, int diff, struct vm_area_struct *vma) { const mm_id_t mm_id = vma->vm_mm->mm_id; @@ -286,9 +286,11 @@ static __always_inline void folio_add_large_mapcount(struct folio *folio, folio->_mm_ids |= FOLIO_MM_IDS_SHARED_BIT; } folio_unlock_large_mapcount(folio); + return new_mapcount_val + 1; } +#define folio_add_large_mapcount folio_add_return_large_mapcount -static __always_inline void folio_sub_large_mapcount(struct folio *folio, +static __always_inline int folio_sub_return_large_mapcount(struct folio *folio, int diff, struct vm_area_struct *vma) { const mm_id_t mm_id = vma->vm_mm->mm_id; @@ -331,7 +333,9 @@ static __always_inline void folio_sub_large_mapcount(struct folio *folio, folio->_mm_ids &= ~FOLIO_MM_IDS_SHARED_BIT; out: folio_unlock_large_mapcount(folio); + return new_mapcount_val + 1; } +#define folio_sub_large_mapcount folio_sub_return_large_mapcount #else /* !CONFIG_MM_ID */ /* * See __folio_rmap_sanity_checks(), we might map large folios even without @@ -350,17 +354,33 @@ static inline void folio_add_large_mapcount(struct folio *folio, atomic_add(diff, &folio->_large_mapcount); } +static inline int folio_add_return_large_mapcount(struct folio *folio, + int diff, struct vm_area_struct *vma) +{ + BUILD_BUG(); +} + static inline void folio_sub_large_mapcount(struct folio *folio, int diff, struct vm_area_struct *vma) { atomic_sub(diff, &folio->_large_mapcount); } + +static inline int folio_sub_return_large_mapcount(struct folio *folio, + int diff, struct vm_area_struct *vma) +{ + BUILD_BUG(); +} #endif /* CONFIG_MM_ID */ #define folio_inc_large_mapcount(folio, vma) \ folio_add_large_mapcount(folio, 1, vma) +#define folio_inc_return_large_mapcount(folio, vma) \ + folio_add_return_large_mapcount(folio, 1, vma) #define folio_dec_large_mapcount(folio, vma) \ folio_sub_large_mapcount(folio, 1, vma) +#define folio_dec_return_large_mapcount(folio, vma) \ + folio_sub_return_large_mapcount(folio, 1, vma) /* RMAP flags, currently only relevant for some anon rmap operations. */ typedef int __bitwise rmap_t; @@ -538,9 +558,11 @@ static __always_inline void __folio_dup_file_rmap(struct folio *folio, break; } - do { - atomic_inc(&page->_mapcount); - } while (page++, --nr_pages > 0); + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) { + do { + atomic_inc(&page->_mapcount); + } while (page++, --nr_pages > 0); + } folio_add_large_mapcount(folio, orig_nr_pages, dst_vma); break; case RMAP_LEVEL_PMD: @@ -638,7 +660,8 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, do { if (PageAnonExclusive(page)) ClearPageAnonExclusive(page); - atomic_inc(&page->_mapcount); + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + atomic_inc(&page->_mapcount); } while (page++, --nr_pages > 0); folio_add_large_mapcount(folio, orig_nr_pages, dst_vma); break; diff --git a/mm/internal.h b/mm/internal.h index 7303ddd9dac04..7a4f81a6edd66 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -84,6 +84,8 @@ void page_writeback_init(void); */ static inline int folio_nr_pages_mapped(const struct folio *folio) { + if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) + return -1; return atomic_read(&folio->_nr_pages_mapped) & FOLIO_PAGES_MAPPED; } @@ -721,7 +723,8 @@ static inline void prep_compound_head(struct page *page, unsigned int order) folio_set_order(folio, order); atomic_set(&folio->_large_mapcount, -1); - atomic_set(&folio->_nr_pages_mapped, 0); + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + atomic_set(&folio->_nr_pages_mapped, 0); if (IS_ENABLED(CONFIG_MM_ID)) { folio->_mm_ids = 0; folio->_mm_id_mapcount[0] = -1; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 08caa92553998..4402672b5d838 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -951,7 +951,8 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) bad_page(page, "nonzero large_mapcount"); goto out; } - if (unlikely(atomic_read(&folio->_nr_pages_mapped))) { + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT) && + unlikely(atomic_read(&folio->_nr_pages_mapped))) { bad_page(page, "nonzero nr_pages_mapped"); goto out; } diff --git a/mm/rmap.c b/mm/rmap.c index 8de415157bc8d..67bb273dfb80d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1258,6 +1258,16 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, break; } + if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) { + nr = folio_add_return_large_mapcount(folio, orig_nr_pages, vma); + if (nr == orig_nr_pages) + /* Was completely unmapped. */ + nr = folio_large_nr_pages(folio); + else + nr = 0; + break; + } + do { first += atomic_inc_and_test(&page->_mapcount); } while (page++, --nr_pages > 0); @@ -1271,6 +1281,18 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, case RMAP_LEVEL_PMD: case RMAP_LEVEL_PUD: first = atomic_inc_and_test(&folio->_entire_mapcount); + if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) { + if (level == RMAP_LEVEL_PMD && first) + *nr_pmdmapped = folio_large_nr_pages(folio); + nr = folio_inc_return_large_mapcount(folio, vma); + if (nr == 1) + /* Was completely unmapped. */ + nr = folio_large_nr_pages(folio); + else + nr = 0; + break; + } + if (first) { nr = atomic_add_return_relaxed(ENTIRELY_MAPPED, mapped); if (likely(nr < ENTIRELY_MAPPED + ENTIRELY_MAPPED)) { @@ -1436,13 +1458,23 @@ static __always_inline void __folio_add_anon_rmap(struct folio *folio, break; } } + + VM_WARN_ON_FOLIO(!folio_test_large(folio) && PageAnonExclusive(page) && + atomic_read(&folio->_mapcount) > 0, folio); for (i = 0; i < nr_pages; i++) { struct page *cur_page = page + i; - /* While PTE-mapping a THP we have a PMD and a PTE mapping. */ - VM_WARN_ON_FOLIO((atomic_read(&cur_page->_mapcount) > 0 || - (folio_test_large(folio) && - folio_entire_mapcount(folio) > 1)) && + VM_WARN_ON_FOLIO(folio_test_large(folio) && + folio_entire_mapcount(folio) > 1 && + PageAnonExclusive(cur_page), folio); + if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) + continue; + + /* + * While PTE-mapping a THP we have a PMD and a PTE + * mapping. + */ + VM_WARN_ON_FOLIO(atomic_read(&cur_page->_mapcount) > 0 && PageAnonExclusive(cur_page), folio); } @@ -1548,20 +1580,23 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, for (i = 0; i < nr; i++) { struct page *page = folio_page(folio, i); - /* increment count (starts at -1) */ - atomic_set(&page->_mapcount, 0); + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + /* increment count (starts at -1) */ + atomic_set(&page->_mapcount, 0); if (exclusive) SetPageAnonExclusive(page); } folio_set_large_mapcount(folio, nr, vma); - atomic_set(&folio->_nr_pages_mapped, nr); + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + atomic_set(&folio->_nr_pages_mapped, nr); } else { nr = folio_large_nr_pages(folio); /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); folio_set_large_mapcount(folio, 1, vma); - atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED); + if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT)) + atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED); if (exclusive) SetPageAnonExclusive(&folio->page); nr_pmdmapped = nr; @@ -1665,6 +1700,19 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, break; } + if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) { + nr = folio_sub_return_large_mapcount(folio, nr_pages, vma); + if (!nr) { + /* Now completely unmapped. */ + nr = folio_nr_pages(folio); + } else { + partially_mapped = nr < folio_large_nr_pages(folio) && + !folio_entire_mapcount(folio); + nr = 0; + } + break; + } + folio_sub_large_mapcount(folio, nr_pages, vma); do { last += atomic_add_negative(-1, &page->_mapcount); @@ -1678,6 +1726,22 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, break; case RMAP_LEVEL_PMD: case RMAP_LEVEL_PUD: + if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) { + last = atomic_add_negative(-1, &folio->_entire_mapcount); + if (level == RMAP_LEVEL_PMD && last) + nr_pmdmapped = folio_large_nr_pages(folio); + nr = folio_dec_return_large_mapcount(folio, vma); + if (!nr) { + /* Now completely unmapped. */ + nr = folio_large_nr_pages(folio); + } else { + partially_mapped = last && + nr < folio_large_nr_pages(folio); + nr = 0; + } + break; + } + folio_dec_large_mapcount(folio, vma); last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) {