From patchwork Thu Aug 29 16:56:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13783460 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7496C87FC8 for ; Thu, 29 Aug 2024 16:58:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4567D6B009B; Thu, 29 Aug 2024 12:58:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 407026B009C; Thu, 29 Aug 2024 12:58:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 280716B009D; Thu, 29 Aug 2024 12:58:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 075D46B009B for ; Thu, 29 Aug 2024 12:58:00 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B7402AA9DE for ; Thu, 29 Aug 2024 16:57:59 +0000 (UTC) X-FDA: 82505890278.04.36BD24D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 051B712000A for ; Thu, 29 Aug 2024 16:57:57 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KOHE1vMi; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724950605; a=rsa-sha256; cv=none; b=kDBgoPfuACu7wZoEpwE19fNR/892mmfKyMeSp/larhO48VRQ0Z9xt8RD2FKnr3RFWgGkXl EHJbtsE9DP2gJusLtl8pGMqsnnrqxSzL5cjG1e16R7IMRYeYRcJLchDi972RPYUtd9vyey cn7ce4Z0sAFLOEqf4BgH1t77NGqGm4E= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KOHE1vMi; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724950605; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kg6KaZoa+GHuxR5fT8B3Y3uXOyNmhSnzACXsGLCOA4A=; b=L5H43dM8S3n4d8bkKAkgb4vNmggoRCeEY9yoYbLJAdPKMwxUJJ5CbvvpeVDy/581uFejsp LBB3txWFqJ7ZBRkU+6zVr99YVBMz2X/02+lU7CkzJ/GY2GoT6CXVbIkMN+m6qCg4O/GTTV i7RbEzkZTVE/OEtKB9vdP/9vGZBKRI4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724950677; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kg6KaZoa+GHuxR5fT8B3Y3uXOyNmhSnzACXsGLCOA4A=; b=KOHE1vMiC5dPz0AhIl67IePIElartihFRIXiRFTPlV2eZ7Y+bYUnj9cDppjvtyuDJjTSM1 TUIiid9ICREN1CpDwRJeANEDpk8GTL5LNELcdqQ78Cgh1inzLzJp62ZW+dllfO0qZy2lAZ QaJ/nxoBZ+Teh+Y/Bk02FjIGr4T452o= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-125-b73DxecZOFi28Pm89Y2ZEg-1; Thu, 29 Aug 2024 12:57:55 -0400 X-MC-Unique: b73DxecZOFi28Pm89Y2ZEg-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E5DFB1954223; Thu, 29 Aug 2024 16:57:50 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.193.245]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 920AC1955F66; Thu, 29 Aug 2024 16:57:41 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, x86@kernel.org, linux-fsdevel@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen Subject: [PATCH v1 07/17] mm/rmap: abstract large mapcount operations for large folios (!hugetlb) Date: Thu, 29 Aug 2024 18:56:10 +0200 Message-ID: <20240829165627.2256514-8-david@redhat.com> In-Reply-To: <20240829165627.2256514-1-david@redhat.com> References: <20240829165627.2256514-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 051B712000A X-Stat-Signature: rc64qhsxhnreux69sdezwuqihb68srzj X-Rspam-User: X-HE-Tag: 1724950677-791990 X-HE-Meta: U2FsdGVkX1/CAsb8SMckg0fOkCs55M/SUE8vRjw0ffLiWH6HX6iQkjeFJpkew2OQpFUhVJFIE44KwLFZb+2Ab8DqSS+4HDkT/Wxp9gL5FEVz91JDjb2gCNGytEoE34Vqrz3j3FktdBnK1u3tV+W1LRK/XBp5lBgNWrsbpHN1XvjnOxzlLTiaXjaaeqecOgE2h16uGGOJziwG41zDh426oL8f1ubecPrlyfpIlMs/IHenQTYO4wF56r7uQ7A3tlA4NOgYZk91/blMyiPAGsLNaDPcL90ugTepu7fPjm6vwICUuwhozP/dd+DD8FGUs9tUkxRKxZZ1s1Ts1VOo3s3OxDctRyyv0i1tuCVKMXNS1BB+YMAIpg10xpNozbyXzKwLkwm9d/5Ge3mXbKo9P0uRj9m/as5kcqJ+tFkdgSNbfyMzwfyjGQcMeo5hFc4oW3yP6pFpL2J5W7AxK2AbbuGIemcClhhJ7us8gGH9jhkpYHFGX6vi5iDh/euLaMHh4/XTuYnfx+5NXlnoQVhUORqMLlFAbvbL5fqioX2BMLxCCFQFFeZdQQJLnVkCCn1z1LOMcFnY/pIQVF2Tg9cQy5s+DZDNj7i9RtQ9OWlv/7oxp0wYcMCct48ocxq7f3tfarsk2sYYCXdUcQHICfu6rDTTPUDJUg/yiwp9XuPFWcC7O5D9vSBIHF22cdEektFdpVy6dTUlwMHf3qLPb5KWb3DgTCglKlvUGgc0fJe4CDCHw/WZGVLHfwEYc3rmH1L8xVfGFGX3Ol41Z+3mziZZchzowPZWCSBpLTH4bwhNxVJxAMTdgTLg6mVLtSwWVP08XGKBkLrgu4hMx7YdClcKR2ne1w8GHON+M/ywsc4w473MYpuYtih2Dw1SeRK1cU/Nu4/YJQS+ZiBMokAwUeSF7MqstE5+DJ78+ZfcOYsQ/01jJamjS7kfxB42d/RnX9EhIy3IE8fdmRz/Z337zSU0OQc LSJ84v/P gaV3+6A9AInjLDg0ulGEj/bgUNRcJCSjC322dg9oead+yJdlwQbu5+9r9WIYvraobJfv/ePb+7Bqa3xZhfs4/CPFkArM9mkZZQdKATb8IehJr1+mnJ/VwptNDkw9spKRC2ntlYfeJtf3+gg+soBTDh6q6FM/XwqjOeh6Ka4jOUFGLH0tCcB4iu+LGkgFl+6IGWyCYfRcN7UAO5iSRpa1uY7YyZ7XBB2s08dMKeEVlPx5EwkQGJIK0cGhJuM29hYVy3boEkpZ7ZY+iFGTlAxvDHiJgIZGzvBxHv8yErN5lsSgl5blNWGjaxg+zKGM2zQBtxmAEtrBnTk+ZVbPJXpEC4zyEOnD6IdSLw0yiPszjPkpO8oYadjUZ525QaQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's abstract the operations so we can extend these operations easily. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 39 +++++++++++++++++++++++++++++++++++---- mm/rmap.c | 14 ++++++-------- 2 files changed, 41 insertions(+), 12 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 9e275986f0ef6..e3b82a04b4acb 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -173,6 +173,37 @@ static inline void anon_vma_merge(struct vm_area_struct *vma, struct anon_vma *folio_get_anon_vma(struct folio *folio); +static inline void folio_set_large_mapcount(struct folio *folio, int mapcount, + struct vm_area_struct *vma) +{ + /* Note: mapcounts start at -1. */ + atomic_set(&folio->_large_mapcount, mapcount - 1); +} + +static inline void folio_add_large_mapcount(struct folio *folio, + int diff, struct vm_area_struct *vma) +{ + atomic_add(diff, &folio->_large_mapcount); +} + +static inline void folio_inc_large_mapcount(struct folio *folio, + struct vm_area_struct *vma) +{ + atomic_inc(&folio->_large_mapcount); +} + +static inline void folio_sub_large_mapcount(struct folio *folio, + int diff, struct vm_area_struct *vma) +{ + atomic_sub(diff, &folio->_large_mapcount); +} + +static inline void folio_dec_large_mapcount(struct folio *folio, + struct vm_area_struct *vma) +{ + atomic_dec(&folio->_large_mapcount); +} + /* RMAP flags, currently only relevant for some anon rmap operations. */ typedef int __bitwise rmap_t; @@ -339,11 +370,11 @@ static __always_inline void __folio_dup_file_rmap(struct folio *folio, do { atomic_inc(&page->_mapcount); } while (page++, --nr_pages > 0); - atomic_add(orig_nr_pages, &folio->_large_mapcount); + folio_add_large_mapcount(folio, orig_nr_pages, dst_vma); break; case RMAP_LEVEL_PMD: atomic_inc(&folio->_entire_mapcount); - atomic_inc(&folio->_large_mapcount); + folio_inc_large_mapcount(folio, dst_vma); break; } } @@ -437,7 +468,7 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, ClearPageAnonExclusive(page); atomic_inc(&page->_mapcount); } while (page++, --nr_pages > 0); - atomic_add(orig_nr_pages, &folio->_large_mapcount); + folio_add_large_mapcount(folio, orig_nr_pages, dst_vma); break; case RMAP_LEVEL_PMD: if (PageAnonExclusive(page)) { @@ -446,7 +477,7 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, ClearPageAnonExclusive(page); } atomic_inc(&folio->_entire_mapcount); - atomic_inc(&folio->_large_mapcount); + folio_inc_large_mapcount(folio, dst_vma); break; } return 0; diff --git a/mm/rmap.c b/mm/rmap.c index ee1bff1638f90..226b188499f91 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1177,7 +1177,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, atomic_add_return_relaxed(first, mapped) < ENTIRELY_MAPPED) nr = first; - atomic_add(orig_nr_pages, &folio->_large_mapcount); + folio_add_large_mapcount(folio, orig_nr_pages, vma); break; case RMAP_LEVEL_PMD: first = atomic_inc_and_test(&folio->_entire_mapcount); @@ -1194,7 +1194,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, nr = 0; } } - atomic_inc(&folio->_large_mapcount); + folio_inc_large_mapcount(folio, vma); break; } return nr; @@ -1450,15 +1450,13 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, SetPageAnonExclusive(page); } - /* increment count (starts at -1) */ - atomic_set(&folio->_large_mapcount, nr - 1); + folio_set_large_mapcount(folio, nr, vma); atomic_set(&folio->_nr_pages_mapped, nr); } else { nr = folio_large_nr_pages(folio); /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); - /* increment count (starts at -1) */ - atomic_set(&folio->_large_mapcount, 0); + folio_set_large_mapcount(folio, 1, vma); atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED); if (exclusive) SetPageAnonExclusive(&folio->page); @@ -1542,7 +1540,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, break; } - atomic_sub(nr_pages, &folio->_large_mapcount); + folio_sub_large_mapcount(folio, nr_pages, vma); do { last += atomic_add_negative(-1, &page->_mapcount); } while (page++, --nr_pages > 0); @@ -1554,7 +1552,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, partially_mapped = nr && atomic_read(mapped); break; case RMAP_LEVEL_PMD: - atomic_dec(&folio->_large_mapcount); + folio_dec_large_mapcount(folio, vma); last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { nr = atomic_sub_return_relaxed(ENTIRELY_MAPPED, mapped);