From patchwork Tue Apr 9 19:22:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13623120 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FF2BC67861 for ; Tue, 9 Apr 2024 19:24:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 232A16B0098; Tue, 9 Apr 2024 15:24:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1BB5B6B0099; Tue, 9 Apr 2024 15:24:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00DD86B009A; Tue, 9 Apr 2024 15:24:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D66416B0098 for ; Tue, 9 Apr 2024 15:24:16 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7501DA0DCB for ; Tue, 9 Apr 2024 19:24:16 +0000 (UTC) X-FDA: 81990969312.20.F9D333D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 4C51C1A0016 for ; Tue, 9 Apr 2024 19:24:14 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cR4Nt0zU; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712690654; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DanfdMkK2y8X/5UQWi5wxydKuxMzBBZcoNekt2NzNQo=; b=q95xjLDeYSeNlqDv4IPkVa6lFo7YwEyGmQ0RBPIRiNyjXV7iJWb8j0x6hj903sAWgP4Abj fL9UQZOQefoMv/b06IAfkiSzzxxSWA6pt+51uRLqY0QLPpPBl7tH9D5CbDmD/CKpJvpRqK iu9vsEb6cdN8rGFihuxRur1lQsR9TD0= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cR4Nt0zU; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712690654; a=rsa-sha256; cv=none; b=1wzlPXHLq0KxAN8J//64JHguIpwG78RERgKIhDy68xqZk8uXv9irijcX0bRRiDq6SNvvpM tRNArd99wsZhPK7BD/n+sLTIwnnvrwOuIgGzvCNoTrRWJlToK2vtrQKTws2R2NfTMzYlgv 2zgxCQHxBdMItbU6BtEfQLABcOgzUYc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1712690653; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DanfdMkK2y8X/5UQWi5wxydKuxMzBBZcoNekt2NzNQo=; b=cR4Nt0zU9mOStfjO1xhb+ffD7bEhX68DET43WdhVOV0xItKGkNnoQ533mXO9Gpr1Dcx+L0 8fIIxmELZvhu7G+rYd4uAqyyDrWRb+2iecM8ZKXPLKHNa0v6jCUDmZqSh1KJjyDiO0wK45 2NR/lFcnVW2Bzb02W6IzUcXRHZHBpm0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-140-eJSj8LVjMUGt1dEE8jgGSA-1; Tue, 09 Apr 2024 15:24:08 -0400 X-MC-Unique: eJSj8LVjMUGt1dEE8jgGSA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E743E806604; Tue, 9 Apr 2024 19:24:06 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.106]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3CAAC40B4982; Tue, 9 Apr 2024 19:24:00 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, linux-doc@vger.kernel.org, cgroups@vger.kernel.org, linux-sh@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Peter Xu , Ryan Roberts , Yin Fengwei , Yang Shi , Zi Yan , Jonathan Corbet , Hugh Dickins , Yoshinori Sato , Rich Felker , John Paul Adrian Glaubitz , Chris Zankel , Max Filippov , Muchun Song , Miaohe Lin , Naoya Horiguchi , Richard Chang Subject: [PATCH v1 03/18] mm/rmap: add fast-path for small folios when adding/removing/duplicating Date: Tue, 9 Apr 2024 21:22:46 +0200 Message-ID: <20240409192301.907377-4-david@redhat.com> In-Reply-To: <20240409192301.907377-1-david@redhat.com> References: <20240409192301.907377-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: 4C51C1A0016 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 93qdyq3q7gemt3wmkcytmozquj5mt8e9 X-HE-Tag: 1712690654-824703 X-HE-Meta: U2FsdGVkX18t9n19zhFccy6ih/2H0x/I3e2Mjfu1z0khI2KTruzISXFbYGMCrxOvlVEVSuSF6jHEQhvUWf+9TbBM4a2iVu8Goe4RdQtsb9ZCTxOwNn4x3DJZnO+ED7F2+H2aq9T/zxStvVMah7+wiWo+sStFBhDQOxKSdK3aY21opzgBi+VOPo7dzQzfSKTewX0Y7sKujbDrzcrF+QPRdEOLvqDutccl6cu9GT8sopEjPQGwmLX1NAdwXK0jJB43ub3TWxidFR8qLKB/pBm847I4w0BctrvymV4uBB4c6WCWBariqUfABQloBHK/dEIupr4c7WCYluy9Gvta8s8b2Md8eFRfR2uLvrWd9EXqDlSaPJS8wpEUcNGbwdDI9qDCw24U7DQ/br7NuZXqVi0WMJQbxilSY5l2N0SXQwsUDWZ2ape5XfwGFbwHdhBUwsNpWhTYqt3LzeILI1mhMaFSKSatxEzpz1vL94CT0JyZrLPh4ALGUX6A/OWw/Lnf+aWEf5zTVhAmKuYt+WYaWEssslLMHSBbEbVmRKl+mbeFjeHvaQRcEG+H+fpWB7IHRTJkSOYlfLVo+TDNvTLY8cO+BPnP/q1r2iLVRgsoaijUUnDFVmXdRvi50jOvkFdpq8Cug23QuJyEM2cbpHgUn1AbwkeKHHghhJ9MdCN60Ri4RmAOOkYpndVqdUFQfyN18oSHMHwVtIUlICvyVCqUd8eNjE2I7TKzXGv8NE0tg+5IO+2hh3WoaQLh+DQv3gUf83zryMC6hpCwtlfGZGnVbmtnA3jx04bN0XJ2+ib7Q1+pT2U3s5FfuVw5DfgnrENJtZjuIXeD9TC9tXiAgP399t2v6f47/nXuxmWc7EvNl4c3TtrJhuZXADJpKatOFNp3I2YY50z6TwMvywMsQ+AM7EriRZUsAnLFEp/1WKQLR5Yh2cz4TPdBHNRgrFiemgjyUcXNEGgXcy7T26Cvz9q6yVz 7FFyFesy 0Baz0LaD9t6h/kq+3n80qlI0YRLobrlrkHO1pLH8PVFu9STYafrCgtCpudY3VZjWclohsMa1w2Hzkhk039wYGbGeBx94gHhW8j3gv0rPtaTWxoeK1IXD8+H0SVjGLmJ+IaTteepo1dWb3qy9yhXSeF03sZa2P5YqbgM75DtG8bxuzld4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's add a fast-path for small folios to all relevant rmap functions. Note that only RMAP_LEVEL_PTE applies. This is a preparation for tracking the mapcount of large folios in a single value. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- include/linux/rmap.h | 13 +++++++++++++ mm/rmap.c | 26 ++++++++++++++++---------- 2 files changed, 29 insertions(+), 10 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 9549d78928bb..327f1ca5a487 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -322,6 +322,11 @@ static __always_inline void __folio_dup_file_rmap(struct folio *folio, switch (level) { case RMAP_LEVEL_PTE: + if (!folio_test_large(folio)) { + atomic_inc(&page->_mapcount); + break; + } + do { atomic_inc(&page->_mapcount); } while (page++, --nr_pages > 0); @@ -405,6 +410,14 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, if (PageAnonExclusive(page + i)) return -EBUSY; } + + if (!folio_test_large(folio)) { + if (PageAnonExclusive(page)) + ClearPageAnonExclusive(page); + atomic_inc(&page->_mapcount); + break; + } + do { if (PageAnonExclusive(page)) ClearPageAnonExclusive(page); diff --git a/mm/rmap.c b/mm/rmap.c index 56b313aa2ebf..4bde6d60db6c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1172,15 +1172,18 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, switch (level) { case RMAP_LEVEL_PTE: + if (!folio_test_large(folio)) { + nr = atomic_inc_and_test(&page->_mapcount); + break; + } + do { first = atomic_inc_and_test(&page->_mapcount); - if (first && folio_test_large(folio)) { + if (first) { first = atomic_inc_return_relaxed(mapped); - first = (first < ENTIRELY_MAPPED); + if (first < ENTIRELY_MAPPED) + nr++; } - - if (first) - nr++; } while (page++, --nr_pages > 0); break; case RMAP_LEVEL_PMD: @@ -1514,15 +1517,18 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, switch (level) { case RMAP_LEVEL_PTE: + if (!folio_test_large(folio)) { + nr = atomic_add_negative(-1, &page->_mapcount); + break; + } + do { last = atomic_add_negative(-1, &page->_mapcount); - if (last && folio_test_large(folio)) { + if (last) { last = atomic_dec_return_relaxed(mapped); - last = (last < ENTIRELY_MAPPED); + if (last < ENTIRELY_MAPPED) + nr++; } - - if (last) - nr++; } while (page++, --nr_pages > 0); break; case RMAP_LEVEL_PMD: