From patchwork Fri Nov 24 13:26:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13467670 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7753FC624B4 for ; Fri, 24 Nov 2023 13:27:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 32CF08D006E; Fri, 24 Nov 2023 08:27:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AF348D0086; Fri, 24 Nov 2023 08:27:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F334E8D006E; Fri, 24 Nov 2023 08:27:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D83328D0084 for ; Fri, 24 Nov 2023 08:27:39 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 6DC80A062A for ; Fri, 24 Nov 2023 13:27:39 +0000 (UTC) X-FDA: 81492925038.21.93A5E7C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 81F3240025 for ; Fri, 24 Nov 2023 13:27:37 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=edevLR9P; spf=pass (imf01.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700832457; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Cfr/GJlqpI78eOuvYmduIMFro8OwYW0A0XSs9rIUuNI=; b=hq5LJjN9ORvhW3q5HWSCmmBF1RSAM8D6514iXcpDoBghZJ8TH1uJ77KVeoESpgyXEQ6XLq yvG7aPDOJEsS6yA3aj5+RQz7ztBzJhexJH7+T3z+hjSzf8Ka98d2mwLCeXMiHwWLGJSW6c H/RuQokNJgsDQ5mMYFiHPDU4Hrcqpuw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700832457; a=rsa-sha256; cv=none; b=rRwe1OZPa2XKFPxeX7bJ+2Q11WolCnQ4v/gAuf6My4Jj3k3c6wnhbRqCdHxAKrNahV/FP8 rkrZs03SE5HmiZFPOzc2HIqByecx1lMCMHPVAbIZJoFfDOctpfwuF9o7r/YSlnBKrVWiOq ciCsvn2el85/52ojlbkxetoeNP6tRw0= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=edevLR9P; spf=pass (imf01.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700832456; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cfr/GJlqpI78eOuvYmduIMFro8OwYW0A0XSs9rIUuNI=; b=edevLR9PUoRVCb52wiTofr7qqMmyXIV75KUhrV3YiriecMiJ3gMc+hKne6zkLNnvIi9aqO TXZaOsulLBX5Mc3C70pZ09EBJo861JVGyTWG3TfdZs394IxdVO23hMnMIoNQymvpyJU0f4 8T/rW2eaSTwJDZDNi/4DIfoa9HM0IHU= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-509-8WRNY9xWPlSFJQ6I882kqQ-1; Fri, 24 Nov 2023 08:27:33 -0500 X-MC-Unique: 8WRNY9xWPlSFJQ6I882kqQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 28DBB1C04357; Fri, 24 Nov 2023 13:27:33 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 953662166B2A; Fri, 24 Nov 2023 13:27:29 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Linus Torvalds , Ryan Roberts , Matthew Wilcox , Hugh Dickins , Yin Fengwei , Yang Shi , Ying Huang , Zi Yan , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , "Paul E. McKenney" Subject: [PATCH WIP v1 17/20] mm/rmap_id: reduce atomic RMW operations when we are the exclusive writer Date: Fri, 24 Nov 2023 14:26:22 +0100 Message-ID: <20231124132626.235350-18-david@redhat.com> In-Reply-To: <20231124132626.235350-1-david@redhat.com> References: <20231124132626.235350-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.6 X-Stat-Signature: na9be34g8y1rxn98i3qrobmsod69nq35 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 81F3240025 X-Rspam-User: X-HE-Tag: 1700832457-147066 X-HE-Meta: U2FsdGVkX19YouEND1wCuqDvam3LhcI8MAUGCaBjLGblsd2e7Shl6aqxlVD9XbSJnFdHpMFqgKxE6ds+ebwH3shktuww0tytU9iKgNnejgTNWcFtRMXRvxMbitn0IFh9olzyXGKg5AS8UXBnXcp94kDuoMs/BANPZTvT4rkFEO6gn4rAouUqxmS3FIa8hnXf5gwlKFAxdDW6bv1aQwi5rSfUSNg4qLypn5/++bDbi1r4EAwDMn7mPgMRsxkjoLtIyQHNWBldgL3Si635NuSeoDbmheMPQ3bgwVkTjp0NYhI/UsQRrbWVE5MWlwGSvQHhou/3pG6GCk5fwze20+CrstjQoRshgTFRs+SE+sU2eFyh8eVFDYVMmrXMsftEGoES2pETeB/2WZsl2/Vrl+NttALG5R3tfqF8VgKtJPQgjT9SNHcfcUPum1E8ZUnqhBqGrNyfMShDqJu2zjqXWBf4RSnLJQmhAmTthe7QertZ3tKY9Q6ceL46HNP6eh8uwWHvfDMMRwtqD7mFr5QvDz/vIxK/RVJFsrOCE3MUNkfh/XbYY+yLrTNJKSkTGoNXwKNw43nYR7/JYVq+pe4gxYhY/DyP7TLJaUpmkzF5Vk4DcgGpnKFZI1DuhtoY5HgwAvWijwP9RnWnprKF2xfpku2WwVzgAmuOGJ6zc+Jlps6df4mE9Wx8E2SBd5IpTpDO2mJLX4Ba1D3U8aIh2l2P4Vyahgoomyi7NEwLHpW4DcHQcfJ2ycK62HQkA3saWyz3EAPml6gajt+xb4VCQdKBKsYoYOYGcLKqN6i97CAOwaM371RgcK+SNrEXpew+uWBdX4zA2fcFVdVpjCBuBgqM+rhwQOPyu/ncPAFXN87XSCVDsqGitaNeV4cCDTnj762lkLZ/xeNBN0arCIrxb7YyDNayBk61wqgMLLWhUvlRQge5U43JAzg7TEs4aiBrOm7qghL7O/tnip4S3Cf+o7ts6/V WaFl9Rm8 WlBZlQ/Cgu5On/ByoFnm7nSSZz+2nP7yicCJgWG+aF9gEHO5W6lQr0utjnJqKYrSbUzKedpdy8WDq10HhYmxOgJ6g9LPDYkFSmWOV1lcs71paq6fdCIB4iACLgWYi2L8aLJ0Y3EW6vSzi79Qq0f66mPyNRwyS670n2IETp/ssnJR2fGlVWWlVRhv9LQ3qYIaO840do86Ex5wlexRl2lWgk4O7FvkdO2M/mA8r X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We can reduce the number of atomic RMW operations when we are the single exclusive writer -- the common case. So instead of always requiring (1) 2 atomic RMW operations for adjusting the atomic seqcount (2) 1 atomic RMW operation for adjusting the total mapcount (3) 1 to 6 atomic RMW operation for adjusting the rmap values We can avoid (2) and (3) if we are the exclusive writer and limit it to the 2 atomic RMW operations from (1). Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 81 +++++++++++++++++++++++++++++++++----------- mm/rmap_id.c | 52 ++++++++++++++++++++++++++++ 2 files changed, 114 insertions(+), 19 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0758dddc5528..538c23d3c0c9 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -291,23 +291,36 @@ static inline void __folio_undo_large_rmap(struct folio *folio) #endif } -static inline void __folio_write_large_rmap_begin(struct folio *folio) +static inline bool __folio_write_large_rmap_begin(struct folio *folio) { + bool exclusive; + VM_WARN_ON_FOLIO(!folio_test_large_rmappable(folio), folio); VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); - raw_write_atomic_seqcount_begin(&folio->_rmap_atomic_seqcount, - false); + + exclusive = raw_write_atomic_seqcount_begin(&folio->_rmap_atomic_seqcount, + true); + if (likely(exclusive)) { + prefetchw(&folio->_rmap_val0); + if (unlikely(folio_order(folio) > RMAP_SUBID_4_MAX_ORDER)) + prefetchw(&folio->_rmap_val4); + } + return exclusive; } -static inline void __folio_write_large_rmap_end(struct folio *folio) +static inline void __folio_write_large_rmap_end(struct folio *folio, + bool exclusive) { - raw_write_atomic_seqcount_end(&folio->_rmap_atomic_seqcount, false); + raw_write_atomic_seqcount_end(&folio->_rmap_atomic_seqcount, + exclusive); } void __folio_set_large_rmap_val(struct folio *folio, int count, struct mm_struct *mm); void __folio_add_large_rmap_val(struct folio *folio, int count, struct mm_struct *mm); +void __folio_add_large_rmap_val_exclusive(struct folio *folio, int count, + struct mm_struct *mm); bool __folio_has_large_matching_rmap_val(struct folio *folio, int count, struct mm_struct *mm); #else @@ -317,12 +330,14 @@ static inline void __folio_prep_large_rmap(struct folio *folio) static inline void __folio_undo_large_rmap(struct folio *folio) { } -static inline void __folio_write_large_rmap_begin(struct folio *folio) +static inline bool __folio_write_large_rmap_begin(struct folio *folio) { VM_WARN_ON_FOLIO(!folio_test_large_rmappable(folio), folio); VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); + return false; } -static inline void __folio_write_large_rmap_end(struct folio *folio) +static inline void __folio_write_large_rmap_end(struct folio *folio, + bool exclusive) { } static inline void __folio_set_large_rmap_val(struct folio *folio, int count, @@ -333,6 +348,10 @@ static inline void __folio_add_large_rmap_val(struct folio *folio, int count, struct mm_struct *mm) { } +static inline void __folio_add_large_rmap_val_exclusive(struct folio *folio, + int count, struct mm_struct *mm) +{ +} #endif /* CONFIG_RMAP_ID */ static inline void folio_set_large_mapcount(struct folio *folio, @@ -348,28 +367,52 @@ static inline void folio_set_large_mapcount(struct folio *folio, static inline void folio_inc_large_mapcount(struct folio *folio, struct vm_area_struct *vma) { - __folio_write_large_rmap_begin(folio); - atomic_inc(&folio->_total_mapcount); - __folio_add_large_rmap_val(folio, 1, vma->vm_mm); - __folio_write_large_rmap_end(folio); + bool exclusive; + + exclusive = __folio_write_large_rmap_begin(folio); + if (likely(exclusive)) { + atomic_set(&folio->_total_mapcount, + atomic_read(&folio->_total_mapcount) + 1); + __folio_add_large_rmap_val_exclusive(folio, 1, vma->vm_mm); + } else { + atomic_inc(&folio->_total_mapcount); + __folio_add_large_rmap_val(folio, 1, vma->vm_mm); + } + __folio_write_large_rmap_end(folio, exclusive); } static inline void folio_add_large_mapcount(struct folio *folio, int count, struct vm_area_struct *vma) { - __folio_write_large_rmap_begin(folio); - atomic_add(count, &folio->_total_mapcount); - __folio_add_large_rmap_val(folio, count, vma->vm_mm); - __folio_write_large_rmap_end(folio); + bool exclusive; + + exclusive = __folio_write_large_rmap_begin(folio); + if (likely(exclusive)) { + atomic_set(&folio->_total_mapcount, + atomic_read(&folio->_total_mapcount) + count); + __folio_add_large_rmap_val_exclusive(folio, count, vma->vm_mm); + } else { + atomic_add(count, &folio->_total_mapcount); + __folio_add_large_rmap_val(folio, count, vma->vm_mm); + } + __folio_write_large_rmap_end(folio, exclusive); } static inline void folio_dec_large_mapcount(struct folio *folio, struct vm_area_struct *vma) { - __folio_write_large_rmap_begin(folio); - atomic_dec(&folio->_total_mapcount); - __folio_add_large_rmap_val(folio, -1, vma->vm_mm); - __folio_write_large_rmap_end(folio); + bool exclusive; + + exclusive = __folio_write_large_rmap_begin(folio); + if (likely(exclusive)) { + atomic_set(&folio->_total_mapcount, + atomic_read(&folio->_total_mapcount) - 1); + __folio_add_large_rmap_val_exclusive(folio, -1, vma->vm_mm); + } else { + atomic_dec(&folio->_total_mapcount); + __folio_add_large_rmap_val(folio, -1, vma->vm_mm); + } + __folio_write_large_rmap_end(folio, exclusive); } /* RMAP flags, currently only relevant for some anon rmap operations. */ diff --git a/mm/rmap_id.c b/mm/rmap_id.c index 421d8d2b646c..5009c6e43965 100644 --- a/mm/rmap_id.c +++ b/mm/rmap_id.c @@ -379,6 +379,58 @@ void __folio_add_large_rmap_val(struct folio *folio, int count, } } +void __folio_add_large_rmap_val_exclusive(struct folio *folio, int count, + struct mm_struct *mm) +{ + const unsigned int order = folio_order(folio); + + /* + * Concurrent rmap value modifications are impossible. We don't care + * about store tearing because readers will realize the concurrent + * updates using the seqcount and simply retry. So adjust the bare + * atomic counter instead. + */ + switch (order) { +#if MAX_ORDER >= RMAP_SUBID_6_MIN_ORDER + case RMAP_SUBID_6_MIN_ORDER ... RMAP_SUBID_6_MAX_ORDER: + folio->_rmap_val0.counter += get_rmap_subid_6(mm, 0) * count; + folio->_rmap_val1.counter += get_rmap_subid_6(mm, 1) * count; + folio->_rmap_val2.counter += get_rmap_subid_6(mm, 2) * count; + folio->_rmap_val3.counter += get_rmap_subid_6(mm, 3) * count; + folio->_rmap_val4.counter += get_rmap_subid_6(mm, 4) * count; + folio->_rmap_val5.counter += get_rmap_subid_6(mm, 5) * count; + break; +#endif +#if MAX_ORDER >= RMAP_SUBID_5_MIN_ORDER + case RMAP_SUBID_5_MIN_ORDER ... RMAP_SUBID_5_MAX_ORDER: + folio->_rmap_val0.counter += get_rmap_subid_5(mm, 0) * count; + folio->_rmap_val1.counter += get_rmap_subid_5(mm, 1) * count; + folio->_rmap_val2.counter += get_rmap_subid_5(mm, 2) * count; + folio->_rmap_val3.counter += get_rmap_subid_5(mm, 3) * count; + folio->_rmap_val4.counter += get_rmap_subid_5(mm, 4) * count; + break; +#endif + case RMAP_SUBID_4_MIN_ORDER ... RMAP_SUBID_4_MAX_ORDER: + folio->_rmap_val0.counter += get_rmap_subid_4(mm, 0) * count; + folio->_rmap_val1.counter += get_rmap_subid_4(mm, 1) * count; + folio->_rmap_val2.counter += get_rmap_subid_4(mm, 2) * count; + folio->_rmap_val3.counter += get_rmap_subid_4(mm, 3) * count; + break; + case RMAP_SUBID_3_MIN_ORDER ... RMAP_SUBID_3_MAX_ORDER: + folio->_rmap_val0.counter += get_rmap_subid_3(mm, 0) * count; + folio->_rmap_val1.counter += get_rmap_subid_3(mm, 1) * count; + folio->_rmap_val2.counter += get_rmap_subid_3(mm, 2) * count; + break; + case RMAP_SUBID_2_MIN_ORDER ... RMAP_SUBID_2_MAX_ORDER: + folio->_rmap_val0.counter += get_rmap_subid_2(mm, 0) * count; + folio->_rmap_val1.counter += get_rmap_subid_2(mm, 1) * count; + break; + default: + folio->_rmap_val0.counter += get_rmap_subid_1(mm); + break; + } +} + bool __folio_has_large_matching_rmap_val(struct folio *folio, int count, struct mm_struct *mm) {