From patchwork Thu Feb 13 22:46:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974128 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37E24C021A0 for ; Thu, 13 Feb 2025 22:47:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 82094280002; Thu, 13 Feb 2025 17:47:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A9A9280001; Thu, 13 Feb 2025 17:47:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64A28280002; Thu, 13 Feb 2025 17:47:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 441B3280001 for ; Thu, 13 Feb 2025 17:47:02 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id EE94412184E for ; Thu, 13 Feb 2025 22:47:01 +0000 (UTC) X-FDA: 83116408242.12.6713050 Received: from mail-oa1-f73.google.com (mail-oa1-f73.google.com [209.85.160.73]) by imf25.hostedemail.com (Postfix) with ESMTP id 3ED6CA0007 for ; Thu, 13 Feb 2025 22:47:00 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="HU/nD9lf"; spf=pass (imf25.hostedemail.com: domain of 3Y3auZwYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com designates 209.85.160.73 as permitted sender) smtp.mailfrom=3Y3auZwYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486820; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=27VMvYX+XeIZWrw3x5wMgd7xnhB8ftl1Uxw2ee/Q3DE=; b=VHppRgDrRs75LVzj4nNN5L5nsmEIh3FL9sjAXupdzuzqCjpoqTGdfDAVb1kzRhWWD9OTy0 IeQ7npge7qDL4T7uy84ru5l9p9L7Hcv5TfQHk+NRZztDl6HvSTVDT9tF7QTKSpwnRv5g0y dzP0Ax5HzjeIiyqoJe1fzTJGdda7PXo= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="HU/nD9lf"; spf=pass (imf25.hostedemail.com: domain of 3Y3auZwYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com designates 209.85.160.73 as permitted sender) smtp.mailfrom=3Y3auZwYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486820; a=rsa-sha256; cv=none; b=Xen0A4+2SoCX38IlAuTyWCv17zwh6K9ImamIODPWTPWcQpIEqCdNNI75FNaqMaHRpIvp5Y FQHKj1GGig1qSs6PyaoL5HArhBiX/adg66RvUp5tBvUEnHeHeawdkRCNpN5ed0mi5jbaTe dbAsIkr42opMBWcUXdul1qT9SODtPBA= Received: by mail-oa1-f73.google.com with SMTP id 586e51a60fabf-2bc6cb1a560so670389fac.0 for ; Thu, 13 Feb 2025 14:47:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486819; x=1740091619; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=27VMvYX+XeIZWrw3x5wMgd7xnhB8ftl1Uxw2ee/Q3DE=; b=HU/nD9lfet8oudkhHt7GsweShppuUGE3hBV0Pvk1FlsKwcMkkp5wg+BQJulxSdvgPF 98IeUvQndvVZUugvfoTayl0rGsVxFfeMtgmgOxDgWO/zMDM1jj07FId0Rf1nQ9Vkbno6 LbWBz8vMLo+5dJhoe4lc3Hde/g9cOvz8mrCHXgrQI5z0frw0copNAtylKMC+VF1cU7/n LAxuN5HHOJdUuvcatMlOPgbMpDEsTQR+ztrscL5lqa39UwV7UpkmLwHO1hhoFYRJ36n3 1hbT+ZW8fkazbWuSL+YcguEk37qryUe/y5emuq+9rttJ//NYagla5gOtsSUosAvQ1SB6 vqbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486819; x=1740091619; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=27VMvYX+XeIZWrw3x5wMgd7xnhB8ftl1Uxw2ee/Q3DE=; b=wnqSwJtcICd3giIlUpJKyhn735kZ8FiKTx0nGTqawB/IOJkh8zRu3K/KMkNW3fNIEb whAvYA4ORRc7FXRjWUSP9GNm3I4hEHpX6g3NUQ8z2H8L9Ey8aH6/mKxqZo50QF7UL+Vc 74WXaiEikwv3L/fDugTXWLLvVbr+2QJvqblmP79s3Yp+3SnPf7zAECuyXRLdhBIS7ldx 9jctMH/yqsfRmj/VY74j9tLCUGc7WyZaZNgV7JEuS2OK/CAWNXUzTbvhICQtgI9CoLoS /JwJ2X+X3doqAuqClP37ZORRMOzVYAip6tO1DfdidusaYvxxHRSK2VTi3L/tnikFypNi gGGw== X-Forwarded-Encrypted: i=1; AJvYcCVNhbXE3rEROW0vN2ho8MlTUTgcfrKymtcPdzOLS/NYL6Amrj+pdnSFqihtxguHVlGYVr3+D4JDXA==@kvack.org X-Gm-Message-State: AOJu0YxFXGDh/vy+OKKoFyfaRYnLTTlW30zXP1wYxhee7RYvUgtKaguI TZDbcIXzfDgYQX1NHseFBYfloJIqu465V206ZPHNRauxh5LcLJiQjwSmWwcnCCX5mHEFd0bMfje JuA== X-Google-Smtp-Source: AGHT+IFqU1mAmYOjarqaEBcrctfJ39DVCGZvcnhZHusudh7jNm8NK23hT9sU/SxfoGJNKCasg65TO0nhtlQ= X-Received: from oabpu8.prod.google.com ([2002:a05:6870:9e88:b0:294:c73f:1037]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6871:67c2:b0:2bc:7c3d:18d8 with SMTP id 586e51a60fabf-2bc7c3d685cmr120712fac.33.1739486819281; Thu, 13 Feb 2025 14:46:59 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:37 -0800 Mime-Version: 1.0 X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-1-surenb@google.com> Subject: [PATCH v10 00/18] reimplement per-vma lock as a refcount From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3ED6CA0007 X-Stat-Signature: cjmw74rkztjog33j5y3qh37917e7sbjr X-Rspam-User: X-HE-Tag: 1739486820-759724 X-HE-Meta: U2FsdGVkX1+g74z0Q1pY/EwGt8eHtzAiAvYXKYtJIjps+EyFOmWnnvZeir47/VZvomkTx0B1ze3NYZ0osYIvd1lo/ZCpi+sEOu+0C5eYfNY8wHIYIe2/7GIBDmDyR+7pbzb9iyVviL0xH+s28/TWmd/qChQFyj1SbHO1ffA+BzgjkYX8DdB2EOOJUULA4ZmzOS2m4CejiuNX/F+ii1AigTWkanAtK/nSBrP4t2W2pchqiZUomK2rKZ/OfuFUYwo8tVSE41pirvmANovUeO+jKtaya9pboZd95RAGvx5VfYXzjuZHnWx24UzwMYUQBEMlflgcH1Ic9WQx544U7wu2E8+FVUDFQep7ef5n92U+LnD2LHVEXPRXlzYhRQ/mKyO3dcGF7gitOdfGBToALa0TEDpOnu4GK4S0rb6mlWK/J7bGM21TPbibMpg6PgkePp0N9+IL7jr4ojHAr5qitMawkXn+DV09vFV2IsYGr9uIfAncfBOvXRbcgk5luGgPwBHluQflFx0tlbSIYe4yn0W4qITRvDDkF456ncgpyyBbrUzy4z/XxCNMAfieGNezS5CwpRYKQ+U8Ea+xF770ohmQ2rNoBGMUo5Cav62liN84fnD7gggeG6MtIk5M3BoWEQvbYbKq+q6DIBS4dGrOiIKq0QORiUGUHyt5YepSqHLMkc8Bxvl3VDD3RVC2R5QKgxHlFGPOHR6yMGo+YDtq+GW/tTh1Q7VTD7HJJFjxIkRJZYKR0UzJr/34KK8eQXaU7Z7Oky3kMV4+idKyruI0xGGQf3sPv5+vSsbCTPdiQxtqwWFAqAuZiF/foUGYq8rC4AzDg6Wg/uJ13Q1f7iruG7pNghw+d500TldOSxiYcns0tU9hRbezYc7dUntAMEd9iD5R84ayOrhw25RKzqwJQwvya5Y/wB009PCuyxh0C0pKXGIXZ8/4PClB+BK3s9vgHuWkXDSKURIcgnP2rIVqKn7 KAfduSAl iqU6rhuRWlHJlf++xjUPkHfG+5r5PpGYru4wMLH13Jp3PZNjU0RzupfTRQ+QOM2VXNRQbkOdBdb3guzDqFtmecIYCWKYk5LGYArl/KxxAHj+OKNE1DAsnnA02sl3FivXV/ilrg9xUxc4nqE4UiuTW4XinpbUN6QYtOPUGa8AE0xSbkR9B6OXblHDFCJMzZSFk1j5Gb7jkxUPNLXmiCeaczjka7w3KXfCtfDr/D4SC0zjXZb6604v4fWqXnZ4SVtLKes5XXrKro+hMAvmOVQtvWwTBR2BtcmpdUwZ5X5aj45zzXK01utdKyzbJl3uYnSqX7+Ha5XJsIGWFT+PDGrk4UtN9Pfxqmu655RgvGpGiEc+hAZeRF6M6WZCxpDhVx82H6DgXOajNAylWV520RBSBAcV7FzylsHkiO2rru8badafVJz8iiwfQWmiz1oF2z47mR+Br+LQujokHIdQxVBOMkkTNdIIKhPy2t/UATo9q4SKBF7zoHpYRlnYP7jwiPVVxtiGpI6MxUBw+wyD7Gic5bR9r7uFoDVqEFsN0Iva0nrM8YY2AXWoHVsiv3P1KUb5xEgmfOxOrqzSq3mgKwmCp+HMbSQPX5Ke6COZ1w11tKikCVvgX9lkdjAboF8F7uysTCvSH/5NgRTuBNLnptvJPa/NRKsKlWmOK9Lfv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Back when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3]. Splitting single logical structure into multiple ones leads to more complicated management, extra pointer dereferences and overall less maintainable code. When that split-away part is a lock, it complicates things even further. With no performance benefits, there are no reasons for this split. Merging the vm_lock back into vm_area_struct also allows vm_area_struct to use SLAB_TYPESAFE_BY_RCU later in this patchset. This patchset: 1. moves vm_lock back into vm_area_struct, aligning it at the cacheline boundary and changing the cache to be cacheline-aligned to minimize cacheline sharing; 2. changes vm_area_struct initialization to mark new vma as detached until it is inserted into vma tree; 3. replaces vm_lock and vma->detached flag with a reference counter; 4. regroups vm_area_struct members to fit them into 3 cachelines; 5. changes vm_area_struct cache to SLAB_TYPESAFE_BY_RCU to allow for their reuse and to minimize call_rcu() calls. Pagefault microbenchmarks show performance improvement: Hmean faults/cpu-1 507926.5547 ( 0.00%) 506519.3692 * -0.28%* Hmean faults/cpu-4 479119.7051 ( 0.00%) 481333.6802 * 0.46%* Hmean faults/cpu-7 452880.2961 ( 0.00%) 455845.6211 * 0.65%* Hmean faults/cpu-12 347639.1021 ( 0.00%) 352004.2254 * 1.26%* Hmean faults/cpu-21 200061.2238 ( 0.00%) 229597.0317 * 14.76%* Hmean faults/cpu-30 145251.2001 ( 0.00%) 164202.5067 * 13.05%* Hmean faults/cpu-48 106848.4434 ( 0.00%) 120641.5504 * 12.91%* Hmean faults/cpu-56 92472.3835 ( 0.00%) 103464.7916 * 11.89%* Hmean faults/sec-1 507566.1468 ( 0.00%) 506139.0811 * -0.28%* Hmean faults/sec-4 1880478.2402 ( 0.00%) 1886795.6329 * 0.34%* Hmean faults/sec-7 3106394.3438 ( 0.00%) 3140550.7485 * 1.10%* Hmean faults/sec-12 4061358.4795 ( 0.00%) 4112477.0206 * 1.26%* Hmean faults/sec-21 3988619.1169 ( 0.00%) 4577747.1436 * 14.77%* Hmean faults/sec-30 3909839.5449 ( 0.00%) 4311052.2787 * 10.26%* Hmean faults/sec-48 4761108.4691 ( 0.00%) 5283790.5026 * 10.98%* Hmean faults/sec-56 4885561.4590 ( 0.00%) 5415839.4045 * 10.85%* Changes since v9 [4]: PATCH [4/18] - Change VM_BUG_ON_VMA() to WARN_ON_ONCE() in vma_assert_{attached|detached}, per Lorenzo Stoakes - Rename vma_iter_store() into vma_iter_store_new(), per Lorenzo Stoakes - Expand changelog, per Lorenzo Stoakes - Update vma tests to check for vma detached state correctness, per Lorenzo Stoakes PATCH [5/18] - Add Reviewed-by, per Lorenzo Stoakes PATCH [6/18] - Add Acked-by, per Lorenzo Stoakes PATCH [7/18] - Refactor the code, per Lorenzo Stoakes - Remove Vlastimil's Acked-by since code is changed PATCH [8/18] - Drop inline for mmap_init_lock(), per Lorenzo Stoakes - Add Reviewed-by, per Lorenzo Stoakes PATCH [9/18] - Add Reviewed-by, per Lorenzo Stoakes PATCH [10/18] - New patch to add refcount_add_not_zero_acquire/refcount_set_release - Add Acked-by #slab, per Vlastimil Babka PATCH [11/18] - Change refcount limit to be used with xxx_acquire functions PATCH [12/18] - Use __refcount_inc_not_zero_limited_acquire() in vma_start_read(), per Hillf Danton - Refactor vma_assert_locked() to avoid vm_refcnt read when CONFIG_DEBUG_VM=n, per Mateusz Guzik - Update changelog, per Wei Yang - Change vma_start_read() to return EAGAIN if vma got isolated and changed lock_vma_under_rcu() back to detect this condition, per Wei Yang - Change VM_BUG_ON_VMA() to WARN_ON_ONCE() when checking vma detached state, per Lorenzo Stoakes - Remove Vlastimil's Reviewed-by since code is changed PATCH [13/18] - Update vm_area_struct for tests, per Lorenzo Stoakes - Add Reviewed-by, per Lorenzo Stoakes PATCH [14/18] - Minimized duplicate code, per Lorenzo Stoakes PATCH [15/18] - Add Reviewed-by, per Lorenzo Stoakes PATCH [17/18] - Use refcount_set_release() in vma_mark_attached(), per Will Deacon PATCH [18/18] - Updated documenation, per Lorenzo Stoakes - Add Reviewed-by, per Lorenzo Stoakes [1] https://lore.kernel.org/all/20230227173632.3292573-34-surenb@google.com/ [2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/ [3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/ [4] https://lore.kernel.org/all/20250111042604.3230628-1-surenb@google.com/ Patchset applies over mm-unstable Suren Baghdasaryan (18): mm: introduce vma_start_read_locked{_nested} helpers mm: move per-vma lock into vm_area_struct mm: mark vma as detached until it's added into vma tree mm: introduce vma_iter_store_attached() to use with attached vmas mm: mark vmas detached upon exit types: move struct rcuwait into types.h mm: allow vma_start_read_locked/vma_start_read_locked_nested to fail mm: move mmap_init_lock() out of the header file mm: uninline the main body of vma_start_write() refcount: provide ops for cases when object's memory can be reused refcount: introduce __refcount_{add|inc}_not_zero_limited_acquire mm: replace vm_lock and detached flag with a reference count mm: move lesser used vma_area_struct members into the last cacheline mm/debug: print vm_refcnt state when dumping the vma mm: remove extra vma_numab_state_init() call mm: prepare lock_vma_under_rcu() for vma reuse possibility mm: make vma cache SLAB_TYPESAFE_BY_RCU docs/mm: document latest changes to vm_lock Documentation/RCU/whatisRCU.rst | 10 + Documentation/core-api/refcount-vs-atomic.rst | 37 +++- Documentation/mm/process_addrs.rst | 44 +++-- include/linux/mm.h | 176 ++++++++++++++---- include/linux/mm_types.h | 75 ++++---- include/linux/mmap_lock.h | 6 - include/linux/rcuwait.h | 13 +- include/linux/refcount.h | 125 +++++++++++++ include/linux/slab.h | 15 +- include/linux/types.h | 12 ++ kernel/fork.c | 129 ++++++------- mm/debug.c | 6 + mm/init-mm.c | 1 + mm/memory.c | 106 ++++++++++- mm/mmap.c | 3 +- mm/nommu.c | 4 +- mm/userfaultfd.c | 38 ++-- mm/vma.c | 27 ++- mm/vma.h | 15 +- tools/include/linux/refcount.h | 5 + tools/testing/vma/linux/atomic.h | 6 + tools/testing/vma/vma.c | 42 ++++- tools/testing/vma/vma_internal.h | 127 ++++++------- 23 files changed, 702 insertions(+), 320 deletions(-) base-commit: 47aa60e930fe7fc2a945e4406e3ad1dfa73bb47c Tested-by: Shivank Garg