From patchwork Mon Dec 16 19:24:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910234 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED637E7717F for ; Mon, 16 Dec 2024 19:24:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F9CB6B00A8; Mon, 16 Dec 2024 14:24:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A9956B00A9; Mon, 16 Dec 2024 14:24:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 271F26B00AA; Mon, 16 Dec 2024 14:24:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0B34C6B00A8 for ; Mon, 16 Dec 2024 14:24:27 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B75B8A0A22 for ; Mon, 16 Dec 2024 19:24:26 +0000 (UTC) X-FDA: 82901797608.30.664D0C3 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf14.hostedemail.com (Postfix) with ESMTP id 2684F100014 for ; Mon, 16 Dec 2024 19:23:51 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3S0M57Ab; spf=pass (imf14.hostedemail.com: domain of 3Z35gZwYKCCgWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3Z35gZwYKCCgWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377051; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=LeGomQCdTlzqmvFZ6fy5rq2UtmzZcMIBE5Kh+oHkUdg=; b=hbG525vaPDTnG3DVGCF4h0bNve9J/a7DDlpj5eORFZ4jthvSdawmZES3LRF8bzUC5uLjlr jrK7ubTlxXSwaHnIcQNMhHVAsok1BLJWCuQ3dg9qQCbuha9d+9eWXCJKOhJRsVg1T8tLws AVX/VYmgPnF0dbd2duWTpYQV8ZHuoNc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3S0M57Ab; spf=pass (imf14.hostedemail.com: domain of 3Z35gZwYKCCgWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3Z35gZwYKCCgWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377051; a=rsa-sha256; cv=none; b=XIp6Wxmsp0Y1HdCeEw8Ap/uiajoN8HeooA6k+EWQ7YauhphQgWDhv4pACPNXX42DIdUWQt VK8GokmKJ63UGmrbrTQH74xR0zoEt2DfDY5II59CRvk4s4LixoABum3hGdgq3WG56IVKTh rvcliZiREjAlGzz53YFyUtxxfGYvEQM= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2eeeb5b7022so4213670a91.0 for ; Mon, 16 Dec 2024 11:24:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377063; x=1734981863; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=LeGomQCdTlzqmvFZ6fy5rq2UtmzZcMIBE5Kh+oHkUdg=; b=3S0M57AbxO7Ya/nx7t3KG3poG4qKc0ZtGqrH0ImBt/626gOQ7qPaWrUE9FDLrUDsbU M3hAWtff3Bpv94L1Eu2CmhaYVyvQu7dqTVNbfuutNBRXmwWlJTdiD0jHdO8p9rvcskUK ISaPCa6AffNu5hm2wMYBqLeWO7h+bH/N7lwEJNZHpgj2u3BfOZNqM9fTpHsStPEEzlQk mkrUXpGJvmDnFXOOqvMstpEVJW8YepadNsEFH1PRnVhB98lYjgs1qDVszcNBGs9mORUR iYD7dqXrpzFvRPZ7GeW+6475INe9/QOO/XDkK11M8E/oeAu/SNW7hJITc/0tuGqBoULI RG+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377063; x=1734981863; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=LeGomQCdTlzqmvFZ6fy5rq2UtmzZcMIBE5Kh+oHkUdg=; b=lxDLaBs1w7uqydgcawFO/MobjVWP4beurxqK6NUMxvuUd5zbELfQlHjrxXwuZcYcnP 5itQU6AI9ca8QgLDZBZ6+Gd9b/8k0589nQgbh83OlBweaOzWjd6HdTTzAhpxPjrnPVXc Cyc6UHTdxNRaMQMlJlRVvRZrSeVBNTflPCEw8pQVNJ0GhAeJltQqj83Ua1C3aDe8pWNu 4qPHXFQGNkN4O1h/LQZx5ulh7L6YtvGfd4Q5HKPbCIBZk+ak37MmGSU2ZC+3KXGYhH3+ uWQ4W7NQD2rSprv6CS/hykHYqcDrQ6TrpFISLrhYSTv8TZj3Y6DJSPmdfZ778bbZCBix 5YnA== X-Forwarded-Encrypted: i=1; AJvYcCWlYLGghQjUCcXzPHM1KlPhcunPVAbvWvB1eWu1o3DB2zhSy7DWRvZVtrl03m+1lIXVI3KaKxTvig==@kvack.org X-Gm-Message-State: AOJu0YwjAFcPJlajM+cq9E4ryjduo874zffEk6HRot86BM62Wm1JpW0q nXX3gpH3758/rMtBVXQW+GxqpbLmIB01F8xe9MWwGWPskmpoGgkuLd+UJnk4tw+lW0OvXecCxbE cJQ== X-Google-Smtp-Source: AGHT+IEdDvpLmKrEQvDiFjGzeYKKUg0/T8qcx1CuYCjryvPmhtOLq4Mr6xjJ+BU35VHJ2P4yHAcB9T2GqoI= X-Received: from pjbsy3.prod.google.com ([2002:a17:90b:2d03:b0:2ef:8d43:14d8]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:ec84:b0:2ee:f80c:6889 with SMTP id 98e67ed59e1d1-2f290dbce03mr20770436a91.33.1734377063393; Mon, 16 Dec 2024 11:24:23 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:03 -0800 Mime-Version: 1.0 X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-1-surenb@google.com> Subject: [PATCH v6 00/16] move per-vma lock into vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 2684F100014 X-Stat-Signature: wb5i483nimrwhuxummdssu6wxx5bajih X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1734377031-558444 X-HE-Meta: U2FsdGVkX19Q9bCZhsDm65b2QJxoH6XNOeRZXnqKShVXA7AGjbcCh1c0q7smU6AfptzJ+ZCrPuN5+6ch4W/K6VKDAcwVDiYmorkHnd+qP7rouIzP6gF8VnOq0Xr5BRaem4rF404x5tntaLjBDiqfzlFWmm+Di7F4GpuvSjVbhRjaKGUXseIKryehxqO8TXl1nEpFjRo3e+OZagqtE54G9yyxjPNQdrphLjKNSQ/ckllbh6JqiSrJPXgamoNQ0kfkkzO0pj3t23nbP6uK2OVsMuh3NKbJ6E8SOEP5UFaoCYPfn1mS9Ozi5RFMMj1CSt9k61/eJX4CulTVAZ3cbIzsjfxaN+Gmu7xacrjzfGbsiCrV+woA8WpJV09q/UyFr6WgjpbszChG4y9kvU50rbURVy80DPT0bTVh9ffek0mlVGaJVhFtl7ev3CRjTaTRYU9K84NWtAIoQb5osVzcDlQEQFIoljNcgHMkHmjLXB15LVrhfhsPDwstDV5wSwpjR22NaM2NUlLETt9+BPUaRcuKLs8Md7Bb2BbMdxt/fTLni/uyp9hGZEND/IrHgsXg/T5XAiqRTM3rxN2INlvT9yjMgAWgZ5v+tye/lBNR+KauB9IfDQ3fMbCh7mb/Exbw8gUOpmVUr3r2CEoAfo95p7+VKVnhLtic5OHZXnuWmIPt7OrkcoZBVMwojO2RLv9o8nLVc8DbpH86UKwlYObo9wlNQ6cYXOxNc0XapgPK7WAtGWWwtT7TUxDviTmp4/RJG1647ocv1o0XMYqfGys8zymSGxnl911jf69OXDMBQh0Yn4c/WJIABO/ugQ1TF6Zo5T3Lv/Aoegj1j1hxx82ktZmj5h/uFwQj5TrEsqCrZDyr8mC8BqM06CO/ZOd67HgFOxsrCm+NSYD0yaF2mArBKSNpvE+99QCCW7FjpfAXEFftL0LLJPvRx3+VMffnFtF+9q4FasvcOzCPG9cx3MozC8m +nP7E2Ij 7uGMCf5YUkqHoEdIdRL6TMjrn84okh3VcPFdt3+NRCp54m17BFp0mQCm+VPG5LDoxzMfzbaioRRIVaDBYIMy4w+W4X1b1fHDRgWJqpMS/ob4urUUPrX6JXV2d1nTe/Udq+ppuuW3s09i2c3ruC5J/joClogSXxbWGjnM174f0Vjc3yecq8KblqcjkTv+n6/ot9ee1+J/3B0/Rr+lv5sFadzAzSZYLPbI17xbx6MxDOqzRIdn+h2hrlh6MFjx0Xd1hCJOTIszeL546InALzkh24dx7OMQN/KJBmMIy9t7t06lPnKaUk/N68f+bfqZH1H5qiwXd6ARvY335zWW33cOlOrw7J9Sph8E/IKEiaG4n2HFT5pR/uHnbHX06yq9x1QrTB7zXBeF+qHGvJMp8dcJOFgpL9woUt0xPleihMX9/lBiFzLvFThb3Xi+rCJB0slyuG8gdLYomkmf3rMIBu7NqTzw8lM0DlBaU1gFQ7VZRThxQEKaGHOi1RKLpHFF1WNE/qLj2ZK9TQX3H8OXyNq2E13UpcOyMh1ILaiVJ9ji3BRjXweOk8nppaTAF5wmTuJxcAW40TCvFtdVGn7se81PGF2vSwB0PsIgBrOXuDNvYWGX7tWzmSMW8h/5+FmUlz6SES8Hj X-Bogosity: Ham, tests=bogofilter, spamicity=0.002205, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Back when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3]. Splitting single logical structure into multiple ones leads to more complicated management, extra pointer dereferences and overall less maintainable code. When that split-away part is a lock, it complicates things even further. With no performance benefits, there are no reasons for this split. Merging the vm_lock back into vm_area_struct also allows vm_area_struct to use SLAB_TYPESAFE_BY_RCU later in this patchset. This patchset: 1. moves vm_lock back into vm_area_struct, aligning it at the cacheline boundary and changing the cache to be cacheline-aligned to minimize cacheline sharing; 2. changes vm_area_struct initialization to mark new vma as detached until it is inserted into vma tree; 3. replaces vm_lock and vma->detached flag with a reference counter; 4. changes vm_area_struct cache to SLAB_TYPESAFE_BY_RCU to allow for their reuse and to minimize call_rcu() calls. Pagefault microbenchmarks show performance improvement: Hmean faults/cpu-1 507926.5547 ( 0.00%) 506519.3692 * -0.28%* Hmean faults/cpu-4 479119.7051 ( 0.00%) 481333.6802 * 0.46%* Hmean faults/cpu-7 452880.2961 ( 0.00%) 455845.6211 * 0.65%* Hmean faults/cpu-12 347639.1021 ( 0.00%) 352004.2254 * 1.26%* Hmean faults/cpu-21 200061.2238 ( 0.00%) 229597.0317 * 14.76%* Hmean faults/cpu-30 145251.2001 ( 0.00%) 164202.5067 * 13.05%* Hmean faults/cpu-48 106848.4434 ( 0.00%) 120641.5504 * 12.91%* Hmean faults/cpu-56 92472.3835 ( 0.00%) 103464.7916 * 11.89%* Hmean faults/sec-1 507566.1468 ( 0.00%) 506139.0811 * -0.28%* Hmean faults/sec-4 1880478.2402 ( 0.00%) 1886795.6329 * 0.34%* Hmean faults/sec-7 3106394.3438 ( 0.00%) 3140550.7485 * 1.10%* Hmean faults/sec-12 4061358.4795 ( 0.00%) 4112477.0206 * 1.26%* Hmean faults/sec-21 3988619.1169 ( 0.00%) 4577747.1436 * 14.77%* Hmean faults/sec-30 3909839.5449 ( 0.00%) 4311052.2787 * 10.26%* Hmean faults/sec-48 4761108.4691 ( 0.00%) 5283790.5026 * 10.98%* Hmean faults/sec-56 4885561.4590 ( 0.00%) 5415839.4045 * 10.85%* Changes since v5 [4] - Added Reviewed-by, per Vlastimil Babka; - Added replacement of vm_lock and vma->detached flag with vm_refcnt, per Peter Zijlstra and Matthew Wilcox - Marked vmas detached during exit_mmap; - Ensureed vmas are in detached state before they are freed; - Changed SLAB_TYPESAFE_BY_RCU patch to not require ctor, leading to a much simpler code; - Removed unnecessary patch [5] - Updated documentation to reflect changes to vm_lock; Patchset applies over mm-unstable after reverting v5 of this patchset [4] (currently 687e99a5faa5-905ab222508a) [1] https://lore.kernel.org/all/20230227173632.3292573-34-surenb@google.com/ [2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/ [3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/ [4] https://lore.kernel.org/all/20241206225204.4008261-1-surenb@google.com/ [5] https://lore.kernel.org/all/20241206225204.4008261-6-surenb@google.com/ Suren Baghdasaryan (16): mm: introduce vma_start_read_locked{_nested} helpers mm: move per-vma lock into vm_area_struct mm: mark vma as detached until it's added into vma tree mm/nommu: fix the last places where vma is not locked before being attached types: move struct rcuwait into types.h mm: allow vma_start_read_locked/vma_start_read_locked_nested to fail mm: move mmap_init_lock() out of the header file mm: uninline the main body of vma_start_write() refcount: introduce __refcount_{add|inc}_not_zero_limited mm: replace vm_lock and detached flag with a reference count mm: enforce vma to be in detached state before freeing mm: remove extra vma_numab_state_init() call mm: introduce vma_ensure_detached() mm: prepare lock_vma_under_rcu() for vma reuse possibility mm: make vma cache SLAB_TYPESAFE_BY_RCU docs/mm: document latest changes to vm_lock Documentation/mm/process_addrs.rst | 44 ++++---- include/linux/mm.h | 162 +++++++++++++++++++++++------ include/linux/mm_types.h | 37 ++++--- include/linux/mmap_lock.h | 6 -- include/linux/rcuwait.h | 13 +-- include/linux/refcount.h | 20 +++- include/linux/slab.h | 6 -- include/linux/types.h | 12 +++ kernel/fork.c | 88 ++++------------ mm/init-mm.c | 1 + mm/memory.c | 75 +++++++++++-- mm/mmap.c | 8 +- mm/nommu.c | 2 + mm/userfaultfd.c | 31 +++--- mm/vma.c | 15 ++- mm/vma.h | 4 +- tools/testing/vma/linux/atomic.h | 5 + tools/testing/vma/vma_internal.h | 96 +++++++++-------- 18 files changed, 378 insertions(+), 247 deletions(-)