From patchwork Fri Dec 6 22:51:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13897896 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9200BE77173 for ; Fri, 6 Dec 2024 22:52:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 108CD6B0306; Fri, 6 Dec 2024 17:52:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0BA7E6B0307; Fri, 6 Dec 2024 17:52:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EEB096B0308; Fri, 6 Dec 2024 17:52:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id CFA126B0306 for ; Fri, 6 Dec 2024 17:52:10 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4D14342EDD for ; Fri, 6 Dec 2024 22:52:10 +0000 (UTC) X-FDA: 82866033642.18.9854CF5 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf23.hostedemail.com (Postfix) with ESMTP id EC5C6140016 for ; Fri, 6 Dec 2024 22:51:56 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Q103EBKP; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3F4BTZwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3F4BTZwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733525512; a=rsa-sha256; cv=none; b=b3F2sTPBfbMYWWWFawKuhnccZK3ECkzIqLaWcYtmpaKl5MpQgYPh+T40f70CVYKq6oOpyp iQo+s6d8Bf0299IYZaG/t9n6s142afa3zMVfdIZ7PgogkaijldVVAPlKkRANpa0ZL9fZiZ CKH3WupjJwuh5ivh8y+ij4TCbC8IuhU= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Q103EBKP; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3F4BTZwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3F4BTZwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733525512; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=dMTBtYHVwLTdC/ExbH0P/+wfgLQciB2EDIuylOcBn5Y=; b=mlEnwi3+w+RTxkHmFtpE7F6f+1nR71GbDGzj4XF7O228LYq3D1GcW405Q5iFuJuIh02pmz n6+QeaW8j3dKUpeZWcQkZAl7k9/RPgF8agP0skDvuPdGKPChEE1KCj+Dv8lqyHF4Nqbdo7 G+fYioCuCRyDMglKf8xGnJUv4samkWU= Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-7251424982fso2243705b3a.0 for ; Fri, 06 Dec 2024 14:52:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733525527; x=1734130327; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=dMTBtYHVwLTdC/ExbH0P/+wfgLQciB2EDIuylOcBn5Y=; b=Q103EBKPuH1LSX1F2gXHL5qnnQfSzbL02MQe6WJ0rHZtMqUjAxFjToY6b715BiZot3 iKTa8eFy6wWaf2ZTM/m87Do5SSZgZ9UdP0xqrz08d3IUxRzxCqHroZ4QzOqZojOZEMjQ wddZuEiItZSMq7zEX0b+IedybB+dSnJs7QEHuATG8ewu6MSR1nnE1PZMfyO+6pH2bU3C 5HXyfPXoxSv4Zuqzx6/frnvc2lTOFNUFjeznZif0CBN2IF+ApMcC2Tf6PgqZg23aodCB oN+emxlfp+E24r553xfwT4lLD+hmCMyl+rQcch4qxTwrifvprmi/N3aL2iaMf5+bIJRZ 5SWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525527; x=1734130327; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=dMTBtYHVwLTdC/ExbH0P/+wfgLQciB2EDIuylOcBn5Y=; b=VpOGcnoXFSKEe+WNovIEudxX7CTyq478eD7JdnL4iHhrePJmaosSeAUZY96ahflyi5 cJlq10j1iToSc6cvtFGZIqWjPVn6RnX6ZPP+iawCuGpVB8oeitnpWMV8WO/Qx7NuZS4a ajWWyMjFF7l2gAYqAgmqzqfEOkOq/kl5jXlFoYOr/orMf7QYRL2dWvpr5NytbK0zCQ1K 2jrbxWOJYD538RUyyEX1BO+xdWoqxgcsGI7kG4RIaaHqPOv3X/PtwVzDHiGyrue5ggPk HzA6Gmh03LZrhYrzKVQzP7sbQ8RAUywMeXgMNFCscHkpEg1+qRU0aEN5jd+XllJRx20z CnEg== X-Forwarded-Encrypted: i=1; AJvYcCVhwUI5xjhRT2Eed5sKaR5e9GPt32P4RqXN0i/+zpRqabwoIS1k4glxp+xgtZUsXLgVoPpwdA64KQ==@kvack.org X-Gm-Message-State: AOJu0Yw5LLWkwC70FDITAPYnEepVPeTUpl2+A76xbZHwb4hDcescYX+g 1CuJL3dWxsUVMf7EEGXJhOfYTYq59eqvpuczl9/v77NBYf39hNywD3SFfegqfjjQahgUj1IK4am NFw== X-Google-Smtp-Source: AGHT+IEfV06JzC0SiVnMxaptnUTDnfB304/9csh16+/GMcszHB4C/gm3cOQerG89/3g7z8wtX+YAxVhCLkE= X-Received: from pfwy40.prod.google.com ([2002:a05:6a00:1ca8:b0:724:f17d:ebd7]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:3016:b0:725:c8ea:b320 with SMTP id d2e1a72fcca58-725c8eab756mr3012011b3a.14.1733525527205; Fri, 06 Dec 2024 14:52:07 -0800 (PST) Date: Fri, 6 Dec 2024 14:51:57 -0800 Mime-Version: 1.0 X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241206225204.4008261-1-surenb@google.com> Subject: [PATCH v5 0/6] move per-vma lock into vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: 16sgchua9dw9znqmi9gp91kzt4gdkbtj X-Rspam-User: X-Rspamd-Queue-Id: EC5C6140016 X-Rspamd-Server: rspam08 X-HE-Tag: 1733525516-881986 X-HE-Meta: U2FsdGVkX1/A/G4WaZ6m7xnfP+qS5sbSfFacKnJHA9PAinoJ4NGiSHTEQrr7wpsAf/XcxrPHmkx2HhM4bWumxIkVU5hvqqy34m5Hfzc3t0niHqOKcRfY3SncOoO1hck0CNA4YQJGYUg8KxOMt1WdBlByORJ0QXlZ6+tv3Qujncjul4VeIjuwU40YhGKO0UieJR1xQNeMTkSAFJT6eVkOv2hCpV2NRJwsCl1O0dnvvjk8yOf3ddTVWR2jLimCQVrzgnFC819zQWhEaQZ9t6pibmqlJpbdSDntq1Eh221fA46MytaNxKiPAYkMAjJ8YyUYnO6c7GOxY7vm+uJsyP75Da8/OOxwMqwreGN3X0tf9BZO7EpW1xsyHRSD6c5kPbsXks/24OBjc5MfKIWDh70XLHtmY2lHYdmNMPOS80Ql2KOL7kZFzJCCcHZcC1hmmJS8uTiPtXRHSZb0KbBgTDZ4jmBuyGoAmE8DFaeIUGRc2hCZhIUNTxOct8vS9ZspHWSe2lXycPt9u+1phGHFoDOByPFawWVLD6XyqPEzzE8lxRrojLQmT+xHj0SF2Qt7jj3SI6hWkrFNfxErgDsYk/OpJ/Ube4RaFJD4DElfeAKfz4p/uLhebQ4bJLRLsViSV2TpsIr4H5fqlpn6YCqf3XBoE9sra1B/W3FRWVhleTsmQ/91Rm8OuuW61/YcJ4eqqmaXbYeeiQJAAWSZ3CHjeMZFckskbhEn56hsd4FGFyQoyeliEMBo2Mi43d1AWoLchvMyB6tGqItFQ7v2lmnsJzIuL4cM7RYU7WekUBO1sdFoO68/gDXtktQ2MAPe+WxAHf94IYPPMF8CSztgj7FEURgwFggPEUIy1VQuixE5Ov/s61Rj13VIKkIgxFwOCApPAPRPu3Wsto0JWSpmOy4lAjx7zRP25G8pzag9qZJS8YuVQ6XyXCahvCIhrmbbeYk9K8p4OoXQWXnes4n+4tob3dw tzB+v5i9 UQ7NlSk4aLJ6N37fkK1rncm7G/PhNV3DOWZDueWfpxPKrZlI64NKROPALqNKjnHE8PPySIxdZ/zE9vZHlVpbtgZGqht4vfI0ZOtJgr+wZYCxc/pPyC367LTi7ytd9UQHZzscDlZlowiknHJE0U8BIeMvmYoBlmHdLD8YCIjBqXqXVQqWyCtYmRIKy0qEc0yZ7O5+HqKc5sWu3d8zqv/tTFivospwnSC8A5hn4PDYRkHVd8YHdiSKtbCHUCwvMcYlJJOyOOmeoXIHNZLY6w+17q4vHWpfJu7f6OlNlYRkuRt+ZVfHGDKo7xiGQuNRiO4k5PSDrxZz/2jXttnlGt7p6ShXI1C3ECh25+FU2JogcRw4TOfoQJxq+4qAtjGW/xocHOQoRb2ZQV2d9DIal68JuJd3W41LrXNkDJg5shXDKWNmtkoxCe88GOeASm2b2j+VccG4qfJmI/OZQ+RUH++A9/NcLSy9ZAPnMUW0weFLvCLE001KI0ISIEjMyMX95JQDohCwgUFObVF/b7ARvAW4t/eyqGqsQ8+/uvTRxL7Qf5qFPmp4lqD5Kn6v4C1XD8VO0IoeQMLkj3vtjgQI9/uN08TIcOoD7XgYuooaWvQUESRnl2bM1rsLIthfgqTh5GS618Xsm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Back when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3]. Splitting single logical structure into multiple ones leads to more complicated management, extra pointer dereferences and overall less maintainable code. When that split-away part is a lock, it complicates things even further. With no performance benefits, there are no reasons for this split. Merging the vm_lock back into vm_area_struct also allows vm_area_struct to use SLAB_TYPESAFE_BY_RCU later in this patchset. This patchset: 1. moves vm_lock back into vm_area_struct, aligning it at the cacheline boundary and changing the cache to be cacheline-aligned to minimize cacheline sharing; 2. changes vm_area_struct initialization to mark new vma as detached until it is inserted into vma tree; 3. changes vm_area_struct cache to SLAB_TYPESAFE_BY_RCU to allow for their reuse and to minimize call_rcu() calls. To avoid bloating vm_area_struct, we introduce unioned freeptr_t field and allow freeptr_offset to be used with ctor in a separate patch. Pagefault microbenchmarks do not show noticeable performance change. Changes since v4 [4] - Added SOBs, per Lorenzo Stoakes and Shakeel Butt; - Changed vma_clear() and vma_copy() to set required vma members individually, per Matthew Wilcox - Added comments in vma_start_read() about new false locked result possibilities, per Vlastimil Babka - Added back freeptr_t into vm_area_struct, as it's harmless and can be used in the future to shrink the structure, per Vlastimil Babka - Fixed the race in vm_area_alloc() when vma->detached was temporarily reset by memcpy() before setting it back, per Vlastimil Babka - Added a patch allowing freeptr_offset to be used with ctor, per Vlastimil Babka Patch applies over linux-next (due to vm_lock change [5] not in mm tree). [1] https://lore.kernel.org/all/20230227173632.3292573-34-surenb@google.com/ [2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/ [3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/ [4] https://lore.kernel.org/all/20241120000826.335387-1-surenb@google.com/ [5] https://lore.kernel.org/all/20241122174416.1367052-2-surenb@google.com/ Suren Baghdasaryan (6): mm: introduce vma_start_read_locked{_nested} helpers mm: move per-vma lock into vm_area_struct mm: mark vma as detached until it's added into vma tree mm: make vma cache SLAB_TYPESAFE_BY_RCU mm/slab: allow freeptr_offset to be used with ctor docs/mm: document latest changes to vm_lock Documentation/mm/process_addrs.rst | 10 +- include/linux/mm.h | 107 ++++++++++++++---- include/linux/mm_types.h | 16 ++- include/linux/slab.h | 11 +- kernel/fork.c | 168 ++++++++++++++++++++--------- mm/memory.c | 17 ++- mm/slub.c | 2 +- mm/userfaultfd.c | 22 +--- mm/vma.c | 8 +- mm/vma.h | 2 + tools/testing/vma/vma_internal.h | 55 ++++------ 11 files changed, 267 insertions(+), 151 deletions(-) base-commit: ebe1b11614e079c5e366ce9bd3c8f44ca0fbcc1b