From patchwork Mon Oct 28 01:08:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13852772 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C503AD1358B for ; Mon, 28 Oct 2024 01:09:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4217B6B0088; Sun, 27 Oct 2024 21:09:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3D2016B0089; Sun, 27 Oct 2024 21:09:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 272776B008A; Sun, 27 Oct 2024 21:09:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0B1E66B0088 for ; Sun, 27 Oct 2024 21:09:10 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 48D9881C01 for ; Mon, 28 Oct 2024 01:08:50 +0000 (UTC) X-FDA: 82721226084.13.53DB35B Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf26.hostedemail.com (Postfix) with ESMTP id C1692140002 for ; Mon, 28 Oct 2024 01:08:50 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=gErwT3Bo; spf=pass (imf26.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730077538; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=gPaLmyZEMQjoZIzwUMT9oiAKI5SFr6ZlIYS/FfB7TZo=; b=EVi6qrvQI1g5EIbby7v4rMqs+pdzwqCV435LS5132zfUbhwEJlLvc2RG9FOFx8Lg70Bm8l zWLkH5Cr6lUfgjSBncnAMD2rhidH1Gtd3KBotuU7DLFI8ewr0BzzzCrVwOQF21mWRGplCo VlwNAvPDuIrZcATWxeVJucxKxptyIAQ= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=gErwT3Bo; spf=pass (imf26.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730077538; a=rsa-sha256; cv=none; b=6uqhPAqhiM6piUURwmG81s6qE4sgesNEdaIvHZpjS+mNyGeqOkYIlQxYYPq4gRVoyx+aEx GmY1Sf4/92PIU+YC16p7r3tp7cvwaw9fv7a6T8fei7boCE8SqTTCcH3nVyJJVlt4N5ziNj fcU+HaBpErR6adOZ/8dH6r+nEguq1r0= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id A7B4F5C3E75; Mon, 28 Oct 2024 01:08:21 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5625EC4CEC3; Mon, 28 Oct 2024 01:09:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730077746; bh=cLHlFgsTefJ/63cMxsPXUMnfpmoCzVWJwou//7LsYNw=; h=From:To:Cc:Subject:Date:From; b=gErwT3Bohbwn/Qs4nZQrC6/plN8hPUAEzZyFdUnMgUIUqTUPjxWcD7YB9PKdeAWeF /aFgyBgP+3e+NJOOCxJsRO31EIHGiT3QtU7SY6/n3vwkWJsViVAW6tBUXvD3teHE4g QSvSYRP0xW1MkYuxGpJ/SJSQzvh5+U7IoVNmIhTPNwbvH9nMjaDpD852M5smDN+8XT z4higS6hVwHgvS8dIpOtc5Gn7gx0X2xHLVA0q72y5WY7wtsvlo9cfRrsUd+fbKdrNg gI6Ifdd54lkGARG2BRQ1cCrPQyEcQXaUSm7OWgBLItuJcymqg4vPaXa0mwX4d03DfR YKvLOq6xH89rA== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, peterz@infradead.org Cc: oleg@redhat.com, rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, shakeel.butt@linux.dev, hannes@cmpxchg.org, Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, david@redhat.com, arnd@arndb.de, richard.weiyang@gmail.com, zhangpeng.00@bytedance.com, linmiaohe@huawei.com, viro@zeniv.linux.org.uk, hca@linux.ibm.com, Andrii Nakryiko Subject: [PATCH v4 tip/perf/core 0/4] uprobes,mm: speculative lockless VMA-to-uprobe lookup Date: Sun, 27 Oct 2024 18:08:14 -0700 Message-ID: <20241028010818.2487581-1-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: C1692140002 X-Stat-Signature: 8cj1hkiehisukjt1oek3j9t73hffcqmd X-Rspam-User: X-HE-Tag: 1730077730-644938 X-HE-Meta: U2FsdGVkX19ljm3/l9HTb9JSrCd3SABk+2eBldRBSljjw2xdP9/5Ix2EwDpfL/jJo+4RiCNXSfuyiJfB9OlyaukG0U+XLNpJggximaErvVV6c9dby3ALC6FwlWXicw9FAYW9aw7mTtIwVwJ4UxK9fvXTOgD4ntBqZdIUpSF3cYOQN6jPlqDenmezn/rPYb5i4dSGjGnBXPtH36d8gimols1rfGQBMXDhz+nbRZdDFs0Uh/nn2Kc4j7ISl+d2m/3RGPOJRw7178nWDo00ApWtWqVsYtow2JXWpbJDuQNENYJhHN48HF0LO0yWMpvlvMSlN56KV1IrviMMoCR8G6LiqI4efEW39vWnos6/CFvuW+RxGOYbEGzfa2HV3YnlOU5HsESXAq22NMUocOMIKV+uVtI3HDS4ynI0Cg/BIV3iS1O9WdfrH9WggOlzClD2ocmk7qK/pK4Z4rIKQqtJFer7qJ5lficbFUNR04/qn4L1o5c9C5DuomBJ3TV7LFoXb3BIdLDo17odeRgxVYclPkJYofeeZmGlokb28ni1hoOtP6znSXUkeeOA+uTflxyBKQSnmANoA4E4bxlDfIsxAsRXSFUF1Y9gzrkWveSQVA+3D3YxsmGmKS4WcFPwSl4Ib+Uy9nTUIOMWI0m2xr9raQtSyIsse6AcoteHszMO5dVl7sg5XbOrcuU/wd/qOrYFGj0uSDR3yaXwusAYlCT9VNZdgNY7GmbXqgPYDabEj8ltxeyLBgrXAEKrLVSL8vtK7MqTCZxJl+DI4y7kQzG3PXbYElQP+bUV3GL6KQKQtH7riJTtOG5U7JwuAfXqoP6Plhfn/zI58d7xqxWIQnEZ5pNa3nd3oFi9lh18U+jOFh853F2pavFb8BRkYg0X864ARKX2emgnYfaLO4NylHn0e3k/fTYKwQFmMzqvt1IC3zss3kE03w/U9EguvyIKooIg6mvSWmDg+gnZl27nqSWL4+J uZcYyzkT p1s8NX5p4tr323BypyaLsWmRa4am7thoxhvC1QPTBRNuOMQSVk5wd9sYa0oKoGNh5h0MB20NGwT7ZZNuam5NVpf2qdXPY2PTNzzFTp9wf8DZmF1eFgZVVX9fsUUnDWML8paAAJlFDdNg41c5hatAUGyHtRYHlH0duRewehVLB9Y7kKnOltwQAuaQLwsxAyc+u103VqjKymTD2Glo865obnNMjtCXDQjQtAM9Z59mDZIM2JDxaL2ikevZJUgr2ZIzkFMyfdjm2vEFkv2QgFwKJdSsYu4ugJ4/myMkdHJqdh2tDa/cRQ8GVkDZGcCrXzvN2zVCI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement speculative (lockless) resolution of VMA to inode to uprobe, bypassing the need to take mmap_lock for reads, if possible. First two patches by Suren adds mm_struct helpers that help detect whether mm_struct was changed, which is used by uprobe logic to validate that speculative results can be trusted after all the lookup logic results in a valid uprobe instance. Patch #3 is a simplification to uprobe VMA flag checking, suggested by Oleg. And, finally, patch #4 is the speculative VMA-to-uprobe resolution logic itself, and is the focal point of this patch set. It makes entry uprobes in common case scale very well with number of CPUs, as we avoid any locking or cache line bouncing between CPUs. See corresponding patch for details and benchmarking results. Note, this patch set assumes that FMODE_BACKING files were switched to have SLAB_TYPE_SAFE_BY_RCU semantics, which was recently done by Christian Brauner in [0]. This change can be pulled into perf/core through stable tags/vfs-6.13.for-bpf.file tag from [1]. [0] https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/commit/?h=vfs-6.13.for-bpf.file&id=8b1bc2590af61129b82a189e9dc7c2804c34400e [1] git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git v3->v4: - rebased and dropped data_race(), given mm_struct uses real seqcount (Peter); v2->v3: - dropped kfree_rcu() patch (Christian); - added data_race() annotations for fields of vma and vma->vm_file which could be modified during speculative lookup (Oleg); - fixed int->long problem in stubs for mmap_lock_speculation_{start,end}(), caught by Kernel test robot; v1->v2: - adjusted vma_end_write_all() comment to point out it should never be called manually now, but I wasn't sure how ACQUIRE/RELEASE comments should be reworded (previously requested by Jann), so I'd appreciate some help there (Jann); - int -> long change for mm_lock_seq, as agreed at LPC2024 (Jann, Suren, Liam); - kfree_rcu_mightsleep() for FMODE_BACKING (Suren, Christian); - vm_flags simplification in find_active_uprobe_rcu() and find_active_uprobe_speculative() (Oleg); - guard(rcu)() simplified find_active_uprobe_speculative() implementation. Andrii Nakryiko (2): uprobes: simplify find_active_uprobe_rcu() VMA checks uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution Suren Baghdasaryan (2): mm: Convert mm_lock_seq to a proper seqcount mm: Introduce mmap_lock_speculation_{begin|end} include/linux/mm.h | 12 ++--- include/linux/mm_types.h | 7 ++- include/linux/mmap_lock.h | 87 ++++++++++++++++++++++++-------- kernel/events/uprobes.c | 47 ++++++++++++++++- kernel/fork.c | 5 +- mm/init-mm.c | 2 +- tools/testing/vma/vma.c | 4 +- tools/testing/vma/vma_internal.h | 4 +- 8 files changed, 129 insertions(+), 39 deletions(-)