From patchwork Tue Sep 3 17:46:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13789160 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBB52CD37AA for ; Tue, 3 Sep 2024 17:46:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DD748D01BF; Tue, 3 Sep 2024 13:46:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 43C7E8D018A; Tue, 3 Sep 2024 13:46:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26B5E8D01BF; Tue, 3 Sep 2024 13:46:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id F372B8D018A for ; Tue, 3 Sep 2024 13:46:37 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 45B451A084B for ; Tue, 3 Sep 2024 17:46:37 +0000 (UTC) X-FDA: 82524156834.20.D81D0F2 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf24.hostedemail.com (Postfix) with ESMTP id 614B4180033 for ; Tue, 3 Sep 2024 17:46:35 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=HI8wfMpG; spf=pass (imf24.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725385469; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2cPfkNPUgpWNCJ64tOTVJ9JwI+/m8MnmG4i1rhglujk=; b=ycrCu5H1jND4Lh4am8WlDlsyBPr/Ufow0EtxSmP4Y/BcDUTfVPS1kkpdKTJyc1kPfneELo 8tXBTc42QsqYx3ZFr5EZmBtfxMxyLu643eOfu3AblGU7vhlmNYsYhN5IoE/H6uiiJlHBjV l0PlqauUreRPgqk/6qi3ZSofhh3Wz5k= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=HI8wfMpG; spf=pass (imf24.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725385469; a=rsa-sha256; cv=none; b=VipVDXm929L8MBhGoOcBK90Wx8Hysu9gN4ak8aLXMUXunNFqBfcQyI9OVViPP4RYBf3CtD /1ZUV1P971EH7y98v5rmvt+6GgSlKW59hvmPYO2xNCVx/Qf0r1IVzqNYCOXLIE+oXGlt7J sSbbqAvyTCMMabFtZuOElWlqkmjOk9A= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 239E65C5A41; Tue, 3 Sep 2024 17:46:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E481FC4CEC6; Tue, 3 Sep 2024 17:46:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725385594; bh=R43LEMwmNgnFNuxwf3tZak7H2OM5C5eCgaYGHiHVLKQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HI8wfMpGnWd5IDk1+SWR2si4h7TTVdySI6NjRRNceULM9jxc1mTi8TSxrcv7qt2/H K2J8ytHFtwPEPduB0Thbxm+VtWE/ANW5HFXHbn8mAUuq0N08HRQGso0NR63YVe2Rtf wMytWtQfWut96YvdyPDW0Lknyiw6MN+OGUacfX3o4ZAc9fXxk+ttv+UQGl8MLnfmGf 1vi5w2koWFj9R87QOtca1gfaUU90bmfRwwTs8Kg/0szmEUvkSEB2PDMvjnkgDrd0hu a+xADk99DZUWTxFD4lQbcBxqcVRD7TQ3qYrqNDFxAhkvQsYUwBlVVsC9qvymVEfd8a RIcY0M0oKgmCg== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, Andrii Nakryiko Subject: [PATCH v5 8/8] uprobes: switch to RCU Tasks Trace flavor for better performance Date: Tue, 3 Sep 2024 10:46:03 -0700 Message-ID: <20240903174603.3554182-9-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20240903174603.3554182-1-andrii@kernel.org> References: <20240903174603.3554182-1-andrii@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 614B4180033 X-Stat-Signature: 64s1cqir8ncfqyuwqri1b6od5dc1df9m X-Rspam-User: X-HE-Tag: 1725385595-655170 X-HE-Meta: U2FsdGVkX194aCuI7V8Mc1jHB8zkWGca6VKxxglLzZgX5Lefy5r2G5HBBp1/kZPO6f/fQWHqcoze+aE1cZcuukmax6B72H3NxboHSE5mPTb2vZ7f875Pp9mwOCdyu2UmprWKfZS7nNOswbuS5xgfB4hFCPgmP6tvaDK+KHqmWxwosFo669DfdXH9AiTzM9wkRnKApp1yqqCcq49f+Gkz22u9WR/xTBKKlEiyOu113NOGXXgG2+jQQueR1RejfxKVut5LLtBR99xYUqrNUF8c12ryq3kxJJFLjSdDi/XeC9X4M9giZA6bD4u3ygZk+0aeNfCiGcz18pwDuOYVY9haNirCpWtOG1mAVxlFZS+tDxDP3Ncqfr6f0EYpVrbFlYjeC9cxZJzyLZ2NUlL2u3QgxMNBVfHdBGfK8h9RoYvpNcQOHkE+DY3dN/rnjPo0cMhTZT7jEdyRRF5warxZMZ3LZwwr0XkLcZz2iURvwKXBmzUKPH+G1CbQ7jL1zNQHTKbnGDX52PLWIl/HQyHo+4WaD0u7loe/bthxqHeAYz0mvAegFR686pdtoQeO0F0XXUtBpjGztoi922Ud/Wk/FhZ4hT6pRc/C85knuEhZlrh+tSA7BGmVwNVmYxiOSSOA4QFHQZRYHvI8NCYDwTu9q0r9O9umH+Bq/rqDv1ua9qqR4oWKuXN/PTW0bJU2MeHcQjGzuqlGe3bGpvP6RAYIchknlRQaXhagYY2ZZ5hZ1ITt4XcG4WCyYLxXu0qR0rGBa6+cMuWeK0T4nzsEYlm4Q/UxY8f7AjKnkJF/tXux+4Kem+POfn5Yervtf3ddVJvKUdRdcvvBUZPJcmMzOKLO/DqB0FBvZyaPMtfSLMohdAXZgZYCVrrGh/aRQYdS4IrO9MujjHTLRU/+wAsLGkOVPGEZ7SLeQJ1uhK01T4gdFBMURN8I+KklJnhkOQSkuUmGrE5LVSFYBC25d5impP/CmMG 3mOlcdS1 UAjnS5VsQzf4BcnfMmb6kg+Dw8Vv/aRnENr5e0YZZrBZfjKNpdi1t9pVUjJcoTJKXU25jUvGlWXCNtPlO0SpePwQgfPOHNCd5wpJePVtLDwxv4sJdgmOhZjqjsUuiBJa79AZTwjVcCL1Bev4WI7uLmPJei1EvbXQXUwJysfoARAVDsaRpsHfiTMGsmS+J2oPIjnFiIjhtGh+SuzJxGpPa5/PfMOQ7SdE4CfxSY0NWbHVQdDR7UnbDRvofU0UkDQ8B/5oRnDDn3jxkfw5HD5B+rdPBgUT5DtjMK48aZcWJ1qNQEZI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch switches uprobes SRCU usage to RCU Tasks Trace flavor, which is optimized for more lightweight and quick readers (at the expense of slower writers, which for uprobes is a fine tradeof) and has better performance and scalability with number of CPUs. Similarly to baseline vs SRCU, we've benchmarked SRCU-based implementation vs RCU Tasks Trace implementation. SRCU ==== uprobe-nop ( 1 cpus): 3.276 ± 0.005M/s ( 3.276M/s/cpu) uprobe-nop ( 2 cpus): 4.125 ± 0.002M/s ( 2.063M/s/cpu) uprobe-nop ( 4 cpus): 7.713 ± 0.002M/s ( 1.928M/s/cpu) uprobe-nop ( 8 cpus): 8.097 ± 0.006M/s ( 1.012M/s/cpu) uprobe-nop (16 cpus): 6.501 ± 0.056M/s ( 0.406M/s/cpu) uprobe-nop (32 cpus): 4.398 ± 0.084M/s ( 0.137M/s/cpu) uprobe-nop (64 cpus): 6.452 ± 0.000M/s ( 0.101M/s/cpu) uretprobe-nop ( 1 cpus): 2.055 ± 0.001M/s ( 2.055M/s/cpu) uretprobe-nop ( 2 cpus): 2.677 ± 0.000M/s ( 1.339M/s/cpu) uretprobe-nop ( 4 cpus): 4.561 ± 0.003M/s ( 1.140M/s/cpu) uretprobe-nop ( 8 cpus): 5.291 ± 0.002M/s ( 0.661M/s/cpu) uretprobe-nop (16 cpus): 5.065 ± 0.019M/s ( 0.317M/s/cpu) uretprobe-nop (32 cpus): 3.622 ± 0.003M/s ( 0.113M/s/cpu) uretprobe-nop (64 cpus): 3.723 ± 0.002M/s ( 0.058M/s/cpu) RCU Tasks Trace =============== uprobe-nop ( 1 cpus): 3.396 ± 0.002M/s ( 3.396M/s/cpu) uprobe-nop ( 2 cpus): 4.271 ± 0.006M/s ( 2.135M/s/cpu) uprobe-nop ( 4 cpus): 8.499 ± 0.015M/s ( 2.125M/s/cpu) uprobe-nop ( 8 cpus): 10.355 ± 0.028M/s ( 1.294M/s/cpu) uprobe-nop (16 cpus): 7.615 ± 0.099M/s ( 0.476M/s/cpu) uprobe-nop (32 cpus): 4.430 ± 0.007M/s ( 0.138M/s/cpu) uprobe-nop (64 cpus): 6.887 ± 0.020M/s ( 0.108M/s/cpu) uretprobe-nop ( 1 cpus): 2.174 ± 0.001M/s ( 2.174M/s/cpu) uretprobe-nop ( 2 cpus): 2.853 ± 0.001M/s ( 1.426M/s/cpu) uretprobe-nop ( 4 cpus): 4.913 ± 0.002M/s ( 1.228M/s/cpu) uretprobe-nop ( 8 cpus): 5.883 ± 0.002M/s ( 0.735M/s/cpu) uretprobe-nop (16 cpus): 5.147 ± 0.001M/s ( 0.322M/s/cpu) uretprobe-nop (32 cpus): 3.738 ± 0.008M/s ( 0.117M/s/cpu) uretprobe-nop (64 cpus): 4.397 ± 0.002M/s ( 0.069M/s/cpu) Peak throughput for uprobes increases from 8 mln/s to 10.3 mln/s (+28%!), and for uretprobes from 5.3 mln/s to 5.8 mln/s (+11%), as we have more work to do on uretprobes side. Even single-thread (no contention) performance is slightly better: 3.276 mln/s to 3.396 mln/s (+3.5%) for uprobes, and 2.055 mln/s to 2.174 mln/s (+5.8%) for uretprobes. Reviewed-by: Oleg Nesterov Signed-off-by: Andrii Nakryiko --- kernel/events/uprobes.c | 38 ++++++++++++++++---------------------- 1 file changed, 16 insertions(+), 22 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 8a464cf38127..1b3990dd9c93 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -26,6 +26,7 @@ #include #include #include +#include #include @@ -42,8 +43,6 @@ static struct rb_root uprobes_tree = RB_ROOT; static DEFINE_RWLOCK(uprobes_treelock); /* serialize rbtree access */ static seqcount_rwlock_t uprobes_seqcount = SEQCNT_RWLOCK_ZERO(uprobes_seqcount, &uprobes_treelock); -DEFINE_STATIC_SRCU(uprobes_srcu); - #define UPROBES_HASH_SZ 13 /* serialize uprobe->pending_list */ static struct mutex uprobes_mmap_mutex[UPROBES_HASH_SZ]; @@ -652,7 +651,7 @@ static void put_uprobe(struct uprobe *uprobe) delayed_uprobe_remove(uprobe, NULL); mutex_unlock(&delayed_uprobe_lock); - call_srcu(&uprobes_srcu, &uprobe->rcu, uprobe_free_rcu); + call_rcu_tasks_trace(&uprobe->rcu, uprobe_free_rcu); } static __always_inline @@ -707,7 +706,7 @@ static struct uprobe *find_uprobe_rcu(struct inode *inode, loff_t offset) struct rb_node *node; unsigned int seq; - lockdep_assert(srcu_read_lock_held(&uprobes_srcu)); + lockdep_assert(rcu_read_lock_trace_held()); do { seq = read_seqcount_begin(&uprobes_seqcount); @@ -935,8 +934,7 @@ static bool filter_chain(struct uprobe *uprobe, struct mm_struct *mm) bool ret = false; down_read(&uprobe->consumer_rwsem); - list_for_each_entry_srcu(uc, &uprobe->consumers, cons_node, - srcu_read_lock_held(&uprobes_srcu)) { + list_for_each_entry_rcu(uc, &uprobe->consumers, cons_node, rcu_read_lock_trace_held()) { ret = consumer_filter(uc, mm); if (ret) break; @@ -1157,7 +1155,7 @@ void uprobe_unregister_sync(void) * unlucky enough caller can free consumer's memory and cause * handler_chain() or handle_uretprobe_chain() to do an use-after-free. */ - synchronize_srcu(&uprobes_srcu); + synchronize_rcu_tasks_trace(); } EXPORT_SYMBOL_GPL(uprobe_unregister_sync); @@ -1241,19 +1239,18 @@ EXPORT_SYMBOL_GPL(uprobe_register); int uprobe_apply(struct uprobe *uprobe, struct uprobe_consumer *uc, bool add) { struct uprobe_consumer *con; - int ret = -ENOENT, srcu_idx; + int ret = -ENOENT; down_write(&uprobe->register_rwsem); - srcu_idx = srcu_read_lock(&uprobes_srcu); - list_for_each_entry_srcu(con, &uprobe->consumers, cons_node, - srcu_read_lock_held(&uprobes_srcu)) { + rcu_read_lock_trace(); + list_for_each_entry_rcu(con, &uprobe->consumers, cons_node, rcu_read_lock_trace_held()) { if (con == uc) { ret = register_for_each_vma(uprobe, add ? uc : NULL); break; } } - srcu_read_unlock(&uprobes_srcu, srcu_idx); + rcu_read_unlock_trace(); up_write(&uprobe->register_rwsem); @@ -2123,8 +2120,7 @@ static void handler_chain(struct uprobe *uprobe, struct pt_regs *regs) current->utask->auprobe = &uprobe->arch; - list_for_each_entry_srcu(uc, &uprobe->consumers, cons_node, - srcu_read_lock_held(&uprobes_srcu)) { + list_for_each_entry_rcu(uc, &uprobe->consumers, cons_node, rcu_read_lock_trace_held()) { int rc = 0; if (uc->handler) { @@ -2162,15 +2158,13 @@ handle_uretprobe_chain(struct return_instance *ri, struct pt_regs *regs) { struct uprobe *uprobe = ri->uprobe; struct uprobe_consumer *uc; - int srcu_idx; - srcu_idx = srcu_read_lock(&uprobes_srcu); - list_for_each_entry_srcu(uc, &uprobe->consumers, cons_node, - srcu_read_lock_held(&uprobes_srcu)) { + rcu_read_lock_trace(); + list_for_each_entry_rcu(uc, &uprobe->consumers, cons_node, rcu_read_lock_trace_held()) { if (uc->ret_handler) uc->ret_handler(uc, ri->func, regs); } - srcu_read_unlock(&uprobes_srcu, srcu_idx); + rcu_read_unlock_trace(); } static struct return_instance *find_next_ret_chain(struct return_instance *ri) @@ -2255,13 +2249,13 @@ static void handle_swbp(struct pt_regs *regs) { struct uprobe *uprobe; unsigned long bp_vaddr; - int is_swbp, srcu_idx; + int is_swbp; bp_vaddr = uprobe_get_swbp_addr(regs); if (bp_vaddr == uprobe_get_trampoline_vaddr()) return uprobe_handle_trampoline(regs); - srcu_idx = srcu_read_lock(&uprobes_srcu); + rcu_read_lock_trace(); uprobe = find_active_uprobe_rcu(bp_vaddr, &is_swbp); if (!uprobe) { @@ -2319,7 +2313,7 @@ static void handle_swbp(struct pt_regs *regs) out: /* arch_uprobe_skip_sstep() succeeded, or restart if can't singlestep */ - srcu_read_unlock(&uprobes_srcu, srcu_idx); + rcu_read_unlock_trace(); } /*