From patchwork Tue Nov 12 16:38:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13872509 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59E0CD42BAB for ; Tue, 12 Nov 2024 16:40:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E36656B0103; Tue, 12 Nov 2024 11:39:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AECAC6B0096; Tue, 12 Nov 2024 11:39:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 703466B00FD; Tue, 12 Nov 2024 11:39:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A774E6B00FB for ; Tue, 12 Nov 2024 11:39:51 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5600D1603F9 for ; Tue, 12 Nov 2024 16:39:51 +0000 (UTC) X-FDA: 82778003616.05.7809299 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf03.hostedemail.com (Postfix) with ESMTP id AD7D220004 for ; Tue, 12 Nov 2024 16:39:29 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Oru17CWM; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=uFishl99; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Oru17CWM; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=uFishl99; spf=pass (imf03.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731429446; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vGD6k8ZGUV4h33x+p3T4Fo15crfC0RACbnBYnZB+C5Q=; b=llPOTfYcpoEYAxcofLVz6Slaybe4piOJFL70bEKdygHEJRRAFCDyLfV6aTxO3S/s2UPPwQ u5tEKMKCPsWpv8KtFW/daD1h+ykyR0OMvSQZiNnDKHd0YujipOIkhfGHgLKizoZmeT/s5l joKw5I/pP+sg9+KVAl0VUdrOxswJkac= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731429446; a=rsa-sha256; cv=none; b=2iksrUhk9c/dceQf5oIeDbxLaP9spsFNKAyGsOFU37lob1KO1izufG33c4ElfJ7Cs4eo6W rVmeuJ6jHwsq3pjxckyUKvdDInJLyDcCCxreLkppEpsKTvQwQBDbuTkgDlL2tsO7cg+7y0 IDmpi9Cqr6kxOqWlJJevGTgpED2swwM= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Oru17CWM; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=uFishl99; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Oru17CWM; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=uFishl99; spf=pass (imf03.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 79E022125C; Tue, 12 Nov 2024 16:39:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1731429587; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vGD6k8ZGUV4h33x+p3T4Fo15crfC0RACbnBYnZB+C5Q=; b=Oru17CWMMNAoa2qCEd8Tw3rDbFwwS5FtalLfzWmc8sbRhSn12+yoGKX6OO+1ot0T+anpVe eOsDO92T2G7TpqrisjSaTDNVHKdUV4+5tPCrBJ39K/+uCywizJNtgSb2CQiT+5uk0TvWed zDFdDWLjBzNA7WjGEKq83Txx4bH1lIU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1731429587; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vGD6k8ZGUV4h33x+p3T4Fo15crfC0RACbnBYnZB+C5Q=; b=uFishl99Dqd2FDrHQvEUljWOvbDB7te1IrlUN+r6T+UrQ1awQa1NaSCMZVkVXcih1TyCe1 k6+LarO+3dEkCzAw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1731429587; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vGD6k8ZGUV4h33x+p3T4Fo15crfC0RACbnBYnZB+C5Q=; b=Oru17CWMMNAoa2qCEd8Tw3rDbFwwS5FtalLfzWmc8sbRhSn12+yoGKX6OO+1ot0T+anpVe eOsDO92T2G7TpqrisjSaTDNVHKdUV4+5tPCrBJ39K/+uCywizJNtgSb2CQiT+5uk0TvWed zDFdDWLjBzNA7WjGEKq83Txx4bH1lIU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1731429587; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vGD6k8ZGUV4h33x+p3T4Fo15crfC0RACbnBYnZB+C5Q=; b=uFishl99Dqd2FDrHQvEUljWOvbDB7te1IrlUN+r6T+UrQ1awQa1NaSCMZVkVXcih1TyCe1 k6+LarO+3dEkCzAw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 5D31F13AC3; Tue, 12 Nov 2024 16:39:47 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id QMxuFtOEM2e6IwAAD6G6ig (envelope-from ); Tue, 12 Nov 2024 16:39:47 +0000 From: Vlastimil Babka Date: Tue, 12 Nov 2024 17:38:47 +0100 Subject: [PATCH RFC 3/6] maple_tree: use percpu sheaves for maple_node_cache MIME-Version: 1.0 Message-Id: <20241112-slub-percpu-caches-v1-3-ddc0bdc27e05@suse.cz> References: <20241112-slub-percpu-caches-v1-0-ddc0bdc27e05@suse.cz> In-Reply-To: <20241112-slub-percpu-caches-v1-0-ddc0bdc27e05@suse.cz> To: Suren Baghdasaryan , "Liam R. Howlett" , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, "Paul E. McKenney" , Lorenzo Stoakes , Matthew Wilcox , Boqun Feng , Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=1640; i=vbabka@suse.cz; h=from:subject:message-id; bh=GNEwsdpTw+alaKMVcf/e8GCDO0WuhBFhaIiXyrGwzUU=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBnM4TI07ZzZ8MVpE6qqbUQruxLEDeCS+rtR9kUE 8Eo0qeSMFKJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZzOEyAAKCRC74LB10kWI moliB/4ku9iAnntGedIvuMOTgOtHGpi71febt1h0Mg0DbTCsyHdWwtgl9EuIjYk1j0G/Yywp7/L TSVrVWBrY1fGHCtF3j2IugkLOrljU6a6uF0EoCBGr+oMRPwGRd3MIWRZ+zZ9na+2GnPqK2hbqMj rT7vQD4S9OMEOY1QHsi0FfQhAH0YzEWV1epnNHABBV1nnJAoerCAPCPirN8Ao/sGN+eVRY6y3Kd UtsszCx2c0yQmT8OdtzJYPeRnl2rWDZfsdVuC00GO8wPx0YLzfRWSidzrAfd0G8lW6MkMQm+kWh WrXADGJR5y8Cx+vF6VoOEsE6KlDfNnGz8pcUZg8dVOt+CPH9 X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: AD7D220004 X-Stat-Signature: re8iqw4hhfwdp8rbotpiz7hbo7tmphyp X-HE-Tag: 1731429569-658333 X-HE-Meta: U2FsdGVkX1936SPbgrxOcLckPimjQh70xhDUQTs+9j5TgJAADpOoY/UkznOKfbUZfxWQXP2Oag4uyHdew26n5ENP1jLPADwYhCJdm9lS4BXh4Nr9LVuAL3Svv5MIwExs9YUTr/bh9lQwJjafowi59sKNHcP+LuBG6V2UVxjeypDunJZWLdsIBmU1asu+1h1Y1hyfn512jBSsQGKdaiGeaufkCzKVaKOeIgU2tCK8ki+z7b9a4WBR0PQSjUW3kveVemeYAPEWUoQRKZKdPZiVw3TxQtsLZeQtRxyBh2GQF5+E5wuzvWlKHXMLZfvQmmKYbDt1IvwN3t/W+yII1ZcoCrlYMmhbJo1UJEGKhlQwXzLg218p26Dzn7WL/8FbBfELZ8EuvyIwOazV3sqdTj9hCxptK9982Y0jTdtdEIcYahr+mvCs3T4LH++3JpJ+HQGZGWOeGZLwqXx2IgCGKat6F0PGfYtuhMIi6NNtsHCcXxoAL3Hae56Ct1enT2zy4mNkw5GWU0cp7RXgFcr6b+SR3ullLkfmC0bVfXPHwsLvUNdWuKNcgZR/C8zN2fsZVpY1zysrPLG8uI1LYx+KhSay7GyG3MmXCRcGjbp5a1P9o35zP5ORo5Zm+YCznRI+43I2PU/FjPWPnu2F9k8eYdju1KJzPuVQZPCPIz6skB++lzgqvKAgQqPHmHugABGbSi+DvNRS/vpRCjD9mxEryWaqseV4L8F4B3BAb4SQyxlaah5ZNjbEnu221PEQeBgrWZZjVoWDFrQpz9S/w2NQvBm1+ZqPEaXLk76qNlSAVKYz57AHO6spQAWHb9TnYzRUXKF6YFzOa0LgcGYPGccd7nkNb4K+JQLdZhQpvK2xfH8BPBJdePjeiIqVKr+DA9KePiVSsK+qi0/plP2ukVKsN75uY4OV/f1mr1PjBbXDr7ZNVkmOuLM4IXtaPRJXpW0Wgw/fvyJknVAhoD5pg9jhHfb WmlfmjyH 4AQvv7nfR07vVPclIvTc94oW1xzw9nH552sp1YJANOwv97VpzZXUWe54Le6Antjs+9TJdLzEbFnqUrZxq9w4vuJJj/mJ6Nh0ViWdWrh7vXKE+F+4+eYYZUx0f46McH6r5x9usp1eZaqZrGo4NySJOhZmEn5UMqv1T3eTYP0uIoqn/+rWa2bsU4ydFVB7iOKWzRT8ngQYcuPtKkvamTCNJzEzCus822EHlBFWyAB3axLVbVhzMAiECqs05fbP631s9Pt/3uhiREEasq3pEuXJrnvv9vNLFaTm4ODYj/JJaEgC+XJHz7lLlOG/1zSn5jGz/cpoOMSlO6/5d0Kz7GORcNM1kAde14cwKLBgxHwu7MQ041BJIocHdCDI0Yg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Setup the maple_node_cache with percpu sheaves of size 32 to hopefully improve its performance. Change the single node rcu freeing in ma_free_rcu() to use kfree_rcu() instead of the custom callback, which allows the rcu_free sheaf batching to be used. Note there are other users of mt_free_rcu() where larger parts of maple tree are submitted to call_rcu() as a whole, and that cannot use the rcu_free sheaf, but it's still possible for maple nodes freed this way to be reused via the barn, even if only some cpus are allowed to process rcu callbacks. Signed-off-by: Vlastimil Babka --- lib/maple_tree.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 3619301dda2ebeaaba8a73837389b6ee3c7e1a3f..c69365e17fcbfe963dcedd0de07335fc6bbdfb27 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -194,7 +194,7 @@ static void mt_free_rcu(struct rcu_head *head) static void ma_free_rcu(struct maple_node *node) { WARN_ON(node->parent != ma_parent_ptr(node)); - call_rcu(&node->rcu, mt_free_rcu); + kfree_rcu(node, rcu); } static void mas_set_height(struct ma_state *mas) @@ -6299,9 +6299,14 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) void __init maple_tree_init(void) { + struct kmem_cache_args args = { + .align = sizeof(struct maple_node), + .sheaf_capacity = 32, + }; + maple_node_cache = kmem_cache_create("maple_node", - sizeof(struct maple_node), sizeof(struct maple_node), - SLAB_PANIC, NULL); + sizeof(struct maple_node), &args, + SLAB_PANIC); } /**