From patchwork Thu Aug 10 16:36:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13349717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01FC4C04FE0 for ; Thu, 10 Aug 2023 16:36:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AEBD06B0075; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A9A6A6B0078; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93AB26B007D; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7A44C6B007B for ; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 2731B1A10CA for ; Thu, 10 Aug 2023 16:36:36 +0000 (UTC) X-FDA: 81108748392.20.FA87886 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf14.hostedemail.com (Postfix) with ESMTP id E611C100002 for ; Thu, 10 Aug 2023 16:36:33 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="b+B/bO77"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=jMSVsxc9; spf=pass (imf14.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691685394; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=Rrrk6AXmKlg6kfpz/5HE6qHTHNnXNplMWbRQ/ZhwN54=; b=0ritzZ9ynBQ0eMRqwtXvgVt7AUYcAGono7oMrKC5VkaDkbZcSfTzT22fES1mG9Wh6DR+jf Mstyb+a5KvmdnKvHYnNUa19o8Cr8spLDEIR8haeUjy69QOR4Zehq5nrkEEhNbaGMtjFtpE IOZnEpGqzVjsa73K1DsRAyAAGeYZeyw= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="b+B/bO77"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=jMSVsxc9; spf=pass (imf14.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691685394; a=rsa-sha256; cv=none; b=Aj1A0iAMVs+AznjqqJN4fkdS3uXo2co6LOw2IwaHWNs6QHxcXd1EZMpYoRrGIYdat91RUk JSWkkjamADqWiPYWn61SMgZQ90ucEUWNxsp0qvNnRv/i8GTfjRF+uQngFwzFbuEstZYIUT ydS2fe5RPZMj6sV8flDhtoun/za84js= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 5F6DE21872; Thu, 10 Aug 2023 16:36:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1691685392; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=Rrrk6AXmKlg6kfpz/5HE6qHTHNnXNplMWbRQ/ZhwN54=; b=b+B/bO774hqjJtmgG2U1kU1fwu3PXO72QjCRfxodkZj80x5FARs431Cl2R5w4bTPenkdyp wia/gyoIWkgkv64jbtwpFLD75iiQ6sQWJu0AiFVEER98L50NmPELQpTop1jGnhTCJ5VXez 70UtOrDd2uexrRVvXZcaYJicuDdbbt8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1691685392; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=Rrrk6AXmKlg6kfpz/5HE6qHTHNnXNplMWbRQ/ZhwN54=; b=jMSVsxc9EUS0W7CDyngB9IDEjhZYVNY3LHcgSZNYtpUkpcVehUzb/4cnYmJb397PiERDZ8 2GywFRdCz0AmQVCQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 2F13F138E0; Thu, 10 Aug 2023 16:36:32 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id TkfPChAS1WSEPQAAMHmgww (envelope-from ); Thu, 10 Aug 2023 16:36:32 +0000 From: Vlastimil Babka To: "Liam R. Howlett" , Matthew Wilcox , Suren Baghdasaryan , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev, Vlastimil Babka Subject: [RFC v2 0/7] SLUB percpu array caches and maple tree nodes Date: Thu, 10 Aug 2023 18:36:28 +0200 Message-ID: <20230810163627.6206-9-vbabka@suse.cz> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-Rspamd-Queue-Id: E611C100002 X-Rspam-User: X-Stat-Signature: ge8hpbows4uyhbto33mfxaudjy6n5rby X-Rspamd-Server: rspam01 X-HE-Tag: 1691685393-827633 X-HE-Meta: U2FsdGVkX1+WchvlCbLwOoaz1q2NAbHEmFCpmTCBXZgIs1lkV54mECSJLWM3KR/R86kXGybq2gZ9IqOWutlH2sGEjNsw2d5Gg3Vchp0pFqkKgDv17XaGDj6tvHuF8J+vE7qZu0QEZE5xLigSWdk+cKxfyq0p/0WiHx0GUXJnj68U2zgPsjhDjB7SnrpvSLeGlmxDDgmPfyl2uN5LY1EHmUdI/fgkEFja6MFGShq4yvBjA2V6hdKJOLvoFsMhqJLGQj2wtTFIw7XN4XZQaa1VI/wq78i53SzE0szEbowdLlHJG2vf+/zCSlNDlNvIajSLr3tw7/CkcBqlM3kqpVWybetYk2/o+y2wLfsFdzFvIH/oCO+/lMbRqDmGh5YSV3Mpfuua7N9drYY11D4hpoMLSbiMuP824ToN80AeGfTwTeJFbR8Cio91Gp9R1K1bAn2d4ymuxx+EC0oQOtk4LTB6Wew/k4q1VUC2dVf7lc6HsksqbxoK/gbEVR2rZZVw/sJrPwYZ2XygjD8Fe/YOTQJsN2BEb233iHsoGoNVnJqT33BeLq2MZSuRTRuoJcNWUcUvSvqFiXbTCR1p06D2jXf9PMEb4w8K/+on8vaMvCIY70uzgksTgB6qoTUnfBkEEkCdYgjdGZwFVMepAk8Zz/G1+MfebZTQLDxKwlTSXjIXqhnC8UY7rfUqeuY+h63GfQnGg0i/z6+hSzCc0bMtQkOtFK3oSUj2/7sQHLlexERC4YU+k2WG5zKDOYGlrxZaLSABpltw4DAswkdwkzCByEr7AGtDC8pzJG9b1Jf0poq3z9VwiksZpDY9pTJ6M7/7rTJXdwPMy/TQdvIpOittEy8pfW/GLGKrbn/LfRKnqmtkqFvRajnTXKb8EILhoLYMFcbi0HlxTsE7dBeWDunLgUc39fFuXIXDbJ8Ib4HuMKdUJvmJYid/GKnU+WNLQGJI7nMi7Ml4tsnXoPPJOvodP50 ahI52JoJ cpBh8Kk+CZ4LpbCV3TDytwUETZTItx9uigTunbYiWBI2ycAfpG8Y579GengwRM+pXFyL9iJnUQh99Rosw+dzgH/OQxVjfAj2lI5TxR1oU1YPUxhehMeigJ0AYQ0C1zImEhlgsSWKndLobHnzKHfDXidoiXPvsnOdEa4/d9jwh70Pcx7mM22AO/YUvscHXgd255+iJA8U4vQkIMr48LcRWLAAPRt0RcwKx8I6MTyHc1XAP09h64bh+1sJ+bQbbCe4qPNrR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Also in git [1]. Changes since v1 [2]: - fix a few bugs - SLAB marked as BROKEN so bots dont complain about missing functions - incorporate Liam's patches, which allows getting rid of preallocations in mas_prealloc() completely. This has reduced the allocation stats further, with the whole series. More notes wrt v1 RFC feedback: - locking is still done as in v1, as it allows remote draining, which should be added before this is suitable for merging - there's currently no bulk freeing/refill of the percpu array, which will eventually be added, but I expect most perf gain for the maple tree use case to come from the avoided preallocations anyway ---- At LSF/MM I've mentioned that I see several use cases for introducing opt-in percpu arrays for caching alloc/free objects in SLUB. This is my first exploration of this idea, speficially for the use case of maple tree nodes. We have brainstormed this use case on IRC last week with Liam and Matthew and this how I understood the requirements: - percpu arrays will be faster thank bulk alloc/free which needs relatively long freelists to work well. Especially in the freeing case we need the nodes to come from the same slab (or small set of those) - preallocation for the worst case of needed nodes for a tree operation that can't reclaim due to locks is wasteful. We could instead expect that most of the time percpu arrays would satisfy the constained allocations, and in the rare cases it does not we can dip into GFP_ATOMIC reserves temporarily. Instead of preallocation just prefill the arrays. - NUMA locality is not a concern as the nodes of a process's VMA tree end up all over the place anyway. So this RFC patchset adds such percpu array in Patch 2. Locking is stolen from Mel's recent page allocator's pcplists implementation so it can avoid disabling IRQs and just disable preemption, but the trylocks can fail in rare situations. Then maple tree is modified in patches 3-6 to benefit from this. This is done in a very crude way as I'm not so familiar with the code. I've briefly tested this with virtme VM boot and checking the stats from CONFIG_SLUB_STATS in sysfs. Patch 2: slub changes implemented including new counters alloc_cpu_cache and free_cpu_cache but maple tree doesn't use them yet (none):/sys/kernel/slab/maple_node # grep . alloc_cpu_cache alloc_*path free_cpu_cache free_*path | cut -d' ' -f1 alloc_cpu_cache:0 alloc_fastpath:54842 alloc_slowpath:8142 free_cpu_cache:0 free_fastpath:32336 free_slowpath:23484 Patch 3: maple node cache creates percpu array with 32 entries, not changed anything else -> some allocs/free satisfied by the array alloc_cpu_cache:11956 alloc_fastpath:40675 alloc_slowpath:7308 free_cpu_cache:12082 free_fastpath:23617 free_slowpath:17956 Patch 4: maple tree nodes bulk alloc/free converted to loop of normal alloc to use percpu array more, because bulk alloc bypasses it -> majority alloc/free now satisfied by percpu array alloc_cpu_cache:54673 alloc_fastpath:4491 alloc_slowpath:737 free_cpu_cache:54759 free_fastpath:332 free_slowpath:4723 Patch 5+6: mas_preallocate() just prefills the percpu array, doesn't preallocate anything mas_store_prealloc() gains a retry loop with mas_nomem(mas, GFP_ATOMIC | __GFP_NOFAIL) -> major drop of alloc/free (the prefills are included in the accounting) alloc_cpu_cache:15036 alloc_fastpath:4651 alloc_slowpath:656 free_cpu_cache:15102 free_fastpath:299 free_slowpath:4835 It would be interesting to see how it affects the workloads that saw regressions from the maple tree introduction, as the slab operations were suspected to be a major factor and now they should be both reduced and made cheaper. Liam R. Howlett (2): maple_tree: Remove MA_STATE_PREALLOC tools: Add SLUB percpu array functions for testing Vlastimil Babka (5): mm, slub: fix bulk alloc and free stats mm, slub: add opt-in slub_percpu_array maple_tree: use slub percpu array maple_tree: avoid bulk alloc/free to use percpu array more maple_tree: replace preallocation with slub percpu array prefill include/linux/slab.h | 4 + include/linux/slub_def.h | 10 ++ lib/maple_tree.c | 60 ++++--- mm/Kconfig | 1 + mm/slub.c | 221 +++++++++++++++++++++++- tools/include/linux/slab.h | 4 + tools/testing/radix-tree/linux.c | 14 ++ tools/testing/radix-tree/linux/kernel.h | 1 + 8 files changed, 286 insertions(+), 29 deletions(-)