From patchwork Tue Nov 12 16:38:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13872506 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 530CDD42BAA for ; Tue, 12 Nov 2024 16:39:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4610E6B00FF; Tue, 12 Nov 2024 11:39:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 375576B0104; Tue, 12 Nov 2024 11:39:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 129CB6B0102; Tue, 12 Nov 2024 11:39:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9C56F6B0096 for ; Tue, 12 Nov 2024 11:39:51 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 50087C0187 for ; Tue, 12 Nov 2024 16:39:51 +0000 (UTC) X-FDA: 82778003280.10.3222AB1 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf29.hostedemail.com (Postfix) with ESMTP id 5964D12001F for ; Tue, 12 Nov 2024 16:38:52 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=kvkqbo5c; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=GSk0LM88; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=kvkqbo5c; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=GSk0LM88; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731429501; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SpHz+sHwXWPoDofFQG0LgmUZpXU7k9OQyeDihc++kFA=; b=OEd0dK2OQsL/h82gKowPupaz8KZT3g2NPRFdl29jugPTx/DXxzWX0pGpozn08DfPuK5p4C qdmWNfNoNPbbaUkTBjrTY7wErUx2qTNrTH+6/L1JV2f0aeCWO2IaS8++6jwqhwjpRzrGkT T6sGw3o27T9cW7Zdk7+gDGkswt+OuR4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731429501; a=rsa-sha256; cv=none; b=FWlh8XxzBzVc6MAbTre9eTfG8rmrMZAfQVDyJ7XrTGpMKD/6VYFFzM3RII2RdnZBdQ+xX6 kHfwa8dhHPNONuoXvLZ3B04Euv0afmun5HB5xS1B7+dnDJJa72n736BjiTSsw0bNNAYy0M nybYat5DYzsJPSVyH0AWYBnsHChjOJw= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=kvkqbo5c; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=GSk0LM88; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=kvkqbo5c; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=GSk0LM88; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 949F12125D; Tue, 12 Nov 2024 16:39:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1731429587; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SpHz+sHwXWPoDofFQG0LgmUZpXU7k9OQyeDihc++kFA=; b=kvkqbo5cMpaS3GfGvzzfDZeh+huJRJkRuyNTVHynFI3Wo3g/2JgvdqVxLB4rHPUH9bGXmc LGj99fzR2/M42vpnh18UZX5tpMRXeQ8PseNL0T8Tzgm5XW+SBq3u7hOMQhop7yq/9hxmkB mElK9vvBzv4eqNzo5RbP85Th6L4sHr8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1731429587; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SpHz+sHwXWPoDofFQG0LgmUZpXU7k9OQyeDihc++kFA=; b=GSk0LM88ofSjfpcfJlljrgmXshtZbauPYbiegzNpaRcSfQUrivxtQ8Pwup+l5w6xVlXNsX 8FBJqKCoLAxDG4CQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1731429587; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SpHz+sHwXWPoDofFQG0LgmUZpXU7k9OQyeDihc++kFA=; b=kvkqbo5cMpaS3GfGvzzfDZeh+huJRJkRuyNTVHynFI3Wo3g/2JgvdqVxLB4rHPUH9bGXmc LGj99fzR2/M42vpnh18UZX5tpMRXeQ8PseNL0T8Tzgm5XW+SBq3u7hOMQhop7yq/9hxmkB mElK9vvBzv4eqNzo5RbP85Th6L4sHr8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1731429587; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SpHz+sHwXWPoDofFQG0LgmUZpXU7k9OQyeDihc++kFA=; b=GSk0LM88ofSjfpcfJlljrgmXshtZbauPYbiegzNpaRcSfQUrivxtQ8Pwup+l5w6xVlXNsX 8FBJqKCoLAxDG4CQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 77C1113301; Tue, 12 Nov 2024 16:39:47 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id sOHxHNOEM2e6IwAAD6G6ig (envelope-from ); Tue, 12 Nov 2024 16:39:47 +0000 From: Vlastimil Babka Date: Tue, 12 Nov 2024 17:38:48 +0100 Subject: [PATCH RFC 4/6] mm, vma: use sheaves for vm_area_struct cache MIME-Version: 1.0 Message-Id: <20241112-slub-percpu-caches-v1-4-ddc0bdc27e05@suse.cz> References: <20241112-slub-percpu-caches-v1-0-ddc0bdc27e05@suse.cz> In-Reply-To: <20241112-slub-percpu-caches-v1-0-ddc0bdc27e05@suse.cz> To: Suren Baghdasaryan , "Liam R. Howlett" , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, "Paul E. McKenney" , Lorenzo Stoakes , Matthew Wilcox , Boqun Feng , Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=3057; i=vbabka@suse.cz; h=from:subject:message-id; bh=dcUmI/Co/tcz1EOGwJFhsClJvt8T9OCwC5fGPuq0Ncw=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBnM4TLe/L4WzzKctMjTmceg/hAmTUewfV1I8r17 2hD6G0cBk2JATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZzOEywAKCRC74LB10kWI mqaxB/9mPs/2SlMtyHeKo4Yp33+gLG5rO3/xEXoM4p8PzNzwd9kDkeGh9LvjpkeZFOfiK9ajRln YHzQzS+l2oB3kBbQP6QQIXITaLRl1Y3m00P1hvonUOrwvhq6IShaqFtYIDX4ZF7sxqHgYW9bwN5 BDkfyktC03/RrtuSROT7ImNG9Fw9ZVaWv1EftiuQa8wf7pYnW/1trRWsCZHTkzU9CYR7gbDiFiq ppE7t4i37j6h9RR5VTjBmtfSDosFmUrtVeLQffEUtbBj+V6hibYXUrXZHkUehovaRTRu44a0apX A8dlvsDgf0w/EcVXcpR2gEqQyrSv5vK8QIXpJckNd7qpLeYs X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspamd-Server: rspam10 X-Stat-Signature: rz4i935jmz5dwcmxspt59u4kiqrsynjy X-Rspamd-Queue-Id: 5964D12001F X-Rspam-User: X-HE-Tag: 1731429532-202017 X-HE-Meta: U2FsdGVkX19349ztpLT0FGPQK/K1/p9OtYg1ICMCAfwfS7q7hIh0q3TqvWlEn16xTUSt+C3fRs/v8/AF58CW6Lgagt6Ds5YF/zDNGJQPS16recnj7IIHb4wo/DQQZaF4YwgdHueqsNI1clG0kQETtkJlRCjlSyR4TL9V0gSE4KJlalGL9ovGl+IOTP0wauUCybPCfXIdTo5juVhayREzQ985jOU3fO4y0e8ITBNsofFY7cnZVYNYltVhUc7rK85pvlJ7myTboAE8g21x0R/UsyBxVq3JglnH2N1IoNTIq1k7C04K78fhlTpXNL0CyvtrT5HhQAdQvfqi4yHCXcHa1TloTKnvdHX2UHWazp145umTi2caJwNZHhp75A0jVBw9qWmTkvXWYSYi8vMwOWjynzRAipWZkxSSsVjBu1TAu+eO837MGeQ+08f77y3FFCxojtEwR/h/eT44LnZpq3XDisEySmBGA2uoWB0obHIKicStJwoBAxvuDwFNbVqLLuWthIVXvBWv98hTZj6X9yEq4Z6Sl+tKBsX4igyelixHoC1KMjWFdlVNNEiucBlBlhs267EdYc7id6WkI0/Px22xoTZYBMc60f7fJQDaaCrUeOhxyrsJnYSk2SXa89+PB83GzFjQiXnFCSE3dvzliycLa+enCNQJsNfHOhG3uki41ZYP152iFqNVrSLe+u1xVJbPPry/1IguEjZ3iB5LX4vwda/QsnZUmb/+UvzqEEdus6yP+SPAZDuphng3V13uMPjryx659kdeqwJdY7cEtz4oDOoP+JMv0vp0usx0URQebxZ/Ew544MPx8a8f6bZI16hV3xOW6s86L1Gzq0wYch7N2wuCjx5PxFqjRZfO/KzYUc4wRhcGa1IqmHzZitaPBOLWDpajlpzoxC/S4maa4PYn2t0JWjp++VC4samVlKne0ogMkS31HZAhDvHmeL4sEZulQvd4+l8fmuXBi/jdFuB o8I6PWQL Bi3UY1tc55TYrnrUnk7zuH6PWAP92eNJ4+2hPM5evIqTYFSCalZwcXe/TxZDy+a2Sb27rlnpkNuuR+aKXOczbXtFgZdpuI9xPPXCFjDWfAB0/BQ1VQyySnVMcOPTMBHF4EdbqRZ2BdcqbvdiFUWDxMxmHtYkYeEu++qYjukGrPDOEtN/PPKq7+MECrKsKj1mO1fxmSPz+ryNb5MSc+JsRMA4+Lt16SiuypXugNhsfWz+rSNUMT4Aw/PKMrNyn/klIYTQJtWOhknJmPDZeI9p6/yPBFMbvjbfMZwJdQZnWAkuaPhmYN/BwKliRb4svRBeQgFRSbDyXx0a34DDySSGRm2iS4pYSGZ2Qr73WesMs+Xt36e4DCOlygervEA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create the vm_area_struct cache with percpu sheaves of size 32 to hopefully improve its performance. For CONFIG_PER_VMA_LOCK, change the vma freeing from custom call_rcu() callback to kfree_rcu() which will perform rcu_free sheaf batching. Since there may be additional structures attached and they are freed only after the grace period, create a __vma_area_rcu_free_dtor() to do that. Note I have not investigated whether vma_numab_state_free() or free_anon_vma_name() must really need to wait for the grace period. For vma_lock_free() ideally we wouldn't free it at all when freeing the vma to the sheaf (or even slab page), but that would require using also a ctor for vmas to allocate the vma lock, and reintroducing dtor support for deallocating the lock when freeing slab pages containing the vmas. The plan is to move vma_lock into vma itself anyway, so if the rest can be freed immediately, the whole destructor support won't be needed anymore. Signed-off-by: Vlastimil Babka --- kernel/fork.c | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 22f43721d031d48fd5be2606e86642334be9735f..9b1ae5aaf6a58fded6c9ac378809296825eba9fa 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -516,22 +516,24 @@ void __vm_area_free(struct vm_area_struct *vma) kmem_cache_free(vm_area_cachep, vma); } -#ifdef CONFIG_PER_VMA_LOCK -static void vm_area_free_rcu_cb(struct rcu_head *head) +static void __vma_area_rcu_free_dtor(void *ptr) { - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); + struct vm_area_struct *vma = ptr; /* The vma should not be locked while being destroyed. */ +#ifdef CONFIG_PER_VMA_LOCK VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock), vma); - __vm_area_free(vma); -} #endif + vma_numab_state_free(vma); + free_anon_vma_name(vma); + vma_lock_free(vma); +} + void vm_area_free(struct vm_area_struct *vma) { #ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); + kfree_rcu(vma, vm_rcu); #else __vm_area_free(vma); #endif @@ -3155,6 +3157,12 @@ void __init mm_cache_init(void) void __init proc_caches_init(void) { + struct kmem_cache_args vm_args = { + .align = __alignof__(struct vm_area_struct), + .sheaf_capacity = 32, + .sheaf_rcu_dtor = __vma_area_rcu_free_dtor, + }; + sighand_cachep = kmem_cache_create("sighand_cache", sizeof(struct sighand_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU| @@ -3172,7 +3180,10 @@ void __init proc_caches_init(void) SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL); - vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT); + vm_area_cachep = kmem_cache_create("vm_area_struct", + sizeof(struct vm_area_struct), &vm_args, + SLAB_PANIC|SLAB_ACCOUNT); + #ifdef CONFIG_PER_VMA_LOCK vma_lock_cachep = KMEM_CACHE(vma_lock, SLAB_PANIC|SLAB_ACCOUNT); #endif