From patchwork Tue Aug 8 09:53:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13345910 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8B65C04E69 for ; Tue, 8 Aug 2023 09:53:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 08D7B6B0075; Tue, 8 Aug 2023 05:53:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E1B3E8D0005; Tue, 8 Aug 2023 05:53:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC4366B007D; Tue, 8 Aug 2023 05:53:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7E8B16B0075 for ; Tue, 8 Aug 2023 05:53:56 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4F795120ACE for ; Tue, 8 Aug 2023 09:53:56 +0000 (UTC) X-FDA: 81100476072.07.37D44DA Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf18.hostedemail.com (Postfix) with ESMTP id 07C6F1C001A for ; Tue, 8 Aug 2023 09:53:53 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=JuHIK5yA; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=8lR8OBOO; spf=pass (imf18.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691488434; a=rsa-sha256; cv=none; b=IGJuG1Obub46w+vqya7LBtqHrEDTfy/C4Qc7W+4TJHZXQH7HOxYgSR/f9/NIdUyLytFubE vSjASDfFatCWA3/M6TreolbyhsZpFrlVnNHfQERLV6RElmOhDbnKnrTUH2SjVH+XSRbOXH Pdd/5TbYoTH9GJh0EIEAUoIWufsgyL0= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=JuHIK5yA; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=8lR8OBOO; spf=pass (imf18.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691488434; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X+E0unmCYoA/5q96uLhFby2/QU/JbMexzKmCXmErmbw=; b=M4ZMqyY+wlskAgH1sDiJd2qnnUsgxMab3FbbB8E3Z+Ae838WyXMiZHos1YdwrtPRzTxmD2 wSO8eWwpP2mcKTP5VPYUnLg2Draf9xI6UNsRXN/LRRMnSfTENh5Ktjx3bJ1QnVk/08Xucn nzgrRTq8zwSz8PxzQ0v66EsircNH584= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id BB09221C21; Tue, 8 Aug 2023 09:53:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1691488432; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X+E0unmCYoA/5q96uLhFby2/QU/JbMexzKmCXmErmbw=; b=JuHIK5yA07vWudODvORFefuFMGecT3WRUyFcG8HubBRT64OpJEFUdDEl52fFQ6gz4H90kv ISXwJNf7MbV3Wd1UHDOlLh6ux3vyjmJtY8aOpXZlfV2j0fAloFWTumEgSdimGKJIpjMF0J glyqRd0MduDW1F5GfTB24ayFXh4xtUo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1691488432; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X+E0unmCYoA/5q96uLhFby2/QU/JbMexzKmCXmErmbw=; b=8lR8OBOOUrNsZP5iXkEKBSPRcGaYP8iGo6k7WS/5Qv5TNNAqaGgIxraAPd11BDjhyHllGM Q5rHU4gmYa6PUnDw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 8F110139E9; Tue, 8 Aug 2023 09:53:52 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id +FJRIrAQ0mSBJQAAMHmgww (envelope-from ); Tue, 08 Aug 2023 09:53:52 +0000 From: Vlastimil Babka To: "Liam R. Howlett" , Matthew Wilcox , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev, Vlastimil Babka Subject: [RFC v1 1/5] mm, slub: fix bulk alloc and free stats Date: Tue, 8 Aug 2023 11:53:44 +0200 Message-ID: <20230808095342.12637-8-vbabka@suse.cz> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230808095342.12637-7-vbabka@suse.cz> References: <20230808095342.12637-7-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 07C6F1C001A X-Stat-Signature: d3xicf1p4ww6nd58434yu9sqpq4uuejo X-Rspam-User: X-HE-Tag: 1691488433-520290 X-HE-Meta: U2FsdGVkX18ooh1cIkUz38q6q0xqYFAynK9NzfmNdbim3Lb2Jt+isQ3Bm6RgMgzh2BJMjK8z3LVZm2cGI3iYzbMEzj1cL58/LZJNFjjnDQSmIQTuBVGhtXNcJoIsONGPsvRJK9RxyOFvWD3gh5RX9KfMi5GppWpGiiPFKe1BOZWWtJEvmO0fIojgeLF5cR6RLC51Gq+EBV+r2jd+0r+F8ObsMPrrW5stfJ4G7g7ks+1nZxenXWHV1cRgc/JkeZmN+RKcH3UNqiZmzelm303Y6kJOP38diGtn6zghVh/AY5zcG1bfoA6GRV043iGavL+kGbDohWyjRSD8yd+ZJQKajzpdgbb8+KXymlzFzo8Tz7tIawM8KRCAM3/2CJQkq+dg8yWP3tG3ln6TLhjLRuBho5eNLjPl7uuYUQVacdeUnoXchPPw47bjEnImjjThsgKYNoNnJb36uecgf7PSE8iaiqSmx7J977bLe3GoywcFMNp8NYUNnq/B5hoP231ydAtCRVbDhgptkXJhRDstxmJlmyiT7rgQcpP8HCoqTlhPHVcWNqHfnmVCRy0XFZjcW+Hv7AdkfU75eY+Ie0hf1gtvSC7BdI6qfUFvwahkEA7W8vSVrqI35ot0HAoLXjObTBdUb8WzgkuCOUpKSDShEdrmqMRj9uz5HWXiEhasZJVAbb81GQi/PfhsaFJFau63zbZLxwdxU/huUac2BNCJQ4R47+PtuSPNTMe/hqBTr8oRpHWFlVmmrtVSiyCvbqt1iS/TJ6JINcKMXF62glewEHmotK8FHMJCayKw2sgJnBt9WKWxyZobzD6BR+I1X3XEIenYQnjB+zdFTPdCqeIHeMWLmA0D/d2n/vuhcU1sMpbDy3oBjBJE7RZcH/Xh16hR5e0RYFs7tt0EJTMb+q/lbLCBn9mHGsyGfDGiTZb9WRQWB4LvSnjUXLdqMyOgNAKSWvLQMyopobrjny8cekM7PUt Pjc0p8sZ fd0q+73vw1ucZShxWZMzcOvja5yevmgwFazs/2z7Nzc/+VueTW+BvKv1xsnQ8/iAquPUObGJC5+5yrXVuO0Bpma+ZJ304zEMUPV9C6FBkAsIUeibOG/0cmoh9p8HYg+NhroEzlgPC9YK0uCXf2vJeP6WYkAXfZiPJQvZHfIvIo8dXClUIylgNzAk8lAfsJoagTpdTDooWZAgliC9FFN58sQ7CGZEKzN87bIqqDEK5naGw+ixemCyvkkH25kgdjGmJHYQ1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The SLUB sysfs stats enabled CONFIG_SLUB_STATS have two deficiencies identified wrt bulk alloc/free operations: - Bulk allocations from cpu freelist are not counted. Add the ALLOC_FASTPATH counter there. - Bulk fastpath freeing will count a list of multiple objects with a single FREE_FASTPATH inc. Add a stat_add() variant to count them all. Signed-off-by: Vlastimil Babka --- mm/slub.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/slub.c b/mm/slub.c index e3b5d5c0eb3a..a9437d48840c 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -341,6 +341,14 @@ static inline void stat(const struct kmem_cache *s, enum stat_item si) #endif } +static inline void stat_add(const struct kmem_cache *s, enum stat_item si, int v) +{ +#ifdef CONFIG_SLUB_STATS + raw_cpu_add(s->cpu_slab->stat[si], v); +#endif +} + + /* * Tracks for which NUMA nodes we have kmem_cache_nodes allocated. * Corresponds to node_state[N_NORMAL_MEMORY], but can temporarily @@ -3776,7 +3784,7 @@ static __always_inline void do_slab_free(struct kmem_cache *s, local_unlock(&s->cpu_slab->lock); } - stat(s, FREE_FASTPATH); + stat_add(s, FREE_FASTPATH, cnt); } #else /* CONFIG_SLUB_TINY */ static void do_slab_free(struct kmem_cache *s, @@ -3978,6 +3986,7 @@ static inline int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, c->freelist = get_freepointer(s, object); p[i] = object; maybe_wipe_obj_freeptr(s, p[i]); + stat(s, ALLOC_FASTPATH); } c->tid = next_tid(c->tid); local_unlock_irqrestore(&s->cpu_slab->lock, irqflags); From patchwork Tue Aug 8 09:53:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13345909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36EF7C001DF for ; Tue, 8 Aug 2023 09:53:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D33258D0001; Tue, 8 Aug 2023 05:53:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CE28E8D0002; Tue, 8 Aug 2023 05:53:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 912528D0001; Tue, 8 Aug 2023 05:53:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7970F6B0074 for ; Tue, 8 Aug 2023 05:53:56 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 42AFB1A05CA for ; Tue, 8 Aug 2023 09:53:56 +0000 (UTC) X-FDA: 81100476072.05.A28E280 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf27.hostedemail.com (Postfix) with ESMTP id 4963340011 for ; Tue, 8 Aug 2023 09:53:54 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=V2Kbm2lI; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=lLwi1ysL; dmarc=none; spf=pass (imf27.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691488434; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TBu9LYlHJ8OjXikD6cSMhoFhFKNv3djoLniF9L00kCU=; b=xZ2gH4tVHgbrAtFXTLvz3IRAtLHudLlqHFzVevIg5x0ZcQL/mTiO26WXHWEyjmnb1I9jBk uTbooD6n3v+bO78CJGU5Sz0GkB+Gh4Emr/jMsyC/tMtqZr/D5nyr3+/1V3C4VqyI2RfWhy UOtF8dHh3DMkroh/rSgiwAgiRUHVrDg= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=V2Kbm2lI; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=lLwi1ysL; dmarc=none; spf=pass (imf27.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691488434; a=rsa-sha256; cv=none; b=tdaMWPrcN5V5+59lqf/khkCPQdoLKZ7W8cngWi8Yh5GN9QMnEw4NBKxPYuoAmEybmnKYRN c83MEZ9gFG8jUcevJGtpk6ciBSzdhwwol1SpisYKFMASKId/g40Hdr0A2/TWbhT/EKO0FW PPKK5FKBq+YkVrHxS0OTiJF80ROcVmY= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id EB1E620318; Tue, 8 Aug 2023 09:53:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1691488432; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TBu9LYlHJ8OjXikD6cSMhoFhFKNv3djoLniF9L00kCU=; b=V2Kbm2lIqct91r38K4Ct7TQaHhljTuhoYVMpRcAt3cKRaekudh4nhIcpG85OITNPFcOyyB 9ny32bzS3cJ9kFOEpuYoc/5Az+gqs0imKKtm2YkXP1T1/JudJTJ8/RU4aO2ZNKp10qyzdy 0cT/eGNDsn8TY4oRAkOVz4/R6bWkim8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1691488432; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TBu9LYlHJ8OjXikD6cSMhoFhFKNv3djoLniF9L00kCU=; b=lLwi1ysLAUCxBlvGR5/1aEpWdfJUdaAtB0i8tJnQ/pz/b9FbcFpkHWfHfHfiX6579mtNR+ sIcdcA5cPJNiLFAw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id BC8AA13451; Tue, 8 Aug 2023 09:53:52 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id mIhcLbAQ0mSBJQAAMHmgww (envelope-from ); Tue, 08 Aug 2023 09:53:52 +0000 From: Vlastimil Babka To: "Liam R. Howlett" , Matthew Wilcox , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev, Vlastimil Babka Subject: [RFC v1 2/5] mm, slub: add opt-in slub_percpu_array Date: Tue, 8 Aug 2023 11:53:45 +0200 Message-ID: <20230808095342.12637-9-vbabka@suse.cz> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230808095342.12637-7-vbabka@suse.cz> References: <20230808095342.12637-7-vbabka@suse.cz> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 4963340011 X-Stat-Signature: s4a4icgzwx6uy4hs654o66iofw9cumrw X-HE-Tag: 1691488434-924899 X-HE-Meta: U2FsdGVkX1/hascs3q8+wNatX+FFKBsJKnA0zsFg0s+y84SexJLL5aAYgBBsWph4l5JC0nJ7Q/LjJlJcak9kh7yRf4j1dXjfholzyh8SGpLP2Q04gAuTZcHG80KmCiG8jrxDYIh8v9Av55patN3CU7hwKD2bU7J8F4mnuievB+gFDRNqxeINw7XouAV0kgeg/UxnxCdquT2F0+rLMQsMg4NIvlhsKwZfd8HDnoN1qZ/YrvJssoTAqeqYVEAdrsuD1uJgh+xPNM8KiDMoSDYFzsrhnluSeQzwiazf2hXZovWZWkKRMvq3/wIRJ3eM94286jMGCSOAK9KRWoXsxLejzrZyf2coUDKouoSQaY6v+fFbsErDjrQnsWVcxFXnY9LrEbje43/sP+LUK8qeqd0exROfCzJqfjwy7LtqSxmCHVFat0F3207sNicbEWKLpDLtzXgjrql6pa0dYNsHTqxnjihFsGZqlNhq4Ro+kNechwsJJbIGQcJdz7KHv/oYMpYzeefFerF+C9/y8Yvl03YPLEDadQny5hCnWS/q0iGrw232TTunjEDCjdicIU2gpOJ9wj38mIhT9ma1WbiglhzNxIYC/CzJ+eyBbDOS4l4WzxlkSXFFxHtrv3Vif7/yKZqUrUBMnvNSfXyL5mSdZryMI2QHD0YYYgfkGNZtv8aVlVObQpxpxExTW5WjMOnVzoBD5/I1S3E1e5l+qPq7VzgHuVe3IUj+xqMWa8L7Tbz0uLWf2AWJZg7QzmzcFs3YAZUoxELQzaHrQZMRDvRJIZbUZGsIzffSCGyzT87gTLk6Q7CHd6CXcDJDPuNHCZQcH2RncifkCJWdLn4xYDk7IWGtpEW+X53ltUCJNSHcoO8NO/Zt2b7X96SH1dmR7f4YbmReqGpaUUT3MxlEUOzjE7+Jtv2uR8TdicNUbzRXJWnuBtN8VAXiVDSkvb6Nzg+CRrIzm5gLkvN94ecXKJafrVD Rcymsb/H m4UAP7huQYVon6jtfuZ5vzndOAo90SJvuDLVwjibxc0gb2F2aU5LHBIefHD0GJHJ80kXitkntqf7Nh+mcbgCFqwPVRz1k02mgStJWdoBPWXBYPErrfmjF336kHuh+et4i4fiVOv6HXm+w/uzg4YgYCPFxOxXkzoZaKcjt2FT77MXK+wdGAMj69TpoW/oc9NvXkJcktR9VsCYfpOTUjJgq6gWWp1yadgLiZTDoPnGtd0jo3mQdY5CoY3lJu/8lfshnPxLin/wur2kEHuvNl59wzGnCKRjyqduUG1KvVhLKysiU+A24ffCGiONTow== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: kmem_cache_setup_percpu_array() will allocate a per-cpu array for caching alloc/free objects of given size for the cache. The cache has to be created with SLAB_NO_MERGE flag. The array is filled by freeing. When empty for alloc or full for freeing, it's simply bypassed by the operation, there's currently no batch freeing/allocations. The locking is copied from the page allocator's pcplists, based on embedded spin locks. Interrupts are not disabled, only preemption (cpu migration on RT). Trylock is attempted to avoid deadlock due to an intnerrupt, trylock failure means the array is bypassed. Sysfs stat counters alloc_cpu_cache and free_cpu_cache count operations that used the percpu array. Bulk allocation bypasses the array, bulk freeing does not. kmem_cache_prefill_percpu_array() can be called to ensure the array on the current cpu to at least the given number of objects. However this is only opportunistic as there's no cpu pinning and the trylocks may always fail. Therefore allocations cannot rely on the array for success even after the prefill. But misses should be rare enough that e.g. GFP_ATOMIC allocations should be acceptable after the refill. The operation is currently not optimized. More TODO/FIXMEs: - NUMA awareness - preferred node currently ignored, __GFP_THISNODE not honored - slub_debug - will not work for allocations from the array. Normally in SLUB implementation the slub_debug kills all fast paths, but that could lead to depleting the reserves if we ignore the prefill and use GFP_ATOMIC. Needs more thought. --- include/linux/slab.h | 4 + include/linux/slub_def.h | 10 ++ mm/slub.c | 210 ++++++++++++++++++++++++++++++++++++++- 3 files changed, 223 insertions(+), 1 deletion(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 848c7c82ad5a..f6c91cbc1544 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -196,6 +196,8 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name, void kmem_cache_destroy(struct kmem_cache *s); int kmem_cache_shrink(struct kmem_cache *s); +int kmem_cache_setup_percpu_array(struct kmem_cache *s, unsigned int count); + /* * Please use this macro to create slab caches. Simply specify the * name of the structure and maybe some flags that are listed above. @@ -494,6 +496,8 @@ void kmem_cache_free(struct kmem_cache *s, void *objp); void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p); int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p); +int kmem_cache_prefill_percpu_array(struct kmem_cache *s, unsigned int count, gfp_t gfp); + static __always_inline void kfree_bulk(size_t size, void **p) { kmem_cache_free_bulk(NULL, size, p); diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index deb90cf4bffb..c85434668419 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -13,8 +13,10 @@ #include enum stat_item { + ALLOC_PERCPU_CACHE, /* Allocation from percpu array cache */ ALLOC_FASTPATH, /* Allocation from cpu slab */ ALLOC_SLOWPATH, /* Allocation by getting a new cpu slab */ + FREE_PERCPU_CACHE, /* Free to percpu array cache */ FREE_FASTPATH, /* Free to cpu slab */ FREE_SLOWPATH, /* Freeing not to cpu slab */ FREE_FROZEN, /* Freeing to frozen slab */ @@ -66,6 +68,13 @@ struct kmem_cache_cpu { }; #endif /* CONFIG_SLUB_TINY */ +struct slub_percpu_array { + spinlock_t lock; + unsigned int count; + unsigned int used; + void * objects[]; +}; + #ifdef CONFIG_SLUB_CPU_PARTIAL #define slub_percpu_partial(c) ((c)->partial) @@ -99,6 +108,7 @@ struct kmem_cache { #ifndef CONFIG_SLUB_TINY struct kmem_cache_cpu __percpu *cpu_slab; #endif + struct slub_percpu_array __percpu *cpu_array; /* Used for retrieving partial slabs, etc. */ slab_flags_t flags; unsigned long min_partial; diff --git a/mm/slub.c b/mm/slub.c index a9437d48840c..7fc9f7c124eb 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -188,6 +188,79 @@ do { \ #define USE_LOCKLESS_FAST_PATH() (false) #endif +/* copy/pasted from mm/page_alloc.c */ + +#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT_RT) +/* + * On SMP, spin_trylock is sufficient protection. + * On PREEMPT_RT, spin_trylock is equivalent on both SMP and UP. + */ +#define pcp_trylock_prepare(flags) do { } while (0) +#define pcp_trylock_finish(flag) do { } while (0) +#else + +/* UP spin_trylock always succeeds so disable IRQs to prevent re-entrancy. */ +#define pcp_trylock_prepare(flags) local_irq_save(flags) +#define pcp_trylock_finish(flags) local_irq_restore(flags) +#endif + +/* + * Locking a pcp requires a PCP lookup followed by a spinlock. To avoid + * a migration causing the wrong PCP to be locked and remote memory being + * potentially allocated, pin the task to the CPU for the lookup+lock. + * preempt_disable is used on !RT because it is faster than migrate_disable. + * migrate_disable is used on RT because otherwise RT spinlock usage is + * interfered with and a high priority task cannot preempt the allocator. + */ +#ifndef CONFIG_PREEMPT_RT +#define pcpu_task_pin() preempt_disable() +#define pcpu_task_unpin() preempt_enable() +#else +#define pcpu_task_pin() migrate_disable() +#define pcpu_task_unpin() migrate_enable() +#endif + +/* + * Generic helper to lookup and a per-cpu variable with an embedded spinlock. + * Return value should be used with equivalent unlock helper. + */ +#define pcpu_spin_lock(type, member, ptr) \ +({ \ + type *_ret; \ + pcpu_task_pin(); \ + _ret = this_cpu_ptr(ptr); \ + spin_lock(&_ret->member); \ + _ret; \ +}) + +#define pcpu_spin_trylock(type, member, ptr) \ +({ \ + type *_ret; \ + pcpu_task_pin(); \ + _ret = this_cpu_ptr(ptr); \ + if (!spin_trylock(&_ret->member)) { \ + pcpu_task_unpin(); \ + _ret = NULL; \ + } \ + _ret; \ +}) + +#define pcpu_spin_unlock(member, ptr) \ +({ \ + spin_unlock(&ptr->member); \ + pcpu_task_unpin(); \ +}) + +/* struct slub_percpu_array specific helpers. */ +#define pca_spin_lock(ptr) \ + pcpu_spin_lock(struct slub_percpu_array, lock, ptr) + +#define pca_spin_trylock(ptr) \ + pcpu_spin_trylock(struct slub_percpu_array, lock, ptr) + +#define pca_spin_unlock(ptr) \ + pcpu_spin_unlock(lock, ptr) + #ifndef CONFIG_SLUB_TINY #define __fastpath_inline __always_inline #else @@ -3326,6 +3399,32 @@ static void *__slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, return p; } +static inline void *alloc_from_pca(struct kmem_cache *s) +{ + unsigned long __maybe_unused UP_flags; + struct slub_percpu_array *pca; + void *object = NULL; + + pcp_trylock_prepare(UP_flags); + pca = pca_spin_trylock(s->cpu_array); + + if (unlikely(!pca)) + goto failed; + + if (likely(pca->used > 0)) { + object = pca->objects[--pca->used]; + pca_spin_unlock(pca); + pcp_trylock_finish(UP_flags); + stat(s, ALLOC_PERCPU_CACHE); + return object; + } + pca_spin_unlock(pca); + +failed: + pcp_trylock_finish(UP_flags); + return NULL; +} + static __always_inline void *__slab_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node, unsigned long addr, size_t orig_size) { @@ -3465,7 +3564,11 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list if (unlikely(object)) goto out; - object = __slab_alloc_node(s, gfpflags, node, addr, orig_size); + if (s->cpu_array) + object = alloc_from_pca(s); + + if (!object) + object = __slab_alloc_node(s, gfpflags, node, addr, orig_size); maybe_wipe_obj_freeptr(s, object); init = slab_want_init_on_alloc(gfpflags, s); @@ -3715,6 +3818,34 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, discard_slab(s, slab); } +static inline bool free_to_pca(struct kmem_cache *s, void *object) +{ + unsigned long __maybe_unused UP_flags; + struct slub_percpu_array *pca; + bool ret = false; + + pcp_trylock_prepare(UP_flags); + pca = pca_spin_trylock(s->cpu_array); + + if (!pca) { + pcp_trylock_finish(UP_flags); + return false; + } + + if (pca->used < pca->count) { + pca->objects[pca->used++] = object; + ret = true; + } + + pca_spin_unlock(pca); + pcp_trylock_finish(UP_flags); + + if (ret) + stat(s, FREE_PERCPU_CACHE); + + return ret; +} + #ifndef CONFIG_SLUB_TINY /* * Fastpath with forced inlining to produce a kfree and kmem_cache_free that @@ -3740,6 +3871,11 @@ static __always_inline void do_slab_free(struct kmem_cache *s, unsigned long tid; void **freelist; + if (s->cpu_array && cnt == 1) { + if (free_to_pca(s, head)) + return; + } + redo: /* * Determine the currently cpus per cpu slab. @@ -3793,6 +3929,11 @@ static void do_slab_free(struct kmem_cache *s, { void *tail_obj = tail ? : head; + if (s->cpu_array && cnt == 1) { + if (free_to_pca(s, head)) + return; + } + __slab_free(s, slab, head, tail_obj, cnt, addr); } #endif /* CONFIG_SLUB_TINY */ @@ -4060,6 +4201,45 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, } EXPORT_SYMBOL(kmem_cache_alloc_bulk); +int kmem_cache_prefill_percpu_array(struct kmem_cache *s, unsigned int count, + gfp_t gfp) +{ + struct slub_percpu_array *pca; + void *objects[32]; + unsigned int used; + unsigned int allocated; + + if (!s->cpu_array) + return -EINVAL; + + /* racy but we don't care */ + pca = raw_cpu_ptr(s->cpu_array); + + used = READ_ONCE(pca->used); + + if (used >= count) + return 0; + + if (pca->count < count) + return -EINVAL; + + count -= used; + + /* TODO fix later */ + if (count > 32) + count = 32; + + for (int i = 0; i < count; i++) + objects[i] = NULL; + allocated = kmem_cache_alloc_bulk(s, gfp, count, &objects[0]); + + for (int i = 0; i < count; i++) { + if (objects[i]) { + kmem_cache_free(s, objects[i]); + } + } + return allocated; +} /* * Object placement in a slab is made very easy because we always start at @@ -5131,6 +5311,30 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) return 0; } +int kmem_cache_setup_percpu_array(struct kmem_cache *s, unsigned int count) +{ + int cpu; + + if (WARN_ON_ONCE(!(s->flags & SLAB_NO_MERGE))) + return -EINVAL; + + s->cpu_array = __alloc_percpu(struct_size(s->cpu_array, objects, count), + sizeof(void *)); + + if (!s->cpu_array) + return -ENOMEM; + + for_each_possible_cpu(cpu) { + struct slub_percpu_array *pca = per_cpu_ptr(s->cpu_array, cpu); + + spin_lock_init(&pca->lock); + pca->count = count; + pca->used = 0; + } + + return 0; +} + #ifdef SLAB_SUPPORTS_SYSFS static int count_inuse(struct slab *slab) { @@ -5908,8 +6112,10 @@ static ssize_t text##_store(struct kmem_cache *s, \ } \ SLAB_ATTR(text); \ +STAT_ATTR(ALLOC_PERCPU_CACHE, alloc_cpu_cache); STAT_ATTR(ALLOC_FASTPATH, alloc_fastpath); STAT_ATTR(ALLOC_SLOWPATH, alloc_slowpath); +STAT_ATTR(FREE_PERCPU_CACHE, free_cpu_cache); STAT_ATTR(FREE_FASTPATH, free_fastpath); STAT_ATTR(FREE_SLOWPATH, free_slowpath); STAT_ATTR(FREE_FROZEN, free_frozen); @@ -5995,8 +6201,10 @@ static struct attribute *slab_attrs[] = { &remote_node_defrag_ratio_attr.attr, #endif #ifdef CONFIG_SLUB_STATS + &alloc_cpu_cache_attr.attr, &alloc_fastpath_attr.attr, &alloc_slowpath_attr.attr, + &free_cpu_cache_attr.attr, &free_fastpath_attr.attr, &free_slowpath_attr.attr, &free_frozen_attr.attr, From patchwork Tue Aug 8 09:53:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13345911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5EDAEC001B0 for ; Tue, 8 Aug 2023 09:54:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 320606B0074; Tue, 8 Aug 2023 05:53:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A7888D0005; Tue, 8 Aug 2023 05:53:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E906C6B0074; Tue, 8 Aug 2023 05:53:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A523D6B007B for ; Tue, 8 Aug 2023 05:53:56 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 76866C0278 for ; Tue, 8 Aug 2023 09:53:56 +0000 (UTC) X-FDA: 81100476072.10.7B229B5 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf24.hostedemail.com (Postfix) with ESMTP id 99C91180009 for ; Tue, 8 Aug 2023 09:53:54 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=TM63PQDL; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=PMYk4+ps; dmarc=none; spf=pass (imf24.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691488434; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=a3DtkVdLGQHDYk90N/OXEPjHa8PNLxLzj/qp7wDJMuQ=; b=RLkzkw9D5jJ81SuBffTTIi/bJ6KLs4tJephWbaBOhjl9BvAX3zQnhCD9e6EiM56mrXwB95 4REf0/MXCqyur4terHe7+1GqcRqm6Ays+q7bY0tSi+yaLkGxAgsdOGg0OgMcjKBZt+Awfa LDmkylNdJzw3sj9eGjUD8dpSG3efQI8= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=TM63PQDL; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=PMYk4+ps; dmarc=none; spf=pass (imf24.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691488434; a=rsa-sha256; cv=none; b=YxLQZjyyRKzAUd/e1c+Pr28ZqAW3hQxeaynx4XHWV1Xq7HsUcBPFy0LdS1ZMW3ErubDKLm 6N9SpzHlXs/XbiL7klzdRqwIezaRddLaClytWqxQXC0GAteUiC4kM6ZDoJJiQhXQYpma0p 26jZLPcpuv/C6+cZSwzXZM+5VzFX0Vs= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 244E520319; Tue, 8 Aug 2023 09:53:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1691488433; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a3DtkVdLGQHDYk90N/OXEPjHa8PNLxLzj/qp7wDJMuQ=; b=TM63PQDL1/kV/0eRyfhNJd9mTnr0x7YZSgjc8gu+OShasFlpsvfTd29e6XuZUTzyIuOzKh epAVCCvXUWmkzTZ+F27l6Dqs/VEedrJReg+43nUQbnAOrW9MhnfDcJEo+daT4fyDfyI7+2 tuJoCLZ60xeq+ygkOsdBQLf9xHRywQI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1691488433; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a3DtkVdLGQHDYk90N/OXEPjHa8PNLxLzj/qp7wDJMuQ=; b=PMYk4+pst66z0H1Ard3n4p3Ej4MlqgpLiQdaPFhRHz+PTuyTYA3lk/x+G2Bsq/DLk5blTG bJlASInXsVjsuJBA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id ED5F6139E9; Tue, 8 Aug 2023 09:53:52 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id IP9CObAQ0mSBJQAAMHmgww (envelope-from ); Tue, 08 Aug 2023 09:53:52 +0000 From: Vlastimil Babka To: "Liam R. Howlett" , Matthew Wilcox , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev, Vlastimil Babka Subject: [RFC v1 3/5] maple_tree: use slub percpu array Date: Tue, 8 Aug 2023 11:53:46 +0200 Message-ID: <20230808095342.12637-10-vbabka@suse.cz> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230808095342.12637-7-vbabka@suse.cz> References: <20230808095342.12637-7-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 99C91180009 X-Stat-Signature: idgqkefpn16si73zq614pp9sjh34fmkj X-Rspam-User: X-HE-Tag: 1691488434-631052 X-HE-Meta: U2FsdGVkX1+Dxqcj5NGmr066YDw0IOO56YeONupMJIpCZoHpIfFtvnFSrvuhDMSjP5AoWbBUAczwk7fdpRbNEayqRRY8Jl3AZp/WT6fnns76Kd8OSnUbfEbNNNHSrRLAi43YtZdu+UiKyxgPsrYof3qY/2M9HS6DTiNVtn7mSHbv6mcY2UGVNzCjsTl1hFeNKYKwW5LFPjGXEA3Vrhil8a3JsxBAR/NmBRymBNVCj1JkY3du0f5XW5olx9Z37Y+IfbjXllM4fUufzyBTZr0KpMXXxXKGJkCfugqb7pDgzS4kR8Jmxu4r3j0gI/XLF+dyB44fX+UlF+KXTRDZswFR0I/e14BTOKN2+Ryici4VYkkav+TG+6xKqnlORBQSmUPteIfZ85a8qQ6J/2j9/kfOuHvsGtF+egOL5nLgBNRpls1XwkxcsTAddQXhG2biNUYRKTWKeQDyWOEvoF+M0bZ3GVHv+kx+w6D+ywHYFuK8tTJ/kLLZ4uwSDdEzSyHkdcG3wUaB+Fg3Psn2i1eEv6Jl7mai5VwEoTY6Zdy28KIdGcLTCYGRiRoKU0FYt/zF8330LjXEwoMato6CHJ8LydrdIhmV34UALxZBN9p73Y+dzCI8jv+J8Krv4gNNvEUa0fRMMMYNkA4Lmn6H7Bq/yXU8lAVdNKeNZRS596XNdjtxAFTcyGgVaSG9tT6l2TV3ZksrBRc/V/3OJGQW8hApLB//cy4O0XAchPuse+YxE/B6jEpeerA8JeICdV8htguDUbwriOnVwC2ZevVx1jSMEmP4XKEbYAGNOVPjrr5Z7hbx820QqjSix6+KGgJ/SxjU5GE7MgtKEMiklXLCPyUFY73TCRhys6JiKu+rpGXIkyVzvNd7OnQtgqxop27QZSrzh8K0qFD0pTsGddMDMB72RYYj9TbN3OUmhn06RBJh5KbrHz+i5Q0gyuD8C2vh2J6CyiYSmma13OE6DI8OF3Udk30 kCKtWnzc xJEOzNVrNyUh62Hb2JQaMZXtrUKg9igNn1xTn4Jk0NvNrwEF7aEuFdhJ2ihUFaNnDl3Bk57xe9TQKIzCfDgl1IN4uJ+aBa2tfOLWvH5YB7rG/++wYr6nCAM9BlyTD9NOhzBCenITrMaLQHVMVJHzXHeeSFmPSAqUf1LrrxII7XRGhr0ZENUUhYWcEGmc8qxpkGPoEhv1H0CXmHerngKWnEpz+3BHT2fdy7mRP3Y3NVaJeLTvzjHBkiMeNi9y3aj6PtNabDkkoEzN6wVav0fv/a1eIfkFfvr1pyUBw52j7XHRbxM0MHcUN24/CWw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just make sure the maple_node_cache has a percpu array of size 32. Will break with CONFIG_SLAB. --- lib/maple_tree.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 4dd73cf936a6..1196d0a17f03 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -6180,9 +6180,16 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) void __init maple_tree_init(void) { + int ret; + maple_node_cache = kmem_cache_create("maple_node", sizeof(struct maple_node), sizeof(struct maple_node), - SLAB_PANIC, NULL); + SLAB_PANIC | SLAB_NO_MERGE, NULL); + + ret = kmem_cache_setup_percpu_array(maple_node_cache, 32); + + if (ret) + pr_warn("error %d creating percpu_array for maple_node_cache\n", ret); } /** From patchwork Tue Aug 8 09:53:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13345912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAEB7C001DF for ; Tue, 8 Aug 2023 09:54:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 652886B007B; Tue, 8 Aug 2023 05:53:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5BE298D0002; Tue, 8 Aug 2023 05:53:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 393A48D0007; Tue, 8 Aug 2023 05:53:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 16D958D0006 for ; Tue, 8 Aug 2023 05:53:57 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D69741C9369 for ; Tue, 8 Aug 2023 09:53:56 +0000 (UTC) X-FDA: 81100476072.05.C03DABB Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf03.hostedemail.com (Postfix) with ESMTP id C87D420013 for ; Tue, 8 Aug 2023 09:53:54 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=sMK735vG; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=qySqY3LE; dmarc=none; spf=pass (imf03.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691488435; a=rsa-sha256; cv=none; b=ovidPJy1pwsU0AqNk/BYxrsl7NxUYz319BF5h3aij1yZxuW9zhQ5ztGOIXKSh1r6j7pGEJ miwQ0rqsG2kKwnqMeG+FYrkdxZb7COLLh5EQo8jHkpEZ5lObgiB57ZR+unE5uz2/DtW0U7 HlawAPQDCNxcci6UVYQiFsFzj39t/AY= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=sMK735vG; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=qySqY3LE; dmarc=none; spf=pass (imf03.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691488435; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i6TQwXZ4v9nk5j9z/G1P0QjfrlA/rpwYdzHk2PVBTRo=; b=8d8fDXBfNWHhy+hTA3npDzXBHZJk7hAlVsn8jrkZnM+4S+GnUxNQn8W6m2pM4vqq7GL00d OR5JFa8mHae9P230I5yGXkflYEqQsOaUjkbGbV6lsRTkYUI+HD/iAOmLUDVaf1FznMb4l0 RlNeI7+DHxWi23uCF/xMZ6uBfOMNjsQ= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 5CA9E2248A; Tue, 8 Aug 2023 09:53:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1691488433; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i6TQwXZ4v9nk5j9z/G1P0QjfrlA/rpwYdzHk2PVBTRo=; b=sMK735vGrxWU1VYbx6RUmceJoQbzNocoGXJWDnHnQMuYYl5OwU3eFj+3qgZ8Ba0mFBaNLq Oq7pXw3+pm/6X820956mlj1IpRt+1zQo128JYQOLdIXM+09UMZwOMLvmaDtJEvC7VeLAxw oWjOyz1mC/TZfMdOelpD4y7kBG9k37Q= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1691488433; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i6TQwXZ4v9nk5j9z/G1P0QjfrlA/rpwYdzHk2PVBTRo=; b=qySqY3LEh48iTLSTZF1kLGV3kFsl/U5DLBfmwlVvoTvtGqeP3Z4gT7neP5svnR3Q21mOnH hlN2F/r3TLQ+AODw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 2649513451; Tue, 8 Aug 2023 09:53:53 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id AFq7CLEQ0mSBJQAAMHmgww (envelope-from ); Tue, 08 Aug 2023 09:53:53 +0000 From: Vlastimil Babka To: "Liam R. Howlett" , Matthew Wilcox , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev, Vlastimil Babka Subject: [RFC v1 4/5] maple_tree: avoid bulk alloc/free to use percpu array more Date: Tue, 8 Aug 2023 11:53:47 +0200 Message-ID: <20230808095342.12637-11-vbabka@suse.cz> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230808095342.12637-7-vbabka@suse.cz> References: <20230808095342.12637-7-vbabka@suse.cz> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: C87D420013 X-Stat-Signature: zaiwt1pfss6dtsrdcosmbjt4nqpdwsm9 X-HE-Tag: 1691488434-818392 X-HE-Meta: U2FsdGVkX1+30L5qL8PTAF5TNYzbft2uni6+pNxVYjI8NqBQJv5Uf98XDrUN5A68pfXkLXlZB46q8sTNKTOcb31AYjQlgJOOkB/Rvx7OIzHKyfcePYQmNKd4op5vi+3pKCXMAeTg10RqWs46WZnB1RynL7Zhn7afPVBdrxOr2/N+Pys2QeC7z8t43J86t7nFhFjFhB+/ayjwQ72NBTPBAFU5MgOY2/UllSXXoXr1NcanfAt5zfSALIEtFeQTZZPDkYHKGA+pMfrLKnwg4dbFVPzao0cQrS1kXcC/7X1eHFweSoqtHo9F/tGkmOclUyW0ewUX3KG+M6Ps+4akBPU57BXoGEEwB77ghfwopuKeSuEmnl1XNs7e/cTaZ4o7gT18si+OPXm3z+og550da8KGYaxqfRjfL+Ii4Ls3Nez0P6kD2n4BI64QUviN+RpF9v+6EZfb8v7sDXTVqsKG+2JqYgIEf/vCHV3xBhIS4ONFK+aHoFbz7ruBkpVnH5bIyDC1gsvXmuPCZ8z+UGMun1/pyxRRSevuIj/IyOh7cm1Eg80pGY1rZGeNMk+GgXpnP8kW7eIQhtpariP8axivgfHNLj06wQICSu8CoP7PKRyMmVzM7VCNToLitLuKfQlOZYJ8zk8dkNbpfDYSNgyw+Fq1sH2TK6Y+e0u9QYxTIyhRpChjJ1XtPLGOb7PPbScO/cqknGA8FynMsiMdHRQfTTKasCxnWxN5t3ntBaQkt4m9dG2NtVx+v8cYcmWUJFa9CgSKNdG1VSMELGPzxEnOVEgqw0c3jERMqN4+5K2ZbB9w2zyz+kTiCn5uxhPQSmPFJYb+LVoQZjsBcL0Cmyp0+MEp/OB1Z1ABG6Ruu2UgpW4cd2eZ5RZQ7ezu0EDmpABuOE7hLz/yMsDZaRmsXdJYtbwUkUz2Msok8PXkpXvfrpX3xxLgnPhLX6runuuXYwj4fa8ra7/OoIiC4+WxHzZw9w5 eAfEem2N xbAILIvFO2cmsuKcnIX6Pa1Dxl4GirsD3+DNZxa1KjS4/UqLWLfg1yTT9Cys3H+v9PURTeyuYheRJBBdqHy6Coik+c28WsHByf+pAbmOBx0B0P2oXG1FYS8qH7dleZteoiF9OSssoIub/JsNR3tNEVIf25TUREorvUL1GxpyArBu2B3P1aXQ8q0XqKEiFnl8iNjrjdOf5ZsTj530GYoBgcVk0Ns1bmP/hjNt2cJpU8jpo6io+RJt6rj9LOK849S6ERwdJadRYHfaTxv54Y4rNZhWXXL/v8wFM1eFuA6yUjO1JYul3ni2ATDHOqQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Using bulk alloc/free on a cache with percpu array should not be necessary and the bulk alloc actually bypasses the array (the prefill functionality currently relies on this). The simplest change is just to convert the respective maple tree wrappers to do a loop of normal alloc/free. --- lib/maple_tree.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 1196d0a17f03..7a8e7c467d7c 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -161,12 +161,19 @@ static inline struct maple_node *mt_alloc_one(gfp_t gfp) static inline int mt_alloc_bulk(gfp_t gfp, size_t size, void **nodes) { - return kmem_cache_alloc_bulk(maple_node_cache, gfp, size, nodes); + int allocated = 0; + for (size_t i = 0; i < size; i++) { + nodes[i] = kmem_cache_alloc(maple_node_cache, gfp); + if (nodes[i]) + allocated++; + } + return allocated; } static inline void mt_free_bulk(size_t size, void __rcu **nodes) { - kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes); + for (size_t i = 0; i < size; i++) + kmem_cache_free(maple_node_cache, nodes[i]); } static void mt_free_rcu(struct rcu_head *head) From patchwork Tue Aug 8 09:53:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13345913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08D12C001B0 for ; Tue, 8 Aug 2023 09:54:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 867278D0002; Tue, 8 Aug 2023 05:53:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 806E06B0080; Tue, 8 Aug 2023 05:53:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4867F6B007D; Tue, 8 Aug 2023 05:53:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0BA438D0002 for ; Tue, 8 Aug 2023 05:53:57 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id BD4F11C9275 for ; Tue, 8 Aug 2023 09:53:56 +0000 (UTC) X-FDA: 81100476072.30.CB8487E Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf09.hostedemail.com (Postfix) with ESMTP id C70F7140014 for ; Tue, 8 Aug 2023 09:53:54 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="0ypz/M4h"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=gHwsy0hi; dmarc=none; spf=pass (imf09.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691488435; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=19m9+YAryd9zQqOJKLPS0ScJg55yhuc9Zn7O2SmAkY8=; b=yy8ZngryVYEUGSTyyUMEY1gtA2Vq/9Fhl/NKDLMCz2hUdSzv9okb1vY+ByCtKYMng0wOUz GidXMG0IT+flc6m+AbGwz5QBLUYiORetfiGmyh+OCVJ0SE4PjvNRux2K2+6mrnNd2ZphYn gVsFICyUIXRX1Uma6nkjh0uZKVZhH7k= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="0ypz/M4h"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=gHwsy0hi; dmarc=none; spf=pass (imf09.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691488435; a=rsa-sha256; cv=none; b=yZl12evcd9c9IxzWG5nDYBF5hkuGTnHuAWRo8S9oZLqhlI2PbHAfsLYK0FEsZR/PXyJ3IR 1XOAgGpsu559/RL8Q5Af//xF1hr7Hprh4UL2r/49gS1ZXvzUel1WuiH2I5Q3nfB0y0XFUT Rm0R1hzpv8XObxHj+Cb8hH9bnbf1bt8= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 7C5062248B; Tue, 8 Aug 2023 09:53:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1691488433; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=19m9+YAryd9zQqOJKLPS0ScJg55yhuc9Zn7O2SmAkY8=; b=0ypz/M4hFEYVg3RIkyKTZpGOtO0OELdvePk4KsvNp0QXKtSQ1NTkwvJ6x9bkVpsk5ACEo+ VdhEFCaw7TRHrCXiaOhP4fR0hWTfeBY2vexIA+Fg3uiY/xr+RGb+kCzEL8ZCXD6x0fD/My 6dCsKBwvmBeEB1+gOW5PonTGZTXEndk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1691488433; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=19m9+YAryd9zQqOJKLPS0ScJg55yhuc9Zn7O2SmAkY8=; b=gHwsy0hibpgfl2awLPIMsMc4dKqncic8vqqIN/UioyCepHOo4bgnZRyoxdFoRXKIOTGPXa oVyI5GXsppN4AdBQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 53972139E9; Tue, 8 Aug 2023 09:53:53 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id eKi8E7EQ0mSBJQAAMHmgww (envelope-from ); Tue, 08 Aug 2023 09:53:53 +0000 From: Vlastimil Babka To: "Liam R. Howlett" , Matthew Wilcox , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev, Vlastimil Babka Subject: [RFC v1 5/5] maple_tree: replace preallocation with slub percpu array prefill Date: Tue, 8 Aug 2023 11:53:48 +0200 Message-ID: <20230808095342.12637-12-vbabka@suse.cz> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230808095342.12637-7-vbabka@suse.cz> References: <20230808095342.12637-7-vbabka@suse.cz> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 7qpd5hg1is8zen77m34iismec5bku833 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C70F7140014 X-HE-Tag: 1691488434-66351 X-HE-Meta: U2FsdGVkX1/+6U+Rqh7IE25Fg0EIVd3dGEisW3Y9098cUQC0OxKyML7qlT7o3c/uhZ40Bh72M2Sj2y93tubT2elfowju4kI6EdPa7t69aVqSlX7PjpvEyO8b2pVESkacVgVuBBNgnVLvL9RIjZzsJf+rzd8wlIPiYGYGCUUGM6XTNTSGKsm8AdVZRCiuvwzv1YZsGszD8ELAefo0k2sku4dNdM6DzYo3YxiyXCu8AxODTk24UWCtJG965WK2DkQJHlX6ES/8rmQgDUC9wXhLOJBW19HLMLU7Qw0xVDhBDNlDxNJauad6nCSyGDPahHJx7oJKfA/VRfu/5EfzMpKQGni6m2UB8Po2If3U2eIy6pbw/DhQurLREeNZUeAKo2HIBvrTdwt4p1Ngwp2j/v9sX5GeRYkoxsqB2eIwgMccFfm2MlPKteoQrucUqyqLc44J9RY/4MvF9sie49MqWkghV7yKAVf9O8EpO7GiWqTXpvjDgqzkviZypemjGY7yw8PX82Jls3NW5/V5piPxYlKNf18vFK/5QpsFjKqQQiC8pNqGFcFFAir+/bYaIzLDp4GfTTSMfaJywmuHMlm0DLJlyD2YkS167Etz+uXU4iRByoxNvQT8nZrAB8UcZebXr5zvKpK4UBwtt/4EoHWv5J0+YAmeBJwhcqV5/v9EnDLAKoLYIP2sIvVcvw4yPXBhfJLWjLggHnWeN4RA0CoyYRCBFg0XupoOYed/nQzT58Gr0Cu54hkqrqw5CJHl62+kRb+tJ4vik0jfEm/q0sq5Vpf1AO/VilRbpazCFIsVoIUA3dWHPojwIqNWR2BdbUJxXVxMy/LHK9EbJ4qfd/kfSpncbE8Rvd70ho1O+d2biX9chchygHRp3Es7y6Hj7zsAGKVTmytN+3JZ5nMoAVYQBXbPhdjyN5lxigt1NwKfJoqDPd3BW6NDJkC5JdFreXHqqfuLWFLhzj5oyAPFi8CGY8g 5p7/XA7r aLs1QY1i5w09R5XpX3E9WWutwSL5SRIc4b9C5VQrI44YbFgSCxURSAfaS9uWF3ZAAir+CqzcttJV5AqyftIgjvehOb0fYhFGskTwNaWLrozNJO1s3DNHRp7iAGhq/9eJgAcj7bj8mxhNuym0f+VAZNgEEOPJogf389GMvSRWfD/05c6ZF+hxO/cJxquxzUdp4xENcQR4x8NkXwJLW/q0BbMWWiwcCAno4sCDo4SfgY4eSNLSzLS1/ZTk1mE52EEMFwPrDtMX/K7WI/h0CG/guj9lWoRAWJ5/5JFVCQVwYd+lgtXdcvWwhHeCAfA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With the percpu array we can try not doing the preallocations in maple tree, and instead make sure the percpu array is prefilled, and using GFP_ATOMIC in places that relied on the preallocation (in case we miss or fail trylock on the array), i.e. mas_store_prealloc(). For now simply add __GFP_NOFAIL there as well. First I tried to change mas_node_count_gfp() to not preallocate anything anywhere, but that lead to warns and panics, even though the other caller mas_node_count() uses GFP_NOWAIT | __GFP_NOWARN so it has no guarantees... So I changed just mas_preallocate(). I let it still to truly preallocate a single node, but maybe it's not necessary? --- lib/maple_tree.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 7a8e7c467d7c..5a209d88c318 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5534,7 +5534,12 @@ void mas_store_prealloc(struct ma_state *mas, void *entry) mas_wr_store_setup(&wr_mas); trace_ma_write(__func__, mas, 0, entry); + +retry: mas_wr_store_entry(&wr_mas); + if (unlikely(mas_nomem(mas, GFP_ATOMIC | __GFP_NOFAIL))) + goto retry; + MAS_WR_BUG_ON(&wr_mas, mas_is_err(mas)); mas_destroy(mas); } @@ -5550,9 +5555,10 @@ EXPORT_SYMBOL_GPL(mas_store_prealloc); int mas_preallocate(struct ma_state *mas, gfp_t gfp) { int ret; + int count = 1 + mas_mt_height(mas) * 3; - mas_node_count_gfp(mas, 1 + mas_mt_height(mas) * 3, gfp); - mas->mas_flags |= MA_STATE_PREALLOC; + mas_node_count_gfp(mas, 1, gfp); + kmem_cache_prefill_percpu_array(maple_node_cache, count, gfp); if (likely(!mas_is_err(mas))) return 0;