From patchwork Thu Aug 10 16:36:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13349716 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9A10C001E0 for ; Thu, 10 Aug 2023 16:36:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 630296B0072; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5DFD56B0075; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A7196B0078; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3427D6B0072 for ; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0642CA06FB for ; Thu, 10 Aug 2023 16:36:36 +0000 (UTC) X-FDA: 81108748392.26.BB3ED0A Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf13.hostedemail.com (Postfix) with ESMTP id 07AF420035 for ; Thu, 10 Aug 2023 16:36:33 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=JWeB4JaU; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=LJ6qkOJe; dmarc=none; spf=pass (imf13.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691685394; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X+E0unmCYoA/5q96uLhFby2/QU/JbMexzKmCXmErmbw=; b=arR1YRk7sekzRHoXvXCQk6hsc5vXNBguL/BkLa6jCejyom8yU282LMJMIVs+KOuUqMU6D+ Wo/BltBLipuISbwiHy7CfjamANaU6Xl2DyYalbSrIfSjpVH0Q/UdvZPu7zvZLEw4Iw0Pd4 Aoa3/OEN5TDl8c3DQBQrnZ8l8IEUhMU= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=JWeB4JaU; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=LJ6qkOJe; dmarc=none; spf=pass (imf13.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691685394; a=rsa-sha256; cv=none; b=YRHWYNGHtm0W7M8/MJuDmbDuD5Z+gYu7KsgZ0dbatJEeCRV0wnvpsVC2fffWK+r66/RtX8 eDguSfU2MoYCIf+/cn6+zmajZPyQdBAPmm0LseaPKPCzrlT8HWWUIE2WxMLnx+I2q5shfm 9OqWiJY6o3hFoVXOVaK10hI6RoGkE+o= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 8EF7621873; Thu, 10 Aug 2023 16:36:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1691685392; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X+E0unmCYoA/5q96uLhFby2/QU/JbMexzKmCXmErmbw=; b=JWeB4JaUfgHvZaQeMrzQj2VqCapH0EynaaBfufBlIRxTx2ECYhL9B4n2CuXBu6jHkxgffL uVIy1YZx4Q6tJiGJvsrFSVobLMkg9U3vkSz/at4f5HT90qx28+/qHEIJYIHKswNOLrQsr4 QED/YEf1+onvGmo/vyJGQ/ZssUBn/y0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1691685392; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X+E0unmCYoA/5q96uLhFby2/QU/JbMexzKmCXmErmbw=; b=LJ6qkOJeM0r8nLNtSMjM6Pg9NO37l/Pq3Eu3JIT5fRdJOyJqXGNu1/AgSzrz+6N0a1tK+A kI0gqf+8T9tuSgDg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 61C11139D1; Thu, 10 Aug 2023 16:36:32 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id CIAVFxAS1WSEPQAAMHmgww (envelope-from ); Thu, 10 Aug 2023 16:36:32 +0000 From: Vlastimil Babka To: "Liam R. Howlett" , Matthew Wilcox , Suren Baghdasaryan , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev, Vlastimil Babka Subject: [RFC v2 1/7] mm, slub: fix bulk alloc and free stats Date: Thu, 10 Aug 2023 18:36:29 +0200 Message-ID: <20230810163627.6206-10-vbabka@suse.cz> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230810163627.6206-9-vbabka@suse.cz> References: <20230810163627.6206-9-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Queue-Id: 07AF420035 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: ohz9o6iboyf56kkx3ertzx4czt9ms94k X-HE-Tag: 1691685393-456110 X-HE-Meta: U2FsdGVkX1/nuVDjkk7g8JbJI3SwFbiQzNgRKCnY48kwMBqWldA3qq5ahCxKR1OrFdhaAKjHqeMdnTt3SnAjdN0HFTzCcXjp34Mn72dfpm6Qr31RN+DN75LbcJnV+/0DS0x0Hh/8lPFMgM6kKcaaUo8EoUCGc7gWrY9K8X05bWd9M7a5tlhd9rMFMviPCusBgb5HQmLHLeQlaX3KdDegNp6xHWTwg2cM4RwKaOa+0LlrIBclaoD1ryLI2zll7egg27KtLqx8NO8UJJgOVDeRWZtJgkxKyIwdwlmVHZoeJNbjInrEMyRPkQAg8GqDtAVjbGRhtZNqcqNKnj+9TKtO4TLa14fyGFOqrR8Cevz99wAvDneU+uaa92/WPB5PdrvBSGb1ydKpgJlhXLIRYYl+dVfZLksI7/nXaSttXXlXXt5KNq7CdTn71u1AtMcYo+qSPmwwu2/sJZ+p1M4ai2llHN+f96vIgdc/rP7rChLZeOdAazTY9fSDO/9k2WoSk9m3FsujOK2p32w/x63sV2AOgeb6vY1QIt4DVTFsX5U2NL98qs6fRhw0s/AXQ9EV+MvUvny1BsYinFb9FvZnCvNtldBIvm6D8ko+ryRCDezIwJgPENFuZ6hjkf9tE9De/pDLLgRNme5Kujk+ba6/gt+e9TbIXpdgJYV2RBBtfCV/U0X4P39GcexbZ5AdRbpBwD9u7KpvZZeQFO+USKlSBum2PAwO9Q7tRIr2YI1eOtgWXD5mScS9AQTF0gvY+yDIiD5T4KZrkVs3MSKoqfRkl5fCASDVdVw02xY2I72cbRazPajdkkwOpCrn3NY2Q+e70mRp/6MRC1tOkCYxDh4/3N2jYzdL/MjUTakMoo7gYx/RTVbx3AGxbCK/5/vlW5VYungVxkd0eoZAwnI0bfq5+Fa3cNq4z9MDQpQWUoc96Tnz40umhQRFy7ur0H9yUJr3csKMYyR/5vJ5UXFrSYuSgvz Aa1Os8B2 MWkxBKkumwqaAr/J3IrEesznZKmj4L/XpOji04TVsQ4v1wNiqSYxs5Ow8ektcc3UgcvlUnyLnqk0VWSUgtDWVhu6HEuesIzUHwDLIhpA1IsfkM7mBVhesXfRKz28b4vJsHRlB1cfmwjzt0w1DSJvK1v9y1Zi9IeWRzEw2RFfYbIH9j0WfXNTLXqGlF3OTjz4TQ9hs/6j3C93sAJQYp/hqKN9fK6gC6cHXNZP+iKOq3w4/wXuiIpHuH/XNq6gn3jyKgnXG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The SLUB sysfs stats enabled CONFIG_SLUB_STATS have two deficiencies identified wrt bulk alloc/free operations: - Bulk allocations from cpu freelist are not counted. Add the ALLOC_FASTPATH counter there. - Bulk fastpath freeing will count a list of multiple objects with a single FREE_FASTPATH inc. Add a stat_add() variant to count them all. Signed-off-by: Vlastimil Babka --- mm/slub.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/slub.c b/mm/slub.c index e3b5d5c0eb3a..a9437d48840c 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -341,6 +341,14 @@ static inline void stat(const struct kmem_cache *s, enum stat_item si) #endif } +static inline void stat_add(const struct kmem_cache *s, enum stat_item si, int v) +{ +#ifdef CONFIG_SLUB_STATS + raw_cpu_add(s->cpu_slab->stat[si], v); +#endif +} + + /* * Tracks for which NUMA nodes we have kmem_cache_nodes allocated. * Corresponds to node_state[N_NORMAL_MEMORY], but can temporarily @@ -3776,7 +3784,7 @@ static __always_inline void do_slab_free(struct kmem_cache *s, local_unlock(&s->cpu_slab->lock); } - stat(s, FREE_FASTPATH); + stat_add(s, FREE_FASTPATH, cnt); } #else /* CONFIG_SLUB_TINY */ static void do_slab_free(struct kmem_cache *s, @@ -3978,6 +3986,7 @@ static inline int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, c->freelist = get_freepointer(s, object); p[i] = object; maybe_wipe_obj_freeptr(s, p[i]); + stat(s, ALLOC_FASTPATH); } c->tid = next_tid(c->tid); local_unlock_irqrestore(&s->cpu_slab->lock, irqflags); From patchwork Thu Aug 10 16:36:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13349718 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF0B8C04A6A for ; Thu, 10 Aug 2023 16:36:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DCE8E6B0078; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD37E6B007E; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B0016B007B; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 76C546B0078 for ; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 45BD1410AD for ; Thu, 10 Aug 2023 16:36:36 +0000 (UTC) X-FDA: 81108748392.20.5F961E5 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf26.hostedemail.com (Postfix) with ESMTP id 14D5614001D for ; Thu, 10 Aug 2023 16:36:33 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=axrt0kxg; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=g5qxQbbi; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691685394; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4EGfAOQz6XY1pXT3KWDaIKebAxzMbhko7rkj0tHhhE0=; b=L0Mp6vkQ34yy1WxCStaT22Nq29lx//CG2jYXIfN08pTYVc5GX45unEdL5PH1qNHQB+B/Tm l7rkrHHrNKfk6lqYezBKWXnNP7JYoQYwo3baFvz5URU31mvLM8ahHA9H3+IDyv+Q7DQ4Cq oc/QgWsYONIG9n0udvo1F1rMPd/cYx8= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=axrt0kxg; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=g5qxQbbi; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691685394; a=rsa-sha256; cv=none; b=rubeYZRcFljjTjWBWBuwVv312Ha4cDg9ycbYHoBqLcG3UtnCfKqz0/sWe1JN1zM0lO6p+r qttnp5kU9ROZjQ2AT53mnlCgXtySFEpDTkBCQB63I3kMNLckW0GtvI9lpLN7Nu5tIW8RzW ZTCEYDABIgQj9QYUDftQQebSrUHTkvU= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id C3EE71F45B; Thu, 10 Aug 2023 16:36:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1691685392; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4EGfAOQz6XY1pXT3KWDaIKebAxzMbhko7rkj0tHhhE0=; b=axrt0kxgXTo5iJR9/RB/ZPx9oJoNVwf4NlPkAjES0QAbRrNPLa3OjPKPN0kNr2/KHTGtzp BQAFShiw1Z61i8my7P8znJ0mhytmUseVm7s7b8xdVnVkkiwkYuXBeg0CVxBX1FH6fYFyb8 U/xjyYxMQ5kyalhxcjfk3qTq8gzI2AI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1691685392; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4EGfAOQz6XY1pXT3KWDaIKebAxzMbhko7rkj0tHhhE0=; b=g5qxQbbiIaGIdqNSVfvnr6oKuz/ZAZnUCeH5CWbOmCgyhBUdV1Xl7+1HiJGFIox9Qufmol 9YrzUGt7xdheeBDw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 90FA7138E0; Thu, 10 Aug 2023 16:36:32 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id qBLNIhAS1WSEPQAAMHmgww (envelope-from ); Thu, 10 Aug 2023 16:36:32 +0000 From: Vlastimil Babka To: "Liam R. Howlett" , Matthew Wilcox , Suren Baghdasaryan , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev, Vlastimil Babka Subject: [RFC v2 2/7] mm, slub: add opt-in slub_percpu_array Date: Thu, 10 Aug 2023 18:36:30 +0200 Message-ID: <20230810163627.6206-11-vbabka@suse.cz> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230810163627.6206-9-vbabka@suse.cz> References: <20230810163627.6206-9-vbabka@suse.cz> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 14D5614001D X-Stat-Signature: nsno9dp1qqb6kkdefj6sdbmankgm5fd7 X-HE-Tag: 1691685393-674500 X-HE-Meta: U2FsdGVkX19HnQf4tzaswhM4LiR6jv1Yk8oCBiuVBYeEbUsU5sF1Hb0ChVnyJTOWhoHQeQAq87ggSLVVRbKmeZt9gJDm7/hkTS6prz7phTH8S45rlnvY96/3kG1PLGHhuyddeP0nvlinWgVGHa7focexBKZBcvKd4GfWOcorcvSSpvkCds4KTZIVMHnkvOQo7FYHXnJO6rRFqhwzRb7iQ/Y4z57eUh015hcH/Gt0h3t6xLZ38gG9VtpMx3q7VPYfDLhuowBAPWa/kn8+JPywovgrSRmvizF8jCPrSeEtpJpaliGneUwwcEyrlThSjbJcQrFlxVnkhkmBb7qscNeV7HPlfitKn21Ln4E9BL4zecSXHXIpne4HUpNjJK1KHyRSr+pw0RCCNIWytsWGuedFsSaSbyCS0g2UUt6eZZcebyQKucStSADWkREbQXefVq5RIlU/StFwFurSG44upUaMqN20sB1GKu3jw6kHqtO28J2v577vhyIM9E8sNl7ItOkGkf9QHXaA2XCCd2uhtm6YM2lnRxJh9MXV0UvdSfOgTPqYregLbgfDe+y3VC+zDt4GA6ajqcyalppyeWTgjle1EVaGr7E3wqun136pWiUYPnR6SFJcc7VjjbOxEh4vniTi3Xz8HfIbF58QSWUspiBqBouO/mJcUdLEWw957rUgFS9uJLVAhdAdGWHuD+els0sEva1sCdnvoDU5bChg4rH4bBthtQQ83ItCxJxKRFf8G8JoB9msLTZaqHz74zcvOOGUVuv/HuoMGQajSDvVorIZpTTfyjh1g6de+CvWsL/AIhFAnMgGMRLAHKM0ABC7EzgsVyy9UXNSt4TX+BnbvQDivOX778vZc5HrDKl9SmSc1r63/k/ap/IqJVQ3ozl+NGDo33N5MARun72pHk2r+jesWI2wvD2aYymuoR23Sv5fgvbBbGbYb2YA3AX10CzfT0RbeGIshs+y5REq5/ywB3p 9bUNdUbc 96TVZhBLwsN6Kvbt8JRF2MnrL01Vdkyf6LDsqCBNtCSMJIMb5dusuPKe045Gz851l2kwOwy1CK1Fd1qZpnK/dvkMWBWtngF5+d+USOoTaceg9ZcL9KAiHJ6jAKzdODrGon9908tdPu+LldHGA2hxjzyRIpUkG861prg20pXCh5tModq9o3VgAHLpOINQxUanE+SVyyC40xp88dVNrg7BRNDTqzw0KtHMAqGGPYW9Sje1MuKd6X5MBXJIeAGWEYyy+Y/Vp0JHIW3+yX+fQvzXy2xzumri5ZqjiDNTR1mXS/jZOa69fbCrA1MOEHQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: kmem_cache_setup_percpu_array() will allocate a per-cpu array for caching alloc/free objects of given size for the cache. The cache has to be created with SLAB_NO_MERGE flag. The array is filled by freeing. When empty for alloc or full for freeing, it's simply bypassed by the operation, there's currently no batch freeing/allocations. The locking is copied from the page allocator's pcplists, based on embedded spin locks. Interrupts are not disabled, only preemption (cpu migration on RT). Trylock is attempted to avoid deadlock due to an intnerrupt, trylock failure means the array is bypassed. Sysfs stat counters alloc_cpu_cache and free_cpu_cache count operations that used the percpu array. Bulk allocation bypasses the array, bulk freeing does not. kmem_cache_prefill_percpu_array() can be called to ensure the array on the current cpu to at least the given number of objects. However this is only opportunistic as there's no cpu pinning and the trylocks may always fail. Therefore allocations cannot rely on the array for success even after the prefill. But misses should be rare enough that e.g. GFP_ATOMIC allocations should be acceptable after the refill. The operation is currently not optimized. Mark SLAB_DEPRECATED as BROKEN so the new APIs don't need to be reimplemented there and the bots don't complain. SLAB has percpu arrays by design but their sizes are determined internally. More TODO/FIXMEs: - NUMA awareness - preferred node currently ignored, __GFP_THISNODE not honored - slub_debug - will not work for allocations from the array. Normally in SLUB implementation the slub_debug kills all fast paths, but that could lead to depleting the reserves if we ignore the prefill and use GFP_ATOMIC. Needs more thought. --- include/linux/slab.h | 4 + include/linux/slub_def.h | 10 ++ mm/Kconfig | 1 + mm/slub.c | 210 ++++++++++++++++++++++++++++++++++++++- 4 files changed, 224 insertions(+), 1 deletion(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 848c7c82ad5a..f6c91cbc1544 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -196,6 +196,8 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name, void kmem_cache_destroy(struct kmem_cache *s); int kmem_cache_shrink(struct kmem_cache *s); +int kmem_cache_setup_percpu_array(struct kmem_cache *s, unsigned int count); + /* * Please use this macro to create slab caches. Simply specify the * name of the structure and maybe some flags that are listed above. @@ -494,6 +496,8 @@ void kmem_cache_free(struct kmem_cache *s, void *objp); void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p); int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p); +int kmem_cache_prefill_percpu_array(struct kmem_cache *s, unsigned int count, gfp_t gfp); + static __always_inline void kfree_bulk(size_t size, void **p) { kmem_cache_free_bulk(NULL, size, p); diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index deb90cf4bffb..c85434668419 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -13,8 +13,10 @@ #include enum stat_item { + ALLOC_PERCPU_CACHE, /* Allocation from percpu array cache */ ALLOC_FASTPATH, /* Allocation from cpu slab */ ALLOC_SLOWPATH, /* Allocation by getting a new cpu slab */ + FREE_PERCPU_CACHE, /* Free to percpu array cache */ FREE_FASTPATH, /* Free to cpu slab */ FREE_SLOWPATH, /* Freeing not to cpu slab */ FREE_FROZEN, /* Freeing to frozen slab */ @@ -66,6 +68,13 @@ struct kmem_cache_cpu { }; #endif /* CONFIG_SLUB_TINY */ +struct slub_percpu_array { + spinlock_t lock; + unsigned int count; + unsigned int used; + void * objects[]; +}; + #ifdef CONFIG_SLUB_CPU_PARTIAL #define slub_percpu_partial(c) ((c)->partial) @@ -99,6 +108,7 @@ struct kmem_cache { #ifndef CONFIG_SLUB_TINY struct kmem_cache_cpu __percpu *cpu_slab; #endif + struct slub_percpu_array __percpu *cpu_array; /* Used for retrieving partial slabs, etc. */ slab_flags_t flags; unsigned long min_partial; diff --git a/mm/Kconfig b/mm/Kconfig index 09130434e30d..84f4dff70d39 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -237,6 +237,7 @@ choice config SLAB_DEPRECATED bool "SLAB (DEPRECATED)" depends on !PREEMPT_RT + depends on BROKEN help Deprecated and scheduled for removal in a few cycles. Replaced by SLUB. diff --git a/mm/slub.c b/mm/slub.c index a9437d48840c..f41c69bac07d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -188,6 +188,79 @@ do { \ #define USE_LOCKLESS_FAST_PATH() (false) #endif +/* copy/pasted from mm/page_alloc.c */ + +#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT_RT) +/* + * On SMP, spin_trylock is sufficient protection. + * On PREEMPT_RT, spin_trylock is equivalent on both SMP and UP. + */ +#define pcp_trylock_prepare(flags) do { } while (0) +#define pcp_trylock_finish(flag) do { } while (0) +#else + +/* UP spin_trylock always succeeds so disable IRQs to prevent re-entrancy. */ +#define pcp_trylock_prepare(flags) local_irq_save(flags) +#define pcp_trylock_finish(flags) local_irq_restore(flags) +#endif + +/* + * Locking a pcp requires a PCP lookup followed by a spinlock. To avoid + * a migration causing the wrong PCP to be locked and remote memory being + * potentially allocated, pin the task to the CPU for the lookup+lock. + * preempt_disable is used on !RT because it is faster than migrate_disable. + * migrate_disable is used on RT because otherwise RT spinlock usage is + * interfered with and a high priority task cannot preempt the allocator. + */ +#ifndef CONFIG_PREEMPT_RT +#define pcpu_task_pin() preempt_disable() +#define pcpu_task_unpin() preempt_enable() +#else +#define pcpu_task_pin() migrate_disable() +#define pcpu_task_unpin() migrate_enable() +#endif + +/* + * Generic helper to lookup and a per-cpu variable with an embedded spinlock. + * Return value should be used with equivalent unlock helper. + */ +#define pcpu_spin_lock(type, member, ptr) \ +({ \ + type *_ret; \ + pcpu_task_pin(); \ + _ret = this_cpu_ptr(ptr); \ + spin_lock(&_ret->member); \ + _ret; \ +}) + +#define pcpu_spin_trylock(type, member, ptr) \ +({ \ + type *_ret; \ + pcpu_task_pin(); \ + _ret = this_cpu_ptr(ptr); \ + if (!spin_trylock(&_ret->member)) { \ + pcpu_task_unpin(); \ + _ret = NULL; \ + } \ + _ret; \ +}) + +#define pcpu_spin_unlock(member, ptr) \ +({ \ + spin_unlock(&ptr->member); \ + pcpu_task_unpin(); \ +}) + +/* struct slub_percpu_array specific helpers. */ +#define pca_spin_lock(ptr) \ + pcpu_spin_lock(struct slub_percpu_array, lock, ptr) + +#define pca_spin_trylock(ptr) \ + pcpu_spin_trylock(struct slub_percpu_array, lock, ptr) + +#define pca_spin_unlock(ptr) \ + pcpu_spin_unlock(lock, ptr) + #ifndef CONFIG_SLUB_TINY #define __fastpath_inline __always_inline #else @@ -3440,6 +3513,32 @@ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s, 0, sizeof(void *)); } +static inline void *alloc_from_pca(struct kmem_cache *s) +{ + unsigned long __maybe_unused UP_flags; + struct slub_percpu_array *pca; + void *object = NULL; + + pcp_trylock_prepare(UP_flags); + pca = pca_spin_trylock(s->cpu_array); + + if (unlikely(!pca)) + goto failed; + + if (likely(pca->used > 0)) { + object = pca->objects[--pca->used]; + pca_spin_unlock(pca); + pcp_trylock_finish(UP_flags); + stat(s, ALLOC_PERCPU_CACHE); + return object; + } + pca_spin_unlock(pca); + +failed: + pcp_trylock_finish(UP_flags); + return NULL; +} + /* * Inlined fastpath so that allocation functions (kmalloc, kmem_cache_alloc) * have the fastpath folded into their functions. So no function call @@ -3465,7 +3564,11 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list if (unlikely(object)) goto out; - object = __slab_alloc_node(s, gfpflags, node, addr, orig_size); + if (s->cpu_array) + object = alloc_from_pca(s); + + if (!object) + object = __slab_alloc_node(s, gfpflags, node, addr, orig_size); maybe_wipe_obj_freeptr(s, object); init = slab_want_init_on_alloc(gfpflags, s); @@ -3715,6 +3818,34 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, discard_slab(s, slab); } +static inline bool free_to_pca(struct kmem_cache *s, void *object) +{ + unsigned long __maybe_unused UP_flags; + struct slub_percpu_array *pca; + bool ret = false; + + pcp_trylock_prepare(UP_flags); + pca = pca_spin_trylock(s->cpu_array); + + if (!pca) { + pcp_trylock_finish(UP_flags); + return false; + } + + if (pca->used < pca->count) { + pca->objects[pca->used++] = object; + ret = true; + } + + pca_spin_unlock(pca); + pcp_trylock_finish(UP_flags); + + if (ret) + stat(s, FREE_PERCPU_CACHE); + + return ret; +} + #ifndef CONFIG_SLUB_TINY /* * Fastpath with forced inlining to produce a kfree and kmem_cache_free that @@ -3740,6 +3871,11 @@ static __always_inline void do_slab_free(struct kmem_cache *s, unsigned long tid; void **freelist; + if (s->cpu_array && cnt == 1) { + if (free_to_pca(s, head)) + return; + } + redo: /* * Determine the currently cpus per cpu slab. @@ -3793,6 +3929,11 @@ static void do_slab_free(struct kmem_cache *s, { void *tail_obj = tail ? : head; + if (s->cpu_array && cnt == 1) { + if (free_to_pca(s, head)) + return; + } + __slab_free(s, slab, head, tail_obj, cnt, addr); } #endif /* CONFIG_SLUB_TINY */ @@ -4060,6 +4201,45 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, } EXPORT_SYMBOL(kmem_cache_alloc_bulk); +int kmem_cache_prefill_percpu_array(struct kmem_cache *s, unsigned int count, + gfp_t gfp) +{ + struct slub_percpu_array *pca; + void *objects[32]; + unsigned int used; + unsigned int allocated; + + if (!s->cpu_array) + return -EINVAL; + + /* racy but we don't care */ + pca = raw_cpu_ptr(s->cpu_array); + + used = READ_ONCE(pca->used); + + if (used >= count) + return 0; + + if (pca->count < count) + return -EINVAL; + + count -= used; + + /* TODO fix later */ + if (count > 32) + count = 32; + + for (int i = 0; i < count; i++) + objects[i] = NULL; + allocated = kmem_cache_alloc_bulk(s, gfp, count, &objects[0]); + + for (int i = 0; i < count; i++) { + if (objects[i]) { + kmem_cache_free(s, objects[i]); + } + } + return allocated; +} /* * Object placement in a slab is made very easy because we always start at @@ -5131,6 +5311,30 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) return 0; } +int kmem_cache_setup_percpu_array(struct kmem_cache *s, unsigned int count) +{ + int cpu; + + if (WARN_ON_ONCE(!(s->flags & SLAB_NO_MERGE))) + return -EINVAL; + + s->cpu_array = __alloc_percpu(struct_size(s->cpu_array, objects, count), + sizeof(void *)); + + if (!s->cpu_array) + return -ENOMEM; + + for_each_possible_cpu(cpu) { + struct slub_percpu_array *pca = per_cpu_ptr(s->cpu_array, cpu); + + spin_lock_init(&pca->lock); + pca->count = count; + pca->used = 0; + } + + return 0; +} + #ifdef SLAB_SUPPORTS_SYSFS static int count_inuse(struct slab *slab) { @@ -5908,8 +6112,10 @@ static ssize_t text##_store(struct kmem_cache *s, \ } \ SLAB_ATTR(text); \ +STAT_ATTR(ALLOC_PERCPU_CACHE, alloc_cpu_cache); STAT_ATTR(ALLOC_FASTPATH, alloc_fastpath); STAT_ATTR(ALLOC_SLOWPATH, alloc_slowpath); +STAT_ATTR(FREE_PERCPU_CACHE, free_cpu_cache); STAT_ATTR(FREE_FASTPATH, free_fastpath); STAT_ATTR(FREE_SLOWPATH, free_slowpath); STAT_ATTR(FREE_FROZEN, free_frozen); @@ -5995,8 +6201,10 @@ static struct attribute *slab_attrs[] = { &remote_node_defrag_ratio_attr.attr, #endif #ifdef CONFIG_SLUB_STATS + &alloc_cpu_cache_attr.attr, &alloc_fastpath_attr.attr, &alloc_slowpath_attr.attr, + &free_cpu_cache_attr.attr, &free_fastpath_attr.attr, &free_slowpath_attr.attr, &free_frozen_attr.attr, From patchwork Thu Aug 10 16:36:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13349719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBCF9C04FDF for ; Thu, 10 Aug 2023 16:36:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1EE676B007D; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C3396B007E; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE0976B0080; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CDD966B007B for ; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 773B21CA0D3 for ; Thu, 10 Aug 2023 16:36:36 +0000 (UTC) X-FDA: 81108748392.22.16A39C4 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf01.hostedemail.com (Postfix) with ESMTP id 92FCD40028 for ; Thu, 10 Aug 2023 16:36:34 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=izuGpZjT; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=p6eqWXON; spf=pass (imf01.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691685394; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=a3DtkVdLGQHDYk90N/OXEPjHa8PNLxLzj/qp7wDJMuQ=; b=wanJrbRAn5xuS30pxeoEcP0esBk8UEJE8eQf4dQ0xQteew/shLFVa1VFF1NE+P0sQkh6ca K3zZF1HXwpB0D9AAbu0nfUzNj3zuhZC8WpEfRoYmCTHtV/NEdxCvgGhP3iXnHBZHl1A102 vZw3Z7g6AoinenCllFIVJ5ad/S6GX6I= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691685394; a=rsa-sha256; cv=none; b=H0P1Nbx7KhO63tx7YHRiCpAzQsLQ7LfAdZKxcNlkHHgut/M54JvAQM+sQH6/k9YuEHmdB9 Boi5cV29qOzwK5jEplkQnOPXyopnlZSkDy2fXyd36zH074vokz91AmSKsZ3jNgBX2tnVZU es/zgCIboh6kPJN14EvkxkFjA7qv1rs= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=izuGpZjT; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=p6eqWXON; spf=pass (imf01.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id F036821878; Thu, 10 Aug 2023 16:36:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1691685392; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a3DtkVdLGQHDYk90N/OXEPjHa8PNLxLzj/qp7wDJMuQ=; b=izuGpZjTDaa1X7FM+8xTm5XGp80s21t8mw95pR1CYB5uPp0dA8c+tZ0jgOTQBfSnlPUZXc vQiMiGKsA7yQKGOWxzUA/EX95/ge5PmBAW/5TI6OTaJmg4IENCoK0N2z6vsWBJv2ji8loT oEXgrJoEYRtt2UOfw5uspqh2z5YapbY= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1691685392; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a3DtkVdLGQHDYk90N/OXEPjHa8PNLxLzj/qp7wDJMuQ=; b=p6eqWXON9YDd4m3Wr/OaVX58a9pQHfuB4SGI9tgsmckkAANRrz+0BOIX9nWMdm6L9jpBej SEImetYJhoV2wgAQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id C432F139D1; Thu, 10 Aug 2023 16:36:32 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id IMZJLxAS1WSEPQAAMHmgww (envelope-from ); Thu, 10 Aug 2023 16:36:32 +0000 From: Vlastimil Babka To: "Liam R. Howlett" , Matthew Wilcox , Suren Baghdasaryan , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev, Vlastimil Babka Subject: [RFC v2 3/7] maple_tree: use slub percpu array Date: Thu, 10 Aug 2023 18:36:31 +0200 Message-ID: <20230810163627.6206-12-vbabka@suse.cz> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230810163627.6206-9-vbabka@suse.cz> References: <20230810163627.6206-9-vbabka@suse.cz> MIME-Version: 1.0 X-Stat-Signature: 3ojzk6yisoicw7x9owabyk6hrg45f1jk X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 92FCD40028 X-Rspam-User: X-HE-Tag: 1691685394-344765 X-HE-Meta: U2FsdGVkX18b6wtLLOTK+T24gFUn+ZxBjOzTfA9h6RL8hujOnJadZU3A/FC0j4GQ0FyBZRFNMYWTQ/oPrPpq0fbzy0HO3s2tHuU9oGQvh7rmI0dRzjyE0VG4TzZ4wtHvXyo9DyfMa08ImfSrWIP6qWelxMXUwKvWmRmZ6IMNCAvLjofCy+KD6+9AZ+/iy2rt6MRmz5JsT9yDpkvcI0AQrBkV/gM3mOQ9yyI6kC1jT+NQP4BTf90hfoPcRRxkmuh5JWe+/FvqDKCJEOxrgCC2LLHLRjGrTjihPYl25OfquInM6vkUzv14fJ/vi+BZodPvwBlvbYVSd8piyGIzX1epVHqn4o8NfB/3h/31L+K1lr0dm4Nmbd0MO8t8T0u8LIFusuOwE8Hz6vaUrzuALBO5x686YxebYdlIN76qzQxjoeEtbvPhMoOdp3MnD12OmZwgJeYPSskRlDEbu9dzjM+788eynt60H0FblVFZXQy5hem4FXQe6S4gwj4THdsIvoBWQj0oT47rYpPg7ntLbrgVTa7iTE+Y3BITQrflkqBFw1KgfHUl8u0J1ONDV5dbCxNLxlBFkIbaJkHhQmj4Z21pDZ7f6EpRCyckdtRr8YTg8rtGGTLECSZTJpoUwSqjAgOsAFR7KtJCDdvYtZxwdLDxO+uF0ymeNvPPncAprEQFGFefXGTfSxPSTRSFTsA6dFvxdd5WHHdqLOuvcMx4j1bBEJPANxxXRPauFbEolkcZvUR3Q7FD3FS9HtL0ld+JLrYdeTG1YpG8mbBcu20t1GopuAgqn1bW7J4smcSr4bH0PcwAi+GOhUTzURbQidSV/jFLxZ6Es5ebccq5TQvI7005dcpEXeu82DOFNKzKSHkcgVpmXTqEKtYutIWcURlpSKH+DH23rpyE9CiCqaYm8E25lkeAWjJNZKn1leZr+mnUYbbjDZvAJm3sO18VZx1wi6sz0L9DcUVuCOB4hawioKR q9p9hYI7 y9yHz9uM7FT5cuFI0PWbeEcUGCjLH5ytBMrHctnLCFsCX+blpJAN1Nm6MfzqlCfUIRcIy03kJ5TdoQjBTNcsASvQ8QXmiEiArxp1a3rev538kI9dtMFUxCHxWcqT2USaoFhg4yYuxPAOVAdsuSeEbDfWJAs9t1nb2TrMBTYDCf50E6hwE2/iyck1ONhb9Ovxyhx56dyWa9UMOcUD3/XOxlDtMfNYTDwZkpOQXeeWglp+B2Adb+vSxiE4eIoHpVUYrwtguBqJEP4V4XMvUXF3KGOtF9VCINP3XNH5AznRK9hSxU2R0/o/xCjZUHg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just make sure the maple_node_cache has a percpu array of size 32. Will break with CONFIG_SLAB. --- lib/maple_tree.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 4dd73cf936a6..1196d0a17f03 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -6180,9 +6180,16 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) void __init maple_tree_init(void) { + int ret; + maple_node_cache = kmem_cache_create("maple_node", sizeof(struct maple_node), sizeof(struct maple_node), - SLAB_PANIC, NULL); + SLAB_PANIC | SLAB_NO_MERGE, NULL); + + ret = kmem_cache_setup_percpu_array(maple_node_cache, 32); + + if (ret) + pr_warn("error %d creating percpu_array for maple_node_cache\n", ret); } /** From patchwork Thu Aug 10 16:36:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13349723 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B079C001E0 for ; Thu, 10 Aug 2023 16:36:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 287CE6B0080; Thu, 10 Aug 2023 12:36:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A39E6B0085; Thu, 10 Aug 2023 12:36:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E44836B0082; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BFBDE6B0081 for ; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4C710B2B54 for ; Thu, 10 Aug 2023 16:36:33 +0000 (UTC) X-FDA: 81108748434.02.A05BF18 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf20.hostedemail.com (Postfix) with ESMTP id 9A9A41C0013 for ; Thu, 10 Aug 2023 16:36:34 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=vzqZq8M1; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NWWqKFNb; dmarc=none; spf=pass (imf20.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691685394; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=19h9QNSgdzIWywJanN9veEOMa6s1Xj+q+Cz0v3FP+f4=; b=Hbn4Ltp4ILQ6e/9cjNy5sSsw6TkShBjnTmjxHEfI+Z1BzbDDaQX+2Hz+vA606vMpQX8kww bhDLFDdRwXjoUB8vTFuZNqaBiRRu/f/BOUaQzOxY6iRZbYjM1dGiiAJ/tD4PL5DCOQIFH0 03uJd6Tm/eHaf7mlhBVIvP6V3pzG6Nk= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=vzqZq8M1; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NWWqKFNb; dmarc=none; spf=pass (imf20.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691685394; a=rsa-sha256; cv=none; b=DcqkSIV76n7D9cSkJqwOeGl8jzF6+0GeEwpjgoeMWfbyiE4/Pb7c8EvoC3lGKPFFeVjS/B UdQgV03M4LDq9CYAZhZXRI9zXAwyzsy0jEUtxX0LroUFjoFlkPMD+YxvHHcXdojwinDuu+ YlJMHQyhHqFN06HUwUav0102/i2mQEk= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 302EE2187B; Thu, 10 Aug 2023 16:36:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1691685393; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=19h9QNSgdzIWywJanN9veEOMa6s1Xj+q+Cz0v3FP+f4=; b=vzqZq8M1QVHjJzcAciJEyysHegc4ljBQedRS1OxPDkK3CHjdyrBTf+D6vESZxsfkhbcdVL MIoyrdIPrePiMNwI+qW9AoKZpWA5gzajN/uBrziCUIySW9YuqYE3Yf3g2M13Ovrbehg/dr vI+tbRZzkrG3tO2oaTs0iBlNzeiRoys= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1691685393; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=19h9QNSgdzIWywJanN9veEOMa6s1Xj+q+Cz0v3FP+f4=; b=NWWqKFNbDZBln4sSAtj33wuORgNzUuDPhyE63EaKkKg3gsXILvYYR+v0b3lf9S97ya6XCE f5wjW2+XbXSXUoCg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id F1D3C138E0; Thu, 10 Aug 2023 16:36:32 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id IBVjOhAS1WSEPQAAMHmgww (envelope-from ); Thu, 10 Aug 2023 16:36:32 +0000 From: Vlastimil Babka To: "Liam R. Howlett" , Matthew Wilcox , Suren Baghdasaryan , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev, Vlastimil Babka Subject: [RFC v2 4/7] maple_tree: avoid bulk alloc/free to use percpu array more Date: Thu, 10 Aug 2023 18:36:32 +0200 Message-ID: <20230810163627.6206-13-vbabka@suse.cz> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230810163627.6206-9-vbabka@suse.cz> References: <20230810163627.6206-9-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9A9A41C0013 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: b1hz4jnszpzuu64kxzbhoauqa1mmjx86 X-HE-Tag: 1691685394-542612 X-HE-Meta: U2FsdGVkX18PTAaBeFP8rv7BiUwbekPLJ8o2MQ9QMaE8xMM36M1vhRQdqKoAj0R/ePcScxuGyeTkXBbbvYIGBscXx5ZLHV+MTy92C9I416GXHoNaNc5mtpVRaZ+3wKQXtt9jSVIgQr0zVsknLvvWvmfsFSb9QVFc/g34A7Y6FfBwh5zIUWk4xyGbBdSjfShRAPy7JWrU58z6B8M1y9eOk8XWVYv/7kcEc4+MTP/hbPCh63yO4HnsTcY8c95qsnhjUN/ExSveoOL99xjXZKFBtfzhiMj1awe+r36/06AXeJq7jXk4Ab9/m62WHbQJHbgfMgdtzvrQCzeBH4cCzvdo5DPayT8fpTZaFY9+47edT7K9+NmqLLTQZjEzFv/GZr2MN18fUcO093SwiQzUXd/3Dj1H2VBu9jDYwJFpXg5sP3eKQwiPKCdcxannu4YOXgqjAD335WUgLpk3RWabimICLoJ5adYguKkn4bBMk3WJb/SySFhHZb3qUWpbnU7fIyn0SML29r9wIgzKyNoJtjtSmJdUKp1/WK/yJHnqHV4K8eTzx3UBceVJ8xcqhh3BGl64clbQMDPTeaSkM3xHLgKpg7GWsKd7o3GKTitNnn8mzZWtlBSV5XPpqDy6pYzxmledhoXuVuPfoF/mXnljDSZ2QtZ8vShWmxVTfdSpgz5BdltzxujKmgWT7m+OlXshbAA7MWsgaPk5y8ynKQ/g9jgpbmIo1mIHcW+VEe0RwgzZ+gO5JM/HcsfI2uAs+TM2LsIwZ6/zNR5vt+lqbMbY84eqKd3CSJ/CzU5PFy7AXGV43Dgkl0HMizo+NZSAghju6KK/B++awUHU9aRkQBX2yDBPnAjNZk5EA82mSHf71hUxDVZdlrB8CvmddZT9ZCNCrKeSm4uOiZQoZ19aCAmuLqXfwhKAkbMUYzoCfxyl2ijrcRVmFJWNSLITVOpbDoMfnYn9hPgOfvyc8q/o6TsjOdB PDuWTj7u 1pgVaFaOaV0QzvPy8r1edqzHFEUxzeW8maAPFWPMh8GrOCd6U+B0W/lEs/2hPBARcyr0XffwDM5LibA4DWHdIVZH99q17h+MdUUN20j3lUv6MhzcoIjqFsJ2so06/NoAnUjw0sBB5+d5eJMxmaTrRkkTzf3XgDPp4mJKzEgABzgOJUEOD+XimdVYrtdKbwpRV4KQZy6IZpRe+NPn49CEvGC0rOg6hMS6VpByNFwkRDxhDAfsO3sZDK4J5ZiO/WBHI7xQvVqoc3N0OgyVOvJb1rJ5WCq/nvbsgmpbshSVbJvAtPkpNE5LVeDAvjA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Using bulk alloc/free on a cache with percpu array should not be necessary and the bulk alloc actually bypasses the array (the prefill functionality currently relies on this). The simplest change for now is just to convert the respective maple tree wrappers to do a loop of normal alloc/free. We can optimize later versions if needed. --- lib/maple_tree.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 1196d0a17f03..926bee7a274a 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -161,12 +161,20 @@ static inline struct maple_node *mt_alloc_one(gfp_t gfp) static inline int mt_alloc_bulk(gfp_t gfp, size_t size, void **nodes) { - return kmem_cache_alloc_bulk(maple_node_cache, gfp, size, nodes); + int i; + + for (i = 0; i < size; i++) { + nodes[i] = kmem_cache_alloc(maple_node_cache, gfp); + if (!nodes[i]) + break; + } + return i; } static inline void mt_free_bulk(size_t size, void __rcu **nodes) { - kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes); + for (size_t i = 0; i < size; i++) + kmem_cache_free(maple_node_cache, nodes[i]); } static void mt_free_rcu(struct rcu_head *head) From patchwork Thu Aug 10 16:36:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13349720 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56C87C001E0 for ; Thu, 10 Aug 2023 16:36:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F36A6B007E; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CC316B007B; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3233C6B0082; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id F295E6B007B for ; Thu, 10 Aug 2023 12:36:36 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id AE9D11CA140 for ; Thu, 10 Aug 2023 16:36:36 +0000 (UTC) X-FDA: 81108748392.18.3ACDC07 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf15.hostedemail.com (Postfix) with ESMTP id A1D9AA0009 for ; Thu, 10 Aug 2023 16:36:34 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="vGJ6/1Wt"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=9x9XyqLN; spf=pass (imf15.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691685394; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=81VWuf1Jqzz86k9wDN7LrYfBK2s1ZDCqD4SXLtwC3uA=; b=RCmA9F2mKm/GCYrRH5iIs6z6bH+WnLmWeBApJJqDt2Tnqvn7L+D2i3kCOgVZduA6oDdpez 84sSO8QTcfOkWzdAlXajI7v+7Irg0a5QQrQ398WKZOoLKBTcrlHKYHu2GcWr6z6NyoOBoe V8atX4fXV3IbaO5vdcmTZRrkqTuFwm8= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="vGJ6/1Wt"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=9x9XyqLN; spf=pass (imf15.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691685394; a=rsa-sha256; cv=none; b=wxEn9/DLz/0rq/gsr7pFoqQA+fr9Vr7NmOmGjmZh/0kWbgnJC1ytMlC+CBcXnV5PSOVcNi 0u1bHBeLXBW9euVdaM4pdNs/yxwLmKYmi/yX4E4i4DHVBOo7JcD6+YH0pB1rcXeJgUnP+Z e9Su07wk8RotdT8cOAxRUZiTJAT1Fjc= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 614A21F750; Thu, 10 Aug 2023 16:36:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1691685393; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=81VWuf1Jqzz86k9wDN7LrYfBK2s1ZDCqD4SXLtwC3uA=; b=vGJ6/1WteLNGdLHhKpW6hrMYCuWBe6WzoYcWBYsk/0kPZb7z0w1EKu7Caou9OwoQqM0QPD hTllK+yJdN9fv7kLVpgI2tCN5xKxsmExHBA4r/mVMBQaMFAfb4YFTFCu538cEnR6H1DKSn dLBkK+T1U+VHRemqTiutB021iD242/0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1691685393; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=81VWuf1Jqzz86k9wDN7LrYfBK2s1ZDCqD4SXLtwC3uA=; b=9x9XyqLN8gJpEQKeJMyMY/R1t7RGpoiajCpyEbPmS092cPUZLAJeo+as7qvBiBUDFncs97 rZ5jw5kHYfhuUuBA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 329D1139D1; Thu, 10 Aug 2023 16:36:33 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id GPC5CxES1WSEPQAAMHmgww (envelope-from ); Thu, 10 Aug 2023 16:36:33 +0000 From: Vlastimil Babka To: "Liam R. Howlett" , Matthew Wilcox , Suren Baghdasaryan , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev, Vlastimil Babka Subject: [RFC v2 5/7] maple_tree: Remove MA_STATE_PREALLOC Date: Thu, 10 Aug 2023 18:36:33 +0200 Message-ID: <20230810163627.6206-14-vbabka@suse.cz> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230810163627.6206-9-vbabka@suse.cz> References: <20230810163627.6206-9-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Queue-Id: A1D9AA0009 X-Rspam-User: X-Stat-Signature: k7r3dcdyhdjiqinyf3okqmojhpuupzeb X-Rspamd-Server: rspam01 X-HE-Tag: 1691685394-99795 X-HE-Meta: U2FsdGVkX1/NbAWfXUoaCJjV+xrqNlv+zJrkbjGz+V44JIYmpKuPwId9wwB/ACZq4snZRhW5DTZHtRm1kuVxo2EHgV6VKZz82OmPEehM2uqZ2MgcYILPDe+FuWTiNxENV8pu/QtR4mBOKqVkSc7IUojA8jyUEdXgY/DohY7hvxLvRq/urYr/QHqO1El105S3VeNl7Mp2lhOkZHfGVHnnVU4iIuZin3LnV2LSGsXwZnO6/1MSQexhsSBDdFKP9XusjdONKdxV1ew9FxQCnhyyJ/zMlCA9EYm4Vb40L5uvDjFFeuXK+GKuIaZm/oHw0+tJZd8tuyAF16TK/ObGZePxkF9Bq3J0lfJREdn8tRGctBwTZxv0mbTZ3Z0fLtfl21qtbs9z4tZTl6dJuDbKEJvEUb2YYQaU97mhsTo8AFu9UjPFkAsuXFAijiXZXPd+m1Ptzt6Js4BpIqwJUQLnpCdUm0vItoK39s+K009V7p2hzOKLB5yel+FsBPOQrm9sD1KibVhUvXg0sQQSEVSlgPC8BBRKsqrTPXthMbPhTTNcN/DyI4umybDALLLiDUl0RueRBjU6QdXzOlyb6LTpoEIPTVcGG30gbK1U/MfRvKP5vEgzpgOfR2EzwzPXFV2d4yoIxVQxo/VCNHPGTpoeOS2zi2wDzSkzojBoOYr9YkYhhi1iAnwFDanU2RbZ++k9kqOhU+Ei7nF3+efUfX1vsLsVqCWAUoSYnlyqA/gXPvoo1EMx/qatXGHBCXRO36DYS5OSBsCsfqjA1NAcl6DmSnDnBwipN2d84qy7Z43YgaN6484r8UQcgtQYrKM/h0jYFLctctv0l68/PLFE3/OhZMNyAfq2B/81jz8eCKrbEgc4c2cBr08spDx93Moz4l76MdaGl4Yd1o3+cNIC9sH8C2vXzCUFtXg0TQNFWkjHOIVOyH7XsJKHxye+6vxrZURaPWseESkIgYI+HwlGjPuOr/F AN5xVlgk 7hhHeRzulx3tAqd5RRz2ITBuzhOqUlesn+M/b96Gn8TStPXVg0fqrbyChLuojsp+JKNxUoZGmUXTvXT8nkFAsovf5EzDYrVYUEG9gU1tqnvqUUWwnzWoYcpYbxcyKVClfiOFpgmFf1cSdpVxg9ExZH+gOXXQjlpkJ0fys0kCBYFRYauXxXPTR2SsDxU/4KzuNktiNrwp/ppt25WxAJdlfuWLQY4zemlAvXBAQqIEjuhc6f0d54omskc3OYzhqfwEfbccB6OKqYrNavJxDWtgnaNsLFFnxlDeHz2QzbAQf6WBEX6QCN37tHqY9UaZAMlZtwrh8ASWOKnWaUME= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" MA_SATE_PREALLOC was added to catch any writes that try to allocate when the maple state is being used in preallocation mode. This can safely be removed in favour of the percpu array of nodes. Note that mas_expected_entries() still expects no allocations during operation and so MA_STATE_BULK can be used in place of preallocations for this case, which is primarily used for forking. Signed-off-by: Liam R. Howlett Signed-off-by: Vlastimil Babka --- lib/maple_tree.c | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 926bee7a274a..8bd4a79537d8 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -68,11 +68,9 @@ * Maple state flags * * MA_STATE_BULK - Bulk insert mode * * MA_STATE_REBALANCE - Indicate a rebalance during bulk insert - * * MA_STATE_PREALLOC - Preallocated nodes, WARN_ON allocation */ #define MA_STATE_BULK 1 #define MA_STATE_REBALANCE 2 -#define MA_STATE_PREALLOC 4 #define ma_parent_ptr(x) ((struct maple_pnode *)(x)) #define ma_mnode_ptr(x) ((struct maple_node *)(x)) @@ -1280,11 +1278,8 @@ static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp) return; mas_set_alloc_req(mas, 0); - if (mas->mas_flags & MA_STATE_PREALLOC) { - if (allocated) - return; - WARN_ON(!allocated); - } + if (mas->mas_flags & MA_STATE_BULK) + return; if (!allocated || mas->alloc->node_count == MAPLE_ALLOC_SLOTS) { node = (struct maple_alloc *)mt_alloc_one(gfp); @@ -5596,7 +5591,7 @@ void mas_destroy(struct ma_state *mas) mas->mas_flags &= ~MA_STATE_REBALANCE; } - mas->mas_flags &= ~(MA_STATE_BULK|MA_STATE_PREALLOC); + mas->mas_flags &= ~MA_STATE_BULK; total = mas_allocated(mas); while (total) { @@ -5645,9 +5640,6 @@ int mas_expected_entries(struct ma_state *mas, unsigned long nr_entries) * of nodes during the operation. */ - /* Optimize splitting for bulk insert in-order */ - mas->mas_flags |= MA_STATE_BULK; - /* * Avoid overflow, assume a gap between each entry and a trailing null. * If this is wrong, it just means allocation can happen during @@ -5664,8 +5656,9 @@ int mas_expected_entries(struct ma_state *mas, unsigned long nr_entries) /* Add working room for split (2 nodes) + new parents */ mas_node_count(mas, nr_nodes + 3); - /* Detect if allocations run out */ - mas->mas_flags |= MA_STATE_PREALLOC; + /* Optimize splitting for bulk insert in-order */ + mas->mas_flags |= MA_STATE_BULK; + if (!mas_is_err(mas)) return 0; From patchwork Thu Aug 10 16:36:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13349721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F7B7C04A6A for ; Thu, 10 Aug 2023 16:36:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9A3AB6B007B; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 851B46B0082; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B9466B0080; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2F2806B0081 for ; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E2C901CA157 for ; Thu, 10 Aug 2023 16:36:36 +0000 (UTC) X-FDA: 81108748392.11.3AA5EE5 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf06.hostedemail.com (Postfix) with ESMTP id EBAFC180022 for ; Thu, 10 Aug 2023 16:36:34 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=ahOAhspP; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=Cj3pdlbc; dmarc=none; spf=pass (imf06.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691685395; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0YgBKGsaXsGLSxrzXPhBkvZQSAlAP3ciKJFIHzEnvRY=; b=MAanK9BL2OgUIyQu89dqGBnsoHQ2Vvk/IQNj1/y04qLtt1j51n6kXoxjyWufe5r83AhlXo 0Xy6pmekWo7QUYqvJpGa06BvD8BLg2j76OOn5mHqvU8m0Z9IqmSR7ogEkDZtYjtht6YPLi aFLKTUPnvhP7AH4Tyiw1NqEXFQm5Dic= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=ahOAhspP; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=Cj3pdlbc; dmarc=none; spf=pass (imf06.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691685395; a=rsa-sha256; cv=none; b=R+s6T3xCy2CSYqEjq5ihmp5G7CgOwmdQ7v8+M2gQ2JxDL0vCDGJLcvztHSkxO/4GuJF2D7 O1lPu3tzHLOhNsvzH13Exg8kZVCKwQtH0rRo0TO6LunIrqQ9kHOD6O3bCQwV5HxRWXN1ds MoWfNE3Yn1OohbATNOyv/kGDeAvwhuI= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 8DD541F855; Thu, 10 Aug 2023 16:36:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1691685393; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0YgBKGsaXsGLSxrzXPhBkvZQSAlAP3ciKJFIHzEnvRY=; b=ahOAhspPOh9gWRokiULaPvSVlO7+FXuxKO4tvOHBGWkFdq68YuUC556iHKh4f3O7IPBUEb 9+o7trNIK7n0OsSei8wrjxwksDOg4QP9utkyO1S6+9mIqAA9POXLcynrS+eRBvJDWA5e2u SJ68T9v+Br03Qj02Wz/h3oGxsa+KMn0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1691685393; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0YgBKGsaXsGLSxrzXPhBkvZQSAlAP3ciKJFIHzEnvRY=; b=Cj3pdlbc0C8qo51lQJTDCKqNdKlP6jg3SiLPuDNcG94XPwf5LjRYss+oyxhSePhcqfnwz9 44KQ/fhVgwG1ijAA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 63EF5138E0; Thu, 10 Aug 2023 16:36:33 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id SEF4FxES1WSEPQAAMHmgww (envelope-from ); Thu, 10 Aug 2023 16:36:33 +0000 From: Vlastimil Babka To: "Liam R. Howlett" , Matthew Wilcox , Suren Baghdasaryan , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev, Vlastimil Babka Subject: [RFC v2 6/7] maple_tree: replace preallocation with slub percpu array prefill Date: Thu, 10 Aug 2023 18:36:34 +0200 Message-ID: <20230810163627.6206-15-vbabka@suse.cz> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230810163627.6206-9-vbabka@suse.cz> References: <20230810163627.6206-9-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Queue-Id: EBAFC180022 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: nm8xh7hh5zbzxiuixe1gwjjkxrau7uxm X-HE-Tag: 1691685394-668329 X-HE-Meta: U2FsdGVkX1+BjuyYeyqdUvLwmAk4KtFHzwSgyF/TJ/GpIkSMUPpcf5DauxJAhFKgFdDEc+VwJuQoOnvR0+cGSoDqhhiSnEH94aqgimkqSHjuIc5JtzO6fkcoXbLTBfM/owTM7wvpSAcYQ3Vo4Eq93azb4amWwvqA5BaWvWjvvMvSqWetezX2j+Jc0I0Q2zc79CNgK+sBpMz9rtwrY3X5+8L1YiI8UScR9DjbGajaDra6EM8vQiky9PQEK2wfwcRfAs3VVGgeZXZuLpqfKjlGIr389IB2UJvyRXkeVx73pmKo45vyzs6HuCwJFUmprQIp+ETdhyUiAeUFf17wICV8BIKz5WfJ0cQe0yNKI5u9pGVSL2AM3FDeDGLClnjRI6CihnQsOIzbUp7cKU+U00upxBamC9ivdVa2XBf5TekemYeD5A1urCArQNC5ir90n7bg5qwpEcUJjfvfWghlpgYq0lLgbriONxuVfp9p2RRRuhx8NO8xR4W1QVCgrrjAnHygYi6ida/KnB/zIFjAcnbv620t6t2B9CFDPPuWAc4QODqQ47N9BGtqVHq1W/g/BKzu+CE+UUZwU1UC08iaR9H/qOW+Jnm92QaSkzisVcdoOoiHeQgxQmiRxOum3Wkg/2vJfiFic+0cM7xjuZIiJi3Q431ctKrT2dRPAp0zzoBs7bMGhSSnsXnGSOBNnCYJvU35HOtZlkC7LQFYII71AIBsdPOu0IvyWzfePqCTtBksr5sNrtXRijJOvuUR0XQZWaq+tBINBqwaXHUKKHfDghYTw/x+raQPBw7uxWORJWQhiu7SP7h+DFUDGLwbU0Be11/boLlgal/sMVbl+3jZomwQSOinEMFztznGGhkd7wI0B7RviTWi5gtRN146Z22ahpIgTqCQUjohp0oFgUGEMrO3wIEVjL6WrDllZr3Dzn0dGTxEXdjXdOPf4K5tu5KeqbFdPHsoI3An/6zVqJI9h4P NZI3JbGr Mugno9dBAgO/snNMpaYt/HEmmB+AD8WdcTpZCEfAsP0/oaEK/vDF+8tAqoOB/tPPcbDp3EkAYV17cApGCW3MO+8PkFLWUJeLNyA+CPH2k0EHjCA5t5FWMZ+xxC2HSDthvM4DzcyAP20UwL74RECfumgjv4qITEWjQLa4xs4Tzduh5j21NBPrHpeffw7is84E3MZTVcCNl0Nv9650JHI4jNH8GfhPXWJEuHFtTIFuXmYOxWLU8xCaYv2UpoMwnyHhzoc6IfVXpONNs1Oalfn6G/LuoGWEloklx64Upl5aJ0yfU6LLWwc+G7YzkJQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With the percpu array we can try not doing the preallocations in maple tree, and instead make sure the percpu array is prefilled, and using GFP_ATOMIC in places that relied on the preallocation (in case we miss or fail trylock on the array), i.e. mas_store_prealloc(). For now simply add __GFP_NOFAIL there as well. --- lib/maple_tree.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 8bd4a79537d8..304f2453fac9 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5530,7 +5530,12 @@ void mas_store_prealloc(struct ma_state *mas, void *entry) mas_wr_store_setup(&wr_mas); trace_ma_write(__func__, mas, 0, entry); + +retry: mas_wr_store_entry(&wr_mas); + if (unlikely(mas_nomem(mas, GFP_ATOMIC | __GFP_NOFAIL))) + goto retry; + MAS_WR_BUG_ON(&wr_mas, mas_is_err(mas)); mas_destroy(mas); } @@ -5545,19 +5550,12 @@ EXPORT_SYMBOL_GPL(mas_store_prealloc); */ int mas_preallocate(struct ma_state *mas, gfp_t gfp) { - int ret; + int count = 1 + mas_mt_height(mas) * 3; - mas_node_count_gfp(mas, 1 + mas_mt_height(mas) * 3, gfp); - mas->mas_flags |= MA_STATE_PREALLOC; - if (likely(!mas_is_err(mas))) - return 0; + // TODO: should probably indicate if it failed the prefill? + kmem_cache_prefill_percpu_array(maple_node_cache, count, gfp); - mas_set_alloc_req(mas, 0); - ret = xa_err(mas->node); - mas_reset(mas); - mas_destroy(mas); - mas_reset(mas); - return ret; + return 0; } EXPORT_SYMBOL_GPL(mas_preallocate); From patchwork Thu Aug 10 16:36:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13349722 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FF09C04FE0 for ; Thu, 10 Aug 2023 16:36:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F0A406B0081; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EBA416B0080; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BDB1F6B0083; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8658A6B0080 for ; Thu, 10 Aug 2023 12:36:37 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 609D51A0306 for ; Thu, 10 Aug 2023 16:36:37 +0000 (UTC) X-FDA: 81108748434.09.9598A03 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf17.hostedemail.com (Postfix) with ESMTP id 32DAC40025 for ; Thu, 10 Aug 2023 16:36:34 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=jFDhujQK; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=PdU68wcF; spf=pass (imf17.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691685395; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cIe5o/xpdmWF4fStDltZQwKQCcPAraOzhHNBE1tj7sI=; b=u6PivWeWEniR2332np6EiqT9yrGvjiliB/axiV0tXXmfEZ30yz5z6d/Y2dleGt2t7ExPI3 I9RB1kxIUfwYxf8xWdh+nm3tU/JxDFz6HzkA9Yj4PZ+uGtwYxSd1WN1tb+qeu6tOBwZtfc fP5JUC80gHb3BraTSBrJ9Kol2UOWNT4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691685395; a=rsa-sha256; cv=none; b=pG8NlmYkxZJBN6Z1GnOtmEv8r4nqV8iMrlLLh6B3T/kEP5y1u64Itfe74yNBrFqjW2hIBI ty2Uc1orZvGJanmtlvVM+OzCQo7TdSLUwCkX7MNzCJqJYl0M4v7rrdKWKOmYC8GdoSCzc4 uYqcfpiSRXxF7qGzA/6RNSrGMb+Uwrc= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=jFDhujQK; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=PdU68wcF; spf=pass (imf17.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B89CB2187D; Thu, 10 Aug 2023 16:36:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1691685393; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cIe5o/xpdmWF4fStDltZQwKQCcPAraOzhHNBE1tj7sI=; b=jFDhujQK9jnwYC5KTNeGJCIlO2KBUw/XTV80/Sckd7ry6MpJADq8984Hcb/JcLyttdyOCc By0JjZpyPSMRUFs60gV0TZ5tiJmU656Gez67m2SNag97I5yOFGfQMRohSq16iolp/mFQ1E r7ASU8hfgP0p2GuDAsPTINkLoM6TUww= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1691685393; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cIe5o/xpdmWF4fStDltZQwKQCcPAraOzhHNBE1tj7sI=; b=PdU68wcF1isVbJVPgMJg1tS6L1fRdxB0QJp3XEvCF5yMZ3vLhenlySUtF9nHAX+uGFzMuT KyYySGhI6o/rPMDQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 8FB1C139D1; Thu, 10 Aug 2023 16:36:33 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id SCB6IhES1WSEPQAAMHmgww (envelope-from ); Thu, 10 Aug 2023 16:36:33 +0000 From: Vlastimil Babka To: "Liam R. Howlett" , Matthew Wilcox , Suren Baghdasaryan , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev, Vlastimil Babka Subject: [RFC v2 7/7] tools: Add SLUB percpu array functions for testing Date: Thu, 10 Aug 2023 18:36:35 +0200 Message-ID: <20230810163627.6206-16-vbabka@suse.cz> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230810163627.6206-9-vbabka@suse.cz> References: <20230810163627.6206-9-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Queue-Id: 32DAC40025 X-Rspam-User: X-Stat-Signature: a6he5fyzwe9w7ooazitbi4dkugc1ogos X-Rspamd-Server: rspam03 X-HE-Tag: 1691685394-110231 X-HE-Meta: U2FsdGVkX19a5qcNYP3B47ceu3r8QePOhJ5c0A9VHu32IHJzUJlmr2nRpcLoLcXhR0e0m4M0uXI5hp6mImOTJPmlxRcuUM2Z9JB09Axspcq8Oh/ng27Rj3UnK7NVPRVc0s9URS/ei68DEao8jqGP4J1Kvo33aZSmv2MhuJyWUgDKWqECNnGxC9g1W72EBcnuEtQmyRRLXN5WL3amk3OJG2yQ6o+AxLoZ6iyw03nWrFMs7uPEqSFBw15hE3BeV3mqazTbPd8aSEZESL+tlMgb4/EAUEihiQWULp43TLzelzraujB4J3TECEWg99oCbKN62CvmXtWkSwWgfTkgF+vLrAOjn1G9irSgYcbwRrlCrHMdLn7U2visNXzrDYuiDMMYAixmUcOwanj4GSRaUOa6NsuVEYg5XXUH5bH8oW8OA7KGkBE93CVYq2+QPI39Vnxeikqaoh5cSdjRQaO+9UOcYfqTvI8KwNcm3dX4wpenSrv3GXorSsbwEtkQ6u1FKCj3Hs3+MF5m0qebB30tQvkEwN1XXO8ylPp9BE+/VYfUuuh/aHlAdYhvta6U+w7dVZBGg2jvyhVZ+eKlazqp2dxyyLjqDqT4o9U8jZQLXoAT+HpbMGTCLn0GnTYRLSz0oxnfgBfOj/RP79MPIEDWsUxhCFAsP37/MN2wkc08QHUUBmj2F06A1N8HdQPJfbARdhUWXRXJ7zkIfuhsf0+Y1gnl/qgvaCT1/ZxxiqhmviW83wd7qDpnut6HTcyLLj01Lrq1G4u2CDbW0od3k/lLe8X4lmnnDbytuKjbStEBniiY3fhO8v7f3hIPv5CCwhk+HESPVO/510DuW9tere8JJNruHdRft3wN1eqXz0Qb0SWAojh0J0gmlv8rx93jsDk5gqTNoMASuSQdtAUUUSt8r4Ow1SoNaFsHGIi/zlA7RxAe8oedUTCTu4vK0YCkw2CXR8QcIFZRfjPM4WgLoMKZBfX WDOkwhgx MwmGH5qCFPI3ZIkPputEl8fZm8rbbNDyMQDxy1T+2mOTjuZ7tak1PpBuJWjeJwo7b2oeC0X4dBoT9s2r/VFHyyOdfJmAcUFY/vGWh/ZMJ03luCq2Y8Eks2I+U1Q8+tgML1vPoJBZB+gAWKH79hmkx034lowJL9YYSGUXBAjA+2BaGSNAJ6VNhm3xNm0ohnKi4FgYvJ21z2mNthUVxDuVpVAo2ag59t5ZFdEffmqblXEXhb0wJgQiEOOCqRwTDPrdeULOPSQ4G5ja7OehwLEKdlXRDmQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Support new percpu array functions to the test code so they can be used in the maple tree testing. Signed-off-by: Liam R. Howlett Signed-off-by: Vlastimil Babka --- tools/include/linux/slab.h | 4 ++++ tools/testing/radix-tree/linux.c | 14 ++++++++++++++ tools/testing/radix-tree/linux/kernel.h | 1 + 3 files changed, 19 insertions(+) diff --git a/tools/include/linux/slab.h b/tools/include/linux/slab.h index 311759ea25e9..1043f9c5ef4e 100644 --- a/tools/include/linux/slab.h +++ b/tools/include/linux/slab.h @@ -7,6 +7,7 @@ #define SLAB_PANIC 2 #define SLAB_RECLAIM_ACCOUNT 0x00020000UL /* Objects are reclaimable */ +#define SLAB_NO_MERGE 0x01000000UL /* Prevent merging with compatible kmem caches */ #define kzalloc_node(size, flags, node) kmalloc(size, flags) @@ -45,4 +46,7 @@ void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list); int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size, void **list); +int kmem_cache_setup_percpu_array(struct kmem_cache *s, unsigned int count); +int kmem_cache_prefill_percpu_array(struct kmem_cache *s, unsigned int count, + gfp_t gfp); #endif /* _TOOLS_SLAB_H */ diff --git a/tools/testing/radix-tree/linux.c b/tools/testing/radix-tree/linux.c index d587a558997f..cbe7937fdd5e 100644 --- a/tools/testing/radix-tree/linux.c +++ b/tools/testing/radix-tree/linux.c @@ -187,6 +187,20 @@ int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size, return size; } +int kmem_cache_setup_percpu_array(struct kmem_cache *s, unsigned int count) +{ + return 0; +} + +int kmem_cache_prefill_percpu_array(struct kmem_cache *s, unsigned int count, + gfp_t gfp) +{ + if (count > s->non_kernel) + return s->non_kernel; + + return count; +} + struct kmem_cache * kmem_cache_create(const char *name, unsigned int size, unsigned int align, unsigned int flags, void (*ctor)(void *)) diff --git a/tools/testing/radix-tree/linux/kernel.h b/tools/testing/radix-tree/linux/kernel.h index c5c9d05f29da..fc75018974de 100644 --- a/tools/testing/radix-tree/linux/kernel.h +++ b/tools/testing/radix-tree/linux/kernel.h @@ -15,6 +15,7 @@ #define printk printf #define pr_err printk +#define pr_warn printk #define pr_info printk #define pr_debug printk #define pr_cont printk