From patchwork Mon Sep 20 15:48:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12505757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,HK_RANDOM_FROM,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39EC7C433F5 for ; Mon, 20 Sep 2021 15:49:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B552D6115B for ; Mon, 20 Sep 2021 15:49:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B552D6115B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id EEA45900002; Mon, 20 Sep 2021 11:49:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E9A606B0072; Mon, 20 Sep 2021 11:49:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D616B900002; Mon, 20 Sep 2021 11:49:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0149.hostedemail.com [216.40.44.149]) by kanga.kvack.org (Postfix) with ESMTP id C2FEB6B006C for ; Mon, 20 Sep 2021 11:49:24 -0400 (EDT) Received: from smtpin40.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 2E29D2CFFB for ; Mon, 20 Sep 2021 15:49:24 +0000 (UTC) X-FDA: 78608386248.40.1FBC748 Received: from mail-pg1-f172.google.com (mail-pg1-f172.google.com [209.85.215.172]) by imf19.hostedemail.com (Postfix) with ESMTP id CE1B6B0000A3 for ; Mon, 20 Sep 2021 15:49:23 +0000 (UTC) Received: by mail-pg1-f172.google.com with SMTP id h3so17785377pgb.7 for ; Mon, 20 Sep 2021 08:49:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=pLEY0q0GQ36ytZataO/Ltnrmp+MHaUX8/R4Sa9g1JnU=; b=bI6+7GFFOdcVQbFBj2offTrqroyvmna9AkgtJx6uw9alggXu+oyxncTYw4dWXcVcqh Stlgtiz8A5+DtN0dOEhTORTi6iC7pz8i+WUp3YlmPa8Ie444eQahBgNRuyIKgTHDJzpn qRe5xvgAsC+09wgCO97qHMMDsEC5ckKNUauJ4miD3j3eQy4ECC2rY8UhBTTtpyngRYIn iDzZtUX/oPub82DJSZZqNaKav6IM2m/5HEL5BpAMJDnvi+dJ8UuFdTUhqcSNEvh6do95 VO7PUrE/bweL+RTXVMkM0/Spny6oZX8ujBcySfVjyZ9aas6bGMLFCoWEdz3uWfp/UHDg jDKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=pLEY0q0GQ36ytZataO/Ltnrmp+MHaUX8/R4Sa9g1JnU=; b=xr8D5nN+h2M7KugYjoPcSmb3GFwqKXFb56E7rQM6YRw60DJdP9h1m3KJBl0rRvQktP 5rZxrgHCwIEATcrvDGCWecWrFHHFjKu6ePQeVMfNtEEKZ8IzGl/2DSP9LNQgMsrJ3Y2N eSFgCbYn72QRI0guYOmBVaETlquhzvTGuGvPOhnf2JVCQggV6bnkWajIUcL+SNaBeNuj 7y5UMX8faXr/9sp5BCbOD+nVr1ZDjfUWCmk/hiH8vyJUNS23G0LY5Y2fZrz1qdXhmwjZ H7n7sBmdkin8OZZjCVDDsBpWUGGeA2i82B4MpaFcjiMy+f4XZsDeN3XMNg0qQ5kcdDtV Bl7w== X-Gm-Message-State: AOAM5318yzpbacHc1h17+4jfMpSUhZUWkwlnhm8jqeV5tVXyCsxpTBKZ ftsRDU0Q46WmlqGhXU1KJHS2nmpV9eo= X-Google-Smtp-Source: ABdhPJy3URQnASeV8LURZeBCnwluXPeiiCH1e30n7vOPDzmBAO77aPYVqExHafravrxNaBP0/SebuQ== X-Received: by 2002:a05:6a00:a8a:b029:30c:a10b:3e3f with SMTP id b10-20020a056a000a8ab029030ca10b3e3fmr25357425pfl.40.1632152962500; Mon, 20 Sep 2021 08:49:22 -0700 (PDT) Received: from kvm.asia-northeast3-a.c.our-ratio-313919.internal (252.229.64.34.bc.googleusercontent.com. [34.64.229.252]) by smtp.gmail.com with ESMTPSA id t6sm2391564pfh.63.2021.09.20.08.49.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Sep 2021 08:49:22 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , linux-kernel@vger.kernel.org, Matthew Wilcox , Jens Axboe , John Garry , linux-block@vger.kernel.org, netdev@vger.kernel.org Subject: [RFC v2 PATCH] mm, sl[au]b: Introduce lockless cache Date: Mon, 20 Sep 2021 15:48:16 +0000 Message-Id: <20210920154816.31832-1-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: CE1B6B0000A3 X-Stat-Signature: 8rz6nojq9mzfgrnzoobhu7rdrojpimb3 Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=bI6+7GFF; spf=pass (imf19.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.172 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1632152963-191308 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is RFC v2 of lockless cache on slab, for situation like IO Polling. It is untested, and just simple proof of concept yet. So there will be things to improve or erroneous code. (I'm sure of it) Any opinions or suggestions will be appreciated a lot! v1 is here: https://lore.kernel.org/linux-mm/20210919164239.49905-1-42.hyeyoo@gmail.com/ Changes since v1: - It was implemented as separate layer from slab, but it is now in slab. - Changed linked list to array Things to think about, or things to work on: - Applying limit, batchcount like SLAB - I suspect if it does make sence to implment it in SLOB/SLAB. - Can we improve it's mechanism depending on SL[AOU]B? - Test needed - Finding and fixing erroneous code :( --- include/linux/slab.h | 23 ++++++++++++++ include/linux/slab_def.h | 2 ++ include/linux/slub_def.h | 1 + mm/slab_common.c | 66 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 92 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 083f3ce550bc..091f514dc8e0 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -120,6 +120,9 @@ /* Slab deactivation flag */ #define SLAB_DEACTIVATED ((slab_flags_t __force)0x10000000U) +/* use percpu lockless cache */ +#define SLAB_LOCKLESS_CACHE ((slab_flags_t __force)0x20000000U) + /* * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. * @@ -327,6 +330,13 @@ enum kmalloc_cache_type { NR_KMALLOC_TYPES }; +#define KMEM_LOCKLESS_CACHE_QUEUE_SIZE 64 + +struct kmem_lockless_cache { + void *queue[KMEM_LOCKLESS_CACHE_QUEUE_SIZE]; + unsigned int size; +}; + #ifndef CONFIG_SLOB extern struct kmem_cache * kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; @@ -429,6 +439,19 @@ void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __malloc; void *kmem_cache_alloc(struct kmem_cache *, gfp_t flags) __assume_slab_alignment __malloc; void kmem_cache_free(struct kmem_cache *, void *); +#ifndef CONFIG_SLOB + +void *kmem_cache_alloc_cached(struct kmem_cache *s, gfp_t gfpflags); +void kmem_cache_free_cached(struct kmem_cache *s, void *p); + +#else + +#define kmem_cache_alloc_cached kmem_cache_alloc +#define kmem_cache_free_cached kmem_cache_free + +#endif /* CONFIG_SLOB */ + + /* * Bulk allocation and freeing operations. These are accelerated in an * allocator specific way to avoid taking locks repeatedly or building diff --git a/include/linux/slab_def.h b/include/linux/slab_def.h index 3aa5e1e73ab6..9f3161f38a8a 100644 --- a/include/linux/slab_def.h +++ b/include/linux/slab_def.h @@ -85,6 +85,8 @@ struct kmem_cache { unsigned int usersize; /* Usercopy region size */ struct kmem_cache_node *node[MAX_NUMNODES]; + + struct kmem_lockless_cache __percpu *cache; /* percpu lockless cache */ }; static inline void *nearest_obj(struct kmem_cache *cache, struct page *page, diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index 85499f0586b0..1dc3527efba8 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -96,6 +96,7 @@ struct kmem_cache { unsigned int object_size;/* The size of an object without metadata */ struct reciprocal_value reciprocal_size; unsigned int offset; /* Free pointer offset */ + struct kmem_lockless_cache __percpu *cache; /* percpu lockless cache */ #ifdef CONFIG_SLUB_CPU_PARTIAL /* Number of per cpu partial objects to keep around */ unsigned int cpu_partial; diff --git a/mm/slab_common.c b/mm/slab_common.c index ec2bb0beed75..5b8e4d5a644d 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -262,6 +262,13 @@ static struct kmem_cache *create_cache(const char *name, s->useroffset = useroffset; s->usersize = usersize; + if (flags & SLAB_LOCKLESS_CACHE) { + s->cache = alloc_percpu(struct kmem_lockless_cache); + if (!s->cache) + goto out_free_cache; + s->cache->size = 0; + } + err = __kmem_cache_create(s, flags); if (err) goto out_free_cache; @@ -424,6 +431,57 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align, } EXPORT_SYMBOL(kmem_cache_create); +/** + * kmem_cache_alloc_cached - try to allocate from cache without lock + * @s: slab cache + * @flags: SLAB flags + * + * Try to allocate from cache without lock. If fails, fill the lockless cache + * using bulk alloc API + * + * Be sure that there's no race condition. + * Must create slab cache with SLAB_LOCKLESS_CACHE flag to use this function. + * + * Return: a pointer to free object on allocation success, NULL on failure. + */ +void *kmem_cache_alloc_cached(struct kmem_cache *s, gfp_t gfpflags) +{ + struct kmem_lockless_cache *cache = this_cpu_ptr(s->cache); + + BUG_ON(!(s->flags & SLAB_LOCKLESS_CACHE)); + + if (cache->size) /* fastpath without lock */ + return cache->queue[--cache->size]; + + /* slowpath */ + cache->size = kmem_cache_alloc_bulk(s, gfpflags, + KMEM_LOCKLESS_CACHE_QUEUE_SIZE, cache->queue); + if (cache->size) + return cache->queue[--cache->size]; + else + return NULL; +} +EXPORT_SYMBOL(kmem_cache_alloc_cached); + +/** + * kmem_cache_free_cached - return object to cache + * @s: slab cache + * @p: pointer to free + */ +void kmem_cache_free_cached(struct kmem_cache *s, void *p) +{ + struct kmem_lockless_cache *cache = this_cpu_ptr(s->cache); + + BUG_ON(!(s->flags & SLAB_LOCKLESS_CACHE)); + + /* Is there better way to do this? */ + if (cache->size == KMEM_LOCKLESS_CACHE_QUEUE_SIZE) + kmem_cache_free(s, cache->queue[--cache->size]); + + cache->queue[cache->size++] = p; +} +EXPORT_SYMBOL(kmem_cache_free_cached); + static void slab_caches_to_rcu_destroy_workfn(struct work_struct *work) { LIST_HEAD(to_destroy); @@ -460,6 +518,8 @@ static void slab_caches_to_rcu_destroy_workfn(struct work_struct *work) static int shutdown_cache(struct kmem_cache *s) { + struct kmem_lockless_cache *cache; + /* free asan quarantined objects */ kasan_cache_shutdown(s); @@ -468,6 +528,12 @@ static int shutdown_cache(struct kmem_cache *s) list_del(&s->list); + if (s->flags & SLAB_LOCKLESS_CACHE) { + cache = this_cpu_ptr(s->cache); + kmem_cache_free_bulk(s, cache->size, cache->queue); + free_percpu(s->cache); + } + if (s->flags & SLAB_TYPESAFE_BY_RCU) { #ifdef SLAB_SUPPORTS_SYSFS sysfs_slab_unlink(s);