From patchwork Fri May 18 07:48:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10408365 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E350A601F9 for ; Fri, 18 May 2018 07:49:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D3C1B2886A for ; Fri, 18 May 2018 07:49:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C85032887D; Fri, 18 May 2018 07:49:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1FAD02886A for ; Fri, 18 May 2018 07:49:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BFC106B0584; Fri, 18 May 2018 03:49:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BAAD66B0585; Fri, 18 May 2018 03:49:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A72786B0586; Fri, 18 May 2018 03:49:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk0-f198.google.com (mail-qk0-f198.google.com [209.85.220.198]) by kanga.kvack.org (Postfix) with ESMTP id 7ACDA6B0584 for ; Fri, 18 May 2018 03:49:50 -0400 (EDT) Received: by mail-qk0-f198.google.com with SMTP id c132-v6so6179010qkg.10 for ; Fri, 18 May 2018 00:49:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=DukDmIDOCuZvOvYV7AxI2za4rxww2vVAjhaHcLiyQT0=; b=gvBO+nJM04L/LBHQzi/oodZs+3TANnd5260pATreOQDbojx7uQR0HM/b1thZHhSB5t 1yVJcCSqx4+EEW5Wp8G6jYgrMGFOIaK3oVpxh8KL1Hzuk7UkLqmfFW768d6fyqqwDHxT 6J8JrJDcydRd6SrdhFHVSQQgcv67N8eGsnvFh73+o9w8NS3PoI6K+Nx07dlvWxyz28yc gj//nYXvUgJezoCanGF9JIVbyh8fYqdSQnyCmKxJwv7oyoEi5xK9eIgCL17Yc0Ua3l7q kGQW+GX9kjCjKuA4kdUD5wNpdcOq45dkn9TkSuKwlw7ejotZC/LaSlxjQHDcxsMAo/ir NxoA== X-Gm-Message-State: ALKqPweqCUqtAfrRBelu0OccMmPk/EB2ocaCO1Tbpq4pCCsuv8Y4h6iu DlqfDVdqZI3Eg0Ji3ztV9kr9Bnzd5zKXIchGJ0nngrdzhftH5dmcsnYSxVpH9rkQaoJrUhaRW0U 3lV+B9iLsJudWbxFTbEaZ8Kc716eDJ0guF30YSxm75x4U+R1PcvaYjGJCIwvxyWqV/lmF0VJFYr wXV5qoCRyjijo63qbJuTixDmrIAfMcQ7RGpSXyFZnQwSxhoxH9PYQWYFRA4rXSkZqu7ompbOXFR s2146MVVEqag+JQlYgP8xVSn1pyL5fiHG8in7X3A+YF/1KpK5LTlb0i425fXPxUH3ef22uCGY1+ FbzniH2PkdIxiC7xfw9gI6rRE6lDFOOi5+vD9qv0OhTADSiyMypb4qM5IVvIBRtv5m6mjYtldls G X-Received: by 2002:a37:7645:: with SMTP id r66-v6mr8242480qkc.394.1526629790216; Fri, 18 May 2018 00:49:50 -0700 (PDT) X-Received: by 2002:a37:7645:: with SMTP id r66-v6mr8242453qkc.394.1526629789043; Fri, 18 May 2018 00:49:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1526629789; cv=none; d=google.com; s=arc-20160816; b=rlk4CT31CUHsCL/Tnan5zq7Ys4LN3sY8TARVJZFqNYvmgnsvOzXgJF6d5aLPuFN3cw /TlCbMkIuruvCi8R78xnde9Akr4uzacjT+9FSlQOtlXENgS7NNxnR7NcNMLIPq31f0nV 1Pn6XaxnXq4i7/NU/5JU0XWnX67xFZJBg5LNprISBetmW2mFYxzJ6WTAHg11fbXhPyBV dcAGmGfBMRIsRoo4/2Dbye755Bd6CrB3fgf35dYkpZDWjGgVQcNbhZG1XMGKDoOQsHbc HHkbaXUQ9D97ehNF+GTsEaDWM2eVDdsNtkWVA6UJLW+eF48h3AdKdA8KQyaa6MQ15OIh C0/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=DukDmIDOCuZvOvYV7AxI2za4rxww2vVAjhaHcLiyQT0=; b=T2UzaHDJSOiVLXNxl/QfE7CNn/FJZwSXSoqKfRsd65EHMrVat5rX8QlzVVSO1oMMes JwcP2zVNuNF10bWpbMbZWkPYHpopavndwGpKwyyrw6vCUERfFUxhint0saUCGkdeyMpn 6hBfmaj4haYxdJ6nqtMuncrrCdY6NsVtptHmwsqSN+iQpCKZ3a0kIrJVZPAEUI29kOdP lpppOs7BZm31h3zgGV88I7dLH0Fv95U71twUnw67z9Ga+s/nCQJXtENKRFEaIZvlLVLQ IrFsjmrclH+3rdWYR2GxU7WjeOv82nM2FZtd6eGoDWBqNA7r56ERaEiaKmEsY4MT0MXj 2sQw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=JrkMKYkt; spf=pass (google.com: domain of kent.overstreet@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=kent.overstreet@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id 50-v6sor4651655qvr.94.2018.05.18.00.49.48 for (Google Transport Security); Fri, 18 May 2018 00:49:49 -0700 (PDT) Received-SPF: pass (google.com: domain of kent.overstreet@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=JrkMKYkt; spf=pass (google.com: domain of kent.overstreet@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=kent.overstreet@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=DukDmIDOCuZvOvYV7AxI2za4rxww2vVAjhaHcLiyQT0=; b=JrkMKYkt+4umxXml1AgHMOTijXy0cc0FExHMlQkFaW1PF9dydW013/DCq29pdqvzEg ZY6o2gT4BGd6vl21lXFGKdSIeqjTQ0XFHFplYW5L0811SKIItrD0jejIheJKW7EPR8Bv AhgCFRq3FtHyDLtMx90fCVwJeHWLRVMMipTVPA1M+8mQoNpxLty3ldTw9pZ+Zm3DhO4i 1hoE0dIBNR4Ydp4LiV2A+n6Uu+0/miUCETE3ASWdRtEXHsm79cFj8l8oYzJbAEAX8tNa wuE2LfDGomL5Ef3ImORVpIWOrtLz+pxuuhwiJK79UqM7WFH0sIjxhyEFCwGldhnoj1l3 HIAA== X-Google-Smtp-Source: AB8JxZoOAlgwuItTMHrmbyK+qvJ0oGWMMyth7XlJnA/zgkkjM+mc0X0zlFZEUlKWB3EbJsEg2LWLDA== X-Received: by 2002:a0c:e64b:: with SMTP id c11-v6mr7934201qvn.73.1526629788660; Fri, 18 May 2018 00:49:48 -0700 (PDT) Received: from localhost.localdomain (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id s64-v6sm5443004qkl.85.2018.05.18.00.49.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 May 2018 00:49:47 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, Jens Axboe , Ingo Molnar Cc: Kent Overstreet Subject: [PATCH 01/10] mempool: Add mempool_init()/mempool_exit() Date: Fri, 18 May 2018 03:48:59 -0400 Message-Id: <20180518074918.13816-2-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180518074918.13816-1-kent.overstreet@gmail.com> References: <20180518074918.13816-1-kent.overstreet@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Allows mempools to be embedded in other structs, getting rid of a pointer indirection from allocation fastpaths. mempool_exit() is safe to call on an uninitialized but zeroed mempool. Signed-off-by: Kent Overstreet Reviewed-by: Johannes Thumshirn --- include/linux/mempool.h | 34 +++++++++++++ mm/mempool.c | 108 ++++++++++++++++++++++++++++++---------- 2 files changed, 115 insertions(+), 27 deletions(-) diff --git a/include/linux/mempool.h b/include/linux/mempool.h index b51f5c430c..0c964ac107 100644 --- a/include/linux/mempool.h +++ b/include/linux/mempool.h @@ -25,6 +25,18 @@ typedef struct mempool_s { wait_queue_head_t wait; } mempool_t; +static inline bool mempool_initialized(mempool_t *pool) +{ + return pool->elements != NULL; +} + +void mempool_exit(mempool_t *pool); +int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id); +int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data); + extern mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); extern mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, @@ -43,6 +55,14 @@ extern void mempool_free(void *element, mempool_t *pool); */ void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data); void mempool_free_slab(void *element, void *pool_data); + +static inline int +mempool_init_slab_pool(mempool_t *pool, int min_nr, struct kmem_cache *kc) +{ + return mempool_init(pool, min_nr, mempool_alloc_slab, + mempool_free_slab, (void *) kc); +} + static inline mempool_t * mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) { @@ -56,6 +76,13 @@ mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) */ void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data); void mempool_kfree(void *element, void *pool_data); + +static inline int mempool_init_kmalloc_pool(mempool_t *pool, int min_nr, size_t size) +{ + return mempool_init(pool, min_nr, mempool_kmalloc, + mempool_kfree, (void *) size); +} + static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) { return mempool_create(min_nr, mempool_kmalloc, mempool_kfree, @@ -68,6 +95,13 @@ static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) */ void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data); void mempool_free_pages(void *element, void *pool_data); + +static inline int mempool_init_page_pool(mempool_t *pool, int min_nr, int order) +{ + return mempool_init(pool, min_nr, mempool_alloc_pages, + mempool_free_pages, (void *)(long)order); +} + static inline mempool_t *mempool_create_page_pool(int min_nr, int order) { return mempool_create(min_nr, mempool_alloc_pages, mempool_free_pages, diff --git a/mm/mempool.c b/mm/mempool.c index 5c9dce3471..df90ace400 100644 --- a/mm/mempool.c +++ b/mm/mempool.c @@ -137,6 +137,28 @@ static void *remove_element(mempool_t *pool, gfp_t flags) return element; } +/** + * mempool_destroy - exit a mempool initialized with mempool_init() + * @pool: pointer to the memory pool which was initialized with + * mempool_init(). + * + * Free all reserved elements in @pool and @pool itself. This function + * only sleeps if the free_fn() function sleeps. + * + * May be called on a zeroed but uninitialized mempool (i.e. allocated with + * kzalloc()). + */ +void mempool_exit(mempool_t *pool) +{ + while (pool->curr_nr) { + void *element = remove_element(pool, GFP_KERNEL); + pool->free(element, pool->pool_data); + } + kfree(pool->elements); + pool->elements = NULL; +} +EXPORT_SYMBOL(mempool_exit); + /** * mempool_destroy - deallocate a memory pool * @pool: pointer to the memory pool which was allocated via @@ -150,15 +172,65 @@ void mempool_destroy(mempool_t *pool) if (unlikely(!pool)) return; - while (pool->curr_nr) { - void *element = remove_element(pool, GFP_KERNEL); - pool->free(element, pool->pool_data); - } - kfree(pool->elements); + mempool_exit(pool); kfree(pool); } EXPORT_SYMBOL(mempool_destroy); +int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id) +{ + spin_lock_init(&pool->lock); + pool->min_nr = min_nr; + pool->pool_data = pool_data; + pool->alloc = alloc_fn; + pool->free = free_fn; + init_waitqueue_head(&pool->wait); + + pool->elements = kmalloc_array_node(min_nr, sizeof(void *), + gfp_mask, node_id); + if (!pool->elements) + return -ENOMEM; + + /* + * First pre-allocate the guaranteed number of buffers. + */ + while (pool->curr_nr < pool->min_nr) { + void *element; + + element = pool->alloc(gfp_mask, pool->pool_data); + if (unlikely(!element)) { + mempool_exit(pool); + return -ENOMEM; + } + add_element(pool, element); + } + + return 0; +} +EXPORT_SYMBOL(mempool_init_node); + +/** + * mempool_init - initialize a memory pool + * @min_nr: the minimum number of elements guaranteed to be + * allocated for this pool. + * @alloc_fn: user-defined element-allocation function. + * @free_fn: user-defined element-freeing function. + * @pool_data: optional private data available to the user-defined functions. + * + * Like mempool_create(), but initializes the pool in (i.e. embedded in another + * structure). + */ +int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data) +{ + return mempool_init_node(pool, min_nr, alloc_fn, free_fn, + pool_data, GFP_KERNEL, NUMA_NO_NODE); + +} +EXPORT_SYMBOL(mempool_init); + /** * mempool_create - create a memory pool * @min_nr: the minimum number of elements guaranteed to be @@ -186,35 +258,17 @@ mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, gfp_t gfp_mask, int node_id) { mempool_t *pool; + pool = kzalloc_node(sizeof(*pool), gfp_mask, node_id); if (!pool) return NULL; - pool->elements = kmalloc_array_node(min_nr, sizeof(void *), - gfp_mask, node_id); - if (!pool->elements) { + + if (mempool_init_node(pool, min_nr, alloc_fn, free_fn, pool_data, + gfp_mask, node_id)) { kfree(pool); return NULL; } - spin_lock_init(&pool->lock); - pool->min_nr = min_nr; - pool->pool_data = pool_data; - init_waitqueue_head(&pool->wait); - pool->alloc = alloc_fn; - pool->free = free_fn; - /* - * First pre-allocate the guaranteed number of buffers. - */ - while (pool->curr_nr < pool->min_nr) { - void *element; - - element = pool->alloc(gfp_mask, pool->pool_data); - if (unlikely(!element)) { - mempool_destroy(pool); - return NULL; - } - add_element(pool, element); - } return pool; } EXPORT_SYMBOL(mempool_create_node);