From patchwork Fri May 18 07:48:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10408491 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8C7E1602CB for ; Fri, 18 May 2018 07:56:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7F4BC28875 for ; Fri, 18 May 2018 07:56:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 735CB28899; Fri, 18 May 2018 07:56:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D3B7528875 for ; Fri, 18 May 2018 07:56:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751577AbeERH4f (ORCPT ); Fri, 18 May 2018 03:56:35 -0400 Received: from mail-qt0-f194.google.com ([209.85.216.194]:33911 "EHLO mail-qt0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751221AbeERHtt (ORCPT ); Fri, 18 May 2018 03:49:49 -0400 Received: by mail-qt0-f194.google.com with SMTP id m5-v6so9177599qti.1; Fri, 18 May 2018 00:49:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=DukDmIDOCuZvOvYV7AxI2za4rxww2vVAjhaHcLiyQT0=; b=JrkMKYkt+4umxXml1AgHMOTijXy0cc0FExHMlQkFaW1PF9dydW013/DCq29pdqvzEg ZY6o2gT4BGd6vl21lXFGKdSIeqjTQ0XFHFplYW5L0811SKIItrD0jejIheJKW7EPR8Bv AhgCFRq3FtHyDLtMx90fCVwJeHWLRVMMipTVPA1M+8mQoNpxLty3ldTw9pZ+Zm3DhO4i 1hoE0dIBNR4Ydp4LiV2A+n6Uu+0/miUCETE3ASWdRtEXHsm79cFj8l8oYzJbAEAX8tNa wuE2LfDGomL5Ef3ImORVpIWOrtLz+pxuuhwiJK79UqM7WFH0sIjxhyEFCwGldhnoj1l3 HIAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=DukDmIDOCuZvOvYV7AxI2za4rxww2vVAjhaHcLiyQT0=; b=HNeIC040PtcFjbQW7ae91cfPETbQ6YzRVfw0yOr0iJa7d/Fjm4drtDR+DyYyHRiSYZ ySOSQRBMvzWxFJcT2/F0dYvNNHrrjHCDFtUj78Ep+93I1HEK3TPfQXAZclKycimBBaOb wfB3U8LQCMCl1aBSeW2zSlTwt5OnRTZfZwdkVt5R6hcduvGLS89P7E6llXutV6B5kOpC Y7/q1fpsPiPkdso+PijwtclZcjoImi9Tund4US6FrhlnD3wkBK+KmP02PmAnjpOx5h9s jdRqQJlY0wijUA0tq43u6gio2uO/rM1PuT4wkcomVqB5TbZg7IMeeeCT+T9zoDfUKwfc 77gQ== X-Gm-Message-State: ALKqPwcotfKZ/GTNbatIJjiiWU34WDMClKjD7L4ZRg4Cr3GPMUsteT5y U5mWiRGOTbRcKPWJlEzhZAJN4XN2cQ== X-Google-Smtp-Source: AB8JxZoOAlgwuItTMHrmbyK+qvJ0oGWMMyth7XlJnA/zgkkjM+mc0X0zlFZEUlKWB3EbJsEg2LWLDA== X-Received: by 2002:a0c:e64b:: with SMTP id c11-v6mr7934201qvn.73.1526629788660; Fri, 18 May 2018 00:49:48 -0700 (PDT) Received: from localhost.localdomain (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id s64-v6sm5443004qkl.85.2018.05.18.00.49.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 May 2018 00:49:47 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, Jens Axboe , Ingo Molnar Cc: Kent Overstreet Subject: [PATCH 01/10] mempool: Add mempool_init()/mempool_exit() Date: Fri, 18 May 2018 03:48:59 -0400 Message-Id: <20180518074918.13816-2-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180518074918.13816-1-kent.overstreet@gmail.com> References: <20180518074918.13816-1-kent.overstreet@gmail.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Allows mempools to be embedded in other structs, getting rid of a pointer indirection from allocation fastpaths. mempool_exit() is safe to call on an uninitialized but zeroed mempool. Signed-off-by: Kent Overstreet Reviewed-by: Johannes Thumshirn --- include/linux/mempool.h | 34 +++++++++++++ mm/mempool.c | 108 ++++++++++++++++++++++++++++++---------- 2 files changed, 115 insertions(+), 27 deletions(-) diff --git a/include/linux/mempool.h b/include/linux/mempool.h index b51f5c430c..0c964ac107 100644 --- a/include/linux/mempool.h +++ b/include/linux/mempool.h @@ -25,6 +25,18 @@ typedef struct mempool_s { wait_queue_head_t wait; } mempool_t; +static inline bool mempool_initialized(mempool_t *pool) +{ + return pool->elements != NULL; +} + +void mempool_exit(mempool_t *pool); +int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id); +int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data); + extern mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); extern mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, @@ -43,6 +55,14 @@ extern void mempool_free(void *element, mempool_t *pool); */ void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data); void mempool_free_slab(void *element, void *pool_data); + +static inline int +mempool_init_slab_pool(mempool_t *pool, int min_nr, struct kmem_cache *kc) +{ + return mempool_init(pool, min_nr, mempool_alloc_slab, + mempool_free_slab, (void *) kc); +} + static inline mempool_t * mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) { @@ -56,6 +76,13 @@ mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) */ void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data); void mempool_kfree(void *element, void *pool_data); + +static inline int mempool_init_kmalloc_pool(mempool_t *pool, int min_nr, size_t size) +{ + return mempool_init(pool, min_nr, mempool_kmalloc, + mempool_kfree, (void *) size); +} + static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) { return mempool_create(min_nr, mempool_kmalloc, mempool_kfree, @@ -68,6 +95,13 @@ static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) */ void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data); void mempool_free_pages(void *element, void *pool_data); + +static inline int mempool_init_page_pool(mempool_t *pool, int min_nr, int order) +{ + return mempool_init(pool, min_nr, mempool_alloc_pages, + mempool_free_pages, (void *)(long)order); +} + static inline mempool_t *mempool_create_page_pool(int min_nr, int order) { return mempool_create(min_nr, mempool_alloc_pages, mempool_free_pages, diff --git a/mm/mempool.c b/mm/mempool.c index 5c9dce3471..df90ace400 100644 --- a/mm/mempool.c +++ b/mm/mempool.c @@ -137,6 +137,28 @@ static void *remove_element(mempool_t *pool, gfp_t flags) return element; } +/** + * mempool_destroy - exit a mempool initialized with mempool_init() + * @pool: pointer to the memory pool which was initialized with + * mempool_init(). + * + * Free all reserved elements in @pool and @pool itself. This function + * only sleeps if the free_fn() function sleeps. + * + * May be called on a zeroed but uninitialized mempool (i.e. allocated with + * kzalloc()). + */ +void mempool_exit(mempool_t *pool) +{ + while (pool->curr_nr) { + void *element = remove_element(pool, GFP_KERNEL); + pool->free(element, pool->pool_data); + } + kfree(pool->elements); + pool->elements = NULL; +} +EXPORT_SYMBOL(mempool_exit); + /** * mempool_destroy - deallocate a memory pool * @pool: pointer to the memory pool which was allocated via @@ -150,15 +172,65 @@ void mempool_destroy(mempool_t *pool) if (unlikely(!pool)) return; - while (pool->curr_nr) { - void *element = remove_element(pool, GFP_KERNEL); - pool->free(element, pool->pool_data); - } - kfree(pool->elements); + mempool_exit(pool); kfree(pool); } EXPORT_SYMBOL(mempool_destroy); +int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id) +{ + spin_lock_init(&pool->lock); + pool->min_nr = min_nr; + pool->pool_data = pool_data; + pool->alloc = alloc_fn; + pool->free = free_fn; + init_waitqueue_head(&pool->wait); + + pool->elements = kmalloc_array_node(min_nr, sizeof(void *), + gfp_mask, node_id); + if (!pool->elements) + return -ENOMEM; + + /* + * First pre-allocate the guaranteed number of buffers. + */ + while (pool->curr_nr < pool->min_nr) { + void *element; + + element = pool->alloc(gfp_mask, pool->pool_data); + if (unlikely(!element)) { + mempool_exit(pool); + return -ENOMEM; + } + add_element(pool, element); + } + + return 0; +} +EXPORT_SYMBOL(mempool_init_node); + +/** + * mempool_init - initialize a memory pool + * @min_nr: the minimum number of elements guaranteed to be + * allocated for this pool. + * @alloc_fn: user-defined element-allocation function. + * @free_fn: user-defined element-freeing function. + * @pool_data: optional private data available to the user-defined functions. + * + * Like mempool_create(), but initializes the pool in (i.e. embedded in another + * structure). + */ +int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data) +{ + return mempool_init_node(pool, min_nr, alloc_fn, free_fn, + pool_data, GFP_KERNEL, NUMA_NO_NODE); + +} +EXPORT_SYMBOL(mempool_init); + /** * mempool_create - create a memory pool * @min_nr: the minimum number of elements guaranteed to be @@ -186,35 +258,17 @@ mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, gfp_t gfp_mask, int node_id) { mempool_t *pool; + pool = kzalloc_node(sizeof(*pool), gfp_mask, node_id); if (!pool) return NULL; - pool->elements = kmalloc_array_node(min_nr, sizeof(void *), - gfp_mask, node_id); - if (!pool->elements) { + + if (mempool_init_node(pool, min_nr, alloc_fn, free_fn, pool_data, + gfp_mask, node_id)) { kfree(pool); return NULL; } - spin_lock_init(&pool->lock); - pool->min_nr = min_nr; - pool->pool_data = pool_data; - init_waitqueue_head(&pool->wait); - pool->alloc = alloc_fn; - pool->free = free_fn; - /* - * First pre-allocate the guaranteed number of buffers. - */ - while (pool->curr_nr < pool->min_nr) { - void *element; - - element = pool->alloc(gfp_mask, pool->pool_data); - if (unlikely(!element)) { - mempool_destroy(pool); - return NULL; - } - add_element(pool, element); - } return pool; } EXPORT_SYMBOL(mempool_create_node);