From patchwork Sun Apr 7 13:08:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yunsheng Lin X-Patchwork-Id: 13620171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7E5FCD11C2 for ; Sun, 7 Apr 2024 13:11:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5EA016B009E; Sun, 7 Apr 2024 09:11:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 572A26B009F; Sun, 7 Apr 2024 09:11:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43A686B00A0; Sun, 7 Apr 2024 09:11:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 26A706B009E for ; Sun, 7 Apr 2024 09:11:33 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E5D89C0AD3 for ; Sun, 7 Apr 2024 13:11:32 +0000 (UTC) X-FDA: 81982772424.01.BB09176 Received: from szxga05-in.huawei.com (szxga05-in.huawei.com [45.249.212.191]) by imf03.hostedemail.com (Postfix) with ESMTP id 6F9A020012 for ; Sun, 7 Apr 2024 13:11:29 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf03.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.191 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712495491; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=li2k89W2YA223dg/YDNZv2vgDgEeU+CpLNgHU6ZpwWQ=; b=Xy2gx3v5ptPx53ZAzC5lTjiRCVz3enYDvHzm01oY6R2c7yprFj2qAdZn24e2pWo+Lfj8qj nS0OUKsb1ExKGGX8eTVtxP6X/e05Hw3Qds96U5LInVEwrQ7RU7yrF807rJHaL8dwfvx1PT /rLH3uDpUds5lB1+1XBHH98QpFlFqWk= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf03.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.191 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712495491; a=rsa-sha256; cv=none; b=S8NxQegZWaq8IrnNSgLwZ1u2Ps4w9sSPRi2O9A0TUUPN+o4Wx+uBd0cSNQrSWoZkPuR3UC uOH0g9mjFOMfzsRt3EIhZgihJ7GCib4mBspzwQZrHiw7xvY/041T+AxYMUj+an7ahGMU6p qj5Zo9X2H03wFu6Kf9zaEtrvdp1dJ6E= Received: from mail.maildlp.com (unknown [172.19.163.17]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4VCCJh3KRmz1GG52; Sun, 7 Apr 2024 21:10:44 +0800 (CST) Received: from dggpemm500005.china.huawei.com (unknown [7.185.36.74]) by mail.maildlp.com (Postfix) with ESMTPS id 661961A0172; Sun, 7 Apr 2024 21:11:27 +0800 (CST) Received: from localhost.localdomain (10.69.192.56) by dggpemm500005.china.huawei.com (7.185.36.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Sun, 7 Apr 2024 21:11:27 +0800 From: Yunsheng Lin To: , , CC: , , Yunsheng Lin , Andrew Morton , Subject: [PATCH net-next v1 11/12] mm: page_frag: add a test module for page_frag Date: Sun, 7 Apr 2024 21:08:48 +0800 Message-ID: <20240407130850.19625-12-linyunsheng@huawei.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20240407130850.19625-1-linyunsheng@huawei.com> References: <20240407130850.19625-1-linyunsheng@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.56] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpemm500005.china.huawei.com (7.185.36.74) X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 6F9A020012 X-Stat-Signature: m1975yfyq55pgjw9x5akdqz4akk65stj X-HE-Tag: 1712495489-908871 X-HE-Meta: U2FsdGVkX18EUP0vkzwj5Yh5W8eQBL6Kgf94lV/UpaEeKA7SfyTvygeob9bvpJn6snZLlutop9i0lPNsdsiS5MC2RMbR7TnlzS3EU+oAom7Bl0WNp3CIWVHbtQ2Vpo8c1B/8mW5mGxUEGgGmuCJTSlnlnXzwDfcarK58zYDQJFi/mNJRXD1MBak34SSwuhBz2SUzRYg8FG2UsX3wuOzTA0mTZqB8eHGJ7bfpncPVWjaULu+ncBkKHCc8j06sPP4QyHSQZNcKaiiuUJotKz2BvgFusIqJr3WGaj9AVq4F5ngr3el0UKgLJJ8viENAklt49/TCBhqCGaDf+DKzQLZxPHFmUjOrr+Urm3l6RnZQtZfpd9g0vV2AIHwSM5bbT7BBzO4mawaT62FSoe1xG42cDlHoFD7PrfF2UMU4osvSN+Hx/HlwMhS2y4YCMsuoB+FHZ6qtRSrxKPtZ2jfeTT+Y28xK5eFVrHXN6QPeudGKenTviy5p+DQ2brS1dGgO5hO4o9Nrz91+RgW/6wwNtyOI43Df0vUK93DhlqmBcMbr6yW+c2OC09ZIxO6VVkcKT4v6GyoQH3kyoFB1+2pBXBHVTms/fEOzZO1e0Z1XMyxLPW2jKBq1L/vQo+Wd3zWYPCwtFuwmn10QDn8lo6Ys0amEejlrdLTRwmHzIkKj2Sxbx1hYr9fwMmJjMIyscvV60cbun1n2KJvCskp+d6JwktFX0M7lVhUlK2v1E5j1kcbAFH6whQiTcCFlZL08IVj6IMyIijzbgVdzeCqCcwKA4Mtl4LB9EirvS6SIgwS9ajwm/k7y/Cll5jgKZ+jdmiaK/4Unf1K4jTtuvuXaVbyP6/Noo4S8m0F2NpY8hUJ5jXrU1nzPNdMfiBvCgfoMqpVggyuozZUCDqvTbt5p5lHxswFeAsJwCIfMDOC45Rvo2wgCVlYMdNynfIAPNRFKR8kmSdN1zWAfEi1y7XATpyjs+Tp IzlUUqt1 rnYwFWurr+px93d+WnY1VMlEfQ+5HSv+3tvbyb2nz1apDw7evXRonZQFiY+KP69vYldrX9V+FPdamgP4MGAHmwR5HE7Hdrq0SPc92yA932KBAwSkgMGTiUdhTcU3hELSZCtmsm8Z+3H7aQeKQU3h/tu1+H6hZIJYZ+sJR5oTVW5WUrDnnJ4/dmFPfgU+vVz188xFhmxfFBM/SJUvItHHSdrMFPqqvLRZqW0nP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Basing on the lib/objpool.c, change it to something like a ptrpool, so that we can utilize that to test the correctness and performance of the page_frag. The testing is done by ensuring that the fragments allocated from a frag_frag_cache instance is pushed into a ptrpool instance in a kthread binded to the first cpu, and a kthread binded to the current node will pop the fragmemt from the ptrpool and call page_frag_alloc_va() to free the fragmemt. We may refactor out the common part between objpool and ptrpool if this ptrpool thing turns out to be helpful for other place. Signed-off-by: Yunsheng Lin --- mm/Kconfig.debug | 8 + mm/Makefile | 1 + mm/page_frag_test.c | 366 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 375 insertions(+) create mode 100644 mm/page_frag_test.c diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index afc72fde0f03..1ebcd45f47d4 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -142,6 +142,14 @@ config DEBUG_PAGE_REF kernel code. However the runtime performance overhead is virtually nil until the tracepoints are actually enabled. +config DEBUG_PAGE_FRAG_TEST + tristate "Test module for page_frag" + default n + depends on m && DEBUG_KERNEL + help + This builds the "page_frag_test" module that is used to test the + correctness and performance of page_frag's implementation. + config DEBUG_RODATA_TEST bool "Testcase for the marking rodata read-only" depends on STRICT_KERNEL_RWX diff --git a/mm/Makefile b/mm/Makefile index 146c481c006f..8b62f5de48a7 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -102,6 +102,7 @@ obj-$(CONFIG_MEMORY_FAILURE) += memory-failure.o obj-$(CONFIG_HWPOISON_INJECT) += hwpoison-inject.o obj-$(CONFIG_DEBUG_KMEMLEAK) += kmemleak.o obj-$(CONFIG_DEBUG_RODATA_TEST) += rodata_test.o +obj-$(CONFIG_DEBUG_PAGE_FRAG_TEST) += page_frag_test.o obj-$(CONFIG_DEBUG_VM_PGTABLE) += debug_vm_pgtable.o obj-$(CONFIG_PAGE_OWNER) += page_owner.o obj-$(CONFIG_MEMORY_ISOLATION) += page_isolation.o diff --git a/mm/page_frag_test.c b/mm/page_frag_test.c new file mode 100644 index 000000000000..e311c2e7ff49 --- /dev/null +++ b/mm/page_frag_test.c @@ -0,0 +1,366 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* + * Test module for page_frag cache + * + * Copyright: linyunsheng@huawei.com + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define OBJPOOL_NR_OBJECT_MAX BIT(24) + +struct objpool_slot { + u32 head; + u32 tail; + u32 last; + u32 mask; + void *entries[]; +} __packed; + +struct objpool_head { + int nr_cpus; + int capacity; + struct objpool_slot **cpu_slots; +}; + +/* initialize percpu objpool_slot */ +static void objpool_init_percpu_slot(struct objpool_head *pool, + struct objpool_slot *slot) +{ + /* initialize elements of percpu objpool_slot */ + slot->mask = pool->capacity - 1; +} + +/* allocate and initialize percpu slots */ +static int objpool_init_percpu_slots(struct objpool_head *pool, + int nr_objs, gfp_t gfp) +{ + int i; + + for (i = 0; i < pool->nr_cpus; i++) { + struct objpool_slot *slot; + int size; + + /* skip the cpu node which could never be present */ + if (!cpu_possible(i)) + continue; + + size = struct_size(slot, entries, pool->capacity); + + /* + * here we allocate percpu-slot & objs together in a single + * allocation to make it more compact, taking advantage of + * warm caches and TLB hits. in default vmalloc is used to + * reduce the pressure of kernel slab system. as we know, + * mimimal size of vmalloc is one page since vmalloc would + * always align the requested size to page size + */ + if (gfp & GFP_ATOMIC) + slot = kmalloc_node(size, gfp, cpu_to_node(i)); + else + slot = __vmalloc_node(size, sizeof(void *), gfp, + cpu_to_node(i), + __builtin_return_address(0)); + if (!slot) + return -ENOMEM; + + memset(slot, 0, size); + pool->cpu_slots[i] = slot; + + objpool_init_percpu_slot(pool, slot); + } + + return 0; +} + +/* cleanup all percpu slots of the object pool */ +static void objpool_fini_percpu_slots(struct objpool_head *pool) +{ + int i; + + if (!pool->cpu_slots) + return; + + for (i = 0; i < pool->nr_cpus; i++) + kvfree(pool->cpu_slots[i]); + kfree(pool->cpu_slots); +} + +/* initialize object pool and pre-allocate objects */ +static int objpool_init(struct objpool_head *pool, int nr_objs, gfp_t gfp) +{ + int rc, capacity, slot_size; + + /* check input parameters */ + if (nr_objs <= 0 || nr_objs > OBJPOOL_NR_OBJECT_MAX) + return -EINVAL; + + /* calculate capacity of percpu objpool_slot */ + capacity = roundup_pow_of_two(nr_objs); + if (!capacity) + return -EINVAL; + + gfp = gfp & ~__GFP_ZERO; + + /* initialize objpool pool */ + memset(pool, 0, sizeof(struct objpool_head)); + pool->nr_cpus = nr_cpu_ids; + pool->capacity = capacity; + slot_size = pool->nr_cpus * sizeof(struct objpool_slot *); + pool->cpu_slots = kzalloc(slot_size, gfp); + if (!pool->cpu_slots) + return -ENOMEM; + + /* initialize per-cpu slots */ + rc = objpool_init_percpu_slots(pool, nr_objs, gfp); + if (rc) + objpool_fini_percpu_slots(pool); + + return rc; +} + +/* adding object to slot, abort if the slot was already full */ +static int objpool_try_add_slot(void *obj, struct objpool_head *pool, int cpu) +{ + struct objpool_slot *slot = pool->cpu_slots[cpu]; + u32 head, tail; + + /* loading tail and head as a local snapshot, tail first */ + tail = READ_ONCE(slot->tail); + + do { + head = READ_ONCE(slot->head); + /* slot is full */ + if (unlikely(tail - head >= pool->capacity)) + return -ENOSPC; + } while (!try_cmpxchg_acquire(&slot->tail, &tail, tail + 1)); + + /* now the tail position is reserved for the given obj */ + WRITE_ONCE(slot->entries[tail & slot->mask], obj); + /* update sequence to make this obj available for pop() */ + smp_store_release(&slot->last, tail + 1); + + return 0; +} + +/* reclaim an object to object pool */ +static int objpool_push(void *obj, struct objpool_head *pool) +{ + unsigned long flags; + int rc; + + /* disable local irq to avoid preemption & interruption */ + raw_local_irq_save(flags); + rc = objpool_try_add_slot(obj, pool, raw_smp_processor_id()); + raw_local_irq_restore(flags); + + return rc; +} + +/* try to retrieve object from slot */ +static void *objpool_try_get_slot(struct objpool_head *pool, int cpu) +{ + struct objpool_slot *slot = pool->cpu_slots[cpu]; + /* load head snapshot, other cpus may change it */ + u32 head = smp_load_acquire(&slot->head); + + while (head != READ_ONCE(slot->last)) { + void *obj; + + /* + * data visibility of 'last' and 'head' could be out of + * order since memory updating of 'last' and 'head' are + * performed in push() and pop() independently + * + * before any retrieving attempts, pop() must guarantee + * 'last' is behind 'head', that is to say, there must + * be available objects in slot, which could be ensured + * by condition 'last != head && last - head <= nr_objs' + * that is equivalent to 'last - head - 1 < nr_objs' as + * 'last' and 'head' are both unsigned int32 + */ + if (READ_ONCE(slot->last) - head - 1 >= pool->capacity) { + head = READ_ONCE(slot->head); + continue; + } + + /* obj must be retrieved before moving forward head */ + obj = READ_ONCE(slot->entries[head & slot->mask]); + + /* move head forward to mark it's consumption */ + if (try_cmpxchg_release(&slot->head, &head, head + 1)) + return obj; + } + + return NULL; +} + +/* allocate an object from object pool */ +static void *objpool_pop(struct objpool_head *pool) +{ + void *obj = NULL; + unsigned long flags; + int i, cpu; + + /* disable local irq to avoid preemption & interruption */ + raw_local_irq_save(flags); + + cpu = raw_smp_processor_id(); + for (i = 0; i < num_possible_cpus(); i++) { + obj = objpool_try_get_slot(pool, cpu); + if (obj) + break; + cpu = cpumask_next_wrap(cpu, cpu_possible_mask, -1, 1); + } + raw_local_irq_restore(flags); + + return obj; +} + +/* release whole objpool forcely */ +static void objpool_free(struct objpool_head *pool) +{ + if (!pool->cpu_slots) + return; + + /* release percpu slots */ + objpool_fini_percpu_slots(pool); +} + +static struct objpool_head ptr_pool; +static int nr_objs = 512; +static int nr_test = 5120000; +static atomic_t nthreads; +static struct completion wait; +struct page_frag_cache test_frag; + +module_param(nr_test, int, 0600); +MODULE_PARM_DESC(nr_test, "number of iterations to test"); + +static int page_frag_pop_thread(void *arg) +{ + struct objpool_head *pool = arg; + int nr = nr_test; + + pr_info("page_frag pop test thread begins on cpu %d\n", + smp_processor_id()); + + while (nr > 0) { + void *obj = objpool_pop(pool); + + if (obj) { + nr--; + page_frag_free_va(obj); + } else { + cond_resched(); + } + } + + if (atomic_dec_and_test(&nthreads)) + complete(&wait); + + pr_info("page_frag pop test thread exits on cpu %d\n", + smp_processor_id()); + + return 0; +} + +static int page_frag_push_thread(void *arg) +{ + struct objpool_head *pool = arg; + int nr = nr_test; + + pr_info("page_frag push test thread begins on cpu %d\n", + smp_processor_id()); + + while (nr > 0) { + unsigned int size = get_random_u32(); + void *va; + int ret; + + size = clamp(size, sizeof(unsigned int), PAGE_SIZE); + va = page_frag_alloc_va(&test_frag, size, GFP_KERNEL); + if (!va) + continue; + + ret = objpool_push(va, pool); + if (ret) { + page_frag_free_va(va); + cond_resched(); + } else { + nr--; + } + } + + pr_info("page_frag push test thread exits on cpu %d\n", + smp_processor_id()); + + if (atomic_dec_and_test(&nthreads)) + complete(&wait); + + return 0; +} + +static int __init page_frag_test_init(void) +{ + struct task_struct *tsk_push, *tsk_pop; + ktime_t start; + u64 duration; + int ret; + + page_frag_cache_init(&test_frag); + atomic_set(&nthreads, 2); + init_completion(&wait); + + ret = objpool_init(&ptr_pool, nr_objs, GFP_KERNEL); + if (ret) + return ret; + + tsk_push = kthread_create_on_cpu(page_frag_push_thread, &ptr_pool, + cpumask_first(cpu_online_mask), + "page_frag_push"); + if (IS_ERR(tsk_push)) + return PTR_ERR(tsk_push); + + tsk_pop = kthread_create(page_frag_pop_thread, &ptr_pool, + "page_frag_pop"); + if (IS_ERR(tsk_pop)) { + kthread_stop(tsk_push); + return PTR_ERR(tsk_pop); + } + + start = ktime_get(); + wake_up_process(tsk_push); + wake_up_process(tsk_pop); + + pr_info("waiting for test to complete\n"); + wait_for_completion(&wait); + + duration = (u64)ktime_us_delta(ktime_get(), start); + pr_info("%d of iterations took: %lluus\n", nr_test, duration); + + objpool_free(&ptr_pool); + page_frag_cache_drain(&test_frag); + + return -EAGAIN; +} + +static void __exit page_frag_test_exit(void) +{ +} + +module_init(page_frag_test_init); +module_exit(page_frag_test_exit); + +MODULE_LICENSE("GPL");