From patchwork Mon Aug 23 20:42:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ritesh Harjani X-Patchwork-Id: 12453655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B612AC4338F for ; Mon, 23 Aug 2021 20:42:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3B10761357 for ; Mon, 23 Aug 2021 20:42:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3B10761357 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 89F338D0001; Mon, 23 Aug 2021 16:42:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 84F8A6B0071; Mon, 23 Aug 2021 16:42:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 717718D0001; Mon, 23 Aug 2021 16:42:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0228.hostedemail.com [216.40.44.228]) by kanga.kvack.org (Postfix) with ESMTP id 584866B006C for ; Mon, 23 Aug 2021 16:42:40 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id E916E8249980 for ; Mon, 23 Aug 2021 20:42:39 +0000 (UTC) X-FDA: 78507518838.14.40CFA5C Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf24.hostedemail.com (Postfix) with ESMTP id 832E6B0000A2 for ; Mon, 23 Aug 2021 20:42:39 +0000 (UTC) Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17NKY73v035222; Mon, 23 Aug 2021 16:42:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : mime-version : content-transfer-encoding; s=pp1; bh=9rXrXUL2EH5bsC/ieEUl845iVI4rREWihUpykiR5eDo=; b=e3pnvMf6fg0Eoct12XvJ1iww9L9bWrJBRnMQUAwTJAyGlsYjP4gK7CzSP05ilB820Dpa R6hiveCXHDPMSj+ClxmWjgB64u+WI17WLmEZZ6WfPTWqm2I6LaRwzJ24PHn+CyKSR8Ym Ow0Go0sHAYxY5awpqagkCqLVqPqgPPbbDzVVQJW8htz2On7JnN30soFfwABLOmgNkcvX xU8WX4AYx8hUewtlR+HjiiK2LuEsCjQLyVChjj+QTeMsA/682Et3UFxPkEbM1m60uWpN LwuRYbElWPzJ2d66NnuPcG+dxnQ1k8zM/NXEwul0amxo7SKyAv0RXPrV/l8tPt3b42V5 iA== Received: from ppma03ams.nl.ibm.com (62.31.33a9.ip4.static.sl-reverse.com [169.51.49.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 3amb23qns6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 16:42:37 -0400 Received: from pps.filterd (ppma03ams.nl.ibm.com [127.0.0.1]) by ppma03ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17NKXS0F001938; Mon, 23 Aug 2021 20:42:35 GMT Received: from b06avi18626390.portsmouth.uk.ibm.com (b06avi18626390.portsmouth.uk.ibm.com [9.149.26.192]) by ppma03ams.nl.ibm.com with ESMTP id 3ajs48bucf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 20:42:35 +0000 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06avi18626390.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17NKcrlk57671958 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Aug 2021 20:38:53 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 22F794C052; Mon, 23 Aug 2021 20:42:33 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B3A2F4C046; Mon, 23 Aug 2021 20:42:32 +0000 (GMT) Received: from localhost (unknown [9.102.2.197]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 23 Aug 2021 20:42:32 +0000 (GMT) From: Ritesh Harjani To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Dennis Zhou , Tejun Heo , Christoph Lameter , Ritesh Harjani Subject: [PATCHv2 1/2] kernel/workqueue: Make schedule_on_each_cpu as EXPORT_SYMBOL_GPL Date: Tue, 24 Aug 2021 02:12:29 +0530 Message-Id: <5afc2a0c4da65e71ccf24fe65396710d34fc662e.1629751104.git.riteshh@linux.ibm.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: IGHOdOMC2wUIWC8IsNTsQ9uDG8Hh0IC5 X-Proofpoint-ORIG-GUID: IGHOdOMC2wUIWC8IsNTsQ9uDG8Hh0IC5 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790 definitions=2021-08-23_04:2021-08-23,2021-08-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 clxscore=1015 malwarescore=0 suspectscore=0 adultscore=0 phishscore=0 spamscore=0 mlxscore=0 mlxlogscore=711 lowpriorityscore=0 impostorscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108230140 Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=e3pnvMf6; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf24.hostedemail.com: domain of riteshh@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=riteshh@linux.ibm.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 832E6B0000A2 X-Stat-Signature: soc9bcmdhx34zbfrtr1n89kb3heqemzb X-HE-Tag: 1629751359-472353 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000031, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make schedule_on_each_cpu as EXPORT_SYMBOL_GPL Signed-off-by: Ritesh Harjani --- [v1 -> v2]: Use EXPORT_SYMBOL_GPL instead of EXPORT_SYMBOL kernel/workqueue.c | 1 + 1 file changed, 1 insertion(+) -- 2.31.1 diff --git a/kernel/workqueue.c b/kernel/workqueue.c index f148eacda55a..993f8983186d 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -3309,6 +3309,7 @@ int schedule_on_each_cpu(work_func_t func) free_percpu(works); return 0; } +EXPORT_SYMBOL_GPL(schedule_on_each_cpu); /** * execute_in_process_context - reliably execute the routine with user context From patchwork Mon Aug 23 20:42:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ritesh Harjani X-Patchwork-Id: 12453657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67F62C4320A for ; Mon, 23 Aug 2021 20:42:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DD14D61357 for ; Mon, 23 Aug 2021 20:42:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org DD14D61357 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 878466B006C; Mon, 23 Aug 2021 16:42:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8018F900002; Mon, 23 Aug 2021 16:42:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 654876B0072; Mon, 23 Aug 2021 16:42:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0121.hostedemail.com [216.40.44.121]) by kanga.kvack.org (Postfix) with ESMTP id 4BDC86B006C for ; Mon, 23 Aug 2021 16:42:44 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CF0D5252B9 for ; Mon, 23 Aug 2021 20:42:43 +0000 (UTC) X-FDA: 78507519006.09.E460FEA Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf22.hostedemail.com (Postfix) with ESMTP id 48DB01909 for ; Mon, 23 Aug 2021 20:42:43 +0000 (UTC) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17NKWjeH024204; Mon, 23 Aug 2021 16:42:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=s9kvZZ7tv1nuCof3fQNvFxMRX7gaaweRp5I/H3IIBSI=; b=qgzwtkA9BVHu76lMz6EHtmEHqAIO1VNTFQ4sMUDEjsyvtjzLkF5jCT8mcpz4cnhfnF4s uJJbTvj+fcrGyW7PDIjT/sze5lHOl884UhTrFa9hfm8qe9/Zuj/Xma8xe4PomAF7uFg+ YI+zLYU2/ga7z/YnGNk5hJ1ueUPA7bNifl3QQrh3ENx25F6uwjJ+Pm988kZMz+otqM6O lVg6xsJN7juB3jizFb+UK3f9CIphc44G0oQVu1/algQWMZAiSNuybJFd4VOzke1hvN1K 6gT+lusdB/IEuqLon//+iFeWdBHXQajqRoi1Z9TXL74BglzFtoVKUMsflI9Kxx3ELOlO Og== Received: from ppma06ams.nl.ibm.com (66.31.33a9.ip4.static.sl-reverse.com [169.51.49.102]) by mx0a-001b2d01.pphosted.com with ESMTP id 3ameuypww8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 16:42:40 -0400 Received: from pps.filterd (ppma06ams.nl.ibm.com [127.0.0.1]) by ppma06ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17NKXMfE009437; Mon, 23 Aug 2021 20:42:38 GMT Received: from b06cxnps3075.portsmouth.uk.ibm.com (d06relay10.portsmouth.uk.ibm.com [9.149.109.195]) by ppma06ams.nl.ibm.com with ESMTP id 3ajrrhbux0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 20:42:38 +0000 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17NKgYUh23265746 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Aug 2021 20:42:34 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AA3034C050; Mon, 23 Aug 2021 20:42:34 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3D2974C04A; Mon, 23 Aug 2021 20:42:34 +0000 (GMT) Received: from localhost (unknown [9.102.2.197]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 23 Aug 2021 20:42:34 +0000 (GMT) From: Ritesh Harjani To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Dennis Zhou , Tejun Heo , Christoph Lameter , Ritesh Harjani , "Aneesh Kumar K . V" , Vaibhav Jain , kernel test robot Subject: [PATCHv2 2/2] lib/percpu_test: Add extra tests in percpu_test Date: Tue, 24 Aug 2021 02:12:30 +0530 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: <5afc2a0c4da65e71ccf24fe65396710d34fc662e.1629751104.git.riteshh@linux.ibm.com> References: <5afc2a0c4da65e71ccf24fe65396710d34fc662e.1629751104.git.riteshh@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: viGg41THcIx1L8Pgp-1rBB4DXGsv4o6L X-Proofpoint-ORIG-GUID: viGg41THcIx1L8Pgp-1rBB4DXGsv4o6L X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790 definitions=2021-08-23_04:2021-08-23,2021-08-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 clxscore=1015 impostorscore=0 lowpriorityscore=0 phishscore=0 adultscore=0 bulkscore=0 spamscore=0 suspectscore=0 priorityscore=1501 mlxlogscore=999 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108230140 Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=qgzwtkA9; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf22.hostedemail.com: domain of riteshh@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=riteshh@linux.ibm.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 48DB01909 X-Stat-Signature: dnp6tjrqwkn8pdibdur8sr4izu1s7hju X-HE-Tag: 1629751363-519605 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: While debugging a issue, we needed to stress test the percpu alloc/free path. Hence added some tests in lib/percpu_test to stress test percpu subsystem for allocation with different sizes. This patch keeps the default behavior of insmod module same for default test. But when given insmod with different option, it can run a percpu_stressd daemon (percpu_test_num=2) which does a stress test evey 10secs unless the module is unloaded. We found this to be helpful in our testing, since with this we could easily excercise percpu allo/free path. Hence cleaned this up for inclusion in percpu_test module. Logs ====== qemu-> sudo insmod /mnt/percpu_test.ko percpu_test_num=0 [ 334.362973] percpu_test: INIT, interval: 1000, max_shift: 13, run_tests: percpu_verify [ 334.364946] TEST Starts: percpu_verify [ 334.365601] TEST Completed: percpu_verify insmod: ERROR: could not insert module /mnt/percpu_test.ko: Resource temporarily unavailable qemu-> sudo insmod /mnt/percpu_test.ko percpu_test_num=1 [ 336.556464] percpu_test: INIT, interval: 1000, max_shift: 13, run_tests: percpu_stress [ 336.558388] TEST Starts: percpu_stress [ 336.560611] TEST Completed: percpu_stress insmod: ERROR: could not insert module /mnt/percpu_test.ko: Resource temporarily unavailable qemu-> sudo insmod /mnt/percpu_test.ko percpu_test_num=2 [ 339.164406] percpu_test: INIT, interval: 1000, max_shift: 13, run_tests: percpu_stressd [ 339.165935] TEST Starts: percpu_stressd [ 339.167033] TEST Completed: percpu_stressd [ 339.167082] DAEMON: starts percpu_stressd [ 339.168498] TEST Starts: percpu_stressd: iter (1) [ 339.182530] TEST Completed: percpu_stressd: iter (1) [ 349.341109] TEST Starts: percpu_stressd: iter (2) [ 349.344447] TEST Completed: percpu_stressd: iter (2) [ 359.580829] TEST Starts: percpu_stressd: iter (3) [ 359.584315] TEST Completed: percpu_stressd: iter (3) [ 369.820471] TEST Starts: percpu_stressd: iter (4) [ 369.844402] TEST Completed: percpu_stressd: iter (4) qemu-> sudo rmmod percpu_test [ 375.001098] percpu_test: EXIT [qemu][~] Cc: Aneesh Kumar K.V Cc: Vaibhav Jain Reported-by: kernel test robot Signed-off-by: Ritesh Harjani --- [v1 -> v2]: Fix warnings from kernel test robot lib/percpu_test.c | 240 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 191 insertions(+), 49 deletions(-) -- 2.31.1 diff --git a/lib/percpu_test.c b/lib/percpu_test.c index 4a3d70bbc1a0..68c57c288dc6 100644 --- a/lib/percpu_test.c +++ b/lib/percpu_test.c @@ -1,4 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include #include /* validate @native and @pcp counter values match @expected */ @@ -14,10 +17,25 @@ (long long)(expected), (long long)(expected)); \ } while (0) -static DEFINE_PER_CPU(long, long_counter); -static DEFINE_PER_CPU(unsigned long, ulong_counter); +/* upto max alloc size tests for percpu var */ +static char __percpu *counters[1 << PAGE_SHIFT]; +static struct task_struct *percpu_stressd_thread; -static int __init percpu_test_init(void) +/* let's not trigger OOM */ +int percpu_alloc_max_size_shift = PAGE_SHIFT - 3; +module_param(percpu_alloc_max_size_shift, int, 0644); +MODULE_PARM_DESC(percpu_alloc_max_size_shift, "max size of allocation in stress test will be upto 1 << percpu_alloc_max_size_shift"); + +static long percpu_stressd_interval = 1 * 10 * HZ; +module_param(percpu_stressd_interval, long, 0644); +MODULE_PARM_DESC(percpu_stressd_interval, "percpu_stressd internal"); + +/* keep the default test same */ +static int percpu_test_num; +module_param(percpu_test_num, int, 0644); +MODULE_PARM_DESC(percpu_test_num, "Test number percpu_test_num"); + +static int percpu_test_verify(void) { /* * volatile prevents compiler from optimizing it uses, otherwise the @@ -26,109 +44,233 @@ static int __init percpu_test_init(void) volatile unsigned int ui_one = 1; long l = 0; unsigned long ul = 0; + long __percpu *long_counter = alloc_percpu(long); + unsigned long __percpu *ulong_counter = alloc_percpu(unsigned long); - pr_info("percpu test start\n"); + if (!long_counter || !ulong_counter) + goto out; + + pr_debug("percpu_test: %s start cpu: %d\n", __func__, smp_processor_id()); preempt_disable(); l += -1; - __this_cpu_add(long_counter, -1); - CHECK(l, long_counter, -1); + __this_cpu_add(*long_counter, -1); + CHECK(l, *long_counter, -1); l += 1; - __this_cpu_add(long_counter, 1); - CHECK(l, long_counter, 0); + __this_cpu_add(*long_counter, 1); + CHECK(l, *long_counter, 0); ul = 0; - __this_cpu_write(ulong_counter, 0); + __this_cpu_write(*ulong_counter, 0); ul += 1UL; - __this_cpu_add(ulong_counter, 1UL); - CHECK(ul, ulong_counter, 1); + __this_cpu_add(*ulong_counter, 1UL); + CHECK(ul, *ulong_counter, 1); ul += -1UL; - __this_cpu_add(ulong_counter, -1UL); - CHECK(ul, ulong_counter, 0); + __this_cpu_add(*ulong_counter, -1UL); + CHECK(ul, *ulong_counter, 0); ul += -(unsigned long)1; - __this_cpu_add(ulong_counter, -(unsigned long)1); - CHECK(ul, ulong_counter, -1); + __this_cpu_add(*ulong_counter, -(unsigned long)1); + CHECK(ul, *ulong_counter, -1); ul = 0; - __this_cpu_write(ulong_counter, 0); + __this_cpu_write(*ulong_counter, 0); ul -= 1; - __this_cpu_dec(ulong_counter); - CHECK(ul, ulong_counter, -1); - CHECK(ul, ulong_counter, ULONG_MAX); + __this_cpu_dec(*ulong_counter); + CHECK(ul, *ulong_counter, -1); + CHECK(ul, *ulong_counter, ULONG_MAX); l += -ui_one; - __this_cpu_add(long_counter, -ui_one); - CHECK(l, long_counter, 0xffffffff); + __this_cpu_add(*long_counter, -ui_one); + CHECK(l, *long_counter, 0xffffffff); l += ui_one; - __this_cpu_add(long_counter, ui_one); - CHECK(l, long_counter, (long)0x100000000LL); + __this_cpu_add(*long_counter, ui_one); + CHECK(l, *long_counter, (long)0x100000000LL); l = 0; - __this_cpu_write(long_counter, 0); + __this_cpu_write(*long_counter, 0); l -= ui_one; - __this_cpu_sub(long_counter, ui_one); - CHECK(l, long_counter, -1); + __this_cpu_sub(*long_counter, ui_one); + CHECK(l, *long_counter, -1); l = 0; - __this_cpu_write(long_counter, 0); + __this_cpu_write(*long_counter, 0); l += ui_one; - __this_cpu_add(long_counter, ui_one); - CHECK(l, long_counter, 1); + __this_cpu_add(*long_counter, ui_one); + CHECK(l, *long_counter, 1); l += -ui_one; - __this_cpu_add(long_counter, -ui_one); - CHECK(l, long_counter, (long)0x100000000LL); + __this_cpu_add(*long_counter, -ui_one); + CHECK(l, *long_counter, (long)0x100000000LL); l = 0; - __this_cpu_write(long_counter, 0); + __this_cpu_write(*long_counter, 0); l -= ui_one; - this_cpu_sub(long_counter, ui_one); - CHECK(l, long_counter, -1); - CHECK(l, long_counter, ULONG_MAX); + this_cpu_sub(*long_counter, ui_one); + CHECK(l, *long_counter, -1); + CHECK(l, *long_counter, ULONG_MAX); ul = 0; - __this_cpu_write(ulong_counter, 0); + __this_cpu_write(*ulong_counter, 0); ul += ui_one; - __this_cpu_add(ulong_counter, ui_one); - CHECK(ul, ulong_counter, 1); + __this_cpu_add(*ulong_counter, ui_one); + CHECK(ul, *ulong_counter, 1); ul = 0; - __this_cpu_write(ulong_counter, 0); + __this_cpu_write(*ulong_counter, 0); ul -= ui_one; - __this_cpu_sub(ulong_counter, ui_one); - CHECK(ul, ulong_counter, -1); - CHECK(ul, ulong_counter, ULONG_MAX); + __this_cpu_sub(*ulong_counter, ui_one); + CHECK(ul, *ulong_counter, -1); + CHECK(ul, *ulong_counter, ULONG_MAX); ul = 3; - __this_cpu_write(ulong_counter, 3); + __this_cpu_write(*ulong_counter, 3); - ul = this_cpu_sub_return(ulong_counter, ui_one); - CHECK(ul, ulong_counter, 2); + ul = this_cpu_sub_return(*ulong_counter, ui_one); + CHECK(ul, *ulong_counter, 2); - ul = __this_cpu_sub_return(ulong_counter, ui_one); - CHECK(ul, ulong_counter, 1); + ul = __this_cpu_sub_return(*ulong_counter, ui_one); + CHECK(ul, *ulong_counter, 1); preempt_enable(); - pr_info("percpu test done\n"); - return -EAGAIN; /* Fail will directly unload the module */ +out: + free_percpu(long_counter); + free_percpu(ulong_counter); + pr_debug("percpu_test: %s done cpu: %d\n", __func__, smp_processor_id()); + + /* + * Keep the default functionality same. + * Fail will directly unload this module. + */ + return -EAGAIN; +} + +static void percpu_test_verify_work(struct work_struct *work) +{ + percpu_test_verify(); +} + +static int percpu_test_stress(void) +{ + int i; + + for (i = 1; i < (1 << percpu_alloc_max_size_shift); i++) { + size_t size = i; + + if (size > PCPU_MIN_ALLOC_SIZE) + break; + counters[i] = (char __percpu *)__alloc_percpu(size, __alignof__(char)); + if (!counters[i]) + break; + cond_resched(); + } + + schedule_on_each_cpu(percpu_test_verify_work); + + for (i = 0; i < (1 << percpu_alloc_max_size_shift); i++) { + free_percpu(counters[i]); + cond_resched(); + } + return -EAGAIN; +} + +static int percpu_stressd(void *v) +{ + int iter = 0; + + pr_info("DAEMON: starts %s\n", __func__); + do { + if (kthread_should_stop()) + break; + iter++; + pr_info("TEST Starts: %s: iter (%d)\n", __func__, iter); + percpu_test_stress(); + pr_info("TEST Completed: %s: iter (%d)\n", __func__, iter); + + set_current_state(TASK_INTERRUPTIBLE); + schedule_timeout(percpu_stressd_interval); + } while (1); + + return 0; +} + +static int percpu_test_stressd(void) +{ + percpu_stressd_thread = kthread_run(percpu_stressd, NULL, "percpu_stressd"); + if (IS_ERR(percpu_stressd_thread)) + percpu_stressd_thread = NULL; + return 0; +} + +enum test_type { + PERCPU_VERIFY, + PERCPU_STRESS, + PERCPU_STRESSD, + NR_TESTS, +}; + +const char *test_names[NR_TESTS] = { + [PERCPU_VERIFY] = "percpu_verify", + [PERCPU_STRESS] = "percpu_stress", + [PERCPU_STRESSD] = "percpu_stressd", +}; + +static int __init percpu_test_init(void) +{ + int i, ret = 0; + typedef int (*percpu_tests)(void); + const percpu_tests test_funcs[NR_TESTS] = { + [PERCPU_VERIFY] = percpu_test_verify, + [PERCPU_STRESS] = percpu_test_stress, + [PERCPU_STRESSD] = percpu_test_stressd, + }; + + /* sanity checks */ + if (percpu_alloc_max_size_shift > PAGE_SHIFT) + percpu_alloc_max_size_shift = PAGE_SHIFT; + if (percpu_test_num > NR_TESTS) + percpu_test_num = NR_TESTS; + + pr_info("percpu_test: INIT, interval: %ld, max_shift: %d, run_tests: %s\n", + percpu_stressd_interval, percpu_alloc_max_size_shift, + percpu_test_num == NR_TESTS ? "run all tests" : + test_names[percpu_test_num]); + + /* run a given test */ + if (percpu_test_num < NR_TESTS) { + pr_info("TEST Starts: %s\n", test_names[percpu_test_num]); + ret = test_funcs[percpu_test_num](); + pr_info("TEST Completed: %s\n", test_names[percpu_test_num]); + goto out; + } + + for (i = 0; i < NR_TESTS; i++) { + pr_info("TEST Starts: %s\n", test_names[i]); + test_funcs[i](); + pr_info("TEST Completed: %s\n", test_names[i]); + } +out: + return ret; } static void __exit percpu_test_exit(void) { + if (percpu_stressd_thread) + kthread_stop(percpu_stressd_thread); + pr_info("percpu_test: EXIT\n"); } module_init(percpu_test_init)