From patchwork Thu Apr 18 14:20:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng Zhang X-Patchwork-Id: 13634848 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4618C04FF8 for ; Thu, 18 Apr 2024 14:20:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8243C6B0089; Thu, 18 Apr 2024 10:20:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 733FF6B0096; Thu, 18 Apr 2024 10:20:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F10E6B0092; Thu, 18 Apr 2024 10:20:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id F267C6B008C for ; Thu, 18 Apr 2024 10:20:25 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 76A1D160993 for ; Thu, 18 Apr 2024 14:20:25 +0000 (UTC) X-FDA: 82022862810.24.F41C683 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by imf24.hostedemail.com (Postfix) with ESMTP id D328718001D for ; Thu, 18 Apr 2024 14:20:22 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf24.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713450023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aBWkZ7C1HbyeYuZQ3wV78ieLpFmpJaU4kWj+BPRe7Ns=; b=DTMw3z2bSG+au9C1/BcoTQveEZu+FwFmpsVXReW2zo/I6Px9E9iizCvLtxsqbSGnGmJ1Cd lRD9jLWGQkktKNQh+ZfpTzNQFRAoYIP7VA87BKwqKz+heLgoxXJInhxuCDdLc5toy/WFi7 zMQRDMkp6m/qgfYSaTnPZX0Z//Iu2CU= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf24.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713450023; a=rsa-sha256; cv=none; b=KcStu2vywi3p6+crczjHLG+20+agz3iGgI0sd9+n2pTw+rGoh3j4B0npcrhnOOyXNaUVgr LDWncs+LG0mioSW/IER29ZbDQuvnuYfkL0iNUiE7kdSrZY84Nm5f/3GOaLiW88MljTs+O2 I5BoiCiu1IXsw4hXPrRTbDD7yhTIZvU= Received: from mail.maildlp.com (unknown [172.19.163.252]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4VL0G20QnKzXlRQ; Thu, 18 Apr 2024 22:16:58 +0800 (CST) Received: from kwepemm600020.china.huawei.com (unknown [7.193.23.147]) by mail.maildlp.com (Postfix) with ESMTPS id 98AD218006B; Thu, 18 Apr 2024 22:20:18 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by kwepemm600020.china.huawei.com (7.193.23.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Thu, 18 Apr 2024 22:20:17 +0800 From: Peng Zhang To: , CC: , , , , , , , , , , , , Subject: [RFC PATCH v2 1/2] percpu_counter: introduce atomic mode for percpu_counter Date: Thu, 18 Apr 2024 22:20:07 +0800 Message-ID: <20240418142008.2775308-2-zhangpeng362@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240418142008.2775308-1-zhangpeng362@huawei.com> References: <20240418142008.2775308-1-zhangpeng362@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To kwepemm600020.china.huawei.com (7.193.23.147) X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: D328718001D X-Stat-Signature: i9t495fupjrpk4qdtb1y9bzetg145zkj X-HE-Tag: 1713450022-117723 X-HE-Meta: U2FsdGVkX18UgQK4mnFXx6sCPeqfF+9Lzk6Mqxr88fINFs07WKw1vTDG4R/On3nkG6qHYlzawy1S//Rd4mlrE0VmI1yX3nStcEUggbhFWisb0ltFK+nGE4ocQY/xAp+ToLd6dvjH/kpOwIu/Zy0NoCmlooQH0AIa1sq/y0gn1UrbfRBOsdi4lPp3FLt2tglxZVkTHSPlGEb6kuyvoXbw+10RCmY2zib7nhHOKUUXctVhaIQBPaRzAIiPyWfxBjW/ynpcszlRhWTjW4de15cgE3NDwVe/X3YUMfMVvbiQsDacnI4AMv8nzttTC3bSpBPYF3PSlAn0H2F+9ZmfLzaOyz1KZNSLsYKgOTemz0EbY+TiB98xqPMhgI5D4I74fqnafW6dQhmG1HjvDtYsprQ4yWDeixedCCVPPGoFSHv+784xlfj5le2/4gZtiqiWwyIOiWPfw4Dn1mC50/zrGDjpLu/1r1j4XYqwzli4y+CM4kyS9fpa9Jjw1ZA6mKkzR5Pb13v7CixN2xbmqjp4FFSsDWGniOK1JACRsjusZpNfdekUL8pbkXHxvGJ/A3DBsWcS9y0oo/e/pEHHvmj7gAkG7JPFlgf5Zou8IvXGJWf95dC6hTsRmiZ39NQDRH9dmQnj2cZXjyk2rr5nFI0bR6xxQhBsao0XRDnwRT0fhCoIabLMPgyMF8Yoo7lapCx5AfhbtNU/PrNbBThSSE6h2ieG6t2iEnbcrHfaEfuSy8I3HboQ9/yUhR2W5P2aPPHUT33VrrC9qsjCo/k9s66k2YH5mTFl/8vMCiCBeAlCr8H5noikyh5LipUlzI8rJjjy7PQd8GS5QPQ/sz4z6OqKYVOnoOgk/HnpySC5Ovrj1cTaQR5nkuXhXLQIdW4XLowwrU2Zg5UI/g44Fy38Q1C0BtoWXISWFNYjbztZ/zt1OVWTOw5JLu+pYMNqeNIj2+GvLON3DR1P3OpvFgQ5+gsfvqi hCNS8a3x MCbu04yKZxBwD9tolNjGcM2UfPjN+t3eHfb9le1RP1eJrOfdTHwOUFm2ElNMsrf7/loRUv24i3mJSDhgQ5JLjJNCBTUL+RGAHc+GL1JOmAVRvpZC77ebAnEtMlE+vJOVAs+/5TN/RqOXqYCpXFgcis09pOpEFinTvEMlhrD6Oy+tSMJmd8nYL5QbKcEwvaVNwE9S9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: ZhangPeng Depending on whether counters is NULL, we can support two modes: atomic mode and perpcu mode. We implement both modes by grouping the s64 count and atomic64_t count_atomic in a union. At the same time, we create the interface for adding and reading in atomic mode and for switching atomic mode to percpu mode. Suggested-by: Jan Kara Signed-off-by: ZhangPeng Signed-off-by: Kefeng Wang --- include/linux/percpu_counter.h | 43 +++++++++++++++++++++++++++++++--- lib/percpu_counter.c | 31 ++++++++++++++++++++++-- 2 files changed, 69 insertions(+), 5 deletions(-) diff --git a/include/linux/percpu_counter.h b/include/linux/percpu_counter.h index 3a44dd1e33d2..160f9734c0bb 100644 --- a/include/linux/percpu_counter.h +++ b/include/linux/percpu_counter.h @@ -21,7 +21,13 @@ struct percpu_counter { raw_spinlock_t lock; - s64 count; + /* Depending on whether counters is NULL, we can support two modes, + * atomic mode using count_atomic and perpcu mode using count. + */ + union { + s64 count; + atomic64_t count_atomic; + }; #ifdef CONFIG_HOTPLUG_CPU struct list_head list; /* All percpu_counters are on a list */ #endif @@ -32,14 +38,14 @@ extern int percpu_counter_batch; int __percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, gfp_t gfp, u32 nr_counters, - struct lock_class_key *key); + struct lock_class_key *key, bool switch_mode); #define percpu_counter_init_many(fbc, value, gfp, nr_counters) \ ({ \ static struct lock_class_key __key; \ \ __percpu_counter_init_many(fbc, value, gfp, nr_counters,\ - &__key); \ + &__key, false); \ }) @@ -130,6 +136,20 @@ static inline bool percpu_counter_initialized(struct percpu_counter *fbc) return (fbc->counters != NULL); } +static inline s64 percpu_counter_atomic_read(struct percpu_counter *fbc) +{ + return atomic64_read(&fbc->count_atomic); +} + +static inline void percpu_counter_atomic_add(struct percpu_counter *fbc, + s64 amount) +{ + atomic64_add(amount, &fbc->count_atomic); +} + +int percpu_counter_switch_to_pcpu_many(struct percpu_counter *fbc, + u32 nr_counters); + #else /* !CONFIG_SMP */ struct percpu_counter { @@ -260,6 +280,23 @@ static inline bool percpu_counter_initialized(struct percpu_counter *fbc) static inline void percpu_counter_sync(struct percpu_counter *fbc) { } + +static inline s64 percpu_counter_atomic_read(struct percpu_counter *fbc) +{ + return fbc->count; +} + +static inline void percpu_counter_atomic_add(struct percpu_counter *fbc, + s64 amount) +{ + percpu_counter_add(fbc, amount); +} + +static inline int percpu_counter_switch_to_pcpu_many(struct percpu_counter *fbc, + u32 nr_counters) +{ + return 0; +} #endif /* CONFIG_SMP */ static inline void percpu_counter_inc(struct percpu_counter *fbc) diff --git a/lib/percpu_counter.c b/lib/percpu_counter.c index 44dd133594d4..95c4e038051a 100644 --- a/lib/percpu_counter.c +++ b/lib/percpu_counter.c @@ -153,7 +153,7 @@ EXPORT_SYMBOL(__percpu_counter_sum); int __percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, gfp_t gfp, u32 nr_counters, - struct lock_class_key *key) + struct lock_class_key *key, bool switch_mode) { unsigned long flags __maybe_unused; size_t counter_size; @@ -174,7 +174,8 @@ int __percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, #ifdef CONFIG_HOTPLUG_CPU INIT_LIST_HEAD(&fbc[i].list); #endif - fbc[i].count = amount; + if (likely(!switch_mode)) + fbc[i].count = amount; fbc[i].counters = (void *)counters + (i * counter_size); debug_percpu_counter_activate(&fbc[i]); @@ -357,6 +358,32 @@ bool __percpu_counter_limited_add(struct percpu_counter *fbc, return good; } +/* + * percpu_counter_switch_to_pcpu_many: Converts struct percpu_counters from + * atomic mode to percpu mode. + */ +int percpu_counter_switch_to_pcpu_many(struct percpu_counter *fbc, + u32 nr_counters) +{ + static struct lock_class_key __key; + unsigned long flags; + bool ret = 0; + + if (percpu_counter_initialized(fbc)) + return 0; + + preempt_disable(); + local_irq_save(flags); + if (likely(!percpu_counter_initialized(fbc))) + ret = __percpu_counter_init_many(fbc, 0, + GFP_ATOMIC|__GFP_NOWARN|__GFP_ZERO, + nr_counters, &__key, true); + local_irq_restore(flags); + preempt_enable(); + + return ret; +} + static int __init percpu_counter_startup(void) { int ret;