From patchwork Fri Apr 12 09:24:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng Zhang X-Patchwork-Id: 13627433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4BE5C4345F for ; Fri, 12 Apr 2024 09:25:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D575F6B0083; Fri, 12 Apr 2024 05:24:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D07CE6B0087; Fri, 12 Apr 2024 05:24:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE1AF6B0089; Fri, 12 Apr 2024 05:24:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9F6F36B0083 for ; Fri, 12 Apr 2024 05:24:57 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6F6331C1182 for ; Fri, 12 Apr 2024 09:24:57 +0000 (UTC) X-FDA: 82000345434.22.CA65537 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf09.hostedemail.com (Postfix) with ESMTP id 01B2A140018 for ; Fri, 12 Apr 2024 09:24:54 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712913895; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Mi93krnxZRxKjJUzpinsMipngaeZpGOt6BBciD7NOoQ=; b=PsnSPgMKhdC8gtVi7LP+BzQYcQ2yj3IIIXFP6hfnfcPYJBfBiSvO5DChUf6/Tb+dOSYEU7 NK6NO+4gVXfXiLbEPhqgGwk06fm89ZWtAN1/3VLHsTpid1nnAXYm06N1Q95dn9E7h/2yQL w1HJaFY4YMQIN23XxhTIygTK6j7XYPk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712913895; a=rsa-sha256; cv=none; b=aWX1H6SVP+PUjqGi0WsbMz2BglWS1DZ8KqifjnLxvm64Divg5J9xaMX8ID6uD6h5Om0TtD hZ84xI0HCiQ/NH2BDCUbEKzn0USNmZ0XBY6fviAPu0ux8bUF91gJIZ97+Zj0kq98NQIz/x y0JIC58edlCKR0jDTTJDbfE5NonGlJo= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of zhangpeng362@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=zhangpeng362@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com Received: from mail.maildlp.com (unknown [172.19.162.254]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4VGB0L1sf0zwS0n; Fri, 12 Apr 2024 17:21:54 +0800 (CST) Received: from kwepemm600020.china.huawei.com (unknown [7.193.23.147]) by mail.maildlp.com (Postfix) with ESMTPS id CF99218007B; Fri, 12 Apr 2024 17:24:51 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by kwepemm600020.china.huawei.com (7.193.23.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Fri, 12 Apr 2024 17:24:50 +0800 From: Peng Zhang To: , CC: , , , , , , , , , , , , Subject: [RFC PATCH 2/3] lazy_percpu_counter: include struct percpu_counter in struct lazy_percpu_counter Date: Fri, 12 Apr 2024 17:24:40 +0800 Message-ID: <20240412092441.3112481-3-zhangpeng362@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240412092441.3112481-1-zhangpeng362@huawei.com> References: <20240412092441.3112481-1-zhangpeng362@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To kwepemm600020.china.huawei.com (7.193.23.147) X-Stat-Signature: zfmb15ts3gt8ynyqh3mostjnpojffymj X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 01B2A140018 X-Rspam-User: X-HE-Tag: 1712913894-917525 X-HE-Meta: U2FsdGVkX199ncQx0NLezlZ43F5gx0NVJm7PyL9Z5qIb2GQWAk81F9JPrqooI0NA7a1KtwmSrn57/NCgo9hN50Zt865PdI2vQuwS6XZaKAeKCt535cowvA+wIlOeJ7k/3Rp299CYdL0ThOoy6dctpj++Q3xXQyZYOZT4nT0zqNsw9IFTt4p3CuzuL7rAXga9mK/z1VnMDPJciWKfYMP23A8gJER0zxiZhMH8BlW2ViLNOGeUuMIxXJEtZTNID6Gl+IbykRVPhZulIEgMaK5osCtGSVbx+yVw/9l16vGUoyyalYse5KCjAGor4uJx8kpzeMsBO+/P1oz/4ei0FmvmD3dwjWB7WCuTjV+AdtsKZa/Tva94JoumaBTDO76Syhpgnjb7wNBDClejlsZdn7/DVaXo+BxdOF9hIEwsFOrncccow7cFmdFCf63YM9xA6z9FSSJdr7/4U/uUt3usMPuhWzuk+MLerCLu2KolVjS9QV/+D12VVZIih75zD4T2gQVZychzWg519X0I+4H40ZS3nTZB50KATwX7Zm93RhQCdTMudi94ZvmUjZzq4b9v2mbzytO9iK9vn/qSwgDQDzz/IPNnB3Wtv2QWjZOidOPExczfitzTiU+UozieX6XS2BbF3aH2U7UQqx3Iy4IVVYezvAQFTeoDM2vvZdg0nD1Fjr+R6FPckpqlwimN1LT+518EUA5dwZst0YyT+zkmJWhGdGRY5rK1rceRGJiM1tvTihCe1dgZJgV79TzjVhXZb1RNJLMWq8G/kvXoyjRAE+fGv9HDacc/FsmhKaW4nvOIB0cFTJQKYR5SHBOWGEiuYHQ/jQsNdmZMaPa9Rf4kqpkAcxkeh9jTEXZhFkiVngX/tP/wnlkQFMY9J6c5CUdSto0rILmnKLa4pImjPigITgHsJQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: ZhangPeng Add the struct percpu_counter fbc to struct lazy_percpu_counter. Convert the u64 __percpu parameter of the lazy percpu counter function to the struct percpu_counter parameter to prepare for converting mm's rss stats into lazy_percpu_counter. Signed-off-by: ZhangPeng Signed-off-by: Kefeng Wang --- include/linux/lazy-percpu-counter.h | 16 ++++-- lib/lazy-percpu-counter.c | 83 +++++++++++++++++++++++------ 2 files changed, 77 insertions(+), 22 deletions(-) diff --git a/include/linux/lazy-percpu-counter.h b/include/linux/lazy-percpu-counter.h index 281b8dd88cb2..03ff24f0128d 100644 --- a/include/linux/lazy-percpu-counter.h +++ b/include/linux/lazy-percpu-counter.h @@ -20,15 +20,21 @@ #define _LINUX_LAZY_PERCPU_COUNTER_H #include +#include #include struct lazy_percpu_counter { atomic64_t v; unsigned long last_wrap; + struct percpu_counter fbc; }; -void lazy_percpu_counter_exit(struct lazy_percpu_counter *c); +void lazy_percpu_counter_destroy_many(struct lazy_percpu_counter *c, + u32 nr_counters); void lazy_percpu_counter_add_slowpath(struct lazy_percpu_counter *c, s64 i); +s64 lazy_percpu_counter_read_positive(struct lazy_percpu_counter *c); +s64 lazy_percpu_counter_sum(struct lazy_percpu_counter *c); +s64 lazy_percpu_counter_sum_positive(struct lazy_percpu_counter *c); /* * We use the high bits of the atomic counter for a secondary counter, which is @@ -48,13 +54,13 @@ void lazy_percpu_counter_add_slowpath(struct lazy_percpu_counter *c, s64 i); */ #define COUNTER_IS_PCPU_BIT 1 -static inline u64 __percpu *lazy_percpu_counter_is_pcpu(u64 v) +static inline struct percpu_counter *lazy_percpu_counter_is_pcpu(u64 v) { if (!(v & COUNTER_IS_PCPU_BIT)) return NULL; v ^= COUNTER_IS_PCPU_BIT; - return (u64 __percpu *)(unsigned long)v; + return (struct percpu_counter *)(unsigned long)v; } /** @@ -66,10 +72,10 @@ static inline u64 __percpu *lazy_percpu_counter_is_pcpu(u64 v) static inline void lazy_percpu_counter_add(struct lazy_percpu_counter *c, s64 i) { u64 v = atomic64_read(&c->v); - u64 __percpu *pcpu_v = lazy_percpu_counter_is_pcpu(v); + struct percpu_counter *pcpu_v = lazy_percpu_counter_is_pcpu(v); if (likely(pcpu_v)) - this_cpu_add(*pcpu_v, i); + percpu_counter_add(pcpu_v, i); else lazy_percpu_counter_add_slowpath(c, i); } diff --git a/lib/lazy-percpu-counter.c b/lib/lazy-percpu-counter.c index e1914207214d..c360903cc02a 100644 --- a/lib/lazy-percpu-counter.c +++ b/lib/lazy-percpu-counter.c @@ -15,45 +15,94 @@ static inline s64 lazy_percpu_counter_atomic_val(s64 v) static void lazy_percpu_counter_switch_to_pcpu(struct lazy_percpu_counter *c) { - u64 __percpu *pcpu_v = alloc_percpu_gfp(u64, GFP_ATOMIC|__GFP_NOWARN); u64 old, new, v; + unsigned long flags; + bool allocated = false; - if (!pcpu_v) - return; - + local_irq_save(flags); preempt_disable(); v = atomic64_read(&c->v); do { - if (lazy_percpu_counter_is_pcpu(v)) { - free_percpu(pcpu_v); - return; + if (lazy_percpu_counter_is_pcpu(v)) + break; + + if (!allocated) { + if (percpu_counter_init(&c->fbc, 0, GFP_ATOMIC|__GFP_NOWARN)) + break; + allocated = true; } old = v; - new = (unsigned long)pcpu_v | 1; + new = (unsigned long)&c->fbc | 1; - *this_cpu_ptr(pcpu_v) = lazy_percpu_counter_atomic_val(v); + percpu_counter_set(&c->fbc, lazy_percpu_counter_atomic_val(v)); } while ((v = atomic64_cmpxchg(&c->v, old, new)) != old); preempt_enable(); + local_irq_restore(flags); } /** - * lazy_percpu_counter_exit: Free resources associated with a - * lazy_percpu_counter + * lazy_percpu_counter_destroy_many: Free resources associated with + * lazy_percpu_counters * - * @c: counter to exit + * @c: counters to exit + * @nr_counters: number of counters */ -void lazy_percpu_counter_exit(struct lazy_percpu_counter *c) +void lazy_percpu_counter_destroy_many(struct lazy_percpu_counter *c, + u32 nr_counters) +{ + struct percpu_counter *pcpu_v; + u32 i; + + for (i = 0; i < nr_counters; i++) { + pcpu_v = lazy_percpu_counter_is_pcpu(atomic64_read(&c[i].v)); + if (pcpu_v) + percpu_counter_destroy(pcpu_v); + } +} +EXPORT_SYMBOL_GPL(lazy_percpu_counter_destroy_many); + +s64 lazy_percpu_counter_read_positive(struct lazy_percpu_counter *c) +{ + s64 v = atomic64_read(&c->v); + struct percpu_counter *pcpu_v = lazy_percpu_counter_is_pcpu(v); + + if (pcpu_v) + return percpu_counter_read_positive(pcpu_v); + + return lazy_percpu_counter_atomic_val(v); +} +EXPORT_SYMBOL_GPL(lazy_percpu_counter_read_positive); + +s64 lazy_percpu_counter_sum(struct lazy_percpu_counter *c) +{ + s64 v = atomic64_read(&c->v); + struct percpu_counter *pcpu_v = lazy_percpu_counter_is_pcpu(v); + + if (pcpu_v) + return percpu_counter_sum(pcpu_v); + + return lazy_percpu_counter_atomic_val(v); +} +EXPORT_SYMBOL_GPL(lazy_percpu_counter_sum); + +s64 lazy_percpu_counter_sum_positive(struct lazy_percpu_counter *c) { - free_percpu(lazy_percpu_counter_is_pcpu(atomic64_read(&c->v))); + s64 v = atomic64_read(&c->v); + struct percpu_counter *pcpu_v = lazy_percpu_counter_is_pcpu(v); + + if (pcpu_v) + return percpu_counter_sum_positive(pcpu_v); + + return lazy_percpu_counter_atomic_val(v); } -EXPORT_SYMBOL_GPL(lazy_percpu_counter_exit); +EXPORT_SYMBOL_GPL(lazy_percpu_counter_sum_positive); void lazy_percpu_counter_add_slowpath(struct lazy_percpu_counter *c, s64 i) { u64 atomic_i; u64 old, v = atomic64_read(&c->v); - u64 __percpu *pcpu_v; + struct percpu_counter *pcpu_v; atomic_i = i << COUNTER_IS_PCPU_BIT; atomic_i &= ~COUNTER_MOD_MASK; @@ -62,7 +111,7 @@ void lazy_percpu_counter_add_slowpath(struct lazy_percpu_counter *c, s64 i) do { pcpu_v = lazy_percpu_counter_is_pcpu(v); if (pcpu_v) { - this_cpu_add(*pcpu_v, i); + percpu_counter_add(pcpu_v, i); return; }