From patchwork Wed Feb 14 08:54:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13556095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9D6BC48BEC for ; Wed, 14 Feb 2024 08:54:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B5C168D0014; Wed, 14 Feb 2024 03:54:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ABDC48D0001; Wed, 14 Feb 2024 03:54:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95DD08D0014; Wed, 14 Feb 2024 03:54:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 81FB48D0001 for ; Wed, 14 Feb 2024 03:54:53 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5454FA2202 for ; Wed, 14 Feb 2024 08:54:53 +0000 (UTC) X-FDA: 81789799266.11.E68A2B7 Received: from out-188.mta1.migadu.com (out-188.mta1.migadu.com [95.215.58.188]) by imf14.hostedemail.com (Postfix) with ESMTP id 8AB2C100016 for ; Wed, 14 Feb 2024 08:54:51 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf14.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.188 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707900891; a=rsa-sha256; cv=none; b=kcKU+bTAQOCkQ5ocYg1Rqn4gAycclZ31dDp10k1pbWk5UvfaRoBIcewhAouq6X+PcEGcow 48w9jCs/xL+Q1WTeAMqlkD6/0uZw681nykvZFDiBWibuV5DU8FlZVUIeLHpGo5idA4d9B0 uLWpl8+Vi+lJCyczHzDRFPJLs9TrKdI= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf14.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.188 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707900891; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1p4VLiafWD3G/v8RUY99fbKtMhISGd+RELaghb++nEw=; b=qnl/FxrG4Sduo3tfdVxx85v5zR0KFEsPPlmyDXgzFGrzpTtDeyu/suMQtuozeLWGgBVTzp b96scLJ6L7gCP0XMFcyGYnVEFY0FTGdzoGlf+wLr2MPxMdMTKeWijt3az7NLY1BPuS7lAr eQKVYFSmPvbd+4kEgv1MhuDxm5rEgew= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Wed, 14 Feb 2024 08:54:38 +0000 Subject: [PATCH v2 2/2] mm/zswap: change zswap_pool kref to percpu_ref MIME-Version: 1.0 Message-Id: <20240210-zswap-global-lru-v2-2-fbee3b11a62e@bytedance.com> References: <20240210-zswap-global-lru-v2-0-fbee3b11a62e@bytedance.com> In-Reply-To: <20240210-zswap-global-lru-v2-0-fbee3b11a62e@bytedance.com> To: Yosry Ahmed , Johannes Weiner , Andrew Morton , Nhat Pham Cc: linux-mm@kvack.org, Chengming Zhou , linux-kernel@vger.kernel.org X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8AB2C100016 X-Stat-Signature: 6wjrdd6wgi18kzmzct6xfd5gpq1rw83q X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspam: Yes X-HE-Tag: 1707900891-691345 X-HE-Meta: U2FsdGVkX1+amttiaceESOMffjyzkWRD534GSBFauvJc0HyL60T6MmVZW5RAHce9c2agweXq3EZ4ed/HqvTjgWg9ct4JfX27HeZr8AlueCJxrMV08bInnYdcMwOmVyPJ7BT7nqLe8LHOUeEnbvN4U6hL9aO0b9Dr10w0F+Xy/eE/ZUjBsM1XUnOa7jaqgR1WY8yCilIZ/CmbFETkXhKureQxNg65Gc8T4scRtvo6lVTdM8vguipeiwUz6YVUHhfHydaPTxy9fnZvkTVI07R5Ad6MHWakz7SX0nVtHAAN3j25DbldSEhWc0LF7dBovIxDJC8IfbfmKyVD1t7pv0otpnWgMGwNwLmuKAYfYLdJaQDZHX9HOHUjPPQgD/QxMQdErRYzLIIJkFsXTnLuNApKvxHB46rBYdWXXF4+8jZ7JsWZGeEd8/vaHtx1elKyFzLKIryCZukELBh0ORtKS2g927qcNPlXyqOBXYGcs4MZsEhMElqN6BGeZB0481G7XMLb6+A0vkJ48GI+xE0q8TwBRK55vLERwp0rYOh0LuQfc5Bx1FRRtunuyv2VeIDEFH3lpHkVurN7+pgNFFtsKeyjp0O7KZ3wpUHUptq2FerhOYJPBEgkWItR87m/9AfLyVkiaYf/F/vWDOH3q4kcz/N27jYXo6ykU7keCNNYhzefnxgESKMPruX48aFk0IGKZeaRkywDA1pvrdLNqrP1VtDN0Ptx7hy51aIVTRdCJsZZd7MMyfrahJGLL26ji4gcSG64RXAn4JuQyUmSE4nOIQuZCsQ8A/BJBwf3OsUrPwPPWZ1nyL8sp3PlWFVJBrwrAq/FepdqMcE9qYKYrDaUFd0Wgz8aqXLg+kXt8elCxG6xkeLzGUVXzgn1ch31T/o4tiCUZGjMqBvN3ViumyAuxuo0gmmCeJ/c+62ZHg9jXqtn50I745Ow0cJH52iUvSTFmtMvECWDqXc44YEIKSFQflT xVjSFEhP LxtbzHCq3axG+v6lPeIV608MsASIhikBneIvxZ+i2pI/AC/KBkDuEO2eS0BWBZETPB+p46sMgQrzuoFEUFKRnTZZUl/ZmLlGcLtQ5B2j+TvAdn8ThsIYk1Win1B6LhU1MsTHI/kmqy6XpUw9J/cWrnmZYwNBooeIpUSvehNqOgrLXQx5HwQdFCym6G93GUK1HTGE3gSQ8GxhmU5XyCSH8boYP2v2ftrmy8L3ecSUF9ZlrzlrZQWwA6zRBhbWAI6PodGCeSveT71SXqetBZJuDYMqEkq0OoS1FbTy9CfO2NDl5+eeMchMNRFAo12kWQm7WfRlVWEKquBAXhv5hzzNnY+k8IgF6j1nX8aeltUyQAutAbz7nQa/ApL4Pxg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All zswap entries will take a reference of zswap_pool when zswap_store(), and drop it when free. Change it to use the percpu_ref is better for scalability performance. Although percpu_ref use a bit more memory which should be ok for our use case, since we almost have only one zswap_pool to be using. The performance gain is for zswap_store/load hotpath. Testing kernel build (32 threads) in tmpfs with memory.max=2GB. (zswap shrinker and writeback enabled with one 50GB swapfile, on a 128 CPUs x86-64 machine, below is the average of 5 runs) mm-unstable zswap-global-lru real 63.20 63.12 user 1061.75 1062.95 sys 268.74 264.44 Signed-off-by: Chengming Zhou --- mm/zswap.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index dbff67d7e1c7..f6470d30d337 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -173,7 +173,7 @@ struct crypto_acomp_ctx { struct zswap_pool { struct zpool *zpools[ZSWAP_NR_ZPOOLS]; struct crypto_acomp_ctx __percpu *acomp_ctx; - struct kref kref; + struct percpu_ref ref; struct list_head list; struct work_struct release_work; struct hlist_node node; @@ -304,6 +304,7 @@ static void zswap_update_total_size(void) /********************************* * pool functions **********************************/ +static void __zswap_pool_empty(struct percpu_ref *ref); static struct zswap_pool *zswap_pool_create(char *type, char *compressor) { @@ -357,13 +358,18 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor) /* being the current pool takes 1 ref; this func expects the * caller to always add the new pool as the current pool */ - kref_init(&pool->kref); + ret = percpu_ref_init(&pool->ref, __zswap_pool_empty, + PERCPU_REF_ALLOW_REINIT, GFP_KERNEL); + if (ret) + goto ref_fail; INIT_LIST_HEAD(&pool->list); zswap_pool_debug("created", pool); return pool; +ref_fail: + cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); error: if (pool->acomp_ctx) free_percpu(pool->acomp_ctx); @@ -436,8 +442,9 @@ static void __zswap_pool_release(struct work_struct *work) synchronize_rcu(); - /* nobody should have been able to get a kref... */ - WARN_ON(kref_get_unless_zero(&pool->kref)); + /* nobody should have been able to get a ref... */ + WARN_ON(percpu_ref_tryget(&pool->ref)); + percpu_ref_exit(&pool->ref); /* pool is now off zswap_pools list and has no references. */ zswap_pool_destroy(pool); @@ -445,11 +452,11 @@ static void __zswap_pool_release(struct work_struct *work) static struct zswap_pool *zswap_pool_current(void); -static void __zswap_pool_empty(struct kref *kref) +static void __zswap_pool_empty(struct percpu_ref *ref) { struct zswap_pool *pool; - pool = container_of(kref, typeof(*pool), kref); + pool = container_of(ref, typeof(*pool), ref); spin_lock(&zswap_pools_lock); @@ -468,12 +475,12 @@ static int __must_check zswap_pool_get(struct zswap_pool *pool) if (!pool) return 0; - return kref_get_unless_zero(&pool->kref); + return percpu_ref_tryget(&pool->ref); } static void zswap_pool_put(struct zswap_pool *pool) { - kref_put(&pool->kref, __zswap_pool_empty); + percpu_ref_put(&pool->ref); } static struct zswap_pool *__zswap_pool_current(void) @@ -603,6 +610,12 @@ static int __zswap_param_set(const char *val, const struct kernel_param *kp, if (!pool) pool = zswap_pool_create(type, compressor); + else { + /* Resurrect percpu_ref to percpu mode. */ + percpu_ref_resurrect(&pool->ref); + /* Drop the ref from zswap_pool_find_get(). */ + zswap_pool_put(pool); + } if (pool) ret = param_set_charp(s, kp); @@ -641,7 +654,7 @@ static int __zswap_param_set(const char *val, const struct kernel_param *kp, * or the new pool we failed to add */ if (put_pool) - zswap_pool_put(put_pool); + percpu_ref_kill(&put_pool->ref); return ret; }