From patchwork Thu Jun 27 07:59:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13713909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A9D6C2BD09 for ; Thu, 27 Jun 2024 08:01:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9987E6B0092; Thu, 27 Jun 2024 04:01:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 947E56B0098; Thu, 27 Jun 2024 04:01:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 836886B0099; Thu, 27 Jun 2024 04:01:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 666FB6B0092 for ; Thu, 27 Jun 2024 04:01:20 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 025751A1BEF for ; Thu, 27 Jun 2024 08:01:19 +0000 (UTC) X-FDA: 82275923520.08.505B413 Received: from out-180.mta1.migadu.com (out-180.mta1.migadu.com [95.215.58.180]) by imf22.hostedemail.com (Postfix) with ESMTP id 06351C0013 for ; Thu, 27 Jun 2024 08:01:17 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=prMdhjOb; spf=pass (imf22.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.180 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719475262; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=Qk1uUCNTkSFH7biQuWiNk/WtpQApJrMKSj8t5jy2L9Y=; b=0SosLGoZYK8I6jnFyaZlwu2CUdKNnLddxnzAwQJGL5dj0hlNyey3moMiNqcNb/go3r0Xnf KjpW1OYbCZy2GWq3rAXFNFJzocAeBWhKjRFNCebb6D60ak/k1M57y19TML/5XcUz5gLTCI 5il76fXJO77zDtym2H6yfmxn/6dEHFg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719475262; a=rsa-sha256; cv=none; b=ke/YiKe06/spCn67sDtOeZbkFFn9HjXaQvFJp6ErxSEJIr/5IKq+gYeUJ5NH81TtqBSmGl NQIte5vqbOirv72MTD/zRrS1CBcU7IEKYmXLtPs0zWa3AjnXChNYiB4ERrba9aC0z0hCzd vyraU5nmdIQPd5kej1yppwgkp9BsaiY= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=prMdhjOb; spf=pass (imf22.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.180 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Envelope-To: minchan@kernel.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1719475275; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=Qk1uUCNTkSFH7biQuWiNk/WtpQApJrMKSj8t5jy2L9Y=; b=prMdhjObneMqmY2jva8GFCZsP3n12mpBu2Rxgv91T1w+M9vi0E9OM0dp2fDXaN3ihBydAt HefswZUzOVRiAfFFc9aLvsjRB9Cg4GWmChSV3X3LRpMPIlFMNCnTMRAHqWbdjA/vhEOPLj pu7TzfAWCnxSNICAuXajabNjUHkZG+w= X-Envelope-To: senozhatsky@chromium.org X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: chengming.zhou@linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou To: minchan@kernel.org, senozhatsky@chromium.org, akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev Subject: [PATCH 1/2] mm/zsmalloc: fix class per-fullness zspage counts Date: Thu, 27 Jun 2024 15:59:58 +0800 Message-Id: <20240627075959.611783-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 06351C0013 X-Stat-Signature: dp7ebs6abp5p4hqyfzp8sxdmn65stho6 X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1719475277-965280 X-HE-Meta: U2FsdGVkX19OoIHP8PrQ8azDJzLaOGfw3x+7/7e7NxcO8Gc+rAmNpsbfIMWruSrS3Zs159ut89vTN15m/YEiSNKrG/IPpXHt9cGbW/nthoC+eRB2C1oWtHRYdCEBk310ueYLl3BAoRdMIaNf2061vAtrrsCiDnS9vsko5ipgJktH5AeiQ6EqjxJ0ttEY9ZhQ0UHKqnOPKGgwFzuTMp8BAHuOliUIdmI98LqnVEymYAZkuyxkorsPk4v5hUboLuWGOy06e+UQoR6AJPAkim1U8OomW6jn7YhG0Ppq6X/Vwluj3Mpa0HSZSl+PQ2pigmkjwbtA2cn+60f7Es44KXn5JjQY4JIGMLZKGm4yX0zSf32Y29qWbF/wQqjwPa4y3iyHk6qi0HY9tmha7VNq5OnGNMLEaHtzORbx9BQCz4xS53qR8ZEuQlFrYUdCeqF/xrYz/zh2M045FdjPaCw3HLmKF1eL3QifnS4dpthvbdqjGYDLcbmYNQoLeFbopOisKAp8EGpRqG1OT3rE7kBUJCgWB0FRttoafzhMFtAEkgupcmLypDf3Es14cm+4h98ODniun61IYpyLGroAIBMG4Lms1pmAq/Bff38OrvKzgiDt1i7fiH9VWXRE1XsWSLdBOv6yDFHn8BaPze0lANT5qztE3YND4y3PAqeJTCv1nTovk4S1L8ASEvyK3WZ+mOEPo7Dl5d3S/D43lLLNkAIN9pcZDyBvbBIsk5ZroCZ/ao7jtwVq3atMGwlALZsMy7ArbU1WcAWLJzgCm4t8XX5y5hCpGVjKXDbRKUQFPK9lTCCY0LS3rfZXkACZZab5WnljXCJyHMhBT0Yxf4xz3XGwE27FPK9MfW9tyM3aQlfNIKAAoNbRmznYxlXHjDHM7H7OrXh5L2AUGZWFGJQxtStBKwiPFCy50Uh7zyH8GvMSlmTnYZI/eausRZfjMXvb1Dnd6ucibYjgiq1PXRtxA/vwZmp XjAciw0P XR0ijD72k7i++jRXJHqPCAmmyYMhidCR1lu9d+dJ0w961Aq8VZq85g3ni/Hw/a5BWyGiZk5IfLuuus1V8lQphTtmuwnlMXJbSP7hKM28h7/TP7T9c+dWkPnmU1imJR4G2fRNH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We always use insert_zspage() and remove_zspage() to update zspage's fullness location, which will account correctly. But this special async free path use "splice" instead of remove_zspage(), so the per-fullness zspage count for ZS_INUSE_RATIO_0 won't decrease. Fix it by decreasing when iterate over the zspage free list. Signed-off-by: Chengming Zhou --- mm/zsmalloc.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index fec1a39e5bbe..7fc25fa4e6b3 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1883,6 +1883,7 @@ static void async_free_zspage(struct work_struct *work) class = zspage_class(pool, zspage); spin_lock(&class->lock); + class_stat_dec(class, ZS_INUSE_RATIO_0, 1); __free_zspage(pool, class, zspage); spin_unlock(&class->lock); } From patchwork Thu Jun 27 07:59:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13713910 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40227C2BD09 for ; Thu, 27 Jun 2024 08:01:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C31026B0099; Thu, 27 Jun 2024 04:01:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE34E6B009A; Thu, 27 Jun 2024 04:01:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A5A266B009B; Thu, 27 Jun 2024 04:01:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 85FF46B0099 for ; Thu, 27 Jun 2024 04:01:23 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9E306141AAC for ; Thu, 27 Jun 2024 08:01:22 +0000 (UTC) X-FDA: 82275923604.17.C593E54 Received: from out-184.mta1.migadu.com (out-184.mta1.migadu.com [95.215.58.184]) by imf19.hostedemail.com (Postfix) with ESMTP id 6B8051A000F for ; Thu, 27 Jun 2024 08:01:20 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=onetD7Ma; spf=pass (imf19.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.184 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719475263; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zLaVUbIzK4uy17+aNn995y1ea9NwFkc4qJomwmgNMH0=; b=FcpyELqlrjWuH29OZys/AjoDRqldlPFQQX1yc5Wn3KXnLDU3E3b0wjDGt+kNEhsh9baxlq O29zZIbWHkcar8C1RNqQ4hpPTaadX8G7br1PqiX3BvIl61BIc9HuPXvaUiU9QXiLJSdNnv LzdSj1CQbVC6T1ByYIAfYcfSSfZN/kY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719475263; a=rsa-sha256; cv=none; b=xOoU30btcvvMIsmzNVG9eCv3SprJ2duFRzHV2+QO2496UttTu8bO2OLkhs34JW+2ot9c3d IqRdN5b4ZgM5e+4wjODaqFcu7dgqeVHYsPvACQzAHmtndgsujupVLIPm4qXFkyhNGQMHaE 9uzhQVdX3N4lkxNSaOgzlz/CUWBsAc0= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=onetD7Ma; spf=pass (imf19.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.184 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Envelope-To: minchan@kernel.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1719475278; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zLaVUbIzK4uy17+aNn995y1ea9NwFkc4qJomwmgNMH0=; b=onetD7MadjrwiRohJi7DIxKPhHmIiojjfSQSPlRGIFGOdstA62rHNjHOpdis3341h4iEJ4 mpoA/KOV+fyTEipHhyH4PAfAo5BXP6ii137lGk7O5MWOnpeVnwc5QeLasHHL/DMeejhxz5 zCx7TZ/1auPzMNE/v/PrtM8qurAKbDQ= X-Envelope-To: senozhatsky@chromium.org X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: chengming.zhou@linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou To: minchan@kernel.org, senozhatsky@chromium.org, akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev Subject: [PATCH 2/2] mm/zsmalloc: move record_obj() into obj_malloc() Date: Thu, 27 Jun 2024 15:59:59 +0800 Message-Id: <20240627075959.611783-2-chengming.zhou@linux.dev> In-Reply-To: <20240627075959.611783-1-chengming.zhou@linux.dev> References: <20240627075959.611783-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Stat-Signature: mjorqwm7tts14sjq7x7ntp81c8uj4ahe X-Rspamd-Queue-Id: 6B8051A000F X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1719475280-131370 X-HE-Meta: U2FsdGVkX1+xbUoGIbcyvL1BD+XpgwfXMjC9O96znbh05TlTdd9xKTbfzZmKW69M8pmxwA+5TGn/BIsyFnXK4MDTyDsm//2hcsuxLZA4sWz9GfkvGr625S/iRkDlWi6oQDFVyvaiKZRn2q8oG307prsD5n8APxSrNUB5qD6YUfGBQfh8zgPvXbQpSDGysK4F85h8psQyhVrNBArvL0B/giYgSw4IFQIszTTSDm/h4emf8gRDA7SY/EGUPgC7W74XAXLFW2qSaWuLsP1JAYJXRJtu1iwMRfrm7yPtoDlZpantV7+UhEDeo3bUVdC9y5X1nOG0JD1LFkEnArgyRMnO9BDGrZSJwALemdjQxkM5Tu58v/7HtQKIJWuEHWlLFlZxpJl4dXEjNha1a8OPlyT0R1p6ORqMK6uBggi9EdJy9cIvXmYxVOhBo1ziOdRCu7d/1fzvJPbFqlPjLyaCJhJVpxbgh1VKi1wh6EvIOJIbhZH4DcDtA4QYccydiAZ9KqWfudo7LQD4EDkBDbbUF9h9YB5h6pxjPNQSW0TogmbCZLI56g7gmfwUqXCnKike5mPQBlX2aOLcp5tj49JHsB3EVHt7CqppR9jlH+92mTi/lnxnK+AJ1tb/diKXjmDrU0bM9iDoLsTbXu8nShxl8YPwCWhW28mh/ND6cya3YWqI/Sp4MCmPU6LPGb/CrbN25W3DNc1eJF/u1pwoSt8mMaHmR2yXRFCrKJT5JhQFldKUpKwh5zN954LPJSOYTxfU/nIGxsFEuAJn7mrm/oOYjESnAy8EHF3sw2X7C/H/EXEBVmazbCGZnvN87v+mFQHnp86xsdfcCbzyKA8+GLD7pQr13WkfVrxPNccIJkQdZD1Ct7AyJxmBcxxmC1nzq3eKz2HDhmGUuUVKqxCG7Fb6xb/AFuOnswiLh+rA/Mu4l/efQ86ov74BJraprnUuo5HbUTvr4i5YQAk/qBOXROgAEbu wb0r9O6/ 6yrOf8XbG+2N9DcxGGNu1GIDqJw6MM2NSoCVJ7aNqnFTDjYMrMR4Pmu0O8QE7a8XBRipu62noOs4ca06mGq2MJxmujTiCn4/fVZmW5m+NXS6PCT0mhNYPscmeY9v7LdBN7bfapU6DTuco4JWDEMlW5G7Qp/Iat/Fr0aGG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We always record_obj() to make handle points to object after obj_malloc(), so simplify the code by moving record_obj() into obj_malloc(). There should be no functional change. Signed-off-by: Chengming Zhou Reviewed-by: Sergey Senozhatsky --- mm/zsmalloc.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 7fc25fa4e6b3..c2f4e62ffb46 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1306,7 +1306,6 @@ static unsigned long obj_malloc(struct zs_pool *pool, void *vaddr; class = pool->size_class[zspage->class]; - handle |= OBJ_ALLOCATED_TAG; obj = get_freeobj(zspage); offset = obj * class->size; @@ -1322,15 +1321,16 @@ static unsigned long obj_malloc(struct zs_pool *pool, set_freeobj(zspage, link->next >> OBJ_TAG_BITS); if (likely(!ZsHugePage(zspage))) /* record handle in the header of allocated chunk */ - link->handle = handle; + link->handle = handle | OBJ_ALLOCATED_TAG; else /* record handle to page->index */ - zspage->first_page->index = handle; + zspage->first_page->index = handle | OBJ_ALLOCATED_TAG; kunmap_atomic(vaddr); mod_zspage_inuse(zspage, 1); obj = location_to_obj(m_page, obj); + record_obj(handle, obj); return obj; } @@ -1348,7 +1348,7 @@ static unsigned long obj_malloc(struct zs_pool *pool, */ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) { - unsigned long handle, obj; + unsigned long handle; struct size_class *class; int newfg; struct zspage *zspage; @@ -1371,10 +1371,9 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) spin_lock(&class->lock); zspage = find_get_zspage(class); if (likely(zspage)) { - obj = obj_malloc(pool, zspage, handle); + obj_malloc(pool, zspage, handle); /* Now move the zspage to another fullness group, if required */ fix_fullness_group(class, zspage); - record_obj(handle, obj); class_stat_inc(class, ZS_OBJS_INUSE, 1); goto out; @@ -1389,10 +1388,9 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) } spin_lock(&class->lock); - obj = obj_malloc(pool, zspage, handle); + obj_malloc(pool, zspage, handle); newfg = get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); - record_obj(handle, obj); atomic_long_add(class->pages_per_zspage, &pool->pages_allocated); class_stat_inc(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage); class_stat_inc(class, ZS_OBJS_INUSE, 1); @@ -1591,7 +1589,6 @@ static void migrate_zspage(struct zs_pool *pool, struct zspage *src_zspage, free_obj = obj_malloc(pool, dst_zspage, handle); zs_object_copy(class, free_obj, used_obj); obj_idx++; - record_obj(handle, free_obj); obj_free(class->size, used_obj); /* Stop if there is no more space */