From patchwork Wed Nov 10 18:54:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Minchan Kim X-Patchwork-Id: 12612577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5050C4332F for ; Wed, 10 Nov 2021 18:54:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7CF4661247 for ; Wed, 10 Nov 2021 18:54:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7CF4661247 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id BA9236B0078; Wed, 10 Nov 2021 13:54:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A969D6B007B; Wed, 10 Nov 2021 13:54:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 932C06B007D; Wed, 10 Nov 2021 13:54:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0042.hostedemail.com [216.40.44.42]) by kanga.kvack.org (Postfix) with ESMTP id 84BC16B0078 for ; Wed, 10 Nov 2021 13:54:42 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 3EE108287DFB for ; Wed, 10 Nov 2021 18:54:42 +0000 (UTC) X-FDA: 78793921836.19.61B5057 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by imf06.hostedemail.com (Postfix) with ESMTP id D964F801A8A0 for ; Wed, 10 Nov 2021 18:54:41 +0000 (UTC) Received: by mail-pf1-f175.google.com with SMTP id m14so3458811pfc.9 for ; Wed, 10 Nov 2021 10:54:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=f4XRsa4d0py4V38UEJRkvGwgasV1XmLgNajIpiYk5Ig=; b=cME2Bx0n24Xt4SjSIywzCjrzmmr1ACxgoz3HgA0u00IXYQ2s8ag4BkYR9Ob5qlUjRK +iMcaU2Z3tWMjphj2BTWy+274IFUeQddnJbwJMzOZ9iVo6OtQNKaWIrxC8ynFHViBs9n jbceAllpy0b6GkbIQVQPWseuWH6KDZX8Q+T9doiiTZYnNZIQUTTDmfj45RNkqHlF6HOx BjcGNtR3IpFHzbbx4gjDXdSuCylsdkH4Eth3aAW85xedtQa+dJniqlkC9KABibsZjf92 IS9+sXSuxaaFfol4tOlZ/6MDKQygKx060b9kwQEDpTc6KG3rNzgM/YW9dQyL13Qh4UWO HanQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=f4XRsa4d0py4V38UEJRkvGwgasV1XmLgNajIpiYk5Ig=; b=LaPyF7trr5LTU9BBk+gYAMRwRUWPqAmLIdhkcTORpK+lu4/qpwmCom4RHu++PWA3Qu elvFbevzDtoxkqMcSl/zdHaUaBKgI3UK3yFOHLkGG8bm1jU9N7QJ4acBmgyPbLVkHJ8s wUnhuz5Zgi+kVsJYcXJFee5YKRFwAYZ1Bjolol2Z0UVttMdwbHBRfJXe50tWdm72A93s xEb9bq7o6uyPeAvR9YBXmtgOjmRtB8ivqA1uJpv7my1jW30LkNp+o/NurUmyjNVzEjhN MYrQID05YetlhRAldsNv7lBodrY7uAK39kVuqNHzIDr/3ZrX+mJZ+R/DgjIsz1LSQIjI i0Mw== X-Gm-Message-State: AOAM530h6vxsm6qP+CaF2SBduhCQWexNxcWuDo5bQ8BoGmbbSVR/LFD0 bZ9E89KjBLfTJhXDpt2VaLc= X-Google-Smtp-Source: ABdhPJwu5o70vPSjBVwwMx2PCumm6qq7tL/a9fcTlAJtGCEm0JRVadhsdd7mIvQUA93Flvm269yizw== X-Received: by 2002:aa7:9511:0:b0:49f:e35c:f6f1 with SMTP id b17-20020aa79511000000b0049fe35cf6f1mr1214454pfp.45.1636570480872; Wed, 10 Nov 2021 10:54:40 -0800 (PST) Received: from bbox-1.mtv.corp.google.com ([2620:15c:211:201:11d4:2de3:ab82:be64]) by smtp.gmail.com with ESMTPSA id g13sm325253pjc.39.2021.11.10.10.54.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Nov 2021 10:54:40 -0800 (PST) From: Minchan Kim To: Andrew Morton Cc: Sergey Senozhatsky , linux-mm , Minchan Kim Subject: [PATCH 3/8] zsmalloc: decouple class actions from zspage works Date: Wed, 10 Nov 2021 10:54:28 -0800 Message-Id: <20211110185433.1981097-4-minchan@kernel.org> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211110185433.1981097-1-minchan@kernel.org> References: <20211110185433.1981097-1-minchan@kernel.org> MIME-Version: 1.0 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=cME2Bx0n; spf=pass (imf06.hostedemail.com: domain of minchan.kim@gmail.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=minchan.kim@gmail.com; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=kernel.org (policy=none) X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D964F801A8A0 X-Stat-Signature: dmxf5ud9ig49enzkwmedxt1kwztd6baj X-HE-Tag: 1636570481-992200 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch moves class stat update out of obj_malloc since it's not related to zspage operation. This is a preparation to introduce new lock scheme in next patch. Signed-off-by: Minchan Kim --- mm/zsmalloc.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 0b073becb91c..52c6431ed5c6 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1360,17 +1360,19 @@ size_t zs_huge_class_size(struct zs_pool *pool) } EXPORT_SYMBOL_GPL(zs_huge_class_size); -static unsigned long obj_malloc(struct size_class *class, +static unsigned long obj_malloc(struct zs_pool *pool, struct zspage *zspage, unsigned long handle) { int i, nr_page, offset; unsigned long obj; struct link_free *link; + struct size_class *class; struct page *m_page; unsigned long m_offset; void *vaddr; + class = pool->size_class[zspage->class]; handle |= OBJ_ALLOCATED_TAG; obj = get_freeobj(zspage); @@ -1394,7 +1396,6 @@ static unsigned long obj_malloc(struct size_class *class, kunmap_atomic(vaddr); mod_zspage_inuse(zspage, 1); - class_stat_inc(class, OBJ_USED, 1); obj = location_to_obj(m_page, obj); @@ -1433,10 +1434,11 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) spin_lock(&class->lock); zspage = find_get_zspage(class); if (likely(zspage)) { - obj = obj_malloc(class, zspage, handle); + obj = obj_malloc(pool, zspage, handle); /* Now move the zspage to another fullness group, if required */ fix_fullness_group(class, zspage); record_obj(handle, obj); + class_stat_inc(class, OBJ_USED, 1); spin_unlock(&class->lock); return handle; @@ -1451,7 +1453,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) } spin_lock(&class->lock); - obj = obj_malloc(class, zspage, handle); + obj = obj_malloc(pool, zspage, handle); newfg = get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); set_zspage_mapping(zspage, class->index, newfg); @@ -1459,6 +1461,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) atomic_long_add(class->pages_per_zspage, &pool->pages_allocated); class_stat_inc(class, OBJ_ALLOCATED, class->objs_per_zspage); + class_stat_inc(class, OBJ_USED, 1); /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); @@ -1468,7 +1471,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) } EXPORT_SYMBOL_GPL(zs_malloc); -static void obj_free(struct size_class *class, unsigned long obj) +static void obj_free(int class_size, unsigned long obj) { struct link_free *link; struct zspage *zspage; @@ -1478,7 +1481,7 @@ static void obj_free(struct size_class *class, unsigned long obj) void *vaddr; obj_to_location(obj, &f_page, &f_objidx); - f_offset = (class->size * f_objidx) & ~PAGE_MASK; + f_offset = (class_size * f_objidx) & ~PAGE_MASK; zspage = get_zspage(f_page); vaddr = kmap_atomic(f_page); @@ -1489,7 +1492,6 @@ static void obj_free(struct size_class *class, unsigned long obj) kunmap_atomic(vaddr); set_freeobj(zspage, f_objidx); mod_zspage_inuse(zspage, -1); - class_stat_dec(class, OBJ_USED, 1); } void zs_free(struct zs_pool *pool, unsigned long handle) @@ -1513,7 +1515,8 @@ void zs_free(struct zs_pool *pool, unsigned long handle) class = zspage_class(pool, zspage); spin_lock(&class->lock); - obj_free(class, obj); + obj_free(class->size, obj); + class_stat_dec(class, OBJ_USED, 1); fullness = fix_fullness_group(class, zspage); if (fullness != ZS_EMPTY) { migrate_read_unlock(zspage); @@ -1671,7 +1674,7 @@ static int migrate_zspage(struct zs_pool *pool, struct size_class *class, } used_obj = handle_to_obj(handle); - free_obj = obj_malloc(class, get_zspage(d_page), handle); + free_obj = obj_malloc(pool, get_zspage(d_page), handle); zs_object_copy(class, free_obj, used_obj); obj_idx++; /* @@ -1683,7 +1686,7 @@ static int migrate_zspage(struct zs_pool *pool, struct size_class *class, free_obj |= BIT(HANDLE_PIN_BIT); record_obj(handle, free_obj); unpin_tag(handle); - obj_free(class, used_obj); + obj_free(class->size, used_obj); } /* Remember last position in this iteration */