From patchwork Thu Jun 15 03:48:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhongkun He X-Patchwork-Id: 13280676 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB11EEB64D9 for ; Thu, 15 Jun 2023 03:48:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 010276B0072; Wed, 14 Jun 2023 23:48:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EDB846B0074; Wed, 14 Jun 2023 23:48:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7BBE6B0078; Wed, 14 Jun 2023 23:48:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C429B6B0072 for ; Wed, 14 Jun 2023 23:48:44 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8D59AA0948 for ; Thu, 15 Jun 2023 03:48:44 +0000 (UTC) X-FDA: 80903600568.13.D78C0EC Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by imf14.hostedemail.com (Postfix) with ESMTP id 00A5F100003 for ; Thu, 15 Jun 2023 03:48:41 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="QuB/9REr"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf14.hostedemail.com: domain of hezhongkun.hzk@bytedance.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=hezhongkun.hzk@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686800922; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=Vbu+qXDQAnJmw8qCM+u2edU9xMFzBPDexgHZg7upYSc=; b=RsJu5Cy2QrpSsP8dgyH9/irQglNFIjw1+hHQlP2N6WmnHw/qjT7FNS115cSaCFVGpLRXIM 1IQa4hHdOxjKclPtpslod93v6D2EpfGzdIl55Ld64EZiaP/FzMHm8Xjdo8JIF9gh0gjlCI zI6VX1rCuoiiYaP+xgXSh4vqMladTnw= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="QuB/9REr"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf14.hostedemail.com: domain of hezhongkun.hzk@bytedance.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=hezhongkun.hzk@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686800922; a=rsa-sha256; cv=none; b=PYWLPYMO6uqPWfVSQGBfG05z2+zVyD7UbpSuChSHo57StJ4J7IKdJ8p5xYRrJ04JwXKkso XS043lubXTPM2mzUZyirxrQ4UfmvgU3vnml64lUR4AFZ1Zf8KdyI0OCcmcqDMeGCYg1PF5 FakSJuYVe96B0Gln0ceAfQumkffyItA= Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-543d90cc675so4144421a12.3 for ; Wed, 14 Jun 2023 20:48:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1686800919; x=1689392919; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Vbu+qXDQAnJmw8qCM+u2edU9xMFzBPDexgHZg7upYSc=; b=QuB/9REryqq6Mgb8L/e4zMTLqzDoDXfDapT3W0Gx/mxECAPvMPdAXb/7xqeOhW6cOS rDSsPeeYrCYrv5l2lI+PFC6KhRdpynj5Nc5W68ZABRG0SVuM4SYMvOjbDQp9yiwdfuFg c3IlUzDHTM3Fe8qKMpcnHym1vs/Tc9pnPa/Z9VZjvYSWuOV+oc3k2SLfRMFyiHBgDHcf WE/hXQy+tuxSMe7jlfmDYO9nNIjXK8xkQy8hrUeKjS7HgAKMRd7kpWv9b+YE46IvVy/u dilXepFYsW49CWNtr+c9qK0KedUGDaNOr6gL61uSNoBrjIPF6bYFFyLINhJRBq0lk7Pf OxnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686800919; x=1689392919; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Vbu+qXDQAnJmw8qCM+u2edU9xMFzBPDexgHZg7upYSc=; b=SQKpX/ufd+25phs4RDjlupv23689neaxu0aDWxgZzTwE7r1AAPHvwFh6qvn+py3OPH OVHo8Ok8FnQJeaN6bZ7XFaXUpkqwFL3QC5lmRZnRK0/3nsdMwTYbIOWYm1RfKkRusxVv z2x17ibu4FhdnoVu6oSnoUYidMsnGEd8dxb6N+8URZQ7Bq7lVIjlEgblPHr+6lzCZe8n yEDjir8M7TstIRxlTxiBqEyZE9deUQ98pwkNcE3xR+BVUuGJLUNqdpdBCaDRGMyKJjUT 2OEJSylyrRCTnv5VudybmdJpiQvtyR6OeZDFe0qYiuH17EQC6xFm6VbCGESvMgYTD7Z2 cJSQ== X-Gm-Message-State: AC+VfDwHgZlRZj/Ckj0WMAzhVR0f9hJhjb6T7jBTtGfJ/PFxeRMsmnMS 2PvzXloeLFyMDdZAc9fWVUkp0A== X-Google-Smtp-Source: ACHHUZ7IJ/OOff5P+kevIRNHk4rSmF9QFuBJH8tCKcPmy/VcWRp1kVAz+QNpkDSFj/YM7GxNJewtnw== X-Received: by 2002:a05:6a21:29ca:b0:11d:d83b:8d80 with SMTP id tv10-20020a056a2129ca00b0011dd83b8d80mr2265679pzb.46.1686800919047; Wed, 14 Jun 2023 20:48:39 -0700 (PDT) Received: from Tower.bytedance.net ([203.208.167.147]) by smtp.gmail.com with ESMTPSA id 19-20020a170902c11300b001b3fb1119fdsm2751713pli.297.2023.06.14.20.48.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jun 2023 20:48:38 -0700 (PDT) From: Zhongkun He To: minchan@kernel.org, senozhatsky@chromium.org, mhocko@suse.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Zhongkun He Subject: [RFC PATCH 1/3] zram: charge the compressed RAM to the page's memcgroup Date: Thu, 15 Jun 2023 11:48:30 +0800 Message-Id: <20230615034830.1361853-1-hezhongkun.hzk@bytedance.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 00A5F100003 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 6yxmpj9rq9etkpws899esa7dgbq4bacf X-HE-Tag: 1686800921-32357 X-HE-Meta: U2FsdGVkX1/X+H2SmDqLMF2vbtKcjixvseyLpQPfAUYqP6rHjjebwvjsTRh0ghYK2PMxB1ZRJDON2BZKtYF4rDURB+PLnq9IYKzkjVRbKPexOIbfiaCeFbNgKAbBvVK7Lk5js5fhLcdKAsCWp7pz13iY34jTx7foH7ZASvh0ALHsYz8H0sGyM0C8WpScR8SPYGw09lUQM592Q3kMjkoJ8ns9PVu4C/hrdY5kDJ3nHvgMJOC52q2i1GAzlrGtM//32b+F93bQB8TZ+IbDhipBgvakIrZQ+Ice/FH3MvGQSV43/uMVUx1BysC5Kg8PshZrC47P7Bi1rKLxPdYXAVLRfePSn+gWfNZ+cn5nk7vIHbRMLwgXiRnan3U4nKlseQRaQmjzoDzw+Aqiwb4yIsqxkAJsB6YINbkUBbXgbJmxi8aYln3kA0sJ7dVMGEPMxklOlzq10cRo0TNpc6SXY3h6pmgun6F0Hol6QqENXJsCtcv730Ytqw0oJhZJzq+GuJgvLCXZAZmYuhASVJxfeFVgpK08fzqOC+g0Q++nuvbFNf5pAyCi0DJKJoJoO2JpKZcw8G+NdvAR6jte3J6U/TC6xwDFTPWupwaOGx7qeEwwXuTmNdFk7T6q5Nt8HZLHErax3R53qxlRFJe1IRt0UwfKv2Ydg9Hw64QjlZQAKLjKlfGbawp0KH9Rico7EuRVuI56TFCS1hHP8vopm13M59jY8tEiLaiRBe5IzccFehpCOa5UuCU2lofEAhlAQSH995MfVEygVpy/bE6cAdwwdsEq0GGgfe5gxvsexJQMNGSMgrGm9ai4r4kFFqcuRhZPapRTycyLU7frk7R6PNWjE3W7eJlu83iHaf7r2b3cKeJzOMRBOK1aoZIH51+cmlV2aIN2hlxz7ugGEJg1h72OrNS4r7PL37A7ZcfDVuUx74Bjrn/E40GLVuSu+MN4GpYEyjcUgWznQyfevuRio6dSAJw wqH6FaMV jg8snx+KlReI5AYQ9p76vPYY8LnKMuigb8dfV2pvLTcMNYaADQWJtEhApZrG0Qur9vkUrcwAtEdJjAHbXEdqMCF50rNJJYtTwTjALeaDL0ZTk224OV/QIUVeJV0d4r9CmOtQdReorGRBcIrqTMmwhTKrpkMOse5GxCIwricVlSx5dVMefssIPuuK+3TQk13W3a9NItplJQ6C2jQrHvjpdfJ6uAqAvxhFRw7Wft6cRI8xIIg237IPtIAAp36J3Wt6b1eltONKLEXdexSBx7e6rqz58vfD08opJNK/WUL50h8aX/AzMHJPFppbJtL+R+FbOrO7dNhi9DWuH3/L5Ypl0kbGI5IzVmKdlgKpvpePbWRqa+C4tndyIxf8T8Ms3z1RPfCpHZqERyABqTQIopH+7mXlgPKC+If1JPRNkKRUEap4v9SFly0RsKzEM5g5Td7JRgZSC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The compressed RAM is currently charged to kernel, not to any memory cgroup, which is not satisfy our usage scenario. if the memory of a task is limited by memcgroup, it will swap out the memory to zram swap device when the memory is insufficient. In that case, the memory limit will have no effect. So, it should makes sense to charge the compressed RAM to the page's memory cgroup. Signed-off-by: Zhongkun He --- drivers/block/zram/zram_drv.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index f6d90f1ba5cf..03b508447473 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -33,6 +33,7 @@ #include #include #include +#include #include "zram_drv.h" @@ -1419,6 +1420,10 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) struct zcomp_strm *zstrm; unsigned long element = 0; enum zram_pageflags flags = 0; + struct mem_cgroup *memcg, *old_memcg; + + memcg = page_memcg(page); + old_memcg = set_active_memcg(memcg); mem = kmap_atomic(page); if (page_same_filled(mem, &element)) { @@ -1426,7 +1431,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) /* Free memory associated with this sector now. */ flags = ZRAM_SAME; atomic64_inc(&zram->stats.same_pages); - goto out; + goto out_free; } kunmap_atomic(mem); @@ -1440,7 +1445,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); pr_err("Compression failed! err=%d\n", ret); zs_free(zram->mem_pool, handle); - return ret; + goto out; } if (comp_len >= huge_class_size) @@ -1470,8 +1475,10 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) handle = zs_malloc(zram->mem_pool, comp_len, GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); + if (IS_ERR_VALUE(handle)) { + ret = PTR_ERR((void *)handle); + goto out; + } if (comp_len != PAGE_SIZE) goto compress_again; @@ -1491,7 +1498,8 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) if (zram->limit_pages && alloced_pages > zram->limit_pages) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_free(zram->mem_pool, handle); - return -ENOMEM; + ret = -ENOMEM; + goto out; } dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); @@ -1506,7 +1514,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_unmap_object(zram->mem_pool, handle); atomic64_add(comp_len, &zram->stats.compr_data_size); -out: +out_free: /* * Free memory associated with this sector * before overwriting unused sectors. @@ -1531,6 +1539,8 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) /* Update stats */ atomic64_inc(&zram->stats.pages_stored); +out: + set_active_memcg(old_memcg); return ret; } From patchwork Thu Jun 15 03:48:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhongkun He X-Patchwork-Id: 13280677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F3C4EB64D9 for ; Thu, 15 Jun 2023 03:49:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BF70F6B0074; Wed, 14 Jun 2023 23:49:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B81638E0002; Wed, 14 Jun 2023 23:49:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9FA268E0001; Wed, 14 Jun 2023 23:49:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8A0796B0074 for ; Wed, 14 Jun 2023 23:49:02 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 529561C8A54 for ; Thu, 15 Jun 2023 03:49:02 +0000 (UTC) X-FDA: 80903601324.30.4A0A104 Received: from mail-qk1-f172.google.com (mail-qk1-f172.google.com [209.85.222.172]) by imf16.hostedemail.com (Postfix) with ESMTP id 9DCC9180003 for ; Thu, 15 Jun 2023 03:48:59 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=TR6OrR3f; spf=pass (imf16.hostedemail.com: domain of hezhongkun.hzk@bytedance.com designates 209.85.222.172 as permitted sender) smtp.mailfrom=hezhongkun.hzk@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686800939; a=rsa-sha256; cv=none; b=P3t4sTivo7hutSjhUyeNJn6X0WWA8Gd/UivnL8YibGdBJ0QlcCODd7DrDTERTvvpfFIpyg BUUhetD7bggJKGwfTqX19U6kJA3+TlGwMDZ4q+z0mHYqiXva7jx8E7MISe/RdUEgIDI2xV R5unerJlYU/BlNRSETiVTJF4ggOLSA0= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=TR6OrR3f; spf=pass (imf16.hostedemail.com: domain of hezhongkun.hzk@bytedance.com designates 209.85.222.172 as permitted sender) smtp.mailfrom=hezhongkun.hzk@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686800939; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=NRjYS/ADH2rjGpNFpYFXCA45Do2mTCUsIGO3/Z8z6+w=; b=XzFg8291Wzt3SQEbicmIWSKJI+b/+IIBw3zW1vSjSjrj0zhQx2gdGHH/b+1yf4Lqh3ZDKA hn/X/EBxhnpcl6LhlYqDnY4k9rGyXsQv4XC3LpWxDWlwwjycd2OS+6Cht8XRmkseeQR+no O3bJ/CXrcb9lO7ItNpcifZiGcIDdO6I= Received: by mail-qk1-f172.google.com with SMTP id af79cd13be357-75ebb3d57d0so274992185a.3 for ; Wed, 14 Jun 2023 20:48:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1686800939; x=1689392939; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=NRjYS/ADH2rjGpNFpYFXCA45Do2mTCUsIGO3/Z8z6+w=; b=TR6OrR3fFDYNkCl+qhYqAsxfyiD8kknZZjEleHF/oB/mIAtrHmUqxT5dh8hGkLm7x6 jETfLCXhDDQw3QbD/HPI3dKgkAtov4zpQaVmJPSO1erj3zR9xOsTFOp/lu60HJH6nsZ/ 4QyKhVFs6hTtM1Nv/n0pMN9cqjRyFBJ1twgQJkxpYVM0qUnn216Whur/qwooRbmn9AeT kof5U1V3m8rwug8zJXXUBQJk+6gYIZBF4L5DgmHRpFHd4qzoGdTpoEVL0cGg3oFV0vDF BLwKzNKm+0bR1j1KZrUOTSyLGxMdfiQRFInPi2+TrbQvj9I13/lac+ftr4xbOUYSQSGb ObmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686800939; x=1689392939; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=NRjYS/ADH2rjGpNFpYFXCA45Do2mTCUsIGO3/Z8z6+w=; b=L6el/nodZzmZDEq70zONKykJ15ZGttf+LnjCGVfcWO0eV9mfcf7+D48DF+ts/4ugrN Ge39H60qQMY9RsAgiDWbNehS/a/t2P7tG0jDl21prI8UoGfFyUi0IJqo87qHqdrK8/hf bsQkmTMJTEt3f0DKHKDp9L3+UYJuBWI08lpFQyNFYUDB315gcfZKm/DkQ8iqUdV+hl27 WtKvw2CJG3HsqclX6dablYsGAnhxLAAv3q/e44E1Ftfsk8DlXkHAdcUrQDNfLh6MTfB6 NKnceKwvoucyQeyacvnK3lgsuQFD8K50v6UHDBWxoZJy3oFMtfVHHj7NbkLUL1LQbbrF ytug== X-Gm-Message-State: AC+VfDxTQRXGVL3rFVkXxavQFpgPX3AlToXLTY5vjTO6by1mxJdN7sWz MaGwWi54GLYrEq9RmtIzRhzRWg== X-Google-Smtp-Source: ACHHUZ76N8XpYmzFAvhMwHbuyElh6B1OPNzdSBFlDx7RhGFePCnX/L9cfOqNK5UXZWfPHlXbijOyhA== X-Received: by 2002:a05:620a:28c8:b0:760:859a:d713 with SMTP id l8-20020a05620a28c800b00760859ad713mr15005467qkp.56.1686800938766; Wed, 14 Jun 2023 20:48:58 -0700 (PDT) Received: from Tower.bytedance.net ([203.208.167.147]) by smtp.gmail.com with ESMTPSA id 21-20020aa79215000000b0064559b58eb8sm11012074pfo.154.2023.06.14.20.48.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jun 2023 20:48:58 -0700 (PDT) From: Zhongkun He To: minchan@kernel.org, senozhatsky@chromium.org, mhocko@suse.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Zhongkun He Subject: [RFC PATCH 2/3] memcg: export obj_cgroup* symbol to charge compressed RAM Date: Thu, 15 Jun 2023 11:48:49 +0800 Message-Id: <20230615034849.1361940-1-hezhongkun.hzk@bytedance.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 9DCC9180003 X-Stat-Signature: eay45bj9xcow84a9cz67mzc6ctep1eg3 X-Rspam-User: X-HE-Tag: 1686800939-195836 X-HE-Meta: U2FsdGVkX18ycQbn1Ef9x/TAOiU800Ki5eecKlufmmQaILdnHRoHfUxlIhgoKPpWYWWY79xQxWhoKu73GNDRzKo91kTLCx+g77KAq/YZzeNnH2NT9/WOK2sBJJlVLhWlKFA0fWQnQVc/ofng5qtLrTqGaIDaZwUDicjF6eY2OpNCu/+vxcHS2pABYPZq5azz+4FxR7fPVTzjr/banV9lavUtudS+9eK2JEvYIr/v1PQCuFCxZqSHo9ApW0Fc3kre/TdHvg+kTRDTOK76gCN8OXst5oER9XKJrnJKV1ZmcxNfu2M4XzqmiHgOW3QrLmsgddR2r2bKP4wzeVJFhwcp7u2aX+481FVbPx1kw5CS5OXe/i/E0j/CP1kEDZCuTf1XkzkUmbtfOUIeQGiJsOTzbuOPKOoUU9IuN8QCG0bjvKc3SzIPQ4YL0GJYckMV1iqbLOdnEGXfGxpBNd67fqoVkS7Fsb/3vBOOZm6YzCjaGixFZ73NHIh7hrsCj0BPSW9Ux4JMRP9eKuJz7qwAiyVjrTV9ukBPGGy0JPOcLVRsEx5AJZ8QJv2g0YPgcOHg3WigRtj5sfBQ88YOrmBBDE0jubYouH2GKYFPpxb6crqwIux9+/WIjf2fQmHT/tJ2vmp58A9lwU8BmWLmoQMYXy8XWO4FqbpgxrjMxaDK+7bycBM50rXS2woW/o6Wuu/KtuThtJm4j6mmhOSNEiysCVXT/Mh7yXXX5RvoYZE5FfAPBC3Ld87K6E2ftBqvBXb+9/0pOiVORmLwyTgeHPLfjqzLmUyPt8WZp697cVfeIy1Gdd1pQduYd95eQ3+vTCYe5+dGY7qXcnxShLbkef0sZ5ze/GKOjGR6FG7QFdiuFYhxBHdcoaOz6ClI+xlbP1t4ERVZBDm4sR5HCMwD28XH2qk1zCrzj4qj+Rr1CJ1KOpS3egHMdq7UgLCkY2xY7lZFzbtURnVDAhvSoEQ5u1wmBzR vf1/yK/v X7E0UBuJN++yqje9ddeR4CYbrsVsrToM+sEZfgJjip6e9kmoeSoKeyM4C+QmGy7Zg4v6d5nOaxkJJkZoHHuTmOYMRm11cdNq2Ti3S1PJP/18o8aKfAqmAYpIgRNiY32S3pUaW+jib5+bDMNe74RgrKsmy5ZfMVcvVuZuQXm2bUPNbwPwuCXmffpx06fobW9otMtAtAOJHhtI54xeH3k9VZAHeMc0D8BV9m9e0XOOiCK4Vx4XHsd/V+tc2lLrwVg7Arb3eWZL0+IDky1ZU9AKyAV32CMnlY+IEvsthR2cRuadhhwmT06u6JHv/ihfVW7Y4CsGgYbHX6r0KSIkJ8zAcRCc7bdQWwLjwoQ10hdMWdcWkLFYNeoPTrkrnw+bX1sccaHS8ghfIM2zlfbXP/aneCoBuYXm8HnpTHbNSHV95S6qpprXKbm8IsASEjHxFFMzvm4pBgh+jnsoO61U= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Export obj_cgroup* symbol to charge compressed memory in zsmalloc module. Signed-off-by: Zhongkun He --- mm/memcontrol.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4b27e245a055..8ba93afacf31 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3064,6 +3064,7 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) rcu_read_unlock(); return objcg; } +EXPORT_SYMBOL(get_obj_cgroup_from_current); struct obj_cgroup *get_obj_cgroup_from_page(struct page *page) { @@ -3431,11 +3432,13 @@ int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size) return ret; } +EXPORT_SYMBOL(obj_cgroup_charge); void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) { refill_obj_stock(objcg, size, true); } +EXPORT_SYMBOL(obj_cgroup_uncharge); #endif /* CONFIG_MEMCG_KMEM */ From patchwork Thu Jun 15 03:49:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhongkun He X-Patchwork-Id: 13280678 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CD69EB64D9 for ; Thu, 15 Jun 2023 03:49:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E910E6B0078; Wed, 14 Jun 2023 23:49:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E187F8E0002; Wed, 14 Jun 2023 23:49:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C92148E0001; Wed, 14 Jun 2023 23:49:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B2BCA6B0078 for ; Wed, 14 Jun 2023 23:49:15 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 70CF8C0970 for ; Thu, 15 Jun 2023 03:49:15 +0000 (UTC) X-FDA: 80903601870.04.701A4BC Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf03.hostedemail.com (Postfix) with ESMTP id B3C7A20002 for ; Thu, 15 Jun 2023 03:49:13 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=RFOEvPUJ; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf03.hostedemail.com: domain of hezhongkun.hzk@bytedance.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=hezhongkun.hzk@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686800953; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=Ru9RQc/YctUd3oofmNo9B/wQzL/S7hlbQkvymAiMmdk=; b=z61srtX+whbFbE4N+kyGe6XI6AZJwbXA0YxVTnb7bj1Y8SNzJ74LX9vpwb7aQ5bYgNHy8u cDCMOqd9uX7AXwNzorV9OLUBoY6xtoe2WltzxqcVnTktzKNJPYMj/pIeVQtVk+6yWuWE8U BETH4gMece1t9gp4s+LS51G8QEAq+F4= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=RFOEvPUJ; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf03.hostedemail.com: domain of hezhongkun.hzk@bytedance.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=hezhongkun.hzk@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686800953; a=rsa-sha256; cv=none; b=RKChyUuRV3Ke2wlyGC0XpgUC/LUaGHfi8tI8DyvJ672PyIQ3Sk2T9b9zjk4ZVy9M1ln1qs HBODiqKDt4MnsqZIrQo4X3CBI2jg9dY9oaVa8skl4BfDML9cG0wkfCmoIx4Hb+j4UD3kmI vZ5lVoxnh6OrcdUI2iNNJR1T0GPj+ls= Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-662f0feafb2so265044b3a.1 for ; Wed, 14 Jun 2023 20:49:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1686800952; x=1689392952; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Ru9RQc/YctUd3oofmNo9B/wQzL/S7hlbQkvymAiMmdk=; b=RFOEvPUJE1xXVdLOmJQw4AYx9wGU3W9LSe9MDMMyjAwqI9uTfSl9XBmBC6yV1Yi4Kh H5ihk+wgHHlCnOOdlsK+YQIKHKcsa0sxfObaLcDMztLxyRgjxloUdUKjfcaCc5G09DE4 T1ZtX0M7w/f+A68B0vXL/ZEVD92E9KMiC/kc5Ymc3uTz6/eN3y0FCb3l4BVRWFuozSP1 zpZda+7WfBf2P6fjRmGIp0NBM73LgZXiCoJsxszp3yFdP3mEa7LVogULdTubmA4Z8Va2 dcEwfzIurYedtwnU1JpEKfxiGNiABEwM5APuMvNbbDATP0bRmEYfjh+gAcGsup4LFKbK uUAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686800952; x=1689392952; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Ru9RQc/YctUd3oofmNo9B/wQzL/S7hlbQkvymAiMmdk=; b=DphcUx3op/TPGUwpqUMFPfgPeh1bW59wVOOrl/12dpmeW2Qlj7GFPUwAReNI38eQR2 Ptbfoico1kGX0U9N6Rpa/kU0pQnaUJou1JGxjCWBeskWtZWlcKINFU6dx4tKepmFNx+E 6f/nrao5dnqDw5450mRIxJg/aAdDNo0HLX8tv6k8ZzIAnN4s6Mt+f0eGAQ5st7cu4Oyb 7LjzrUWkPIsSlQW8HFQagWWpOJV84HyUhXcE4hhlTHK0u3SYrcsQKEKOoDeBsCAi6xZ6 petG7dVfYGgLJKkguu3VtpDQWiUNtVRZ8XA9C+IzXJ9qt/A1rYruY7jigzM8ccIY+4/0 RZuA== X-Gm-Message-State: AC+VfDyPkbqNEBeP9bepg7uUxpwEXpAzArarstmjSkhFkobD1R716Zv2 yMhMhf9/dASxt2vdwzzpKYB8Og== X-Google-Smtp-Source: ACHHUZ4uuJz+fwISKlsP9Eu7UCx/w4kIpOrZngm6hKKNWnuEab1aS4tbt3/zosNjXGxk7SIc4kGFVg== X-Received: by 2002:a05:6a00:2e17:b0:650:1a64:d8d3 with SMTP id fc23-20020a056a002e1700b006501a64d8d3mr4983241pfb.14.1686800952295; Wed, 14 Jun 2023 20:49:12 -0700 (PDT) Received: from Tower.bytedance.net ([203.208.167.147]) by smtp.gmail.com with ESMTPSA id u6-20020a62ed06000000b00661694f1469sm6090369pfh.74.2023.06.14.20.49.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jun 2023 20:49:11 -0700 (PDT) From: Zhongkun He To: minchan@kernel.org, senozhatsky@chromium.org, mhocko@suse.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Zhongkun He Subject: [RFC PATCH 3/3] zsmalloc: charge the zspage's object to obj_cgroup Date: Thu, 15 Jun 2023 11:49:05 +0800 Message-Id: <20230615034905.1362034-1-hezhongkun.hzk@bytedance.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: B3C7A20002 X-Stat-Signature: 53pzw6och1onjyzmgy9z368nyq41fmcu X-Rspam-User: X-HE-Tag: 1686800953-4434 X-HE-Meta: U2FsdGVkX1/clvy+7beK19Q3ci/dfHNPoPlWsVxdEMjON4K+gNzB4t0M0GrRmJ38cLwr2QGuDq//GZ9VvYM3wttWUbf8rWCMNAQmJlKEEQasK8qeaK7QSKoQnacYocjYKcFyiVzUXKk71Rhj4Hb3rBYbwM34fgjzGxYFvBWCRxzqbHk+ZLGB0gyjoOoUfrq8pOnndNzx/3LfGM3+kJUi/kdCay8r5fZ8IuJmkrErDuD2diTOnimnY1qeXMUL6ASdzsBFWhOk0dA6mCdHLbG/I/L+GXiKn3KMp4eM/gXsHVEhA2iersUNot769aQl4xRd/4qUPTtyhV4riFLguMXS067tlbNZkbsaFVnzzrxhtAR07VNiPBWl6sd+/Xp6f4dP5FeWjLSAehPXPP9aK5p33lwxozVN5y54qMK+EyQuCM7jLVGZJ7+JhF9TZbHCAToKkh4avfRduhU2a0Z11uxSRVyVhBbiycqAbX4v6LT+uIv/XVl8T48vzes67hom+T3jYQS6ZpfZYAwOX55nEJ0rcg473Vgw6ytoaRg3U1NKbWBQtu2al4VHgMmT419C0SZvYSSmWP+BPrdtcABMXXBMo7rZk7ZyZ+JWKvX9HZgl9Q9d3l+bV8T7glAzNDfQmFJq+5raZiARb4BfIHYzZvha9yLqyTGmBzkPY+35l2XbIL32nqFH+NMu5zH+2xKvqLoh/Rv2Dq1wN33n7uMuqg4l0Ijv6guOABMWdVMYKPXrsP/wIKVeVSwRyZays4Si8eWRkTamTp0EF/VPvEX5BKG9UNSbOO3BQUoQ8pvXgJAKVl+pa0Hgxu1PETDDb9KZkBkmxFZhH52aiyhZuBs54mpvasLClijqUtqnYfpoqhlvwFTMsbSUPF0aWTDH+baq2+uy9bz/MlqXiGxomfpX6mV5ClqOKr5zRIgxW1IXOplFvPZaB7lY3YbnJ1uUQ46VNTR3XaKG4wNZsQV39wwPwmh JPkdXzWr W4QjRiB7etDCEJjmmolVQtoBREgEB+HtKdTUq1SyerfRiisTJwQrJTWr53DRC0m3WcrxZrJNKy/kNyUj6KR9vfMo85E8ChhDTldinonbHNpkH5y3gybqPbSCDVRAXa4PHhJn6kbCv6YAv0u6dNCoZUKWm5/GyYtU5GT51mbih9SE8T3+aDFImJZ1Ae4BkVRR+dh+rkFzkmzLlmHYc8npdfUA+sBpfGIJjPJe55kAsJ7WbuM7Bk0wBJEvSHQTMsfH7vqmKSL2Tl5jRvuIOeI/I/+FPbTT4AlFx4rehNjwoF+GPbiolTu7jfiMnKewXN39PWFl5lzZAOm154R/oK1r/ETNLRhrG4BF7rkA50P/IUVwCU5aEbn8HIUagvXqF6enfobwh8VmR21ofaT+VAVm0XqnNSv0dVFiaWBMlqpcOQiaoDPZiq0SXkxBKqTE6QU1iDK/KRIzalzUIgdk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The zspage's object is not charge to any memory cgroup.The memory of a task will have no limit if the zram swap device is large enough even though the task is limited by memory cgroup. So, it should be necessary to charge the zspage's object to obj_cgroup,just like slub. Signed-off-by: Zhongkun He --- mm/zsmalloc.c | 196 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 191 insertions(+), 5 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 02f7f414aade..a5d1c484dd8f 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -62,6 +62,7 @@ #include #include #include +#include #define ZSPAGE_MAGIC 0x58 @@ -311,6 +312,7 @@ static bool ZsHugePage(struct zspage *zspage) static void migrate_lock_init(struct zspage *zspage); static void migrate_read_lock(struct zspage *zspage); static void migrate_read_unlock(struct zspage *zspage); +static void zs_free_page_obj_cgroups(struct zspage *zspage); #ifdef CONFIG_COMPACTION static void migrate_write_lock(struct zspage *zspage); @@ -973,6 +975,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, free_handles(pool, class, zspage); next = page = get_first_page(zspage); + zs_free_page_obj_cgroups(zspage); do { VM_BUG_ON_PAGE(!PageLocked(page), page); next = get_next_page(page); @@ -1420,6 +1423,170 @@ size_t zs_huge_class_size(struct zs_pool *pool) } EXPORT_SYMBOL_GPL(zs_huge_class_size); + +#ifdef CONFIG_MEMCG_KMEM + +static inline struct obj_cgroup **page_objcgs_check(struct page *page) +{ + unsigned long memcg_data = READ_ONCE(page->memcg_data); + + if (!memcg_data || !(memcg_data & MEMCG_DATA_OBJCGS)) + return NULL; + + VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_KMEM, page); + + return (struct obj_cgroup **)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); +} + + +static void zs_free_page_obj_cgroups(struct zspage *zspage) +{ + struct obj_cgroup **objcgs; + struct page *page = zspage->first_page; + + objcgs = page_objcgs_check(page); + if (!objcgs) + return; + + kfree(objcgs); + page->memcg_data = 0; +} + +static inline size_t zs_obj_full_size(struct size_class *class) +{ + return class->size + sizeof(struct obj_cgroup *); +} + +static inline void zs_objcg_exchange(struct page *s_page, unsigned int s_index, + struct page *d_page, unsigned int d_index) +{ + struct page *sf_page, *df_page; + struct obj_cgroup **s_objcgs, **d_objcgs; + + sf_page = get_first_page(get_zspage(s_page)); + df_page = get_first_page(get_zspage(d_page)); + + s_objcgs = page_objcgs_check(sf_page); + d_objcgs = page_objcgs_check(df_page); + + if (!s_objcgs || !d_objcgs) + return; + + d_objcgs[d_index] = s_objcgs[s_index]; + s_objcgs[s_index] = NULL; +} + +void zs_alloc_obj_cgroups(struct zspage *zspage, struct size_class *class) +{ + struct page *page = zspage->first_page; + int objects = class->objs_per_zspage; + unsigned long memcg_data = 0; + void *vec; + + vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), + GFP_NOWAIT|__GFP_NOWARN| + __GFP_ZERO|__GFP_MOVABLE, + page_to_nid(page)); + + if (vec) + memcg_data = (unsigned long)vec | MEMCG_DATA_OBJCGS; + page->memcg_data = memcg_data; +} + +static inline bool zs_obj_pre_alloc_hook(struct size_class *class, + struct obj_cgroup **objcgp, gfp_t flags) +{ + struct obj_cgroup *objcg; + + *objcgp = NULL; + + objcg = get_obj_cgroup_from_current(); + if (!objcg) + return true; + + if (obj_cgroup_charge(objcg, flags, zs_obj_full_size(class))) { + obj_cgroup_put(objcg); + return false; + } + + *objcgp = objcg; + return true; +} + +static inline void zs_obj_post_alloc_hook(struct size_class *class, + struct zspage *zspage, + unsigned int idx, + struct obj_cgroup *objcg) +{ + struct obj_cgroup **objcgs = NULL; + + if (!objcg) + return; + + if (zspage) + objcgs = page_objcgs_check(zspage->first_page); + + if (!objcgs) { + obj_cgroup_uncharge(objcg, zs_obj_full_size(class)); + obj_cgroup_put(objcg); + return; + } + + objcgs[idx] = objcg; +} + +static inline void zs_obj_free_hook(int class_size, struct zspage *zspage, + unsigned int idx) +{ + struct obj_cgroup **objcgs, *objcg; + + objcgs = page_objcgs_check(zspage->first_page); + + if (WARN_ON(!objcgs)) + return; + + objcg = objcgs[idx]; + if (!objcg) + return; + + objcgs[idx] = NULL; + obj_cgroup_uncharge(objcg, class_size + sizeof(struct obj_cgroup *)); + obj_cgroup_put(objcg); +} +#else + +static void zs_free_page_obj_cgroups(struct zspage *zspage) +{ +} + +static inline void zs_objcg_exchange(struct page *s_page, unsigned int s_index, + struct page *d_page, unsigned int d_index) +{ +} + +int zs_alloc_obj_cgroups(struct zspage *zspage, struct size_class *class) +{ + return 0; +} + +static inline bool zs_obj_pre_alloc_hook(struct size_class *class, + struct obj_cgroup **objcgp, gfp_t flags) +{ + return true; +} + +static inline void zs_obj_post_alloc_hook(struct size_class *class, + struct page *page, + unsigned int idx, + struct obj_cgroup *objcg) +{ +} +static inline void zs_obj_free_hook(int class_size, struct zspage *zspage, + unsigned int idx) +{ +} +#endif + static unsigned long obj_malloc(struct zs_pool *pool, struct zspage *zspage, unsigned long handle) { @@ -1475,7 +1642,8 @@ static unsigned long obj_malloc(struct zs_pool *pool, */ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) { - unsigned long handle, obj; + unsigned long handle, obj, index; + struct obj_cgroup *objcg; struct size_class *class; int newfg; struct zspage *zspage; @@ -1491,17 +1659,23 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) size += ZS_HANDLE_SIZE; class = pool->size_class[get_size_class_index(size)]; + if (!zs_obj_pre_alloc_hook(class, &objcg, gfp)) { + cache_free_handle(pool, handle); + return (unsigned long)ERR_PTR(-ENOMEM); + } + /* pool->lock effectively protects the zpage migration */ spin_lock(&pool->lock); zspage = find_get_zspage(class); if (likely(zspage)) { + index = get_freeobj(zspage); obj = obj_malloc(pool, zspage, handle); /* Now move the zspage to another fullness group, if required */ fix_fullness_group(class, zspage); record_obj(handle, obj); class_stat_inc(class, ZS_OBJS_INUSE, 1); - goto out; + goto out_unlock; } spin_unlock(&pool->lock); @@ -1509,9 +1683,12 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) zspage = alloc_zspage(pool, class, gfp); if (!zspage) { cache_free_handle(pool, handle); - return (unsigned long)ERR_PTR(-ENOMEM); + handle = (unsigned long)ERR_PTR(-ENOMEM); + goto out; } + zs_alloc_obj_cgroups(zspage, class); + index = get_freeobj(zspage); spin_lock(&pool->lock); obj = obj_malloc(pool, zspage, handle); newfg = get_fullness_group(class, zspage); @@ -1524,7 +1701,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); -out: +out_unlock: #ifdef CONFIG_ZPOOL /* Add/move zspage to beginning of LRU */ if (!list_empty(&zspage->lru)) @@ -1533,6 +1710,8 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) #endif spin_unlock(&pool->lock); +out: + zs_obj_post_alloc_hook(class, zspage, index, objcg); return handle; } @@ -1573,7 +1752,7 @@ static void obj_free(int class_size, unsigned long obj, unsigned long *handle) f_page->index = 0; set_freeobj(zspage, f_objidx); } - + zs_obj_free_hook(class_size, zspage, f_objidx); kunmap_atomic(vaddr); mod_zspage_inuse(zspage, -1); } @@ -1640,6 +1819,7 @@ static void zs_object_copy(struct size_class *class, unsigned long dst, obj_to_location(src, &s_page, &s_objidx); obj_to_location(dst, &d_page, &d_objidx); + zs_objcg_exchange(s_page, s_objidx, d_page, d_objidx); s_off = (class->size * s_objidx) & ~PAGE_MASK; d_off = (class->size * d_objidx) & ~PAGE_MASK; @@ -1960,6 +2140,12 @@ static void replace_sub_page(struct size_class *class, struct zspage *zspage, int idx = 0; page = get_first_page(zspage); +#ifdef CONFIG_MEMCG_KMEM + if (page == oldpage) { + newpage->memcg_data = page->memcg_data; + page->memcg_data = 0; + } +#endif do { if (page == oldpage) pages[idx] = newpage;