From patchwork Wed Oct 4 19:36:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13409406 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 940EFE7C4EF for ; Wed, 4 Oct 2023 19:36:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BABAC6B01A7; Wed, 4 Oct 2023 15:36:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B34636B01A8; Wed, 4 Oct 2023 15:36:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9AD766B01A9; Wed, 4 Oct 2023 15:36:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 870636B01A7 for ; Wed, 4 Oct 2023 15:36:26 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 52882B414D for ; Wed, 4 Oct 2023 19:36:26 +0000 (UTC) X-FDA: 81308785572.28.F261930 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf01.hostedemail.com (Postfix) with ESMTP id 6ED7B40013 for ; Wed, 4 Oct 2023 19:36:24 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bLwjmvX1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696448184; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=Oxe/lUUklxbHu0vEIlUOVm4PBVuobqclnUB653St8qw=; b=Na5HymcKiZVEn0bvsTp9K/cH8C+ZrNR5rIIUDxlvepCSoPZIx1MkF/LyvMzL/ziRpPGmW7 Q3u7hr53AId5lPY0cEARcuyAek2AZw5gGAbJ384bQG+EiBjXjwI99sYkfqEhynsER/zbwH hRBe3jskWKjNpXFvEk8dMR0eXRtnvZo= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bLwjmvX1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696448184; a=rsa-sha256; cv=none; b=AZfymS//Am2z5fQahDmM6E6SvN4g+esSCVPdzgzEaBMqKYSY1VdR91L/Xgc6PqF8739RlP AwOYB6uikvgZeTl+81TmfCtAy1YtNZkuASeBN/bc9Dv2C6Qe6L8u1dAB8JE0S+bd1bUP/x cYB0hN0XX7xd/HxbPESY0BX9zVaqqko= Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-2776d882214so110315a91.2 for ; Wed, 04 Oct 2023 12:36:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696448183; x=1697052983; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Oxe/lUUklxbHu0vEIlUOVm4PBVuobqclnUB653St8qw=; b=bLwjmvX1aTv9ZhvxG7NaX62hohsY1TT8e19qiZ1EEuLO+Zw+RztuzwohGXaZJbzXbl PUp5BtvEq3WKLEmtErFuSzDzqGFnpLz/vp/vj9iU6KNrHZtY7B+pAPP9Lt9wA2kjhxr4 I+3Eo6Feiz/BE32BWiYRs6BpFI1SIQSEo2n2TMO1pUR+6f89MsF4yrEcw5H+87DaokGw Ks5PxkoGsfVtl1q0muWCo34qCXqdX2lZ4WlQU9Z3Cfq+MIUwKl6tyeFK7GNG/C+PXyJy G5mbLPPIebiZ8Fjddjg49xIQu/EHpn5Yi0AVEEz2FSchdxbS+YUtRg0yjMUsllc4Q+/H xeMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696448183; x=1697052983; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Oxe/lUUklxbHu0vEIlUOVm4PBVuobqclnUB653St8qw=; b=wMpsIPO5Rs1nEwjzrzOtS910Ar5sMH8Kqu+jW17zqnShsnd0qNquM8IO0kESMXdzdr nd4golf5fLgCRC/oIv0EG3Jx4vHXjJNwHcMK02iAu1cE3m/0ezzRVUbhc6WaWZunMTDe H2zcLPpWiUkskdkATJET7tMtsLVULspTKvOVxoW0Z7VKwnCpGVfoLoOW9RuxwQy5IDNz vtPh1l6N5+zInDZFy7NNElOxCnaH4+reS7STnb1m1N1zZ7EBu+7eH6W0Dqp+7og/vKtM jLu7hT5K+jaPtWMW1NPxdUv3zxg+HX/9a+qtp8cOCscIbUvwYjauOJqpE8EGxiD/dCXo gruA== X-Gm-Message-State: AOJu0YwTmntj8uONPmk+kaSomF7gsa4+JCZJqxdr6bHhQHHP4Y7aDMCG rCbGtaanRR8eKNkgsCCl/Hg= X-Google-Smtp-Source: AGHT+IGC2IF1Rwb6lyP1BbCO/xIxVJzbxzbXytefjWzvm4+Km4BvCbHu0QkrrRljkhcMchfRMGJoEA== X-Received: by 2002:a17:90a:ce92:b0:274:6135:8363 with SMTP id g18-20020a17090ace9200b0027461358363mr2908761pju.33.1696448183010; Wed, 04 Oct 2023 12:36:23 -0700 (PDT) Received: from localhost (fwdproxy-prn-113.fbsv.net. [2a03:2880:ff:71::face:b00c]) by smtp.gmail.com with ESMTPSA id bv6-20020a17090af18600b002630c9d78aasm1961226pjb.5.2023.10.04.12.36.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 12:36:22 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: riel@surriel.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, tj@kernel.org, lizefan.x@bytedance.com, shuah@kernel.org, mike.kravetz@oracle.com, yosryahmed@google.com, fvdl@google.com, linux-mm@kvack.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH v2] memcontrol: only transfer the memcg data for migration Date: Wed, 4 Oct 2023 12:36:22 -0700 Message-Id: <20231004193622.900383-1-nphamcs@gmail.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 6ED7B40013 X-Stat-Signature: hwy6eymxdmhusbxb9pt9kb51ffg3ot6c X-HE-Tag: 1696448184-834170 X-HE-Meta: U2FsdGVkX1+W8rDXuhwBXpB7AfNzZZ3NZE8fKvSCAOOMON3+qyGbelxQRci7jwI2mKxpUrOnTqqsOVRq1BjR4WAdpjqkHoDcA6QwzqXk/UrD6ODQV5ESwBYQP0KEAet3vsI9A2mncl/+pSjo1POQHRm7Bt/SsBTJ+qp8KkAh8zCCll1cwyPxg+4ECtmuLF8rDEHNkrEwScyhMSY0vx9xCYd5f5szuBsLuQR9NtZv7wntpxhZONX1hhJA9X+JDqM84LoogjxOU8u5ylnoc3TYpfu/tH6N5qy91EJIjq8GwoDeU02q4T8X6p4iKnWK2KjdlZufOjA/pI7vcM8qXeewICiiCkxGW4YY5+kNy25SJkk+U6R7Li2tlYf8cPVdsB4BG7vccVcPv6HuNniUG3h+3ZYH3nMEu6/KKjqHqzNY//clRDAqaWIMfaYmJZRDr5kljc7dtFyKTkpNgsmXn8NRnnJoybVA/6WTEK9ah5XLjadsLa/ruA9yGucSyHNEHFYiPOgMCIPPXQJHJTaVf2NVk2ylLvqCLLTOU8h3ViTljZnzkq79tV466UecV3YVzoKSraZgm6JYj4E+Ec917GXf7P1Q81eYy80cNmY2mK9jPRddq88g4H7y8BRJJBQAsN0C23nrGdGJNtFtN7DCBrG474qE281yELhdBN5+Rxfpq1xIExtUpbqc6I+rapV+klT6vV3HIWnadtF6O6ZX06CYL6gZPog9pJ22YlSHQC/ZVlVapKKD8MT3p4tQNtMYmf+8aPtscAc30ffopduiBUJOcBfteXHbLAaQ1vKUvFob8DsD7CwwCbz53CMmsvTRAeGUDXztdYKbks6YJez0Vj9vt24Hzij8mkXhBWacZdWKG2K5ClBs9LSwcPJkYVXWP/aVBwT01dPlvIhr9uq5iTbpw06nQ6Gc4unG8vKgAIy8HthrD1Ik8WPGp7t+Sm0Ac8oU/ZJdhH66OJGyEA2D8zF 0R1PHQJB fdTz/keQSU0u2goGfcLYPbA9am59sLLZf8A6GB6Z9Q6cYz0LDymorr7p1oCS1IbDV+bnfDM0osin4fUDXOZXW8VaEATsdqZWXGwz9Ydyc/Rfd0i/9LYKqlEln33UfBF5TLhNv4r/ST6D/AKuDxThnHfq4aUVkORmp+3aH+sNNcbQxUlPkDuivX3KEcPEG91IsJUmwa6PbSiEUZUgLD0tJ5QbjR9zuB8Wg1nraukayZiJ9yHkzi/6kooOzaNL2jxoGsTjPY3NvdINNNF33bwpvpqdLmb1HGszo+kkGkChadKoj+9plPp4Fqun7L4qP+WXGUAdCotwPTxgWcler1O5YHSRwAHVRigB52nQrPC/E60/sIK+7qE83qy1GoVQTdBlcr59BVGtTm1r5n4TK0Gpc6gyT5tW3eGT/rWSNYGYnA2iZzoqnPbnUJI6+3zeuNia75Td2/C5uIW4J/gsxvATNQuuEwGV+7Kzj1NY2fSRYxXBQhWrwDsGTxC7Q7v9zXp6I6fKndNkLjgw9ua/IVxyGrANYdw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For most migration use cases, only transfer the memcg data from the old folio to the new folio, and clear the old folio's memcg data. No charging and uncharging will be done. This shaves off some work on the migration path, and avoids the temporary double charging of a folio during its migration. The only exception is replace_page_cache_folio(), which will use the old mem_cgroup_migrate() (now renamed to mem_cgroup_replace_folio). In that context, the isolation of the old page isn't quite as thorough as with migration, so we cannot use our new implementation directly. This patch is the result of the following discussion on the new hugetlb memcg accounting behavior: https://lore.kernel.org/lkml/20231003171329.GB314430@monkey/ This should be added as the second prep patch in the following series: https://lore.kernel.org/all/20231003001828.2554080-1-nphamcs@gmail.com/ (hugetlb memcg accounting) and should go right before the following patch: hugetlb: memcg: account hugetlb-backed memory in memory controller Reported-by: Mike Kravetz Closes: https://lore.kernel.org/lkml/20231003171329.GB314430@monkey/ Suggested-by: Johannes Weiner Signed-off-by: Nhat Pham Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 7 ++++++ mm/filemap.c | 2 +- mm/memcontrol.c | 45 +++++++++++++++++++++++++++++++++++--- 3 files changed, 50 insertions(+), 4 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 42bf7e9b1a2f..5daf14da3759 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -708,6 +708,8 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages); +void mem_cgroup_replace_folio(struct folio *old, struct folio *new); + void mem_cgroup_migrate(struct folio *old, struct folio *new); /** @@ -1285,6 +1287,11 @@ static inline void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, { } +static inline void mem_cgroup_replace_folio(struct folio *old, + struct folio *new) +{ +} + static inline void mem_cgroup_migrate(struct folio *old, struct folio *new) { } diff --git a/mm/filemap.c b/mm/filemap.c index 9481ffaf24e6..673745219c82 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -819,7 +819,7 @@ void replace_page_cache_folio(struct folio *old, struct folio *new) new->mapping = mapping; new->index = offset; - mem_cgroup_migrate(old, new); + mem_cgroup_replace_folio(old, new); xas_lock_irq(&xas); xas_store(&xas, new); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0219befeae38..2ccf0a53cd71 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7281,16 +7281,17 @@ void __mem_cgroup_uncharge_list(struct list_head *page_list) } /** - * mem_cgroup_migrate - Charge a folio's replacement. + * mem_cgroup_replace_folio - Charge a folio's replacement. * @old: Currently circulating folio. * @new: Replacement folio. * * Charge @new as a replacement folio for @old. @old will - * be uncharged upon free. + * be uncharged upon free. This is only used by the page cache + * (in replace_page_cache_folio()). * * Both folios must be locked, @new->mapping must be set up. */ -void mem_cgroup_migrate(struct folio *old, struct folio *new) +void mem_cgroup_replace_folio(struct folio *old, struct folio *new) { struct mem_cgroup *memcg; long nr_pages = folio_nr_pages(new); @@ -7329,6 +7330,44 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new) local_irq_restore(flags); } +/** + * mem_cgroup_migrate - Transfer the memcg data from the old to the new folio. + * @old: Currently circulating folio. + * @new: Replacement folio. + * + * Transfer the memcg data from the old folio to the new folio for migration. + * The old folio's data info will be cleared. Note that the memory counters + * will remain unchanged throughout the process. + * + * Both folios must be locked, @new->mapping must be set up. + */ +void mem_cgroup_migrate(struct folio *old, struct folio *new) +{ + struct mem_cgroup *memcg; + + VM_BUG_ON_FOLIO(!folio_test_locked(old), old); + VM_BUG_ON_FOLIO(!folio_test_locked(new), new); + VM_BUG_ON_FOLIO(folio_test_anon(old) != folio_test_anon(new), new); + VM_BUG_ON_FOLIO(folio_nr_pages(old) != folio_nr_pages(new), new); + + if (mem_cgroup_disabled()) + return; + + memcg = folio_memcg(old); + /* + * Note that it is normal to see !memcg for a hugetlb folio. + * For e.g, itt could have been allocated when memory_hugetlb_accounting + * was not selected. + */ + VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(old) && !memcg, old); + if (!memcg) + return; + + /* Transfer the charge and the css ref */ + commit_charge(new, memcg); + old->memcg_data = 0; +} + DEFINE_STATIC_KEY_FALSE(memcg_sockets_enabled_key); EXPORT_SYMBOL(memcg_sockets_enabled_key);