From patchwork Tue Jun 21 12:56:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12889209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6EFAC433EF for ; Tue, 21 Jun 2022 12:58:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 70BD28E0003; Tue, 21 Jun 2022 08:58:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 694076B0074; Tue, 21 Jun 2022 08:58:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 44AE68E0003; Tue, 21 Jun 2022 08:58:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3485C6B0072 for ; Tue, 21 Jun 2022 08:58:23 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 19BD960E6B for ; Tue, 21 Jun 2022 12:58:23 +0000 (UTC) X-FDA: 79602246486.29.72F70FC Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by imf10.hostedemail.com (Postfix) with ESMTP id 719E9C0015 for ; Tue, 21 Jun 2022 12:58:22 +0000 (UTC) Received: by mail-pg1-f178.google.com with SMTP id f65so13048716pgc.7 for ; Tue, 21 Jun 2022 05:58:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=R9q0ch1BG5yy3uNoTKVJadkT0JjG8vQh5rsCMZAiDeM=; b=X1ymgAzrECJrNEqT0zu/qYqioj+MK14RMF+IxcAS9y0uCpcuMThML5qsMfwkh5Syqb a+aAQz+na0wNKw5ONdOQH1YzYWTCg9aI9VoZg1AiFiGcFm5GNrdmqoniS/jFYAWZB9U0 x7hnUcvnD2Uwvu2dL/27+ftKyQYgBnzgc38FLw8CWdGOZanoyKBMLvP036HHHz7f77de /57RM1HmPaKRHnE6aN43rnTmSuQnUSdoWtTVyMJC9jbL9ijPukp0Xg3q3OywkZdjchvG /bQ8emT1SlpJguuihXT//Xhcgv32+AVxjjynF7sfuZCkEaISt9UJisvUrCMCsdduD4Ug QLPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=R9q0ch1BG5yy3uNoTKVJadkT0JjG8vQh5rsCMZAiDeM=; b=pyYlW/DpfVAWRyLOJno7ShsBQJoCWp3/RUfSOsdTl2/rraNITdJSaqRo/Fkw9IGDRa FzBH+PeSpIE1H9H7ILk70bNqetk6oY/3QPNumhA4Krf/SFtYpKJ7xfPagVIAGUDbTM/A 1yLorMw6EfMzDbO9ukBXtRMbftwWv6pk+oAMf7FbgL62iFwN59HIE7DbIHa0kqqjnet5 P2+Kpnp2/mms359LwI4zjJCsEG8WIKdZlY9VBSVL8OKMaxJh3xlJl6/YlAmJkDVpoAwh yw77bBCEtDPbJTxG7hEPG4AxqjJWAQ6GY8dPmSnMnP+RlMUE+0y/0lk69H6F7brXTNq4 b90Q== X-Gm-Message-State: AJIora+U74ZWIEI4RC00MnpcqLQgsVrrwTi5F6p8ISeVGyG7iOKpz8F6 9yvztBXRE8E8wQwYcENSjoBJGg== X-Google-Smtp-Source: AGRyM1vJmCnTypP6KX9NHCb9AyM5aDswtoVUk0Wth7I38QLP7dlJmEjPt2tfj2n1yyM2/5DwhBTAJQ== X-Received: by 2002:a63:5155:0:b0:400:14af:a1e6 with SMTP id r21-20020a635155000000b0040014afa1e6mr25830864pgl.71.1655816301516; Tue, 21 Jun 2022 05:58:21 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id e3-20020a170903240300b0015ea3a491a1sm10643134plo.191.2022.06.21.05.58.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 05:58:21 -0700 (PDT) From: Muchun Song To: akpm@linux-foundation.org, hannes@cmpxchg.org, longman@redhat.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, duanxiongchun@bytedance.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v6 08/11] mm: memcontrol: introduce memcg_reparent_ops Date: Tue, 21 Jun 2022 20:56:55 +0800 Message-Id: <20220621125658.64935-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220621125658.64935-1-songmuchun@bytedance.com> References: <20220621125658.64935-1-songmuchun@bytedance.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655816302; a=rsa-sha256; cv=none; b=JEjF98vhlSoXgZGpIAusVuLLbHfruV0n3LradDSTge4AbNziVlERBJJxQqED/wXHZZhyjc vwO1Lt5YinnEniCtWQ3abXDvdCJcG/AqNIkeWlgoIIbeYT2Vh1H0pp5RBNmfCqCbJGYNsh L+LFF1iLdk8VPssmuYlq4sqNhbrWdgk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655816302; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R9q0ch1BG5yy3uNoTKVJadkT0JjG8vQh5rsCMZAiDeM=; b=578LoFQxoFy4LreaIJUTnP7xJfkB+OGLIYCFkL+j5a3pS9rdjK/69OHiiYb5sDKaRPD8uk TnCQR0jh/hdqpSeOPvEy5LdAH3EexNKVWtao/ibAn5n4rV+lNywX4pxz3EnIfzaVg4blMe KbUBFpyjhmVAxGjN9keb/tWMSMs6iFs= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=X1ymgAzr; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf10.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.178 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=X1ymgAzr; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf10.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.178 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 719E9C0015 X-Stat-Signature: 4bhbupsepjd8di7c378md53msipjyr37 X-HE-Tag: 1655816302-371158 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the previous patch, we know how to make the lruvec lock safe when LRU pages are reparented. We should do something like following. memcg_reparent_objcgs(memcg) 1) lock // lruvec belongs to memcg and lruvec_parent belongs to parent memcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); 2) relocate from current memcg to its parent // Move all the pages from the lruvec list to the parent lruvec list. 3) unlock spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); Apart from the page lruvec lock, the deferred split queue lock (THP only) also needs to do something similar. So we extract the necessary three steps in the memcg_reparent_objcgs(). memcg_reparent_objcgs(memcg) 1) lock memcg_reparent_ops->lock(memcg, parent); 2) relocate memcg_reparent_ops->relocate(memcg, reparent); 3) unlock memcg_reparent_ops->unlock(memcg, reparent); Now there are two different locks (e.g. lruvec lock and deferred split queue lock) need to use this infrastructure. In the next patch, we will use those APIs to make those locks safe when the LRU pages reparented. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 20 +++++++++++++++ mm/memcontrol.c | 62 ++++++++++++++++++++++++++++++++++++---------- 2 files changed, 69 insertions(+), 13 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index a8bd4bb39502..63dbdef60cbd 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -353,6 +353,26 @@ struct mem_cgroup { struct mem_cgroup_per_node *nodeinfo[]; }; +struct memcg_reparent_ops { + /* + * Note that interrupt is disabled before calling those callbacks, + * so the interrupt should remain disabled when leaving those callbacks. + */ + void (*lock)(struct mem_cgroup *src, struct mem_cgroup *dst); + void (*relocate)(struct mem_cgroup *src, struct mem_cgroup *dst); + void (*unlock)(struct mem_cgroup *src, struct mem_cgroup *dst); +}; + +#define DEFINE_MEMCG_REPARENT_OPS(name) \ + const struct memcg_reparent_ops memcg_##name##_reparent_ops = { \ + .lock = name##_reparent_lock, \ + .relocate = name##_reparent_relocate, \ + .unlock = name##_reparent_unlock, \ + } + +#define DECLARE_MEMCG_REPARENT_OPS(name) \ + extern const struct memcg_reparent_ops memcg_##name##_reparent_ops + /* * size of first charge trial. "32" comes from vmscan.c's magic value. * TODO: maybe necessary to use big numbers in big irons. diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 346a954e190e..6ef3a264054e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -335,24 +335,60 @@ static struct obj_cgroup *obj_cgroup_alloc(void) return objcg; } -static void memcg_reparent_objcgs(struct mem_cgroup *memcg) +static void objcg_reparent_lock(struct mem_cgroup *src, struct mem_cgroup *dst) +{ + spin_lock(&objcg_lock); +} + +static void objcg_reparent_relocate(struct mem_cgroup *src, struct mem_cgroup *dst) { struct obj_cgroup *objcg, *iter; - struct mem_cgroup *parent = parent_mem_cgroup(memcg); - objcg = rcu_replace_pointer(memcg->objcg, NULL, true); + objcg = rcu_replace_pointer(src->objcg, NULL, true); + /* 1) Ready to reparent active objcg. */ + list_add(&objcg->list, &src->objcg_list); + /* 2) Reparent active objcg and already reparented objcgs to dst. */ + list_for_each_entry(iter, &src->objcg_list, list) + WRITE_ONCE(iter->memcg, dst); + /* 3) Move already reparented objcgs to the dst's list */ + list_splice(&src->objcg_list, &dst->objcg_list); +} + +static void objcg_reparent_unlock(struct mem_cgroup *src, struct mem_cgroup *dst) +{ + spin_unlock(&objcg_lock); +} - spin_lock_irq(&objcg_lock); +static DEFINE_MEMCG_REPARENT_OPS(objcg); - /* 1) Ready to reparent active objcg. */ - list_add(&objcg->list, &memcg->objcg_list); - /* 2) Reparent active objcg and already reparented objcgs to parent. */ - list_for_each_entry(iter, &memcg->objcg_list, list) - WRITE_ONCE(iter->memcg, parent); - /* 3) Move already reparented objcgs to the parent's list */ - list_splice(&memcg->objcg_list, &parent->objcg_list); - - spin_unlock_irq(&objcg_lock); +static const struct memcg_reparent_ops *memcg_reparent_ops[] = { + &memcg_objcg_reparent_ops, +}; + +#define DEFINE_MEMCG_REPARENT_FUNC(phase) \ + static void memcg_reparent_##phase(struct mem_cgroup *src, \ + struct mem_cgroup *dst) \ + { \ + int i; \ + \ + for (i = 0; i < ARRAY_SIZE(memcg_reparent_ops); i++) \ + memcg_reparent_ops[i]->phase(src, dst); \ + } + +DEFINE_MEMCG_REPARENT_FUNC(lock) +DEFINE_MEMCG_REPARENT_FUNC(relocate) +DEFINE_MEMCG_REPARENT_FUNC(unlock) + +static void memcg_reparent_objcgs(struct mem_cgroup *src) +{ + struct mem_cgroup *dst = parent_mem_cgroup(src); + struct obj_cgroup *objcg = rcu_dereference_protected(src->objcg, true); + + local_irq_disable(); + memcg_reparent_lock(src, dst); + memcg_reparent_relocate(src, dst); + memcg_reparent_unlock(src, dst); + local_irq_enable(); percpu_ref_kill(&objcg->refcnt); }