From patchwork Sat Aug 14 05:25:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12436503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E41EBC4338F for ; Sat, 14 Aug 2021 05:26:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 96ACC60F00 for ; Sat, 14 Aug 2021 05:26:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 96ACC60F00 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 3D6CA8D000A; Sat, 14 Aug 2021 01:26:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3879B8D0002; Sat, 14 Aug 2021 01:26:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24EE88D000A; Sat, 14 Aug 2021 01:26:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0192.hostedemail.com [216.40.44.192]) by kanga.kvack.org (Postfix) with ESMTP id 0956B8D0002 for ; Sat, 14 Aug 2021 01:26:21 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A48758249980 for ; Sat, 14 Aug 2021 05:26:20 +0000 (UTC) X-FDA: 78472550520.30.81BE118 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf12.hostedemail.com (Postfix) with ESMTP id 5A1D9100089E for ; Sat, 14 Aug 2021 05:26:20 +0000 (UTC) Received: by mail-pl1-f175.google.com with SMTP id d1so14716755pll.1 for ; Fri, 13 Aug 2021 22:26:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fPIEUgCv2CfVB3RBhBUqutTabZnAlwkc6TmF8u0la+M=; b=evzFsEQ5cP5VAHIL/eesP/y9sy4kvCZ9J4vPwMM2WZeM3i2WpFmYbgXqFhi7udyynW 1aMUdu+vZBYPWaxURBSa7XvTgGwbtY2Yti0mSH1AAERVh8UGsXqrKAKvcvMj2UnJJQ+a JXOPRC9/lmWRFRzm2zDJac+b32GcJJmKA64coKjcw4dkIOsFQL7IGMMaX7O1/Ay+jfqa to8CPnjiQh/hSZjI3V0a8RTln46tcbOf+p/S72IZunpbIOzatgr0PJFm0XxBXm11xIr8 osFk2afW32RSq/oNKn00qTjBLBWtlMZYo4XNh5NjDYGIbPRh/nkRSYmm+Moiv2c+QijM 7rVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fPIEUgCv2CfVB3RBhBUqutTabZnAlwkc6TmF8u0la+M=; b=lY0Qcb04kmZW9Ace44OKfi9lpfrsD47fHn6/WUi55ijN1mofLdG40q9Dm6A7cfWzwW 9hGd4XHb0M7GtfUcAHhoNFDg1Q7AAU+jXA1wOy5WrV2GOwXUOfcHbfqvFkqQKt/CYOzu qBd0eQiioxSEM9ZtgHdPYXNtUf13CICT/TWYD8Velrefx5qgyTPpFF7em6SiRS6Nyl9U Z4Ap8DC+edILRrJGSYn/5cgCegqm8wS4+/U/mCacpZY8w0wj28T3kdBQG8Pa/+60SwiV ulMx7Z+Y/Nr1PtdAauhFOx42XiHKntAVnscC+V9Im4UlHh3c99zmeCpFmpZ5AFrh9tky ftgQ== X-Gm-Message-State: AOAM532udvLMEq1VzQmCaB1HaXKYWOAnUsGtEBYCGlCNE9rtq+H0RK+b euk0Y16aopQZl6YbEuoYrJqnLQ== X-Google-Smtp-Source: ABdhPJzxwFjAn1im6gsgSEHt+LwD4iGMz1KIVynJ4deShVJLjmTgUF214h03/eIQOp5o9YqHY/X2sA== X-Received: by 2002:a17:90a:c087:: with SMTP id o7mr5996988pjs.57.1628918779474; Fri, 13 Aug 2021 22:26:19 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.237]) by smtp.gmail.com with ESMTPSA id s5sm4783133pgp.81.2021.08.13.22.26.13 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Aug 2021 22:26:19 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alexs@kernel.org, smuchun@gmail.com, zhengqi.arch@bytedance.com, Muchun Song Subject: [PATCH v1 08/12] mm: memcontrol: introduce memcg_reparent_ops Date: Sat, 14 Aug 2021 13:25:15 +0800 Message-Id: <20210814052519.86679-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210814052519.86679-1-songmuchun@bytedance.com> References: <20210814052519.86679-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5A1D9100089E Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=evzFsEQ5; spf=pass (imf12.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam01 X-Stat-Signature: yibmfn73jeubccqb19chqmqeu1a5rydj X-HE-Tag: 1628918780-781456 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the previous patch, we know how to make the lruvec lock safe when LRU pages are reparented. We should do something like following. memcg_reparent_objcgs(memcg) 1) lock // lruvec belongs to memcg and lruvec_parent belongs to parent memcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); 2) do reparent // Move all the pages from the lruvec list to the parent lruvec list. 3) unlock spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); Apart from the page lruvec lock, the deferred split queue lock (THP only) also needs to do something similar. So we extract the necessary three steps in the memcg_reparent_objcgs(). memcg_reparent_objcgs(memcg) 1) lock memcg_reparent_ops->lock(memcg, parent); 2) reparent memcg_reparent_ops->reparent(memcg, reparent); 3) unlock memcg_reparent_ops->unlock(memcg, reparent); Now there are two different locks (e.g. lruvec lock and deferred split queue lock) need to use this infrastructure. In the next patch, we will use those APIs to make those locks safe when the LRU pages reparented. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 7 +++++++ mm/memcontrol.c | 43 +++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 48 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 431fc606f6f9..d85c03f2d76d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -352,6 +352,13 @@ struct mem_cgroup { struct mem_cgroup_per_node *nodeinfo[]; }; +struct memcg_reparent_ops { + /* Irq is disabled before calling those callbacks. */ + void (*lock)(struct mem_cgroup *memcg, struct mem_cgroup *parent); + void (*unlock)(struct mem_cgroup *memcg, struct mem_cgroup *parent); + void (*reparent)(struct mem_cgroup *memcg, struct mem_cgroup *parent); +}; + /* * size of first charge trial. "32" comes from vmscan.c's magic value. * TODO: maybe necessary to use big numbers in big irons. diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0eca3cf6cede..8cfb4221c36a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -344,6 +344,35 @@ static struct obj_cgroup *obj_cgroup_alloc(void) return objcg; } +static const struct memcg_reparent_ops *memcg_reparent_ops[] = {}; + +static void memcg_reparent_lock(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(memcg_reparent_ops); i++) + memcg_reparent_ops[i]->lock(memcg, parent); +} + +static void memcg_reparent_unlock(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(memcg_reparent_ops); i++) + memcg_reparent_ops[i]->unlock(memcg, parent); +} + +static void memcg_do_reparent(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(memcg_reparent_ops); i++) + memcg_reparent_ops[i]->reparent(memcg, parent); +} + static void memcg_reparent_objcgs(struct mem_cgroup *memcg) { struct obj_cgroup *objcg, *iter; @@ -353,9 +382,13 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg) if (!parent) parent = root_mem_cgroup; + local_irq_disable(); + + memcg_reparent_lock(memcg, parent); + objcg = rcu_replace_pointer(memcg->objcg, NULL, true); - spin_lock_irq(&css_set_lock); + spin_lock(&css_set_lock); /* 1) Ready to reparent active objcg. */ list_add(&objcg->list, &memcg->objcg_list); @@ -365,7 +398,13 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg) /* 3) Move already reparented objcgs to the parent's list */ list_splice(&memcg->objcg_list, &parent->objcg_list); - spin_unlock_irq(&css_set_lock); + spin_unlock(&css_set_lock); + + memcg_do_reparent(memcg, parent); + + memcg_reparent_unlock(memcg, parent); + + local_irq_enable(); percpu_ref_kill(&objcg->refcnt); }