From patchwork Thu Sep 16 13:47:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12499135 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56BFBC433EF for ; Thu, 16 Sep 2021 13:53:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0CC9E60EB4 for ; Thu, 16 Sep 2021 13:53:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 0CC9E60EB4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id A7DE36B0080; Thu, 16 Sep 2021 09:53:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A2E0C6B0081; Thu, 16 Sep 2021 09:53:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F555900002; Thu, 16 Sep 2021 09:53:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0206.hostedemail.com [216.40.44.206]) by kanga.kvack.org (Postfix) with ESMTP id 7F5606B0080 for ; Thu, 16 Sep 2021 09:53:04 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 8DC742FD96 for ; Thu, 16 Sep 2021 13:53:03 +0000 (UTC) X-FDA: 78593577846.28.BF68F3F Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf17.hostedemail.com (Postfix) with ESMTP id 55D9FF0003A4 for ; Thu, 16 Sep 2021 13:53:03 +0000 (UTC) Received: by mail-pf1-f171.google.com with SMTP id y17so5899479pfl.13 for ; Thu, 16 Sep 2021 06:53:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9zJ+rajzCodDQWeoI37cULUx5xfm+EYhpR+j50QMba8=; b=UH0q2kMvwpE+oXiVGNhyaG/DpfCTcD3yJP4RI4OdyNI3bcpaFQG+yzqb8NVQKg1DpN DOUPe7YQUg3P01t3908CUfgU/X1z6/ooubNFka5e13rDo+0bEhnFU/wbaqU6bTOP2sjj ARz2nxJHBlGoK7BXo6ZKwbwWkMHew6yfg7dmmPwtmSmOxnVwzGTs4hVBSAcnaAqu3Bw3 dFopBR5Z7ZNZTkQxdsyqoEBiicslqlwytz0mt3/JfKVOkrmmuX82jNFNyDBYuLmvAikv U/COC/LXn5akRPaFXjlFHPLYnCbAj5jOLctJSqtysE8hPp08hvYBr7Ujy+BO89yjik3b 9UTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9zJ+rajzCodDQWeoI37cULUx5xfm+EYhpR+j50QMba8=; b=RAWxfjYktSf76Fb5sPeo5vj3vj062TGd4/7zGGcViG0cqNnV/GKBHXwnAJTvwM5s5I kaEXnvXwVOK55SBRhY3A5MCCgbSgzdNN9KwBzwV337sCKDS0H2aa0eAvrgnqPmxhry7P qa+6py8yPkWNN8fgnIRvolUks3AsCAQ8wP1uwKOSJ9BaitfTIP3NnzBQqafiQOfFGqq+ jAh4pCV/oiYLzfkHOCmA5Dsv750GQ0REThLBIl6nb1fyl5ii6ZtDozTt3EcxfxOHuFjg bRqoDJrWQjVmYv8daOSz1vBAHFZmZ1kQf+PyTV1iTontcL4rtMzJzkIv3I3+J+S50i/B pOzg== X-Gm-Message-State: AOAM5300xpEPIqR3VHgtO+i8uJFpJpq1p+O4TX4qof3ufR/tnz75MJ0n uI5G1aWDDbuFlMDv+RW8gDgNpg== X-Google-Smtp-Source: ABdhPJwxGZVjBFXnUvLHMSj361dGfdo0pyqFzMu2FSl2ZkHX6t6WAMc0bR0oqwQWyHCKbeJyZ6uePw== X-Received: by 2002:a63:da14:: with SMTP id c20mr5099842pgh.155.1631800382477; Thu, 16 Sep 2021 06:53:02 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.226]) by smtp.gmail.com with ESMTPSA id o9sm3617443pfh.217.2021.09.16.06.52.56 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 16 Sep 2021 06:53:02 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alexs@kernel.org, smuchun@gmail.com, zhengqi.arch@bytedance.com, Muchun Song Subject: [PATCH v2 09/13] mm: memcontrol: introduce memcg_reparent_ops Date: Thu, 16 Sep 2021 21:47:44 +0800 Message-Id: <20210916134748.67712-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210916134748.67712-1-songmuchun@bytedance.com> References: <20210916134748.67712-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 55D9FF0003A4 X-Stat-Signature: radchxiberpjokn3kc6dt8tro3f4ku5b Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=UH0q2kMv; spf=pass (imf17.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-HE-Tag: 1631800383-358357 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the previous patch, we know how to make the lruvec lock safe when the LRU pages reparented. We should do something like following. memcg_reparent_objcgs(memcg) 1) lock // lruvec belongs to memcg and lruvec_parent belongs to parent memcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); 2) do reparent // Move all the pages from the lruvec list to the parent lruvec list. 3) unlock spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); Apart from the page lruvec lock, the deferred split queue lock (THP only) also needs to do something similar. So we extract the necessary three steps in the memcg_reparent_objcgs(). memcg_reparent_objcgs(memcg) 1) lock memcg_reparent_ops->lock(memcg, parent); 2) reparent memcg_reparent_ops->reparent(memcg, reparent); 3) unlock memcg_reparent_ops->unlock(memcg, reparent); Now there are two different locks (e.g. lruvec lock and deferred split queue lock) need to use this infrastructure. In the next patch, we will use those APIs to make those locks safe when the LRU pages reparented. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 7 +++++++ mm/memcontrol.c | 43 +++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 48 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ab3cd844e91d..18344c1f4333 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -350,6 +350,13 @@ struct mem_cgroup { struct mem_cgroup_per_node *nodeinfo[]; }; +struct memcg_reparent_ops { + /* Irq is disabled before calling those callbacks. */ + void (*lock)(struct mem_cgroup *memcg, struct mem_cgroup *parent); + void (*unlock)(struct mem_cgroup *memcg, struct mem_cgroup *parent); + void (*reparent)(struct mem_cgroup *memcg, struct mem_cgroup *parent); +}; + /* * size of first charge trial. "32" comes from vmscan.c's magic value. * TODO: maybe necessary to use big numbers in big irons. diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 16db5b39cb81..3a73fd192734 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -333,6 +333,35 @@ static struct obj_cgroup *obj_cgroup_alloc(void) return objcg; } +static const struct memcg_reparent_ops *memcg_reparent_ops[] = {}; + +static void memcg_reparent_lock(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(memcg_reparent_ops); i++) + memcg_reparent_ops[i]->lock(memcg, parent); +} + +static void memcg_reparent_unlock(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(memcg_reparent_ops); i++) + memcg_reparent_ops[i]->unlock(memcg, parent); +} + +static void memcg_do_reparent(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(memcg_reparent_ops); i++) + memcg_reparent_ops[i]->reparent(memcg, parent); +} + static void memcg_reparent_objcgs(struct mem_cgroup *memcg) { struct obj_cgroup *objcg, *iter; @@ -342,9 +371,13 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg) if (!parent) parent = root_mem_cgroup; + local_irq_disable(); + + memcg_reparent_lock(memcg, parent); + objcg = rcu_replace_pointer(memcg->objcg, NULL, true); - spin_lock_irq(&css_set_lock); + spin_lock(&css_set_lock); /* 1) Ready to reparent active objcg. */ list_add(&objcg->list, &memcg->objcg_list); @@ -354,7 +387,13 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg) /* 3) Move already reparented objcgs to the parent's list */ list_splice(&memcg->objcg_list, &parent->objcg_list); - spin_unlock_irq(&css_set_lock); + spin_unlock(&css_set_lock); + + memcg_do_reparent(memcg, parent); + + memcg_reparent_unlock(memcg, parent); + + local_irq_enable(); percpu_ref_kill(&objcg->refcnt); }