From patchwork Wed Sep 25 17:10:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13812339 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70897CCF9E7 for ; Wed, 25 Sep 2024 17:10:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0009D6B00BA; Wed, 25 Sep 2024 13:10:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ECA9D6B00BB; Wed, 25 Sep 2024 13:10:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D1E066B00BC; Wed, 25 Sep 2024 13:10:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AB77A6B00BA for ; Wed, 25 Sep 2024 13:10:44 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 623FA1C441F for ; Wed, 25 Sep 2024 17:10:44 +0000 (UTC) X-FDA: 82603900008.05.8571269 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by imf22.hostedemail.com (Postfix) with ESMTP id 76F7CC000D for ; Wed, 25 Sep 2024 17:10:42 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Gws4B9O4; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727284226; a=rsa-sha256; cv=none; b=cq2xU6PsxhiKmecoYsWDC+R7gPn/86BV4iRt+WiMjIZ+ALH6wZjEPRUNRqwry4p32z6U7a rWC8g7P8prDPK4OUUir5DAmNk9BqAQQF5BMN8vKx2ybQtcDmru7gIKDavhQelbzUhP01IO lZ+PQADmfRgeP+CAREn77YNZK9IZlKo= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Gws4B9O4; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727284226; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eLsxaOIfoQivSOjlmGHEUGtgy/OnwR5U8Z8UVuFepyI=; b=RHdHJsLEwfDusnvMSj5RajGLfMyNhcdYe2g4/qUEDqXn85bTTruHT+r6cGg1dS3noClzfv yeOTneLaDA9wevwOofxeFug5GISsZ5U5vv/EGir4Er2xN0/xkicIrRzKP2wjtPYAb3MYJz 0+omFylh7x51Nj2oekF1g2lpHSkZOT0= Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-7179069d029so108348b3a.2 for ; Wed, 25 Sep 2024 10:10:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1727284241; x=1727889041; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=eLsxaOIfoQivSOjlmGHEUGtgy/OnwR5U8Z8UVuFepyI=; b=Gws4B9O4kmO2CAEoF+UOsvDq3TUbdJiocB8AdqlzNfE7n5D5wHx7fsTWnHAqrlhkGo 01RcMOlDQndYXCQ+5/mc+yXEhNbA7M3VD/RKVCplab2TT9LzSNthV+TGz2Ou/7hdcM11 Pj4zZ8rPyoxXL0Vr2Br/8B4NRlJPEnY3o7G7GxrcAtRhvwoJ80ymgZ6/hCHIB5JxMUpL n3NCrokkyOY+5r45ksHpZIvs7WXePJ7OKxknuxHpVM1wYYkiEigxc2TOiJzHz8Wlgn5m BNYGaULsAZQFKBvSQfITHrP16Hk35S0FiVB5yz8UerpXpNZZd9nIRa1buPfwUpr6BMTu CGrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727284241; x=1727889041; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=eLsxaOIfoQivSOjlmGHEUGtgy/OnwR5U8Z8UVuFepyI=; b=UD15IhmAlY9w/bus8xMngK3Gmm/t957yeU5Xo+K4JbJXlUJbcmtGMTwPQoO0CkaK+9 mxZ1EAq5kjJ00rlbkNMDA+8VVk9TGtE3gGAsr/k5r3gscVkG1q+q5VEGSR9iWqNYO7L6 fJioe1HGDpq0D/2DwGWi2RnCNK8jZzOrhrxKb3klqn7TZ8L+iVOS6mN+kW9lZsF7s16S ElU5vL1K5mdQsgii5IwXyGc5+5iMOoAOoW7/8/3Sn+ICkjJL0nOVjdRoQXAksQMyZB2x zftp5PwhZDfA90zY9spaH1r3r3w+E/GoXPiwp22AvP52SMSuiqrRFQ2kjL8RdpD3mUGB Xzkw== X-Gm-Message-State: AOJu0Yx7kZG/DsGOvhzw3ttimDgNeaX93Wx8GrQXbQ4dLlrUXriYgjGs 0Bzn4U5dtSRcYzJnlwKT99tNP2t4Hq6m30DJHHu595t/U352lZa8jBX2ZywT X-Google-Smtp-Source: AGHT+IGCY98J4UUFHErBPNttuT/k9GAjpJknUsJQjnf9gUlRn4cQ8ORRz/CstFqvhiI7cD0KN8IdZw== X-Received: by 2002:a05:6a20:d497:b0:1d0:56b1:1c59 with SMTP id adf61e73a8af0-1d4d4b860c3mr4704612637.32.1727284240487; Wed, 25 Sep 2024 10:10:40 -0700 (PDT) Received: from KASONG-MC4.tencent.com ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71b032eb4f7sm2437900b3a.181.2024.09.25.10.10.37 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 25 Sep 2024 10:10:40 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , Johannes Weiner , Roman Gushchin , Waiman Long , Shakeel Butt , Michal Hocko , Chengming Zhou , Qi Zheng , Muchun Song , Kairui Song Subject: [PATCH v2 4/6] mm/list_lru: simplify reparenting and initial allocation Date: Thu, 26 Sep 2024 01:10:18 +0800 Message-ID: <20240925171020.32142-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20240925171020.32142-1-ryncsn@gmail.com> References: <20240925171020.32142-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: wuxysrn3xe6oaud4wjjkr7uxp8crgfeu X-Rspamd-Queue-Id: 76F7CC000D X-Rspamd-Server: rspam02 X-HE-Tag: 1727284242-126161 X-HE-Meta: U2FsdGVkX19XxlK1O6NIe1+Gpum3YDisp+RwfWkQa9BZbAL7NioK4KyI9m/qlqpLzHmIrkTyrXAVFFBPh9TK2Q/ELnZunR/fC2WF0OnM8o0KG3zI32BJeVxjxveGXSiOkY+qephE9+BJPPbONh8yLMmzUff6XvZsKn1cosYy0w+Di9z82WGqhB2nhx21glw2G9FVHCS0j9DFqOYGV1mTqthXxFmcQ+o5xuip1iPP9+4QEyzO/vRMYsOXUd06NxA7oj0bdfuDgiF62cnAFVwmx+pXC3VRXl8y/7QSPIPBr90n9TrE9f0YIT0iO4aRqvZlwWdFqvMOvA00OcYReyu1goilnqNJ+2ldhTH3lTRCG32VRPwyFzYHKXZhIZ4B1z+OhAuHmNB14CpSzSHQbT2DayWEmvNY4Qs6Xa7iwP9BzUELXodBdbxioApGaQfFKub6MgU1736b/I9M55xjl9s36raLwf6Apl+wnbkwH+N0lEQHTVeZzvvh/aAQjvzfcxEVpiQQKmN3gYXm0k+vJe5pDJj/NYRwUHwes3jUnBIvId2J/N63XvO3Ddxqs/u7SZjJiXQiqd3zR4L+X7uuA3WE5ilY7D+KDmh8MReaZ4vOvbfJ7qFE9kVafdYo57zjJqhWbuDWjUPAan1KuQIe9w5DARJFslVzOuhp5d99+VE5VHck3juNBxoBg9lLPzgN5gV+yI1FPSpPOnouNH8Wm1yRAyYpXXvD4fA0h7nN4rUe7wKeZTGHbaInjtyCLkRxDYttQXjsGWM/kE6WMmnGrcejnIYia1KsxJMeoHUDGnrax2biql/zp+3GBC3giEZXzbO3vBHhG20Vz77yL5k+ssadAaO4DEv5+8M30PpAVwb2blK/DIzoce1Q9IUH00Z6wLaoy0gH8bAqUiinbZXHNUfTOHCc5T83zx2AMgGcbx5cJn0SPBHDg5VC2LvC5tnsca31Sp6qamA8PF3HtNmZLU1 4usGYUOy fuLlssYXqvMbUsOrWPlc1bsMVLdkxWAxGNSs7vGXtap2Hk8fGW/C4vvw24FIHqv8vCkH+8uKyWmR7AkYdnVkImCByFw3uXZu3XBOZUmespBhj1ndHVKCdHe7+4E0X6uioaR7mz6+qYA8b2nGJvtS3Veca70kSSpfX8rq+jef8yw0TIF32yxaOPl8x9oDUDu/aTwwWM+bKrIEqMrONWbOSnQ9goq/mBjd9E+W7bj/omb8SAkyUkLFJpEOsZSfdnJVRxbQsU3S/Y9CmbSM2R3YgREQAzqRQ/MGzmDwP+TsIPZedH3ymOC1W0hmRrUiNZ/uI+sUXW/yCSqx1DHwk0w5ZzAfBYHEIE9zRYzYW3PY3JIgncxhsQSTkUfNjtlawmGQ6WbPtiyBGxg3fhKaw+FmL2km+lgmNWDvBdWpboBGlQCcyC+QjnbUTA2l5J0lHRLlELU0qy1FOluH1iJWVjVXx4HQAKvre+kson0qpyR2BVXfy3C+fFAFXRwSViHYkpdNWwRQkD4q7KFNVKsfg5rCu5kD7JeoFVaPwtMCJAZ2x3hp0IXDsRZi09Y44lBJWyvZ4kdHu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Currently, there is a lot of code for detecting reparent racing using kmemcg_id as the synchronization flag. And an intermediate table is required to record and compare the kmemcg_id. We can simplify this by just checking the cgroup css status, skip if cgroup is being offlined. On the reparenting side, ensure no more allocation is on going and no further allocation will occur by using the XArray lock as barrier. Combined with a O(n^2) top-down walk for the allocation, we get rid of the intermediate table allocation completely. Despite being O(n^2), it should be actually faster because it's not practical to have a very deep cgroup level, and in most cases the parent cgroup should have been allocated already. This also avoided changing kmemcg_id before reparenting, making cgroups have a stable index for list_lru_memcg. After this change it's possible that a dying cgroup will see a NULL value in XArray corresponding to the kmemcg_id, because the kmemcg_id will point to an empty slot. In such case, just fallback to use its parent. As a result the code is simpler, following test also showed a very slight performance gain (12 test runs): prepare() { mkdir /tmp/test-fs modprobe brd rd_nr=1 rd_size=16777216 mkfs.xfs -f /dev/ram0 mount -t xfs /dev/ram0 /tmp/test-fs for i in $(seq 10000); do seq 8000 > "/tmp/test-fs/$i" done mkdir -p /sys/fs/cgroup/system.slice/bench/test/1 echo +memory > /sys/fs/cgroup/system.slice/bench/cgroup.subtree_control echo +memory > /sys/fs/cgroup/system.slice/bench/test/cgroup.subtree_control echo +memory > /sys/fs/cgroup/system.slice/bench/test/1/cgroup.subtree_control echo 768M > /sys/fs/cgroup/system.slice/bench/memory.max } do_test() { read_worker() { mkdir -p "/sys/fs/cgroup/system.slice/bench/test/1/$1" echo $BASHPID > "/sys/fs/cgroup/system.slice/bench/test/1/$1/cgroup.procs" read -r __TMP < "/tmp/test-fs/$1"; } read_in_all() { for i in $(seq 10000); do read_worker "$i" & done; wait } echo 3 > /proc/sys/vm/drop_caches time read_in_all for i in $(seq 1 10000); do rmdir "/sys/fs/cgroup/system.slice/bench/test/1/$i" &>/dev/null done } Before: real 0m3.498s user 0m11.037s sys 0m35.872s real 1m33.860s user 0m11.593s sys 3m1.169s real 1m31.883s user 0m11.265s sys 2m59.198s real 1m32.394s user 0m11.294s sys 3m1.616s real 1m31.017s user 0m11.379s sys 3m1.349s real 1m31.931s user 0m11.295s sys 2m59.863s real 1m32.758s user 0m11.254s sys 2m59.538s real 1m35.198s user 0m11.145s sys 3m1.123s real 1m30.531s user 0m11.393s sys 2m58.089s real 1m31.142s user 0m11.333s sys 3m0.549s After: real 0m3.489s user 0m10.943s sys 0m36.036s real 1m10.893s user 0m11.495s sys 2m38.545s real 1m29.129s user 0m11.382s sys 3m1.601s real 1m29.944s user 0m11.494s sys 3m1.575s real 1m31.208s user 0m11.451s sys 2m59.693s real 1m25.944s user 0m11.327s sys 2m56.394s real 1m28.599s user 0m11.312s sys 3m0.162s real 1m26.746s user 0m11.538s sys 2m55.462s real 1m30.668s user 0m11.475s sys 3m2.075s real 1m29.258s user 0m11.292s sys 3m0.780s Which is slightly faster in real time. Signed-off-by: Kairui Song --- mm/list_lru.c | 178 +++++++++++++++++++++----------------------------- mm/zswap.c | 7 +- 2 files changed, 77 insertions(+), 108 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index b54f092d4d65..172b16146e15 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -59,6 +59,20 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) } return &lru->node[nid].lru; } + +static inline struct list_lru_one * +list_lru_from_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg) +{ + struct list_lru_one *l; +again: + l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); + if (likely(l)) + return l; + + memcg = parent_mem_cgroup(memcg); + VM_WARN_ON(!css_is_dying(&memcg->css)); + goto again; +} #else static void list_lru_register(struct list_lru *lru) { @@ -83,6 +97,12 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) { return &lru->node[nid].lru; } + +static inline struct list_lru_one * +list_lru_from_memcg(struct list_lru *lru, int nid, int idx) +{ + return &lru->node[nid].lru; +} #endif /* CONFIG_MEMCG */ /* The caller must ensure the memcg lifetime. */ @@ -94,7 +114,7 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, spin_lock(&nlru->lock); if (list_empty(item)) { - l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); + l = list_lru_from_memcg(lru, nid, memcg); list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) @@ -133,7 +153,7 @@ bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid, spin_lock(&nlru->lock); if (!list_empty(item)) { - l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); + l = list_lru_from_memcg(lru, nid, memcg); list_del_init(item); l->nr_items--; nlru->nr_items--; @@ -355,20 +375,6 @@ static struct list_lru_memcg *memcg_init_list_lru_one(gfp_t gfp) return mlru; } -static void memcg_list_lru_free(struct list_lru *lru, int src_idx) -{ - struct list_lru_memcg *mlru = xa_erase_irq(&lru->xa, src_idx); - - /* - * The __list_lru_walk_one() can walk the list of this node. - * We need kvfree_rcu() here. And the walking of the list - * is under lru->node[nid]->lock, which can serve as a RCU - * read-side critical section. - */ - if (mlru) - kvfree_rcu(mlru, rcu); -} - static inline void memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) { if (memcg_aware) @@ -393,22 +399,18 @@ static void memcg_destroy_list_lru(struct list_lru *lru) } static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, - int src_idx, struct mem_cgroup *dst_memcg) + struct list_lru_one *src, + struct mem_cgroup *dst_memcg) { struct list_lru_node *nlru = &lru->node[nid]; - int dst_idx = dst_memcg->kmemcg_id; - struct list_lru_one *src, *dst; + struct list_lru_one *dst; /* * Since list_lru_{add,del} may be called under an IRQ-safe lock, * we have to use IRQ-safe primitives here to avoid deadlock. */ spin_lock_irq(&nlru->lock); - - src = list_lru_from_memcg_idx(lru, nid, src_idx); - if (!src) - goto out; - dst = list_lru_from_memcg_idx(lru, nid, dst_idx); + dst = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(dst_memcg)); list_splice_init(&src->list, &dst->list); @@ -417,46 +419,43 @@ static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0; } -out: spin_unlock_irq(&nlru->lock); } void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent) { - struct cgroup_subsys_state *css; struct list_lru *lru; - int src_idx = memcg->kmemcg_id, i; - - /* - * Change kmemcg_id of this cgroup and all its descendants to the - * parent's id, and then move all entries from this cgroup's list_lrus - * to ones of the parent. - */ - rcu_read_lock(); - css_for_each_descendant_pre(css, &memcg->css) { - struct mem_cgroup *child; - - child = mem_cgroup_from_css(css); - WRITE_ONCE(child->kmemcg_id, parent->kmemcg_id); - } - rcu_read_unlock(); + int i; - /* - * With kmemcg_id set to parent, holding the lock of each list_lru_node - * below can prevent list_lru_{add,del,isolate} from touching the lru, - * safe to reparent. - */ mutex_lock(&list_lrus_mutex); list_for_each_entry(lru, &memcg_list_lrus, list) { + struct list_lru_memcg *mlru; + XA_STATE(xas, &lru->xa, memcg->kmemcg_id); + + /* + * Lock the Xarray to ensure no on going list_lru_memcg + * allocation and further allocation will see css_is_dying(). + */ + xas_lock_irq(&xas); + mlru = xas_store(&xas, NULL); + xas_unlock_irq(&xas); + if (!mlru) + continue; + + /* + * With Xarray value set to NULL, holding the lru lock below + * prevents list_lru_{add,del,isolate} from touching the lru, + * safe to reparent. + */ for_each_node(i) - memcg_reparent_list_lru_node(lru, i, src_idx, parent); + memcg_reparent_list_lru_node(lru, i, &mlru->node[i], parent); /* * Here all list_lrus corresponding to the cgroup are guaranteed * to remain empty, we can safely free this lru, any further * memcg_list_lru_alloc() call will simply bail out. */ - memcg_list_lru_free(lru, src_idx); + kvfree_rcu(mlru, rcu); } mutex_unlock(&list_lrus_mutex); } @@ -472,77 +471,48 @@ static inline bool memcg_list_lru_allocated(struct mem_cgroup *memcg, int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, gfp_t gfp) { - int i; unsigned long flags; - struct list_lru_memcg_table { - struct list_lru_memcg *mlru; - struct mem_cgroup *memcg; - } *table; + struct list_lru_memcg *mlru; + struct mem_cgroup *pos, *parent; XA_STATE(xas, &lru->xa, 0); if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru)) return 0; gfp &= GFP_RECLAIM_MASK; - table = kmalloc_array(memcg->css.cgroup->level, sizeof(*table), gfp); - if (!table) - return -ENOMEM; - /* * Because the list_lru can be reparented to the parent cgroup's * list_lru, we should make sure that this cgroup and all its * ancestors have allocated list_lru_memcg. */ - for (i = 0; memcg; memcg = parent_mem_cgroup(memcg), i++) { - if (memcg_list_lru_allocated(memcg, lru)) - break; - - table[i].memcg = memcg; - table[i].mlru = memcg_init_list_lru_one(gfp); - if (!table[i].mlru) { - while (i--) - kfree(table[i].mlru); - kfree(table); - return -ENOMEM; + do { + /* + * Keep finding the farest parent that wasn't populated + * until found memcg itself. + */ + pos = memcg; + parent = parent_mem_cgroup(pos); + while (!memcg_list_lru_allocated(parent, lru)) { + pos = parent; + parent = parent_mem_cgroup(pos); } - } - - xas_lock_irqsave(&xas, flags); - while (i--) { - int index = READ_ONCE(table[i].memcg->kmemcg_id); - struct list_lru_memcg *mlru = table[i].mlru; - xas_set(&xas, index); -retry: - if (unlikely(index < 0 || xas_error(&xas) || xas_load(&xas))) { - kfree(mlru); - } else { - xas_store(&xas, mlru); - if (xas_error(&xas) == -ENOMEM) { - xas_unlock_irqrestore(&xas, flags); - if (xas_nomem(&xas, gfp)) - xas_set_err(&xas, 0); - xas_lock_irqsave(&xas, flags); - /* - * The xas lock has been released, this memcg - * can be reparented before us. So reload - * memcg id. More details see the comments - * in memcg_reparent_list_lrus(). - */ - index = READ_ONCE(table[i].memcg->kmemcg_id); - if (index < 0) - xas_set_err(&xas, 0); - else if (!xas_error(&xas) && index != xas.xa_index) - xas_set(&xas, index); - goto retry; + mlru = memcg_init_list_lru_one(gfp); + if (!mlru) + return -ENOMEM; + xas_set(&xas, pos->kmemcg_id); + do { + xas_lock_irqsave(&xas, flags); + if (!xas_load(&xas) && !css_is_dying(&pos->css)) { + xas_store(&xas, mlru); + if (!xas_error(&xas)) + mlru = NULL; } - } - } - /* xas_nomem() is used to free memory instead of memory allocation. */ - if (xas.xa_alloc) - xas_nomem(&xas, gfp); - xas_unlock_irqrestore(&xas, flags); - kfree(table); + xas_unlock_irqrestore(&xas, flags); + } while (xas_nomem(&xas, gfp)); + if (mlru) + kfree(mlru); + } while (pos != memcg && !css_is_dying(&pos->css)); return xas_error(&xas); } diff --git a/mm/zswap.c b/mm/zswap.c index 449914ea9919..95a28301b7cd 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -704,12 +704,11 @@ static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *entry) /* * Note that it is safe to use rcu_read_lock() here, even in the face of - * concurrent memcg offlining. Thanks to the memcg->kmemcg_id indirection - * used in list_lru lookup, only two scenarios are possible: + * concurrent memcg offlining: * - * 1. list_lru_add() is called before memcg->kmemcg_id is updated. The + * 1. list_lru_add() is called before list_lru_memcg is erased. The * new entry will be reparented to memcg's parent's list_lru. - * 2. list_lru_add() is called after memcg->kmemcg_id is updated. The + * 2. list_lru_add() is called after list_lru_memcg is erased. The * new entry will be added directly to memcg's parent's list_lru. * * Similar reasoning holds for list_lru_del().