From patchwork Fri Oct 1 19:09:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12531627 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9769C433FE for ; Fri, 1 Oct 2021 19:10:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 89FD561AD1 for ; Fri, 1 Oct 2021 19:10:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 89FD561AD1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 32D3094013E; Fri, 1 Oct 2021 15:10:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2DE15940121; Fri, 1 Oct 2021 15:10:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CA8A94013E; Fri, 1 Oct 2021 15:10:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0070.hostedemail.com [216.40.44.70]) by kanga.kvack.org (Postfix) with ESMTP id 0F280940121 for ; Fri, 1 Oct 2021 15:10:19 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C902E2B37F for ; Fri, 1 Oct 2021 19:10:18 +0000 (UTC) X-FDA: 78648809316.36.45FE0E6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 7DF1819BC for ; Fri, 1 Oct 2021 19:10:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633115418; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:in-reply-to:in-reply-to:references:references; bh=sFCP5YMQVpSD6yFGcGrxnQTLJHGtWuJlEliBV/TO+yU=; b=L3HeoNeVVzlMKikvGs5OzWTVpqJtZItUgUb8rDrv7e0s8AjS5xGnot1CusBKV6EYdM129A Vf2L9eqd35anYitMDwu2r7if2AT+nGjUqP5brdQ5M2jnrdxGN0XtsZKBC7i2wMwRZgZcYW XtdnQS9rDP8MDnu+DojTzFl1FX54ZOY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-327-_OTOcvk9PryE4TMuHiT51w-1; Fri, 01 Oct 2021 15:10:16 -0400 X-MC-Unique: _OTOcvk9PryE4TMuHiT51w-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C3C5E107B0EF; Fri, 1 Oct 2021 19:10:14 +0000 (UTC) Received: from llong.com (unknown [10.22.33.73]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8003960BD8; Fri, 1 Oct 2021 19:10:12 +0000 (UTC) From: Waiman Long To: Johannes Weiner , Michal Hocko , Vladimir Davydov , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Shakeel Butt , Muchun Song , Waiman Long Subject: [PATCH 1/3] mm, memcg: Don't put offlined memcg into local stock Date: Fri, 1 Oct 2021 15:09:36 -0400 Message-Id: <20211001190938.14050-2-longman@redhat.com> In-Reply-To: <20211001190938.14050-1-longman@redhat.com> References: <20211001190938.14050-1-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 7DF1819BC X-Stat-Signature: qgr7tkyrqnsyqjwuyqpqf4zqrzk86gsw Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=L3HeoNeV; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf22.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=longman@redhat.com X-HE-Tag: 1633115418-636023 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When freeing a page associated with an offlined memcg, refill_stock() will put it into local stock delaying its demise until another memcg comes in to take its place in the stock. To avoid that, we now check for offlined memcg and go directly in this case to the slowpath for the uncharge via the repurposed cancel_charge() function. Signed-off-by: Waiman Long Reported-by: kernel test robot Reported-by: kernel test robot Reviewed-by: Waiman Long --- mm/memcontrol.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4b32896d87a2..4568363062c1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2167,6 +2167,8 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) return ret; } +static void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages); + /* * Returns stocks cached in percpu and reset cached information. */ @@ -2178,9 +2180,7 @@ static void drain_stock(struct memcg_stock_pcp *stock) return; if (stock->nr_pages) { - page_counter_uncharge(&old->memory, stock->nr_pages); - if (do_memsw_account()) - page_counter_uncharge(&old->memsw, stock->nr_pages); + cancel_charge(old, stock->nr_pages); stock->nr_pages = 0; } @@ -2219,6 +2219,14 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) struct memcg_stock_pcp *stock; unsigned long flags; + /* + * An offlined memcg shouldn't be put into stock. + */ + if (unlikely(memcg->kmem_state != KMEM_ONLINE)) { + cancel_charge(memcg, nr_pages); + return; + } + local_irq_save(flags); stock = this_cpu_ptr(&memcg_stock); @@ -2732,7 +2740,6 @@ static inline int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, return try_charge_memcg(memcg, gfp_mask, nr_pages); } -#if defined(CONFIG_MEMCG_KMEM) || defined(CONFIG_MMU) static void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages) { if (mem_cgroup_is_root(memcg)) @@ -2742,7 +2749,6 @@ static void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages) if (do_memsw_account()) page_counter_uncharge(&memcg->memsw, nr_pages); } -#endif static void commit_charge(struct folio *folio, struct mem_cgroup *memcg) { From patchwork Fri Oct 1 19:09:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12531631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FBD2C433EF for ; Fri, 1 Oct 2021 19:10:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C4AB66137A for ; Fri, 1 Oct 2021 19:10:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C4AB66137A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 576CD940140; Fri, 1 Oct 2021 15:10:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 527A5940121; Fri, 1 Oct 2021 15:10:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C5D2940140; Fri, 1 Oct 2021 15:10:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0157.hostedemail.com [216.40.44.157]) by kanga.kvack.org (Postfix) with ESMTP id 2E372940121 for ; Fri, 1 Oct 2021 15:10:22 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id EA1CD3A2BE for ; Fri, 1 Oct 2021 19:10:21 +0000 (UTC) X-FDA: 78648809442.21.5B385FC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 74FD53001DA2 for ; Fri, 1 Oct 2021 19:10:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633115421; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:in-reply-to:in-reply-to:references:references; bh=4NNmOdozgW34250h3i+XZDY/eQfLDbJdb5QbTC9VPyU=; b=OWCUKVzgJjApLLcSlSARA0N48z6Vx1xtqEFqQQSi8uiFBumD0zvRu8RV4qXay/4Ro1P6a3 Zk5iAgoZBqP5ZLPS6K8O9yvsS99pakcbD8U0CeM+k6tT5uZoD0hwjwmWWMh0rqkMz33e0X gWI2DWltxizz9z1/wyBMYMyNZaeRRWA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-576-QPseJOL4PxykHrj_plzBZg-1; Fri, 01 Oct 2021 15:10:17 -0400 X-MC-Unique: QPseJOL4PxykHrj_plzBZg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3082B107B0F3; Fri, 1 Oct 2021 19:10:16 +0000 (UTC) Received: from llong.com (unknown [10.22.33.73]) by smtp.corp.redhat.com (Postfix) with ESMTP id EA75C60BD8; Fri, 1 Oct 2021 19:10:14 +0000 (UTC) From: Waiman Long To: Johannes Weiner , Michal Hocko , Vladimir Davydov , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Shakeel Butt , Muchun Song , Waiman Long Subject: [PATCH 2/3] mm, memcg: Remove obsolete memcg_free_kmem() Date: Fri, 1 Oct 2021 15:09:37 -0400 Message-Id: <20211001190938.14050-3-longman@redhat.com> In-Reply-To: <20211001190938.14050-1-longman@redhat.com> References: <20211001190938.14050-1-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 74FD53001DA2 X-Stat-Signature: w9z3qohcfzieiyjphps46dibcc6ouww3 Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OWCUKVzg; spf=none (imf09.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=longman@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1633115421-606981 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since commit d648bcc7fe65 ("mm: kmem: make memcg_kmem_enabled() irreversible"), the only thing memcg_free_kmem() does is to call memcg_offline_kmem() when the memcg is still online. However, memcg_offline_kmem() is only called from mem_cgroup_css_free() which cannot be reached if the memcg hasn't been offlined first. As this function now serves no purpose, we should just remove it. Signed-off-by: Waiman Long --- mm/memcontrol.c | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4568363062c1..8177f253a127 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3673,13 +3673,6 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) memcg_free_cache_id(kmemcg_id); } - -static void memcg_free_kmem(struct mem_cgroup *memcg) -{ - /* css_alloc() failed, offlining didn't happen */ - if (unlikely(memcg->kmem_state == KMEM_ONLINE)) - memcg_offline_kmem(memcg); -} #else static int memcg_online_kmem(struct mem_cgroup *memcg) { @@ -3688,9 +3681,6 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) static void memcg_offline_kmem(struct mem_cgroup *memcg) { } -static void memcg_free_kmem(struct mem_cgroup *memcg) -{ -} #endif /* CONFIG_MEMCG_KMEM */ static int memcg_update_kmem_max(struct mem_cgroup *memcg, @@ -5325,7 +5315,6 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); free_shrinker_info(memcg); - memcg_free_kmem(memcg); mem_cgroup_free(memcg); } From patchwork Fri Oct 1 19:09:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12531629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4A34C433FE for ; Fri, 1 Oct 2021 19:10:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4FEBB6137A for ; Fri, 1 Oct 2021 19:10:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4FEBB6137A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id E29D294013F; Fri, 1 Oct 2021 15:10:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DD8E4940121; Fri, 1 Oct 2021 15:10:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC71994013F; Fri, 1 Oct 2021 15:10:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0161.hostedemail.com [216.40.44.161]) by kanga.kvack.org (Postfix) with ESMTP id BD3D8940121 for ; Fri, 1 Oct 2021 15:10:21 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 84B128249980 for ; Fri, 1 Oct 2021 19:10:21 +0000 (UTC) X-FDA: 78648809442.04.D99386C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 3E809F000345 for ; Fri, 1 Oct 2021 19:10:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633115420; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:in-reply-to:in-reply-to:references:references; bh=eTKxhjw7Yo5S8XqVnMv4HlnJDXI4fKHSCti6PPjnd+Y=; b=KTY1RDQbAmdVwupYMjlcf8uuZfB1GHGtsiSS74AAK7Swv9LMN1reuenj1UXhpvcfvufVoa D80HIqVuYn6lwAxQSdSbQxKJEAFMwaNdZtKO1vnqP5dj7/RRUSFvFrLY4pJH2Jv1Es7Lz3 gptLzLtsntACh6HuSdiCWWIsKkMsQA8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-466-iuOMRinSNDSYTE6u2ziHIg-1; Fri, 01 Oct 2021 15:10:19 -0400 X-MC-Unique: iuOMRinSNDSYTE6u2ziHIg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8AFC78145E5; Fri, 1 Oct 2021 19:10:17 +0000 (UTC) Received: from llong.com (unknown [10.22.33.73]) by smtp.corp.redhat.com (Postfix) with ESMTP id 55E2F60BD8; Fri, 1 Oct 2021 19:10:16 +0000 (UTC) From: Waiman Long To: Johannes Weiner , Michal Hocko , Vladimir Davydov , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Shakeel Butt , Muchun Song , Waiman Long Subject: [PATCH 3/3] mm, memcg: Ensure valid memcg from objcg within a RCU critical section Date: Fri, 1 Oct 2021 15:09:38 -0400 Message-Id: <20211001190938.14050-4-longman@redhat.com> In-Reply-To: <20211001190938.14050-1-longman@redhat.com> References: <20211001190938.14050-1-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 3E809F000345 X-Stat-Signature: k35ey6kp78nywsc15qf3zz1jpe49bm1o Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KTY1RDQb; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf16.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=longman@redhat.com X-HE-Tag: 1633115421-585682 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To ensure that a to-be-offlined memcg fetched from objcg remains valid (has non-zero reference count) within a RCU critical section, a synchronize_rcu() call is inserted at the end of memcg_offline_kmem(). With that change, we no longer need to use css_tryget() in get_mem_cgroup_from_objcg() as the final css_put() in css_killed_work_fn() would not have been called yet. The obj_cgroup_uncharge_pages() function is simplifed to perform the whole uncharge operation within a RCU critical section saving a css_get()/css_put() pair. Signed-off-by: Waiman Long --- mm/memcontrol.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 8177f253a127..1dbb37d96e49 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2769,10 +2769,8 @@ static struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *objcg) struct mem_cgroup *memcg; rcu_read_lock(); -retry: memcg = obj_cgroup_memcg(objcg); - if (unlikely(!css_tryget(&memcg->css))) - goto retry; + css_get(&memcg->css); rcu_read_unlock(); return memcg; @@ -2947,13 +2945,14 @@ static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, { struct mem_cgroup *memcg; - memcg = get_mem_cgroup_from_objcg(objcg); + rcu_read_lock(); + memcg = obj_cgroup_memcg(objcg); if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) page_counter_uncharge(&memcg->kmem, nr_pages); refill_stock(memcg, nr_pages); - css_put(&memcg->css); + rcu_read_unlock(); } /* @@ -3672,6 +3671,13 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) memcg_drain_all_list_lrus(kmemcg_id, parent); memcg_free_cache_id(kmemcg_id); + + /* + * To ensure that a to-be-offlined memcg fetched from objcg remains + * valid within a RCU critical section, we need to wait here until + * the a grace period has elapsed. + */ + synchronize_rcu(); } #else static int memcg_online_kmem(struct mem_cgroup *memcg)