From patchwork Fri May 3 20:18:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13653401 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E2C2C10F16 for ; Fri, 3 May 2024 20:19:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 902976B0092; Fri, 3 May 2024 16:19:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B1B16B0093; Fri, 3 May 2024 16:19:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72BF86B0095; Fri, 3 May 2024 16:19:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4CDCB6B0092 for ; Fri, 3 May 2024 16:19:03 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CCD588079C for ; Fri, 3 May 2024 20:19:02 +0000 (UTC) X-FDA: 82078198524.09.94F9C74 Received: from out-174.mta1.migadu.com (out-174.mta1.migadu.com [95.215.58.174]) by imf21.hostedemail.com (Postfix) with ESMTP id A55061C0009 for ; Fri, 3 May 2024 20:19:00 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="vWM/xn2u"; spf=pass (imf21.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.174 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714767540; a=rsa-sha256; cv=none; b=UK6jlZAJJuYzBbIIPbdeKInzwwsXxukX4j2phqrxxCxMxgyaFE7v1kD/QojKqrhPqY39RA e8Mnn8zcdQKEMD9x4DQWAxmCbmC2WOIaheyTb2lH45LbQ4z1omSRsf2yNZsGQaAXG6fVji 4iItVzmVNvftEGIhC6yYY+fPCUA5QeA= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="vWM/xn2u"; spf=pass (imf21.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.174 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714767540; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CFkZrf+dM1tVK36JDqPspeMNDuJI8Dzfelt+0AW+5E8=; b=0YOPjSs5O4N012gmcvENOWA3nxedLKWqHE283Q0KhWRf0cGNJhqkuQF2sv9ztEGoAekvUT CCnq1WeHLSlM7MbNuYqiamvepvbm/Tj0MOCdbcjTz9vbdiwdX1kxHQ2jmyJ5Ka5DsvCdYL nkjkMobOtNwEj0jgJltuLYiHVfGzVU0= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1714767539; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CFkZrf+dM1tVK36JDqPspeMNDuJI8Dzfelt+0AW+5E8=; b=vWM/xn2uztR0kVJVznl+MWnB716gH9m2QmOJP+reJrlv493o2yq4sat0Wt99PyRdOFyd54 EDXeiCaElTjH/PNYs+RPQzEtBIURfEF96X9qJrtLZkFpKN9jIG29Ih5KsOrL+vvN0tGggt VpCpmFyzajJGi+C460U/iMjLPuHeTYQ= From: Roman Gushchin To: Andrew Morton Cc: Muchun Song , Johannes Weiner , Michal Hocko , Shakeel Butt , Frank van der Linden , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Roman Gushchin Subject: [PATCH v1 3/4] mm: memcg: don't call propagate_protected_usage() needlessly Date: Fri, 3 May 2024 13:18:34 -0700 Message-ID: <20240503201835.2969707-4-roman.gushchin@linux.dev> In-Reply-To: <20240503201835.2969707-1-roman.gushchin@linux.dev> References: <20240503201835.2969707-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Stat-Signature: 5oujn4daz83aefwfcdo71f8bh9819uqo X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A55061C0009 X-HE-Tag: 1714767540-302076 X-HE-Meta: U2FsdGVkX1/kO+tOtWoLBJiVCcdTMiIl8nYDsVNpPg9ecCzDPKJHXAbBAzGIvAwJon3kWXEge6I8FJtG80NrDto1VWU9P/4r/uHlUjc5fGazCjXaUBN7yPe8Jr3wV7zngLZ3peGatDkNh52remUra479qaSoUVq9T20PzUQou+OGMGbJ7NN78x82Kew9xqu6U8/ngfrpwA3jeVffB1ZlYg4XQgp5l5l4jE3pzmu05UzCrEliG46gBU3hsyGuLf2UM+vVRB3eKJt/ARK9M1ikNauzFRz5zcC9QY6DJHkmvQ+kxGE75ZeQ6BZhURwuo0QSqX9QHpaNmKdenWYgPr8jg2bzbYXkPFLLP6F1PAh+6F5w9TYnoABlAYGFp2GBnDMdvCKU2tHIxZxSq41F768zw9znsY+olnfg4okVjax4yJ2yprBJocJ5tq0odE9i9Pi/Qxe0Tm/QDck50BzbIEVi8klS4tp+rXcue0zHPvPXPYIJaF6YrFLPyrczxknXb2T5ySzbzGCZzp1j5idYIP0cR0iH/esA5Oz9ZZ9x7vs4NDoa4AWV5WE1i9HWiVLLVtmvsRBBUnpTrdm0V17ICkvYPX/VlXeFYU2lm1UreioZmle61zBw96HOJ29Pfzgn0HG3EQxPW7iEHhti83c1l3YadCgP+1JuYPN7fvX/8REtbUhs5DLjCR+2QaA8AdW/QMt/5x1lwUHsIsF11iPPrJ/q5F1HjD7gPvBzc3WQwtpO909URKUSSLkTxTLuYSGbs9FVzdaVmiaGn8t6kee24SgCcC5E8n/4xClDhGrnSXh49c3Cd5gy/Jxq8GBGlOiB8GcY4pa5u/8m00tWHf9KH7/muCicKDQ1lccSlV4nHtvxjXIaLzElBBkX/xJzuI+XkABVPQ76f8nrl8kEPid684T0j9itU+1KMB5/gYfa/qSEez4OgKBJwQ/Vg9wWHuzDJmGm6QmlFo5iKxy2eZOu9Vp 9j6yaqy+ ydaZ9lqY2cHAuRqisbWtCAHTusbuzrvkv1Etm38OJBiLRv9HIAwR0pw4+w6kXAUbfsQUgnZK7pjHKesX+Nh04IGgS+TfTudD9Bom3p+XrxjaBBDZh5438QeyE4N8/QHIocfA0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Memory protection (min/low) requires a constant tracking of the usage of protected memory. propagate_protected_usage() is called each time and does a number of operations even in cases when the actual memory protection functionality is not supported (e.g. hugetlb cgroups or memcg swap counters). It's an obvious inefficiency, which can be addressed by calling propagate_protected_usage() optionally and only for the right counter type. It eliminates a number of operations from hot paths. Signed-off-by: Roman Gushchin --- include/linux/page_counter.h | 8 +++++++- mm/hugetlb_cgroup.c | 2 +- mm/memcontrol.c | 4 ++-- mm/page_counter.c | 15 ++++++++++++--- 4 files changed, 22 insertions(+), 7 deletions(-) diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h index ae6cc080b78b..5d963f54fcb8 100644 --- a/include/linux/page_counter.h +++ b/include/linux/page_counter.h @@ -62,6 +62,7 @@ struct page_counter { /* Keep all the read most fields in a separete cacheline. */ CACHELINE_PADDING(_pad2_); + bool protection_support; unsigned long min; unsigned long low; unsigned long high[__MCT_NR_ITEMS]; @@ -75,8 +76,12 @@ struct page_counter { #define PAGE_COUNTER_MAX (LONG_MAX / PAGE_SIZE) #endif +/* + * Protection is supported only for the first counter (with id 0). + */ static inline void page_counter_init(struct page_counter *counter, - struct page_counter *parent) + struct page_counter *parent, + bool protection_support) { int i; @@ -86,6 +91,7 @@ static inline void page_counter_init(struct page_counter *counter, } counter->parent = parent; + counter->protection_support = protection_support; } static inline unsigned long page_counter_read(struct page_counter *counter, diff --git a/mm/hugetlb_cgroup.c b/mm/hugetlb_cgroup.c index 069c7f164dc5..81cb78d0714f 100644 --- a/mm/hugetlb_cgroup.c +++ b/mm/hugetlb_cgroup.c @@ -79,7 +79,7 @@ static void hugetlb_cgroup_init(struct hugetlb_cgroup *h_cgroup, int idx; page_counter_init(&h_cgroup->res, - parent_h_cgroup ? &parent_h_cgroup->res : NULL); + parent_h_cgroup ? &parent_h_cgroup->res : NULL, false); for (idx = 0; idx < HUGE_MAX_HSTATE; idx++) { unsigned long limit; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f4511506ef1f..46becae5ff99 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5761,11 +5761,11 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) if (parent) { WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent)); WRITE_ONCE(memcg->oom_kill_disable, READ_ONCE(parent->oom_kill_disable)); - page_counter_init(&memcg->memory, &parent->memory); + page_counter_init(&memcg->memory, &parent->memory, true); } else { init_memcg_stats(); init_memcg_events(); - page_counter_init(&memcg->memory, NULL); + page_counter_init(&memcg->memory, NULL, true); root_mem_cgroup = memcg; return &memcg->css; } diff --git a/mm/page_counter.c b/mm/page_counter.c index b6ca3adbc226..5a27e3141ff3 100644 --- a/mm/page_counter.c +++ b/mm/page_counter.c @@ -60,7 +60,8 @@ void page_counter_cancel(struct page_counter *counter, new = 0; atomic_long_set(&counter->usage[id], new); } - propagate_protected_usage(counter, new); + if (counter->protection_support && id == 0) + propagate_protected_usage(counter, new); } /** @@ -76,12 +77,14 @@ void page_counter_charge(struct page_counter *counter, unsigned long nr_pages) { struct page_counter *c; + bool track_protection = counter->protection_support && (id == 0); for (c = counter; c; c = c->parent) { long new; new = atomic_long_add_return(nr_pages, &c->usage[id]); - propagate_protected_usage(c, new); + if (track_protection) + propagate_protected_usage(c, new); /* * This is indeed racy, but we can live with some * inaccuracy in the watermark. @@ -107,6 +110,7 @@ bool page_counter_try_charge(struct page_counter *counter, struct page_counter **fail) { struct page_counter *c; + bool track_protection = counter->protection_support && (id == 0); for (c = counter; c; c = c->parent) { long new; @@ -136,7 +140,8 @@ bool page_counter_try_charge(struct page_counter *counter, *fail = c; goto failed; } - propagate_protected_usage(c, new); + if (track_protection) + propagate_protected_usage(c, new); /* * Just like with failcnt, we can live with some * inaccuracy in the watermark. @@ -226,6 +231,8 @@ void page_counter_set_min(struct page_counter *counter, unsigned long nr_pages) { struct page_counter *c; + WARN_ON_ONCE(!counter->protection_support); + WRITE_ONCE(counter->min, nr_pages); for (c = counter; c; c = c->parent) @@ -243,6 +250,8 @@ void page_counter_set_low(struct page_counter *counter, unsigned long nr_pages) { struct page_counter *c; + WARN_ON_ONCE(!counter->protection_support); + WRITE_ONCE(counter->low, nr_pages); for (c = counter; c; c = c->parent)