From patchwork Wed Sep 7 04:35:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 12968432 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9F6BECAAD3 for ; Wed, 7 Sep 2022 04:36:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 65B806B0072; Wed, 7 Sep 2022 00:36:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 60B978D0007; Wed, 7 Sep 2022 00:36:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AC3B8D0006; Wed, 7 Sep 2022 00:36:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 36DE56B0072 for ; Wed, 7 Sep 2022 00:36:27 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F1651140BBC for ; Wed, 7 Sep 2022 04:36:26 +0000 (UTC) X-FDA: 79884027972.27.80D6346 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf04.hostedemail.com (Postfix) with ESMTP id 927684007B for ; Wed, 7 Sep 2022 04:36:25 +0000 (UTC) Received: by mail-pg1-f201.google.com with SMTP id m188-20020a633fc5000000b00434dccacd4aso944127pga.10 for ; Tue, 06 Sep 2022 21:36:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=khSX7Y4iv18onStJMmBrPhwF58VNn8i03Anirden96Q=; b=gemiZCbExV47bKx8wOek1aQ0u8hzbt+EnZzyGvpEX6V3hdl7+FzjhOMQFdg/BAePWs g1Phtex5qyjXcE0CH8qo/myIMYCKBorj4O/lYfXNAuZQE1nIXh6Y/UorToCbyrfqTc08 w41m5Q1yfn7rEabS42x1kF04pr4Fz4DhfpdBQ7PAZvq2tOJI4kuvrPxvdDoYGIeFXtU7 73wZp+rS4GUGDo8QZ9/BnYm7kUNxwjCsKt+VQJT//oAjo2bR/bpBZ7YFQnfETArjdfst yH5HuLvanOXbdhRwhVJR8If5nQVhIbTM14rXsNmexTs4BoP70k0Ts+2cN/b8E1sehylv kQmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=khSX7Y4iv18onStJMmBrPhwF58VNn8i03Anirden96Q=; b=YqZ6OEZLvS0EKowzp4eyV9S5MEb6JLtd06Run1rp5SH14kKvtX46GzR9kVkwmJsULp Pl3znkG0lLavaQMXbSdIrOkx/q9YTl6I4TmuoFlpxY7c2oLTWrSHcL39W6YKXfyiApbw ebnywLhmGTP1q6QUPqIWeI53czr6mQ1ljk2yT33RzNJO/udQR/2wWxV3ZX/WPLAFEGBu LORIZ36mVX8n3k81iRTA2bkEWUAtPTvJJ0BpvGyyj186zXmktwPu61Nn6Dl6FHGoDvUb IUL3xO57pnoU49i+QfMApTrdjc7guCdI9ITfx+m4asdRZ+635BvQ6iBCjMKlXYJr0Gbf 56Rw== X-Gm-Message-State: ACgBeo0GQuqcmahv9I4mDc7vThj3I4TqG1Y63cyGPwhJZydOUqbPAB85 SeRRWU9P2rz7vWOieBnUhgKtKtoZ3wUeBw== X-Google-Smtp-Source: AA6agR5/ynpD4fcrcx0c8sHHlPJFfJQPloK/P5POaEbeKl2ov/6eY/SmFki7BrZyFFL0jRWVJtkjvAuQBWyeJQ== X-Received: from shakeelb.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:262e]) (user=shakeelb job=sendgmr) by 2002:a17:903:40c9:b0:176:e58c:f082 with SMTP id t9-20020a17090340c900b00176e58cf082mr1969898pld.60.1662525384705; Tue, 06 Sep 2022 21:36:24 -0700 (PDT) Date: Wed, 7 Sep 2022 04:35:37 +0000 In-Reply-To: <20220907043537.3457014-1-shakeelb@google.com> Mime-Version: 1.0 References: <20220907043537.3457014-1-shakeelb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220907043537.3457014-4-shakeelb@google.com> Subject: [PATCH 3/3] memcg: reduce size of memcg vmstats structures From: Shakeel Butt To: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song Cc: Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Shakeel Butt ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=gemiZCbE; spf=pass (imf04.hostedemail.com: domain of 3yB8YYwgKCIc3slvppwmrzzrwp.nzxwty58-xxv6lnv.z2r@flex--shakeelb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3yB8YYwgKCIc3slvppwmrzzrwp.nzxwty58-xxv6lnv.z2r@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662525385; a=rsa-sha256; cv=none; b=XQ66Hxb9YcWgW0tl5+FvRhccr0caIX1MfhHuZOUKNSqktWZ0QBpxb8zXCbR7kwtJWHQvSJ L9A1AANAv3H11UxdLpIOorYKZqzcuI8DS6z7XnqFLXlhqhPDhNb9L9qW2R52tUiN1evZIg azYamXEv8ZwK9bcib78xd2s29+B2GpA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662525385; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=khSX7Y4iv18onStJMmBrPhwF58VNn8i03Anirden96Q=; b=ks0iOFFM1dGf/tTJSiF8pxjz8EzmzGR11+7bDNRE1WRSgzCrK0vs5HhUGFfTF9QpP/WIeS jLcsqqCmCXa26ICKI+Sr7qKLVhUte+GTZPLeKhWTz/QLR2HgHRyXnldBfhh3DhpQJZG3lA lv99XPFN5dmF5Kgr5opQL03LeEbyCAI= Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=gemiZCbE; spf=pass (imf04.hostedemail.com: domain of 3yB8YYwgKCIc3slvppwmrzzrwp.nzxwty58-xxv6lnv.z2r@flex--shakeelb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3yB8YYwgKCIc3slvppwmrzzrwp.nzxwty58-xxv6lnv.z2r@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: x5j4p15ioazgh7o79m5uprwtuaq16oy7 X-Rspamd-Queue-Id: 927684007B X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1662525385-277760 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The struct memcg_vmstats and struct memcg_vmstats_percpu contains two arrays each for events of size NR_VM_EVENT_ITEMS which can be as large as 110. However the memcg v1 only uses 4 of those while memcg v2 uses 15. The union of both is 17. On a 64 bit system, we are wasting approximately ((110 - 17) * 8 * 2) * (nr_cpus + 1) bytes which is significant on large machines. This patch reduces the size of the given structures by adding one indirection and only stores array of events which are actually used by the memcg code. With this patch, the size of memcg_vmstats has reduced from 2544 bytes to 1056 bytes while the size of memcg_vmstats_percpu has reduced from 2568 bytes to 1080 bytes. Signed-off-by: Shakeel Butt Acked-by: Roman Gushchin Signed-off-by: Andrew Morton --- mm/memcontrol.c | 52 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d0ccc16ed416..a60012be6140 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -671,6 +671,8 @@ static void flush_memcg_stats_dwork(struct work_struct *w) /* Subset of vm_event_item to report for memcg event stats */ static const unsigned int memcg_vm_event_stat[] = { + PGPGIN, + PGPGOUT, PGSCAN_KSWAPD, PGSCAN_DIRECT, PGSTEAL_KSWAPD, @@ -692,14 +694,30 @@ static const unsigned int memcg_vm_event_stat[] = { #endif }; +#define NR_MEMCG_EVENTS ARRAY_SIZE(memcg_vm_event_stat) +static int mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly; + +static void init_memcg_events(void) +{ + int i; + + for (i = 0; i < NR_MEMCG_EVENTS; ++i) + mem_cgroup_events_index[memcg_vm_event_stat[i]] = i + 1; +} + +static inline int memcg_events_index(enum vm_event_item idx) +{ + return mem_cgroup_events_index[idx] - 1; +} + struct memcg_vmstats_percpu { /* Local (CPU and cgroup) page state & events */ long state[MEMCG_NR_STAT]; - unsigned long events[NR_VM_EVENT_ITEMS]; + unsigned long events[NR_MEMCG_EVENTS]; /* Delta calculation for lockless upward propagation */ long state_prev[MEMCG_NR_STAT]; - unsigned long events_prev[NR_VM_EVENT_ITEMS]; + unsigned long events_prev[NR_MEMCG_EVENTS]; /* Cgroup1: threshold notifications & softlimit tree updates */ unsigned long nr_page_events; @@ -709,11 +727,11 @@ struct memcg_vmstats_percpu { struct memcg_vmstats { /* Aggregated (CPU and subtree) page state & events */ long state[MEMCG_NR_STAT]; - unsigned long events[NR_VM_EVENT_ITEMS]; + unsigned long events[NR_MEMCG_EVENTS]; /* Pending child counts during tree propagation */ long state_pending[MEMCG_NR_STAT]; - unsigned long events_pending[NR_VM_EVENT_ITEMS]; + unsigned long events_pending[NR_MEMCG_EVENTS]; }; unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) @@ -873,24 +891,34 @@ void __mod_lruvec_kmem_state(void *p, enum node_stat_item idx, int val) void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, unsigned long count) { - if (mem_cgroup_disabled()) + int index = memcg_events_index(idx); + + if (mem_cgroup_disabled() || index < 0) return; memcg_stats_lock(); - __this_cpu_add(memcg->vmstats_percpu->events[idx], count); + __this_cpu_add(memcg->vmstats_percpu->events[index], count); memcg_rstat_updated(memcg, count); memcg_stats_unlock(); } static unsigned long memcg_events(struct mem_cgroup *memcg, int event) { - return READ_ONCE(memcg->vmstats->events[event]); + int index = memcg_events_index(event); + + if (index < 0) + return 0; + return READ_ONCE(memcg->vmstats->events[index]); } static unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) { long x = 0; int cpu; + int index = memcg_events_index(event); + + if (index < 0) + return 0; for_each_possible_cpu(cpu) x += per_cpu(memcg->vmstats_percpu->events[event], cpu); @@ -1564,10 +1592,15 @@ static void memory_stat_format(struct mem_cgroup *memcg, char *buf, int bufsize) memcg_events(memcg, PGSTEAL_KSWAPD) + memcg_events(memcg, PGSTEAL_DIRECT)); - for (i = 0; i < ARRAY_SIZE(memcg_vm_event_stat); i++) + for (i = 0; i < ARRAY_SIZE(memcg_vm_event_stat); i++) { + if (memcg_vm_event_stat[i] == PGPGIN || + memcg_vm_event_stat[i] == PGPGOUT) + continue; + seq_buf_printf(&s, "%s %lu\n", vm_event_name(memcg_vm_event_stat[i]), memcg_events(memcg, memcg_vm_event_stat[i])); + } /* The above should easily fit into one page */ WARN_ON_ONCE(seq_buf_has_overflowed(&s)); @@ -5309,6 +5342,7 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) page_counter_init(&memcg->kmem, &parent->kmem); page_counter_init(&memcg->tcpmem, &parent->tcpmem); } else { + init_memcg_events(); page_counter_init(&memcg->memory, NULL); page_counter_init(&memcg->swap, NULL); page_counter_init(&memcg->kmem, NULL); @@ -5477,7 +5511,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) parent->vmstats->state_pending[i] += delta; } - for (i = 0; i < NR_VM_EVENT_ITEMS; i++) { + for (i = 0; i < NR_MEMCG_EVENTS; i++) { delta = memcg->vmstats->events_pending[i]; if (delta) memcg->vmstats->events_pending[i] = 0;