From patchwork Wed Sep 7 04:35:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 12968430 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3658C38145 for ; Wed, 7 Sep 2022 04:36:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1047E8D0003; Wed, 7 Sep 2022 00:36:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B4676B0073; Wed, 7 Sep 2022 00:36:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E97768D0003; Wed, 7 Sep 2022 00:36:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D81AA6B0072 for ; Wed, 7 Sep 2022 00:36:10 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id AFFF880685 for ; Wed, 7 Sep 2022 04:36:10 +0000 (UTC) X-FDA: 79884027300.29.DDB80CB Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf26.hostedemail.com (Postfix) with ESMTP id 48C64140096 for ; Wed, 7 Sep 2022 04:36:10 +0000 (UTC) Received: by mail-pg1-f202.google.com with SMTP id m188-20020a633fc5000000b00434dccacd4aso943786pga.10 for ; Tue, 06 Sep 2022 21:36:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=+ENncUdfOxJ4q/8LZk6DI50+XMWx6BeUwv0z4tw0gRY=; b=ZequgtabdjcCJmJYwHcuirxjXWYVxqBEGoqYBTddWUrFGxayaRDdkPPL35s5tGIzQ7 oohJe3ane8IE/6VPhwguj78pXQyNyZra47KKc+c8xJKEHGorIcMzWmrmRlwLoA2dHS95 NKjaOI9d+CsoGs/PtD5hTrcJmdjy8ugZzPcm9WEnuiDhAQdfC8LPWQBoQHOijfpRjQwf z49WK7LH8anfLpxSIpaWSL7mxkmp44ZTKIN3Eyolkfra/wn0h5geUklvCLjRgu/NHGQX q+Fnut9idgOpogDkYMQfi517Q0/kauc+cWYzh2xco62FeXGuOpev9tG1hUyiyQYR/ABl 2HFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=+ENncUdfOxJ4q/8LZk6DI50+XMWx6BeUwv0z4tw0gRY=; b=odqw0oP78Kg7C5DKRhSXqO06lLA8n5RF5eN7JKR1x89baKJzCd1wl8+b2THVDRHgNw sZi8pH9uCSVIKkR3W7lzfBp14D3Pa8q6TvQpJgkDOeRJmHhX4hSys6b6lJ+dubKqSwfn woW1m7OBGTu+qciS4K2HPXqH+KzoZ+dE0R7/zvPm61pEhl5d3X6n0YWmXi7N+1Ks5XFv zmE5wSxOgYMj+5jthaae+oErIpEFuL9W84E6cdoOtNUn09Ji9ikOdTku6dnl+LXWhaI5 Pc44vCxTzU0kKtIUslksyBmrySG0nU1U27NbQMtRVuuOJ1rYytxfWODwIlbgd6bgCcoQ VYfA== X-Gm-Message-State: ACgBeo2X4EOwnRYv3yweEQfTnmRMwOY9eCbeFjWX+Y9MOeHhc0CdR6ZL JnNwbMzqF1koQvyEwoX7ivwEIkPVQHWdog== X-Google-Smtp-Source: AA6agR4Tjj9UlQLSGOInlnJphRscUgepBu0Rn+Rxd2copVybQoevmw94qxSMGgA6pLc+G3qI0NNwikm2QKECmw== X-Received: from shakeelb.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:262e]) (user=shakeelb job=sendgmr) by 2002:a17:90b:10a:b0:200:2849:235f with SMTP id p10-20020a17090b010a00b002002849235fmr175469pjz.1.1662525368959; Tue, 06 Sep 2022 21:36:08 -0700 (PDT) Date: Wed, 7 Sep 2022 04:35:35 +0000 In-Reply-To: <20220907043537.3457014-1-shakeelb@google.com> Mime-Version: 1.0 References: <20220907043537.3457014-1-shakeelb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220907043537.3457014-2-shakeelb@google.com> Subject: [PATCH 1/3] memcg: extract memcg_vmstats from struct mem_cgroup From: Shakeel Butt To: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song Cc: Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Shakeel Butt ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Zequgtab; spf=pass (imf26.hostedemail.com: domain of 3uB8YYwgKCHcncVfZZgWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--shakeelb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3uB8YYwgKCHcncVfZZgWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662525370; a=rsa-sha256; cv=none; b=ShcVe/kUGdP5tfU9zyn5CEHdz7fxL2cCz19RKUY+5fLiK/0IoeXgdPlQStUAJi+FqvXSDK CtIpKfp9RKJzjQvSfr2ZyrAGSjuymPiAd9CZKeb7EgAhJTbrrfOYe4YU54tKL6B8f/JRHO 7pANAdNUqSoB/EIjhwo395z/UX9QPWY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662525370; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+ENncUdfOxJ4q/8LZk6DI50+XMWx6BeUwv0z4tw0gRY=; b=Jph9SXI9q5HU7BKGlHM2Rj7X0QVahTNkWH+yk4PJDl7qP+e3vIYPfpuctJMzCZ4j5kUWka NwWPGn8CAtyWb2akTURmn/FJbJdXp+w5Q64viQN3+hjHPH5Um5tQZeFQoY4xjMoxvB4cLJ 963tE22+6zFtu8gbQ6cW8Pa1webvmg8= X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 48C64140096 X-Rspam-User: Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Zequgtab; spf=pass (imf26.hostedemail.com: domain of 3uB8YYwgKCHcncVfZZgWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--shakeelb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3uB8YYwgKCHcncVfZZgWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: y43583ihrjh49m7ugqfru514ibf8kg6o X-HE-Tag: 1662525370-874469 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparatory patch to reduce the memory overhead of memory cgroup. The struct memcg_vmstats is the largest object embedded into the struct mem_cgroup. This patch extracts struct memcg_vmstats from struct mem_cgroup to ease the following patches in reducing the size of struct memcg_vmstats. Signed-off-by: Shakeel Butt --- include/linux/memcontrol.h | 37 +++---------------------- mm/memcontrol.c | 57 ++++++++++++++++++++++++++++++++------ 2 files changed, 52 insertions(+), 42 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ca0df42662ad..dc7d40e575d5 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -80,29 +80,8 @@ enum mem_cgroup_events_target { MEM_CGROUP_NTARGETS, }; -struct memcg_vmstats_percpu { - /* Local (CPU and cgroup) page state & events */ - long state[MEMCG_NR_STAT]; - unsigned long events[NR_VM_EVENT_ITEMS]; - - /* Delta calculation for lockless upward propagation */ - long state_prev[MEMCG_NR_STAT]; - unsigned long events_prev[NR_VM_EVENT_ITEMS]; - - /* Cgroup1: threshold notifications & softlimit tree updates */ - unsigned long nr_page_events; - unsigned long targets[MEM_CGROUP_NTARGETS]; -}; - -struct memcg_vmstats { - /* Aggregated (CPU and subtree) page state & events */ - long state[MEMCG_NR_STAT]; - unsigned long events[NR_VM_EVENT_ITEMS]; - - /* Pending child counts during tree propagation */ - long state_pending[MEMCG_NR_STAT]; - unsigned long events_pending[NR_VM_EVENT_ITEMS]; -}; +struct memcg_vmstats_percpu; +struct memcg_vmstats; struct mem_cgroup_reclaim_iter { struct mem_cgroup *position; @@ -298,7 +277,7 @@ struct mem_cgroup { CACHELINE_PADDING(_pad1_); /* memory.stat */ - struct memcg_vmstats vmstats; + struct memcg_vmstats *vmstats; /* memory.events */ atomic_long_t memory_events[MEMCG_NR_MEMORY_EVENTS]; @@ -1001,15 +980,7 @@ static inline void mod_memcg_page_state(struct page *page, rcu_read_unlock(); } -static inline unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) -{ - long x = READ_ONCE(memcg->vmstats.state[idx]); -#ifdef CONFIG_SMP - if (x < 0) - x = 0; -#endif - return x; -} +unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx); static inline unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0a1a8a846870..b195d4ca2a72 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -669,6 +669,40 @@ static void flush_memcg_stats_dwork(struct work_struct *w) queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } +struct memcg_vmstats_percpu { + /* Local (CPU and cgroup) page state & events */ + long state[MEMCG_NR_STAT]; + unsigned long events[NR_VM_EVENT_ITEMS]; + + /* Delta calculation for lockless upward propagation */ + long state_prev[MEMCG_NR_STAT]; + unsigned long events_prev[NR_VM_EVENT_ITEMS]; + + /* Cgroup1: threshold notifications & softlimit tree updates */ + unsigned long nr_page_events; + unsigned long targets[MEM_CGROUP_NTARGETS]; +}; + +struct memcg_vmstats { + /* Aggregated (CPU and subtree) page state & events */ + long state[MEMCG_NR_STAT]; + unsigned long events[NR_VM_EVENT_ITEMS]; + + /* Pending child counts during tree propagation */ + long state_pending[MEMCG_NR_STAT]; + unsigned long events_pending[NR_VM_EVENT_ITEMS]; +}; + +unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) +{ + long x = READ_ONCE(memcg->vmstats->state[idx]); +#ifdef CONFIG_SMP + if (x < 0) + x = 0; +#endif + return x; +} + /** * __mod_memcg_state - update cgroup memory statistics * @memcg: the memory cgroup @@ -827,7 +861,7 @@ void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, static unsigned long memcg_events(struct mem_cgroup *memcg, int event) { - return READ_ONCE(memcg->vmstats.events[event]); + return READ_ONCE(memcg->vmstats->events[event]); } static unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) @@ -5170,6 +5204,7 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg) for_each_node(node) free_mem_cgroup_per_node_info(memcg, node); + kfree(memcg->vmstats); free_percpu(memcg->vmstats_percpu); kfree(memcg); } @@ -5199,6 +5234,10 @@ static struct mem_cgroup *mem_cgroup_alloc(void) goto fail; } + memcg->vmstats = kzalloc(sizeof(struct memcg_vmstats), GFP_KERNEL); + if (!memcg->vmstats) + goto fail; + memcg->vmstats_percpu = alloc_percpu_gfp(struct memcg_vmstats_percpu, GFP_KERNEL_ACCOUNT); if (!memcg->vmstats_percpu) @@ -5418,9 +5457,9 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) * below us. We're in a per-cpu loop here and this is * a global counter, so the first cycle will get them. */ - delta = memcg->vmstats.state_pending[i]; + delta = memcg->vmstats->state_pending[i]; if (delta) - memcg->vmstats.state_pending[i] = 0; + memcg->vmstats->state_pending[i] = 0; /* Add CPU changes on this level since the last flush */ v = READ_ONCE(statc->state[i]); @@ -5433,15 +5472,15 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) continue; /* Aggregate counts on this level and propagate upwards */ - memcg->vmstats.state[i] += delta; + memcg->vmstats->state[i] += delta; if (parent) - parent->vmstats.state_pending[i] += delta; + parent->vmstats->state_pending[i] += delta; } for (i = 0; i < NR_VM_EVENT_ITEMS; i++) { - delta = memcg->vmstats.events_pending[i]; + delta = memcg->vmstats->events_pending[i]; if (delta) - memcg->vmstats.events_pending[i] = 0; + memcg->vmstats->events_pending[i] = 0; v = READ_ONCE(statc->events[i]); if (v != statc->events_prev[i]) { @@ -5452,9 +5491,9 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) if (!delta) continue; - memcg->vmstats.events[i] += delta; + memcg->vmstats->events[i] += delta; if (parent) - parent->vmstats.events_pending[i] += delta; + parent->vmstats->events_pending[i] += delta; } for_each_node_state(nid, N_MEMORY) { From patchwork Wed Sep 7 04:35:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 12968431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FCF5C38145 for ; Wed, 7 Sep 2022 04:36:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 34E748D0005; Wed, 7 Sep 2022 00:36:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2FE7D6B0073; Wed, 7 Sep 2022 00:36:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C6058D0005; Wed, 7 Sep 2022 00:36:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0F9776B0072 for ; Wed, 7 Sep 2022 00:36:19 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E14BB1A098B for ; Wed, 7 Sep 2022 04:36:18 +0000 (UTC) X-FDA: 79884027636.06.10F92F5 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf24.hostedemail.com (Postfix) with ESMTP id A21C4180099 for ; Wed, 7 Sep 2022 04:36:18 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id l12-20020a25ad4c000000b006a8e04c284dso6748903ybe.11 for ; Tue, 06 Sep 2022 21:36:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=7xWB4XnyIc05hqp29+Ov/me4j17Vvs2h2ECTnGXauQA=; b=SDmUz9LkUVm/33cr2fKGG4SmxGsrw9fW609UlO1Lh2n9kJAvfCfnCBYoBieiLK1YI4 MPXnoJKs7HxJOBMeo211kZrqk641hONA1Abc3kl7lVLPSE2aHCTST/ZAGe/af7Ia1j+Y HSkzyUCEB5QeLvw41TwrGd/TwUv4YIM4WP861pR3jQm7dOBwu7Q0zlxU3M2K1wwpBOpB CpJIkspZw9r5LITpRw9Xg679OMVQjNafq6M/iV99mdZ3p+Qr3ZpmS2S+7O4Mi0TM+q+w DxfU/ss15LI61VWwYkkQPDsa/qz7JkfaqkZ89EbLkvsgKRetTJhSaS1Vm/9WvgNffImX t82A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=7xWB4XnyIc05hqp29+Ov/me4j17Vvs2h2ECTnGXauQA=; b=Z+PluvADz/S0j5tXEFz5lZ2GdB57SosEiATL1FjUunaWxPckw9VgRMfdvlNBtab3am Aao+oUkmTyjyN2AbFOPplzFI3lER5W6dqXSN1icjhj0xYvvpN4AovKUgXxByzesTQdC2 Y7R3TovEoUpFW+0FPrdmpIshc+A0gIjR5tzusE0eAg+0wpZPO5ZctHeXQFqURzj5HcwO e4jeb2i8PSEZ2ltXYNJO+Vpli3/l7OmEY07HJzAdxlcaFWK5ANwCpsRokVSqQI6k7wcO Q/KtCSES/UUIbsq38xlPi4ZgfoJNMbdhrif7I0NOQPLbNi3hB2/kWzgnd0CF8Zl165gg SCTA== X-Gm-Message-State: ACgBeo0yN+vQR9pxNf6RBb55yYa/xug2bNQkH8nGn8sBYUsxJT8P/dkK ylKGCWud7QVmv1BW/sfcfRAyFmj0yQV1DQ== X-Google-Smtp-Source: AA6agR41qApOeAsNTHvyqJlEKDWWr4jqOOSYR9bBI0ISwpTGkM58sXdfS6K35j4AxALrx//Yb7zv/515y/Evow== X-Received: from shakeelb.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:262e]) (user=shakeelb job=sendgmr) by 2002:a25:bb44:0:b0:677:24c:308f with SMTP id b4-20020a25bb44000000b00677024c308fmr1554313ybk.433.1662525377841; Tue, 06 Sep 2022 21:36:17 -0700 (PDT) Date: Wed, 7 Sep 2022 04:35:36 +0000 In-Reply-To: <20220907043537.3457014-1-shakeelb@google.com> Mime-Version: 1.0 References: <20220907043537.3457014-1-shakeelb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220907043537.3457014-3-shakeelb@google.com> Subject: [PATCH 2/3] memcg: rearrange code From: Shakeel Butt To: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song Cc: Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Shakeel Butt ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662525378; a=rsa-sha256; cv=none; b=WgKC+qHdgphh0ZXITd78yVO79kmC3JhFLBvVskzq6NW4EzJOASzb69B3VI/0lIihYX7QlU vjRTPQwLgIMNZMIxCJiLuybNsh7iHj/Um7cQAH4N/c7bEq27T0j8X3Bm3eXlGX9iylyjCI hSj3udpx358lqtFy98rmLfNrMLieeMY= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=SDmUz9Lk; spf=pass (imf24.hostedemail.com: domain of 3wR8YYwgKCIAwleoiipfksskpi.gsqpmry1-qqozego.svk@flex--shakeelb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3wR8YYwgKCIAwleoiipfksskpi.gsqpmry1-qqozego.svk@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662525378; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7xWB4XnyIc05hqp29+Ov/me4j17Vvs2h2ECTnGXauQA=; b=FqLMrpGmMu3lP/LiTrjiLnYbexoDa411XMhPv3slMh+iIGEDr9JVlQQl6G1+RumfbOfUmP kiuU+RBzlhu0+kxuOPqCAG1e+jG+MUgeXikX6nnaTbS2Mb+dgQLWN40m2HUrJky8KIXFeY u87l2K40zYESJygGlnCs9QpqjERfepA= X-Stat-Signature: 91y8z9wtsxedkr8y8ippd8po5qhyi8ms X-Rspamd-Queue-Id: A21C4180099 Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=SDmUz9Lk; spf=pass (imf24.hostedemail.com: domain of 3wR8YYwgKCIAwleoiipfksskpi.gsqpmry1-qqozego.svk@flex--shakeelb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3wR8YYwgKCIAwleoiipfksskpi.gsqpmry1-qqozego.svk@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam11 X-Rspam-User: X-HE-Tag: 1662525378-137795 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparatory patch for easing the review of the follow up patch which will reduce the memory overhead of memory cgroups. Signed-off-by: Shakeel Butt --- mm/memcontrol.c | 46 +++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b195d4ca2a72..d0ccc16ed416 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -669,6 +669,29 @@ static void flush_memcg_stats_dwork(struct work_struct *w) queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } +/* Subset of vm_event_item to report for memcg event stats */ +static const unsigned int memcg_vm_event_stat[] = { + PGSCAN_KSWAPD, + PGSCAN_DIRECT, + PGSTEAL_KSWAPD, + PGSTEAL_DIRECT, + PGFAULT, + PGMAJFAULT, + PGREFILL, + PGACTIVATE, + PGDEACTIVATE, + PGLAZYFREE, + PGLAZYFREED, +#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP) + ZSWPIN, + ZSWPOUT, +#endif +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + THP_FAULT_ALLOC, + THP_COLLAPSE_ALLOC, +#endif +}; + struct memcg_vmstats_percpu { /* Local (CPU and cgroup) page state & events */ long state[MEMCG_NR_STAT]; @@ -1501,29 +1524,6 @@ static inline unsigned long memcg_page_state_output(struct mem_cgroup *memcg, return memcg_page_state(memcg, item) * memcg_page_state_unit(item); } -/* Subset of vm_event_item to report for memcg event stats */ -static const unsigned int memcg_vm_event_stat[] = { - PGSCAN_KSWAPD, - PGSCAN_DIRECT, - PGSTEAL_KSWAPD, - PGSTEAL_DIRECT, - PGFAULT, - PGMAJFAULT, - PGREFILL, - PGACTIVATE, - PGDEACTIVATE, - PGLAZYFREE, - PGLAZYFREED, -#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP) - ZSWPIN, - ZSWPOUT, -#endif -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - THP_FAULT_ALLOC, - THP_COLLAPSE_ALLOC, -#endif -}; - static void memory_stat_format(struct mem_cgroup *memcg, char *buf, int bufsize) { struct seq_buf s; From patchwork Wed Sep 7 04:35:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 12968432 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9F6BECAAD3 for ; Wed, 7 Sep 2022 04:36:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 65B806B0072; Wed, 7 Sep 2022 00:36:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 60B978D0007; Wed, 7 Sep 2022 00:36:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AC3B8D0006; Wed, 7 Sep 2022 00:36:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 36DE56B0072 for ; Wed, 7 Sep 2022 00:36:27 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F1651140BBC for ; Wed, 7 Sep 2022 04:36:26 +0000 (UTC) X-FDA: 79884027972.27.80D6346 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf04.hostedemail.com (Postfix) with ESMTP id 927684007B for ; Wed, 7 Sep 2022 04:36:25 +0000 (UTC) Received: by mail-pg1-f201.google.com with SMTP id m188-20020a633fc5000000b00434dccacd4aso944127pga.10 for ; Tue, 06 Sep 2022 21:36:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=khSX7Y4iv18onStJMmBrPhwF58VNn8i03Anirden96Q=; b=gemiZCbExV47bKx8wOek1aQ0u8hzbt+EnZzyGvpEX6V3hdl7+FzjhOMQFdg/BAePWs g1Phtex5qyjXcE0CH8qo/myIMYCKBorj4O/lYfXNAuZQE1nIXh6Y/UorToCbyrfqTc08 w41m5Q1yfn7rEabS42x1kF04pr4Fz4DhfpdBQ7PAZvq2tOJI4kuvrPxvdDoYGIeFXtU7 73wZp+rS4GUGDo8QZ9/BnYm7kUNxwjCsKt+VQJT//oAjo2bR/bpBZ7YFQnfETArjdfst yH5HuLvanOXbdhRwhVJR8If5nQVhIbTM14rXsNmexTs4BoP70k0Ts+2cN/b8E1sehylv kQmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=khSX7Y4iv18onStJMmBrPhwF58VNn8i03Anirden96Q=; b=YqZ6OEZLvS0EKowzp4eyV9S5MEb6JLtd06Run1rp5SH14kKvtX46GzR9kVkwmJsULp Pl3znkG0lLavaQMXbSdIrOkx/q9YTl6I4TmuoFlpxY7c2oLTWrSHcL39W6YKXfyiApbw ebnywLhmGTP1q6QUPqIWeI53czr6mQ1ljk2yT33RzNJO/udQR/2wWxV3ZX/WPLAFEGBu LORIZ36mVX8n3k81iRTA2bkEWUAtPTvJJ0BpvGyyj186zXmktwPu61Nn6Dl6FHGoDvUb IUL3xO57pnoU49i+QfMApTrdjc7guCdI9ITfx+m4asdRZ+635BvQ6iBCjMKlXYJr0Gbf 56Rw== X-Gm-Message-State: ACgBeo0GQuqcmahv9I4mDc7vThj3I4TqG1Y63cyGPwhJZydOUqbPAB85 SeRRWU9P2rz7vWOieBnUhgKtKtoZ3wUeBw== X-Google-Smtp-Source: AA6agR5/ynpD4fcrcx0c8sHHlPJFfJQPloK/P5POaEbeKl2ov/6eY/SmFki7BrZyFFL0jRWVJtkjvAuQBWyeJQ== X-Received: from shakeelb.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:262e]) (user=shakeelb job=sendgmr) by 2002:a17:903:40c9:b0:176:e58c:f082 with SMTP id t9-20020a17090340c900b00176e58cf082mr1969898pld.60.1662525384705; Tue, 06 Sep 2022 21:36:24 -0700 (PDT) Date: Wed, 7 Sep 2022 04:35:37 +0000 In-Reply-To: <20220907043537.3457014-1-shakeelb@google.com> Mime-Version: 1.0 References: <20220907043537.3457014-1-shakeelb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220907043537.3457014-4-shakeelb@google.com> Subject: [PATCH 3/3] memcg: reduce size of memcg vmstats structures From: Shakeel Butt To: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song Cc: Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Shakeel Butt ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=gemiZCbE; spf=pass (imf04.hostedemail.com: domain of 3yB8YYwgKCIc3slvppwmrzzrwp.nzxwty58-xxv6lnv.z2r@flex--shakeelb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3yB8YYwgKCIc3slvppwmrzzrwp.nzxwty58-xxv6lnv.z2r@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662525385; a=rsa-sha256; cv=none; b=XQ66Hxb9YcWgW0tl5+FvRhccr0caIX1MfhHuZOUKNSqktWZ0QBpxb8zXCbR7kwtJWHQvSJ L9A1AANAv3H11UxdLpIOorYKZqzcuI8DS6z7XnqFLXlhqhPDhNb9L9qW2R52tUiN1evZIg azYamXEv8ZwK9bcib78xd2s29+B2GpA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662525385; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=khSX7Y4iv18onStJMmBrPhwF58VNn8i03Anirden96Q=; b=ks0iOFFM1dGf/tTJSiF8pxjz8EzmzGR11+7bDNRE1WRSgzCrK0vs5HhUGFfTF9QpP/WIeS jLcsqqCmCXa26ICKI+Sr7qKLVhUte+GTZPLeKhWTz/QLR2HgHRyXnldBfhh3DhpQJZG3lA lv99XPFN5dmF5Kgr5opQL03LeEbyCAI= Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=gemiZCbE; spf=pass (imf04.hostedemail.com: domain of 3yB8YYwgKCIc3slvppwmrzzrwp.nzxwty58-xxv6lnv.z2r@flex--shakeelb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3yB8YYwgKCIc3slvppwmrzzrwp.nzxwty58-xxv6lnv.z2r@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: x5j4p15ioazgh7o79m5uprwtuaq16oy7 X-Rspamd-Queue-Id: 927684007B X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1662525385-277760 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The struct memcg_vmstats and struct memcg_vmstats_percpu contains two arrays each for events of size NR_VM_EVENT_ITEMS which can be as large as 110. However the memcg v1 only uses 4 of those while memcg v2 uses 15. The union of both is 17. On a 64 bit system, we are wasting approximately ((110 - 17) * 8 * 2) * (nr_cpus + 1) bytes which is significant on large machines. This patch reduces the size of the given structures by adding one indirection and only stores array of events which are actually used by the memcg code. With this patch, the size of memcg_vmstats has reduced from 2544 bytes to 1056 bytes while the size of memcg_vmstats_percpu has reduced from 2568 bytes to 1080 bytes. Signed-off-by: Shakeel Butt Acked-by: Roman Gushchin Signed-off-by: Andrew Morton --- mm/memcontrol.c | 52 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d0ccc16ed416..a60012be6140 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -671,6 +671,8 @@ static void flush_memcg_stats_dwork(struct work_struct *w) /* Subset of vm_event_item to report for memcg event stats */ static const unsigned int memcg_vm_event_stat[] = { + PGPGIN, + PGPGOUT, PGSCAN_KSWAPD, PGSCAN_DIRECT, PGSTEAL_KSWAPD, @@ -692,14 +694,30 @@ static const unsigned int memcg_vm_event_stat[] = { #endif }; +#define NR_MEMCG_EVENTS ARRAY_SIZE(memcg_vm_event_stat) +static int mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly; + +static void init_memcg_events(void) +{ + int i; + + for (i = 0; i < NR_MEMCG_EVENTS; ++i) + mem_cgroup_events_index[memcg_vm_event_stat[i]] = i + 1; +} + +static inline int memcg_events_index(enum vm_event_item idx) +{ + return mem_cgroup_events_index[idx] - 1; +} + struct memcg_vmstats_percpu { /* Local (CPU and cgroup) page state & events */ long state[MEMCG_NR_STAT]; - unsigned long events[NR_VM_EVENT_ITEMS]; + unsigned long events[NR_MEMCG_EVENTS]; /* Delta calculation for lockless upward propagation */ long state_prev[MEMCG_NR_STAT]; - unsigned long events_prev[NR_VM_EVENT_ITEMS]; + unsigned long events_prev[NR_MEMCG_EVENTS]; /* Cgroup1: threshold notifications & softlimit tree updates */ unsigned long nr_page_events; @@ -709,11 +727,11 @@ struct memcg_vmstats_percpu { struct memcg_vmstats { /* Aggregated (CPU and subtree) page state & events */ long state[MEMCG_NR_STAT]; - unsigned long events[NR_VM_EVENT_ITEMS]; + unsigned long events[NR_MEMCG_EVENTS]; /* Pending child counts during tree propagation */ long state_pending[MEMCG_NR_STAT]; - unsigned long events_pending[NR_VM_EVENT_ITEMS]; + unsigned long events_pending[NR_MEMCG_EVENTS]; }; unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) @@ -873,24 +891,34 @@ void __mod_lruvec_kmem_state(void *p, enum node_stat_item idx, int val) void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, unsigned long count) { - if (mem_cgroup_disabled()) + int index = memcg_events_index(idx); + + if (mem_cgroup_disabled() || index < 0) return; memcg_stats_lock(); - __this_cpu_add(memcg->vmstats_percpu->events[idx], count); + __this_cpu_add(memcg->vmstats_percpu->events[index], count); memcg_rstat_updated(memcg, count); memcg_stats_unlock(); } static unsigned long memcg_events(struct mem_cgroup *memcg, int event) { - return READ_ONCE(memcg->vmstats->events[event]); + int index = memcg_events_index(event); + + if (index < 0) + return 0; + return READ_ONCE(memcg->vmstats->events[index]); } static unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) { long x = 0; int cpu; + int index = memcg_events_index(event); + + if (index < 0) + return 0; for_each_possible_cpu(cpu) x += per_cpu(memcg->vmstats_percpu->events[event], cpu); @@ -1564,10 +1592,15 @@ static void memory_stat_format(struct mem_cgroup *memcg, char *buf, int bufsize) memcg_events(memcg, PGSTEAL_KSWAPD) + memcg_events(memcg, PGSTEAL_DIRECT)); - for (i = 0; i < ARRAY_SIZE(memcg_vm_event_stat); i++) + for (i = 0; i < ARRAY_SIZE(memcg_vm_event_stat); i++) { + if (memcg_vm_event_stat[i] == PGPGIN || + memcg_vm_event_stat[i] == PGPGOUT) + continue; + seq_buf_printf(&s, "%s %lu\n", vm_event_name(memcg_vm_event_stat[i]), memcg_events(memcg, memcg_vm_event_stat[i])); + } /* The above should easily fit into one page */ WARN_ON_ONCE(seq_buf_has_overflowed(&s)); @@ -5309,6 +5342,7 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) page_counter_init(&memcg->kmem, &parent->kmem); page_counter_init(&memcg->tcpmem, &parent->tcpmem); } else { + init_memcg_events(); page_counter_init(&memcg->memory, NULL); page_counter_init(&memcg->swap, NULL); page_counter_init(&memcg->kmem, NULL); @@ -5477,7 +5511,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) parent->vmstats->state_pending[i] += delta; } - for (i = 0; i < NR_VM_EVENT_ITEMS; i++) { + for (i = 0; i < NR_MEMCG_EVENTS; i++) { delta = memcg->vmstats->events_pending[i]; if (delta) memcg->vmstats->events_pending[i] = 0;