From patchwork Tue Feb 9 16:32:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 12078555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42C09C433E6 for ; Tue, 9 Feb 2021 16:33:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6EEAF64EBA for ; Tue, 9 Feb 2021 16:33:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6EEAF64EBA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F365B6B006C; Tue, 9 Feb 2021 11:33:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EF1756B006E; Tue, 9 Feb 2021 11:33:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DAF1D6B0070; Tue, 9 Feb 2021 11:33:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0039.hostedemail.com [216.40.44.39]) by kanga.kvack.org (Postfix) with ESMTP id C291F6B006C for ; Tue, 9 Feb 2021 11:33:29 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 84A4E180ACC2C for ; Tue, 9 Feb 2021 16:33:29 +0000 (UTC) X-FDA: 77799274938.03.C18063F Received: from mail-qv1-f51.google.com (mail-qv1-f51.google.com [209.85.219.51]) by imf07.hostedemail.com (Postfix) with ESMTP id D00AEA0000FA for ; Tue, 9 Feb 2021 16:33:27 +0000 (UTC) Received: by mail-qv1-f51.google.com with SMTP id ew18so8954479qvb.4 for ; Tue, 09 Feb 2021 08:33:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=XqwbjVKT92bgGmTwHneB7S6uqWbAlrDany7fE7oGT1Y=; b=wl5iSzgZjMYgWkqOkI6F1OA6EQOr0VBwWz+1jGJaqi9r0iUCCJMwGfNy2ZgOyJmrST vI9XWIPnkXj3pyoS9MC9dwZeK338IHOw9TivjHI1cpLlskJGdf5qqx53svtAV2vJ1MP+ 4gjuaJl6/GvvI9OK4j1vD+s8KCK/lEJ3wnl16G3olAhs+NxlVEHkzb/mrooKe86FFG86 K8jc53i5n0xi4ZZKuID1hBG8PLcyVmjfJ00zIKKsi8iZzFGEwYc8oDqm+IViUkuIzBHA TPd5DYSVcdaXjzcwLruGKXCJgCuxRWnF/I3rqeoMLw37Dnmv1MGK8blqtFqMuuXyLG9m I1IA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=XqwbjVKT92bgGmTwHneB7S6uqWbAlrDany7fE7oGT1Y=; b=ElJKm/XOiUShWLDX5HKkeQK8iKihe67YbAxfi6PuqJwMY3L358xHkZ8qYZWYR4a2k7 ZbQvLaCZlnIhZxEAb1uLEm1dY2M+CsV5Lu6Tpcq9PivJBU4+tzwivPrPJb2tDZmM+oBy jh71dLWzoG04xCB7f4NNMjGqo5FESpGcTqtyHgYpg8cDZi9oY0hYBvaGexg8GAMFZIH7 OSXWSoqVWqJfKnIGJyniHY2OinaqOSTvnOVJ8FvBFXpXd4lr/Qm9HEWMJaZR6GhXWi4e 8KQp46GP6oFddRs/hLDyHo90JSOiLGBYlgehet4dqK8v2MBLWptRN7YaD93R7MPPuvvo Vc0Q== X-Gm-Message-State: AOAM5337K5+1B/jOucRB/WhnA8SrYvzQMrBnTj+576BX8RyZiC5bjdzW Gvwd4AKDcQ9QpXyJNOPISKesKg== X-Google-Smtp-Source: ABdhPJx0wrSrroGhdi/SB2WSKmIinjBZ9twNIoDRE69zDqkSKMnz4qGrEtx2k+a0Kcyg4wM2ewXlmw== X-Received: by 2002:a0c:814e:: with SMTP id 72mr12762669qvc.16.1612888407771; Tue, 09 Feb 2021 08:33:27 -0800 (PST) Received: from localhost (70.44.39.90.res-cmts.bus.ptd.net. [70.44.39.90]) by smtp.gmail.com with ESMTPSA id h12sm18296262qko.29.2021.02.09.08.33.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 08:33:26 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Tejun Heo , Michal Hocko , Roman Gushchin , Shakeel Butt , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 0/8] mm: memcontrol: switch to rstat Date: Tue, 9 Feb 2021 11:32:56 -0500 Message-Id: <20210209163304.77088-1-hannes@cmpxchg.org> X-Mailer: git-send-email 2.30.0 MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D00AEA0000FA X-Stat-Signature: 8dcta5nin1x1o96nimqcw6hg6kohit4m Received-SPF: none (cmpxchg.org>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=mail-qv1-f51.google.com; client-ip=209.85.219.51 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1612888407-363628 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is version 3 of the memcg rstat patches. Updates since v2: - optimize root level in io controller's rstat flush callback (Tejun) - add comments and changelog notes on root level optimization (Tejun) - eliminate mem_cgroup_usage() rstat flush for leaf cgroups (Shakeel) - collected review & ack tags Updates since v1: - added cgroup selftest output (see test section below) (thanks Roman) - patched cgroup selftest so error expectations match kernel implementation - added Fixes: tag to 'mm: memcontrol: fix cpuhotplug statistics flushing' (Shakeel) - collected review & ack tags - added rstat overview to 'mm: memcontrol: switch to rstat' changelog (Michal) - simplified memcg_flush_lruvec_page_state() and removed cpu==-1 case (Michal) --- This series converts memcg stats tracking to the streamlined rstat infrastructure provided by the cgroup core code. rstat is already used by the CPU controller and the IO controller. This change is motivated by recent accuracy problems in memcg's custom stats code, as well as the benefits of sharing common infra with other controllers. The current memcg implementation does batched tree aggregation on the write side: local stat changes are cached in per-cpu counters, which are then propagated upward in batches when a threshold (32 pages) is exceeded. This is cheap, but the error introduced by the lazy upward propagation adds up: 32 pages times CPUs times cgroups in the subtree. We've had complaints from service owners that the stats do not reliably track and react to allocation behavior as expected, sometimes swallowing the results of entire test applications. The original memcg stat implementation used to do tree aggregation exclusively on the read side: local stats would only ever be tracked in per-cpu counters, and a memory.stat read would iterate the entire subtree and sum those counters up. This didn't keep up with the times: - Cgroup trees are much bigger now. We switched to lazily-freed cgroups, where deleted groups would hang around until their remaining page cache has been reclaimed. This can result in large subtrees that are expensive to walk, while most of the groups are idle and their statistics don't change much anymore. - Automated monitoring increased. With the proliferation of userspace oom killing, proactive reclaim, and higher-resolution logging of workload trends in general, top-level stat files are polled at least once a second in many deployments. - The lifetime of cgroups got shorter. Where most cgroup setups in the past would have a few large policy-oriented cgroups for everything running on the system, newer cgroup deployments tend to create one group per application - which gets deleted again as the processes exit. An aggregation scheme that doesn't retain child data inside the parents loses event history of the subtree. Rstat addresses all three of those concerns through intelligent, persistent read-side aggregation. As statistics change at the local level, rstat tracks - on a per-cpu basis - only those parts of a subtree that have changes pending and require aggregation. The actual aggregation occurs on the colder read side - which can now skip over (potentially large) numbers of recently idle cgroups. --- The test_kmem cgroup selftest is currently failing due to excessive cumulative vmstat drift from 100 subgroups: ok 1 test_kmem_basic memory.current = 8810496 slab + anon + file + kernel_stack = 17074568 slab = 6101384 anon = 946176 file = 0 kernel_stack = 10027008 not ok 2 test_kmem_memcg_deletion ok 3 test_kmem_proc_kpagecgroup ok 4 test_kmem_kernel_stacks ok 5 test_kmem_dead_cgroups ok 6 test_percpu_basic As you can see, memory.stat items far exceed memory.current. The kernel stack alone is bigger than all of charged memory. That's because the memory of the test has been uncharged from memory.current, but the negative vmstat deltas are still sitting in the percpu caches. The test at this time isn't even counting percpu, pagetables etc. yet, which would further contribute to the error. The last patch in the series updates the test to include them - as well as reduces the vmstat tolerances in general to only expect page_counter batching. With all patches applied, the (now more stringent) test succeeds: ok 1 test_kmem_basic ok 2 test_kmem_memcg_deletion ok 3 test_kmem_proc_kpagecgroup ok 4 test_kmem_kernel_stacks ok 5 test_kmem_dead_cgroups ok 6 test_percpu_basic --- A kernel build test confirms that overhead is comparable. Two kernels are built simultaneously in a nested tree with several idle siblings: root - kernelbuild - one - two - three - four - build-a (defconfig, make -j16) `- build-b (defconfig, make -j16) `- idle-1 `- ... `- idle-9 During the builds, kernelbuild/memory.stat is read once a second. A perf diff shows that the changes in cycle distribution is minimal. Top 10 kernel symbols: 0.09% +0.08% [kernel.kallsyms] [k] __mod_memcg_lruvec_state 0.00% +0.06% [kernel.kallsyms] [k] cgroup_rstat_updated 0.08% -0.05% [kernel.kallsyms] [k] __mod_memcg_state.part.0 0.16% -0.04% [kernel.kallsyms] [k] release_pages 0.00% +0.03% [kernel.kallsyms] [k] __count_memcg_events 0.01% +0.03% [kernel.kallsyms] [k] mem_cgroup_charge_statistics.constprop.0 0.10% -0.02% [kernel.kallsyms] [k] get_mem_cgroup_from_mm 0.05% -0.02% [kernel.kallsyms] [k] mem_cgroup_update_lru_size 0.57% +0.01% [kernel.kallsyms] [k] asm_exc_page_fault --- The on-demand aggregated stats are now fully accurate: $ grep -e nr_inactive_file /proc/vmstat | awk '{print($1,$2*4096)}'; \ grep -e inactive_file /sys/fs/cgroup/memory.stat vanilla: patched: nr_inactive_file 1574105088 nr_inactive_file 1027801088 inactive_file 1577410560 inactive_file 1027801088 --- block/blk-cgroup.c | 17 +- include/linux/memcontrol.h | 119 ++++------- kernel/cgroup/cgroup.c | 34 +-- kernel/cgroup/rstat.c | 63 +++--- mm/memcontrol.c | 305 +++++++++++++-------------- tools/testing/selftests/cgroup/test_kmem.c | 22 +- 6 files changed, 267 insertions(+), 293 deletions(-) Based on v5.11-rc6-mmotm-2021-02-02-20-19.