From patchwork Tue Feb 18 03:14:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 13978850 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56E9EC021A9 for ; Tue, 18 Feb 2025 03:15:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B19CA2800B1; Mon, 17 Feb 2025 22:15:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AC8E92800A4; Mon, 17 Feb 2025 22:15:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F5272800B1; Mon, 17 Feb 2025 22:15:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 676C62800A4 for ; Mon, 17 Feb 2025 22:15:05 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 298E1161848 for ; Tue, 18 Feb 2025 03:15:05 +0000 (UTC) X-FDA: 83131598970.28.D22BB0E Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf11.hostedemail.com (Postfix) with ESMTP id 498DF4000B for ; Tue, 18 Feb 2025 03:15:03 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=jGkTPzDq; spf=pass (imf11.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739848503; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LKC2oA1YaKWB10hzGh7Kd/lm3Yr8DEdtBJf40vcnIWI=; b=jqZ91fijZiqQ8QH8bZB0llMkVjRXj2dk0npebNCKM83EmR7jbVsX/I2LDakBDj3uab/QsE NFqDs9OcLIUobPz3evTnkwKQzwVnJnarsrXy0whe5iE0f/d3nT7aA4zzRU9i7kpAOJ1+0+ wk+1NARaWamjAIudYzeFyCzUldoA4OI= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=jGkTPzDq; spf=pass (imf11.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739848503; a=rsa-sha256; cv=none; b=o3ZhwuejPhSfNEp8q+zxrRTN/9+VQuOFrBMQeN3ivuSZIL6SbSpUr1uj1GfhKXVqGxREuB rH10u7svKVWjtLwaZ37IBVjLqq94WoWYTPloRUP/5OiblIMd3ITZTEU2XEAbtoc8RLbvcl 95/KDUr+ssuN7Y+8OsacIeF9+Bl/E6o= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-221206dbd7eso31229855ad.2 for ; Mon, 17 Feb 2025 19:15:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739848502; x=1740453302; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LKC2oA1YaKWB10hzGh7Kd/lm3Yr8DEdtBJf40vcnIWI=; b=jGkTPzDqYpjv4n2glrfKEY7Wz5+6k3NjyJe93o6nMQF46nB+3RDT6tkXXBOLqHeRg3 se7jr1Ta5Qe0EkD0Rtb0fW/nh8zS65K4qfCQraS+PguKIM5mwI5nB1vL8qUOgE1jPlSf bHf9wuFqtdDsPxhM3S4bOIL+ZaG4FK9YMCmz//Puo3IY8vcMrTZsQ8I+QP2n793NCu0/ 4YmrDOkSSGmzeTkCj9ATK7EQn1IKeMTF0aNKEzm+Fjzu+qQwR1pD3Mjc8gTpl0xdWsZx EJD6KJAoe8z3x79NyI99Ibdta55QrGX1y1CYZaqKgEvwBa4aZ6ApPzOQxLq0GFvuGlKn R7hA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739848502; x=1740453302; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LKC2oA1YaKWB10hzGh7Kd/lm3Yr8DEdtBJf40vcnIWI=; b=GbGXJUuBmVxl4EL2chnZOJ3q/25qOBfAWXg7Ak8FVPZ1yzEL8qRui4Z3IMAJTgm9D+ 8Q+HnmNKxRVJOIXFb7EFwvfxFIqmT3+4ow9stX2Peb4nNEt0UnDbW/DAReSm0XnNkx3p VXgDocDbkavmtDnJWLBlZgT9P8LLH72bx7GJkvjt5RU3NwR4InGCq1nVFZ+w5eshS0uz VsJw5PPvnWzEXe0+PVmnQDXAdfTuzODMVcqJe06wQkXi3j6YySO8vCA0aS46pMvFPFj/ F/ZMLD5Xp2oUAMrYok8AA3yrzrsYQHdKFFIsRhr4Y42GYNEBhWEcP/SAmPA87xf4RR0o yQOg== X-Gm-Message-State: AOJu0YxBeYJllsukm/N8B/LxAzqSZsWcxHts7Qn2aU5mHx0jlmRdbj5g JzAUatAJVWJMURDawr4kbZJb6sErE0RRj1lBJWUk5xhTRY4Fu0Ps X-Gm-Gg: ASbGnctowyVZuSpYs7kQXVFRyRXfLT8kQz+O0d81L4YJdmr9MO+McB+G3DJ5+UASNLM dJv5G0qqqgdMqdaLnZvGsknv1fVat9VMkiBpZ2TbY9YHhfd5euntIDZBCq/7PoAhbVlFLUyBOWt 6kK6f+14UrhpMKimbIEYwCI6RGqMaQLQeX3Gvk0iL6AH3ehO8ZZcxWMkPgYdY4wiT9QSofhJxe6 7KLl6xjN8RkiLan9tGDcH/ROsekMAD5h7fZClqliHA0LN7N9cJFwQgAfboP57pRGQx7VPrnru3c baRHgA3k3Df/IMMHQp46c+P9BReyxaAs+m/wq8eVPnhcewPv0EW6 X-Google-Smtp-Source: AGHT+IF+DcmK8IyN/NoZE/SgRoHiYX03btr3kUIPJKJtKWJG3YwfRKa/liO8vC71Fr0zQV87n/GPlg== X-Received: by 2002:a05:6a21:6b05:b0:1ee:5f3f:cde0 with SMTP id adf61e73a8af0-1ee8cbf7bcamr24262982637.34.1739848502002; Mon, 17 Feb 2025 19:15:02 -0800 (PST) Received: from saturn.. (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732425466besm8763451b3a.9.2025.02.17.19.14.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Feb 2025 19:15:01 -0800 (PST) From: JP Kobryn To: shakeel.butt@linux.dev, tj@kernel.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 01/11] cgroup: move rstat pointers into struct of their own Date: Mon, 17 Feb 2025 19:14:38 -0800 Message-ID: <20250218031448.46951-2-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250218031448.46951-1-inwardvessel@gmail.com> References: <20250218031448.46951-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 498DF4000B X-Stat-Signature: 4f8fpuc1h1xcqqz9ftrnbghhi3issymw X-HE-Tag: 1739848503-350894 X-HE-Meta: U2FsdGVkX1+xQcaAIBCz74tXlwbVBK0ay3s/QhTsvEVjVeWXvCyhAu9dl0lvPIzRvLKrfxQjR7oQA16W3ONx3rSIfK5Cpiv/gU1IMHceYU3PxtGDX4pfahSSEbBCRqhCOhzaLbLhf0ggTyUpbMDwBkw0uQx9OEBEy9DcXa8LF42nqp/UkKc74KA1EFolmBlbXtvCd5fpw5H27KSHjz1YM2KC967z7LIY7IHSclW1bqQ1R7VcjnzgIdywYETJ8TAKHs1ve5XkX8EqobGH1PkwhVnM1IZvQOiNppnV1g42f7mFHr2SvuInALs3EMicadrNOxJ1KPKBgYLmswvzYlkwt8FR3o6T7YHG/SeWpecSCakoYv9/DUN+5RsD4NmP+QZOJrXMgPO5ES1NJr90D8pG7Dz5PGi/RSABTpQFYVgUQRoDcD7wxpedj7ZMu/yKW1GPfGc+uvrTlmO7mnSJGqsMRHHtducjFSRWP2e+E4S9v2PHLf0KesbpcSepqPehL9Rr+Xb6IbkW0gJpSMTbiJS7rVaocBDn9gLqAlY3EmhHAmVkwn9v8Up3esmAtdEsbZATUY6KaE9lPfKeIAhG+S88FIB1HSyar80iWhv44zZcwn58yI8+9aR7cxBTkZIOBsw8JH44mRSC0xpw5bIAeg7AOeKDd9/CIOWW2Bcx7SlXyGj+OJhiKPNFxn6KwxvRE/B8ln7fMSKE/f9v1EfFA4yYrn3MwQk738IeQdTg6pUcaVLVmFTZV9b/GW0wJOUvjQVxSUUIzI8fFyn7JNZzLeSuDQ6b/dbvrVE7u40Ht/giKpaVO6V2wK6/IPr1nFJ0N527kH0toNPH8ble7B3ODo/utotvoHuhYsx86ZfyLA06DF+WJCfaAjMvOIQ1iiHC8Xdhneh1RQQ0+C1RMvppgPYxuQ8CMaon3FmNigrcnat/jt4eR8TjQ4kkgq60whdXjfktRvlvCqcraae7QWK4IiX 6tnB9NfM yjBPS5TP3tiG3x8LWFhYJ4zi8wS7nqifHTzFLyUWkLlm4PbZjgPZFKnCc5rEUpAUrY8kdNYCWi8Ei0SN0MKWr234orXCi4wTxRP9T8cDANwNNDeanMQ8qRw2dy7+RH+55cntvEWEcIKB5B/2CEbYEETfb4kNfSH35HL6i7z1MhYRw7PNiuBg/DLSzsbSOBobxyCm8QKL/8FWvT/8hiXNRjlkhY+CgtJux1BJoPTtqK7RdDUaAnWbUblZRW8MI4WQJCLvquoORALJMM4omasxiF9T7A4TwUfMPxTtZmXd9Wu2NzFIGZgttXOKvUIJ6h+2gPZUnUkILlvltpKHGsWKn0xcTV4dUk72rRu7327kiNqeViN5IEjNaAshx/w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The rstat infrastructure makes use of pointers for list management. These pointers only exist as fields in the cgroup struct, so moving them into their own struct will allow them to be used elsewhere. The base stat entities are included with them for now. Signed-off-by: JP Kobryn --- include/linux/cgroup-defs.h | 90 +----------------- include/linux/cgroup_rstat.h | 92 +++++++++++++++++++ kernel/cgroup/cgroup.c | 3 +- kernel/cgroup/rstat.c | 27 +++--- .../selftests/bpf/progs/btf_type_tag_percpu.c | 4 +- 5 files changed, 112 insertions(+), 104 deletions(-) create mode 100644 include/linux/cgroup_rstat.h diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index 1b20d2d8ef7c..6b6cc027fe70 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -17,7 +17,7 @@ #include #include #include -#include +#include #include #include #include @@ -321,78 +321,6 @@ struct css_set { struct rcu_head rcu_head; }; -struct cgroup_base_stat { - struct task_cputime cputime; - -#ifdef CONFIG_SCHED_CORE - u64 forceidle_sum; -#endif - u64 ntime; -}; - -/* - * rstat - cgroup scalable recursive statistics. Accounting is done - * per-cpu in cgroup_rstat_cpu which is then lazily propagated up the - * hierarchy on reads. - * - * When a stat gets updated, the cgroup_rstat_cpu and its ancestors are - * linked into the updated tree. On the following read, propagation only - * considers and consumes the updated tree. This makes reading O(the - * number of descendants which have been active since last read) instead of - * O(the total number of descendants). - * - * This is important because there can be a lot of (draining) cgroups which - * aren't active and stat may be read frequently. The combination can - * become very expensive. By propagating selectively, increasing reading - * frequency decreases the cost of each read. - * - * This struct hosts both the fields which implement the above - - * updated_children and updated_next - and the fields which track basic - * resource statistics on top of it - bsync, bstat and last_bstat. - */ -struct cgroup_rstat_cpu { - /* - * ->bsync protects ->bstat. These are the only fields which get - * updated in the hot path. - */ - struct u64_stats_sync bsync; - struct cgroup_base_stat bstat; - - /* - * Snapshots at the last reading. These are used to calculate the - * deltas to propagate to the global counters. - */ - struct cgroup_base_stat last_bstat; - - /* - * This field is used to record the cumulative per-cpu time of - * the cgroup and its descendants. Currently it can be read via - * eBPF/drgn etc, and we are still trying to determine how to - * expose it in the cgroupfs interface. - */ - struct cgroup_base_stat subtree_bstat; - - /* - * Snapshots at the last reading. These are used to calculate the - * deltas to propagate to the per-cpu subtree_bstat. - */ - struct cgroup_base_stat last_subtree_bstat; - - /* - * Child cgroups with stat updates on this cpu since the last read - * are linked on the parent's ->updated_children through - * ->updated_next. - * - * In addition to being more compact, singly-linked list pointing - * to the cgroup makes it unnecessary for each per-cpu struct to - * point back to the associated cgroup. - * - * Protected by per-cpu cgroup_rstat_cpu_lock. - */ - struct cgroup *updated_children; /* terminated by self cgroup */ - struct cgroup *updated_next; /* NULL iff not on the list */ -}; - struct cgroup_freezer_state { /* Should the cgroup and its descendants be frozen. */ bool freeze; @@ -517,23 +445,9 @@ struct cgroup { struct cgroup *old_dom_cgrp; /* used while enabling threaded */ /* per-cpu recursive resource statistics */ - struct cgroup_rstat_cpu __percpu *rstat_cpu; + struct cgroup_rstat rstat; struct list_head rstat_css_list; - /* - * Add padding to separate the read mostly rstat_cpu and - * rstat_css_list into a different cacheline from the following - * rstat_flush_next and *bstat fields which can have frequent updates. - */ - CACHELINE_PADDING(_pad_); - - /* - * A singly-linked list of cgroup structures to be rstat flushed. - * This is a scratch field to be used exclusively by - * cgroup_rstat_flush_locked() and protected by cgroup_rstat_lock. - */ - struct cgroup *rstat_flush_next; - /* cgroup basic resource statistics */ struct cgroup_base_stat last_bstat; struct cgroup_base_stat bstat; diff --git a/include/linux/cgroup_rstat.h b/include/linux/cgroup_rstat.h new file mode 100644 index 000000000000..f95474d6f8ab --- /dev/null +++ b/include/linux/cgroup_rstat.h @@ -0,0 +1,92 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_RSTAT_H +#define _LINUX_RSTAT_H + +#include + +struct cgroup_rstat_cpu; + +/* + * rstat - cgroup scalable recursive statistics. Accounting is done + * per-cpu in cgroup_rstat_cpu which is then lazily propagated up the + * hierarchy on reads. + * + * When a stat gets updated, the cgroup_rstat_cpu and its ancestors are + * linked into the updated tree. On the following read, propagation only + * considers and consumes the updated tree. This makes reading O(the + * number of descendants which have been active since last read) instead of + * O(the total number of descendants). + * + * This is important because there can be a lot of (draining) cgroups which + * aren't active and stat may be read frequently. The combination can + * become very expensive. By propagating selectively, increasing reading + * frequency decreases the cost of each read. + * + * This struct hosts both the fields which implement the above - + * updated_children and updated_next - and the fields which track basic + * resource statistics on top of it - bsync, bstat and last_bstat. + */ +struct cgroup_rstat { + struct cgroup_rstat_cpu __percpu *rstat_cpu; + + /* + * Add padding to separate the read mostly rstat_cpu and + * rstat_css_list into a different cacheline from the following + * rstat_flush_next and containing struct fields which can have + * frequent updates. + */ + CACHELINE_PADDING(_pad_); + struct cgroup *rstat_flush_next; +}; + +struct cgroup_base_stat { + struct task_cputime cputime; + +#ifdef CONFIG_SCHED_CORE + u64 forceidle_sum; +#endif + u64 ntime; +}; + +struct cgroup_rstat_cpu { + /* + * Child cgroups with stat updates on this cpu since the last read + * are linked on the parent's ->updated_children through + * ->updated_next. + * + * In addition to being more compact, singly-linked list pointing + * to the cgroup makes it unnecessary for each per-cpu struct to + * point back to the associated cgroup. + */ + struct cgroup *updated_children; /* terminated by self */ + struct cgroup *updated_next; /* NULL if not on the list */ + + /* + * ->bsync protects ->bstat. These are the only fields which get + * updated in the hot path. + */ + struct u64_stats_sync bsync; + struct cgroup_base_stat bstat; + + /* + * Snapshots at the last reading. These are used to calculate the + * deltas to propagate to the global counters. + */ + struct cgroup_base_stat last_bstat; + + /* + * This field is used to record the cumulative per-cpu time of + * the cgroup and its descendants. Currently it can be read via + * eBPF/drgn etc, and we are still trying to determine how to + * expose it in the cgroupfs interface. + */ + struct cgroup_base_stat subtree_bstat; + + /* + * Snapshots at the last reading. These are used to calculate the + * deltas to propagate to the per-cpu subtree_bstat. + */ + struct cgroup_base_stat last_subtree_bstat; +}; + +#endif /* _LINUX_RSTAT_H */ diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index d9061bd55436..03a3a4da49f1 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -31,6 +31,7 @@ #include "cgroup-internal.h" #include +#include #include #include #include @@ -164,7 +165,7 @@ static struct static_key_true *cgroup_subsys_on_dfl_key[] = { static DEFINE_PER_CPU(struct cgroup_rstat_cpu, cgrp_dfl_root_rstat_cpu); /* the default hierarchy */ -struct cgroup_root cgrp_dfl_root = { .cgrp.rstat_cpu = &cgrp_dfl_root_rstat_cpu }; +struct cgroup_root cgrp_dfl_root = { .cgrp.rstat.rstat_cpu = &cgrp_dfl_root_rstat_cpu }; EXPORT_SYMBOL_GPL(cgrp_dfl_root); /* diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 5877974ece92..7e7879d88c38 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -16,7 +16,7 @@ static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu); static struct cgroup_rstat_cpu *cgroup_rstat_cpu(struct cgroup *cgrp, int cpu) { - return per_cpu_ptr(cgrp->rstat_cpu, cpu); + return per_cpu_ptr(cgrp->rstat.rstat_cpu, cpu); } /* @@ -149,24 +149,24 @@ static struct cgroup *cgroup_rstat_push_children(struct cgroup *head, struct cgroup *parent, *grandchild; struct cgroup_rstat_cpu *crstatc; - child->rstat_flush_next = NULL; + child->rstat.rstat_flush_next = NULL; next_level: while (chead) { child = chead; - chead = child->rstat_flush_next; + chead = child->rstat.rstat_flush_next; parent = cgroup_parent(child); /* updated_next is parent cgroup terminated */ while (child != parent) { - child->rstat_flush_next = head; + child->rstat.rstat_flush_next = head; head = child; crstatc = cgroup_rstat_cpu(child, cpu); grandchild = crstatc->updated_children; if (grandchild != child) { /* Push the grand child to the next level */ crstatc->updated_children = child; - grandchild->rstat_flush_next = ghead; + grandchild->rstat.rstat_flush_next = ghead; ghead = grandchild; } child = crstatc->updated_next; @@ -238,7 +238,7 @@ static struct cgroup *cgroup_rstat_updated_list(struct cgroup *root, int cpu) /* Push @root to the list first before pushing the children */ head = root; - root->rstat_flush_next = NULL; + root->rstat.rstat_flush_next = NULL; child = rstatc->updated_children; rstatc->updated_children = root; if (child != root) @@ -310,7 +310,7 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp) for_each_possible_cpu(cpu) { struct cgroup *pos = cgroup_rstat_updated_list(cgrp, cpu); - for (; pos; pos = pos->rstat_flush_next) { + for (; pos; pos = pos->rstat.rstat_flush_next) { struct cgroup_subsys_state *css; cgroup_base_stat_flush(pos, cpu); @@ -387,9 +387,10 @@ int cgroup_rstat_init(struct cgroup *cgrp) int cpu; /* the root cgrp has rstat_cpu preallocated */ - if (!cgrp->rstat_cpu) { - cgrp->rstat_cpu = alloc_percpu(struct cgroup_rstat_cpu); - if (!cgrp->rstat_cpu) + if (!cgrp->rstat.rstat_cpu) { + cgrp->rstat.rstat_cpu = alloc_percpu( + struct cgroup_rstat_cpu); + if (!cgrp->rstat.rstat_cpu) return -ENOMEM; } @@ -419,8 +420,8 @@ void cgroup_rstat_exit(struct cgroup *cgrp) return; } - free_percpu(cgrp->rstat_cpu); - cgrp->rstat_cpu = NULL; + free_percpu(cgrp->rstat.rstat_cpu); + cgrp->rstat.rstat_cpu = NULL; } void __init cgroup_rstat_boot(void) @@ -503,7 +504,7 @@ cgroup_base_stat_cputime_account_begin(struct cgroup *cgrp, unsigned long *flags { struct cgroup_rstat_cpu *rstatc; - rstatc = get_cpu_ptr(cgrp->rstat_cpu); + rstatc = get_cpu_ptr(cgrp->rstat.rstat_cpu); *flags = u64_stats_update_begin_irqsave(&rstatc->bsync); return rstatc; } diff --git a/tools/testing/selftests/bpf/progs/btf_type_tag_percpu.c b/tools/testing/selftests/bpf/progs/btf_type_tag_percpu.c index 38f78d9345de..035412265c3c 100644 --- a/tools/testing/selftests/bpf/progs/btf_type_tag_percpu.c +++ b/tools/testing/selftests/bpf/progs/btf_type_tag_percpu.c @@ -45,7 +45,7 @@ int BPF_PROG(test_percpu2, struct bpf_testmod_btf_type_tag_2 *arg) SEC("tp_btf/cgroup_mkdir") int BPF_PROG(test_percpu_load, struct cgroup *cgrp, const char *path) { - g = (__u64)cgrp->rstat_cpu->updated_children; + g = (__u64)cgrp->rstat.rstat_cpu->updated_children; return 0; } @@ -56,7 +56,7 @@ int BPF_PROG(test_percpu_helper, struct cgroup *cgrp, const char *path) __u32 cpu; cpu = bpf_get_smp_processor_id(); - rstat = (struct cgroup_rstat_cpu *)bpf_per_cpu_ptr(cgrp->rstat_cpu, cpu); + rstat = (struct cgroup_rstat_cpu *)bpf_per_cpu_ptr(cgrp->rstat.rstat_cpu, cpu); if (rstat) { /* READ_ONCE */ *(volatile int *)rstat; From patchwork Tue Feb 18 03:14:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 13978851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3536DC021AB for ; Tue, 18 Feb 2025 03:15:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB5072800B5; Mon, 17 Feb 2025 22:15:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B648B2800A4; Mon, 17 Feb 2025 22:15:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 990192800B5; Mon, 17 Feb 2025 22:15:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6D1622800A4 for ; Mon, 17 Feb 2025 22:15:07 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DCC424FFC9 for ; Tue, 18 Feb 2025 03:15:06 +0000 (UTC) X-FDA: 83131599012.21.D30A2A5 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf07.hostedemail.com (Postfix) with ESMTP id F2CE540005 for ; Tue, 18 Feb 2025 03:15:04 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ACK0Al0P; spf=pass (imf07.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739848505; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HgpO8b1EeRoUbQ9L53xVbhgb2e+wvzP8W/zHZ0N0bI0=; b=c1bPjQQZTyfMVik1gLE1A0/dkRwu5cAkvqgh85pGP0NqmdZOiip/bFIWbHRVPCsj39VDuo CD+xfCM6CsJpNgE8LUMPpSWexnqzSnRjZ7u9t6VLFMybYh6tc/l1ny4+LZTevaixS4YDD0 Q1530zzUvU+4bo5SdRLScafhXgKSk9E= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ACK0Al0P; spf=pass (imf07.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739848505; a=rsa-sha256; cv=none; b=SychkcQ9H/1bqIOcOu75J055wU/4LmeENnuNIv8j2wq8JrGr1ywHBd6VK/iAQ00F7aYMrH JvdLETlX23gchc8TN+Vq95mq+ArH5gVps2JgS0dVqLBFFs8EHZZJ1imEfwIA9ol7T7Is8A fwyiJBharunBIVwUFmas35hHIQXlRQI= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-220dc3831e3so71140785ad.0 for ; Mon, 17 Feb 2025 19:15:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739848504; x=1740453304; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HgpO8b1EeRoUbQ9L53xVbhgb2e+wvzP8W/zHZ0N0bI0=; b=ACK0Al0PSMmg88P3M76M00r6sKQnTArARol+M9D5SuID+fZI5rHThs3cRvYbhlV4kL 2Qb4j3mTjHKVh3lhQ5diyCqVloxrbTc+P8Zp4MNsfbjs5m5vXPnREwovuNVPOrzNbl+e FQOuN5he7sD0wjy9uvoBJP5A6z3ntm4vTr+QofTMnpP/ClkdxW5uUsYDUySNPxA32tVD I8dja0SmO8v2OckmQtO76hNGW/K5fCtKXtEqd45hKeXV6qAZqjnaYy8MJVXGJyQhcqRD c0fwDT7PeuJ3VFqVwcA+lqNdw0M7JvADGoOS1Uen5Ngo2v7m3FBemOk3GzVuBGm5Yndv Nwgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739848504; x=1740453304; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HgpO8b1EeRoUbQ9L53xVbhgb2e+wvzP8W/zHZ0N0bI0=; b=ltxAkqVjonA4Jl/v3ebiQ2EXHup5ILqvdovLHO3gC0mLfoUo0kpAZ7J5WFlNV5TCs1 hnQLbHJEuHtVmRqQ5JoUoIFbjzLwbhgGm5YPzqPVPRPaRlQJd83Tq5FUMoO7dudxMdTM e5R2u/QgJgvZi3VjFfOoBBG1A/8yZYThodIGpU/5tBXFrQkEyXZHWttOW15b6D6MWxn8 G+wj5uegpUnia6Olg2s3JcjOlSUeWRFC24Dct5ImtgH6blM6g+Hdr7QDP7LawZQw4Htn MNSTYRKJ/RdliJV5KqtX0X8ctnIbyXbHozmXtXmGnZv8kw7Y9Fp9BipQfxUOMByXOpTj DrPQ== X-Gm-Message-State: AOJu0YxT3H0TofylT5lEebumgWodm8Dza1O5NetuhWbKYRDOmGgRq8vp PrjqRffbpp8t/q46A2cbpvnDZk/LJ6MLqOg5gUmNtBmAhL8udg+N X-Gm-Gg: ASbGncv+6TAKE7QqWwIo8t2UPIJ/9TdbyJAb88iaxRrfpfzb5es6sBYtYNnRh20s1bu 8KgMCBGbqqZYBTxT+tN2ri8o1KtpRcglJydN5uvFZxzmBnE84hjoJ0va0GIStkOWmxbd68n7I46 JgAMtzZL9mstO0qogcueRWL+X41Ovg85CO22tlF7zcAAwaCc0JnMNvV9JaSQplNP8TsIUQSquWI KFSd440F01t8amBzibX973r+Bod6Tmre4jpVAzlTvOi5hgZ7GkndzMQUiQZ+K0s6eKBzkz95VOk fr9DE2uY3MhzOqXI+tZ0fQnD1S/KJpEJOo69paBkZ6JR6fYgDO1d X-Google-Smtp-Source: AGHT+IGvVJgmZUZLNteiLjqooGGcN/Vh5bbt2bNgloW1Ge4ngfDQUgubE+o3sAAi/KBVt1FkRsrBsw== X-Received: by 2002:aa7:9912:0:b0:732:5875:eb95 with SMTP id d2e1a72fcca58-7325875ee4fmr19388335b3a.4.1739848503581; Mon, 17 Feb 2025 19:15:03 -0800 (PST) Received: from saturn.. (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732425466besm8763451b3a.9.2025.02.17.19.15.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Feb 2025 19:15:02 -0800 (PST) From: JP Kobryn To: shakeel.butt@linux.dev, tj@kernel.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 02/11] cgroup: add level of indirection for cgroup_rstat struct Date: Mon, 17 Feb 2025 19:14:39 -0800 Message-ID: <20250218031448.46951-3-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250218031448.46951-1-inwardvessel@gmail.com> References: <20250218031448.46951-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: F2CE540005 X-Stat-Signature: 8kg8sjz856jn9a1acewpd38km7ch13dw X-HE-Tag: 1739848504-803323 X-HE-Meta: U2FsdGVkX1/9aVm9NS1elxccaj/cbWk4TfSL3CIwhDzv10mHTb376su8AkBs6QfkSQh7Hs6tPqjqYU5SsIjAFiRfAVLcFJ7WOh2pwgaKDmmMCvA944VgePEVeanmJELM/WKPDPeXZdd6BnuToGUtH5RJF1Duj94Gy2ptKhv8OzJSl+bMOxmqJ/VJ2VAjUF8zUKgnDZULe+CQmQd1uH+HXTPxTE7jed4dcDuXPqqus1FFBdP9MswvrNVjYE6ty6+PLFXia75qz5KnUJp4Y1v6m/USyNCOO0kqg0fJ17XtLWEWeipTOehhMCesQ78+15dCmXmXil8UnwyNPbYSSLQYvrF3Jy5MfJndxHun7Pw8RFlkJsZfTqenN3v/pMteDwIGpZWGQVDECD3xVXtljcEsxUeCSpcz8jv5U7WbXnngbnJtz+GOuCIdkBMHekUYcNB1mANREjx19dx3WYjh+bA90I39buZopLJrkAMSLJwevXxlABjXmeP0RFH9No82EZFTqmE9pKCJAIGN3N4WGV88knwvn+NnCsXJ1bgAvYCWGOJoEaxz8tzRwzeJGLcccHQPMTDLacPcPAHPlSGJlmutwR2a0BKTRLtROZZRvJxkl82KX3E7GQP3XDPZhmY1oXCaBh+OALpSeENz6bbRCHDg/TdmTZ4yw+fxCaAvCFRJanHjM3b1tNVYnRzXz1fCjOPHe7dwg/d2zrJHSiJMVJI6x0f0WC/wmAbpHmS3SzHvw1cSBiolN74OW6cdgWLKdTm1JgdSW/H3VWuqkvDp05LFJ6utBCr+qFrbIxm9Rv7SfmNoRT83dL5dJ1GB4h+qmlMYaf8v2UEtQYVeQGfjUROI4CTi6x1E/d5H2ow3qg3Aijey9iX6KhYP3UOMUzE6ChidFsr+fwlWmrHlCu/5EURY/Vf29YQEYWWRaDVxxmUURuHQdWOp+oCFZuaOtpaCbZTzCnKB3oUz3Tewo5u25bP 9GGXnz+L NncQ/z62izOuRNiqpHWXIz4FkZAR9cRRCp3DA+ZcY/fqWfIOkCTdh50pUM6nx9mfogsDsfUCccmWWFC7UjR+hNU2RfBFyabyugTsdks+GxvTKLRUxYUGKJlorqxeolHOr8cz0u7tX/7yWowkNw9585TWsm9hEk0A7rqckqLR5Q5t6ibbepQU2/AlAZ230qn7q5rPzg2VB6fQB8eok0g+4lKb/aDuaAStS25OCB0uv/ohcEBjT0LisxHupWUI/EQhFJt0LfvVoGJOF9t+A1cQU48zxnUswGfaplZDkr8AMHsY2EJA5t+ybsDMNcx+aG4YsGczoVS3YmTlcGzcXdF+AOzQ3oOFgBRGo4RO7sfcA3tm0lK3Zc8UB2UQ5uDCvbTO8kI+/5CXdzHA3Zh9av3qEKy7ybiCajyf+e0LN9n82MiVr35xUIbtGcoCSrag7I/H/9XaIPNt+s1I0zc4PeEH5JVR7W1hitkrAVwnvCTZ2H7cxGbA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Change the type of rstat node from cgroup to the new cgroup_rstat struct. Then for the rstat updated/flush api calls, add double under versions that accept references to the cgroup_rstat struct. This new level of indirection will allow for extending the public api further. i.e. the cgroup_rstat struct can be embedded in a new type of object and a public api can be added for that new type. Signed-off-by: JP Kobryn --- include/linux/cgroup_rstat.h | 6 +- kernel/cgroup/cgroup-internal.h | 4 +- kernel/cgroup/cgroup.c | 12 +- kernel/cgroup/rstat.c | 204 ++++++++++++++++++++------------ 4 files changed, 141 insertions(+), 85 deletions(-) diff --git a/include/linux/cgroup_rstat.h b/include/linux/cgroup_rstat.h index f95474d6f8ab..780b826ea364 100644 --- a/include/linux/cgroup_rstat.h +++ b/include/linux/cgroup_rstat.h @@ -36,7 +36,7 @@ struct cgroup_rstat { * frequent updates. */ CACHELINE_PADDING(_pad_); - struct cgroup *rstat_flush_next; + struct cgroup_rstat *rstat_flush_next; }; struct cgroup_base_stat { @@ -58,8 +58,8 @@ struct cgroup_rstat_cpu { * to the cgroup makes it unnecessary for each per-cpu struct to * point back to the associated cgroup. */ - struct cgroup *updated_children; /* terminated by self */ - struct cgroup *updated_next; /* NULL if not on the list */ + struct cgroup_rstat *updated_children; /* terminated by self */ + struct cgroup_rstat *updated_next; /* NULL if not on the list */ /* * ->bsync protects ->bstat. These are the only fields which get diff --git a/kernel/cgroup/cgroup-internal.h b/kernel/cgroup/cgroup-internal.h index c964dd7ff967..03139018da43 100644 --- a/kernel/cgroup/cgroup-internal.h +++ b/kernel/cgroup/cgroup-internal.h @@ -269,8 +269,8 @@ int cgroup_task_count(const struct cgroup *cgrp); /* * rstat.c */ -int cgroup_rstat_init(struct cgroup *cgrp); -void cgroup_rstat_exit(struct cgroup *cgrp); +int cgroup_rstat_init(struct cgroup_rstat *rstat); +void cgroup_rstat_exit(struct cgroup_rstat *rstat); void cgroup_rstat_boot(void); void cgroup_base_stat_cputime_show(struct seq_file *seq); diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 03a3a4da49f1..02d6c11ccccb 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -1359,7 +1359,7 @@ static void cgroup_destroy_root(struct cgroup_root *root) cgroup_unlock(); - cgroup_rstat_exit(cgrp); + cgroup_rstat_exit(&cgrp->rstat); kernfs_destroy_root(root->kf_root); cgroup_free_root(root); } @@ -2133,7 +2133,7 @@ int cgroup_setup_root(struct cgroup_root *root, u16 ss_mask) if (ret) goto destroy_root; - ret = cgroup_rstat_init(root_cgrp); + ret = cgroup_rstat_init(&root_cgrp->rstat); if (ret) goto destroy_root; @@ -2175,7 +2175,7 @@ int cgroup_setup_root(struct cgroup_root *root, u16 ss_mask) goto out; exit_stats: - cgroup_rstat_exit(root_cgrp); + cgroup_rstat_exit(&root_cgrp->rstat); destroy_root: kernfs_destroy_root(root->kf_root); root->kf_root = NULL; @@ -5436,7 +5436,7 @@ static void css_free_rwork_fn(struct work_struct *work) cgroup_put(cgroup_parent(cgrp)); kernfs_put(cgrp->kn); psi_cgroup_free(cgrp); - cgroup_rstat_exit(cgrp); + cgroup_rstat_exit(&cgrp->rstat); kfree(cgrp); } else { /* @@ -5687,7 +5687,7 @@ static struct cgroup *cgroup_create(struct cgroup *parent, const char *name, if (ret) goto out_free_cgrp; - ret = cgroup_rstat_init(cgrp); + ret = cgroup_rstat_init(&cgrp->rstat); if (ret) goto out_cancel_ref; @@ -5780,7 +5780,7 @@ static struct cgroup *cgroup_create(struct cgroup *parent, const char *name, out_kernfs_remove: kernfs_remove(cgrp->kn); out_stat_exit: - cgroup_rstat_exit(cgrp); + cgroup_rstat_exit(&cgrp->rstat); out_cancel_ref: percpu_ref_exit(&cgrp->self.refcnt); out_free_cgrp: diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 7e7879d88c38..13090dda56aa 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -14,9 +14,20 @@ static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_cpu_lock); static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu); -static struct cgroup_rstat_cpu *cgroup_rstat_cpu(struct cgroup *cgrp, int cpu) +static struct cgroup_rstat_cpu *rstat_cpu(struct cgroup_rstat *rstat, int cpu) { - return per_cpu_ptr(cgrp->rstat.rstat_cpu, cpu); + return per_cpu_ptr(rstat->rstat_cpu, cpu); +} + +static struct cgroup_rstat *rstat_parent(struct cgroup_rstat *rstat) +{ + struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + struct cgroup *parent = cgroup_parent(cgrp); + + if (!parent) + return NULL; + + return &parent->rstat; } /* @@ -73,17 +84,9 @@ void _cgroup_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, raw_spin_unlock_irqrestore(cpu_lock, flags); } -/** - * cgroup_rstat_updated - keep track of updated rstat_cpu - * @cgrp: target cgroup - * @cpu: cpu on which rstat_cpu was updated - * - * @cgrp's rstat_cpu on @cpu was updated. Put it on the parent's matching - * rstat_cpu->updated_children list. See the comment on top of - * cgroup_rstat_cpu definition for details. - */ -__bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) +static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu) { + struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); unsigned long flags; @@ -95,15 +98,15 @@ __bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) * instead of NULL, we can tell whether @cgrp is on the list by * testing the next pointer for NULL. */ - if (data_race(cgroup_rstat_cpu(cgrp, cpu)->updated_next)) + if (data_race(rstat_cpu(rstat, cpu)->updated_next)) return; flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, true); /* put @cgrp and all ancestors on the corresponding updated lists */ while (true) { - struct cgroup_rstat_cpu *rstatc = cgroup_rstat_cpu(cgrp, cpu); - struct cgroup *parent = cgroup_parent(cgrp); + struct cgroup_rstat_cpu *rstatc = rstat_cpu(rstat, cpu); + struct cgroup_rstat *parent = rstat_parent(rstat); struct cgroup_rstat_cpu *prstatc; /* @@ -115,20 +118,34 @@ __bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) /* Root has no parent to link it to, but mark it busy */ if (!parent) { - rstatc->updated_next = cgrp; + rstatc->updated_next = rstat; break; } - prstatc = cgroup_rstat_cpu(parent, cpu); + prstatc = rstat_cpu(parent, cpu); rstatc->updated_next = prstatc->updated_children; - prstatc->updated_children = cgrp; + prstatc->updated_children = rstat; - cgrp = parent; + rstat = parent; } _cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, true); } +/** + * cgroup_rstat_updated - keep track of updated rstat_cpu + * @cgrp: target cgroup + * @cpu: cpu on which rstat_cpu was updated + * + * @cgrp's rstat_cpu on @cpu was updated. Put it on the parent's matching + * rstat_cpu->updated_children list. See the comment on top of + * cgroup_rstat_cpu definition for details. + */ +__bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) +{ + __cgroup_rstat_updated(&cgrp->rstat, cpu); +} + /** * cgroup_rstat_push_children - push children cgroups into the given list * @head: current head of the list (= subtree root) @@ -141,32 +158,32 @@ __bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) * into a singly linked list built from the tail backward like "pushing" * cgroups into a stack. The root is pushed by the caller. */ -static struct cgroup *cgroup_rstat_push_children(struct cgroup *head, - struct cgroup *child, int cpu) +static struct cgroup_rstat *cgroup_rstat_push_children( + struct cgroup_rstat *head, struct cgroup_rstat *child, int cpu) { - struct cgroup *chead = child; /* Head of child cgroup level */ - struct cgroup *ghead = NULL; /* Head of grandchild cgroup level */ - struct cgroup *parent, *grandchild; + struct cgroup_rstat *chead = child; /* Head of child cgroup level */ + struct cgroup_rstat *ghead = NULL; /* Head of grandchild cgroup level */ + struct cgroup_rstat *parent, *grandchild; struct cgroup_rstat_cpu *crstatc; - child->rstat.rstat_flush_next = NULL; + child->rstat_flush_next = NULL; next_level: while (chead) { child = chead; - chead = child->rstat.rstat_flush_next; - parent = cgroup_parent(child); + chead = child->rstat_flush_next; + parent = rstat_parent(child); /* updated_next is parent cgroup terminated */ while (child != parent) { - child->rstat.rstat_flush_next = head; + child->rstat_flush_next = head; head = child; - crstatc = cgroup_rstat_cpu(child, cpu); + crstatc = rstat_cpu(child, cpu); grandchild = crstatc->updated_children; if (grandchild != child) { /* Push the grand child to the next level */ crstatc->updated_children = child; - grandchild->rstat.rstat_flush_next = ghead; + grandchild->rstat_flush_next = ghead; ghead = grandchild; } child = crstatc->updated_next; @@ -200,14 +217,16 @@ static struct cgroup *cgroup_rstat_push_children(struct cgroup *head, * within the children list and terminated by the parent cgroup. An exception * here is the cgroup root whose updated_next can be self terminated. */ -static struct cgroup *cgroup_rstat_updated_list(struct cgroup *root, int cpu) +static struct cgroup_rstat *cgroup_rstat_updated_list( + struct cgroup_rstat *root, int cpu) { + struct cgroup *cgrp = container_of(root, typeof(*cgrp), rstat); raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); - struct cgroup_rstat_cpu *rstatc = cgroup_rstat_cpu(root, cpu); - struct cgroup *head = NULL, *parent, *child; + struct cgroup_rstat_cpu *rstatc = rstat_cpu(root, cpu); + struct cgroup_rstat *head = NULL, *parent, *child; unsigned long flags; - flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, root, false); + flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, false); /* Return NULL if this subtree is not on-list */ if (!rstatc->updated_next) @@ -217,17 +236,17 @@ static struct cgroup *cgroup_rstat_updated_list(struct cgroup *root, int cpu) * Unlink @root from its parent. As the updated_children list is * singly linked, we have to walk it to find the removal point. */ - parent = cgroup_parent(root); + parent = rstat_parent(root); if (parent) { struct cgroup_rstat_cpu *prstatc; - struct cgroup **nextp; + struct cgroup_rstat **nextp; - prstatc = cgroup_rstat_cpu(parent, cpu); + prstatc = rstat_cpu(parent, cpu); nextp = &prstatc->updated_children; while (*nextp != root) { struct cgroup_rstat_cpu *nrstatc; - nrstatc = cgroup_rstat_cpu(*nextp, cpu); + nrstatc = rstat_cpu(*nextp, cpu); WARN_ON_ONCE(*nextp == parent); nextp = &nrstatc->updated_next; } @@ -238,13 +257,13 @@ static struct cgroup *cgroup_rstat_updated_list(struct cgroup *root, int cpu) /* Push @root to the list first before pushing the children */ head = root; - root->rstat.rstat_flush_next = NULL; + root->rstat_flush_next = NULL; child = rstatc->updated_children; rstatc->updated_children = root; if (child != root) head = cgroup_rstat_push_children(head, child, cpu); unlock_ret: - _cgroup_rstat_cpu_unlock(cpu_lock, cpu, root, flags, false); + _cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, false); return head; } @@ -300,24 +319,26 @@ static inline void __cgroup_rstat_unlock(struct cgroup *cgrp, int cpu_in_loop) } /* see cgroup_rstat_flush() */ -static void cgroup_rstat_flush_locked(struct cgroup *cgrp) +static void cgroup_rstat_flush_locked(struct cgroup_rstat *rstat) __releases(&cgroup_rstat_lock) __acquires(&cgroup_rstat_lock) { + struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); int cpu; lockdep_assert_held(&cgroup_rstat_lock); for_each_possible_cpu(cpu) { - struct cgroup *pos = cgroup_rstat_updated_list(cgrp, cpu); + struct cgroup_rstat *pos = cgroup_rstat_updated_list(rstat, cpu); - for (; pos; pos = pos->rstat.rstat_flush_next) { + for (; pos; pos = pos->rstat_flush_next) { + struct cgroup *pos_cgroup = container_of(pos, struct cgroup, rstat); struct cgroup_subsys_state *css; - cgroup_base_stat_flush(pos, cpu); - bpf_rstat_flush(pos, cgroup_parent(pos), cpu); + cgroup_base_stat_flush(pos_cgroup, cpu); + bpf_rstat_flush(pos_cgroup, cgroup_parent(pos_cgroup), cpu); rcu_read_lock(); - list_for_each_entry_rcu(css, &pos->rstat_css_list, + list_for_each_entry_rcu(css, &pos_cgroup->rstat_css_list, rstat_css_node) css->ss->css_rstat_flush(css, cpu); rcu_read_unlock(); @@ -333,6 +354,17 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp) } } +static void __cgroup_rstat_flush(struct cgroup_rstat *rstat) +{ + struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + + might_sleep(); + + __cgroup_rstat_lock(cgrp, -1); + cgroup_rstat_flush_locked(rstat); + __cgroup_rstat_unlock(cgrp, -1); +} + /** * cgroup_rstat_flush - flush stats in @cgrp's subtree * @cgrp: target cgroup @@ -348,11 +380,17 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp) */ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) { - might_sleep(); + __cgroup_rstat_flush(&cgrp->rstat); +} + +static void __cgroup_rstat_flush_hold(struct cgroup_rstat *rstat) + __acquires(&cgroup_rstat_lock) +{ + struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + might_sleep(); __cgroup_rstat_lock(cgrp, -1); - cgroup_rstat_flush_locked(cgrp); - __cgroup_rstat_unlock(cgrp, -1); + cgroup_rstat_flush_locked(rstat); } /** @@ -365,63 +403,81 @@ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) * This function may block. */ void cgroup_rstat_flush_hold(struct cgroup *cgrp) - __acquires(&cgroup_rstat_lock) { - might_sleep(); - __cgroup_rstat_lock(cgrp, -1); - cgroup_rstat_flush_locked(cgrp); + __cgroup_rstat_flush_hold(&cgrp->rstat); } /** * cgroup_rstat_flush_release - release cgroup_rstat_flush_hold() * @cgrp: cgroup used by tracepoint */ -void cgroup_rstat_flush_release(struct cgroup *cgrp) +static void __cgroup_rstat_flush_release(struct cgroup_rstat *rstat) __releases(&cgroup_rstat_lock) { + struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + __cgroup_rstat_unlock(cgrp, -1); } -int cgroup_rstat_init(struct cgroup *cgrp) +/** + * cgroup_rstat_flush_release - release cgroup_rstat_flush_hold() + * @cgrp: cgroup used by tracepoint + */ +void cgroup_rstat_flush_release(struct cgroup *cgrp) { - int cpu; + __cgroup_rstat_flush_release(&cgrp->rstat); +} - /* the root cgrp has rstat_cpu preallocated */ - if (!cgrp->rstat.rstat_cpu) { - cgrp->rstat.rstat_cpu = alloc_percpu( - struct cgroup_rstat_cpu); - if (!cgrp->rstat.rstat_cpu) - return -ENOMEM; - } +static void __cgroup_rstat_init(struct cgroup_rstat *rstat) +{ + int cpu; /* ->updated_children list is self terminated */ for_each_possible_cpu(cpu) { - struct cgroup_rstat_cpu *rstatc = cgroup_rstat_cpu(cgrp, cpu); + struct cgroup_rstat_cpu *rstatc = rstat_cpu(rstat, cpu); - rstatc->updated_children = cgrp; + rstatc->updated_children = rstat; u64_stats_init(&rstatc->bsync); } +} + +int cgroup_rstat_init(struct cgroup_rstat *rstat) +{ + /* the root cgrp has rstat_cpu preallocated */ + if (!rstat->rstat_cpu) { + rstat->rstat_cpu = alloc_percpu(struct cgroup_rstat_cpu); + if (!rstat->rstat_cpu) + return -ENOMEM; + } + + __cgroup_rstat_init(rstat); return 0; } -void cgroup_rstat_exit(struct cgroup *cgrp) +static void __cgroup_rstat_exit(struct cgroup_rstat *rstat) { int cpu; - cgroup_rstat_flush(cgrp); - /* sanity check */ for_each_possible_cpu(cpu) { - struct cgroup_rstat_cpu *rstatc = cgroup_rstat_cpu(cgrp, cpu); + struct cgroup_rstat_cpu *rstatc = rstat_cpu(rstat, cpu); - if (WARN_ON_ONCE(rstatc->updated_children != cgrp) || + if (WARN_ON_ONCE(rstatc->updated_children != rstat) || WARN_ON_ONCE(rstatc->updated_next)) return; } - free_percpu(cgrp->rstat.rstat_cpu); - cgrp->rstat.rstat_cpu = NULL; + free_percpu(rstat->rstat_cpu); + rstat->rstat_cpu = NULL; +} + +void cgroup_rstat_exit(struct cgroup_rstat *rstat) +{ + struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + + cgroup_rstat_flush(cgrp); + __cgroup_rstat_exit(rstat); } void __init cgroup_rstat_boot(void) @@ -462,7 +518,7 @@ static void cgroup_base_stat_sub(struct cgroup_base_stat *dst_bstat, static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu) { - struct cgroup_rstat_cpu *rstatc = cgroup_rstat_cpu(cgrp, cpu); + struct cgroup_rstat_cpu *rstatc = rstat_cpu(&cgrp->rstat, cpu); struct cgroup *parent = cgroup_parent(cgrp); struct cgroup_rstat_cpu *prstatc; struct cgroup_base_stat delta; @@ -492,7 +548,7 @@ static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu) cgroup_base_stat_add(&cgrp->last_bstat, &delta); delta = rstatc->subtree_bstat; - prstatc = cgroup_rstat_cpu(parent, cpu); + prstatc = rstat_cpu(&parent->rstat, cpu); cgroup_base_stat_sub(&delta, &rstatc->last_subtree_bstat); cgroup_base_stat_add(&prstatc->subtree_bstat, &delta); cgroup_base_stat_add(&rstatc->last_subtree_bstat, &delta); From patchwork Tue Feb 18 03:14:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 13978852 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEAE0C021AA for ; Tue, 18 Feb 2025 03:15:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 094FE2800B9; Mon, 17 Feb 2025 22:15:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F37042800A4; Mon, 17 Feb 2025 22:15:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8AC72800B9; Mon, 17 Feb 2025 22:15:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A96172800A4 for ; Mon, 17 Feb 2025 22:15:08 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5A686B516B for ; Tue, 18 Feb 2025 03:15:08 +0000 (UTC) X-FDA: 83131599096.08.A983FD1 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf16.hostedemail.com (Postfix) with ESMTP id 69232180006 for ; Tue, 18 Feb 2025 03:15:06 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NWaX+L8w; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf16.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739848506; a=rsa-sha256; cv=none; b=GCeSN44OLZVNYiiabrGBqc6lgR/PT4modbvA788shc8ICrcQVz8atC/nZTyIAD13OFAUYo XzOBTYaT1z4HqhCTwT4Z8kyisXlQo1gvHBjYBr09GeNd8tfnsXBVmAWC1JPKvkJnc0MznN yoXKGhyhct5d74oO53uszXuylknFCus= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NWaX+L8w; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf16.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739848506; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/UwRbDpwwiM5OHo47KEwhQuET3ryRjJIsdR5W9Qkkts=; b=zBOF7nHLeB+XaYwtcNGjfzx/+Qvs587D5tMks10b8QeSKEK27OwF14nyPniz/Acsb/3X/Y /ZBg08E8oInl5vTpeCF0MR2iCbSjGc2ETpMFg/D0omwG1WDJuRrhFODSpMVrWRROR983I5 /Mjjrd06XQgaVcxYfvQXZAfK9hXzMmE= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-220d39a5627so71563155ad.1 for ; Mon, 17 Feb 2025 19:15:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739848505; x=1740453305; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/UwRbDpwwiM5OHo47KEwhQuET3ryRjJIsdR5W9Qkkts=; b=NWaX+L8wMUVyw4vKTY/9ywO0f4fbt90aL1onTZS9blYpjU5XnrrhN7fDNSNME8vOK6 nx72uN+wtDzwH1sGtWatqwAG/r+pxwZO4hZgegRlEMbHTFoQMnD2yEMR5iTv4/njHwqd cmntohXlxFlLUtt3ukIF8xP0NE9FQOrc/cQJYI9S/I4TY0CMcGvMKBax+s/h9KAuAt3A o8v5gVa3PSJgpHE+K8ud5v24vnYlVTF2mUFv/f79RPnsHOEn1XfXuJf2nJj/Tn3uy3S9 v4r2EQKLCgccsLj3raUnFWh+SFILvXO1+ZWwjCKtbp6SeGbSJeA/61jOiNnho8nv6ein LbWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739848505; x=1740453305; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/UwRbDpwwiM5OHo47KEwhQuET3ryRjJIsdR5W9Qkkts=; b=Mqb/vMw9gUDk7lhPSZVuEATnXgCMwZH+5qYKJs21CIqE7R4jG6gYms6+x2ZXJ5FO2B h9O6K3Rx1lQbW5fmneOKV5JL6mdg4N7q6ibI0pYkb2D7MYNQ4Mk+PC9f+6H00U/jeuVI hKVcg5T0BdZFzFaU3Ssm5UzMqNpsrsveW12KGYenYnu7sFQpl9HKwDHpronpU04nNNSW Z6aAgkZ2L/Jdyjsa7JMM6TMy93dMJGBY+VSsiH47y/b2k+wsytzfDia3Y3BRGiCQ+rIN MoANxUPZn2JEWSaSP0Hsk0jvhNGN2rhWSq0BRAHDzOmrxRt4A1AupxvSkDTNQhW9QtWa g0XQ== X-Gm-Message-State: AOJu0YwnFogY52jNMi4DF+V41g8oh+sQ3Pfnwzj2PcD4w0rXCvFcwrPd n5mwgeMU7ec4/wApcMtD/dz49rw8FSscTegj8V5DAW0sPgKVfvHs X-Gm-Gg: ASbGncuklG6owznCYOGC2CAkmzRfg+zN+0rkcNDpPFkBMxbnk2RmCjEbrEkRgkwUojv iMLejZDjFdqHNYXdRGHQSMIP8jXI3ZmmaitVsjyX3j7E7v3ULc9fq4LKKvcgbKmeZIAYV2x8SuK qAx+cbQnokgpDN13e0nQnhwYID56Tc4QJ7ir8nGhMXZJHykkVKQhTWC58Klj4FyUlTE1NEkDdnO S0eMn93JGKmJN22QmGoo4iEZ0ih418iOnefQU8fNkDX+U564scJEr6fU48+JJ9v1DBRulbAfcB0 rEaE7/JadGBxmWXsl0nuseavqkQVo4CFMVzoxiZBddsrFwDJJKmZ X-Google-Smtp-Source: AGHT+IFOvjlSJbep9cTBMal2GTTkcqYRhr58+YYHWmPTnoxu5Wh1hKr9YEqq6/k/u2c1lVSxE6Idrg== X-Received: by 2002:a05:6a20:d49b:b0:1ee:c598:7a7d with SMTP id adf61e73a8af0-1eec5987c38mr2688332637.41.1739848505171; Mon, 17 Feb 2025 19:15:05 -0800 (PST) Received: from saturn.. (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732425466besm8763451b3a.9.2025.02.17.19.15.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Feb 2025 19:15:04 -0800 (PST) From: JP Kobryn To: shakeel.butt@linux.dev, tj@kernel.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 03/11] cgroup: move cgroup_rstat from cgroup to cgroup_subsys_state Date: Mon, 17 Feb 2025 19:14:40 -0800 Message-ID: <20250218031448.46951-4-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250218031448.46951-1-inwardvessel@gmail.com> References: <20250218031448.46951-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 69232180006 X-Stat-Signature: 8zbgux17jpfuh11k68k7hqfm3dpyj8q8 X-Rspam-User: X-HE-Tag: 1739848506-966249 X-HE-Meta: U2FsdGVkX1/9jaZhbpkCQEt7W885bJgy+YQb8EDefg47YOJjOrCQgOTmZXKDc4BUPODZ+T/xwdFkAzGGBr3IMY+6LFhTwcG17Ajsw1YSO6BeyD2utKDijJkxk2HY3JqyLwfRBMe5mfdeaaP5lQhTvUptbgcjpfnMBUhQA4C1H+nNfb6DL11pwnNHwkF0kRPyCEQbX5st8bMAA+w/cAbcjaxjh4/E4lsDT975pvwOsGPCGqUFSQ1qCkmtEsvCVsulLI8aWDXQ3NjKwR+TNf73qe5+fKjIAad76co5F/DyB8BTcD6qb0GbPHGYjc2laoiyv0pbxmw7IYkC+859UbX+VG9U/ptsEdOCqOLwqu4tr8Aj62fJEIF57fuVR+lGHBmIJ8O8gNfN7jjuJWpqfRD0qsJghy9WZv9tuzxqREq9GjAl0d92eEa0y7hoFuZSKySSzHoFYOCCixjs+qH+W27lGAPPWC9pm5GXbMDc5ueF6DljltSiWC0cO91njZHORmQNON9ZUjWNWhVB8II1FJ6+lo41jtcmXY2NnMOmYwX/3ocxhgc9XPxzVYkL13mgrlc40yRnzNUlQ7LmdwdZRjVKUJ8KcOg0PN81toyn44ArqKhd/FaIKr+ThN/SEc4cmDsoB0MUf2mWgMbcpC0cc2zlvBEDMkIFbVi6yDcrIUKI0OPaNcenubSRRo1QO4T5z+ftviwe8+ehwvvVi8zmXUTOQaekAEeGDenmHN6kEIiOaRVNbAL/C927LM4aSv6KEePuwskGL2YOm1uL9xIWkiiH2gvcPJgxwlze0NcY4d1zXRZD36bBLcQud9aGOLtkVTKS0nW1HtXyWPCiqcbI1maRhqEjsCueKVEtP64D+Ir+6G7atKdjM51FbJz4AP2oiNsWUAMJnL8d79gMfJCpGMP6+thk7+NCsPq+JlKRTD984uLYn8kPYjN5kIJtC+1knPw0dipUfndU4XNZsZZO2X+ AAfBycAZ tudrSUvaZT+tTaJwcdGA+ockDVF8niCbdbA/hCLi+C/r/FAD+lxA5dmwyayMSeUgFSLBH5idHLbpfaMwsw1QyoI8zVmV5ImAKUafpxX8HhQFjTT+5opW/4t1R1dSJKXONzhIw4QZds19AlDkoZRLsZabulq2xPH/S7u+VX3KjpdIxN16BRion57TKwSuDg8ZRJKuSgc7gPNsocSH1Ff+lTbJurFvSrG+qMLJQFZFVzqGb0JsUyjP8zOol3kZM4lCNwt19gtK/S45+XMxIsj03SigP9MvQi1SWG+dcCigftAtbpRfoLL2rIQBZgxyYQmPhDmjIZA4EVEJGuUEtXIviXoxKjNQnNdDTfWfVKziW9MoRHYYddgwlxCB8XII4RzYvHAzV/eMJSun2pn0ufpK5qC/QLo8+VebD83gtgW3ycA1HnkQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Each cgroup owns a single cgroup_rstat instance. This means that a tree of pending rstat updates can contain changes from different subsystems. Because of this arrangement, when one subsystem is flushed via the public api cgroup_rstat_flushed(), all other subsystems with pending updates will also be flushed. Remove the cgroup_rstat instance from the cgroup and instead give one instance to each cgroup_subsys_state. Separate rstat trees will now exist for each unique subsystem. This separation allows for subsystems to make updates and flushes without the side effects of other subsystems. i.e. flushing the cpu stats does not cause the memory stats to be flushed and vice versa. The change in cgroup_rstat ownership from cgroup to cgroup_subsys_state allows for direct flushing of the css, so the rcu list management entities and operations previously tied to the cgroup which were used for managing a list of subsystem states with pending flushes are removed. In terms of client code, public api calls were changed to now accept the cgroup_subsystem_state so that when flushing or updating, a specific subsystem is associated with the call. Signed-off-by: JP Kobryn --- block/blk-cgroup.c | 4 +- include/linux/cgroup-defs.h | 10 +- include/linux/cgroup.h | 8 +- kernel/cgroup/cgroup-internal.h | 4 +- kernel/cgroup/cgroup.c | 64 ++++++---- kernel/cgroup/rstat.c | 117 ++++++++++-------- mm/memcontrol.c | 4 +- .../selftests/bpf/progs/btf_type_tag_percpu.c | 5 +- .../bpf/progs/cgroup_hierarchical_stats.c | 8 +- 9 files changed, 123 insertions(+), 101 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 9ed93d91d754..6a0680d8ce6a 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1201,7 +1201,7 @@ static int blkcg_print_stat(struct seq_file *sf, void *v) if (!seq_css(sf)->parent) blkcg_fill_root_iostats(); else - cgroup_rstat_flush(blkcg->css.cgroup); + cgroup_rstat_flush(&blkcg->css); rcu_read_lock(); hlist_for_each_entry_rcu(blkg, &blkcg->blkg_list, blkcg_node) { @@ -2186,7 +2186,7 @@ void blk_cgroup_bio_start(struct bio *bio) } u64_stats_update_end_irqrestore(&bis->sync, flags); - cgroup_rstat_updated(blkcg->css.cgroup, cpu); + cgroup_rstat_updated(&blkcg->css, cpu); put_cpu(); } diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index 6b6cc027fe70..81ec56860ee5 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -180,9 +180,6 @@ struct cgroup_subsys_state { struct list_head sibling; struct list_head children; - /* flush target list anchored at cgrp->rstat_css_list */ - struct list_head rstat_css_node; - /* * PI: Subsys-unique ID. 0 is unused and root is always 1. The * matching css can be looked up using css_from_id(). @@ -222,6 +219,9 @@ struct cgroup_subsys_state { * Protected by cgroup_mutex. */ int nr_descendants; + + /* per-cpu recursive resource statistics */ + struct cgroup_rstat rstat; }; /* @@ -444,10 +444,6 @@ struct cgroup { struct cgroup *dom_cgrp; struct cgroup *old_dom_cgrp; /* used while enabling threaded */ - /* per-cpu recursive resource statistics */ - struct cgroup_rstat rstat; - struct list_head rstat_css_list; - /* cgroup basic resource statistics */ struct cgroup_base_stat last_bstat; struct cgroup_base_stat bstat; diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index f8ef47f8a634..eec970622419 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -687,10 +687,10 @@ static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen) /* * cgroup scalable recursive statistics. */ -void cgroup_rstat_updated(struct cgroup *cgrp, int cpu); -void cgroup_rstat_flush(struct cgroup *cgrp); -void cgroup_rstat_flush_hold(struct cgroup *cgrp); -void cgroup_rstat_flush_release(struct cgroup *cgrp); +void cgroup_rstat_updated(struct cgroup_subsys_state *css, int cpu); +void cgroup_rstat_flush(struct cgroup_subsys_state *css); +void cgroup_rstat_flush_hold(struct cgroup_subsys_state *css); +void cgroup_rstat_flush_release(struct cgroup_subsys_state *css); /* * Basic resource stats. diff --git a/kernel/cgroup/cgroup-internal.h b/kernel/cgroup/cgroup-internal.h index 03139018da43..87d062baff90 100644 --- a/kernel/cgroup/cgroup-internal.h +++ b/kernel/cgroup/cgroup-internal.h @@ -269,8 +269,8 @@ int cgroup_task_count(const struct cgroup *cgrp); /* * rstat.c */ -int cgroup_rstat_init(struct cgroup_rstat *rstat); -void cgroup_rstat_exit(struct cgroup_rstat *rstat); +int cgroup_rstat_init(struct cgroup_subsys_state *css); +void cgroup_rstat_exit(struct cgroup_subsys_state *css); void cgroup_rstat_boot(void); void cgroup_base_stat_cputime_show(struct seq_file *seq); diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 02d6c11ccccb..916e9c5a1fd7 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -165,7 +165,9 @@ static struct static_key_true *cgroup_subsys_on_dfl_key[] = { static DEFINE_PER_CPU(struct cgroup_rstat_cpu, cgrp_dfl_root_rstat_cpu); /* the default hierarchy */ -struct cgroup_root cgrp_dfl_root = { .cgrp.rstat.rstat_cpu = &cgrp_dfl_root_rstat_cpu }; +struct cgroup_root cgrp_dfl_root = { + .cgrp.self.rstat.rstat_cpu = &cgrp_dfl_root_rstat_cpu +}; EXPORT_SYMBOL_GPL(cgrp_dfl_root); /* @@ -1359,7 +1361,7 @@ static void cgroup_destroy_root(struct cgroup_root *root) cgroup_unlock(); - cgroup_rstat_exit(&cgrp->rstat); + cgroup_rstat_exit(&cgrp->self); kernfs_destroy_root(root->kf_root); cgroup_free_root(root); } @@ -1864,13 +1866,6 @@ int rebind_subsystems(struct cgroup_root *dst_root, u16 ss_mask) } spin_unlock_irq(&css_set_lock); - if (ss->css_rstat_flush) { - list_del_rcu(&css->rstat_css_node); - synchronize_rcu(); - list_add_rcu(&css->rstat_css_node, - &dcgrp->rstat_css_list); - } - /* default hierarchy doesn't enable controllers by default */ dst_root->subsys_mask |= 1 << ssid; if (dst_root == &cgrp_dfl_root) { @@ -2053,7 +2048,6 @@ static void init_cgroup_housekeeping(struct cgroup *cgrp) cgrp->dom_cgrp = cgrp; cgrp->max_descendants = INT_MAX; cgrp->max_depth = INT_MAX; - INIT_LIST_HEAD(&cgrp->rstat_css_list); prev_cputime_init(&cgrp->prev_cputime); for_each_subsys(ss, ssid) @@ -2133,7 +2127,7 @@ int cgroup_setup_root(struct cgroup_root *root, u16 ss_mask) if (ret) goto destroy_root; - ret = cgroup_rstat_init(&root_cgrp->rstat); + ret = cgroup_rstat_init(&root_cgrp->self); if (ret) goto destroy_root; @@ -2175,7 +2169,7 @@ int cgroup_setup_root(struct cgroup_root *root, u16 ss_mask) goto out; exit_stats: - cgroup_rstat_exit(&root_cgrp->rstat); + cgroup_rstat_exit(&root_cgrp->self); destroy_root: kernfs_destroy_root(root->kf_root); root->kf_root = NULL; @@ -3240,6 +3234,12 @@ static int cgroup_apply_control_enable(struct cgroup *cgrp) css = css_create(dsct, ss); if (IS_ERR(css)) return PTR_ERR(css); + + if (css->ss && css->ss->css_rstat_flush) { + ret = cgroup_rstat_init(css); + if (ret) + goto err_out; + } } WARN_ON_ONCE(percpu_ref_is_dying(&css->refcnt)); @@ -3253,6 +3253,21 @@ static int cgroup_apply_control_enable(struct cgroup *cgrp) } return 0; + +err_out: + cgroup_for_each_live_descendant_pre(dsct, d_css, cgrp) { + for_each_subsys(ss, ssid) { + struct cgroup_subsys_state *css = cgroup_css(dsct, ss); + + if (!(cgroup_ss_mask(dsct) & (1 << ss->id))) + continue; + + if (css && css->ss && css->ss->css_rstat_flush) + cgroup_rstat_exit(css); + } + } + + return ret; } /** @@ -5436,7 +5451,7 @@ static void css_free_rwork_fn(struct work_struct *work) cgroup_put(cgroup_parent(cgrp)); kernfs_put(cgrp->kn); psi_cgroup_free(cgrp); - cgroup_rstat_exit(&cgrp->rstat); + cgroup_rstat_exit(&cgrp->self); kfree(cgrp); } else { /* @@ -5464,11 +5479,7 @@ static void css_release_work_fn(struct work_struct *work) if (ss) { struct cgroup *parent_cgrp; - /* css release path */ - if (!list_empty(&css->rstat_css_node)) { - cgroup_rstat_flush(cgrp); - list_del_rcu(&css->rstat_css_node); - } + cgroup_rstat_flush(css); cgroup_idr_replace(&ss->css_idr, NULL, css->id); if (ss->css_released) @@ -5494,7 +5505,7 @@ static void css_release_work_fn(struct work_struct *work) /* cgroup release path */ TRACE_CGROUP_PATH(release, cgrp); - cgroup_rstat_flush(cgrp); + cgroup_rstat_flush(&cgrp->self); spin_lock_irq(&css_set_lock); for (tcgrp = cgroup_parent(cgrp); tcgrp; @@ -5542,7 +5553,6 @@ static void init_and_link_css(struct cgroup_subsys_state *css, css->id = -1; INIT_LIST_HEAD(&css->sibling); INIT_LIST_HEAD(&css->children); - INIT_LIST_HEAD(&css->rstat_css_node); css->serial_nr = css_serial_nr_next++; atomic_set(&css->online_cnt, 0); @@ -5551,9 +5561,6 @@ static void init_and_link_css(struct cgroup_subsys_state *css, css_get(css->parent); } - if (ss->css_rstat_flush) - list_add_rcu(&css->rstat_css_node, &cgrp->rstat_css_list); - BUG_ON(cgroup_css(cgrp, ss)); } @@ -5659,7 +5666,6 @@ static struct cgroup_subsys_state *css_create(struct cgroup *cgrp, err_list_del: list_del_rcu(&css->sibling); err_free_css: - list_del_rcu(&css->rstat_css_node); INIT_RCU_WORK(&css->destroy_rwork, css_free_rwork_fn); queue_rcu_work(cgroup_destroy_wq, &css->destroy_rwork); return ERR_PTR(err); @@ -5687,7 +5693,7 @@ static struct cgroup *cgroup_create(struct cgroup *parent, const char *name, if (ret) goto out_free_cgrp; - ret = cgroup_rstat_init(&cgrp->rstat); + ret = cgroup_rstat_init(&cgrp->self); if (ret) goto out_cancel_ref; @@ -5780,7 +5786,7 @@ static struct cgroup *cgroup_create(struct cgroup *parent, const char *name, out_kernfs_remove: kernfs_remove(cgrp->kn); out_stat_exit: - cgroup_rstat_exit(&cgrp->rstat); + cgroup_rstat_exit(&cgrp->self); out_cancel_ref: percpu_ref_exit(&cgrp->self.refcnt); out_free_cgrp: @@ -6092,6 +6098,9 @@ static void __init cgroup_init_subsys(struct cgroup_subsys *ss, bool early) } else { css->id = cgroup_idr_alloc(&ss->css_idr, css, 1, 2, GFP_KERNEL); BUG_ON(css->id < 0); + + if (css->ss && css->ss->css_rstat_flush) + BUG_ON(cgroup_rstat_init(css)); } /* Update the init_css_set to contain a subsys @@ -6193,6 +6202,9 @@ int __init cgroup_init(void) css->id = cgroup_idr_alloc(&ss->css_idr, css, 1, 2, GFP_KERNEL); BUG_ON(css->id < 0); + + if (css->ss && css->ss->css_rstat_flush) + BUG_ON(cgroup_rstat_init(css)); } else { cgroup_init_subsys(ss, false); } diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 13090dda56aa..a32bcd7942a5 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -21,13 +21,13 @@ static struct cgroup_rstat_cpu *rstat_cpu(struct cgroup_rstat *rstat, int cpu) static struct cgroup_rstat *rstat_parent(struct cgroup_rstat *rstat) { - struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); - struct cgroup *parent = cgroup_parent(cgrp); + struct cgroup_subsys_state *css = container_of( + rstat, typeof(*css), rstat); - if (!parent) + if (!css->parent) return NULL; - return &parent->rstat; + return &(css->parent->rstat); } /* @@ -86,7 +86,9 @@ void _cgroup_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu) { - struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + struct cgroup_subsys_state *css = container_of( + rstat, typeof(*css), rstat); + struct cgroup *cgrp = css->cgroup; raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); unsigned long flags; @@ -95,7 +97,7 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu) * temporary inaccuracies, which is fine. * * Because @parent's updated_children is terminated with @parent - * instead of NULL, we can tell whether @cgrp is on the list by + * instead of NULL, we can tell whether @rstat is on the list by * testing the next pointer for NULL. */ if (data_race(rstat_cpu(rstat, cpu)->updated_next)) @@ -103,7 +105,7 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu) flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, true); - /* put @cgrp and all ancestors on the corresponding updated lists */ + /* put @rstat and all ancestors on the corresponding updated lists */ while (true) { struct cgroup_rstat_cpu *rstatc = rstat_cpu(rstat, cpu); struct cgroup_rstat *parent = rstat_parent(rstat); @@ -134,16 +136,16 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu) /** * cgroup_rstat_updated - keep track of updated rstat_cpu - * @cgrp: target cgroup + * @css: target cgroup subsystem state * @cpu: cpu on which rstat_cpu was updated * - * @cgrp's rstat_cpu on @cpu was updated. Put it on the parent's matching + * @css's rstat_cpu on @cpu was updated. Put it on the parent's matching * rstat_cpu->updated_children list. See the comment on top of * cgroup_rstat_cpu definition for details. */ -__bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) +__bpf_kfunc void cgroup_rstat_updated(struct cgroup_subsys_state *css, int cpu) { - __cgroup_rstat_updated(&cgrp->rstat, cpu); + __cgroup_rstat_updated(&css->rstat, cpu); } /** @@ -220,7 +222,9 @@ static struct cgroup_rstat *cgroup_rstat_push_children( static struct cgroup_rstat *cgroup_rstat_updated_list( struct cgroup_rstat *root, int cpu) { - struct cgroup *cgrp = container_of(root, typeof(*cgrp), rstat); + struct cgroup_subsys_state *css = container_of( + root, typeof(*css), rstat); + struct cgroup *cgrp = css->cgroup; raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); struct cgroup_rstat_cpu *rstatc = rstat_cpu(root, cpu); struct cgroup_rstat *head = NULL, *parent, *child; @@ -322,7 +326,9 @@ static inline void __cgroup_rstat_unlock(struct cgroup *cgrp, int cpu_in_loop) static void cgroup_rstat_flush_locked(struct cgroup_rstat *rstat) __releases(&cgroup_rstat_lock) __acquires(&cgroup_rstat_lock) { - struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + struct cgroup_subsys_state *css = container_of( + rstat, typeof(*css), rstat); + struct cgroup *cgrp = css->cgroup; int cpu; lockdep_assert_held(&cgroup_rstat_lock); @@ -331,17 +337,16 @@ static void cgroup_rstat_flush_locked(struct cgroup_rstat *rstat) struct cgroup_rstat *pos = cgroup_rstat_updated_list(rstat, cpu); for (; pos; pos = pos->rstat_flush_next) { - struct cgroup *pos_cgroup = container_of(pos, struct cgroup, rstat); - struct cgroup_subsys_state *css; + struct cgroup_subsys_state *pos_css = container_of( + pos, typeof(*pos_css), rstat); + struct cgroup *pos_cgroup = pos_css->cgroup; - cgroup_base_stat_flush(pos_cgroup, cpu); - bpf_rstat_flush(pos_cgroup, cgroup_parent(pos_cgroup), cpu); + if (!pos_css->ss) + cgroup_base_stat_flush(pos_cgroup, cpu); + else + pos_css->ss->css_rstat_flush(pos_css, cpu); - rcu_read_lock(); - list_for_each_entry_rcu(css, &pos_cgroup->rstat_css_list, - rstat_css_node) - css->ss->css_rstat_flush(css, cpu); - rcu_read_unlock(); + bpf_rstat_flush(pos_cgroup, cgroup_parent(pos_cgroup), cpu); } /* play nice and yield if necessary */ @@ -356,7 +361,9 @@ static void cgroup_rstat_flush_locked(struct cgroup_rstat *rstat) static void __cgroup_rstat_flush(struct cgroup_rstat *rstat) { - struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + struct cgroup_subsys_state *css = container_of( + rstat, typeof(*css), rstat); + struct cgroup *cgrp = css->cgroup; might_sleep(); @@ -366,27 +373,29 @@ static void __cgroup_rstat_flush(struct cgroup_rstat *rstat) } /** - * cgroup_rstat_flush - flush stats in @cgrp's subtree - * @cgrp: target cgroup + * cgroup_rstat_flush - flush stats in @css's rstat subtree + * @css: target cgroup subsystem state * - * Collect all per-cpu stats in @cgrp's subtree into the global counters - * and propagate them upwards. After this function returns, all cgroups in - * the subtree have up-to-date ->stat. + * Collect all per-cpu stats in @css's subtree into the global counters + * and propagate them upwards. After this function returns, all rstat + * nodes in the subtree have up-to-date ->stat. * - * This also gets all cgroups in the subtree including @cgrp off the + * This also gets all rstat nodes in the subtree including @css off the * ->updated_children lists. * * This function may block. */ -__bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) +__bpf_kfunc void cgroup_rstat_flush(struct cgroup_subsys_state *css) { - __cgroup_rstat_flush(&cgrp->rstat); + __cgroup_rstat_flush(&css->rstat); } static void __cgroup_rstat_flush_hold(struct cgroup_rstat *rstat) __acquires(&cgroup_rstat_lock) { - struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + struct cgroup_subsys_state *css = container_of( + rstat, typeof(*css), rstat); + struct cgroup *cgrp = css->cgroup; might_sleep(); __cgroup_rstat_lock(cgrp, -1); @@ -394,38 +403,40 @@ static void __cgroup_rstat_flush_hold(struct cgroup_rstat *rstat) } /** - * cgroup_rstat_flush_hold - flush stats in @cgrp's subtree and hold - * @cgrp: target cgroup + * cgroup_rstat_flush_hold - flush stats in @css's rstat subtree and hold + * @css: target subsystem state * - * Flush stats in @cgrp's subtree and prevent further flushes. Must be + * Flush stats in @css's rstat subtree and prevent further flushes. Must be * paired with cgroup_rstat_flush_release(). * * This function may block. */ -void cgroup_rstat_flush_hold(struct cgroup *cgrp) +void cgroup_rstat_flush_hold(struct cgroup_subsys_state *css) { - __cgroup_rstat_flush_hold(&cgrp->rstat); + __cgroup_rstat_flush_hold(&css->rstat); } /** * cgroup_rstat_flush_release - release cgroup_rstat_flush_hold() - * @cgrp: cgroup used by tracepoint + * @rstat: rstat node used to find associated cgroup used by tracepoint */ static void __cgroup_rstat_flush_release(struct cgroup_rstat *rstat) __releases(&cgroup_rstat_lock) { - struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + struct cgroup_subsys_state *css = container_of( + rstat, typeof(*css), rstat); + struct cgroup *cgrp = css->cgroup; __cgroup_rstat_unlock(cgrp, -1); } /** * cgroup_rstat_flush_release - release cgroup_rstat_flush_hold() - * @cgrp: cgroup used by tracepoint + * @css: css that was previously used for the call to flush hold */ -void cgroup_rstat_flush_release(struct cgroup *cgrp) +void cgroup_rstat_flush_release(struct cgroup_subsys_state *css) { - __cgroup_rstat_flush_release(&cgrp->rstat); + __cgroup_rstat_flush_release(&css->rstat); } static void __cgroup_rstat_init(struct cgroup_rstat *rstat) @@ -441,8 +452,10 @@ static void __cgroup_rstat_init(struct cgroup_rstat *rstat) } } -int cgroup_rstat_init(struct cgroup_rstat *rstat) +int cgroup_rstat_init(struct cgroup_subsys_state *css) { + struct cgroup_rstat *rstat = &css->rstat; + /* the root cgrp has rstat_cpu preallocated */ if (!rstat->rstat_cpu) { rstat->rstat_cpu = alloc_percpu(struct cgroup_rstat_cpu); @@ -472,11 +485,11 @@ static void __cgroup_rstat_exit(struct cgroup_rstat *rstat) rstat->rstat_cpu = NULL; } -void cgroup_rstat_exit(struct cgroup_rstat *rstat) +void cgroup_rstat_exit(struct cgroup_subsys_state *css) { - struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + struct cgroup_rstat *rstat = &css->rstat; - cgroup_rstat_flush(cgrp); + cgroup_rstat_flush(css); __cgroup_rstat_exit(rstat); } @@ -518,7 +531,7 @@ static void cgroup_base_stat_sub(struct cgroup_base_stat *dst_bstat, static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu) { - struct cgroup_rstat_cpu *rstatc = rstat_cpu(&cgrp->rstat, cpu); + struct cgroup_rstat_cpu *rstatc = rstat_cpu(&(cgrp->self.rstat), cpu); struct cgroup *parent = cgroup_parent(cgrp); struct cgroup_rstat_cpu *prstatc; struct cgroup_base_stat delta; @@ -548,7 +561,7 @@ static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu) cgroup_base_stat_add(&cgrp->last_bstat, &delta); delta = rstatc->subtree_bstat; - prstatc = rstat_cpu(&parent->rstat, cpu); + prstatc = rstat_cpu(&(parent->self.rstat), cpu); cgroup_base_stat_sub(&delta, &rstatc->last_subtree_bstat); cgroup_base_stat_add(&prstatc->subtree_bstat, &delta); cgroup_base_stat_add(&rstatc->last_subtree_bstat, &delta); @@ -560,7 +573,7 @@ cgroup_base_stat_cputime_account_begin(struct cgroup *cgrp, unsigned long *flags { struct cgroup_rstat_cpu *rstatc; - rstatc = get_cpu_ptr(cgrp->rstat.rstat_cpu); + rstatc = get_cpu_ptr(cgrp->self.rstat.rstat_cpu); *flags = u64_stats_update_begin_irqsave(&rstatc->bsync); return rstatc; } @@ -570,7 +583,7 @@ static void cgroup_base_stat_cputime_account_end(struct cgroup *cgrp, unsigned long flags) { u64_stats_update_end_irqrestore(&rstatc->bsync, flags); - cgroup_rstat_updated(cgrp, smp_processor_id()); + cgroup_rstat_updated(&cgrp->self, smp_processor_id()); put_cpu_ptr(rstatc); } @@ -673,12 +686,12 @@ void cgroup_base_stat_cputime_show(struct seq_file *seq) u64 usage, utime, stime, ntime; if (cgroup_parent(cgrp)) { - cgroup_rstat_flush_hold(cgrp); + cgroup_rstat_flush_hold(&cgrp->self); usage = cgrp->bstat.cputime.sum_exec_runtime; cputime_adjust(&cgrp->bstat.cputime, &cgrp->prev_cputime, &utime, &stime); ntime = cgrp->bstat.ntime; - cgroup_rstat_flush_release(cgrp); + cgroup_rstat_flush_release(&cgrp->self); } else { /* cgrp->bstat of root is not actually used, reuse it */ root_cgroup_cputime(&cgrp->bstat); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 46f8b372d212..88c2c8e610b1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -579,7 +579,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) if (!val) return; - cgroup_rstat_updated(memcg->css.cgroup, cpu); + cgroup_rstat_updated(&memcg->css, cpu); statc = this_cpu_ptr(memcg->vmstats_percpu); for (; statc; statc = statc->parent) { stats_updates = READ_ONCE(statc->stats_updates) + abs(val); @@ -611,7 +611,7 @@ static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force) if (mem_cgroup_is_root(memcg)) WRITE_ONCE(flush_last_time, jiffies_64); - cgroup_rstat_flush(memcg->css.cgroup); + cgroup_rstat_flush(&memcg->css); } /* diff --git a/tools/testing/selftests/bpf/progs/btf_type_tag_percpu.c b/tools/testing/selftests/bpf/progs/btf_type_tag_percpu.c index 035412265c3c..310cd51e12e8 100644 --- a/tools/testing/selftests/bpf/progs/btf_type_tag_percpu.c +++ b/tools/testing/selftests/bpf/progs/btf_type_tag_percpu.c @@ -45,7 +45,7 @@ int BPF_PROG(test_percpu2, struct bpf_testmod_btf_type_tag_2 *arg) SEC("tp_btf/cgroup_mkdir") int BPF_PROG(test_percpu_load, struct cgroup *cgrp, const char *path) { - g = (__u64)cgrp->rstat.rstat_cpu->updated_children; + g = (__u64)cgrp->self.rstat.rstat_cpu->updated_children; return 0; } @@ -56,7 +56,8 @@ int BPF_PROG(test_percpu_helper, struct cgroup *cgrp, const char *path) __u32 cpu; cpu = bpf_get_smp_processor_id(); - rstat = (struct cgroup_rstat_cpu *)bpf_per_cpu_ptr(cgrp->rstat.rstat_cpu, cpu); + rstat = (struct cgroup_rstat_cpu *)bpf_per_cpu_ptr( + cgrp->self.rstat.rstat_cpu, cpu); if (rstat) { /* READ_ONCE */ *(volatile int *)rstat; diff --git a/tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c b/tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c index c74362854948..10c803c8dc70 100644 --- a/tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c +++ b/tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c @@ -37,8 +37,8 @@ struct { __type(value, struct attach_counter); } attach_counters SEC(".maps"); -extern void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) __ksym; -extern void cgroup_rstat_flush(struct cgroup *cgrp) __ksym; +extern void cgroup_rstat_updated(struct cgroup_subsys_state *css, int cpu) __ksym; +extern void cgroup_rstat_flush(struct cgroup_subsys_state *css) __ksym; static uint64_t cgroup_id(struct cgroup *cgrp) { @@ -75,7 +75,7 @@ int BPF_PROG(counter, struct cgroup *dst_cgrp, struct task_struct *leader, else if (create_percpu_attach_counter(cg_id, 1)) return 0; - cgroup_rstat_updated(dst_cgrp, bpf_get_smp_processor_id()); + cgroup_rstat_updated(&dst_cgrp->self, bpf_get_smp_processor_id()); return 0; } @@ -141,7 +141,7 @@ int BPF_PROG(dumper, struct bpf_iter_meta *meta, struct cgroup *cgrp) return 1; /* Flush the stats to make sure we get the most updated numbers */ - cgroup_rstat_flush(cgrp); + cgroup_rstat_flush(&cgrp->self); total_counter = bpf_map_lookup_elem(&attach_counters, &cg_id); if (!total_counter) { From patchwork Tue Feb 18 03:14:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 13978853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCBB3C021A9 for ; Tue, 18 Feb 2025 03:15:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 39E622800BB; Mon, 17 Feb 2025 22:15:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 327632800A4; Mon, 17 Feb 2025 22:15:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17A662800BB; Mon, 17 Feb 2025 22:15:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id EA7BA2800A4 for ; Mon, 17 Feb 2025 22:15:09 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A8F6950008 for ; Tue, 18 Feb 2025 03:15:09 +0000 (UTC) X-FDA: 83131599138.05.01A94B4 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf01.hostedemail.com (Postfix) with ESMTP id CC54A40007 for ; Tue, 18 Feb 2025 03:15:07 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dxGYpN8B; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739848507; a=rsa-sha256; cv=none; b=ks85h91fIsLRCwKmcurevpuUFXaZVHfd8luA+Ip1ktkwjNRrpghw5MLJ3Jtz6SChoYtsLr 9RT4P+7pzaRObAISTgWHb56gTB6xh/5S6lZhXXAQP91+KWmVYTu8PIN8sZ7RbGsTSOvxIc 84mmqID66ZZy2MyctiUxZM67OFvOdo8= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dxGYpN8B; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739848507; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jAioZutszBaS0Z/YoqfTJqqVrbVraKzr8HB8FcWf8VM=; b=JI9J/AEMNB2WN/uTpmOKlAWYSEcKBkNmjMfZbzI56gFHYZPVqzlyimU+52r8e7Ng+vmpS4 h4wa4f/tdEbSuQv+IpPxF+m/3WJSDuSsNY7aHL/+KYyznv7oeoDxWNTvUYJliZYN2dZZqj Z3j1fzTDQcUyTnEE7mcMwqy2x/WSq5U= Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2fc3fa00323so5464389a91.3 for ; Mon, 17 Feb 2025 19:15:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739848507; x=1740453307; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jAioZutszBaS0Z/YoqfTJqqVrbVraKzr8HB8FcWf8VM=; b=dxGYpN8B+7kvTW151cA7yx7TnAA+Dnc4eRYHwWpmF/DZXtJqiP+yHdJ097/fM6IDBy yBNNZhRG/aB4cc5vWII77Kjb1eMLA3rWXP3GlWW0Uv+QYihxpzj2N4jyQlwecdnnrFbE f9ThoJ0Bk5I2wHjPgVdrgUbYxFOZUNn7E7ssr2fbndTrSC1xj1uRLPsvzdbE/46UgS+J SHVqFCO7SpNP+15ghs2OIpwrog05WvijCUOkyN7fUuWjHLtpsJBMS7KCpHE9IUhIxfhW n75XV4uVvQEQmSw+ZR1O9ig/kj/74kDL5ftAt1yStUsrOQXnFiID7CAMjORX356QzKSf hv4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739848507; x=1740453307; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jAioZutszBaS0Z/YoqfTJqqVrbVraKzr8HB8FcWf8VM=; b=Uq6mj3Dfmuw1wPRkw7UnJt5kPFNzVK1m2mUDfU8Qnphwf3b9c6ADzuKP23DrYSSzNV yNXz9Y8ZWx5SVT/s3MpjOTGe+8LjtoXSHDJs7v/oId3QyLHXzrnccCq0K55HEsXqec70 MOsA1xA1Ohg8wan7AZ924f3oMRSZVzndoCUhINVefDsjuLt6foHhiAdUqKUrdupa44sy iJjwdJfr3Yuchc8k8OotqEraTY1cA8bPOkBiSCVEJcJnLWzRJNANTZKEsO0pswhqzi+D mS0hFBWXBuw2mWpXXCGGQNS/qXoA0QO/IaHPlD7QxpvbkgWDw5caWGY7+ilT8bgjYFZ0 BvRw== X-Gm-Message-State: AOJu0YxMCU3eBskjYe/kKe4ZM2peg6VfkH/pu8jfrL9pg2xNsMbuj2Zi mvYzGvQ/zcCNgbgXid62tviPGHpaQZgFBgS0oZ3JA81U5XuI0jgK X-Gm-Gg: ASbGnctjvbYyNvw3/YwN9jcDlHwO5kZ2YGB7wByemIUYqEP5hXO9smV1DJ9NsFsBRHh +q8EjyzWA2H0lh4Za06sphnVg3syHNpb78hN3KL9WC3Qy7C1+BMo1b6SOb0lKQfEByyWwuEKN6E HhQwSnqNXgIaXJpynOk11Z6vcBpMx0S1HbxeB2J8WA9FpufrTDxusm8ksTgzLui87HiKDLI6YTM DrlkIA+g0rsRc0AY5H0ekqgToxg4Kn1/JzA2AgxuLObuFIER0mykahg+bUdTq67ZRiDPB2awBXN qO7z1ODnPlSTAUHkwuFXe53XzLjn+tZGt7WuXRK8fvBCLSuG8pcO X-Google-Smtp-Source: AGHT+IHo+onZRqN7Qw0BKGqYOQIOyp+AShjQ0rjEZCDEjA45rubie5vL5mfqBgXSfDiXivgdQ4eXKg== X-Received: by 2002:aa7:8893:0:b0:730:a40f:e6dc with SMTP id d2e1a72fcca58-732618f3469mr19540551b3a.23.1739848506699; Mon, 17 Feb 2025 19:15:06 -0800 (PST) Received: from saturn.. (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732425466besm8763451b3a.9.2025.02.17.19.15.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Feb 2025 19:15:06 -0800 (PST) From: JP Kobryn To: shakeel.butt@linux.dev, tj@kernel.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 04/11] cgroup: introduce cgroup_rstat_ops Date: Mon, 17 Feb 2025 19:14:41 -0800 Message-ID: <20250218031448.46951-5-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250218031448.46951-1-inwardvessel@gmail.com> References: <20250218031448.46951-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: CC54A40007 X-Stat-Signature: wbk6g191i5suabwctduiisi1q789rmjy X-Rspam-User: X-HE-Tag: 1739848507-649722 X-HE-Meta: U2FsdGVkX1+1rTIykh4fdIyYc2/5c9WrH/xOCQ8RQA9iumNjmrBsCuVhwyaWDSlrjoKfMQOied6xpui0nTuOz0rooBpZX4MngTWO0eRcxDQuEBKM1FtZIXVMBP8x/VuhTDnSU8m07LcZ9yaGTmeaLnhlx1vrZqrlvsDwz7ZViSMaRokYS3Nv3V06TgNtArW+K64mFh1tQQuJ5Pcu33RydQf7pNXPZNQh0Oy+MnU4pF1+qTv36+3zFldGuRJhysRyyF/IeDz/ewBx6meQE9fY7iKiXEwCW3eZ1ZnsbTZaREoZwnS3e4wUhinqvL55oRMBiU8Oc14p/BZFPiTWeQGrCm5wDrQyrwW5tvC95tRzS+Ec8GwKxKfz72JO0UaWeXGLmHvImCUtCiOk4tgkq7A2i5thcXJ94iGw0JxyyUBkv0fZFd/Is1c26A1OIsQZHn7AzbLpjMxGL+aejrdTOkU4Ka+0c9scL4QQvoE6alABKtSIiQjpRIZXLbRBHIgJuEPxHCXYpvgZZrHNbjOR5xljVxPfcGFdLpCFz9Wikoz//Yt/lJx1UzkljFj/dxencpQTdfCJ1p1IaP/j88CbVrzHQHW6gTAdqJzq6cNCUk7nyXdLohweVcPdVJlHHqn5smQoZGfCQOUaQn6gx5L0rdTpNAPrCfSgVyhIdgf27Ih7fKApcq8A55l4XTIDyKA/dXL+D0nUppvJW9bM7ji9q4YkcjHI3EwVVaBvAn70sB2YU/nLZEY3xoaiXeiDzH3gO+j48yfqZFaEHrdeFMiCWv/0tcPQRtDe1vbS41nMT7x+Aj1rMYyA4/HzH6+YVO4NxZUFwJ9++KYi7sQcH4ztLAZyk1O58UUfd/b8eTjg2AycAACkGqetpmF58aatHcq0FRJPiN1sRL3zci6XRJuiEZHCqu2x5C+zIfmgJZpMGmRQxyBul8yZEqSDNQ8TR6NZAIxYCyJCCV7H3POmZ+8m1Ao hnrDWy5H zkKHP9kmUppXyuGd/o+vBv3tEmY2qHz0MfABM620hR1Wdt/s3YQ6ZAZAUDqKI2KU8OcBbIpKKeU0/xlYvfubl6lS0I6f1Z9/V/HAqP/aSulBTM0Fqy/g4fiTsTWy3J2AFcsIEurnWuzrEFqtMP+B1WgzgprsoCp359RLyQmchMSqqVnkZT7ddOF9YAK3tKLaQe1RpJ9HZnnlN1pN8LBtUTNQPh+dFR3KwW7TwCarj9dxpQUp73ddLjAsdxGOlOnC9KCW9NqyI+pQuyoNfiFFMMEKv8+hWXZhYANC2ndUGhuBUADx0Sh98G/ldFmhVTfv1BekPe8JnlinPGkjUvxudHjdHJCCkLnHfHHECPZFVbeOo9rSC+oS35PtLxTgQpodz48oZfDKw8Xize6ASeduTkVMNbbpfz6iCdenhJ2B3uZ1zRk8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The cgroup_rstat_ops interface provides a way for type-specific operations to be hidden from the common rstat operations. Use it to decouple the cgroup_subsys_type from within the internal rstat updated/flush routines. The new ops interface allows for greater extensibility in terms of future changes. i.e. public updated/flush api's can be created that accept a arbitrary types, as long as that type has an associated ops interface. Signed-off-by: JP Kobryn --- kernel/cgroup/rstat.c | 131 +++++++++++++++++++++++++++--------------- 1 file changed, 85 insertions(+), 46 deletions(-) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index a32bcd7942a5..a8bb304e49c4 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -9,6 +9,12 @@ #include +struct cgroup_rstat_ops { + struct cgroup_rstat *(*parent_fn)(struct cgroup_rstat *); + struct cgroup *(*cgroup_fn)(struct cgroup_rstat *); + void (*flush_fn)(struct cgroup_rstat *, int); +}; + static DEFINE_SPINLOCK(cgroup_rstat_lock); static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_cpu_lock); @@ -19,7 +25,17 @@ static struct cgroup_rstat_cpu *rstat_cpu(struct cgroup_rstat *rstat, int cpu) return per_cpu_ptr(rstat->rstat_cpu, cpu); } -static struct cgroup_rstat *rstat_parent(struct cgroup_rstat *rstat) +static inline bool is_base_css(struct cgroup_subsys_state *css) +{ + /* css for base stats has no subsystem */ + if (!css->ss) + return true; + + return false; +} + +static struct cgroup_rstat *rstat_parent_via_css( + struct cgroup_rstat *rstat) { struct cgroup_subsys_state *css = container_of( rstat, typeof(*css), rstat); @@ -30,6 +46,33 @@ static struct cgroup_rstat *rstat_parent(struct cgroup_rstat *rstat) return &(css->parent->rstat); } +static struct cgroup *rstat_cgroup_via_css(struct cgroup_rstat *rstat) +{ + struct cgroup_subsys_state *css = + container_of(rstat, struct cgroup_subsys_state, rstat); + + return css->cgroup; +} + +static void rstat_flush_via_css(struct cgroup_rstat *rstat, int cpu) +{ + struct cgroup_subsys_state *css = container_of( + rstat, typeof(*css), rstat); + + if (is_base_css(css)) { + cgroup_base_stat_flush(css->cgroup, cpu); + return; + } + + css->ss->css_rstat_flush(css, cpu); +} + +static struct cgroup_rstat_ops rstat_css_ops = { + .parent_fn = rstat_parent_via_css, + .cgroup_fn = rstat_cgroup_via_css, + .flush_fn = rstat_flush_via_css, +}; + /* * Helper functions for rstat per CPU lock (cgroup_rstat_cpu_lock). * @@ -84,11 +127,11 @@ void _cgroup_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, raw_spin_unlock_irqrestore(cpu_lock, flags); } -static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu) +static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu, + struct cgroup_rstat_ops *ops) { - struct cgroup_subsys_state *css = container_of( - rstat, typeof(*css), rstat); - struct cgroup *cgrp = css->cgroup; + struct cgroup *cgrp; + raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); unsigned long flags; @@ -103,12 +146,13 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu) if (data_race(rstat_cpu(rstat, cpu)->updated_next)) return; + cgrp = ops->cgroup_fn(rstat); flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, true); /* put @rstat and all ancestors on the corresponding updated lists */ while (true) { struct cgroup_rstat_cpu *rstatc = rstat_cpu(rstat, cpu); - struct cgroup_rstat *parent = rstat_parent(rstat); + struct cgroup_rstat *parent = ops->parent_fn(rstat); struct cgroup_rstat_cpu *prstatc; /* @@ -145,7 +189,7 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu) */ __bpf_kfunc void cgroup_rstat_updated(struct cgroup_subsys_state *css, int cpu) { - __cgroup_rstat_updated(&css->rstat, cpu); + __cgroup_rstat_updated(&css->rstat, cpu, &rstat_css_ops); } /** @@ -161,7 +205,8 @@ __bpf_kfunc void cgroup_rstat_updated(struct cgroup_subsys_state *css, int cpu) * cgroups into a stack. The root is pushed by the caller. */ static struct cgroup_rstat *cgroup_rstat_push_children( - struct cgroup_rstat *head, struct cgroup_rstat *child, int cpu) + struct cgroup_rstat *head, struct cgroup_rstat *child, int cpu, + struct cgroup_rstat_ops *ops) { struct cgroup_rstat *chead = child; /* Head of child cgroup level */ struct cgroup_rstat *ghead = NULL; /* Head of grandchild cgroup level */ @@ -174,7 +219,7 @@ static struct cgroup_rstat *cgroup_rstat_push_children( while (chead) { child = chead; chead = child->rstat_flush_next; - parent = rstat_parent(child); + parent = ops->parent_fn(child); /* updated_next is parent cgroup terminated */ while (child != parent) { @@ -220,16 +265,15 @@ static struct cgroup_rstat *cgroup_rstat_push_children( * here is the cgroup root whose updated_next can be self terminated. */ static struct cgroup_rstat *cgroup_rstat_updated_list( - struct cgroup_rstat *root, int cpu) + struct cgroup_rstat *root, int cpu, struct cgroup_rstat_ops *ops) { - struct cgroup_subsys_state *css = container_of( - root, typeof(*css), rstat); - struct cgroup *cgrp = css->cgroup; raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); struct cgroup_rstat_cpu *rstatc = rstat_cpu(root, cpu); struct cgroup_rstat *head = NULL, *parent, *child; + struct cgroup *cgrp; unsigned long flags; + cgrp = ops->cgroup_fn(root); flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, false); /* Return NULL if this subtree is not on-list */ @@ -240,7 +284,7 @@ static struct cgroup_rstat *cgroup_rstat_updated_list( * Unlink @root from its parent. As the updated_children list is * singly linked, we have to walk it to find the removal point. */ - parent = rstat_parent(root); + parent = ops->parent_fn(root); if (parent) { struct cgroup_rstat_cpu *prstatc; struct cgroup_rstat **nextp; @@ -265,7 +309,7 @@ static struct cgroup_rstat *cgroup_rstat_updated_list( child = rstatc->updated_children; rstatc->updated_children = root; if (child != root) - head = cgroup_rstat_push_children(head, child, cpu); + head = cgroup_rstat_push_children(head, child, cpu, ops); unlock_ret: _cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, false); return head; @@ -323,34 +367,30 @@ static inline void __cgroup_rstat_unlock(struct cgroup *cgrp, int cpu_in_loop) } /* see cgroup_rstat_flush() */ -static void cgroup_rstat_flush_locked(struct cgroup_rstat *rstat) +static void cgroup_rstat_flush_locked(struct cgroup_rstat *rstat, + struct cgroup_rstat_ops *ops) __releases(&cgroup_rstat_lock) __acquires(&cgroup_rstat_lock) { - struct cgroup_subsys_state *css = container_of( - rstat, typeof(*css), rstat); - struct cgroup *cgrp = css->cgroup; int cpu; lockdep_assert_held(&cgroup_rstat_lock); for_each_possible_cpu(cpu) { - struct cgroup_rstat *pos = cgroup_rstat_updated_list(rstat, cpu); + struct cgroup_rstat *pos = cgroup_rstat_updated_list( + rstat, cpu, ops); for (; pos; pos = pos->rstat_flush_next) { - struct cgroup_subsys_state *pos_css = container_of( - pos, typeof(*pos_css), rstat); - struct cgroup *pos_cgroup = pos_css->cgroup; - - if (!pos_css->ss) - cgroup_base_stat_flush(pos_cgroup, cpu); - else - pos_css->ss->css_rstat_flush(pos_css, cpu); + struct cgroup *pos_cgroup = ops->cgroup_fn(pos); + ops->flush_fn(pos, cpu); bpf_rstat_flush(pos_cgroup, cgroup_parent(pos_cgroup), cpu); } /* play nice and yield if necessary */ if (need_resched() || spin_needbreak(&cgroup_rstat_lock)) { + struct cgroup *cgrp; + + cgrp = ops->cgroup_fn(rstat); __cgroup_rstat_unlock(cgrp, cpu); if (!cond_resched()) cpu_relax(); @@ -359,16 +399,15 @@ static void cgroup_rstat_flush_locked(struct cgroup_rstat *rstat) } } -static void __cgroup_rstat_flush(struct cgroup_rstat *rstat) +static void __cgroup_rstat_flush(struct cgroup_rstat *rstat, + struct cgroup_rstat_ops *ops) { - struct cgroup_subsys_state *css = container_of( - rstat, typeof(*css), rstat); - struct cgroup *cgrp = css->cgroup; + struct cgroup *cgrp; might_sleep(); - + cgrp = ops->cgroup_fn(rstat); __cgroup_rstat_lock(cgrp, -1); - cgroup_rstat_flush_locked(rstat); + cgroup_rstat_flush_locked(rstat, ops); __cgroup_rstat_unlock(cgrp, -1); } @@ -387,19 +426,19 @@ static void __cgroup_rstat_flush(struct cgroup_rstat *rstat) */ __bpf_kfunc void cgroup_rstat_flush(struct cgroup_subsys_state *css) { - __cgroup_rstat_flush(&css->rstat); + __cgroup_rstat_flush(&css->rstat, &rstat_css_ops); } -static void __cgroup_rstat_flush_hold(struct cgroup_rstat *rstat) +static void __cgroup_rstat_flush_hold(struct cgroup_rstat *rstat, + struct cgroup_rstat_ops *ops) __acquires(&cgroup_rstat_lock) { - struct cgroup_subsys_state *css = container_of( - rstat, typeof(*css), rstat); - struct cgroup *cgrp = css->cgroup; + struct cgroup *cgrp; might_sleep(); + cgrp = ops->cgroup_fn(rstat); __cgroup_rstat_lock(cgrp, -1); - cgroup_rstat_flush_locked(rstat); + cgroup_rstat_flush_locked(rstat, ops); } /** @@ -413,20 +452,20 @@ static void __cgroup_rstat_flush_hold(struct cgroup_rstat *rstat) */ void cgroup_rstat_flush_hold(struct cgroup_subsys_state *css) { - __cgroup_rstat_flush_hold(&css->rstat); + __cgroup_rstat_flush_hold(&css->rstat, &rstat_css_ops); } /** * cgroup_rstat_flush_release - release cgroup_rstat_flush_hold() * @rstat: rstat node used to find associated cgroup used by tracepoint */ -static void __cgroup_rstat_flush_release(struct cgroup_rstat *rstat) +static void __cgroup_rstat_flush_release(struct cgroup_rstat *rstat, + struct cgroup_rstat_ops *ops) __releases(&cgroup_rstat_lock) { - struct cgroup_subsys_state *css = container_of( - rstat, typeof(*css), rstat); - struct cgroup *cgrp = css->cgroup; + struct cgroup *cgrp; + cgrp = ops->cgroup_fn(rstat); __cgroup_rstat_unlock(cgrp, -1); } @@ -436,7 +475,7 @@ static void __cgroup_rstat_flush_release(struct cgroup_rstat *rstat) */ void cgroup_rstat_flush_release(struct cgroup_subsys_state *css) { - __cgroup_rstat_flush_release(&css->rstat); + __cgroup_rstat_flush_release(&css->rstat, &rstat_css_ops); } static void __cgroup_rstat_init(struct cgroup_rstat *rstat) From patchwork Tue Feb 18 03:14:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 13978854 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0E62C021AB for ; Tue, 18 Feb 2025 03:15:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A08CD2800BC; Mon, 17 Feb 2025 22:15:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B6BF2800A4; Mon, 17 Feb 2025 22:15:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 834A52800BC; Mon, 17 Feb 2025 22:15:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5E6642800A4 for ; Mon, 17 Feb 2025 22:15:11 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 25196C17F6 for ; Tue, 18 Feb 2025 03:15:11 +0000 (UTC) X-FDA: 83131599222.07.52F9B14 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf26.hostedemail.com (Postfix) with ESMTP id 4AD50140006 for ; Tue, 18 Feb 2025 03:15:09 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LpM+kM14; spf=pass (imf26.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739848509; a=rsa-sha256; cv=none; b=KawwpehDXszOsBg+4K9KagaHBxf/5zHCirRjH1B6GB7PClR4sxksIj60yG2HOeSveqChlo jwOGPTaENUHphAx3G7N6Z6DgcEby770g7KRxYHJ2HA1Hua+XD4qcXBP7J6F53wgEpfLcfg eEJG5CAkc7R0DtXti6TRBG85eQPyvbE= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LpM+kM14; spf=pass (imf26.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739848509; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZOkghzrE1OsBJ+MkQl3TgUWPiNlLeJUW6eoVzOAbjEY=; b=yzWoLByf40hJqJC8YQ2mQAecE2ldImXw+ZMnI2DGTxPOlNIq5Do+U43s000W5v7W1vW3cE Mho8H3CM8k8SIgRMTD8am166j+nNmAbDKKHnwd5t17FdSid5zgdmVwFvv5/iPvuAHVL2Gr BjEIvppsmQwN92etrIVhuIoEeFVvxWk= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-220d39a5627so71563525ad.1 for ; Mon, 17 Feb 2025 19:15:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739848508; x=1740453308; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZOkghzrE1OsBJ+MkQl3TgUWPiNlLeJUW6eoVzOAbjEY=; b=LpM+kM14vfKU2/qTFyYI8W/X7O+WmCvNVDprENBFxJVkGbmPGK+gGs45POQtoOc2hD q2+nGyO1A5+/c3YqiCSjt0QXHgK7ZYtkwoFeqBDwYpi0hZAlhIgUfVH5/a8zKWg0Wjex rVGlKWbem+2wVnmC+IAsUa15jnG29Vu7JGSa80yL0OivNAsnAW3jrTypnpqtnzoqV3+N qHVFMHbj7XbM16pwJ4cmyUvJfAg7rJaL732IHqbFYzgsDEAxe3/p3VG92NKMzdunX6jG bjggTnsw+Xn3kXvW8/8XTtwi41YXBv9A3TYusOs6DxcBAmraeNiRjd5z+DPCByEK/j1w rahw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739848508; x=1740453308; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZOkghzrE1OsBJ+MkQl3TgUWPiNlLeJUW6eoVzOAbjEY=; b=tFiQjZa9hT0c5/ZZhbwcVuU12LMO4l5SwCc9fr4yBvL/XWAlZTu3fHFQ/oM+u0yPxb P6fFJZUDXb5ZuNbnl1ttebbGOhSh7jbrszUCXzG3BXLOHr1aolhmLd2X0T7zHVkCOSMF Xy0WczfvAk/aWYv/A1pktQElECZo8D8iudKmB2JQWr1JAV6nSRkHDKnOUQ0oTM8TNFUI Bm49E86v3Y8IIyK6KTZ8uCp5Prbbur9NsUYynO+iWZ9oxia4MumNy7jTQLgLpq1JQp8Z r2lNCqJ+BurRHy13tSUKXwo1C3qycFznLsnn72sWFclABoVxgjemL69e0gKraHg/T/Pt eiMw== X-Gm-Message-State: AOJu0YzgGjr27Gmma+UuNs+ezF7ZuZ28rr7IlbekkvxC9fvzp14xLoHs gD+XQvGqoKO9YqXKPkwGaTJrs2X/y0aod0wffsykROFUzRTpfTGC X-Gm-Gg: ASbGncuF4lQyLiq4S9ZxyQ5u+FxSZhxdYP0rCR7I1eO07+v8wYMaCBrWjdBaV8cZy/H wgmozFo8drVlGmrk1B0xJrLPa9AetlNp+XocP8zoyZOnfX5fpyBhiJEtbVNF2Fhq8LMae9lVkiO rJozktrq+RzagmFMhjsxbx0Y3wrg9e+EflqCSxqDr3cqOqeWa1HVQXpRARlTso+b4YRiDKFRbOc JJA3kKMHhbU93T+iNqT11WrEacnWv5pAq+LmLH/QbFgxgVBP4mWPhnhemDAX/WS8YIegoJG1YwS STxK2GgFmT1NR3GTW6N7QGQ+R6CYavgZkwC2PtqyaWcnARYfXOFY X-Google-Smtp-Source: AGHT+IGyYGmJDhosxecR54xtKwc083QS170NIxJt7j0joSa3t2yuUZP3qQKRf4iiTOCbXpwa78B98A== X-Received: by 2002:a05:6a21:3995:b0:1ee:c390:58a4 with SMTP id adf61e73a8af0-1eec3905ce7mr4547041637.2.1739848508183; Mon, 17 Feb 2025 19:15:08 -0800 (PST) Received: from saturn.. (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732425466besm8763451b3a.9.2025.02.17.19.15.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Feb 2025 19:15:07 -0800 (PST) From: JP Kobryn To: shakeel.butt@linux.dev, tj@kernel.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 05/11] cgroup: separate rstat for bpf cgroups Date: Mon, 17 Feb 2025 19:14:42 -0800 Message-ID: <20250218031448.46951-6-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250218031448.46951-1-inwardvessel@gmail.com> References: <20250218031448.46951-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4AD50140006 X-Stat-Signature: inaji5mjt8hmu1c5jtndgnncrukwbg75 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1739848509-400073 X-HE-Meta: U2FsdGVkX1+5Z0DbYLcGPOoIDatx1bg+apRdw1syZcNjvqX1LJ2e/Jykwepz9Q4xsXNXDbkDexFrREwHpREgyJCXof3WXnDCd7mcSFDyaAd4+lUAoN1iAqlGQskQPt5g/bP8F9H9mWKbzXPLk3+DVXH23kCBts0HSyTG67Oqns+q8MdMebt/pjqsOs0EegfEvXdA+hJHd4PHrMcqso4VX2U7knO7yDp1rqzNe+AChLm+J+LCXhBAAAl8FpgKnfII4HxofhNZjRL61xDFSMSCMW61XgYmamfXMueHe+BuNaljci2Rdcpwp17vks5X1r3+jpGcK1JeWQ4+Lfqrko3GkyLFbNdoseqhSjmqrLD0HLhI5tzMBP3F3hRdrrJCIGtxI3etiss1gP/zKlyNxiWmfc0O+IlVDUM/AujScPSBOmHx1tN4Hp6h3+kn//ZGqneujWlFSeVMTyN86gXkoaYxoktUB67M9SojRAmpz266s4GEB745N15oSQLXa+ZrVykIy2p0ftO0oKi5aLVT1mddHV6nOffPBv47dBHqr1a5dtHpMeyIooQeovMq6bgBljc1xzMWqk39K/zOBfLZQsktZE6TrgcLmegBBcszNcZmfDToJgg6zliUiwLH/yI5rBRlH8dMZnYBMROcSOC5GGr5cc0pN6nVQlgB3pCtduY2UN5ZOXrZQ/OYomG2Lfgbz6MLiqWn7FdRjdEpO+2q738+165r2LPUrID6IO+ySM6r5ZIRrxHuq38SFjuWBDaN7fg3Lo7+kOPhXXvhKMdsjDpL1GI8GegNaVppZ5+psL3PjjHHIpBtAr5Cs+I+g2z3Z3sQYyg2f7Jc5Ul2aCdrJ9xoj3XqIfLKbVWw040F4l5GslxmxRbqQqV2YZF4HmELBoqdVkYA4VQ7Enp4eWK3daS8dqae8e1UJoB6kPSr5G8ycBBOa7B/DuIVcbvhxj0Y+elxrM3TDDwE3BIKXtlrWrj 34W0LHWb 9TN5XlRBT/195c7ZFLHIaD0fNQSsu5qLiUKatAT+gLgU+gt+RPnFWPrGbcNb+3/n7RG+Cd/AW4RRwNZrfJnEbpu5w7VkmwOvElBtswxBj/Chi/xHpAkyQCL2o2AnYOvfigNdqZ2Gq+Ga7z8t2UsQKAlW7zwe1C4ASL8ZtsIx+sZTuUjIPOYAAyYKVrTus6vninZ1lXxQ6qZtvMoWjIeH2qsuLrusoq8Bf/4v+rV1XrjBXXcMeFf/9ff1ZaCcAg0gjm0VCaJ5xaomTppkbaEU+yVsQdqpF6bA/kTQRLd4os1NIbwysHxE95a02jb0nOO+ROzbFfQsTa9hty94DXLzkMX8OshDt/vCB2PcgG22UUBHfI74ZcNNdD8gbaQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The processing of bpf cgroup stats is tied to the rstat actions of other subsystems. Make changes to have them updated/flushed independently. Give the cgroup_bpf struct its own cgroup_rstat instance and define a new cgroup_rstat_ops instance specifically for the cgroup_bpf. Then replace the kfunc status of the existing updated/flush api calls with non-kfunc status. As an alternative, create new updated/flush kfuncs specifically for bpf cgroups. In these new kfuncs, make use of the bpf-specific rstat ops to plumb back in to the existing rstat routines. Where applicable, use pre-processor conditionals to define bpf rstat related stuff. Signed-off-by: JP Kobryn --- include/linux/bpf-cgroup-defs.h | 3 + include/linux/cgroup.h | 3 + kernel/bpf/cgroup.c | 6 ++ kernel/cgroup/cgroup-internal.h | 5 + kernel/cgroup/rstat.c | 95 ++++++++++++++++--- .../selftests/bpf/progs/btf_type_tag_percpu.c | 4 +- .../bpf/progs/cgroup_hierarchical_stats.c | 8 +- 7 files changed, 107 insertions(+), 17 deletions(-) diff --git a/include/linux/bpf-cgroup-defs.h b/include/linux/bpf-cgroup-defs.h index 0985221d5478..e68359f861fb 100644 --- a/include/linux/bpf-cgroup-defs.h +++ b/include/linux/bpf-cgroup-defs.h @@ -75,6 +75,9 @@ struct cgroup_bpf { /* cgroup_bpf is released using a work queue */ struct work_struct release_work; + + /* per-cpu recursive resource statistics */ + struct cgroup_rstat rstat; }; #else /* CONFIG_CGROUP_BPF */ diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index eec970622419..253ce4bff576 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -836,6 +836,9 @@ static inline bool cgroup_task_frozen(struct task_struct *task) #endif /* !CONFIG_CGROUPS */ #ifdef CONFIG_CGROUP_BPF +void bpf_cgroup_rstat_updated(struct cgroup *cgrp, int cpu); +void bpf_cgroup_rstat_flush(struct cgroup *cgrp); + static inline void cgroup_bpf_get(struct cgroup *cgrp) { percpu_ref_get(&cgrp->bpf.refcnt); diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c index 46e5db65dbc8..72bcfdbda6b1 100644 --- a/kernel/bpf/cgroup.c +++ b/kernel/bpf/cgroup.c @@ -210,6 +210,7 @@ void cgroup_bpf_offline(struct cgroup *cgrp) { cgroup_get(cgrp); percpu_ref_kill(&cgrp->bpf.refcnt); + bpf_cgroup_rstat_exit(&cgrp->bpf); } static void bpf_cgroup_storages_free(struct bpf_cgroup_storage *storages[]) @@ -490,6 +491,10 @@ int cgroup_bpf_inherit(struct cgroup *cgrp) if (ret) return ret; + ret = bpf_cgroup_rstat_init(&cgrp->bpf); + if (ret) + goto cleanup_ref; + for (p = cgroup_parent(cgrp); p; p = cgroup_parent(p)) cgroup_bpf_get(p); @@ -513,6 +518,7 @@ int cgroup_bpf_inherit(struct cgroup *cgrp) for (p = cgroup_parent(cgrp); p; p = cgroup_parent(p)) cgroup_bpf_put(p); +cleanup_ref: percpu_ref_exit(&cgrp->bpf.refcnt); return -ENOMEM; diff --git a/kernel/cgroup/cgroup-internal.h b/kernel/cgroup/cgroup-internal.h index 87d062baff90..bba1a1794de2 100644 --- a/kernel/cgroup/cgroup-internal.h +++ b/kernel/cgroup/cgroup-internal.h @@ -274,6 +274,11 @@ void cgroup_rstat_exit(struct cgroup_subsys_state *css); void cgroup_rstat_boot(void); void cgroup_base_stat_cputime_show(struct seq_file *seq); +#ifdef CONFIG_CGROUP_BPF +int bpf_cgroup_rstat_init(struct cgroup_bpf *bpf); +void bpf_cgroup_rstat_exit(struct cgroup_bpf *bpf); +#endif /* CONFIG_CGROUP_BPF */ + /* * namespace.c */ diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index a8bb304e49c4..14dd8217db64 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -73,6 +73,47 @@ static struct cgroup_rstat_ops rstat_css_ops = { .flush_fn = rstat_flush_via_css, }; +#ifdef CONFIG_CGROUP_BPF +__weak noinline void bpf_rstat_flush(struct cgroup *cgrp, + struct cgroup *parent, int cpu); + +static struct cgroup *rstat_cgroup_via_bpf(struct cgroup_rstat *rstat) +{ + struct cgroup_bpf *bpf = container_of(rstat, typeof(*bpf), rstat); + struct cgroup *cgrp = container_of(bpf, typeof(*cgrp), bpf); + + return cgrp; +} + +static struct cgroup_rstat *rstat_parent_via_bpf( + struct cgroup_rstat *rstat) +{ + struct cgroup *cgrp, *cgrp_parent; + + cgrp = rstat_cgroup_via_bpf(rstat); + cgrp_parent = cgroup_parent(cgrp); + if (!cgrp_parent) + return NULL; + + return &(cgrp_parent->bpf.rstat); +} + +static void rstat_flush_via_bpf(struct cgroup_rstat *rstat, int cpu) +{ + struct cgroup *cgrp, *cgrp_parent; + + cgrp = rstat_cgroup_via_bpf(rstat); + cgrp_parent = cgroup_parent(cgrp); + bpf_rstat_flush(cgrp, cgrp_parent, cpu); +} + +static struct cgroup_rstat_ops rstat_bpf_ops = { + .parent_fn = rstat_parent_via_bpf, + .cgroup_fn = rstat_cgroup_via_bpf, + .flush_fn = rstat_flush_via_bpf, +}; +#endif /* CONFIG_CGROUP_BPF */ + /* * Helper functions for rstat per CPU lock (cgroup_rstat_cpu_lock). * @@ -187,11 +228,18 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu, * rstat_cpu->updated_children list. See the comment on top of * cgroup_rstat_cpu definition for details. */ -__bpf_kfunc void cgroup_rstat_updated(struct cgroup_subsys_state *css, int cpu) +void cgroup_rstat_updated(struct cgroup_subsys_state *css, int cpu) { __cgroup_rstat_updated(&css->rstat, cpu, &rstat_css_ops); } +#ifdef CONFIG_CGROUP_BPF +__bpf_kfunc void bpf_cgroup_rstat_updated(struct cgroup *cgroup, int cpu) +{ + __cgroup_rstat_updated(&(cgroup->bpf.rstat), cpu, &rstat_bpf_ops); +} +#endif /* CONFIG_CGROUP_BPF */ + /** * cgroup_rstat_push_children - push children cgroups into the given list * @head: current head of the list (= subtree root) @@ -330,8 +378,7 @@ static struct cgroup_rstat *cgroup_rstat_updated_list( __bpf_hook_start(); -__weak noinline void bpf_rstat_flush(struct cgroup *cgrp, - struct cgroup *parent, int cpu) +void bpf_rstat_flush(struct cgroup *cgrp, struct cgroup *parent, int cpu) { } @@ -379,12 +426,8 @@ static void cgroup_rstat_flush_locked(struct cgroup_rstat *rstat, struct cgroup_rstat *pos = cgroup_rstat_updated_list( rstat, cpu, ops); - for (; pos; pos = pos->rstat_flush_next) { - struct cgroup *pos_cgroup = ops->cgroup_fn(pos); - + for (; pos; pos = pos->rstat_flush_next) ops->flush_fn(pos, cpu); - bpf_rstat_flush(pos_cgroup, cgroup_parent(pos_cgroup), cpu); - } /* play nice and yield if necessary */ if (need_resched() || spin_needbreak(&cgroup_rstat_lock)) { @@ -424,11 +467,18 @@ static void __cgroup_rstat_flush(struct cgroup_rstat *rstat, * * This function may block. */ -__bpf_kfunc void cgroup_rstat_flush(struct cgroup_subsys_state *css) +void cgroup_rstat_flush(struct cgroup_subsys_state *css) { __cgroup_rstat_flush(&css->rstat, &rstat_css_ops); } +#ifdef CONFIG_CGROUP_BPF +__bpf_kfunc void bpf_cgroup_rstat_flush(struct cgroup *cgroup) +{ + __cgroup_rstat_flush(&(cgroup->bpf.rstat), &rstat_bpf_ops); +} +#endif /* CONFIG_CGROUP_BPF */ + static void __cgroup_rstat_flush_hold(struct cgroup_rstat *rstat, struct cgroup_rstat_ops *ops) __acquires(&cgroup_rstat_lock) @@ -532,6 +582,27 @@ void cgroup_rstat_exit(struct cgroup_subsys_state *css) __cgroup_rstat_exit(rstat); } +#ifdef CONFIG_CGROUP_BPF +int bpf_cgroup_rstat_init(struct cgroup_bpf *bpf) +{ + struct cgroup_rstat *rstat = &bpf->rstat; + + rstat->rstat_cpu = alloc_percpu(struct cgroup_rstat_cpu); + if (!rstat->rstat_cpu) + return -ENOMEM; + + __cgroup_rstat_init(rstat); + + return 0; +} + +void bpf_cgroup_rstat_exit(struct cgroup_bpf *bpf) +{ + __cgroup_rstat_flush(&bpf->rstat, &rstat_bpf_ops); + __cgroup_rstat_exit(&bpf->rstat); +} +#endif /* CONFIG_CGROUP_BPF */ + void __init cgroup_rstat_boot(void) { int cpu; @@ -754,10 +825,11 @@ void cgroup_base_stat_cputime_show(struct seq_file *seq) cgroup_force_idle_show(seq, &cgrp->bstat); } +#ifdef CONFIG_CGROUP_BPF /* Add bpf kfuncs for cgroup_rstat_updated() and cgroup_rstat_flush() */ BTF_KFUNCS_START(bpf_rstat_kfunc_ids) -BTF_ID_FLAGS(func, cgroup_rstat_updated) -BTF_ID_FLAGS(func, cgroup_rstat_flush, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_cgroup_rstat_updated) +BTF_ID_FLAGS(func, bpf_cgroup_rstat_flush, KF_SLEEPABLE) BTF_KFUNCS_END(bpf_rstat_kfunc_ids) static const struct btf_kfunc_id_set bpf_rstat_kfunc_set = { @@ -771,3 +843,4 @@ static int __init bpf_rstat_kfunc_init(void) &bpf_rstat_kfunc_set); } late_initcall(bpf_rstat_kfunc_init); +#endif /* CONFIG_CGROUP_BPF */ diff --git a/tools/testing/selftests/bpf/progs/btf_type_tag_percpu.c b/tools/testing/selftests/bpf/progs/btf_type_tag_percpu.c index 310cd51e12e8..da15ada56218 100644 --- a/tools/testing/selftests/bpf/progs/btf_type_tag_percpu.c +++ b/tools/testing/selftests/bpf/progs/btf_type_tag_percpu.c @@ -45,7 +45,7 @@ int BPF_PROG(test_percpu2, struct bpf_testmod_btf_type_tag_2 *arg) SEC("tp_btf/cgroup_mkdir") int BPF_PROG(test_percpu_load, struct cgroup *cgrp, const char *path) { - g = (__u64)cgrp->self.rstat.rstat_cpu->updated_children; + g = (__u64)cgrp->bpf.rstat.rstat_cpu->updated_children; return 0; } @@ -57,7 +57,7 @@ int BPF_PROG(test_percpu_helper, struct cgroup *cgrp, const char *path) cpu = bpf_get_smp_processor_id(); rstat = (struct cgroup_rstat_cpu *)bpf_per_cpu_ptr( - cgrp->self.rstat.rstat_cpu, cpu); + cgrp->bpf.rstat.rstat_cpu, cpu); if (rstat) { /* READ_ONCE */ *(volatile int *)rstat; diff --git a/tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c b/tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c index 10c803c8dc70..24450dd4d3f3 100644 --- a/tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c +++ b/tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c @@ -37,8 +37,8 @@ struct { __type(value, struct attach_counter); } attach_counters SEC(".maps"); -extern void cgroup_rstat_updated(struct cgroup_subsys_state *css, int cpu) __ksym; -extern void cgroup_rstat_flush(struct cgroup_subsys_state *css) __ksym; +extern void bpf_cgroup_rstat_updated(struct cgroup *cgrp, int cpu) __ksym; +extern void bpf_cgroup_rstat_flush(struct cgroup *cgrp) __ksym; static uint64_t cgroup_id(struct cgroup *cgrp) { @@ -75,7 +75,7 @@ int BPF_PROG(counter, struct cgroup *dst_cgrp, struct task_struct *leader, else if (create_percpu_attach_counter(cg_id, 1)) return 0; - cgroup_rstat_updated(&dst_cgrp->self, bpf_get_smp_processor_id()); + bpf_cgroup_rstat_updated(dst_cgrp, bpf_get_smp_processor_id()); return 0; } @@ -141,7 +141,7 @@ int BPF_PROG(dumper, struct bpf_iter_meta *meta, struct cgroup *cgrp) return 1; /* Flush the stats to make sure we get the most updated numbers */ - cgroup_rstat_flush(&cgrp->self); + bpf_cgroup_rstat_flush(cgrp); total_counter = bpf_map_lookup_elem(&attach_counters, &cg_id); if (!total_counter) { From patchwork Tue Feb 18 03:14:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 13978855 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA7EBC021A9 for ; Tue, 18 Feb 2025 03:15:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 17BF82800BD; Mon, 17 Feb 2025 22:15:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 12B892800A4; Mon, 17 Feb 2025 22:15:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC0EF2800BD; Mon, 17 Feb 2025 22:15:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CB5182800A4 for ; Mon, 17 Feb 2025 22:15:12 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8DF01B5267 for ; Tue, 18 Feb 2025 03:15:12 +0000 (UTC) X-FDA: 83131599264.27.4C31A17 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf27.hostedemail.com (Postfix) with ESMTP id BE74A40008 for ; Tue, 18 Feb 2025 03:15:10 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=QmxS5CzF; spf=pass (imf27.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.216.50 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739848510; a=rsa-sha256; cv=none; b=sUX0WfSCIjuxFjfqNAYhfnRcsuCSSgPZKgsxfmDXBlJ2cz/LS2S6i73ZeTH0QTKBFm55bU wJG8XLWq81eaUeKEcTWkWJaXnW8gyZBff008gqMD1omDtTNtUTfng8bMulbE5hnGLdxl0p v6Zht7XLKIPjdvHQl2I5t9Kaizs6hS4= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=QmxS5CzF; spf=pass (imf27.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.216.50 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739848510; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mO3hb/+vVe+5fAxNnnDqDkFO7GypFlKfFguhCd8DtOs=; b=mk6asr7tAaSWCs0jxzu/pF4O3Iveyle4D2auVZa4E1/hLYmNAzzGlsYRhUXzM0mDzq5jc8 pGFZ5qDPAogvjbeSD0dfxcvGd9OfJCdevgcyRaYx8ZWZfcU6mEosrQ+0vYT3hN3KtuFYIm ue50x60wyNdbS/jbo7LsHBy7a2NjlAM= Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-2fa8ada6662so9287215a91.1 for ; Mon, 17 Feb 2025 19:15:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739848510; x=1740453310; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mO3hb/+vVe+5fAxNnnDqDkFO7GypFlKfFguhCd8DtOs=; b=QmxS5CzFtPMhChUinDD99kfEZbLLKTyi2c3J6zLmmkf94PYV6RQoy/o+YnJhlqwTdL g3iVgS0maPDI3Ypk3qU4jUm+dW9Z9BplDzqqR66a/peJDXXUmEuVZCuQjbHOnpWnsyO9 nbAM+CYsbkYvBaXtVi4DMJ0xnIX09RDRZWDxKH73LKXZgiSqksSycAZaU/WsBFDN1rsa SxQU+04UFb03BtHSR10c+fJJKiUhTW/ahfvZJKZ2BItnD4/+QyljJ1iacWcKE8j3Ei0v S74NHrE1WOSIgchVab1LCeDrNkI/RHa50FgkQShh4lFOk5yNy8dl0x5FAxTD2K6O+4T+ MwLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739848510; x=1740453310; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mO3hb/+vVe+5fAxNnnDqDkFO7GypFlKfFguhCd8DtOs=; b=GCq6aJwHVWt3ZEZx9qDHYb4QbNJNSOtFvzCAzpl2facjyw7gRsO05uprh0ReamkpNK 8nF1A52ROx60YPeXCCNnaBANwbhcfnLrnvkvDYGwPAKIglI9yHSmRji2aZQGCuyUzomC 8KdyR2MDaJofS1pc7eTGFTKiC+ShoDKjC2U+pmbTpbFxMKLBVHRgdqLozNf8N41Lu+KN S6TJUpGAPp+TbAhUFMPJi539Ymak0SU/RbAiMBHY8C+HNjm7+v7oPB/NS0+wmZb5gYbo uWudBSq9nMkEO9lceJAS/qNJRi84mzoGRzWJUVPPW5NXSgxfOHjRPbm3rR9iLYZgKdCw L46A== X-Gm-Message-State: AOJu0Yy0tDDQw8uBYv/Y/XHSLv2NRsMXGzw2Nu6TAVJKmveprIf4nOkI LoMQ9yUhjn5o889l8PywrLtubzIYhx07K0qFN0naet7ldgNU+ZD3 X-Gm-Gg: ASbGncsmY1H9V9Jnr0NwCZiIZNyY4W5lOuOIj7GjK2iU240juauJJdA+wSGTojDVt/9 LWVbcVkRS7x5Zcf7vQ1srwNava29JnP4JekJVGgghqX2XP2o3do6D/s4DU4iC3b56EmfQy8Oxsg CtqXrd2MfTKtsAlcUtkrc5XEHxvmGJ1hBenf2Jv0eoXPOeyK4Z0VFhoDr25XuZbs2g2JbV0ZAqF eHMPUjsQ2kbsUOueADhWmLjZT7k6gzOWZLsRNgC4YWQfyyMH5TLhe4hEnEMzi3a5jbgfU9J7HLz 65x6hk6hdELitTRxguEqw1O5gzgQh8vEQlcJuylz4cSwuTE3RKiB X-Google-Smtp-Source: AGHT+IHcdvcOMi0AxH886B36LGeN6TPfeRO/YTGzu+no+b1L3ofR6FaqviJ5YbkBRETnjOFBfr1SiA== X-Received: by 2002:a05:6a00:198c:b0:730:8a0a:9ef9 with SMTP id d2e1a72fcca58-7326190d9edmr18330498b3a.22.1739848509625; Mon, 17 Feb 2025 19:15:09 -0800 (PST) Received: from saturn.. (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732425466besm8763451b3a.9.2025.02.17.19.15.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Feb 2025 19:15:09 -0800 (PST) From: JP Kobryn To: shakeel.butt@linux.dev, tj@kernel.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 06/11] cgroup: rstat lock indirection Date: Mon, 17 Feb 2025 19:14:43 -0800 Message-ID: <20250218031448.46951-7-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250218031448.46951-1-inwardvessel@gmail.com> References: <20250218031448.46951-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BE74A40008 X-Stat-Signature: znjhr9wuz34qpnex1t4i6tajm93ee9st X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1739848510-472832 X-HE-Meta: U2FsdGVkX1+Trwg3CNj9zusZSwR7++BsOuoreS8RJw6NqEvC1p6B+UG30KwvymRz/+Kg2GfQiVfeixSOfbrzpYjODa0QriPlQC3Cc9TvQrA3kHkZw+myrHuDke0Ou97uNM7IostL5wiPa2nvvwannCZEaSnSsgatoCN5axc94njWZyNC/UB4dZHoXca7y5GYMWFOkjZYUlIP6jCh+ExkDbBvZQjYHgpALvOKFN6wJc/c3hfllYkPi1OHk0OAnMoPKlG6B6Kux8w0lGJ4HevcV6rK2KC7XSwB9hRdlknTmp9R27p/qBYUSBJOMgtDbhrVnuDdY/U1YYGYh+CkQsyJfb4ZDKAqNQm4vyr/DvjNPvr8Z13gwGo25+gCBZ47r93jwpowmTBRba3ryup2pbR8kO5oojn7xo8tkoTHDH5prSUGthPu+eMaEfzQHwKo/IWL01x7+gCYnw27Z+KSz1bJscZ5ajRay8rHDb5+zahkVK6apq3vM1BN4RQwp+foJ5LHujhN7gjL2KBtFRCYZRJUIRCYQl0Qy9SibAMbGmN8Ngm2SoklPrT9pJQJEHZ3DBax2GBds/d3wVIyhG/ajdlYQXW1OwCLdzKCRPVffh78pX/SqRZMkX5pspKMt/bIKr7AoImuaKrNQ3IjVVEmtaO2CDZpeEI1ZAGr+zBQCpsmOFbx+4JuhwvmBsuNzgZJ6faKSevoNJ1fC50EcBmFDS9J3hICMj6ab3W+2jn25SA4ARTU281X70twvN/W9KV8483bLHfzi7kOws8Xd9vzTut41FZHdPPkArsQgBy5QfnJYt/BRGIb3KD3Mi+oiwCJVzjmlOF5JUEXV76VE/Em/BQOYojHFQGqqjhPFqdHWIGTk2EsiU9Xv1Ul9+9IyN5V3cijLypoRmkFBqgQAlCkrON2LN8Tpg4DRqyxGHl40lJeaDoaTT2AAwQRLO7YcmdSUMnazOr23FpW1ioHkEfADwq yOvamtfL 1J/X4guzDWxdLU89qmQPCJ/alc5dNXD9pNf7VjrMA/YgzNPonUcdHhs/z4S6qEw7F16Jr4v/X5/t8s0Jqd7x5DfWgmBmEQJbsRHQ+sE0hcqAJAN9SMqjT2mxHm35Tfnk3wl/xAirUGr1aztFcXnfS5S0sSPI1Yaz5upY7CUydOWnyQzlvRiI1ra5v7B7HjaY3cQwSSsJx798ddojf7nIweexNhlb6B/Zcafy2bw+/3rVTwKopLNdx0Q1pEm+RiBnsk+4W75uQeb1Csr/W7xRcRS7l/LZJYqHlXh+DmyZmVgVCL377TNayd+w+FO9GbPVmYwxKzjOediIOnyMb3RCoVht2ffQBh3tBY7+22XJBB71W9EFJQxddz2vRnA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.010120, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Instead of accessing the target lock directly via global var, access it indirectly in the form of a new parameter. Also change the ordering of the parameters to be consistent with the related per-cpu locking function _cgroup_rstat_cpu_lock(). Signed-off-by: JP Kobryn Reviewed-by: Shakeel Butt --- kernel/cgroup/rstat.c | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 14dd8217db64..26c75629bca2 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -385,7 +385,7 @@ void bpf_rstat_flush(struct cgroup *cgrp, struct cgroup *parent, int cpu) __bpf_hook_end(); /* - * Helper functions for locking cgroup_rstat_lock. + * Helper functions for locking. * * This makes it easier to diagnose locking issues and contention in * production environments. The parameter @cpu_in_loop indicate lock @@ -393,24 +393,26 @@ __bpf_hook_end(); * value -1 is used when obtaining the main lock else this is the CPU * number processed last. */ -static inline void __cgroup_rstat_lock(struct cgroup *cgrp, int cpu_in_loop) - __acquires(&cgroup_rstat_lock) +static inline void __cgroup_rstat_lock(spinlock_t *lock, + struct cgroup *cgrp, int cpu_in_loop) + __acquires(lock) { bool contended; - contended = !spin_trylock_irq(&cgroup_rstat_lock); + contended = !spin_trylock_irq(lock); if (contended) { trace_cgroup_rstat_lock_contended(cgrp, cpu_in_loop, contended); - spin_lock_irq(&cgroup_rstat_lock); + spin_lock_irq(lock); } trace_cgroup_rstat_locked(cgrp, cpu_in_loop, contended); } -static inline void __cgroup_rstat_unlock(struct cgroup *cgrp, int cpu_in_loop) - __releases(&cgroup_rstat_lock) +static inline void __cgroup_rstat_unlock(spinlock_t *lock, + struct cgroup *cgrp, int cpu_in_loop) + __releases(lock) { trace_cgroup_rstat_unlock(cgrp, cpu_in_loop, false); - spin_unlock_irq(&cgroup_rstat_lock); + spin_unlock_irq(lock); } /* see cgroup_rstat_flush() */ @@ -434,10 +436,10 @@ static void cgroup_rstat_flush_locked(struct cgroup_rstat *rstat, struct cgroup *cgrp; cgrp = ops->cgroup_fn(rstat); - __cgroup_rstat_unlock(cgrp, cpu); + __cgroup_rstat_unlock(&cgroup_rstat_lock, cgrp, cpu); if (!cond_resched()) cpu_relax(); - __cgroup_rstat_lock(cgrp, cpu); + __cgroup_rstat_lock(&cgroup_rstat_lock, cgrp, cpu); } } } @@ -449,9 +451,9 @@ static void __cgroup_rstat_flush(struct cgroup_rstat *rstat, might_sleep(); cgrp = ops->cgroup_fn(rstat); - __cgroup_rstat_lock(cgrp, -1); + __cgroup_rstat_lock(&cgroup_rstat_lock, cgrp, -1); cgroup_rstat_flush_locked(rstat, ops); - __cgroup_rstat_unlock(cgrp, -1); + __cgroup_rstat_unlock(&cgroup_rstat_lock, cgrp, -1); } /** @@ -487,7 +489,7 @@ static void __cgroup_rstat_flush_hold(struct cgroup_rstat *rstat, might_sleep(); cgrp = ops->cgroup_fn(rstat); - __cgroup_rstat_lock(cgrp, -1); + __cgroup_rstat_lock(&cgroup_rstat_lock, cgrp, -1); cgroup_rstat_flush_locked(rstat, ops); } @@ -516,7 +518,7 @@ static void __cgroup_rstat_flush_release(struct cgroup_rstat *rstat, struct cgroup *cgrp; cgrp = ops->cgroup_fn(rstat); - __cgroup_rstat_unlock(cgrp, -1); + __cgroup_rstat_unlock(&cgroup_rstat_lock, cgrp, -1); } /** From patchwork Tue Feb 18 03:14:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 13978856 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B0D9C021A9 for ; Tue, 18 Feb 2025 03:15:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8F72D2800BE; Mon, 17 Feb 2025 22:15:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A64C2800A4; Mon, 17 Feb 2025 22:15:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F92E2800BE; Mon, 17 Feb 2025 22:15:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4D3242800A4 for ; Mon, 17 Feb 2025 22:15:15 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 10ADF1C9940 for ; Tue, 18 Feb 2025 03:15:15 +0000 (UTC) X-FDA: 83131599390.18.A866CF0 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf25.hostedemail.com (Postfix) with ESMTP id 38FB8A000A for ; Tue, 18 Feb 2025 03:15:13 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YtkmVOkI; spf=pass (imf25.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739848513; a=rsa-sha256; cv=none; b=v0NufTtCJn+c8RWBKD1mtDy9FDnmJIRZCifqWj6coVFSw5IqkYxlq138VW9x+rToq3Tbhd LE0gYLCchhVa/S9mvL6SCZ6sCvbS/B8rwSk7EKnqtz/2QsDQ989jkPOanyPfKkXJiRQsmA vBBawr6cZXFxP5OOJ2nGbJc8wl8YFnk= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YtkmVOkI; spf=pass (imf25.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739848513; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i6fGNeZWtM3FGTNz/p/Bk400jphjU7NfeytH78W3bTI=; b=xi81XLDV5sTF5aFe5CgOn7KhPx5esSAIahGVeGYDEtOacYoFnRBZ/XEqe/MZz3uj9Tm60q erRyKA/Fxue8kmHdaykPErX8Bm4XBT0NzzR6N0cEhauL//c5a/14craSWGH5bA6NhyDBTl Zt3Xfuq6uRFhUvIDbR0m9E01HBznyS8= Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2fbfa8c73a6so9271966a91.2 for ; Mon, 17 Feb 2025 19:15:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739848512; x=1740453312; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=i6fGNeZWtM3FGTNz/p/Bk400jphjU7NfeytH78W3bTI=; b=YtkmVOkIOm6Xc46CHt7RoBm+sQo0VC4tdKjQB0BQ4AOOKUYK+NBGe9PpJWhdH0yfvM KGZ8z+t0f0/QnxJjxw1p6LToyD8/33Pxi0SP+uuAX+H0ZxzcINcTAWn4/wCq0EYXXJW4 mWuX0Zy8gxR1SlXarEy48NkeOvsCiEiWaccKakKrju4fnc95sO0dg+ej8Yuxq2UBx8p5 dGtm+viUhgqSRU5YkRW8eDOXlimlvVZNwMUjC8T93EUake+sbVyNL1Y9Hj+cEKkuXU98 KS39yGEOOE9UTKkUhGVvqMiRx7jAaEamuEbuC5XFpBaT1RKGNtIjBC818CIUekSS0FpW SG6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739848512; x=1740453312; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i6fGNeZWtM3FGTNz/p/Bk400jphjU7NfeytH78W3bTI=; b=tai6z1+n5zFMnfQdicO6UXY4jZf2XesSBgJwnGiA7pNLAQO9jeAXQUYgDH9JnFte5F vVfz6XEy8xeo0o+Do2353Hn6afxONWofe0rgpEJ53Knq/NJttfZo818PBeQvvJc+GjSF Aayrg/92FmUzZEimCNNqV+o7l09MyWF0CnGhlLIkZM3+6ZK6wBfTzooQW33Dq6w6965m uaM2XhlQKiYA90Tp8JvV7b1Vuc3l7d32yfswnzyfxAX6dyfx3wlWwVTI9+RD22wfXnKW PmsxxbqsTtKJTd7iV9B6S17zhvq7Oer7FTbQmV2LPtqJsr/Rz/QzkbI5A4nOnsPIWLhz zukQ== X-Gm-Message-State: AOJu0YyWqD7kO6sW2TVx5LKZd7KaH2XHEDUa4BGGUWwaJ19xr6ui6Ha+ IwFrS/ZXZqwz3PqTw4oubl3ASUMd3kE8HjSq2lnhcT0gMTO4HT3x X-Gm-Gg: ASbGnctzNEEmqtzvWEn+OBnZYcTkocS3rjWQ1OxAG0cEHLfCU7Fmh/2sIl6HvouF15L a+CcI/YtVg2ceVIpeeUeJhXk9UaPGvrZXfB9M31iEnUOZ05lRo6acJAiO80/EukywK/6UDJ50tT 3AWX18wNc//OTo+BGSx9xv+TykOUD8KUJaQLPjecSyB8/6T4qsSx9VDMTt1SwLCot1TwpIpwjy6 GBUD+CjZeMJFU1CuOOXx1GxobN8Ifayz/wcziohgZkpdRo2TkDRPRToMbRrC6Q5eScJ1avVEMkr rvy+mf3cyqfIzy77GrYox4t4GGjtHioRTDW/8pOe29PHNvK/6Is6 X-Google-Smtp-Source: AGHT+IEdc2CfR68/O4SRSN2xNZsygVxwsIe5F71fhvH7zgtYBNKyQ2HCTgHwt2ieGZ7qbMGNDozeDQ== X-Received: by 2002:a05:6a20:4323:b0:1ee:83b8:7c6b with SMTP id adf61e73a8af0-1ee8cc084demr18831160637.39.1739848512098; Mon, 17 Feb 2025 19:15:12 -0800 (PST) Received: from saturn.. (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732425466besm8763451b3a.9.2025.02.17.19.15.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Feb 2025 19:15:11 -0800 (PST) From: JP Kobryn To: shakeel.butt@linux.dev, tj@kernel.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 07/11] cgroup: fetch cpu-specific lock in rstat cpu lock helpers Date: Mon, 17 Feb 2025 19:14:44 -0800 Message-ID: <20250218031448.46951-8-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250218031448.46951-1-inwardvessel@gmail.com> References: <20250218031448.46951-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 38FB8A000A X-Stat-Signature: 97kf9exndayoztdqqd6y19a3fywy9gs1 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1739848513-990169 X-HE-Meta: U2FsdGVkX18dmBaMJdjPXHo1n3mm5Dkv+SZ6FMOlKxENnB0zU6tl1oVRw7qCrEMxq7kIxKVX8l8koP2mtkyz41Jv7JAhZs5x90w93uDOG8ebJ3hrqjsRSbXioDBCgd96Zd2LgynKOMAADwPOMio1cbo4zaR5VRE/6KI8IpUv0eAsuOd+NPwl5JdixZvLX89ELv+u2Oxn2Z4ekmtcHOduJkY1gdT+09BVRDng4OsDVkkzdhPBqDbeCGbch1abzz0vk2F666djRvbO7lfjHlepTFmwKRJD+3Ii8zHIuLDDBbTwr+lP0iAqzQkL1g+xd4d6Hv1L3MkT3T14XxcWcfOTUdWEZT/BgQuYb2wUGHJVzO84RYAtbW3I/lAboHDKAIdRR8nZWgLmzUYdvdcga7OyLnzx8kmCgiK9MnURX2jBD0UUD1RlIPxsFl18IW2iF3y0S6mYV37yVGe12Q0OUQR9Gck64l5GZoypRaiA4zDB1fwG+PRdhHoVUBzN/znoWdI+7K7NEDy4Q6rPCvzIV4YIWnW5jSFrqV++DixgYmHWLBanic4DhQbGb/IGUYehr5hBzugJLTrPqzSEdtrGHkckBwtaGkHfZiFoMI5aDKz9VBMdR08+NOCklSIfBlLUGTYzXYQmMUBf/801GcEiHDFt4kXeU278BdS8Wq57QTSsTW8S9Ya+tiYdjxVeI5l5feKr9pXN5IZcE1NSG3PdeUMR906qqcHIBHKkL8v2V5QdX0xOWm/Ck3jwO6bn1wbj5o9FbtUUAmnv8CMGMMxUOB6UGr0VWXGq8qCXBEtGhEIGq7wmv9vvCalZ6j1OytnbBR3un+K+9nQESJZ315Lxbht78AJbCy6oASAppAe/iXcZ3XKNC64p4DRTLa2KLYbfUqg/hfgptq58LL+DV/EJzYKl66hmubJBqx7z9/mO6BEQunGutk9lYIg/iprNjr7mmerB0XgjKl8EEcUYvUK6w3s E/3+fd52 Pm+Hr59jWL+DK7Z/s4IIzUyZmWGXwEl4VPD1XNM7trtAVa845Qfigxqb5Ff3ATvyPAFvPJGAhonMbrf8tiNaHAbJvyz/UKXTVuMjLDTZ86QWvBd6LhThrc5PdScOlV8tjnEY2bskFV1wgyEgz7grbrYAlFnwoS/0UGuIyjWkYdRSwFPfAzrc3f97Ty2drTEe3d+bjt3Uo4hcyIHQLay9MJ7h/6KvcYZ/7/CjxSj6he+HuD2TDOf3SSf90JA+4U8/Xz3xEtHwaI6K3M4QNS+UjrWwT+/beJh7MwfR/qyhp344P2sBQ2UGa7WwPZ0ZH5Z/uR70Bx4mntdT4pmfGGTE5jG2A8bNr34XnDLSNelh/Ja/l0YbSO7/xAbAHKQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.006186, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The lock/unlock helper functions for per-cpu locks accept a cpu argument. This makes them appear as if the cpu will be used as the offset off of the base per-cpu pointer. But in fact, the cpu is only used as a tracepoint argument. Change the functions so that the cpu is also used primarily for looking up the lock specific to this cpu. This means the call sites can be adjusted to not have to perform the offset prior to calling this function. Note that this follows suit with other functions in the rstat source - functions that accept a cpu argument perform the per-cpu pointer lookup within as opposed to having clients lookup in advance. Signed-off-by: JP Kobryn Reviewed-by: Shakeel Butt --- kernel/cgroup/rstat.c | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 26c75629bca2..4cb0f3ffc1db 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -115,7 +115,12 @@ static struct cgroup_rstat_ops rstat_bpf_ops = { #endif /* CONFIG_CGROUP_BPF */ /* - * Helper functions for rstat per CPU lock (cgroup_rstat_cpu_lock). + * Helper functions for rstat per-cpu locks. + * @lock: pointer to per-cpu lock variable + * @cpu: the cpu to use for getting the cpu-specific lock + * @cgrp: the associated cgroup + * @fast_path: whether this function is called while updating + * in the fast path or flushing in the NON-fast path * * This makes it easier to diagnose locking issues and contention in * production environments. The parameter @fast_path determine the @@ -123,19 +128,20 @@ static struct cgroup_rstat_ops rstat_bpf_ops = { * operations without handling high-frequency fast-path "update" events. */ static __always_inline -unsigned long _cgroup_rstat_cpu_lock(raw_spinlock_t *cpu_lock, int cpu, +unsigned long _cgroup_rstat_cpu_lock(raw_spinlock_t *lock, int cpu, struct cgroup *cgrp, const bool fast_path) { + raw_spinlock_t *cpu_lock = per_cpu_ptr(lock, cpu); unsigned long flags; bool contended; /* - * The _irqsave() is needed because cgroup_rstat_lock is - * spinlock_t which is a sleeping lock on PREEMPT_RT. Acquiring - * this lock with the _irq() suffix only disables interrupts on - * a non-PREEMPT_RT kernel. The raw_spinlock_t below disables - * interrupts on both configurations. The _irqsave() ensures - * that interrupts are always disabled and later restored. + * The _irqsave() is needed because the locks used for flushing + * are spinlock_t which is a sleeping lock on PREEMPT_RT. + * Acquiring this lock with the _irq() suffix only disables + * interrupts on a non-PREEMPT_RT kernel. The raw_spinlock_t below + * disables interrupts on both configurations. The _irqsave() + * ensures that interrupts are always disabled and later restored. */ contended = !raw_spin_trylock_irqsave(cpu_lock, flags); if (contended) { @@ -156,10 +162,12 @@ unsigned long _cgroup_rstat_cpu_lock(raw_spinlock_t *cpu_lock, int cpu, } static __always_inline -void _cgroup_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, +void _cgroup_rstat_cpu_unlock(raw_spinlock_t *lock, int cpu, struct cgroup *cgrp, unsigned long flags, const bool fast_path) { + raw_spinlock_t *cpu_lock = per_cpu_ptr(lock, cpu); + if (fast_path) trace_cgroup_rstat_cpu_unlock_fastpath(cgrp, cpu, false); else @@ -172,8 +180,6 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu, struct cgroup_rstat_ops *ops) { struct cgroup *cgrp; - - raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); unsigned long flags; /* @@ -188,7 +194,7 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu, return; cgrp = ops->cgroup_fn(rstat); - flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, true); + flags = _cgroup_rstat_cpu_lock(&cgroup_rstat_cpu_lock, cpu, cgrp, true); /* put @rstat and all ancestors on the corresponding updated lists */ while (true) { @@ -216,7 +222,7 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu, rstat = parent; } - _cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, true); + _cgroup_rstat_cpu_unlock(&cgroup_rstat_cpu_lock, cpu, cgrp, flags, true); } /** @@ -315,14 +321,13 @@ static struct cgroup_rstat *cgroup_rstat_push_children( static struct cgroup_rstat *cgroup_rstat_updated_list( struct cgroup_rstat *root, int cpu, struct cgroup_rstat_ops *ops) { - raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); struct cgroup_rstat_cpu *rstatc = rstat_cpu(root, cpu); struct cgroup_rstat *head = NULL, *parent, *child; struct cgroup *cgrp; unsigned long flags; cgrp = ops->cgroup_fn(root); - flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, false); + flags = _cgroup_rstat_cpu_lock(&cgroup_rstat_cpu_lock, cpu, cgrp, false); /* Return NULL if this subtree is not on-list */ if (!rstatc->updated_next) @@ -359,7 +364,7 @@ static struct cgroup_rstat *cgroup_rstat_updated_list( if (child != root) head = cgroup_rstat_push_children(head, child, cpu, ops); unlock_ret: - _cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, false); + _cgroup_rstat_cpu_unlock(&cgroup_rstat_cpu_lock, cpu, cgrp, flags, false); return head; } From patchwork Tue Feb 18 03:14:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 13978857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28999C021AA for ; Tue, 18 Feb 2025 03:15:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 574582800C0; Mon, 17 Feb 2025 22:15:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EA3D2800A4; Mon, 17 Feb 2025 22:15:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 216132800C0; Mon, 17 Feb 2025 22:15:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F0D112800A4 for ; Mon, 17 Feb 2025 22:15:16 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A5ED9B5404 for ; Tue, 18 Feb 2025 03:15:16 +0000 (UTC) X-FDA: 83131599432.15.B621796 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by imf13.hostedemail.com (Postfix) with ESMTP id CE1B220002 for ; Tue, 18 Feb 2025 03:15:14 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=XkvByCTU; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf13.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739848514; a=rsa-sha256; cv=none; b=P6qY899KxIImJNriAvhDpuzy/Kp020CDzYa6MVhAiUYo5+sB5rBi9bwBMT7FhM+vcxy+Zn 1XL64YLODQZJJ80vVbwmutCcsXSlQsebIjPVl18+Scw7/T+Uofgtaw9iLYeKYQ8s7Ge13O XW98QyRDq43qfN9yJvc6kf3lbp+7vFk= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=XkvByCTU; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf13.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739848514; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vXMdWzZanXBYybZlSePKoVtUVs1ExwxhVBifYgxpSeU=; b=IUhOZ2WJneG5HMNdB58mJ8zKlml4wgUtPoOoi7MwmNI4RjG2kYdd1/yhB2vrSFR8Kc7qrs MpjRjDoXPXX363M6TkHlxaxReoVfjjbTmmDr3DN4RMUhlinPB1KUflyGYyRpNP2iBbQk36 +oqZEO/k/wRxBBMZXEYcG7e2zHC7o3w= Received: by mail-pj1-f48.google.com with SMTP id 98e67ed59e1d1-2fbfc9ff0b9so7512899a91.2 for ; Mon, 17 Feb 2025 19:15:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739848514; x=1740453314; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vXMdWzZanXBYybZlSePKoVtUVs1ExwxhVBifYgxpSeU=; b=XkvByCTUtvvFQwmcTCOC89pzKAFN+QvM7kyaBxsQnGAiEMrQ+GsX+L2Bq36hcz6JUf bFLBVAPQC/Os7DmwipaRThZ+WYDUIcTtFkv7u8ISacK9GhyeXcE2MAf9L2YCYB3HVcy/ 3CNbbDota22xFGbxAs1/xnphdTbWrGWc3y/rY32o/hla6TzpA8HPagBWUUER0jVDMpfZ PYLp3HWBBxtG/cM/wInRUcKL/4wiaJIDuc0pqommQlMeo77OBYzthjBfAYFVP4ZK03dp s+nVg6/4iDiBAcmYRbi+3QuQX3cECZhOa7LhA7POPQp21yXxvGE7dbyxGAy5Xr2IaCxj 9hCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739848514; x=1740453314; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vXMdWzZanXBYybZlSePKoVtUVs1ExwxhVBifYgxpSeU=; b=DAxvoiNRWlCtXWxXHHA5dtX6KqPY8Ok+z30epTtDd4jBjpFlZ2ZvvxGARzencDW9tp 0u8t4wNeG/Hsp52oP9S1BWdBzqzm7d9hpojL1Yjgcb65ANsCjIdw7LkoMZdVOhifrLmA MBIVr46N0LFm/xKJtCW8NIJvc2Nh7zKsduvScsq2Nq6UbP+2SCRJ8vpoZYO6oZ7EZBzq sEZFlph3oGTNkZXbLvDRReXGqMbJ4Uhkm8mB7DI5eX8Gxe3QW2iTK1Vy7VFZT+B6B2he ppUR9KkjpPOL8rUe+sk6RRYuat2OXUXt0nD+MrmscIfO17h2Vs0xsb4xmfGx2f480Okk IVqg== X-Gm-Message-State: AOJu0YyWeu4jpS117MR0b+CuL0mEFhbD+rPqfsi8NN2kiDWT6BsTK95O EX3z1mod6C+EGos/X5eK/5MFxDyU1vAZ11WxTn8b6YZuD9u7N9e8 X-Gm-Gg: ASbGncs429YwtwPtoTm/j7r5eZmFBEAic8ZWMEDFnC6/oFN+9vUb6OKgosM9GSSkCrT HWs6zOUpRxoHb+MSD12WOTYzyH5vHBZna/zaZ6OHwiD8ifYaMrC1mQuMDwCJX/WmueUdWWQ6dhv RJVvl7t0oxfi5xs4PacTMmMZRap+41HOtH4CjmKzDvn8SvGF25K+ZCbP9i0D3aqrXAO3rnx0guD gXslQd45GCX4WfRFZWPEhBm2OnaEUgCmfSaBq9k5b9l2wIN8OLKdDxdLl7fux2eCpxFaEGB4/6I MSMyq1dsld3Q+ArCXLVBKiG63wwMCTXD3RAGuT0YYPAU3xfTfbxf X-Google-Smtp-Source: AGHT+IFwOfJPEGgpHYiuDd9xb8LF4nSMRJ63khUyuJwnxeuzlQfZkRMasq2TDIXK5ON9SxXG0Sczxw== X-Received: by 2002:a05:6a00:22cc:b0:730:9467:b4af with SMTP id d2e1a72fcca58-732619005bfmr17987615b3a.23.1739848513618; Mon, 17 Feb 2025 19:15:13 -0800 (PST) Received: from saturn.. (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732425466besm8763451b3a.9.2025.02.17.19.15.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Feb 2025 19:15:13 -0800 (PST) From: JP Kobryn To: shakeel.butt@linux.dev, tj@kernel.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 08/11] cgroup: rstat cpu lock indirection Date: Mon, 17 Feb 2025 19:14:45 -0800 Message-ID: <20250218031448.46951-9-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250218031448.46951-1-inwardvessel@gmail.com> References: <20250218031448.46951-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: CE1B220002 X-Rspamd-Server: rspam12 X-Stat-Signature: 83utpey9aso5gk1pbdi3fq6j8117awzr X-HE-Tag: 1739848514-84060 X-HE-Meta: U2FsdGVkX18iD84KPmZiE+ny3C6CiK503aUMDq4/DL7HOYqzVHIVIRJhj5+EDDNfw8K9nLudqcCLILkNeLU0VYRUJ5ZnLa2uK6rs9o2ATSTrOo7D+lxXRbTc6UBOuCV888xLYqEly71zALIipoGifM4P1g6P4xbV/brCPBdAB+YeJMXUNky9FxD3JdgaDRUnDOI2r2g/gO9sQB+9doeKTC0hsFVa411TGqPcjsmj2ms5k1cmiDLtrsEhUDORDJlZxSKijL/M7vGjR8nvBVIAgDUBSBTt6fW+2OGNLHfYM94cSGOpDxzNPk7ByVAJgDV58etGdGb3F07YOzztvfjAAn0xljFmH5L9Vdwx+zvR4DGVWhMBOK1IGEGBGz473JSMmYQeF64L/Ka/v4Vh6CuRVkR0ETh+bQIt2s2lnG2v6xgGhdZwTmVUI3kvu1ebKM0Yl0JNoxvdUE2KDXYR3l6eQUry8o+3GeGgd8NIEwnz3EFjnZ233Dt4DAZcgQ1/mFkNn8u05JlAYPGigtXAVGdEIKCUvvmAMITKoUnXCJLv1sGoKjzZedKTxhwt6icqhomWCgG7oUDQparnxXX9RaP+SyGaREdM7Dn3SlaGnIoTQ8b0F9cfoWTiGM6dUV95KHour+ItzyTROfs0ySWdWRck//LBi1Z3gyF6zz7CMccf7+vzH10l1YrZAJaXbfKlO1C4PQSIvre3esNVbLSKCP9M/SpB+tV8i5QdtGR0fdxTGUmLo967Ss4Omewf1lNkBjFiWa28MS7posy2tl4aUnF7ixLHru2gmmvHFM+CXXGBjx1O7MH1ivW7evKJzJLED4Dnu4P6lj4dl2vBKiaaMVNSA0a7GH/9T63uzFisFbKwT3pBkXxjuCUhQLtRdwA0yJlG5bvhuhB3N3uY+lZB5oy0Z7HohhkcMiuraGbOe7A/9JQXitasiTM0e37sR/c+HXk7HtJth3fVTxEWvitdtYU OqB2tnCV JDONrgEX68dGKqaAZShgtCqrmjHhA2E75Mv7A65FAWHC2LyyoPV1xiEjpCDl0AH17ujO0kbXhmK85+krkWYRQrGRTGWHknt7jUfol2PwRBJRfDr7sIvy/D4VoUPF7QKjH8qXsCaycs/LFgZ5TnvNnseuIfa00Jl6r983zTkVSm0yE37hlrmylZ2R+KXtcl1d6ex8k86xN5fuwEINuCsSU3OG8zhb/X5UnnV2j6My1clPj5/Fs1JhXshQkGpBgZYafe3gENx2zjhkD3YHDpAfMjCh13XCnR0FLDo/URp1PeZaNymjXMYGK2rIhDF4K9hGagMgMgIk9l5mdeoMwbeJundhcMFKYeX+bSKPucDWxJX8CCmdGl6u2pKhKMA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Where functions access the global per-cpu lock, change their signature to accept the lock instead as a paremeter. Change the code within these functions to only access the parameter. This indirection allows for future code to accept different locks, increasing extensibity. For example, a new lock could be added specifically for the bpf cgroups and it would not contend with the existing lock. Signed-off-by: JP Kobryn Reviewed-by: Shakeel Butt --- kernel/cgroup/rstat.c | 74 +++++++++++++++++++++++++------------------ 1 file changed, 43 insertions(+), 31 deletions(-) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 4cb0f3ffc1db..9f6da3ea3c8c 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -177,7 +177,7 @@ void _cgroup_rstat_cpu_unlock(raw_spinlock_t *lock, int cpu, } static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu, - struct cgroup_rstat_ops *ops) + struct cgroup_rstat_ops *ops, raw_spinlock_t *cpu_lock) { struct cgroup *cgrp; unsigned long flags; @@ -194,7 +194,7 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu, return; cgrp = ops->cgroup_fn(rstat); - flags = _cgroup_rstat_cpu_lock(&cgroup_rstat_cpu_lock, cpu, cgrp, true); + flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, true); /* put @rstat and all ancestors on the corresponding updated lists */ while (true) { @@ -222,7 +222,7 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu, rstat = parent; } - _cgroup_rstat_cpu_unlock(&cgroup_rstat_cpu_lock, cpu, cgrp, flags, true); + _cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, true); } /** @@ -236,13 +236,15 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu, */ void cgroup_rstat_updated(struct cgroup_subsys_state *css, int cpu) { - __cgroup_rstat_updated(&css->rstat, cpu, &rstat_css_ops); + __cgroup_rstat_updated(&css->rstat, cpu, &rstat_css_ops, + &cgroup_rstat_cpu_lock); } #ifdef CONFIG_CGROUP_BPF __bpf_kfunc void bpf_cgroup_rstat_updated(struct cgroup *cgroup, int cpu) { - __cgroup_rstat_updated(&(cgroup->bpf.rstat), cpu, &rstat_bpf_ops); + __cgroup_rstat_updated(&(cgroup->bpf.rstat), cpu, &rstat_bpf_ops, + &cgroup_rstat_cpu_lock); } #endif /* CONFIG_CGROUP_BPF */ @@ -319,7 +321,8 @@ static struct cgroup_rstat *cgroup_rstat_push_children( * here is the cgroup root whose updated_next can be self terminated. */ static struct cgroup_rstat *cgroup_rstat_updated_list( - struct cgroup_rstat *root, int cpu, struct cgroup_rstat_ops *ops) + struct cgroup_rstat *root, int cpu, struct cgroup_rstat_ops *ops, + raw_spinlock_t *cpu_lock) { struct cgroup_rstat_cpu *rstatc = rstat_cpu(root, cpu); struct cgroup_rstat *head = NULL, *parent, *child; @@ -327,7 +330,7 @@ static struct cgroup_rstat *cgroup_rstat_updated_list( unsigned long flags; cgrp = ops->cgroup_fn(root); - flags = _cgroup_rstat_cpu_lock(&cgroup_rstat_cpu_lock, cpu, cgrp, false); + flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, false); /* Return NULL if this subtree is not on-list */ if (!rstatc->updated_next) @@ -364,7 +367,7 @@ static struct cgroup_rstat *cgroup_rstat_updated_list( if (child != root) head = cgroup_rstat_push_children(head, child, cpu, ops); unlock_ret: - _cgroup_rstat_cpu_unlock(&cgroup_rstat_cpu_lock, cpu, cgrp, flags, false); + _cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, false); return head; } @@ -422,43 +425,46 @@ static inline void __cgroup_rstat_unlock(spinlock_t *lock, /* see cgroup_rstat_flush() */ static void cgroup_rstat_flush_locked(struct cgroup_rstat *rstat, - struct cgroup_rstat_ops *ops) - __releases(&cgroup_rstat_lock) __acquires(&cgroup_rstat_lock) + struct cgroup_rstat_ops *ops, spinlock_t *lock, + raw_spinlock_t *cpu_lock) + __releases(lock) __acquires(lock) { int cpu; - lockdep_assert_held(&cgroup_rstat_lock); + lockdep_assert_held(lock); for_each_possible_cpu(cpu) { struct cgroup_rstat *pos = cgroup_rstat_updated_list( - rstat, cpu, ops); + rstat, cpu, ops, cpu_lock); for (; pos; pos = pos->rstat_flush_next) ops->flush_fn(pos, cpu); /* play nice and yield if necessary */ - if (need_resched() || spin_needbreak(&cgroup_rstat_lock)) { + if (need_resched() || spin_needbreak(lock)) { struct cgroup *cgrp; cgrp = ops->cgroup_fn(rstat); - __cgroup_rstat_unlock(&cgroup_rstat_lock, cgrp, cpu); + __cgroup_rstat_unlock(lock, cgrp, cpu); if (!cond_resched()) cpu_relax(); - __cgroup_rstat_lock(&cgroup_rstat_lock, cgrp, cpu); + __cgroup_rstat_lock(lock, cgrp, cpu); } } } static void __cgroup_rstat_flush(struct cgroup_rstat *rstat, - struct cgroup_rstat_ops *ops) + struct cgroup_rstat_ops *ops, spinlock_t *lock, + raw_spinlock_t *cpu_lock) + __acquires(lock) __releases(lock) { struct cgroup *cgrp; might_sleep(); cgrp = ops->cgroup_fn(rstat); - __cgroup_rstat_lock(&cgroup_rstat_lock, cgrp, -1); - cgroup_rstat_flush_locked(rstat, ops); - __cgroup_rstat_unlock(&cgroup_rstat_lock, cgrp, -1); + __cgroup_rstat_lock(lock, cgrp, -1); + cgroup_rstat_flush_locked(rstat, ops, lock, cpu_lock); + __cgroup_rstat_unlock(lock, cgrp, -1); } /** @@ -476,26 +482,29 @@ static void __cgroup_rstat_flush(struct cgroup_rstat *rstat, */ void cgroup_rstat_flush(struct cgroup_subsys_state *css) { - __cgroup_rstat_flush(&css->rstat, &rstat_css_ops); + __cgroup_rstat_flush(&css->rstat, &rstat_css_ops, + &cgroup_rstat_lock, &cgroup_rstat_cpu_lock); } #ifdef CONFIG_CGROUP_BPF __bpf_kfunc void bpf_cgroup_rstat_flush(struct cgroup *cgroup) { - __cgroup_rstat_flush(&(cgroup->bpf.rstat), &rstat_bpf_ops); + __cgroup_rstat_flush(&(cgroup->bpf.rstat), &rstat_bpf_ops, + &cgroup_rstat_lock, &cgroup_rstat_cpu_lock); } #endif /* CONFIG_CGROUP_BPF */ static void __cgroup_rstat_flush_hold(struct cgroup_rstat *rstat, - struct cgroup_rstat_ops *ops) - __acquires(&cgroup_rstat_lock) + struct cgroup_rstat_ops *ops, spinlock_t *lock, + raw_spinlock_t *cpu_lock) + __acquires(lock) { struct cgroup *cgrp; might_sleep(); cgrp = ops->cgroup_fn(rstat); - __cgroup_rstat_lock(&cgroup_rstat_lock, cgrp, -1); - cgroup_rstat_flush_locked(rstat, ops); + __cgroup_rstat_lock(lock, cgrp, -1); + cgroup_rstat_flush_locked(rstat, ops, lock, cpu_lock); } /** @@ -509,7 +518,8 @@ static void __cgroup_rstat_flush_hold(struct cgroup_rstat *rstat, */ void cgroup_rstat_flush_hold(struct cgroup_subsys_state *css) { - __cgroup_rstat_flush_hold(&css->rstat, &rstat_css_ops); + __cgroup_rstat_flush_hold(&css->rstat, &rstat_css_ops, + &cgroup_rstat_lock, &cgroup_rstat_cpu_lock); } /** @@ -517,13 +527,13 @@ void cgroup_rstat_flush_hold(struct cgroup_subsys_state *css) * @rstat: rstat node used to find associated cgroup used by tracepoint */ static void __cgroup_rstat_flush_release(struct cgroup_rstat *rstat, - struct cgroup_rstat_ops *ops) - __releases(&cgroup_rstat_lock) + struct cgroup_rstat_ops *ops, spinlock_t *lock) + __releases(lock) { struct cgroup *cgrp; cgrp = ops->cgroup_fn(rstat); - __cgroup_rstat_unlock(&cgroup_rstat_lock, cgrp, -1); + __cgroup_rstat_unlock(lock, cgrp, -1); } /** @@ -532,7 +542,8 @@ static void __cgroup_rstat_flush_release(struct cgroup_rstat *rstat, */ void cgroup_rstat_flush_release(struct cgroup_subsys_state *css) { - __cgroup_rstat_flush_release(&css->rstat, &rstat_css_ops); + __cgroup_rstat_flush_release(&css->rstat, &rstat_css_ops, + &cgroup_rstat_lock); } static void __cgroup_rstat_init(struct cgroup_rstat *rstat) @@ -605,7 +616,8 @@ int bpf_cgroup_rstat_init(struct cgroup_bpf *bpf) void bpf_cgroup_rstat_exit(struct cgroup_bpf *bpf) { - __cgroup_rstat_flush(&bpf->rstat, &rstat_bpf_ops); + __cgroup_rstat_flush(&bpf->rstat, &rstat_bpf_ops, + &cgroup_rstat_lock, &cgroup_rstat_cpu_lock); __cgroup_rstat_exit(&bpf->rstat); } #endif /* CONFIG_CGROUP_BPF */ From patchwork Tue Feb 18 03:14:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 13978858 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21EA9C021AA for ; Tue, 18 Feb 2025 03:15:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8F62F2800C1; Mon, 17 Feb 2025 22:15:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A5532800A4; Mon, 17 Feb 2025 22:15:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6FAAC2800C1; Mon, 17 Feb 2025 22:15:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 499CB2800A4 for ; Mon, 17 Feb 2025 22:15:19 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0E34C817A1 for ; Tue, 18 Feb 2025 03:15:19 +0000 (UTC) X-FDA: 83131599558.21.EC38BAB Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by imf19.hostedemail.com (Postfix) with ESMTP id 34DB51A000F for ; Tue, 18 Feb 2025 03:15:17 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=N7Mym30q; spf=pass (imf19.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739848517; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CD1jatsWdmIEiP5eGI+IqcgznEGBHRgmN/6oH6WnB14=; b=sXrJ0m6x3YVigGUixSiN9fVsUIMDqZ7JcVsTh8kryNbYiOLIVeFOT215HJ1WAaRWqAPewa CAEM8es89+5AL0OMscKQCsvppb2TDxp+19v6n/vHS3MuyWeoEPiNlLIJRDyoM2lHWybS2q wAeYCwQZnnY1BSH2uKPclyQUg1/4c7I= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=N7Mym30q; spf=pass (imf19.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739848517; a=rsa-sha256; cv=none; b=iDU072WLpYXsT+p3zLs3mSZW1Yoe8OaQUamH34iE/P2IVOgZGw/q1DTtW5ILA75E7huFwy jpxiKVODGKfiQDtOAd8r1yYOEj9To2RXmJ0aW+uGo1iQN4YNoIxLwOEXYjzEoQRoJGyTjy ur2ybd4yT4EkOWSR+OnznzqM8C9qxcg= Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2fbffe0254fso9257164a91.3 for ; Mon, 17 Feb 2025 19:15:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739848516; x=1740453316; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CD1jatsWdmIEiP5eGI+IqcgznEGBHRgmN/6oH6WnB14=; b=N7Mym30q8NpOYKQ/ExU4FHnsLLIPIG2bWgEojls1e0LFSXpTSQII2238GYt5oTyrVz jXttfNbLcdG7Mtnigc0NBdXvtoy4jWI3qORjTiZoJI1A5mcp5+JrGpjQ0bbxJCwH0Cxz brPlz+p9yTCSyJrTHIaI3Dve3NUBaE7TNW9Ti7pVcHuOrdVU0YGZ8OqjdqrBK4X/Jc6R SP0QtdoxakAfzEQL2vZzhAuffjF8gZ6aMYcwlu9wZawJX57bRMr92AQic2MOYG39RCPH jHcA4DHqOqkhmrpqv/d9dpG5PJXPXz1lXgBb1f53LbzCy1NvPbQE5yDs0K4kLuzXQirz FV/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739848516; x=1740453316; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CD1jatsWdmIEiP5eGI+IqcgznEGBHRgmN/6oH6WnB14=; b=LZh0/ROW6i5CXqJhsvwT6+xZaDaaKsDe5xNcHbxnnnmyzE1ceReZbcNAF/SIDAFlOX UvkBhNpVTp0c26vcX+dxGBo+1BbkkfPCRjaIxXEWnbGkjKw5zEodxMpdy4xYBDE67Spl eHoYxNnVd8Tqo33Og2bT+ywm50zA/IQtBFQLkCdvyhE8q3k7PFv/uWEvpzLpgceLZszT pUew6F8XMLGiBA2KTPOIJmlkhfbFjBoh2qgEhlUoMSLl1ZguT5DlKhM4VrGagsDymeBa n4Zb8X2Fd6j+R1htRUXITg5RXfwtMYWqvHVnPREFFaqciZKbq12lRdw1w9MUPsSXGoEm BqSw== X-Gm-Message-State: AOJu0YzuXzuSm41JghfxyR6vGELo9VoNDkwGT2HAYWmCe1FhcY6P3acz mjv3g+ekO0Aq2fQeV3kQrjaR/NTvpuMGJGWE7ZyokmvjPdSwFIbKifkiKQ== X-Gm-Gg: ASbGncsvmdmH/8UdKKnboPmNCJygDeauBIcvZiIsj10bPAOOt3SyGLKw7PGfqV4UU2g Qeb8Ii/6SUcIc5D3JzHJWfJFA8u+miUFUYDhINP4tzqa4ZX659XYpSqmqRFl0C15gP1o19G/wWA R3umyTdrC192SHd/sm1kd1cXn7tO/cYS6xKSsSbkQEcrFzSwtgFJOwsygxMqpoM7jjvHqqrjvAY A4ksviHcTLnBB0fzYvIY6FmasTkVbXJGkMrmucrmYbkYgDPpORMrKkGqcUUS5Yg142begCkp8BR o2BP7lOqcKgwDl92VqGUqg8hYA1iGYzrnf4s4BxDVso5AIBJxu1/ X-Google-Smtp-Source: AGHT+IEZ40Ep2/y9fUsKsPY3ph6FBvkIsGW6s0OHsKOFL28M9YkkbAUChBqTJ0u+bam9kXnHkApFwA== X-Received: by 2002:a05:6a00:124b:b0:732:1bad:e245 with SMTP id d2e1a72fcca58-7326179e7d8mr16437947b3a.7.1739848516068; Mon, 17 Feb 2025 19:15:16 -0800 (PST) Received: from saturn.. (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732425466besm8763451b3a.9.2025.02.17.19.15.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Feb 2025 19:15:14 -0800 (PST) From: JP Kobryn To: shakeel.butt@linux.dev, tj@kernel.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 09/11] cgroup: separate rstat locks for bpf cgroups Date: Mon, 17 Feb 2025 19:14:46 -0800 Message-ID: <20250218031448.46951-10-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250218031448.46951-1-inwardvessel@gmail.com> References: <20250218031448.46951-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 34DB51A000F X-Stat-Signature: zji1intd3ftikzbzydjg5zcg47umrigw X-HE-Tag: 1739848517-743642 X-HE-Meta: U2FsdGVkX1/mG24W8WubGYhyIOD37ShOIqCKjHjG0B2TNRjYhjdJVHBLg/ZHkaPs1KqSKbOXRVMVqwS2/L1nkKfIaZbhzUnVeRNxNe8K20ShPT52PHSzXJGTDBIirEa9pss/SA0R9uu4yEBoa/M4Mwh5uiQk8nDm/M8uiJ/rEGGiEg+hEXcUjMiaea/jXzbGDaN8NkENFlOH3Xvrxk037W0ReJwyzjLLhzIb1vpN1/CnlZF2IRez02Yuk6llx4Gfa1Oy5zbJ+WXuWqnBifXXgLF93U9S78KhffLnGsJpPqtcI0W+T186YBT2riE7cfaSG1ubFUHwB9KhbhfXTqvB4Jy1v6mBF+znQY3+3OYhBAhebpFiOy1Ja3dsOIXv1+6+JtuDxuFpfjbnJkSo+FOiNB9gzz/s0hj6589NyezbwKRe7LOxSg/4pQ6bJwuMu1dbqnQ3VzjSpHq7j6nibtmtvuHSyk42BqfCFnkjnrQl3aYLUbJx+0G9de++8JzXErDnASxOElD2xXbltyfTkbtQ7tILWUxN28pOBFTzjzJm6tVVt0ZGdB9WaQhf139O9GfghD1bicYALaiUImpPE3urBXTWEDjWs130NGfhTo5j2AzzIsNoSjbSY5OzaZ2lZc5YrD26mls/HcBOfKjSv8oZBSvzJj9tck6X8DX8o5bIeP+h/kommKjJ3MOJT0Zi+PWp175WrSrGItcRgfnQghX+RdEYt4A52mCQy6atNUpeOQkzcm8j/kSTImuRetY/MvfQp9DJxBZzYU5fCAWLoKXMXe9qwpIg8vz2CgbGQLQX1xTLsGcZ7Bm+fvHwudxX/xhW5OltUWTbPJQNJmpdZzJiMKGsdzR+yWvHWmsXiCsvpm4RE7Dcg+WacJV/509Mf3MFhOK3bqYUpQdra330zdBcQnv4jQ7lEkljHl0h/c7dU33YZb6zosw028roYD3XZv32SIZcVcq41IKMZ7XXBGV SdIregk0 Tt4dU6FriEqwt5WF3K9rdLfQEtCX3zb889a6xx3tDTplLNEPOMd69UhgA36kNGg0trj0O7szKaJxrmYz06yvup0ymF1e0qd7Lc8UMD7/8+2ooItbTnnQtdvsm4RsOoGQBIP6lsKxP1doE+csY9+6iSq0JFXp6Z0DNFz77Kx34CdAQiJ75wpwiYeCPb8ZWwysU/Mg+PjvFnbM7+KUynz/7IDCIm85T9dHfVZgJBGn+ZmdsvdElTH0n3fR+42mNnYnguYDR1dqu+01Wdvb6KqSv8piiwH7zjpvQtU/0G6ujHb3fgJcL9N7s1g292yjSU9nZ6HuZguEZCf69sJweCc93fIReaSsSgWZdcnbF/Syf2sFB/4DaP2RE9Dsdgg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use new locks with the rstat entities specific to bpf cgroups. Having these locks avoids contention with subsystems such as memory or io while updating/flushing bpf cgroup stats. Signed-off-by: JP Kobryn --- kernel/cgroup/rstat.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 9f6da3ea3c8c..7d9abfd644ca 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -18,6 +18,11 @@ struct cgroup_rstat_ops { static DEFINE_SPINLOCK(cgroup_rstat_lock); static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_cpu_lock); +#ifdef CONFIG_CGROUP_BPF +static DEFINE_SPINLOCK(cgroup_rstat_bpf_lock); +static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_bpf_cpu_lock); +#endif /* CONFIG_CGROUP_BPF */ + static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu); static struct cgroup_rstat_cpu *rstat_cpu(struct cgroup_rstat *rstat, int cpu) @@ -244,7 +249,7 @@ void cgroup_rstat_updated(struct cgroup_subsys_state *css, int cpu) __bpf_kfunc void bpf_cgroup_rstat_updated(struct cgroup *cgroup, int cpu) { __cgroup_rstat_updated(&(cgroup->bpf.rstat), cpu, &rstat_bpf_ops, - &cgroup_rstat_cpu_lock); + &cgroup_rstat_bpf_cpu_lock); } #endif /* CONFIG_CGROUP_BPF */ @@ -490,7 +495,7 @@ void cgroup_rstat_flush(struct cgroup_subsys_state *css) __bpf_kfunc void bpf_cgroup_rstat_flush(struct cgroup *cgroup) { __cgroup_rstat_flush(&(cgroup->bpf.rstat), &rstat_bpf_ops, - &cgroup_rstat_lock, &cgroup_rstat_cpu_lock); + &cgroup_rstat_bpf_lock, &cgroup_rstat_bpf_cpu_lock); } #endif /* CONFIG_CGROUP_BPF */ @@ -617,7 +622,7 @@ int bpf_cgroup_rstat_init(struct cgroup_bpf *bpf) void bpf_cgroup_rstat_exit(struct cgroup_bpf *bpf) { __cgroup_rstat_flush(&bpf->rstat, &rstat_bpf_ops, - &cgroup_rstat_lock, &cgroup_rstat_cpu_lock); + &cgroup_rstat_bpf_lock, &cgroup_rstat_bpf_cpu_lock); __cgroup_rstat_exit(&bpf->rstat); } #endif /* CONFIG_CGROUP_BPF */ @@ -626,8 +631,13 @@ void __init cgroup_rstat_boot(void) { int cpu; - for_each_possible_cpu(cpu) + for_each_possible_cpu(cpu) { raw_spin_lock_init(per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)); + +#ifdef CONFIG_CGROUP_BPF + raw_spin_lock_init(per_cpu_ptr(&cgroup_rstat_bpf_cpu_lock, cpu)); +#endif /* CONFIG_CGROUP_BPF */ + } } /* From patchwork Tue Feb 18 03:14:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 13978859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEB80C021A9 for ; Tue, 18 Feb 2025 03:15:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01CF52800C2; Mon, 17 Feb 2025 22:15:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F0FF02800A4; Mon, 17 Feb 2025 22:15:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D885D2800C2; Mon, 17 Feb 2025 22:15:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B67322800A4 for ; Mon, 17 Feb 2025 22:15:20 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7611CC1819 for ; Tue, 18 Feb 2025 03:15:20 +0000 (UTC) X-FDA: 83131599600.14.8115F49 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf22.hostedemail.com (Postfix) with ESMTP id A1410C0002 for ; Tue, 18 Feb 2025 03:15:18 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JdCBwfNG; spf=pass (imf22.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739848518; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vg+1tuAH5Cml38p4hzH7HNleCRescAO8xBmjt4Ynbfc=; b=ab84Yht9d3vLNOVbu9J4pz3dm+U2HqYDc8gib4dWom9//BOWHpk3tM15tZM/RCzZLTVmlX +p3U8gxLjfIClHLWh/pp7uBrJFGhLnVrUhkaDf6nYUNcjzf1VyHDvHm5zDVmMpP6g5IljW k8+Jm4Sb1svBDMLhyLUhUCfK1SFgclQ= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JdCBwfNG; spf=pass (imf22.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739848518; a=rsa-sha256; cv=none; b=6sl0SOwrQHOAY6/SBvjR28S84l348CZXVaCb9RqXLK6eDaiXTgumCPwC9Lwz6x7wTgWTo3 CLEVMpHtDQtbJ9mNnWViSjpv+rnDDSEGZxidXMv4hh/xqmippvKHJ//9CfZ1atnz0R0B4P mYgn6ENHk1P/T0LYyN5iraeY+THkUqw= Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-220c8cf98bbso99969755ad.1 for ; Mon, 17 Feb 2025 19:15:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739848517; x=1740453317; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vg+1tuAH5Cml38p4hzH7HNleCRescAO8xBmjt4Ynbfc=; b=JdCBwfNGsJ9FY/I9be92wLkmNWJfdv3S0PPTDyHRvv1D9KEXngxCUNtyVwcHVp1bQK FrtbOZNnPabnzNTuVdQUjOTuPSGRfYLKFwjbfHNDwtHgvid+AoATf+siXqWaoU+/KS4K DM2ZPTmRSxybgGnbflwY0FyM1zWbJrXx5u8FOwP0iZx7Rt4NILf05NhcWs9jB8Uo4DzI 58dfnQUzvLFNMPTWCgmVzaw+uErjvV6dMrVL4z2r9pYKg/h8EQD6PeNRAyZOVXNS84S1 HFc4lD+4PxEy5zTg3RXlbD/TsPHIQAi+1X+hQaMSBm9AKH4SATpxfYBMFYQ7BNlFTaID B6+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739848517; x=1740453317; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vg+1tuAH5Cml38p4hzH7HNleCRescAO8xBmjt4Ynbfc=; b=Id95Iq/SCkJDyk6UggAyCmgyAz/XpIB1ak3mXXXjEH62YB+kt7IdO+64Fkih9Stjqr PDsiqFHmqx0DC1wfdhXwYk1Q51k/xTtPQ7llPu+PKKqgayIy8/agXQlheVtq3OyTrDCk 9GDXjy2UC4dBRF24WETFmq1fRE9JsiJSWlu8L2EFq0XGduWSn2WikmS2HCbjBSBcHU8P /dnnLwQQ1SXUdPG5YismzQ5ShMz8sj0xWMNdOGelYhA02Sb4SM1b5TbcCJ0h9Vqoooo4 UP1Fq5WHiX82C3i6LbcCpECzBmNzxoW+S9hKoReH4ga41TC2p99aslqbvGH09cKIEwmv BSvQ== X-Gm-Message-State: AOJu0YxzHgb5RSSpoGA5n8VM3TU59B2u5yNFNjhxhBFzFZLfnCLI8Gy7 oFrLN/4MTAxVRSk73+gazxCG+YtMgqK5BJataHPs15TjVimw9vQR X-Gm-Gg: ASbGnctmSYoCA4Gz0+sp4950qq/wcAi8RKNE2CgMB3aWYV99nIQCoHENcHXHfzDP+cg W+79Sth6+wfckdita7JjmEjM6DFUh/wHd2E5cCyBkdmT73pdBWJyWzdiEoF7F3YrRDTdjelFUXl Q5mPApnMSI24sJC7VU+OvMnW1okkf7H2Aj1oppnaFTVxUrwOriwm1B1vGtpRs4kmmks5q120ZOd TXx5CtP+7hsuZPY8lUMgskMYvZPOzIqjUipX7A8kJqRW04apMwba5Ev6xL/E5rgzirdwelq/oXf 7O9yU+F6dBpTxV9Ls3zYQnWKZLfua4ObgfjVAvJBceVd49hqgISf X-Google-Smtp-Source: AGHT+IGi97sy9lc8oEV2Lvmu4nX/Oz5UjlOTjclhhjOUsPwyOtWg+KBN0Ea6IXWS4MZ0ObmmEo85Dg== X-Received: by 2002:a05:6a20:244a:b0:1ed:a6d8:3439 with SMTP id adf61e73a8af0-1ee8cb7dfd1mr22632952637.22.1739848517528; Mon, 17 Feb 2025 19:15:17 -0800 (PST) Received: from saturn.. (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732425466besm8763451b3a.9.2025.02.17.19.15.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Feb 2025 19:15:17 -0800 (PST) From: JP Kobryn To: shakeel.butt@linux.dev, tj@kernel.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 10/11] cgroup: separate rstat locks for subsystems Date: Mon, 17 Feb 2025 19:14:47 -0800 Message-ID: <20250218031448.46951-11-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250218031448.46951-1-inwardvessel@gmail.com> References: <20250218031448.46951-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A1410C0002 X-Stat-Signature: huk9q9m3m1xcrj3wqwhmr8baf4obr656 X-Rspam-User: X-HE-Tag: 1739848518-534619 X-HE-Meta: U2FsdGVkX1//zBZWGEr6xWuiyhJbAlbm/J7bA4YCTSV2TthKonEwcyE+LP7iDKsKFYFtt2Fm7FufMMp+ocOhoQvYZRYfv6k7eEJ47i7TxYLzpHB3YvaK+xOVmP6i2tK7uDJrgXnHHWk6GAfReCqJA6k/DY41GrtaywGmJx/mxSgP9GpMnNIf357w0eFUmUtm9fkUXOpg1VatumuC1y+b6KMYfjDBaub3bqT8vcu3bn/DVJ+fjsXx5JcESX7wtVkZ2qlrBUrz+ufWeNWmo0o52wCvE/rKjrSc+XrrJRZoekz3mc3xw1WQS9CpUAJYgtvw+r7FyayRXF0CHEC4LMgQKFElSqPgmg99DqpkGUqcdZtZI7bvc1LxgZeDwTGPpBRfokAB+gTd4D61QiqBwJTg8iYWcdxSTv8Qa6Atg8A0lp5fOjivCcqYEH2zlLTlSUpnBip0rZQX3BAc1Pq+0tCj/net3TDJ/b7D7UsfzP4dCOMK/NNc0ybAUiS0JZ6TuxQTIxRekJtNwTbzR+05TYahBcS6Xfz2byItFCUvTilPQTAVBM3icQCxeSXzV03FMBv3RX+3tUI2Tc5QYeImEpUOTVw0G8zyqRz+BPcW3DIeqsyqhNpX1tbQwNbf6BKTyVtAyVI9rhahVy4vqAGeZyjSO2JMrQeyVT+l/8NCfzGe/by1PpZ3Jk1Q9mWQy5LbMyIrfaSJpZ9ZGvSKd8IfQJ2c8IYCCb3sNB5V8FcfsHVpuSCNRD+RIrd/MwrvaV9uHwCPipvDvcQMx0sAA5qiM7Dbkm6OQnP2iTGXS0sJnnHl0ELcRgeVE0hIW938CS/cXfMXyavA/ucWqOOY2QVCjT+6E/N8eRJZ+fLUUJpm1FirpKcIw7tvaz/JzUEQzBBQnn7mSOIFDkbvIQ9lEkZYH2tiBk1rGPIYO3ByR52Mxf/AA6GttlCnAiuhYxGBBbZ5m7VY9pF1OFYjVQlxIi99kvq GFXHzLDg mmNe6yoyF8xTUvsXla/WmsOVo6bwrzGe/QsiXkI6BDq8UqzEmtaqPyYpk/o2CIFbjzImtkAVxze5nVd7HJhR1NQDt4wnKLmm7DTrM3KztyiHbqtXdNWceBltoSh3hZ/id4lbm5g5V+m7/dKS6wr7OI/Fxz6xbf2iecHkPPcgz2VDw19dQqqbM+eKCaiEwN+WmN3mI9HRapEWfiuhRoQSpS69IvKvWjcSQnQCwBD+iDP108DXaftv7bjQMB/MGmrcVODKFfmQz2mYdbAZdwzE3SnOupSBoiJ8ieJ0Wg8OboPTz3vKxJFez3xBkwQzyVrvPzJfUI9rOVbFHdLqRGSMteosUSAK/euSBFwdf+F9To+gig8poG/rr7r3Y6g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add new rstat locks for each subsystem. When handling cgroup subsystem states, distinguish between states associated with formal subsystems (memory, io, etc) and the base stats subsystem state (represented by cgroup::self). This change is made to prevent contention when updating/flushing stats. Signed-off-by: JP Kobryn --- kernel/cgroup/rstat.c | 68 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 58 insertions(+), 10 deletions(-) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 7d9abfd644ca..93b97bddec9c 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -15,8 +15,11 @@ struct cgroup_rstat_ops { void (*flush_fn)(struct cgroup_rstat *, int); }; -static DEFINE_SPINLOCK(cgroup_rstat_lock); -static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_cpu_lock); +static DEFINE_SPINLOCK(cgroup_rstat_base_lock); +static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_base_cpu_lock); + +static spinlock_t cgroup_rstat_subsys_lock[CGROUP_SUBSYS_COUNT]; +static raw_spinlock_t __percpu cgroup_rstat_subsys_cpu_lock[CGROUP_SUBSYS_COUNT]; #ifdef CONFIG_CGROUP_BPF static DEFINE_SPINLOCK(cgroup_rstat_bpf_lock); @@ -241,8 +244,14 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu, */ void cgroup_rstat_updated(struct cgroup_subsys_state *css, int cpu) { - __cgroup_rstat_updated(&css->rstat, cpu, &rstat_css_ops, - &cgroup_rstat_cpu_lock); + raw_spinlock_t *cpu_lock; + + if (is_base_css(css)) + cpu_lock = &cgroup_rstat_base_cpu_lock; + else + cpu_lock = &cgroup_rstat_subsys_cpu_lock[css->ss->id]; + + __cgroup_rstat_updated(&css->rstat, cpu, &rstat_css_ops, cpu_lock); } #ifdef CONFIG_CGROUP_BPF @@ -487,8 +496,19 @@ static void __cgroup_rstat_flush(struct cgroup_rstat *rstat, */ void cgroup_rstat_flush(struct cgroup_subsys_state *css) { + spinlock_t *lock; + raw_spinlock_t *cpu_lock; + + if (is_base_css(css)) { + lock = &cgroup_rstat_base_lock; + cpu_lock = &cgroup_rstat_base_cpu_lock; + } else { + lock = &cgroup_rstat_subsys_lock[css->ss->id]; + cpu_lock = &cgroup_rstat_subsys_cpu_lock[css->ss->id]; + } + __cgroup_rstat_flush(&css->rstat, &rstat_css_ops, - &cgroup_rstat_lock, &cgroup_rstat_cpu_lock); + lock, cpu_lock); } #ifdef CONFIG_CGROUP_BPF @@ -523,8 +543,19 @@ static void __cgroup_rstat_flush_hold(struct cgroup_rstat *rstat, */ void cgroup_rstat_flush_hold(struct cgroup_subsys_state *css) { + spinlock_t *lock; + raw_spinlock_t *cpu_lock; + + if (is_base_css(css)) { + lock = &cgroup_rstat_base_lock; + cpu_lock = &cgroup_rstat_base_cpu_lock; + } else { + lock = &cgroup_rstat_subsys_lock[css->ss->id]; + cpu_lock = &cgroup_rstat_subsys_cpu_lock[css->ss->id]; + } + __cgroup_rstat_flush_hold(&css->rstat, &rstat_css_ops, - &cgroup_rstat_lock, &cgroup_rstat_cpu_lock); + lock, cpu_lock); } /** @@ -547,8 +578,14 @@ static void __cgroup_rstat_flush_release(struct cgroup_rstat *rstat, */ void cgroup_rstat_flush_release(struct cgroup_subsys_state *css) { - __cgroup_rstat_flush_release(&css->rstat, &rstat_css_ops, - &cgroup_rstat_lock); + spinlock_t *lock; + + if (is_base_css(css)) + lock = &cgroup_rstat_base_lock; + else + lock = &cgroup_rstat_subsys_lock[css->ss->id]; + + __cgroup_rstat_flush_release(&css->rstat, &rstat_css_ops, lock); } static void __cgroup_rstat_init(struct cgroup_rstat *rstat) @@ -629,10 +666,21 @@ void bpf_cgroup_rstat_exit(struct cgroup_bpf *bpf) void __init cgroup_rstat_boot(void) { - int cpu; + struct cgroup_subsys *ss; + int cpu, ssid; + + for_each_subsys(ss, ssid) { + spin_lock_init(&cgroup_rstat_subsys_lock[ssid]); + + for_each_possible_cpu(cpu) { + raw_spinlock_t *cpu_lock = + per_cpu_ptr(&cgroup_rstat_subsys_cpu_lock[ssid], cpu); + raw_spin_lock_init(cpu_lock); + } + } for_each_possible_cpu(cpu) { - raw_spin_lock_init(per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)); + raw_spin_lock_init(per_cpu_ptr(&cgroup_rstat_base_cpu_lock, cpu)); #ifdef CONFIG_CGROUP_BPF raw_spin_lock_init(per_cpu_ptr(&cgroup_rstat_bpf_cpu_lock, cpu)); From patchwork Tue Feb 18 03:14:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 13978860 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D12CAC021AA for ; Tue, 18 Feb 2025 03:15:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 884702800C3; Mon, 17 Feb 2025 22:15:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 80C342800A4; Mon, 17 Feb 2025 22:15:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 685212800C3; Mon, 17 Feb 2025 22:15:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 471F82800A4 for ; Mon, 17 Feb 2025 22:15:22 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 05A4350025 for ; Tue, 18 Feb 2025 03:15:22 +0000 (UTC) X-FDA: 83131599684.15.A96B1AD Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf23.hostedemail.com (Postfix) with ESMTP id 3CE6A140004 for ; Tue, 18 Feb 2025 03:15:20 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NnAk8Qld; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739848520; a=rsa-sha256; cv=none; b=3+VMy0iB0KVHS2bU3bYXIyPA2wmonkFUeWkdsTTTj2cAk3+RIaIpKnDyPHGYQKK2Cls8nV QCY2dvQRFFIuH1njCrxX1xEDb+XVdz6wGqhnw74Kg4Tou5zk8aDBgHojRYKvM6MeR1fOaa SHw8SmlWeKE+0yICMkQc9hHGT/CpbO4= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NnAk8Qld; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739848520; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LAAYrMp38Dbf1h36ApVAoOaASM18TrELMrU+bZs87zk=; b=8piIoLL23L1Vc8BvJCmXWc4ZKi2Gb4l6nghrwIguBRQWvHDnfZ2PeOEtDdUSgy+CZaaVfy 24DReVb9wXireZuUrgaH7SLTaw71QanTygCxTuZMSunNS3xdnpIMF7+ODUu1ldXoaTPmd9 MT0MseFBnkGHS+rlNTHxQtqI5PcxLs4= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-220d39a5627so71564965ad.1 for ; Mon, 17 Feb 2025 19:15:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739848519; x=1740453319; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LAAYrMp38Dbf1h36ApVAoOaASM18TrELMrU+bZs87zk=; b=NnAk8QldNVsD3gsrQdMrBIzHOigu6KQLWbX5NhfK6GjoJyPBGbH/sByIvb0ocjEnFb DylbRSv3ds8Iuim/d45RbaAC4CmmoQ8fPuEm4+AiY/uEZh+uzXfUIRUXTtnfziHUQTYt Y8Y4NxoJT1qhDwZw0vO7vHUngnExKXaO8/6OJiv/ny1voqrnjHCC23QowehzZarvrr8/ BhbcbuIBc5/84qGIRtFImkedK4vJa5qpevCQQq1l+PdHAycTFdMOpfnHZCybWZIO1lXe yNDBni2+MVz1b/8nTE9sL3/hvTJy6hjTK+JyQWdOKPtrj9HTbN6DLCYeao5wFNHynQTF baVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739848519; x=1740453319; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LAAYrMp38Dbf1h36ApVAoOaASM18TrELMrU+bZs87zk=; b=PNGOE4S0g8DVF5drXIvByMmreTu7+x5RUjJZVXCqUa3lBj6eknL6pIdnnQdh+9micw Xkv6CcVjQijiiWEtAovpGq79XGMziZbL2BsymZMeyHiH8hF9H/2leUDiuSvJ2neEYaqP 64BVz2y/DKgA5dhYZ+opl2GEH1ReuOxpFKAhLGcYI9qDu7/nEHwrjhV5YZEixOkNy8kQ LdOPzdebha67AcsEK/uwmV6p4RFN8d3vdCTx9/+nf+Ae1kOo9gAsZchAQ6O7x0u+nhL4 6vCv3HXfhdVb6UEB7LoWahzphgOGBQ/WNNvDKAcg63BEiz6SdumuXMuQlgkxJx//T0XN gtfA== X-Gm-Message-State: AOJu0YziNm3ITIu0YsE8ZHo9e3CKmpQhuYGbEA7+9szApbCa9rrqzF8y s8BY0+JmdJMyEFG6cUrQj9i9pO5gW6A31NCOgAw6ZCo7prbB3Rgg X-Gm-Gg: ASbGnct659x122ooRLbqO9GSWFfV9P3BIZlicVv8ekHVlmjKeWqzkhQ35uFDr7c0dUf 741XTKzvqEpXEZVr4B9b623zh9AUlo1JHdQ/YnM1KfbYT826nzDwRrHOa7gShjveR5G+eMbU2dx nIpggZE4almjrMkRV0ga6hL4EfY/Mk6G0u5hTH8cXPflNVEO0ujllGQugYC0OvhCzADaw9g4wLL FieZcRBNvLXPpestdVt7aBc/tgAJ1OCjZupJkY9/Da2oS/0pW6PixaAVbn87aGAydAUqbH3qbQt 4WV4bHctV7ByG+vs2SWhcgtXRsL/w3mFuSYSsjT09LavoyjgbS95 X-Google-Smtp-Source: AGHT+IF+gu3zHN9lsAlpDAPTExtIPOeyPRNrR7IkVwslhKU3Ky0hB0noFRxqnw8wTCYxNUTtfjP8iQ== X-Received: by 2002:a05:6a21:6da7:b0:1ee:5d05:a18f with SMTP id adf61e73a8af0-1ee8cc25a7cmr22443182637.35.1739848519160; Mon, 17 Feb 2025 19:15:19 -0800 (PST) Received: from saturn.. (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732425466besm8763451b3a.9.2025.02.17.19.15.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Feb 2025 19:15:18 -0800 (PST) From: JP Kobryn To: shakeel.butt@linux.dev, tj@kernel.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 11/11] cgroup: separate rstat list pointers from base stats Date: Mon, 17 Feb 2025 19:14:48 -0800 Message-ID: <20250218031448.46951-12-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250218031448.46951-1-inwardvessel@gmail.com> References: <20250218031448.46951-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 3CE6A140004 X-Stat-Signature: z45ch4f6krmeaz4u6xm46bq5dpkok8cc X-Rspam-User: X-HE-Tag: 1739848520-447398 X-HE-Meta: U2FsdGVkX19nlzeZ2LFYDCUEI7ZLClD9sxSJArbstud9gA2GAD4FhQjpSK3QJYCu9IPN+MIfgY9jc85esIoq829pvkNQ6kdLp/JBUcBOjQDTD/OhVJOrZLwCDJJ87KTOppu80xQZ2fBgRSH80xj/qfmQYh0ACGs03ZXcSM2DuY7npYeJI5u3IymRnftr58I7ovHh4m4hpwQmYtJMOgKVWhc1ER8LmtBRLdG9+kPfwLSZj20gTWSJSiG4o9gZUbc/TAq2DPlvPYaA4L5Cmt+durAtgC3uhnPwWKv3fUotqquNyNrIo8sQKPM6Vjpaju8XaN73zNGnfNWbdAi/mCDy2TaTJ+xU3yduWeidrlQ4sRggZfhWAHkAl7h8ekknLbF4HnH5S6f7y4IfaGVHse4mXvJbdvuEkogbGiQ30FJ62Yvg6ZCrm6I8Fn3NLTubBLlifKultamESoxjuXJRgzVIV8bMhEnnF9u1zF+AN1c/gAkBf7V7Vr8tPp6dQi1jcFqJDKCF1TEUNCWOJ3Il3dpsAxbSwpe8JKnwM0McOO7mplOuh/9aLWQUB2RG5a4mOfO6NmqJ1cJgQZevC9IhHgcKbe68nk+49vZHKUISQVK1PP+Z658vWz7aMDPrPciqUkAELV58oeZWenKNqfAxY+AqB77g9IVt1go6pQ8AdTZgK1U0dNfdEQYhiy1Lm4zwV50JAkNTj90CR8JD5uBKpeJHlgRVchce/KAn81+ZI18nAcnQ5GzpmNltbCjfJX8QOHW5qVvPaLirOtjYXmq9GEoy0wbtfoOPdoAK3eZdbPdt99/Aa4GnX6IYbRCJLBBkB7vR/i6gbFsWo8qyMGYIKyFFnnBkc0azIsPo/2LqyAAsN3OeU92P+bS2kmDVG29r997OmAcWB0wLMRxTbC0GSxN3N9tB8m7voW3tWjWMmCEoD2Cq7w64Wd2U6jgE06Co2Zh92hFlNGKXVVRThDLBkX9 svojQk+S 0/do5Milo6Gs3MosrXgfVWS4w0Bbut/6nx1j5Cxtj/bSn/jwvznIlaqhAejgUXYM0xBx7uLxqExtOCBiE+8JQfYEehR4DD06G4tUe6Wjfgdwmlx43opwOXOUnbjkn1XifR+ErUgyzRlxeu5qBcRT3wxW1IkwrVp1WYSBZ7ddrRdJxzNG6DvOjWzTuISq44l6sMgELFPJGsL0GOHohi9L6znf2oLFYAspxUQqNP47Vx6vP8H48yFpQpLMZbdbhFJAX4mdH25emvAvUmYaipA+WQQJl+eGONPxYOnRj1vYJ3Yz2BPqvCAf8RkkHNWIOn4S/lePVn+eOJGunYVaW3SbcxVwJyOmK8WKVxFxOCYY7KnnwHYYzoNKo1Fc2kw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.003618, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A majority of the cgroup_rstat_cpu struct size is made up of the base stat entities. Since only the "self" subsystem state makes use of these, move them into a struct of their own. This allows for a new compact cgroup_rstat_cpu struct that the formal subsystems can make use of. Where applicable, decide on whether to allocate the compact or the full struct including the base stats. Signed-off-by: JP Kobryn --- include/linux/cgroup_rstat.h | 8 +++++- kernel/cgroup/rstat.c | 55 +++++++++++++++++++++++++++--------- 2 files changed, 48 insertions(+), 15 deletions(-) diff --git a/include/linux/cgroup_rstat.h b/include/linux/cgroup_rstat.h index 780b826ea364..fc26c0aa91ef 100644 --- a/include/linux/cgroup_rstat.h +++ b/include/linux/cgroup_rstat.h @@ -27,7 +27,10 @@ struct cgroup_rstat_cpu; * resource statistics on top of it - bsync, bstat and last_bstat. */ struct cgroup_rstat { - struct cgroup_rstat_cpu __percpu *rstat_cpu; + union { + struct cgroup_rstat_cpu __percpu *rstat_cpu; + struct cgroup_rstat_base_cpu __percpu *rstat_base_cpu; + }; /* * Add padding to separate the read mostly rstat_cpu and @@ -60,7 +63,10 @@ struct cgroup_rstat_cpu { */ struct cgroup_rstat *updated_children; /* terminated by self */ struct cgroup_rstat *updated_next; /* NULL if not on the list */ +}; +struct cgroup_rstat_base_cpu { + struct cgroup_rstat_cpu self; /* * ->bsync protects ->bstat. These are the only fields which get * updated in the hot path. diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 93b97bddec9c..6b14241d0924 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -33,6 +33,12 @@ static struct cgroup_rstat_cpu *rstat_cpu(struct cgroup_rstat *rstat, int cpu) return per_cpu_ptr(rstat->rstat_cpu, cpu); } +static struct cgroup_rstat_base_cpu *rstat_base_cpu( + struct cgroup_rstat *rstat, int cpu) +{ + return per_cpu_ptr(rstat->rstat_base_cpu, cpu); +} + static inline bool is_base_css(struct cgroup_subsys_state *css) { /* css for base stats has no subsystem */ @@ -597,6 +603,18 @@ static void __cgroup_rstat_init(struct cgroup_rstat *rstat) struct cgroup_rstat_cpu *rstatc = rstat_cpu(rstat, cpu); rstatc->updated_children = rstat; + } +} + +static void __cgroup_rstat_base_init(struct cgroup_rstat *rstat) +{ + int cpu; + + /* ->updated_children list is self terminated */ + for_each_possible_cpu(cpu) { + struct cgroup_rstat_base_cpu *rstatc = rstat_base_cpu(rstat, cpu); + + rstatc->self.updated_children = rstat; u64_stats_init(&rstatc->bsync); } } @@ -607,13 +625,21 @@ int cgroup_rstat_init(struct cgroup_subsys_state *css) /* the root cgrp has rstat_cpu preallocated */ if (!rstat->rstat_cpu) { - rstat->rstat_cpu = alloc_percpu(struct cgroup_rstat_cpu); - if (!rstat->rstat_cpu) - return -ENOMEM; + if (is_base_css(css)) { + rstat->rstat_base_cpu = alloc_percpu(struct cgroup_rstat_base_cpu); + if (!rstat->rstat_base_cpu) + return -ENOMEM; + + __cgroup_rstat_base_init(rstat); + } else { + rstat->rstat_cpu = alloc_percpu(struct cgroup_rstat_cpu); + if (!rstat->rstat_cpu) + return -ENOMEM; + + __cgroup_rstat_init(rstat); + } } - __cgroup_rstat_init(rstat); - return 0; } @@ -718,9 +744,10 @@ static void cgroup_base_stat_sub(struct cgroup_base_stat *dst_bstat, static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu) { - struct cgroup_rstat_cpu *rstatc = rstat_cpu(&(cgrp->self.rstat), cpu); + struct cgroup_rstat_base_cpu *rstatc = rstat_base_cpu( + &(cgrp->self.rstat), cpu); struct cgroup *parent = cgroup_parent(cgrp); - struct cgroup_rstat_cpu *prstatc; + struct cgroup_rstat_base_cpu *prstatc; struct cgroup_base_stat delta; unsigned seq; @@ -748,25 +775,25 @@ static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu) cgroup_base_stat_add(&cgrp->last_bstat, &delta); delta = rstatc->subtree_bstat; - prstatc = rstat_cpu(&(parent->self.rstat), cpu); + prstatc = rstat_base_cpu(&(parent->self.rstat), cpu); cgroup_base_stat_sub(&delta, &rstatc->last_subtree_bstat); cgroup_base_stat_add(&prstatc->subtree_bstat, &delta); cgroup_base_stat_add(&rstatc->last_subtree_bstat, &delta); } } -static struct cgroup_rstat_cpu * +static struct cgroup_rstat_base_cpu * cgroup_base_stat_cputime_account_begin(struct cgroup *cgrp, unsigned long *flags) { - struct cgroup_rstat_cpu *rstatc; + struct cgroup_rstat_base_cpu *rstatc; - rstatc = get_cpu_ptr(cgrp->self.rstat.rstat_cpu); + rstatc = get_cpu_ptr(cgrp->self.rstat.rstat_base_cpu); *flags = u64_stats_update_begin_irqsave(&rstatc->bsync); return rstatc; } static void cgroup_base_stat_cputime_account_end(struct cgroup *cgrp, - struct cgroup_rstat_cpu *rstatc, + struct cgroup_rstat_base_cpu *rstatc, unsigned long flags) { u64_stats_update_end_irqrestore(&rstatc->bsync, flags); @@ -776,7 +803,7 @@ static void cgroup_base_stat_cputime_account_end(struct cgroup *cgrp, void __cgroup_account_cputime(struct cgroup *cgrp, u64 delta_exec) { - struct cgroup_rstat_cpu *rstatc; + struct cgroup_rstat_base_cpu *rstatc; unsigned long flags; rstatc = cgroup_base_stat_cputime_account_begin(cgrp, &flags); @@ -787,7 +814,7 @@ void __cgroup_account_cputime(struct cgroup *cgrp, u64 delta_exec) void __cgroup_account_cputime_field(struct cgroup *cgrp, enum cpu_usage_stat index, u64 delta_exec) { - struct cgroup_rstat_cpu *rstatc; + struct cgroup_rstat_base_cpu *rstatc; unsigned long flags; rstatc = cgroup_base_stat_cputime_account_begin(cgrp, &flags);