From patchwork Tue Feb 18 03:14:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 13978851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3536DC021AB for ; Tue, 18 Feb 2025 03:15:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB5072800B5; Mon, 17 Feb 2025 22:15:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B648B2800A4; Mon, 17 Feb 2025 22:15:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 990192800B5; Mon, 17 Feb 2025 22:15:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6D1622800A4 for ; Mon, 17 Feb 2025 22:15:07 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DCC424FFC9 for ; Tue, 18 Feb 2025 03:15:06 +0000 (UTC) X-FDA: 83131599012.21.D30A2A5 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf07.hostedemail.com (Postfix) with ESMTP id F2CE540005 for ; Tue, 18 Feb 2025 03:15:04 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ACK0Al0P; spf=pass (imf07.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739848505; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HgpO8b1EeRoUbQ9L53xVbhgb2e+wvzP8W/zHZ0N0bI0=; b=c1bPjQQZTyfMVik1gLE1A0/dkRwu5cAkvqgh85pGP0NqmdZOiip/bFIWbHRVPCsj39VDuo CD+xfCM6CsJpNgE8LUMPpSWexnqzSnRjZ7u9t6VLFMybYh6tc/l1ny4+LZTevaixS4YDD0 Q1530zzUvU+4bo5SdRLScafhXgKSk9E= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ACK0Al0P; spf=pass (imf07.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739848505; a=rsa-sha256; cv=none; b=SychkcQ9H/1bqIOcOu75J055wU/4LmeENnuNIv8j2wq8JrGr1ywHBd6VK/iAQ00F7aYMrH JvdLETlX23gchc8TN+Vq95mq+ArH5gVps2JgS0dVqLBFFs8EHZZJ1imEfwIA9ol7T7Is8A fwyiJBharunBIVwUFmas35hHIQXlRQI= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-220dc3831e3so71140785ad.0 for ; Mon, 17 Feb 2025 19:15:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739848504; x=1740453304; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HgpO8b1EeRoUbQ9L53xVbhgb2e+wvzP8W/zHZ0N0bI0=; b=ACK0Al0PSMmg88P3M76M00r6sKQnTArARol+M9D5SuID+fZI5rHThs3cRvYbhlV4kL 2Qb4j3mTjHKVh3lhQ5diyCqVloxrbTc+P8Zp4MNsfbjs5m5vXPnREwovuNVPOrzNbl+e FQOuN5he7sD0wjy9uvoBJP5A6z3ntm4vTr+QofTMnpP/ClkdxW5uUsYDUySNPxA32tVD I8dja0SmO8v2OckmQtO76hNGW/K5fCtKXtEqd45hKeXV6qAZqjnaYy8MJVXGJyQhcqRD c0fwDT7PeuJ3VFqVwcA+lqNdw0M7JvADGoOS1Uen5Ngo2v7m3FBemOk3GzVuBGm5Yndv Nwgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739848504; x=1740453304; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HgpO8b1EeRoUbQ9L53xVbhgb2e+wvzP8W/zHZ0N0bI0=; b=ltxAkqVjonA4Jl/v3ebiQ2EXHup5ILqvdovLHO3gC0mLfoUo0kpAZ7J5WFlNV5TCs1 hnQLbHJEuHtVmRqQ5JoUoIFbjzLwbhgGm5YPzqPVPRPaRlQJd83Tq5FUMoO7dudxMdTM e5R2u/QgJgvZi3VjFfOoBBG1A/8yZYThodIGpU/5tBXFrQkEyXZHWttOW15b6D6MWxn8 G+wj5uegpUnia6Olg2s3JcjOlSUeWRFC24Dct5ImtgH6blM6g+Hdr7QDP7LawZQw4Htn MNSTYRKJ/RdliJV5KqtX0X8ctnIbyXbHozmXtXmGnZv8kw7Y9Fp9BipQfxUOMByXOpTj DrPQ== X-Gm-Message-State: AOJu0YxT3H0TofylT5lEebumgWodm8Dza1O5NetuhWbKYRDOmGgRq8vp PrjqRffbpp8t/q46A2cbpvnDZk/LJ6MLqOg5gUmNtBmAhL8udg+N X-Gm-Gg: ASbGncv+6TAKE7QqWwIo8t2UPIJ/9TdbyJAb88iaxRrfpfzb5es6sBYtYNnRh20s1bu 8KgMCBGbqqZYBTxT+tN2ri8o1KtpRcglJydN5uvFZxzmBnE84hjoJ0va0GIStkOWmxbd68n7I46 JgAMtzZL9mstO0qogcueRWL+X41Ovg85CO22tlF7zcAAwaCc0JnMNvV9JaSQplNP8TsIUQSquWI KFSd440F01t8amBzibX973r+Bod6Tmre4jpVAzlTvOi5hgZ7GkndzMQUiQZ+K0s6eKBzkz95VOk fr9DE2uY3MhzOqXI+tZ0fQnD1S/KJpEJOo69paBkZ6JR6fYgDO1d X-Google-Smtp-Source: AGHT+IGvVJgmZUZLNteiLjqooGGcN/Vh5bbt2bNgloW1Ge4ngfDQUgubE+o3sAAi/KBVt1FkRsrBsw== X-Received: by 2002:aa7:9912:0:b0:732:5875:eb95 with SMTP id d2e1a72fcca58-7325875ee4fmr19388335b3a.4.1739848503581; Mon, 17 Feb 2025 19:15:03 -0800 (PST) Received: from saturn.. (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732425466besm8763451b3a.9.2025.02.17.19.15.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Feb 2025 19:15:02 -0800 (PST) From: JP Kobryn To: shakeel.butt@linux.dev, tj@kernel.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 02/11] cgroup: add level of indirection for cgroup_rstat struct Date: Mon, 17 Feb 2025 19:14:39 -0800 Message-ID: <20250218031448.46951-3-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250218031448.46951-1-inwardvessel@gmail.com> References: <20250218031448.46951-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: F2CE540005 X-Stat-Signature: 8kg8sjz856jn9a1acewpd38km7ch13dw X-HE-Tag: 1739848504-803323 X-HE-Meta: U2FsdGVkX1/9aVm9NS1elxccaj/cbWk4TfSL3CIwhDzv10mHTb376su8AkBs6QfkSQh7Hs6tPqjqYU5SsIjAFiRfAVLcFJ7WOh2pwgaKDmmMCvA944VgePEVeanmJELM/WKPDPeXZdd6BnuToGUtH5RJF1Duj94Gy2ptKhv8OzJSl+bMOxmqJ/VJ2VAjUF8zUKgnDZULe+CQmQd1uH+HXTPxTE7jed4dcDuXPqqus1FFBdP9MswvrNVjYE6ty6+PLFXia75qz5KnUJp4Y1v6m/USyNCOO0kqg0fJ17XtLWEWeipTOehhMCesQ78+15dCmXmXil8UnwyNPbYSSLQYvrF3Jy5MfJndxHun7Pw8RFlkJsZfTqenN3v/pMteDwIGpZWGQVDECD3xVXtljcEsxUeCSpcz8jv5U7WbXnngbnJtz+GOuCIdkBMHekUYcNB1mANREjx19dx3WYjh+bA90I39buZopLJrkAMSLJwevXxlABjXmeP0RFH9No82EZFTqmE9pKCJAIGN3N4WGV88knwvn+NnCsXJ1bgAvYCWGOJoEaxz8tzRwzeJGLcccHQPMTDLacPcPAHPlSGJlmutwR2a0BKTRLtROZZRvJxkl82KX3E7GQP3XDPZhmY1oXCaBh+OALpSeENz6bbRCHDg/TdmTZ4yw+fxCaAvCFRJanHjM3b1tNVYnRzXz1fCjOPHe7dwg/d2zrJHSiJMVJI6x0f0WC/wmAbpHmS3SzHvw1cSBiolN74OW6cdgWLKdTm1JgdSW/H3VWuqkvDp05LFJ6utBCr+qFrbIxm9Rv7SfmNoRT83dL5dJ1GB4h+qmlMYaf8v2UEtQYVeQGfjUROI4CTi6x1E/d5H2ow3qg3Aijey9iX6KhYP3UOMUzE6ChidFsr+fwlWmrHlCu/5EURY/Vf29YQEYWWRaDVxxmUURuHQdWOp+oCFZuaOtpaCbZTzCnKB3oUz3Tewo5u25bP 9GGXnz+L NncQ/z62izOuRNiqpHWXIz4FkZAR9cRRCp3DA+ZcY/fqWfIOkCTdh50pUM6nx9mfogsDsfUCccmWWFC7UjR+hNU2RfBFyabyugTsdks+GxvTKLRUxYUGKJlorqxeolHOr8cz0u7tX/7yWowkNw9585TWsm9hEk0A7rqckqLR5Q5t6ibbepQU2/AlAZ230qn7q5rPzg2VB6fQB8eok0g+4lKb/aDuaAStS25OCB0uv/ohcEBjT0LisxHupWUI/EQhFJt0LfvVoGJOF9t+A1cQU48zxnUswGfaplZDkr8AMHsY2EJA5t+ybsDMNcx+aG4YsGczoVS3YmTlcGzcXdF+AOzQ3oOFgBRGo4RO7sfcA3tm0lK3Zc8UB2UQ5uDCvbTO8kI+/5CXdzHA3Zh9av3qEKy7ybiCajyf+e0LN9n82MiVr35xUIbtGcoCSrag7I/H/9XaIPNt+s1I0zc4PeEH5JVR7W1hitkrAVwnvCTZ2H7cxGbA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Change the type of rstat node from cgroup to the new cgroup_rstat struct. Then for the rstat updated/flush api calls, add double under versions that accept references to the cgroup_rstat struct. This new level of indirection will allow for extending the public api further. i.e. the cgroup_rstat struct can be embedded in a new type of object and a public api can be added for that new type. Signed-off-by: JP Kobryn --- include/linux/cgroup_rstat.h | 6 +- kernel/cgroup/cgroup-internal.h | 4 +- kernel/cgroup/cgroup.c | 12 +- kernel/cgroup/rstat.c | 204 ++++++++++++++++++++------------ 4 files changed, 141 insertions(+), 85 deletions(-) diff --git a/include/linux/cgroup_rstat.h b/include/linux/cgroup_rstat.h index f95474d6f8ab..780b826ea364 100644 --- a/include/linux/cgroup_rstat.h +++ b/include/linux/cgroup_rstat.h @@ -36,7 +36,7 @@ struct cgroup_rstat { * frequent updates. */ CACHELINE_PADDING(_pad_); - struct cgroup *rstat_flush_next; + struct cgroup_rstat *rstat_flush_next; }; struct cgroup_base_stat { @@ -58,8 +58,8 @@ struct cgroup_rstat_cpu { * to the cgroup makes it unnecessary for each per-cpu struct to * point back to the associated cgroup. */ - struct cgroup *updated_children; /* terminated by self */ - struct cgroup *updated_next; /* NULL if not on the list */ + struct cgroup_rstat *updated_children; /* terminated by self */ + struct cgroup_rstat *updated_next; /* NULL if not on the list */ /* * ->bsync protects ->bstat. These are the only fields which get diff --git a/kernel/cgroup/cgroup-internal.h b/kernel/cgroup/cgroup-internal.h index c964dd7ff967..03139018da43 100644 --- a/kernel/cgroup/cgroup-internal.h +++ b/kernel/cgroup/cgroup-internal.h @@ -269,8 +269,8 @@ int cgroup_task_count(const struct cgroup *cgrp); /* * rstat.c */ -int cgroup_rstat_init(struct cgroup *cgrp); -void cgroup_rstat_exit(struct cgroup *cgrp); +int cgroup_rstat_init(struct cgroup_rstat *rstat); +void cgroup_rstat_exit(struct cgroup_rstat *rstat); void cgroup_rstat_boot(void); void cgroup_base_stat_cputime_show(struct seq_file *seq); diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 03a3a4da49f1..02d6c11ccccb 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -1359,7 +1359,7 @@ static void cgroup_destroy_root(struct cgroup_root *root) cgroup_unlock(); - cgroup_rstat_exit(cgrp); + cgroup_rstat_exit(&cgrp->rstat); kernfs_destroy_root(root->kf_root); cgroup_free_root(root); } @@ -2133,7 +2133,7 @@ int cgroup_setup_root(struct cgroup_root *root, u16 ss_mask) if (ret) goto destroy_root; - ret = cgroup_rstat_init(root_cgrp); + ret = cgroup_rstat_init(&root_cgrp->rstat); if (ret) goto destroy_root; @@ -2175,7 +2175,7 @@ int cgroup_setup_root(struct cgroup_root *root, u16 ss_mask) goto out; exit_stats: - cgroup_rstat_exit(root_cgrp); + cgroup_rstat_exit(&root_cgrp->rstat); destroy_root: kernfs_destroy_root(root->kf_root); root->kf_root = NULL; @@ -5436,7 +5436,7 @@ static void css_free_rwork_fn(struct work_struct *work) cgroup_put(cgroup_parent(cgrp)); kernfs_put(cgrp->kn); psi_cgroup_free(cgrp); - cgroup_rstat_exit(cgrp); + cgroup_rstat_exit(&cgrp->rstat); kfree(cgrp); } else { /* @@ -5687,7 +5687,7 @@ static struct cgroup *cgroup_create(struct cgroup *parent, const char *name, if (ret) goto out_free_cgrp; - ret = cgroup_rstat_init(cgrp); + ret = cgroup_rstat_init(&cgrp->rstat); if (ret) goto out_cancel_ref; @@ -5780,7 +5780,7 @@ static struct cgroup *cgroup_create(struct cgroup *parent, const char *name, out_kernfs_remove: kernfs_remove(cgrp->kn); out_stat_exit: - cgroup_rstat_exit(cgrp); + cgroup_rstat_exit(&cgrp->rstat); out_cancel_ref: percpu_ref_exit(&cgrp->self.refcnt); out_free_cgrp: diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 7e7879d88c38..13090dda56aa 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -14,9 +14,20 @@ static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_cpu_lock); static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu); -static struct cgroup_rstat_cpu *cgroup_rstat_cpu(struct cgroup *cgrp, int cpu) +static struct cgroup_rstat_cpu *rstat_cpu(struct cgroup_rstat *rstat, int cpu) { - return per_cpu_ptr(cgrp->rstat.rstat_cpu, cpu); + return per_cpu_ptr(rstat->rstat_cpu, cpu); +} + +static struct cgroup_rstat *rstat_parent(struct cgroup_rstat *rstat) +{ + struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + struct cgroup *parent = cgroup_parent(cgrp); + + if (!parent) + return NULL; + + return &parent->rstat; } /* @@ -73,17 +84,9 @@ void _cgroup_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, raw_spin_unlock_irqrestore(cpu_lock, flags); } -/** - * cgroup_rstat_updated - keep track of updated rstat_cpu - * @cgrp: target cgroup - * @cpu: cpu on which rstat_cpu was updated - * - * @cgrp's rstat_cpu on @cpu was updated. Put it on the parent's matching - * rstat_cpu->updated_children list. See the comment on top of - * cgroup_rstat_cpu definition for details. - */ -__bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) +static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu) { + struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); unsigned long flags; @@ -95,15 +98,15 @@ __bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) * instead of NULL, we can tell whether @cgrp is on the list by * testing the next pointer for NULL. */ - if (data_race(cgroup_rstat_cpu(cgrp, cpu)->updated_next)) + if (data_race(rstat_cpu(rstat, cpu)->updated_next)) return; flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, true); /* put @cgrp and all ancestors on the corresponding updated lists */ while (true) { - struct cgroup_rstat_cpu *rstatc = cgroup_rstat_cpu(cgrp, cpu); - struct cgroup *parent = cgroup_parent(cgrp); + struct cgroup_rstat_cpu *rstatc = rstat_cpu(rstat, cpu); + struct cgroup_rstat *parent = rstat_parent(rstat); struct cgroup_rstat_cpu *prstatc; /* @@ -115,20 +118,34 @@ __bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) /* Root has no parent to link it to, but mark it busy */ if (!parent) { - rstatc->updated_next = cgrp; + rstatc->updated_next = rstat; break; } - prstatc = cgroup_rstat_cpu(parent, cpu); + prstatc = rstat_cpu(parent, cpu); rstatc->updated_next = prstatc->updated_children; - prstatc->updated_children = cgrp; + prstatc->updated_children = rstat; - cgrp = parent; + rstat = parent; } _cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, true); } +/** + * cgroup_rstat_updated - keep track of updated rstat_cpu + * @cgrp: target cgroup + * @cpu: cpu on which rstat_cpu was updated + * + * @cgrp's rstat_cpu on @cpu was updated. Put it on the parent's matching + * rstat_cpu->updated_children list. See the comment on top of + * cgroup_rstat_cpu definition for details. + */ +__bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) +{ + __cgroup_rstat_updated(&cgrp->rstat, cpu); +} + /** * cgroup_rstat_push_children - push children cgroups into the given list * @head: current head of the list (= subtree root) @@ -141,32 +158,32 @@ __bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) * into a singly linked list built from the tail backward like "pushing" * cgroups into a stack. The root is pushed by the caller. */ -static struct cgroup *cgroup_rstat_push_children(struct cgroup *head, - struct cgroup *child, int cpu) +static struct cgroup_rstat *cgroup_rstat_push_children( + struct cgroup_rstat *head, struct cgroup_rstat *child, int cpu) { - struct cgroup *chead = child; /* Head of child cgroup level */ - struct cgroup *ghead = NULL; /* Head of grandchild cgroup level */ - struct cgroup *parent, *grandchild; + struct cgroup_rstat *chead = child; /* Head of child cgroup level */ + struct cgroup_rstat *ghead = NULL; /* Head of grandchild cgroup level */ + struct cgroup_rstat *parent, *grandchild; struct cgroup_rstat_cpu *crstatc; - child->rstat.rstat_flush_next = NULL; + child->rstat_flush_next = NULL; next_level: while (chead) { child = chead; - chead = child->rstat.rstat_flush_next; - parent = cgroup_parent(child); + chead = child->rstat_flush_next; + parent = rstat_parent(child); /* updated_next is parent cgroup terminated */ while (child != parent) { - child->rstat.rstat_flush_next = head; + child->rstat_flush_next = head; head = child; - crstatc = cgroup_rstat_cpu(child, cpu); + crstatc = rstat_cpu(child, cpu); grandchild = crstatc->updated_children; if (grandchild != child) { /* Push the grand child to the next level */ crstatc->updated_children = child; - grandchild->rstat.rstat_flush_next = ghead; + grandchild->rstat_flush_next = ghead; ghead = grandchild; } child = crstatc->updated_next; @@ -200,14 +217,16 @@ static struct cgroup *cgroup_rstat_push_children(struct cgroup *head, * within the children list and terminated by the parent cgroup. An exception * here is the cgroup root whose updated_next can be self terminated. */ -static struct cgroup *cgroup_rstat_updated_list(struct cgroup *root, int cpu) +static struct cgroup_rstat *cgroup_rstat_updated_list( + struct cgroup_rstat *root, int cpu) { + struct cgroup *cgrp = container_of(root, typeof(*cgrp), rstat); raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); - struct cgroup_rstat_cpu *rstatc = cgroup_rstat_cpu(root, cpu); - struct cgroup *head = NULL, *parent, *child; + struct cgroup_rstat_cpu *rstatc = rstat_cpu(root, cpu); + struct cgroup_rstat *head = NULL, *parent, *child; unsigned long flags; - flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, root, false); + flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, false); /* Return NULL if this subtree is not on-list */ if (!rstatc->updated_next) @@ -217,17 +236,17 @@ static struct cgroup *cgroup_rstat_updated_list(struct cgroup *root, int cpu) * Unlink @root from its parent. As the updated_children list is * singly linked, we have to walk it to find the removal point. */ - parent = cgroup_parent(root); + parent = rstat_parent(root); if (parent) { struct cgroup_rstat_cpu *prstatc; - struct cgroup **nextp; + struct cgroup_rstat **nextp; - prstatc = cgroup_rstat_cpu(parent, cpu); + prstatc = rstat_cpu(parent, cpu); nextp = &prstatc->updated_children; while (*nextp != root) { struct cgroup_rstat_cpu *nrstatc; - nrstatc = cgroup_rstat_cpu(*nextp, cpu); + nrstatc = rstat_cpu(*nextp, cpu); WARN_ON_ONCE(*nextp == parent); nextp = &nrstatc->updated_next; } @@ -238,13 +257,13 @@ static struct cgroup *cgroup_rstat_updated_list(struct cgroup *root, int cpu) /* Push @root to the list first before pushing the children */ head = root; - root->rstat.rstat_flush_next = NULL; + root->rstat_flush_next = NULL; child = rstatc->updated_children; rstatc->updated_children = root; if (child != root) head = cgroup_rstat_push_children(head, child, cpu); unlock_ret: - _cgroup_rstat_cpu_unlock(cpu_lock, cpu, root, flags, false); + _cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, false); return head; } @@ -300,24 +319,26 @@ static inline void __cgroup_rstat_unlock(struct cgroup *cgrp, int cpu_in_loop) } /* see cgroup_rstat_flush() */ -static void cgroup_rstat_flush_locked(struct cgroup *cgrp) +static void cgroup_rstat_flush_locked(struct cgroup_rstat *rstat) __releases(&cgroup_rstat_lock) __acquires(&cgroup_rstat_lock) { + struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); int cpu; lockdep_assert_held(&cgroup_rstat_lock); for_each_possible_cpu(cpu) { - struct cgroup *pos = cgroup_rstat_updated_list(cgrp, cpu); + struct cgroup_rstat *pos = cgroup_rstat_updated_list(rstat, cpu); - for (; pos; pos = pos->rstat.rstat_flush_next) { + for (; pos; pos = pos->rstat_flush_next) { + struct cgroup *pos_cgroup = container_of(pos, struct cgroup, rstat); struct cgroup_subsys_state *css; - cgroup_base_stat_flush(pos, cpu); - bpf_rstat_flush(pos, cgroup_parent(pos), cpu); + cgroup_base_stat_flush(pos_cgroup, cpu); + bpf_rstat_flush(pos_cgroup, cgroup_parent(pos_cgroup), cpu); rcu_read_lock(); - list_for_each_entry_rcu(css, &pos->rstat_css_list, + list_for_each_entry_rcu(css, &pos_cgroup->rstat_css_list, rstat_css_node) css->ss->css_rstat_flush(css, cpu); rcu_read_unlock(); @@ -333,6 +354,17 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp) } } +static void __cgroup_rstat_flush(struct cgroup_rstat *rstat) +{ + struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + + might_sleep(); + + __cgroup_rstat_lock(cgrp, -1); + cgroup_rstat_flush_locked(rstat); + __cgroup_rstat_unlock(cgrp, -1); +} + /** * cgroup_rstat_flush - flush stats in @cgrp's subtree * @cgrp: target cgroup @@ -348,11 +380,17 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp) */ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) { - might_sleep(); + __cgroup_rstat_flush(&cgrp->rstat); +} + +static void __cgroup_rstat_flush_hold(struct cgroup_rstat *rstat) + __acquires(&cgroup_rstat_lock) +{ + struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + might_sleep(); __cgroup_rstat_lock(cgrp, -1); - cgroup_rstat_flush_locked(cgrp); - __cgroup_rstat_unlock(cgrp, -1); + cgroup_rstat_flush_locked(rstat); } /** @@ -365,63 +403,81 @@ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) * This function may block. */ void cgroup_rstat_flush_hold(struct cgroup *cgrp) - __acquires(&cgroup_rstat_lock) { - might_sleep(); - __cgroup_rstat_lock(cgrp, -1); - cgroup_rstat_flush_locked(cgrp); + __cgroup_rstat_flush_hold(&cgrp->rstat); } /** * cgroup_rstat_flush_release - release cgroup_rstat_flush_hold() * @cgrp: cgroup used by tracepoint */ -void cgroup_rstat_flush_release(struct cgroup *cgrp) +static void __cgroup_rstat_flush_release(struct cgroup_rstat *rstat) __releases(&cgroup_rstat_lock) { + struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + __cgroup_rstat_unlock(cgrp, -1); } -int cgroup_rstat_init(struct cgroup *cgrp) +/** + * cgroup_rstat_flush_release - release cgroup_rstat_flush_hold() + * @cgrp: cgroup used by tracepoint + */ +void cgroup_rstat_flush_release(struct cgroup *cgrp) { - int cpu; + __cgroup_rstat_flush_release(&cgrp->rstat); +} - /* the root cgrp has rstat_cpu preallocated */ - if (!cgrp->rstat.rstat_cpu) { - cgrp->rstat.rstat_cpu = alloc_percpu( - struct cgroup_rstat_cpu); - if (!cgrp->rstat.rstat_cpu) - return -ENOMEM; - } +static void __cgroup_rstat_init(struct cgroup_rstat *rstat) +{ + int cpu; /* ->updated_children list is self terminated */ for_each_possible_cpu(cpu) { - struct cgroup_rstat_cpu *rstatc = cgroup_rstat_cpu(cgrp, cpu); + struct cgroup_rstat_cpu *rstatc = rstat_cpu(rstat, cpu); - rstatc->updated_children = cgrp; + rstatc->updated_children = rstat; u64_stats_init(&rstatc->bsync); } +} + +int cgroup_rstat_init(struct cgroup_rstat *rstat) +{ + /* the root cgrp has rstat_cpu preallocated */ + if (!rstat->rstat_cpu) { + rstat->rstat_cpu = alloc_percpu(struct cgroup_rstat_cpu); + if (!rstat->rstat_cpu) + return -ENOMEM; + } + + __cgroup_rstat_init(rstat); return 0; } -void cgroup_rstat_exit(struct cgroup *cgrp) +static void __cgroup_rstat_exit(struct cgroup_rstat *rstat) { int cpu; - cgroup_rstat_flush(cgrp); - /* sanity check */ for_each_possible_cpu(cpu) { - struct cgroup_rstat_cpu *rstatc = cgroup_rstat_cpu(cgrp, cpu); + struct cgroup_rstat_cpu *rstatc = rstat_cpu(rstat, cpu); - if (WARN_ON_ONCE(rstatc->updated_children != cgrp) || + if (WARN_ON_ONCE(rstatc->updated_children != rstat) || WARN_ON_ONCE(rstatc->updated_next)) return; } - free_percpu(cgrp->rstat.rstat_cpu); - cgrp->rstat.rstat_cpu = NULL; + free_percpu(rstat->rstat_cpu); + rstat->rstat_cpu = NULL; +} + +void cgroup_rstat_exit(struct cgroup_rstat *rstat) +{ + struct cgroup *cgrp = container_of(rstat, typeof(*cgrp), rstat); + + cgroup_rstat_flush(cgrp); + __cgroup_rstat_exit(rstat); } void __init cgroup_rstat_boot(void) @@ -462,7 +518,7 @@ static void cgroup_base_stat_sub(struct cgroup_base_stat *dst_bstat, static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu) { - struct cgroup_rstat_cpu *rstatc = cgroup_rstat_cpu(cgrp, cpu); + struct cgroup_rstat_cpu *rstatc = rstat_cpu(&cgrp->rstat, cpu); struct cgroup *parent = cgroup_parent(cgrp); struct cgroup_rstat_cpu *prstatc; struct cgroup_base_stat delta; @@ -492,7 +548,7 @@ static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu) cgroup_base_stat_add(&cgrp->last_bstat, &delta); delta = rstatc->subtree_bstat; - prstatc = cgroup_rstat_cpu(parent, cpu); + prstatc = rstat_cpu(&parent->rstat, cpu); cgroup_base_stat_sub(&delta, &rstatc->last_subtree_bstat); cgroup_base_stat_add(&prstatc->subtree_bstat, &delta); cgroup_base_stat_add(&rstatc->last_subtree_bstat, &delta);