From patchwork Wed Mar 19 22:16:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 14023227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39006C36000 for ; Wed, 19 Mar 2025 22:17:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 77A7328000A; Wed, 19 Mar 2025 18:17:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FE9E280002; Wed, 19 Mar 2025 18:17:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5605C28000A; Wed, 19 Mar 2025 18:17:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2A6FB280002 for ; Wed, 19 Mar 2025 18:17:13 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 880EDA9C22 for ; Wed, 19 Mar 2025 22:17:14 +0000 (UTC) X-FDA: 83239712388.07.68606CE Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf26.hostedemail.com (Postfix) with ESMTP id D8C21140008 for ; Wed, 19 Mar 2025 22:17:12 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Y3VlD8Br; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf26.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742422632; a=rsa-sha256; cv=none; b=fAm4dT/sNJzqGMZ5B0MDQzCt745DG27PTeriSNwO6i8+SxCFk5b5swpIZEBvcHT/tFQb7j J5IOn2A4G9kr2xy4TyHEmwPUMfRgrlZR8QJP3SgfRtwLd6YQ4D81cqkVbeOI+wRoAo1KsK nprDPen6lAKmI3CJ0CIEW32zLBvuKF0= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Y3VlD8Br; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf26.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742422632; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yBVQ9JLe0YyGvw+H4jsTKvC5B30TGlzUOUpyXhkBvvk=; b=HZ/ayAiI+rca5h9on0CxJHPhuj9zmNQpB8UHRkZhO8b1M1A8D7HAeQt0859coW5FblkOyr HjxPjTTQyQHrPn630I5fRmqMzNAkSN3ZAUyJ0/xFyluXLVI0JqdgT7UedEwDw1fvJhsddR 9iJ3yrZ7bH8YZYQo0cJukX4vOqEvQ20= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-224171d6826so1395575ad.3 for ; Wed, 19 Mar 2025 15:17:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742422632; x=1743027432; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yBVQ9JLe0YyGvw+H4jsTKvC5B30TGlzUOUpyXhkBvvk=; b=Y3VlD8BrNAU+3l0mYXJKo9QhBvNw+2g8nEFj3nST6krJmhYF9dxPL1TwfDIjAns/my XX3QhPfg0ryVZ6PavdwCSgwRom8OhTCk8RCfzXEAqOoLKYGtNNMmUeX5h1xTpFNo1A7L Wk008ijQUKe8fRHjECHQURI0C3zffMDzNHH4kSvhI8gApuH8D1PglHGkZ0RryoJICoKv VQ5Yw/q7HGgY8GENahXlXmJm02WCOcVhQ9JenNSv3tO/ru/YPuHI/dBuxQ2QkNlnZdmA Ci11oKt9Zc9CWy0FDuF+hm6fcwVEXjV82W865wIxlmEn1viKrUx8q7tDmT1gIUbbfGWT tG1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742422632; x=1743027432; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yBVQ9JLe0YyGvw+H4jsTKvC5B30TGlzUOUpyXhkBvvk=; b=xJwD+1Y6fFCEOrqR+tO/g2Lz2Rvr/74V5eGcI9zRN88OwmZuIuKCwYe4tWmq9KIYXp fPJPaH43XhRHTP9oB58tHAx/maOfost0JCHwC//5rO28qywJvONPuyzXpWgPSSmgLQLC tXO3BF/GWuTpWCKTfdm5lmAHLughTs09TJuMmaN0dnO4HScxdk7ErBLMxeYVC5WmHI6v L4+wRckJg3BsC1T6bsrySov/UNKc//nWwGQXJhbnm4VFzMegDYIxdUi7WTbpwbLwwwS+ yq3sgXuRuzzAm1HaMDhEOuEzseQvxBvekXsSNxsdAzMhtL89PuRaQZ7GPM/b+N0IUyY1 l7qg== X-Gm-Message-State: AOJu0YymvbXA7oZpUKBK9RvfJwGRDhZwP6R/aW8u39Q5tORanACNslxi TtQsSNHeb/Pip9AJHZIiudiIgOQAAaA+oeDwlu64VEVxSPygdbAH X-Gm-Gg: ASbGncsjwOMo9p0518fY0KaQQ8Jq/jMu20I1jwV+Vj6nT5eFeR0wibIFuPP1ur3Ou9d B4xet9qiAEmvFLokqWspjBClBZ1dVQ2zl5yzofFKh/ZPbrvTeZoxg+qTHz1YS2EqK6bNdewY3zU IGih3y2RR8cSS0RQZstwtt5YBHo4OtxipaNjrwznnb8piLz+0WUBb61X9nNQ2cjqqoGHvg+Temb EdCGsapx1j5qdSPcJ2sDnpBHMOpkirS2d8swe9UR/5XyHQF9UOF1isMY8AJUd+prEVJkpIHz5rz kiYNrHfEt4U90AVQjCzX+G8Rwa2uwgkKKjFwXFob958awpgPhMGRBfQNSM6QClg55gwtLAyO X-Google-Smtp-Source: AGHT+IFkBKXNjicIFUdDP35mA4KEP43YFXh5t0TytgASyjMgPQgr3FsiJbwrqnL3GZ6l81MVO6Gx0Q== X-Received: by 2002:a17:902:e747:b0:220:efc8:60b1 with SMTP id d9443c01a7336-22649a68215mr65921775ad.39.1742422631621; Wed, 19 Mar 2025 15:17:11 -0700 (PDT) Received: from jpkobryn-fedora-PF5CFKNC.thefacebook.com ([2620:10d:c090:500::4:39d5]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-737116af372sm12253977b3a.160.2025.03.19.15.17.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Mar 2025 15:17:11 -0700 (PDT) From: JP Kobryn To: tj@kernel.org, shakeel.butt@linux.dev, yosryahmed@google.com, mkoutny@suse.com, hannes@cmpxchg.org, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 3/4] cgroup: use subsystem-specific rstat locks to avoid contention Date: Wed, 19 Mar 2025 15:16:33 -0700 Message-ID: <20250319221634.71128-4-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250319221634.71128-1-inwardvessel@gmail.com> References: <20250319221634.71128-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: D8C21140008 X-Stat-Signature: fzp8ktgckmhip5zu1hbmi5ccq5e1u7wz X-Rspamd-Server: rspam06 X-HE-Tag: 1742422632-996337 X-HE-Meta: U2FsdGVkX18RtFBc3x7SgVMmPkTXNBpMPDSRTlb87Av6/Iq33B6cFD6dO+8RuFK6j/vnLnjh3x1H6NmDQkoGVo9IPzGUDNJ7IaJMNXvQhy1ZDc9p3cjVhjMZLKLg6JLOyXeYdEhhAeOO9JRBW4jJmf3ID9Kw1OnEJEg/YAIi6Y3E2Ah0tzIveNgCpT82WyX4RGntGa8/Oyyfn3JK8O+iPxR0EsRGbYSUotctPtS+hYfoHzu84Tg89i+qneHaeXFXkg+7ctV5RfmToWq9e5w5osKJ0FbSpNpXNbPBoLGhsm795ihty9/Xk68olYLE0CYLbjvD1pe7dkTQ37JPJi2dSfzUlItHOfeZB6ibOFg7ZByOgLL8tvr5yhiW7YjbnllEG4wekotSky3mSuNyEnnVt+VATxce8WwCxWg7sat6JrTQswy8u2nCwltiRQH3C7Zod0CeqkrRnRwh9lhgPUSLyiNm5l+hvblJtxhw/IbdYW4qWJ6d8TKjV3/1LONSLVc4ZL5PzFHxUZI5No8ngIEjI8xu8QY1xyxQ8jS66vbB9lkZrk6hwdwBNiiYEFsVbTdDqSr6HWLPARxcM+nZ5TDLZnHK65I+x83877wi06kz+izslLwoABi9yTaW/LK7YkR4MVESQ7YjfvY4GYri/lG/opXJHIxhfaIA2ewc3NaKdUXblOlXUIepyHctMMKBBINfRQ8LhGA+f+e3eClLya6XbMltFDJndJlKAJfJa3REii3qiQNNsL+t8LZ4RCsRk8Hxbkw8g2HLu+mW3O6+cKu3uUbDxdq99cEEqXW0LxT0tDdKbl2WAjrvgKxAJzB93XDljtVcXsMcnfwRriNUtxhm5PC5HQ6HwiY7+zdqa3rqyQMuq5JvE5jMIlokbaX/sP4JQooAsqykq6lwZviWBsPmGDvm/IAuYWOBHHA0yYt0u7cU1tcGgPoW8m6QXy1gZAQWnp80hSGhg43c5aGyznl ix8M8N3I wC2C9ZyiVyqsT9/k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: It is possible to eliminate contention between subsystems when updating/flushing stats by using subsystem-specific locks. Let the existing rstat locks be dedicated to the base stats and rename them to reflect it. Add similar locks to the cgroup_subsys struct for use with individual subsystems. To make use of the new locks, change the existing lock helper functions to accept a reference to a css and use css->ss to access the locks or use the static locks for base stats when css->ss is NULL. Signed-off-by: JP Kobryn --- block/blk-cgroup.c | 2 +- include/linux/cgroup-defs.h | 12 ++- include/trace/events/cgroup.h | 10 ++- kernel/cgroup/cgroup-internal.h | 2 +- kernel/cgroup/cgroup.c | 10 ++- kernel/cgroup/rstat.c | 145 +++++++++++++++++++++----------- 6 files changed, 122 insertions(+), 59 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index cd9521f4f607..34d72bbdd5e5 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1022,7 +1022,7 @@ static void __blkcg_rstat_flush(struct blkcg *blkcg, int cpu) /* * For covering concurrent parent blkg update from blkg_release(). * - * When flushing from cgroup, cgroup_rstat_lock is always held, so + * When flushing from cgroup, the subsystem lock is always held, so * this lock won't cause contention most of time. */ raw_spin_lock_irqsave(&blkg_stat_lock, flags); diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index 031f55a9ac49..0ffc8438c6d9 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -223,7 +223,10 @@ struct cgroup_subsys_state { /* * A singly-linked list of css structures to be rstat flushed. * This is a scratch field to be used exclusively by - * cgroup_rstat_flush_locked() and protected by cgroup_rstat_lock. + * cgroup_rstat_flush_locked(). + * + * protected by rstat_base_lock when css is cgroup::self + * protected by css->ss->lock otherwise */ struct cgroup_subsys_state *rstat_flush_next; }; @@ -391,7 +394,9 @@ struct css_rstat_cpu { * to the cgroup makes it unnecessary for each per-cpu struct to * point back to the associated cgroup. * - * Protected by per-cpu cgroup_rstat_cpu_lock. + * Protected by per-cpu rstat_base_cpu_lock when css->ss == NULL + * otherwise, + * Protected by per-cpu css->ss->rstat_cpu_lock */ struct cgroup_subsys_state *updated_children; /* terminated by self */ struct cgroup_subsys_state *updated_next; /* NULL if not on list */ @@ -779,6 +784,9 @@ struct cgroup_subsys { * specifies the mask of subsystems that this one depends on. */ unsigned int depends_on; + + spinlock_t lock; + raw_spinlock_t __percpu *percpu_lock; }; extern struct percpu_rw_semaphore cgroup_threadgroup_rwsem; diff --git a/include/trace/events/cgroup.h b/include/trace/events/cgroup.h index af2755bda6eb..ec3a95bf4981 100644 --- a/include/trace/events/cgroup.h +++ b/include/trace/events/cgroup.h @@ -231,7 +231,10 @@ DECLARE_EVENT_CLASS(cgroup_rstat, __entry->cpu, __entry->contended) ); -/* Related to global: cgroup_rstat_lock */ +/* Related to locks: + * rstat_base_lock when handling cgroup::self + * css->ss->lock otherwise + */ DEFINE_EVENT(cgroup_rstat, cgroup_rstat_lock_contended, TP_PROTO(struct cgroup *cgrp, int cpu, bool contended), @@ -253,7 +256,10 @@ DEFINE_EVENT(cgroup_rstat, cgroup_rstat_unlock, TP_ARGS(cgrp, cpu, contended) ); -/* Related to per CPU: cgroup_rstat_cpu_lock */ +/* Related to per CPU locks: + * rstat_base_cpu_lock when handling cgroup::self + * css->ss->cpu_lock otherwise + */ DEFINE_EVENT(cgroup_rstat, cgroup_rstat_cpu_lock_contended, TP_PROTO(struct cgroup *cgrp, int cpu, bool contended), diff --git a/kernel/cgroup/cgroup-internal.h b/kernel/cgroup/cgroup-internal.h index d4b75fba9a54..513bfce3bc23 100644 --- a/kernel/cgroup/cgroup-internal.h +++ b/kernel/cgroup/cgroup-internal.h @@ -271,7 +271,7 @@ int cgroup_task_count(const struct cgroup *cgrp); */ int css_rstat_init(struct cgroup_subsys_state *css); void css_rstat_exit(struct cgroup_subsys_state *css); -void cgroup_rstat_boot(void); +int ss_rstat_init(struct cgroup_subsys *ss); void cgroup_base_stat_cputime_show(struct seq_file *seq); /* diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 1e21065dec0e..3e8948805f67 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -6085,8 +6085,10 @@ static void __init cgroup_init_subsys(struct cgroup_subsys *ss, bool early) css->id = cgroup_idr_alloc(&ss->css_idr, css, 1, 2, GFP_KERNEL); BUG_ON(css->id < 0); - if (ss->css_rstat_flush) + if (ss->css_rstat_flush) { + BUG_ON(ss_rstat_init(ss)); BUG_ON(css_rstat_init(css)); + } } /* Update the init_css_set to contain a subsys @@ -6163,7 +6165,7 @@ int __init cgroup_init(void) BUG_ON(cgroup_init_cftypes(NULL, cgroup_psi_files)); BUG_ON(cgroup_init_cftypes(NULL, cgroup1_base_files)); - cgroup_rstat_boot(); + BUG_ON(ss_rstat_init(NULL)); get_user_ns(init_cgroup_ns.user_ns); @@ -6193,8 +6195,10 @@ int __init cgroup_init(void) GFP_KERNEL); BUG_ON(css->id < 0); - if (ss->css_rstat_flush) + if (ss->css_rstat_flush) { + BUG_ON(ss_rstat_init(ss)); BUG_ON(css_rstat_init(css)); + } } else { cgroup_init_subsys(ss, false); } diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index a28c00b11736..ffd7ac6bcefc 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -9,8 +9,8 @@ #include -static DEFINE_SPINLOCK(cgroup_rstat_lock); -static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_cpu_lock); +static DEFINE_SPINLOCK(rstat_base_lock); +static DEFINE_PER_CPU(raw_spinlock_t, rstat_base_cpu_lock); static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu); @@ -20,8 +20,24 @@ static struct css_rstat_cpu *css_rstat_cpu( return per_cpu_ptr(css->rstat_cpu, cpu); } +static spinlock_t *ss_rstat_lock(struct cgroup_subsys *ss) +{ + if (ss) + return &ss->lock; + + return &rstat_base_lock; +} + +static raw_spinlock_t *ss_rstat_cpu_lock(struct cgroup_subsys *ss, int cpu) +{ + if (ss) + return per_cpu_ptr(ss->percpu_lock, cpu); + + return per_cpu_ptr(&rstat_base_cpu_lock, cpu); +} + /* - * Helper functions for rstat per CPU lock (cgroup_rstat_cpu_lock). + * Helper functions for rstat per CPU locks. * * This makes it easier to diagnose locking issues and contention in * production environments. The parameter @fast_path determine the @@ -29,20 +45,23 @@ static struct css_rstat_cpu *css_rstat_cpu( * operations without handling high-frequency fast-path "update" events. */ static __always_inline -unsigned long _cgroup_rstat_cpu_lock(raw_spinlock_t *cpu_lock, int cpu, - struct cgroup *cgrp, const bool fast_path) +unsigned long _css_rstat_cpu_lock(struct cgroup_subsys_state *css, int cpu, + const bool fast_path) { + struct cgroup *cgrp = css->cgroup; + raw_spinlock_t *cpu_lock; unsigned long flags; bool contended; /* - * The _irqsave() is needed because cgroup_rstat_lock is - * spinlock_t which is a sleeping lock on PREEMPT_RT. Acquiring - * this lock with the _irq() suffix only disables interrupts on - * a non-PREEMPT_RT kernel. The raw_spinlock_t below disables - * interrupts on both configurations. The _irqsave() ensures - * that interrupts are always disabled and later restored. + * The _irqsave() is needed because the locks used for flushing are + * spinlock_t which is a sleeping lock on PREEMPT_RT. Acquiring this lock + * with the _irq() suffix only disables interrupts on a non-PREEMPT_RT + * kernel. The raw_spinlock_t below disables interrupts on both + * configurations. The _irqsave() ensures that interrupts are always + * disabled and later restored. */ + cpu_lock = ss_rstat_cpu_lock(css->ss, cpu); contended = !raw_spin_trylock_irqsave(cpu_lock, flags); if (contended) { if (fast_path) @@ -62,15 +81,18 @@ unsigned long _cgroup_rstat_cpu_lock(raw_spinlock_t *cpu_lock, int cpu, } static __always_inline -void _cgroup_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, - struct cgroup *cgrp, unsigned long flags, - const bool fast_path) +void _css_rstat_cpu_unlock(struct cgroup_subsys_state *css, int cpu, + unsigned long flags, const bool fast_path) { + struct cgroup *cgrp = css->cgroup; + raw_spinlock_t *cpu_lock; + if (fast_path) trace_cgroup_rstat_cpu_unlock_fastpath(cgrp, cpu, false); else trace_cgroup_rstat_cpu_unlock(cgrp, cpu, false); + cpu_lock = ss_rstat_cpu_lock(css->ss, cpu); raw_spin_unlock_irqrestore(cpu_lock, flags); } @@ -85,8 +107,6 @@ void _cgroup_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, */ void css_rstat_updated(struct cgroup_subsys_state *css, int cpu) { - struct cgroup *cgrp = css->cgroup; - raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); unsigned long flags; /* @@ -100,7 +120,7 @@ void css_rstat_updated(struct cgroup_subsys_state *css, int cpu) if (data_race(css_rstat_cpu(css, cpu)->updated_next)) return; - flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, true); + flags = _css_rstat_cpu_lock(css, cpu, true); /* put @css and all ancestors on the corresponding updated lists */ while (true) { @@ -128,7 +148,7 @@ void css_rstat_updated(struct cgroup_subsys_state *css, int cpu) css = parent; } - _cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, true); + _css_rstat_cpu_unlock(css, cpu, flags, true); } __bpf_kfunc void bpf_cgroup_rstat_updated(struct cgroup *cgrp, int cpu) @@ -211,13 +231,11 @@ static struct cgroup_subsys_state *cgroup_rstat_push_children( static struct cgroup_subsys_state *css_rstat_updated_list( struct cgroup_subsys_state *root, int cpu) { - struct cgroup *cgrp = root->cgroup; - raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); struct css_rstat_cpu *rstatc = css_rstat_cpu(root, cpu); struct cgroup_subsys_state *head = NULL, *parent, *child; unsigned long flags; - flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, false); + flags = _css_rstat_cpu_lock(root, cpu, false); /* Return NULL if this subtree is not on-list */ if (!rstatc->updated_next) @@ -254,7 +272,7 @@ static struct cgroup_subsys_state *css_rstat_updated_list( if (child != root) head = cgroup_rstat_push_children(head, child, cpu); unlock_ret: - _cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, false); + _css_rstat_cpu_unlock(root, cpu, flags, false); return head; } @@ -281,7 +299,7 @@ __weak noinline void bpf_rstat_flush(struct cgroup *cgrp, __bpf_hook_end(); /* - * Helper functions for locking cgroup_rstat_lock. + * Helper functions for locking. * * This makes it easier to diagnose locking issues and contention in * production environments. The parameter @cpu_in_loop indicate lock @@ -289,35 +307,44 @@ __bpf_hook_end(); * value -1 is used when obtaining the main lock else this is the CPU * number processed last. */ -static inline void __cgroup_rstat_lock(struct cgroup *cgrp, int cpu_in_loop) - __acquires(&cgroup_rstat_lock) +static inline void __css_rstat_lock(struct cgroup_subsys_state *css, + int cpu_in_loop) + __acquires(lock) { + struct cgroup *cgrp = css->cgroup; + spinlock_t *lock; bool contended; - contended = !spin_trylock_irq(&cgroup_rstat_lock); + lock = ss_rstat_lock(css->ss); + contended = !spin_trylock_irq(lock); if (contended) { trace_cgroup_rstat_lock_contended(cgrp, cpu_in_loop, contended); - spin_lock_irq(&cgroup_rstat_lock); + spin_lock_irq(lock); } trace_cgroup_rstat_locked(cgrp, cpu_in_loop, contended); } -static inline void __cgroup_rstat_unlock(struct cgroup *cgrp, int cpu_in_loop) - __releases(&cgroup_rstat_lock) +static inline void __css_rstat_unlock(struct cgroup_subsys_state *css, + int cpu_in_loop) + __releases(lock) { + struct cgroup *cgrp = css->cgroup; + spinlock_t *lock; + + lock = ss_rstat_lock(css->ss); trace_cgroup_rstat_unlock(cgrp, cpu_in_loop, false); - spin_unlock_irq(&cgroup_rstat_lock); + spin_unlock_irq(lock); } -/* see css_rstat_flush() */ +/* see css_rstat_flush() + * + * it is required that callers have previously acquired a lock via + * __css_rstat_lock(css) + */ static void css_rstat_flush_locked(struct cgroup_subsys_state *css) - __releases(&cgroup_rstat_lock) __acquires(&cgroup_rstat_lock) { - struct cgroup *cgrp = css->cgroup; int cpu; - lockdep_assert_held(&cgroup_rstat_lock); - for_each_possible_cpu(cpu) { struct cgroup_subsys_state *pos; @@ -332,11 +359,11 @@ static void css_rstat_flush_locked(struct cgroup_subsys_state *css) } /* play nice and yield if necessary */ - if (need_resched() || spin_needbreak(&cgroup_rstat_lock)) { - __cgroup_rstat_unlock(cgrp, cpu); + if (need_resched() || spin_needbreak(ss_rstat_lock(css->ss))) { + __css_rstat_unlock(css, cpu); if (!cond_resched()) cpu_relax(); - __cgroup_rstat_lock(cgrp, cpu); + __css_rstat_lock(css, cpu); } } } @@ -356,13 +383,10 @@ static void css_rstat_flush_locked(struct cgroup_subsys_state *css) */ void css_rstat_flush(struct cgroup_subsys_state *css) { - struct cgroup *cgrp = css->cgroup; - might_sleep(); - - __cgroup_rstat_lock(cgrp, -1); + __css_rstat_lock(css, -1); css_rstat_flush_locked(css); - __cgroup_rstat_unlock(cgrp, -1); + __css_rstat_unlock(css, -1); } __bpf_kfunc void bpf_cgroup_rstat_flush(struct cgroup *cgrp) @@ -381,10 +405,8 @@ __bpf_kfunc void bpf_cgroup_rstat_flush(struct cgroup *cgrp) */ void css_rstat_flush_hold(struct cgroup_subsys_state *css) { - struct cgroup *cgrp = css->cgroup; - might_sleep(); - __cgroup_rstat_lock(cgrp, -1); + __css_rstat_lock(css, -1); css_rstat_flush_locked(css); } @@ -394,8 +416,7 @@ void css_rstat_flush_hold(struct cgroup_subsys_state *css) */ void css_rstat_flush_release(struct cgroup_subsys_state *css) { - struct cgroup *cgrp = css->cgroup; - __cgroup_rstat_unlock(cgrp, -1); + __css_rstat_unlock(css, -1); } int css_rstat_init(struct cgroup_subsys_state *css) @@ -439,12 +460,36 @@ void css_rstat_exit(struct cgroup_subsys_state *css) css->rstat_cpu = NULL; } -void __init cgroup_rstat_boot(void) +/** + * ss_rstat_init - subsystem-specific rstat initialization + * @ss: target subsystem + * + * If @ss is NULL, the static locks associated with the base stats + * are initialized. If @ss is non-NULL, the subsystem-specific locks + * are initialized. + */ +int __init ss_rstat_init(struct cgroup_subsys *ss) { int cpu; + if (!ss) { + spin_lock_init(&rstat_base_lock); + + for_each_possible_cpu(cpu) + raw_spin_lock_init(per_cpu_ptr(&rstat_base_cpu_lock, cpu)); + + return 0; + } + + spin_lock_init(&ss->lock); + ss->percpu_lock = alloc_percpu(raw_spinlock_t); + if (!ss->percpu_lock) + return -ENOMEM; + for_each_possible_cpu(cpu) - raw_spin_lock_init(per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)); + raw_spin_lock_init(per_cpu_ptr(ss->percpu_lock, cpu)); + + return 0; } /*