From patchwork Tue Feb 18 03:14:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 13978856 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B0D9C021A9 for ; Tue, 18 Feb 2025 03:15:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8F72D2800BE; Mon, 17 Feb 2025 22:15:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A64C2800A4; Mon, 17 Feb 2025 22:15:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F92E2800BE; Mon, 17 Feb 2025 22:15:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4D3242800A4 for ; Mon, 17 Feb 2025 22:15:15 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 10ADF1C9940 for ; Tue, 18 Feb 2025 03:15:15 +0000 (UTC) X-FDA: 83131599390.18.A866CF0 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf25.hostedemail.com (Postfix) with ESMTP id 38FB8A000A for ; Tue, 18 Feb 2025 03:15:13 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YtkmVOkI; spf=pass (imf25.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739848513; a=rsa-sha256; cv=none; b=v0NufTtCJn+c8RWBKD1mtDy9FDnmJIRZCifqWj6coVFSw5IqkYxlq138VW9x+rToq3Tbhd LE0gYLCchhVa/S9mvL6SCZ6sCvbS/B8rwSk7EKnqtz/2QsDQ989jkPOanyPfKkXJiRQsmA vBBawr6cZXFxP5OOJ2nGbJc8wl8YFnk= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YtkmVOkI; spf=pass (imf25.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739848513; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i6fGNeZWtM3FGTNz/p/Bk400jphjU7NfeytH78W3bTI=; b=xi81XLDV5sTF5aFe5CgOn7KhPx5esSAIahGVeGYDEtOacYoFnRBZ/XEqe/MZz3uj9Tm60q erRyKA/Fxue8kmHdaykPErX8Bm4XBT0NzzR6N0cEhauL//c5a/14craSWGH5bA6NhyDBTl Zt3Xfuq6uRFhUvIDbR0m9E01HBznyS8= Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2fbfa8c73a6so9271966a91.2 for ; Mon, 17 Feb 2025 19:15:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739848512; x=1740453312; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=i6fGNeZWtM3FGTNz/p/Bk400jphjU7NfeytH78W3bTI=; b=YtkmVOkIOm6Xc46CHt7RoBm+sQo0VC4tdKjQB0BQ4AOOKUYK+NBGe9PpJWhdH0yfvM KGZ8z+t0f0/QnxJjxw1p6LToyD8/33Pxi0SP+uuAX+H0ZxzcINcTAWn4/wCq0EYXXJW4 mWuX0Zy8gxR1SlXarEy48NkeOvsCiEiWaccKakKrju4fnc95sO0dg+ej8Yuxq2UBx8p5 dGtm+viUhgqSRU5YkRW8eDOXlimlvVZNwMUjC8T93EUake+sbVyNL1Y9Hj+cEKkuXU98 KS39yGEOOE9UTKkUhGVvqMiRx7jAaEamuEbuC5XFpBaT1RKGNtIjBC818CIUekSS0FpW SG6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739848512; x=1740453312; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i6fGNeZWtM3FGTNz/p/Bk400jphjU7NfeytH78W3bTI=; b=tai6z1+n5zFMnfQdicO6UXY4jZf2XesSBgJwnGiA7pNLAQO9jeAXQUYgDH9JnFte5F vVfz6XEy8xeo0o+Do2353Hn6afxONWofe0rgpEJ53Knq/NJttfZo818PBeQvvJc+GjSF Aayrg/92FmUzZEimCNNqV+o7l09MyWF0CnGhlLIkZM3+6ZK6wBfTzooQW33Dq6w6965m uaM2XhlQKiYA90Tp8JvV7b1Vuc3l7d32yfswnzyfxAX6dyfx3wlWwVTI9+RD22wfXnKW PmsxxbqsTtKJTd7iV9B6S17zhvq7Oer7FTbQmV2LPtqJsr/Rz/QzkbI5A4nOnsPIWLhz zukQ== X-Gm-Message-State: AOJu0YyWqD7kO6sW2TVx5LKZd7KaH2XHEDUa4BGGUWwaJ19xr6ui6Ha+ IwFrS/ZXZqwz3PqTw4oubl3ASUMd3kE8HjSq2lnhcT0gMTO4HT3x X-Gm-Gg: ASbGnctzNEEmqtzvWEn+OBnZYcTkocS3rjWQ1OxAG0cEHLfCU7Fmh/2sIl6HvouF15L a+CcI/YtVg2ceVIpeeUeJhXk9UaPGvrZXfB9M31iEnUOZ05lRo6acJAiO80/EukywK/6UDJ50tT 3AWX18wNc//OTo+BGSx9xv+TykOUD8KUJaQLPjecSyB8/6T4qsSx9VDMTt1SwLCot1TwpIpwjy6 GBUD+CjZeMJFU1CuOOXx1GxobN8Ifayz/wcziohgZkpdRo2TkDRPRToMbRrC6Q5eScJ1avVEMkr rvy+mf3cyqfIzy77GrYox4t4GGjtHioRTDW/8pOe29PHNvK/6Is6 X-Google-Smtp-Source: AGHT+IEdc2CfR68/O4SRSN2xNZsygVxwsIe5F71fhvH7zgtYBNKyQ2HCTgHwt2ieGZ7qbMGNDozeDQ== X-Received: by 2002:a05:6a20:4323:b0:1ee:83b8:7c6b with SMTP id adf61e73a8af0-1ee8cc084demr18831160637.39.1739848512098; Mon, 17 Feb 2025 19:15:12 -0800 (PST) Received: from saturn.. (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732425466besm8763451b3a.9.2025.02.17.19.15.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Feb 2025 19:15:11 -0800 (PST) From: JP Kobryn To: shakeel.butt@linux.dev, tj@kernel.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 07/11] cgroup: fetch cpu-specific lock in rstat cpu lock helpers Date: Mon, 17 Feb 2025 19:14:44 -0800 Message-ID: <20250218031448.46951-8-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250218031448.46951-1-inwardvessel@gmail.com> References: <20250218031448.46951-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 38FB8A000A X-Stat-Signature: 97kf9exndayoztdqqd6y19a3fywy9gs1 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1739848513-990169 X-HE-Meta: U2FsdGVkX18dmBaMJdjPXHo1n3mm5Dkv+SZ6FMOlKxENnB0zU6tl1oVRw7qCrEMxq7kIxKVX8l8koP2mtkyz41Jv7JAhZs5x90w93uDOG8ebJ3hrqjsRSbXioDBCgd96Zd2LgynKOMAADwPOMio1cbo4zaR5VRE/6KI8IpUv0eAsuOd+NPwl5JdixZvLX89ELv+u2Oxn2Z4ekmtcHOduJkY1gdT+09BVRDng4OsDVkkzdhPBqDbeCGbch1abzz0vk2F666djRvbO7lfjHlepTFmwKRJD+3Ii8zHIuLDDBbTwr+lP0iAqzQkL1g+xd4d6Hv1L3MkT3T14XxcWcfOTUdWEZT/BgQuYb2wUGHJVzO84RYAtbW3I/lAboHDKAIdRR8nZWgLmzUYdvdcga7OyLnzx8kmCgiK9MnURX2jBD0UUD1RlIPxsFl18IW2iF3y0S6mYV37yVGe12Q0OUQR9Gck64l5GZoypRaiA4zDB1fwG+PRdhHoVUBzN/znoWdI+7K7NEDy4Q6rPCvzIV4YIWnW5jSFrqV++DixgYmHWLBanic4DhQbGb/IGUYehr5hBzugJLTrPqzSEdtrGHkckBwtaGkHfZiFoMI5aDKz9VBMdR08+NOCklSIfBlLUGTYzXYQmMUBf/801GcEiHDFt4kXeU278BdS8Wq57QTSsTW8S9Ya+tiYdjxVeI5l5feKr9pXN5IZcE1NSG3PdeUMR906qqcHIBHKkL8v2V5QdX0xOWm/Ck3jwO6bn1wbj5o9FbtUUAmnv8CMGMMxUOB6UGr0VWXGq8qCXBEtGhEIGq7wmv9vvCalZ6j1OytnbBR3un+K+9nQESJZ315Lxbht78AJbCy6oASAppAe/iXcZ3XKNC64p4DRTLa2KLYbfUqg/hfgptq58LL+DV/EJzYKl66hmubJBqx7z9/mO6BEQunGutk9lYIg/iprNjr7mmerB0XgjKl8EEcUYvUK6w3s E/3+fd52 Pm+Hr59jWL+DK7Z/s4IIzUyZmWGXwEl4VPD1XNM7trtAVa845Qfigxqb5Ff3ATvyPAFvPJGAhonMbrf8tiNaHAbJvyz/UKXTVuMjLDTZ86QWvBd6LhThrc5PdScOlV8tjnEY2bskFV1wgyEgz7grbrYAlFnwoS/0UGuIyjWkYdRSwFPfAzrc3f97Ty2drTEe3d+bjt3Uo4hcyIHQLay9MJ7h/6KvcYZ/7/CjxSj6he+HuD2TDOf3SSf90JA+4U8/Xz3xEtHwaI6K3M4QNS+UjrWwT+/beJh7MwfR/qyhp344P2sBQ2UGa7WwPZ0ZH5Z/uR70Bx4mntdT4pmfGGTE5jG2A8bNr34XnDLSNelh/Ja/l0YbSO7/xAbAHKQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.006186, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The lock/unlock helper functions for per-cpu locks accept a cpu argument. This makes them appear as if the cpu will be used as the offset off of the base per-cpu pointer. But in fact, the cpu is only used as a tracepoint argument. Change the functions so that the cpu is also used primarily for looking up the lock specific to this cpu. This means the call sites can be adjusted to not have to perform the offset prior to calling this function. Note that this follows suit with other functions in the rstat source - functions that accept a cpu argument perform the per-cpu pointer lookup within as opposed to having clients lookup in advance. Signed-off-by: JP Kobryn Reviewed-by: Shakeel Butt --- kernel/cgroup/rstat.c | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 26c75629bca2..4cb0f3ffc1db 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -115,7 +115,12 @@ static struct cgroup_rstat_ops rstat_bpf_ops = { #endif /* CONFIG_CGROUP_BPF */ /* - * Helper functions for rstat per CPU lock (cgroup_rstat_cpu_lock). + * Helper functions for rstat per-cpu locks. + * @lock: pointer to per-cpu lock variable + * @cpu: the cpu to use for getting the cpu-specific lock + * @cgrp: the associated cgroup + * @fast_path: whether this function is called while updating + * in the fast path or flushing in the NON-fast path * * This makes it easier to diagnose locking issues and contention in * production environments. The parameter @fast_path determine the @@ -123,19 +128,20 @@ static struct cgroup_rstat_ops rstat_bpf_ops = { * operations without handling high-frequency fast-path "update" events. */ static __always_inline -unsigned long _cgroup_rstat_cpu_lock(raw_spinlock_t *cpu_lock, int cpu, +unsigned long _cgroup_rstat_cpu_lock(raw_spinlock_t *lock, int cpu, struct cgroup *cgrp, const bool fast_path) { + raw_spinlock_t *cpu_lock = per_cpu_ptr(lock, cpu); unsigned long flags; bool contended; /* - * The _irqsave() is needed because cgroup_rstat_lock is - * spinlock_t which is a sleeping lock on PREEMPT_RT. Acquiring - * this lock with the _irq() suffix only disables interrupts on - * a non-PREEMPT_RT kernel. The raw_spinlock_t below disables - * interrupts on both configurations. The _irqsave() ensures - * that interrupts are always disabled and later restored. + * The _irqsave() is needed because the locks used for flushing + * are spinlock_t which is a sleeping lock on PREEMPT_RT. + * Acquiring this lock with the _irq() suffix only disables + * interrupts on a non-PREEMPT_RT kernel. The raw_spinlock_t below + * disables interrupts on both configurations. The _irqsave() + * ensures that interrupts are always disabled and later restored. */ contended = !raw_spin_trylock_irqsave(cpu_lock, flags); if (contended) { @@ -156,10 +162,12 @@ unsigned long _cgroup_rstat_cpu_lock(raw_spinlock_t *cpu_lock, int cpu, } static __always_inline -void _cgroup_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, +void _cgroup_rstat_cpu_unlock(raw_spinlock_t *lock, int cpu, struct cgroup *cgrp, unsigned long flags, const bool fast_path) { + raw_spinlock_t *cpu_lock = per_cpu_ptr(lock, cpu); + if (fast_path) trace_cgroup_rstat_cpu_unlock_fastpath(cgrp, cpu, false); else @@ -172,8 +180,6 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu, struct cgroup_rstat_ops *ops) { struct cgroup *cgrp; - - raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); unsigned long flags; /* @@ -188,7 +194,7 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu, return; cgrp = ops->cgroup_fn(rstat); - flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, true); + flags = _cgroup_rstat_cpu_lock(&cgroup_rstat_cpu_lock, cpu, cgrp, true); /* put @rstat and all ancestors on the corresponding updated lists */ while (true) { @@ -216,7 +222,7 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu, rstat = parent; } - _cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, true); + _cgroup_rstat_cpu_unlock(&cgroup_rstat_cpu_lock, cpu, cgrp, flags, true); } /** @@ -315,14 +321,13 @@ static struct cgroup_rstat *cgroup_rstat_push_children( static struct cgroup_rstat *cgroup_rstat_updated_list( struct cgroup_rstat *root, int cpu, struct cgroup_rstat_ops *ops) { - raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); struct cgroup_rstat_cpu *rstatc = rstat_cpu(root, cpu); struct cgroup_rstat *head = NULL, *parent, *child; struct cgroup *cgrp; unsigned long flags; cgrp = ops->cgroup_fn(root); - flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, false); + flags = _cgroup_rstat_cpu_lock(&cgroup_rstat_cpu_lock, cpu, cgrp, false); /* Return NULL if this subtree is not on-list */ if (!rstatc->updated_next) @@ -359,7 +364,7 @@ static struct cgroup_rstat *cgroup_rstat_updated_list( if (child != root) head = cgroup_rstat_push_children(head, child, cpu, ops); unlock_ret: - _cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, false); + _cgroup_rstat_cpu_unlock(&cgroup_rstat_cpu_lock, cpu, cgrp, flags, false); return head; }