From patchwork Tue Feb 18 03:14:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: JP Kobryn X-Patchwork-Id: 13978859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEB80C021A9 for ; Tue, 18 Feb 2025 03:15:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01CF52800C2; Mon, 17 Feb 2025 22:15:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F0FF02800A4; Mon, 17 Feb 2025 22:15:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D885D2800C2; Mon, 17 Feb 2025 22:15:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B67322800A4 for ; Mon, 17 Feb 2025 22:15:20 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7611CC1819 for ; Tue, 18 Feb 2025 03:15:20 +0000 (UTC) X-FDA: 83131599600.14.8115F49 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf22.hostedemail.com (Postfix) with ESMTP id A1410C0002 for ; Tue, 18 Feb 2025 03:15:18 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JdCBwfNG; spf=pass (imf22.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739848518; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vg+1tuAH5Cml38p4hzH7HNleCRescAO8xBmjt4Ynbfc=; b=ab84Yht9d3vLNOVbu9J4pz3dm+U2HqYDc8gib4dWom9//BOWHpk3tM15tZM/RCzZLTVmlX +p3U8gxLjfIClHLWh/pp7uBrJFGhLnVrUhkaDf6nYUNcjzf1VyHDvHm5zDVmMpP6g5IljW k8+Jm4Sb1svBDMLhyLUhUCfK1SFgclQ= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JdCBwfNG; spf=pass (imf22.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739848518; a=rsa-sha256; cv=none; b=6sl0SOwrQHOAY6/SBvjR28S84l348CZXVaCb9RqXLK6eDaiXTgumCPwC9Lwz6x7wTgWTo3 CLEVMpHtDQtbJ9mNnWViSjpv+rnDDSEGZxidXMv4hh/xqmippvKHJ//9CfZ1atnz0R0B4P mYgn6ENHk1P/T0LYyN5iraeY+THkUqw= Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-220c8cf98bbso99969755ad.1 for ; Mon, 17 Feb 2025 19:15:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739848517; x=1740453317; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vg+1tuAH5Cml38p4hzH7HNleCRescAO8xBmjt4Ynbfc=; b=JdCBwfNGsJ9FY/I9be92wLkmNWJfdv3S0PPTDyHRvv1D9KEXngxCUNtyVwcHVp1bQK FrtbOZNnPabnzNTuVdQUjOTuPSGRfYLKFwjbfHNDwtHgvid+AoATf+siXqWaoU+/KS4K DM2ZPTmRSxybgGnbflwY0FyM1zWbJrXx5u8FOwP0iZx7Rt4NILf05NhcWs9jB8Uo4DzI 58dfnQUzvLFNMPTWCgmVzaw+uErjvV6dMrVL4z2r9pYKg/h8EQD6PeNRAyZOVXNS84S1 HFc4lD+4PxEy5zTg3RXlbD/TsPHIQAi+1X+hQaMSBm9AKH4SATpxfYBMFYQ7BNlFTaID B6+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739848517; x=1740453317; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vg+1tuAH5Cml38p4hzH7HNleCRescAO8xBmjt4Ynbfc=; b=Id95Iq/SCkJDyk6UggAyCmgyAz/XpIB1ak3mXXXjEH62YB+kt7IdO+64Fkih9Stjqr PDsiqFHmqx0DC1wfdhXwYk1Q51k/xTtPQ7llPu+PKKqgayIy8/agXQlheVtq3OyTrDCk 9GDXjy2UC4dBRF24WETFmq1fRE9JsiJSWlu8L2EFq0XGduWSn2WikmS2HCbjBSBcHU8P /dnnLwQQ1SXUdPG5YismzQ5ShMz8sj0xWMNdOGelYhA02Sb4SM1b5TbcCJ0h9Vqoooo4 UP1Fq5WHiX82C3i6LbcCpECzBmNzxoW+S9hKoReH4ga41TC2p99aslqbvGH09cKIEwmv BSvQ== X-Gm-Message-State: AOJu0YxzHgb5RSSpoGA5n8VM3TU59B2u5yNFNjhxhBFzFZLfnCLI8Gy7 oFrLN/4MTAxVRSk73+gazxCG+YtMgqK5BJataHPs15TjVimw9vQR X-Gm-Gg: ASbGnctmSYoCA4Gz0+sp4950qq/wcAi8RKNE2CgMB3aWYV99nIQCoHENcHXHfzDP+cg W+79Sth6+wfckdita7JjmEjM6DFUh/wHd2E5cCyBkdmT73pdBWJyWzdiEoF7F3YrRDTdjelFUXl Q5mPApnMSI24sJC7VU+OvMnW1okkf7H2Aj1oppnaFTVxUrwOriwm1B1vGtpRs4kmmks5q120ZOd TXx5CtP+7hsuZPY8lUMgskMYvZPOzIqjUipX7A8kJqRW04apMwba5Ev6xL/E5rgzirdwelq/oXf 7O9yU+F6dBpTxV9Ls3zYQnWKZLfua4ObgfjVAvJBceVd49hqgISf X-Google-Smtp-Source: AGHT+IGi97sy9lc8oEV2Lvmu4nX/Oz5UjlOTjclhhjOUsPwyOtWg+KBN0Ea6IXWS4MZ0ObmmEo85Dg== X-Received: by 2002:a05:6a20:244a:b0:1ed:a6d8:3439 with SMTP id adf61e73a8af0-1ee8cb7dfd1mr22632952637.22.1739848517528; Mon, 17 Feb 2025 19:15:17 -0800 (PST) Received: from saturn.. (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732425466besm8763451b3a.9.2025.02.17.19.15.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Feb 2025 19:15:17 -0800 (PST) From: JP Kobryn To: shakeel.butt@linux.dev, tj@kernel.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 10/11] cgroup: separate rstat locks for subsystems Date: Mon, 17 Feb 2025 19:14:47 -0800 Message-ID: <20250218031448.46951-11-inwardvessel@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250218031448.46951-1-inwardvessel@gmail.com> References: <20250218031448.46951-1-inwardvessel@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A1410C0002 X-Stat-Signature: huk9q9m3m1xcrj3wqwhmr8baf4obr656 X-Rspam-User: X-HE-Tag: 1739848518-534619 X-HE-Meta: U2FsdGVkX1//zBZWGEr6xWuiyhJbAlbm/J7bA4YCTSV2TthKonEwcyE+LP7iDKsKFYFtt2Fm7FufMMp+ocOhoQvYZRYfv6k7eEJ47i7TxYLzpHB3YvaK+xOVmP6i2tK7uDJrgXnHHWk6GAfReCqJA6k/DY41GrtaywGmJx/mxSgP9GpMnNIf357w0eFUmUtm9fkUXOpg1VatumuC1y+b6KMYfjDBaub3bqT8vcu3bn/DVJ+fjsXx5JcESX7wtVkZ2qlrBUrz+ufWeNWmo0o52wCvE/rKjrSc+XrrJRZoekz3mc3xw1WQS9CpUAJYgtvw+r7FyayRXF0CHEC4LMgQKFElSqPgmg99DqpkGUqcdZtZI7bvc1LxgZeDwTGPpBRfokAB+gTd4D61QiqBwJTg8iYWcdxSTv8Qa6Atg8A0lp5fOjivCcqYEH2zlLTlSUpnBip0rZQX3BAc1Pq+0tCj/net3TDJ/b7D7UsfzP4dCOMK/NNc0ybAUiS0JZ6TuxQTIxRekJtNwTbzR+05TYahBcS6Xfz2byItFCUvTilPQTAVBM3icQCxeSXzV03FMBv3RX+3tUI2Tc5QYeImEpUOTVw0G8zyqRz+BPcW3DIeqsyqhNpX1tbQwNbf6BKTyVtAyVI9rhahVy4vqAGeZyjSO2JMrQeyVT+l/8NCfzGe/by1PpZ3Jk1Q9mWQy5LbMyIrfaSJpZ9ZGvSKd8IfQJ2c8IYCCb3sNB5V8FcfsHVpuSCNRD+RIrd/MwrvaV9uHwCPipvDvcQMx0sAA5qiM7Dbkm6OQnP2iTGXS0sJnnHl0ELcRgeVE0hIW938CS/cXfMXyavA/ucWqOOY2QVCjT+6E/N8eRJZ+fLUUJpm1FirpKcIw7tvaz/JzUEQzBBQnn7mSOIFDkbvIQ9lEkZYH2tiBk1rGPIYO3ByR52Mxf/AA6GttlCnAiuhYxGBBbZ5m7VY9pF1OFYjVQlxIi99kvq GFXHzLDg mmNe6yoyF8xTUvsXla/WmsOVo6bwrzGe/QsiXkI6BDq8UqzEmtaqPyYpk/o2CIFbjzImtkAVxze5nVd7HJhR1NQDt4wnKLmm7DTrM3KztyiHbqtXdNWceBltoSh3hZ/id4lbm5g5V+m7/dKS6wr7OI/Fxz6xbf2iecHkPPcgz2VDw19dQqqbM+eKCaiEwN+WmN3mI9HRapEWfiuhRoQSpS69IvKvWjcSQnQCwBD+iDP108DXaftv7bjQMB/MGmrcVODKFfmQz2mYdbAZdwzE3SnOupSBoiJ8ieJ0Wg8OboPTz3vKxJFez3xBkwQzyVrvPzJfUI9rOVbFHdLqRGSMteosUSAK/euSBFwdf+F9To+gig8poG/rr7r3Y6g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add new rstat locks for each subsystem. When handling cgroup subsystem states, distinguish between states associated with formal subsystems (memory, io, etc) and the base stats subsystem state (represented by cgroup::self). This change is made to prevent contention when updating/flushing stats. Signed-off-by: JP Kobryn --- kernel/cgroup/rstat.c | 68 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 58 insertions(+), 10 deletions(-) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 7d9abfd644ca..93b97bddec9c 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -15,8 +15,11 @@ struct cgroup_rstat_ops { void (*flush_fn)(struct cgroup_rstat *, int); }; -static DEFINE_SPINLOCK(cgroup_rstat_lock); -static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_cpu_lock); +static DEFINE_SPINLOCK(cgroup_rstat_base_lock); +static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_base_cpu_lock); + +static spinlock_t cgroup_rstat_subsys_lock[CGROUP_SUBSYS_COUNT]; +static raw_spinlock_t __percpu cgroup_rstat_subsys_cpu_lock[CGROUP_SUBSYS_COUNT]; #ifdef CONFIG_CGROUP_BPF static DEFINE_SPINLOCK(cgroup_rstat_bpf_lock); @@ -241,8 +244,14 @@ static void __cgroup_rstat_updated(struct cgroup_rstat *rstat, int cpu, */ void cgroup_rstat_updated(struct cgroup_subsys_state *css, int cpu) { - __cgroup_rstat_updated(&css->rstat, cpu, &rstat_css_ops, - &cgroup_rstat_cpu_lock); + raw_spinlock_t *cpu_lock; + + if (is_base_css(css)) + cpu_lock = &cgroup_rstat_base_cpu_lock; + else + cpu_lock = &cgroup_rstat_subsys_cpu_lock[css->ss->id]; + + __cgroup_rstat_updated(&css->rstat, cpu, &rstat_css_ops, cpu_lock); } #ifdef CONFIG_CGROUP_BPF @@ -487,8 +496,19 @@ static void __cgroup_rstat_flush(struct cgroup_rstat *rstat, */ void cgroup_rstat_flush(struct cgroup_subsys_state *css) { + spinlock_t *lock; + raw_spinlock_t *cpu_lock; + + if (is_base_css(css)) { + lock = &cgroup_rstat_base_lock; + cpu_lock = &cgroup_rstat_base_cpu_lock; + } else { + lock = &cgroup_rstat_subsys_lock[css->ss->id]; + cpu_lock = &cgroup_rstat_subsys_cpu_lock[css->ss->id]; + } + __cgroup_rstat_flush(&css->rstat, &rstat_css_ops, - &cgroup_rstat_lock, &cgroup_rstat_cpu_lock); + lock, cpu_lock); } #ifdef CONFIG_CGROUP_BPF @@ -523,8 +543,19 @@ static void __cgroup_rstat_flush_hold(struct cgroup_rstat *rstat, */ void cgroup_rstat_flush_hold(struct cgroup_subsys_state *css) { + spinlock_t *lock; + raw_spinlock_t *cpu_lock; + + if (is_base_css(css)) { + lock = &cgroup_rstat_base_lock; + cpu_lock = &cgroup_rstat_base_cpu_lock; + } else { + lock = &cgroup_rstat_subsys_lock[css->ss->id]; + cpu_lock = &cgroup_rstat_subsys_cpu_lock[css->ss->id]; + } + __cgroup_rstat_flush_hold(&css->rstat, &rstat_css_ops, - &cgroup_rstat_lock, &cgroup_rstat_cpu_lock); + lock, cpu_lock); } /** @@ -547,8 +578,14 @@ static void __cgroup_rstat_flush_release(struct cgroup_rstat *rstat, */ void cgroup_rstat_flush_release(struct cgroup_subsys_state *css) { - __cgroup_rstat_flush_release(&css->rstat, &rstat_css_ops, - &cgroup_rstat_lock); + spinlock_t *lock; + + if (is_base_css(css)) + lock = &cgroup_rstat_base_lock; + else + lock = &cgroup_rstat_subsys_lock[css->ss->id]; + + __cgroup_rstat_flush_release(&css->rstat, &rstat_css_ops, lock); } static void __cgroup_rstat_init(struct cgroup_rstat *rstat) @@ -629,10 +666,21 @@ void bpf_cgroup_rstat_exit(struct cgroup_bpf *bpf) void __init cgroup_rstat_boot(void) { - int cpu; + struct cgroup_subsys *ss; + int cpu, ssid; + + for_each_subsys(ss, ssid) { + spin_lock_init(&cgroup_rstat_subsys_lock[ssid]); + + for_each_possible_cpu(cpu) { + raw_spinlock_t *cpu_lock = + per_cpu_ptr(&cgroup_rstat_subsys_cpu_lock[ssid], cpu); + raw_spin_lock_init(cpu_lock); + } + } for_each_possible_cpu(cpu) { - raw_spin_lock_init(per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)); + raw_spin_lock_init(per_cpu_ptr(&cgroup_rstat_base_cpu_lock, cpu)); #ifdef CONFIG_CGROUP_BPF raw_spin_lock_init(per_cpu_ptr(&cgroup_rstat_bpf_cpu_lock, cpu));