From patchwork Tue Aug 22 18:41:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mateusz Guzik X-Patchwork-Id: 13361323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06D2DEE49B1 for ; Tue, 22 Aug 2023 18:42:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E0BF280057; Tue, 22 Aug 2023 14:42:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 790C4280040; Tue, 22 Aug 2023 14:42:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63134280057; Tue, 22 Aug 2023 14:42:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 52F89280040 for ; Tue, 22 Aug 2023 14:42:03 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2132A1C8B81 for ; Tue, 22 Aug 2023 18:42:03 +0000 (UTC) X-FDA: 81152610126.29.3B4FBBF Received: from mail-lf1-f43.google.com (mail-lf1-f43.google.com [209.85.167.43]) by imf08.hostedemail.com (Postfix) with ESMTP id 08DE316000D for ; Tue, 22 Aug 2023 18:42:00 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b="IpUdBzg/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.167.43 as permitted sender) smtp.mailfrom=mjguzik@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692729721; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZcKf8TCfmSL/jTvJtJ1dXOWDhnfVF83USQ3Eyo19kik=; b=BI1ySbZqw5ys/Gz4T0hoCN3/BKPBf34t3E23bqqubNkXjLNgQM2d52EvzlJ55JjhjgFfEt wIBwnl4kTVxQcLK+ZGltaRrOpVD+0KXlePCaGaDo9oiqR6MsFYqEn1wKA0ccF/eIoHW4pr kWPSLgSu0KzYi/Cfn9KY0N19J24bqkA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b="IpUdBzg/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.167.43 as permitted sender) smtp.mailfrom=mjguzik@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692729721; a=rsa-sha256; cv=none; b=yU/1yaEcv2OzWNZIS0CgCBWuJ6EvBVoYWwfhXUtlvMomPW6rOVsxSgdcpqmcFseJHIJB+Y I+pZvf0n7gleh5n9YZSdXTCP5cjmnmNlBLl6qF+LQhokj7imFXAxsCl2xaz5hSJsKSRRax O7MTtcxcJwNpsKmZh/e0UivUJrS5X5c= Received: by mail-lf1-f43.google.com with SMTP id 2adb3069b0e04-4ff8a1746e0so7378601e87.0 for ; Tue, 22 Aug 2023 11:42:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692729719; x=1693334519; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZcKf8TCfmSL/jTvJtJ1dXOWDhnfVF83USQ3Eyo19kik=; b=IpUdBzg/DdaYRMpbl13X84cDQiYDpmUCgeXaiBqeMbuJoHM1oXSNizZuih3VSwvYMl aZDcfvFa0yOhVkBUIFUaneixc6ZJOUXIfss7UU5ctP6d6wxplb/U69CS6fbvW4JWED1U cGtKnSfUOCEe/NmDiPVOE65upZgDhxf2tK9WwygoOomfip5wkdzE8lSCrdiZWcfowj0d BoM8ULxp4AnpR74B5yg7idNT4bwx7b7DBgaLziZSk3R5i4B8YZHLUjcGja68LIRiEvMf 8HqcuccQTe/izpkHLUmtgmGy7DU4obCcA1WIkptReUG/nQjmH/0FpxoTSIlOZchMrU/D rcCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692729719; x=1693334519; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZcKf8TCfmSL/jTvJtJ1dXOWDhnfVF83USQ3Eyo19kik=; b=g/2vnD5D9GUwOnDpSeZI34lHlhUoUx0zXlK4Y31eIsAssuce9YIF6uCDsQBnawVc+r 6+p1mTaxnBA3/+kxPOBv2ztS+uOz4KZUfX/T6Tt4lQqljaaM5Ay2K1h4NJjfUasXxHmm 0wymO53iy9g4FXIMB9e+UGzgjdXdc+hlP46rz577H+ThBvTP8BezJxOFocVfHhfHSzCX ddd1qZA58+pCIs7gSJj8qTrWpImx0xnalZ66YC1LmSEWshpc643HtzZIWuNfx49ecsMw Oc4DdT7BVpcE0IcdHBvwm7D0HtYDiC9DoubHwMcKIh5tePOmAOxovn0UCjs9h/KMAEbj YE+g== X-Gm-Message-State: AOJu0YxIVg7bIo3RmOl8iTXCN58GknwcDYYxAXAX1oxpSGvIVsB+jWSf wzhmtR/88VfDPI9tQcK7fwQ= X-Google-Smtp-Source: AGHT+IFwNeb+x8+h4MA5Q285B17LTQ1gswDV1LNkxSm1vOHyvMZAe4lJw/QPCraL5VibXY/m5vFGcQ== X-Received: by 2002:a05:6512:1296:b0:4fe:993:2218 with SMTP id u22-20020a056512129600b004fe09932218mr9935537lfs.31.1692729718947; Tue, 22 Aug 2023 11:41:58 -0700 (PDT) Received: from f.. (cst-prg-85-121.cust.vodafone.cz. [46.135.85.121]) by smtp.gmail.com with ESMTPSA id w9-20020a056402070900b00525683f9b2fsm7945317edx.5.2023.08.22.11.41.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Aug 2023 11:41:58 -0700 (PDT) From: Mateusz Guzik To: linux-kernel@vger.kernel.org Cc: dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, linux-mm@kvack.org, Mateusz Guzik Subject: [PATCH v2 1/2] pcpcntr: add group allocation/free Date: Tue, 22 Aug 2023 20:41:51 +0200 Message-Id: <20230822184152.2194558-2-mjguzik@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230822184152.2194558-1-mjguzik@gmail.com> References: <20230822184152.2194558-1-mjguzik@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 08DE316000D X-Stat-Signature: 1nhmozq1kyxigad7zrsduq8xumjsu5i3 X-Rspam-User: X-HE-Tag: 1692729720-928837 X-HE-Meta: U2FsdGVkX19YTjTDcTzV0NNY41pPKJGcSEnstuBLzWbqoR3JwmToOaIHZiXeOvon9C+J4Klru8iL5u6TMudT/6uBe0uwoWvwBp+bWWtW4yXheBUnE38n55ek02fx5nOdw8vZMT2vHdMMzvy4HwYvtJoYid2uV23zNAU3vhe+mWvoafavoR8MlWCf71DMFoElqJ418AErxPOWr4iGFNQH+iff/WeY6LFH/qJVkPvr5HJLM6bHVLxeI6fR9aFEA/Hk6L6jNiULZAlBhB3roD4Ow8DsRK6SynbLY2Vm+oA9Gp+DgbkdkIk3jYaCS82G7TqgSta1EsI/DvWGGg72ngDYbvM9iq863ZF9f3BMpNoT1vzotFwvYK6ri0j19qboUQSJe0H5sEsZBL5aFKSzclEQytVc80YB8REUguCXhqPzVM1lGjJdrG35YA7yNlc4toTkaydCouAbwVyM7zbM8dgPk3lkc/izCI/SkP8jUVkQOrpC0AmHSXht7ITsIL69t7mzKUdXwHmNhTmQnFZ6Y14+fv3OO+STxVkosjl6+PGjOYf62Z1HPzxSRKM3Qs2x1lUwAxsgi3iIJztX1btpUCnjvmi1GHYgZWCpkMal7wsYCSJDo8wSHNTJnfGiklVYhYNh0TTYm+9HWyukOGUi5bxYD3Fapd2CVdrtj2JKiAuNaO+icLKnnre0nu8L+LHhaiQqkMwxSoCg56an0TCIiDXOK7oC3Zag9CXhrr8uYq+9vqf6f0xDKNzLbW140F+R1yf8DNUk8TL0aVH1rLZgwR+YH0NTTrMagmu0KIaW2Kht+aod6ZwloEMllHQXiykoualGDBy6ZczBGAQUzYYnsVsp4S/XkRP3pMfJyqLdcDz31gZoNIPcA0eUWlaBDj6Q/KYn/nw7rJoo0DxkxzgRiKNf0Xy6UZnTH5cjFcgX8gmrWvdDrLGWPApv6Ie76+9qk0d3boW3WflkckeoeJzH0Q6 Reb84k2E 50V0Pf8UiozGC97uZdPLFAq3N003fzcwkm++VzsslzP9Al/jddlJlI4m4coaUJZVacR8ndsCDDhUpZLThtmHLmCVPIVe1V5ddvzFoEFjMzPVuENwnJ7Ty9fjcQXzFdwbhPfhwTJrKqp+NFkKEwJg4aOJXkyTWuaPtn4jdFAks78WYZSArXD3NEAdxF5rKySLEKxnJpyM6ouSAOPIm0NKhFrJLiWNKTTmClzAbG4cDcUQvhCYsWNfAQtfJlQXVXHtKVyewbh0Nt5lkXZzXm2/sgxYqFkdboPcyb2RCS8QZMR9sZRsKK9lImxSpJEVckxQM8QQp7DV/WuGDxb3aD8yiak7dRWFanpe7ylGX+I3J5IfsU1WgNc9ViSXRrNxnN743Dn4JCsBhnccRRqoW3bdd3K4lmkO+lVGdp6Jy1Nhkm3zrevKgfLlwAxsBfyuv9jSme/6d X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocations and frees are globally serialized on the pcpu lock (and the CPU hotplug lock if enabled, which is the case on Debian). At least one frequent consumer allocates 4 back-to-back counters (and frees them in the same manner), exacerbating the problem. While this does not fully remedy scalability issues, it is a step towards that goal and provides immediate relief. Signed-off-by: Mateusz Guzik --- include/linux/percpu_counter.h | 20 ++++++++--- lib/percpu_counter.c | 61 +++++++++++++++++++++++----------- 2 files changed, 57 insertions(+), 24 deletions(-) diff --git a/include/linux/percpu_counter.h b/include/linux/percpu_counter.h index 75b73c83bc9d..518a4088b964 100644 --- a/include/linux/percpu_counter.h +++ b/include/linux/percpu_counter.h @@ -30,17 +30,27 @@ struct percpu_counter { extern int percpu_counter_batch; -int __percpu_counter_init(struct percpu_counter *fbc, s64 amount, gfp_t gfp, - struct lock_class_key *key); +int __percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, gfp_t gfp, + u32 nr_counters, struct lock_class_key *key); -#define percpu_counter_init(fbc, value, gfp) \ +#define percpu_counter_init_many(fbc, value, gfp, nr_counters) \ ({ \ static struct lock_class_key __key; \ \ - __percpu_counter_init(fbc, value, gfp, &__key); \ + __percpu_counter_init_many(fbc, value, gfp, nr_counters,\ + &__key); \ }) -void percpu_counter_destroy(struct percpu_counter *fbc); + +#define percpu_counter_init(fbc, value, gfp) \ + percpu_counter_init_many(fbc, value, gfp, 1) + +void percpu_counter_destroy_many(struct percpu_counter *fbc, u32 nr_counters); +static inline void percpu_counter_destroy(struct percpu_counter *fbc) +{ + percpu_counter_destroy_many(fbc, 1); +} + void percpu_counter_set(struct percpu_counter *fbc, s64 amount); void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount, s32 batch); diff --git a/lib/percpu_counter.c b/lib/percpu_counter.c index 5004463c4f9f..9338b27f1cdd 100644 --- a/lib/percpu_counter.c +++ b/lib/percpu_counter.c @@ -151,48 +151,71 @@ s64 __percpu_counter_sum(struct percpu_counter *fbc) } EXPORT_SYMBOL(__percpu_counter_sum); -int __percpu_counter_init(struct percpu_counter *fbc, s64 amount, gfp_t gfp, - struct lock_class_key *key) +int __percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, gfp_t gfp, + u32 nr_counters, struct lock_class_key *key) { unsigned long flags __maybe_unused; - - raw_spin_lock_init(&fbc->lock); - lockdep_set_class(&fbc->lock, key); - fbc->count = amount; - fbc->counters = alloc_percpu_gfp(s32, gfp); - if (!fbc->counters) + size_t counter_size; + s32 __percpu *counters; + u32 i; + + counter_size = ALIGN(sizeof(*counters), __alignof__(*counters)); + counters = __alloc_percpu_gfp(nr_counters * counter_size, + __alignof__(*counters), gfp); + if (!counters) { + fbc[0].counters = NULL; return -ENOMEM; + } - debug_percpu_counter_activate(fbc); + for (i = 0; i < nr_counters; i++) { + raw_spin_lock_init(&fbc[i].lock); + lockdep_set_class(&fbc[i].lock, key); +#ifdef CONFIG_HOTPLUG_CPU + INIT_LIST_HEAD(&fbc[i].list); +#endif + fbc[i].count = amount; + fbc[i].counters = (void *)counters + (i * counter_size); + + debug_percpu_counter_activate(&fbc[i]); + } #ifdef CONFIG_HOTPLUG_CPU - INIT_LIST_HEAD(&fbc->list); spin_lock_irqsave(&percpu_counters_lock, flags); - list_add(&fbc->list, &percpu_counters); + for (i = 0; i < nr_counters; i++) + list_add(&fbc[i].list, &percpu_counters); spin_unlock_irqrestore(&percpu_counters_lock, flags); #endif return 0; } -EXPORT_SYMBOL(__percpu_counter_init); +EXPORT_SYMBOL(__percpu_counter_init_many); -void percpu_counter_destroy(struct percpu_counter *fbc) +void percpu_counter_destroy_many(struct percpu_counter *fbc, u32 nr_counters) { unsigned long flags __maybe_unused; + u32 i; + + if (WARN_ON_ONCE(!fbc)) + return; - if (!fbc->counters) + if (!fbc[0].counters) return; - debug_percpu_counter_deactivate(fbc); + for (i = 0; i < nr_counters; i++) + debug_percpu_counter_deactivate(&fbc[i]); #ifdef CONFIG_HOTPLUG_CPU spin_lock_irqsave(&percpu_counters_lock, flags); - list_del(&fbc->list); + for (i = 0; i < nr_counters; i++) + list_del(&fbc[i].list); spin_unlock_irqrestore(&percpu_counters_lock, flags); #endif - free_percpu(fbc->counters); - fbc->counters = NULL; + + free_percpu(fbc[0].counters); + + for (i = 0; i < nr_counters; i++) + fbc[i].counters = NULL; } -EXPORT_SYMBOL(percpu_counter_destroy); +EXPORT_SYMBOL(percpu_counter_destroy_many); int percpu_counter_batch __read_mostly = 32; EXPORT_SYMBOL(percpu_counter_batch);