From patchwork Mon Aug 21 20:28:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mateusz Guzik X-Patchwork-Id: 13359794 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AE05EE4996 for ; Mon, 21 Aug 2023 20:28:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A3BD794000C; Mon, 21 Aug 2023 16:28:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9EB7F8E0012; Mon, 21 Aug 2023 16:28:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B3B794000C; Mon, 21 Aug 2023 16:28:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 788DF8E0012 for ; Mon, 21 Aug 2023 16:28:55 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4B922809CC for ; Mon, 21 Aug 2023 20:28:55 +0000 (UTC) X-FDA: 81149250630.09.71E6F85 Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) by imf19.hostedemail.com (Postfix) with ESMTP id 737B11A0009 for ; Mon, 21 Aug 2023 20:28:53 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=HaTnWJYE; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.218.52 as permitted sender) smtp.mailfrom=mjguzik@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692649733; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dcR7quqcdfLIpDV/BrSlU217Tc1djQCifnGeh+jON88=; b=j2e+7/P9raAKLmeJwXJOsSLxc98ucLvqFt/xyBiGtpUSarwg3EnKh4DsWoUAwCmyG5Nw2a B9DmTWyXKMCC2sPrDtGop1WMdJ4Mj2S+ktkiKumNnZ8zoVCWAT671JS6JAdj7q7EHiTURR R4bs8mR1pWRPPflfIhwR8wIDXYEybRQ= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=HaTnWJYE; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.218.52 as permitted sender) smtp.mailfrom=mjguzik@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692649733; a=rsa-sha256; cv=none; b=CEUAaotkAS7PatebnkUtzVK7BePI8d6zZBzqLPU8TI5tKB1bRjfFuSikHgTj+3D3WQD++3 L3d0a8+hKmQNMU2Mmq/rGwycE0IWfX7cw376wV5EwsTKYAQXKCsgYqoESrKQ7BTnkocRGK sUkSzhgYirx2iLvu1WjAqGgtWt4UPX0= Received: by mail-ej1-f52.google.com with SMTP id a640c23a62f3a-99c93638322so775153366b.1 for ; Mon, 21 Aug 2023 13:28:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692649732; x=1693254532; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dcR7quqcdfLIpDV/BrSlU217Tc1djQCifnGeh+jON88=; b=HaTnWJYE97VIyQjGgW8oXxjKuiWkeUDWXj46fGZH1mL4dDpwmq7tmfpX5SktuZoiP5 hUtFQbd9VD/uRaVPfssMjRXUDKkjwj2A6IZCCObm/PzrXt7RdvIk2/Cx4n+EwScduqvq 5zdLlfCcmOL+WnnWtVzhfdFeWiK/jVpRbHz0NrJBS3m51vhUAsJ9GDYSkLvN5V35GGPk LknmQ4pZAN4UBzap1Q9GHexrXPs5Msp2Y7p+LDXQgxRUj4sz/RO9X+apujJZ7Kp0PLnR DMtpZchQSUEMIRLKVAZZ2I3YyabBIp1fK5tZvPcgOUQXZ8aW3aduoK0roeYwuO9Ki75r nQvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692649732; x=1693254532; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dcR7quqcdfLIpDV/BrSlU217Tc1djQCifnGeh+jON88=; b=jRBRMMjlrzfZJOeNSG8bTN35M6wBmnqAr41maqUtxWM8RwxGzB5LdIhnT+3bVYfQou l2W6a9d8FCcnornB6WCsC6oyDB5ubPscbGr3xJhYOnCqSxtpSYHUNDEgnHVbXz/3Kltv 9Kqe5drl5OXlY0yrxaXnJqpN72xFah4+yr4gFwd1UJwKhJhHEZhDoN3KNiF78BnZBgXz frht2mjuWcCdb16roXODASl4eMG0txNhzDX5WsZ9HTpK0qK75+svS09FP3q0pccRdpbf OqvuFYbAVLvEmKCSEMBjWMKJu1s1vEPNLDd4aAoUtj8ukyiPGijCtsKCc9vQF/G5mj6e RL6Q== X-Gm-Message-State: AOJu0Yy59TiHQ9b+sEZiOnb2OsnR40RoE8ARXfXjSeLdJPQo7SftO8Md /ffLCr53OJPuItaY5eb64jg= X-Google-Smtp-Source: AGHT+IEEFRgYy+w0a0gW6U6MQsAvYVD8DCQhrRJusGPWmQcAHJH5nXLue/UF1odPjC5zuNn88lHHTw== X-Received: by 2002:a17:907:6d9a:b0:99b:cadd:c2ee with SMTP id sb26-20020a1709076d9a00b0099bcaddc2eemr8773312ejc.29.1692649732008; Mon, 21 Aug 2023 13:28:52 -0700 (PDT) Received: from f.. (cst-prg-85-121.cust.vodafone.cz. [46.135.85.121]) by smtp.gmail.com with ESMTPSA id k26-20020a1709062a5a00b00997cce73cc7sm7084450eje.29.2023.08.21.13.28.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Aug 2023 13:28:51 -0700 (PDT) From: Mateusz Guzik To: linux-kernel@vger.kernel.org Cc: dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, linux-mm@kvack.org, Mateusz Guzik Subject: [PATCH 1/2] pcpcntr: add group allocation/free Date: Mon, 21 Aug 2023 22:28:28 +0200 Message-Id: <20230821202829.2163744-2-mjguzik@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230821202829.2163744-1-mjguzik@gmail.com> References: <20230821202829.2163744-1-mjguzik@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 737B11A0009 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: i45p5jpdi5cfqiymmakek3ptoondsewa X-HE-Tag: 1692649733-659572 X-HE-Meta: U2FsdGVkX1/Q/D5NfSLpLiXmrhVQdBjcxSJJ/YmHt6CmaY2dhJs5dMky6e63jcXZ3QktwhKZIqpQwibrcPVm2U1RaIOM3YPHM+wXTrOtNpaLGYd0zikXdlYvW5CqbB9ufmgtozoUzuGRv41371i7nSM3ifDX48S3ufbbCuiifhJ4kg1pdVETLj/pOzDO3gWf6F5fh0H4rBVn4Lai5KZ3MM+C07HwZMPA1zIK4HtQC4kSL4YYP5xtrfHbKEMSVzmFR8qg15eMueLHbu2kfepxzPZVM2oecTadpN61OZOLQCIaJflGNIAyRu86OLowSFp3v3DORim6D3zc8C2uwXDnCuu+S5D3TebHJ48HVGH8BIU/5dCqZCEgh0UU4w/8WCCJyDPOjP/x5/B8zLL/3vChqh0bYuJ3ia53yiWOSbg/kF5Cnr7hji0uYvjT0P7fsI1Xz9jSrTKdEYgifwMovIMILEQ0/uxkDeLL2FduDkaHG1LJOlzGwsmef3qiu9+OxEEUnpIle+FSLCR80YvZsq+0JcuxTkKjUHab74YZSwFDqbTRt5Kb29EOD9wpeAMefoYdWKmVAzm8b+8g7XSOLUs/LIOfciVYu9NxrQEWfhff/u+6fqaVsGX3zypcunmmrnS5m4ASB1VTc6uQMI8L0i1Fpdj6/hogxUdNkff+5sIDhiUgm8jFldBtb051q/ctdiVRzxVC1VuFUTGUJ28rO7QpVX3+DakKNzApQywCJ0XkiCQ8Cf8NJcAtgJ0fAGlAXTRItnOr9G1YWS23kY8SdZfgapxl0Xez7DHgCAvLiwRMr33QGnPEBlUtc+gk1/jZtvYZsryI//iHnTBamRr/kk9RcZ+UcOTDaCMPOeGiMipE2DyWDn+boBKSJfUsGDHRc926s2SbPAVJRe7GlhhRFdRpyeC2ZuLITdrC7DhgWoT4RgCSX+hKOQhIsutZKo9yid+Ax/seceEQYvOPrVncI+j XZMD6Y6/ BLqvjPcYXamgbnHs/x5kykQ0Eqitayex1wthPJJ8ALLjDCGn9WXmKT9L6uqUK+tyw7FoyAQT/wSlEtKj2BZMoHy+hgTUetti/HY3/WuBrrZsqFtXKaNLV19j78mcyrJKhWYuYwgfNGu/8HHEfrOL7OL2CgAVAgg6QhUk1a+wT3htGh2gd60yjTjXS44o8zu27pjdmkp8AQvhjXttzIn0c0fxcTZEY2aa4im7TKepGEbk9uXTE4FJ+aaqdonwbcUwZpioc5acx6pD4xmisR2Q+656cilsoLEjhAUYbLJG4eJDWdfeleJd61kTAsw073uKBzbWtu05Z3PeH63/GqOF1YBe4M5aS1kB7ibZXMC0sYVtx5LmhbKs//81VeUBDQOjOsyo6bgGbeUkc7JPZfvq81j/JGaZ/oe4xuXqIql6H11TaesO2fVBvI648xaGYjPR3Hy4A X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocations and frees are globally serialized on the pcpu lock (and the CPU hotplug lock if enabled, which is the case on Debian). At least one frequent consumer allocates 4 back-to-back counters (and frees them in the same manner), exacerbating the problem. While this does not fully remedy scalability issues, it is a step towards that goal and provides immediate relief. Signed-off-by: Mateusz Guzik Reviewed-by: Vegard Nossum --- include/linux/percpu_counter.h | 19 ++++++++--- lib/percpu_counter.c | 61 ++++++++++++++++++++++++---------- 2 files changed, 57 insertions(+), 23 deletions(-) diff --git a/include/linux/percpu_counter.h b/include/linux/percpu_counter.h index 75b73c83bc9d..ff5850b07124 100644 --- a/include/linux/percpu_counter.h +++ b/include/linux/percpu_counter.h @@ -30,17 +30,26 @@ struct percpu_counter { extern int percpu_counter_batch; -int __percpu_counter_init(struct percpu_counter *fbc, s64 amount, gfp_t gfp, - struct lock_class_key *key); +int __percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, gfp_t gfp, + struct lock_class_key *key, u32 count); -#define percpu_counter_init(fbc, value, gfp) \ +#define percpu_counter_init_many(fbc, value, gfp, count) \ ({ \ static struct lock_class_key __key; \ \ - __percpu_counter_init(fbc, value, gfp, &__key); \ + __percpu_counter_init_many(fbc, value, gfp, &__key, count);\ }) -void percpu_counter_destroy(struct percpu_counter *fbc); + +#define percpu_counter_init(fbc, value, gfp) \ + percpu_counter_init_many(fbc, value, gfp, 1) + +void percpu_counter_destroy_many(struct percpu_counter *fbc, u32 count); +static inline void percpu_counter_destroy(struct percpu_counter *fbc) +{ + percpu_counter_destroy_many(fbc, 1); +} + void percpu_counter_set(struct percpu_counter *fbc, s64 amount); void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount, s32 batch); diff --git a/lib/percpu_counter.c b/lib/percpu_counter.c index 5004463c4f9f..2a33cf23df55 100644 --- a/lib/percpu_counter.c +++ b/lib/percpu_counter.c @@ -151,48 +151,73 @@ s64 __percpu_counter_sum(struct percpu_counter *fbc) } EXPORT_SYMBOL(__percpu_counter_sum); -int __percpu_counter_init(struct percpu_counter *fbc, s64 amount, gfp_t gfp, - struct lock_class_key *key) +int __percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, gfp_t gfp, + struct lock_class_key *key, u32 count) { unsigned long flags __maybe_unused; + s32 __percpu *counters; + u32 i; - raw_spin_lock_init(&fbc->lock); - lockdep_set_class(&fbc->lock, key); - fbc->count = amount; - fbc->counters = alloc_percpu_gfp(s32, gfp); - if (!fbc->counters) + counters = __alloc_percpu_gfp(sizeof(*counters) * count, + sizeof(*counters), gfp); + if (!counters) { + fbc[0].counters = NULL; return -ENOMEM; + } - debug_percpu_counter_activate(fbc); + for (i = 0; i < count; i++) { + raw_spin_lock_init(&fbc[i].lock); + lockdep_set_class(&fbc[i].lock, key); +#ifdef CONFIG_HOTPLUG_CPU + INIT_LIST_HEAD(&fbc[i].list); +#endif + fbc[i].count = amount; + fbc[i].counters = &counters[i]; + + debug_percpu_counter_activate(&fbc[i]); + } #ifdef CONFIG_HOTPLUG_CPU - INIT_LIST_HEAD(&fbc->list); spin_lock_irqsave(&percpu_counters_lock, flags); - list_add(&fbc->list, &percpu_counters); + for (i = 0; i < count; i++) { + list_add(&fbc[i].list, &percpu_counters); + } spin_unlock_irqrestore(&percpu_counters_lock, flags); #endif return 0; } -EXPORT_SYMBOL(__percpu_counter_init); +EXPORT_SYMBOL(__percpu_counter_init_many); -void percpu_counter_destroy(struct percpu_counter *fbc) +void percpu_counter_destroy_many(struct percpu_counter *fbc, u32 count) { unsigned long flags __maybe_unused; + u32 i; - if (!fbc->counters) + if (WARN_ON_ONCE(!fbc)) return; - debug_percpu_counter_deactivate(fbc); + if (!fbc[0].counters) + return; + + for (i = 0; i < count; i++) { + debug_percpu_counter_deactivate(&fbc[i]); + } #ifdef CONFIG_HOTPLUG_CPU spin_lock_irqsave(&percpu_counters_lock, flags); - list_del(&fbc->list); + for (i = 0; i < count; i++) { + list_del(&fbc[i].list); + } spin_unlock_irqrestore(&percpu_counters_lock, flags); #endif - free_percpu(fbc->counters); - fbc->counters = NULL; + + free_percpu(fbc[0].counters); + + for (i = 0; i < count; i++) { + fbc[i].counters = NULL; + } } -EXPORT_SYMBOL(percpu_counter_destroy); +EXPORT_SYMBOL(percpu_counter_destroy_many); int percpu_counter_batch __read_mostly = 32; EXPORT_SYMBOL(percpu_counter_batch);