From patchwork Thu Aug 25 00:05:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 12954102 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2C80C3F6B0 for ; Thu, 25 Aug 2022 00:05:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 11E99940008; Wed, 24 Aug 2022 20:05:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0A7686B0075; Wed, 24 Aug 2022 20:05:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E64A5940008; Wed, 24 Aug 2022 20:05:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D506C6B0074 for ; Wed, 24 Aug 2022 20:05:30 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A3A19160921 for ; Thu, 25 Aug 2022 00:05:30 +0000 (UTC) X-FDA: 79836170820.16.B8CB936 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf10.hostedemail.com (Postfix) with ESMTP id 4CE0EC0036 for ; Thu, 25 Aug 2022 00:05:29 +0000 (UTC) Received: by mail-pf1-f202.google.com with SMTP id a21-20020a62bd15000000b005360da6b25aso6820913pff.23 for ; Wed, 24 Aug 2022 17:05:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc; bh=X0aXbVnjIsqv6IZlugAyzwT5Jn0BIh8DUeidfr3NTwU=; b=pizx5jtbolFH74HPRP+JXhQp2kt4f4Gvo5c8BahsoqOTkDMpUC21gySwAqRyERcsxp RxiJ7ZTDPE4UhNy7ASniRjeSQfLiOp7A52n7UcNJaRcgoNbt5hJSwRUZ2tkpQ2Q3baJ8 4cQTvaVR9bW07R8J1S2oVdOXktAWKIkD5EuCtdCBJ7DOtVg6boyaG97qE7d++5ICk+ah 2bvtXldISERsWobmMdnkvmtae/Ky7rkElbKgMUHUw96nEyyamlUTC7zNBQtNgsfupFiv 1ehcbs9orlmOrXVk8eSGqQ8mVmxBPOLIJjcaHvhtzjRgZpozbsey811NndPtqmQMB85j qK7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc; bh=X0aXbVnjIsqv6IZlugAyzwT5Jn0BIh8DUeidfr3NTwU=; b=UsWsJp3Znxe8JxqeH1q0jmoQdsabvP8TNQx1DIq0OzOxAJZFFnHxeGpESN5IMTyuZ3 Yhnx4lkDjkRWdFT0r5nirLCHr3+B2ZeQiUadYlMDl17l/jOb8fWKxziJVPV9ZydKtF93 P7gjXcLaaKKgSPXjmWGaMjyKeaYeYpGFfZh1q+XMeSqf2/MFya4f3uHKoZgRLxn9zRWO Mslb27qGyw3T2Nga6Ed9W64LRvwNGtqcTxyV693jwlesjfl4trQ1w5Gnvo3n+wMozHW3 q/dBx5u+2+s67bKpTmWfmHoX7hZNkD0Tn+U67gPWT+CcBVlVYgunOO2ZB3+0jZvRD+a2 P4Xw== X-Gm-Message-State: ACgBeo0fxXuFdzK4Me/qJlNGJb/qqahmnyJLdWdXsdIcfyBBXCeprZ3h 0l8Wt+ziw23vRq9a1sJ74rpTrou4/tKBQg== X-Google-Smtp-Source: AA6agR5XSREtrswBTLHTLhIxt0y3r3MXI7uBpxQYmDyvEFx98x6p9m1M8gUdoBcShWUI0C8PD7kVrWRRQWbPPw== X-Received: from shakeelb.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:262e]) (user=shakeelb job=sendgmr) by 2002:a05:6a00:4147:b0:52e:2d56:17c8 with SMTP id bv7-20020a056a00414700b0052e2d5617c8mr1426905pfb.51.1661385928246; Wed, 24 Aug 2022 17:05:28 -0700 (PDT) Date: Thu, 25 Aug 2022 00:05:04 +0000 In-Reply-To: <20220825000506.239406-1-shakeelb@google.com> Message-Id: <20220825000506.239406-2-shakeelb@google.com> Mime-Version: 1.0 References: <20220825000506.239406-1-shakeelb@google.com> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog Subject: [PATCH v2 1/3] mm: page_counter: remove unneeded atomic ops for low/min From: Shakeel Butt To: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song Cc: " =?utf-8?q?Michal_Koutn=C3=BD?= " , Eric Dumazet , Soheil Hassas Yeganeh , Feng Tang , Oliver Sang , Andrew Morton , lkp@lists.01.org, cgroups@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Shakeel Butt ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661385929; a=rsa-sha256; cv=none; b=JAiCV49RdgLdOlceFvnVJO0ASGuIsvC4j90mgN/3pcYCNSoI0Z4fNhZdhE0m4LKh5yXF8F qloa2CpkFTKFxGYF3xsttQZxjEUL8a9VDOIdAojXZmi0BxA8j0BXpNkxJh9JVRqpOBHnh1 oeUvqCDnymTGNxOIVRb/5JlOsA5h0Kw= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=pizx5jtb; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3yLwGYwgKCHsrgZjddkafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--shakeelb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3yLwGYwgKCHsrgZjddkafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--shakeelb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661385929; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X0aXbVnjIsqv6IZlugAyzwT5Jn0BIh8DUeidfr3NTwU=; b=b/A5BiIt0fphT9u5vaPThJA/+bQThBu3ChgGnURh4ljYuWSjyTGETmp0TbKjrZV8rn8zU0 szvJ+eHN9MGGx9lwiu6gePjALqm2BM4Teak9Lu6SbXozrTx+T+BB8QvJ9QgcqByvBoqIs8 vq/tDW/ft9LeN8Ao1Vf3AYVx5ZKYbSM= Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=pizx5jtb; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3yLwGYwgKCHsrgZjddkafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--shakeelb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3yLwGYwgKCHsrgZjddkafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--shakeelb.bounces.google.com X-Rspamd-Queue-Id: 4CE0EC0036 X-Rspamd-Server: rspam02 X-Stat-Signature: d4mhsugpc9me39qt14kaympiiabpmhwb X-Rspam-User: X-HE-Tag: 1661385929-987364 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For cgroups using low or min protections, the function propagate_protected_usage() was doing an atomic xchg() operation irrespectively. We can optimize out this atomic operation for one specific scenario where the workload is using the protection (i.e. min > 0) and the usage is above the protection (i.e. usage > min). This scenario is actually very common where the users want a part of their workload to be protected against the external reclaim. Though this optimization does introduce a race when the usage is around the protection and concurrent charges and uncharged trip it over or under the protection. In such cases, we might see lower effective protection but the subsequent charge/uncharge will correct it. To evaluate the impact of this optimization, on a 72 CPUs machine, we ran the following workload in a three level of cgroup hierarchy with top level having min and low setup appropriately to see if this optimization is effective for the mentioned case. $ netserver -6 # 36 instances of netperf with following params $ netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K Results (average throughput of netperf): Without (6.0-rc1) 10482.7 Mbps With patch 14542.5 Mbps (38.7% improvement) With the patch, the throughput improved by 38.7% Signed-off-by: Shakeel Butt Reported-by: kernel test robot Acked-by: Soheil Hassas Yeganeh Reviewed-by: Feng Tang Acked-by: Roman Gushchin Acked-by: Michal Hocko --- Changes since v1: - Commit message update with more detail on which scenario is getting optimized and possible race condition. mm/page_counter.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/page_counter.c b/mm/page_counter.c index eb156ff5d603..47711aa28161 100644 --- a/mm/page_counter.c +++ b/mm/page_counter.c @@ -17,24 +17,23 @@ static void propagate_protected_usage(struct page_counter *c, unsigned long usage) { unsigned long protected, old_protected; - unsigned long low, min; long delta; if (!c->parent) return; - min = READ_ONCE(c->min); - if (min || atomic_long_read(&c->min_usage)) { - protected = min(usage, min); + protected = min(usage, READ_ONCE(c->min)); + old_protected = atomic_long_read(&c->min_usage); + if (protected != old_protected) { old_protected = atomic_long_xchg(&c->min_usage, protected); delta = protected - old_protected; if (delta) atomic_long_add(delta, &c->parent->children_min_usage); } - low = READ_ONCE(c->low); - if (low || atomic_long_read(&c->low_usage)) { - protected = min(usage, low); + protected = min(usage, READ_ONCE(c->low)); + old_protected = atomic_long_read(&c->low_usage); + if (protected != old_protected) { old_protected = atomic_long_xchg(&c->low_usage, protected); delta = protected - old_protected; if (delta) From patchwork Thu Aug 25 00:05:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 12954103 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 567ADC3F6B0 for ; Thu, 25 Aug 2022 00:05:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E671F940009; Wed, 24 Aug 2022 20:05:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DEDF86B0075; Wed, 24 Aug 2022 20:05:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C684F940009; Wed, 24 Aug 2022 20:05:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B5C5D6B0074 for ; Wed, 24 Aug 2022 20:05:35 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8808F120219 for ; Thu, 25 Aug 2022 00:05:35 +0000 (UTC) X-FDA: 79836171030.04.6C2BAD6 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf10.hostedemail.com (Postfix) with ESMTP id F1DE8C0038 for ; Thu, 25 Aug 2022 00:05:34 +0000 (UTC) Received: by mail-pf1-f202.google.com with SMTP id a21-20020a62bd15000000b005360da6b25aso6821043pff.23 for ; Wed, 24 Aug 2022 17:05:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc; bh=PYwH6PVPNEHbvaxrK4rS2xJuwZB9u4thO5MZW7XvPbI=; b=P+EUKtE6ricb8wdTGimfBl8wJsULmlf1U1ls8bGlVkUaNforVbvQ3tC93r2Tpdhx/X hu4ZaMawAO1E12zWi9v+4nP6DUb1SVdPpLc3AZ/xudS37LWaRSiaGz4e0Ka35xNar8nS etTkXsMGtCUsHLIDJkq4yN6bXebVUWkeBXO36LV/lhjXudjbUt/w2JGO2qgaytESq7HV 6902pIR1fNvHv7YRbB2LtQuxsm8du6S+P2rMcQlo6xTb/H6Yi7+CtEya63vTqQFwHjtL G+IvhJMvzorYcD57EwbY2fI611avPsNj/SezAfStYMECqaRueXSKqxPmfC43aGfBG3ct BUhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc; bh=PYwH6PVPNEHbvaxrK4rS2xJuwZB9u4thO5MZW7XvPbI=; b=tHz3mtFwHv6wf9lcOFqIwqfwVOZGSY15AjD2tzuEEmEdup9udVIBt+OPsaDPIcQ71+ rtyC9V2H/pxOUaOogfm3Lj17a/MDlz7xORbtdVl5BnGPUXoC5j7c66c4hxYcSfKxsSMx UM7WG6LVjdDGdNcn+1e5LnTRhIn6RiIX5oCek7/B/Ytdv/4kYHcy/0M6IuFd4iZK+XHw tmmx16NsXea/HEER1Nj7JtKYx+LdN2rr642jeCAJMdvI1ze6T/MTMI03Z6aqdukr2hke BFkXXMeQC2p81qx9+GT6vVYMUvySbGjGMUSnJVwMPrA6sEROrZJUaok6isP9ef9Fa7W0 GcrQ== X-Gm-Message-State: ACgBeo0uAXGEqxevm6LgqKk7MWjCIEXAPQNKJySaOy7tFhb0/Jc8Qor6 uigXNslBGR53d2NI0xmZ6Z57SDrkuThtTQ== X-Google-Smtp-Source: AA6agR7GEZzDTz56S0z9ZFwwo3+8A2cJS+hBXK4FDB5SFen3ixUf21xrVf8m+nkxp75ju1W8OqgypD5lC+ctAw== X-Received: from shakeelb.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:262e]) (user=shakeelb job=sendgmr) by 2002:a17:90b:a14:b0:1fa:bc6e:e5e8 with SMTP id gg20-20020a17090b0a1400b001fabc6ee5e8mr118633pjb.1.1661385933916; Wed, 24 Aug 2022 17:05:33 -0700 (PDT) Date: Thu, 25 Aug 2022 00:05:05 +0000 In-Reply-To: <20220825000506.239406-1-shakeelb@google.com> Message-Id: <20220825000506.239406-3-shakeelb@google.com> Mime-Version: 1.0 References: <20220825000506.239406-1-shakeelb@google.com> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog Subject: [PATCH v2 2/3] mm: page_counter: rearrange struct page_counter fields From: Shakeel Butt To: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song Cc: " =?utf-8?q?Michal_Koutn=C3=BD?= " , Eric Dumazet , Soheil Hassas Yeganeh , Feng Tang , Oliver Sang , Andrew Morton , lkp@lists.01.org, cgroups@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Shakeel Butt ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661385935; a=rsa-sha256; cv=none; b=D8rL+RsRONbDPEeHjAZqReauvbPmaD4w5p+p0Cd+UcBUBe6SKCVmbss9FKdaFB1ve+9mqK xTREEKLknnIaG1iQU0vrMWERDSP8DcQdNsKmGsC/RG/WLX5RATtZUpdX92z5SND18sHDFb 8W7rLjdCC4z5Ej8LZpNN9r77NLi91bg= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=P+EUKtE6; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3zbwGYwgKCIAwleoiipfksskpi.gsqpmry1-qqozego.svk@flex--shakeelb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3zbwGYwgKCIAwleoiipfksskpi.gsqpmry1-qqozego.svk@flex--shakeelb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661385935; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PYwH6PVPNEHbvaxrK4rS2xJuwZB9u4thO5MZW7XvPbI=; b=NcREBk6vU8QkJCGSMnqzeZs0SBPJdhHrkTptWnM3yKYSCy1qfnw9TWo4xbMV428q/Qwc3L yt59GXnWfJ1sECVWKMWAGV/xQbcsp3OtZfhlSgiOPAaFFU2T6qOki1Cf7+ejm07HBvG3oZ pWnvVSmj+pHpZltL9zckUPaJkztx5jw= Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=P+EUKtE6; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3zbwGYwgKCIAwleoiipfksskpi.gsqpmry1-qqozego.svk@flex--shakeelb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3zbwGYwgKCIAwleoiipfksskpi.gsqpmry1-qqozego.svk@flex--shakeelb.bounces.google.com X-Rspamd-Queue-Id: F1DE8C0038 X-Rspamd-Server: rspam02 X-Stat-Signature: iaoxrh8x3u9xtw45wcy3s4yppug38mgq X-Rspam-User: X-HE-Tag: 1661385934-160524 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With memcg v2 enabled, memcg->memory.usage is a very hot member for the workloads doing memcg charging on multiple CPUs concurrently. Particularly the network intensive workloads. In addition, there is a false cache sharing between memory.usage and memory.high on the charge path. This patch moves the usage into a separate cacheline and move all the read most fields into separate cacheline. To evaluate the impact of this optimization, on a 72 CPUs machine, we ran the following workload in a three level of cgroup hierarchy. $ netserver -6 # 36 instances of netperf with following params $ netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K Results (average throughput of netperf): Without (6.0-rc1) 10482.7 Mbps With patch 12413.7 Mbps (18.4% improvement) With the patch, the throughput improved by 18.4%. One side-effect of this patch is the increase in the size of struct mem_cgroup. For example with this patch on 64 bit build, the size of struct mem_cgroup increased from 4032 bytes to 4416 bytes. However for the performance improvement, this additional size is worth it. In addition there are opportunities to reduce the size of struct mem_cgroup like deprecation of kmem and tcpmem page counters and better packing. Signed-off-by: Shakeel Butt Reported-by: kernel test robot Reviewed-by: Feng Tang Acked-by: Soheil Hassas Yeganeh Acked-by: Roman Gushchin Acked-by: Michal Hocko --- Changes since v1: - Updated the commit message - Make struct page_counter cache align. include/linux/page_counter.h | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h index 679591301994..78a1c934e416 100644 --- a/include/linux/page_counter.h +++ b/include/linux/page_counter.h @@ -3,15 +3,26 @@ #define _LINUX_PAGE_COUNTER_H #include +#include #include #include +#if defined(CONFIG_SMP) +struct pc_padding { + char x[0]; +} ____cacheline_internodealigned_in_smp; +#define PC_PADDING(name) struct pc_padding name +#else +#define PC_PADDING(name) +#endif + struct page_counter { + /* + * Make sure 'usage' does not share cacheline with any other field. The + * memcg->memory.usage is a hot member of struct mem_cgroup. + */ atomic_long_t usage; - unsigned long min; - unsigned long low; - unsigned long high; - unsigned long max; + PC_PADDING(_pad1_); /* effective memory.min and memory.min usage tracking */ unsigned long emin; @@ -23,18 +34,18 @@ struct page_counter { atomic_long_t low_usage; atomic_long_t children_low_usage; - /* legacy */ unsigned long watermark; unsigned long failcnt; - /* - * 'parent' is placed here to be far from 'usage' to reduce - * cache false sharing, as 'usage' is written mostly while - * parent is frequently read for cgroup's hierarchical - * counting nature. - */ + /* Keep all the read most fields in a separete cacheline. */ + PC_PADDING(_pad2_); + + unsigned long min; + unsigned long low; + unsigned long high; + unsigned long max; struct page_counter *parent; -}; +} ____cacheline_internodealigned_in_smp; #if BITS_PER_LONG == 32 #define PAGE_COUNTER_MAX LONG_MAX From patchwork Thu Aug 25 00:05:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 12954104 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12647C04AA5 for ; Thu, 25 Aug 2022 00:05:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A947094000A; Wed, 24 Aug 2022 20:05:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A1D0D6B0075; Wed, 24 Aug 2022 20:05:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BE3F94000A; Wed, 24 Aug 2022 20:05:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7B2D96B0074 for ; Wed, 24 Aug 2022 20:05:40 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 53B161C7123 for ; Thu, 25 Aug 2022 00:05:40 +0000 (UTC) X-FDA: 79836171240.25.B8F9F10 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf22.hostedemail.com (Postfix) with ESMTP id 11712C0044 for ; Thu, 25 Aug 2022 00:05:39 +0000 (UTC) Received: by mail-pj1-f73.google.com with SMTP id b5-20020a17090a12c500b001faa33a11dbso1540148pjg.1 for ; Wed, 24 Aug 2022 17:05:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc; bh=3ANOEI0BBXDszhHsMjNjVkJxW2dSCbobxCznkiQMg0M=; b=M8tSmqMgyZLVy2Ldt7OEwatTGNnYnfDpQ69BuMQNcXOyVDB1zOG1S5rA/96I/PJKxE 4+npbWP3pWJ08Qbrxcia+8i6j7Gu69xnnOJNpGf1qEh/CTh6F27WQ4nVQLW4A9J5Lhh9 wZHGTsioVx7toZiUJ8cHixmiZOFP7E6zdyNZLONxSLuMdImMycE2Fyt+XuqZ+8IhLHsR C6dFsjW4r02bqFmtyRS3Kq6G3VP5sgtlfNBMT7ztQEx6AL13xu9pwUVXfdMd6YEAnByA YGozttZzHIvY64hF0IBSZ5RbI4tZvRxI5hd9vrRa+K0+Wr8X++i6Jrnvoe/mNBWb+0ko v09Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc; bh=3ANOEI0BBXDszhHsMjNjVkJxW2dSCbobxCznkiQMg0M=; b=3debY8phhaVOegr9l7XyetWhYoC0+gvngoVKcFwycpkQ1GJOfGVxnxtoxcCk8oFN2w 2bRF6UiP2DYEiVP0RBCdaJiDCoAbPBrDDzncYUPtbZZHmJIIreDWF+F47l7s5sYwqai7 4ijxFWJwgSWhk71SALHi1Df6Xd41urj6GpOzIJiN9sno4LJHUSub4x9/OgMlyFOkULtN /WqS0XJ5zT7Rds14kfrmHryfb7pjJOh4akD4l9v0FnUtjGZubTsXzPYMD2n7DUlZTPKH CckFnXwA3qtQwdgL2KOr8e1s4eDfewAkwMGiyiaf6LxML3NyNXsgGIrfViN7wBqzLZnZ 6hvA== X-Gm-Message-State: ACgBeo2FSMMsxATvx0cBe5rmaochQfQZzgE/MwMsYUPOcbGutOew386J b1MDHsST4y/cAbAbBC4W4yElRiOirlJo+w== X-Google-Smtp-Source: AA6agR4pbPM00Y2dhBDzDYXIyyPDDsgaDbSO1VcMlYVbQBY5g87vni0ByGdo33smXqzxyN0vznInx5Zn2YarNQ== X-Received: from shakeelb.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:262e]) (user=shakeelb job=sendgmr) by 2002:a05:6a00:1803:b0:536:e9a4:89d4 with SMTP id y3-20020a056a00180300b00536e9a489d4mr1451224pfa.28.1661385939060; Wed, 24 Aug 2022 17:05:39 -0700 (PDT) Date: Thu, 25 Aug 2022 00:05:06 +0000 In-Reply-To: <20220825000506.239406-1-shakeelb@google.com> Message-Id: <20220825000506.239406-4-shakeelb@google.com> Mime-Version: 1.0 References: <20220825000506.239406-1-shakeelb@google.com> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog Subject: [PATCH v2 3/3] memcg: increase MEMCG_CHARGE_BATCH to 64 From: Shakeel Butt To: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song Cc: " =?utf-8?q?Michal_Koutn=C3=BD?= " , Eric Dumazet , Soheil Hassas Yeganeh , Feng Tang , Oliver Sang , Andrew Morton , lkp@lists.01.org, cgroups@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Shakeel Butt ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661385940; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3ANOEI0BBXDszhHsMjNjVkJxW2dSCbobxCznkiQMg0M=; b=R6nF2m1XOUp5IoGdH99rS9PXqLmFewCOZ0ErOuz0ibGPpEic+qj2puoImXf/QI4JEDCc5l xyq+oBpzvCd+dyiaatpUmXehe98/4uel1vNnkcRBM1BqeKPOpPRJCFNr+A4u9+U0knqW+g Kh08afwt1Qv6ziizf5hvyFLZcVYXe0c= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=M8tSmqMg; spf=pass (imf22.hostedemail.com: domain of 307wGYwgKCIY2rkuoovlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--shakeelb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=307wGYwgKCIY2rkuoovlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661385940; a=rsa-sha256; cv=none; b=RXRbIFn+IqTELozHxfYBSGWdjO+C30Z1J4ZPtX7UK95FiGD6NDuZ1uqRrpiP80+2bVh4uy V5peeGvzIGAggyCnQLTXW/iKJBq59Wc3hfE6TDBP5p8YZT34HiDMZyQu74urFhQTJu3Idt HlKuGy9AQYv0xJmZ4RHWhPnyI/HCJbs= X-Rspamd-Queue-Id: 11712C0044 Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=M8tSmqMg; spf=pass (imf22.hostedemail.com: domain of 307wGYwgKCIY2rkuoovlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--shakeelb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=307wGYwgKCIY2rkuoovlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: c6dwagd7b3o9k6h8brhdp1msbf4m1mxj X-HE-Tag: 1661385939-451902 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For several years, MEMCG_CHARGE_BATCH was kept at 32 but with bigger machines and the network intensive workloads requiring througput in Gbps, 32 is too small and makes the memcg charging path a bottleneck. For now, increase it to 64 for easy acceptance to 6.0. We will need to revisit this in future for ever increasing demand of higher performance. Please note that the memcg charge path drain the per-cpu memcg charge stock, so there should not be any oom behavior change. Though it does have impact on rstat flushing and high limit reclaim backoff. To evaluate the impact of this optimization, on a 72 CPUs machine, we ran the following workload in a three level of cgroup hierarchy. $ netserver -6 # 36 instances of netperf with following params $ netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K Results (average throughput of netperf): Without (6.0-rc1) 10482.7 Mbps With patch 17064.7 Mbps (62.7% improvement) With the patch, the throughput improved by 62.7%. Signed-off-by: Shakeel Butt Reported-by: kernel test robot Acked-by: Soheil Hassas Yeganeh Reviewed-by: Feng Tang Acked-by: Roman Gushchin Acked-by: Michal Hocko Acked-by: Muchun Song --- Changes since v1: - Updated commit message include/linux/memcontrol.h | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4d31ce55b1c0..70ae91188e16 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -354,10 +354,11 @@ struct mem_cgroup { }; /* - * size of first charge trial. "32" comes from vmscan.c's magic value. - * TODO: maybe necessary to use big numbers in big irons. + * size of first charge trial. + * TODO: maybe necessary to use big numbers in big irons or dynamic based of the + * workload. */ -#define MEMCG_CHARGE_BATCH 32U +#define MEMCG_CHARGE_BATCH 64U extern struct mem_cgroup *root_mem_cgroup;