From patchwork Mon Aug 22 00:17:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 12950095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 441D1C00140 for ; Mon, 22 Aug 2022 00:19:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D45BD8E0001; Sun, 21 Aug 2022 20:19:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CF4166B0075; Sun, 21 Aug 2022 20:19:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBD078E0001; Sun, 21 Aug 2022 20:19:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AF6EB6B0074 for ; Sun, 21 Aug 2022 20:19:32 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 83AAA160410 for ; Mon, 22 Aug 2022 00:19:32 +0000 (UTC) X-FDA: 79825319784.27.38B6960 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf07.hostedemail.com (Postfix) with ESMTP id B547D4006A for ; Mon, 22 Aug 2022 00:17:53 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-32a115757b6so162367027b3.13 for ; Sun, 21 Aug 2022 17:17:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc; bh=PKFRaxD4qQqEIFqu3chPSEEEaimoA7SPNFa30ZMS/uU=; b=HizZvrBQiCe/lTGkboVezrNH98vpHv6pPqN3AxEJH/hFrU99YqU4n2tDb7vx0BDvjD bQrrXzlJxZNS5VEi+6ur1985jYp1mqaVxFam9zQg29fgy4j8FDwrvk2EWIEw9zZcLyy8 ZrbSnLWlxJzAlcK4inCb0YBuOPVU8Pf6wgn3U0AS+YKBi3N8fLf+SDkLifHnsuVpXtE9 VW+MgYxWYsl4FT5vPrWtjoQKUyoGIDwGhmbfEnhOfVukKNjsvqjWPYthJA+lrTU4JyRJ Sj3VMg2YYPUcwhLJDwgqi0QQBI7r5yYyU5d+lRAwuifJcVjqGV647qwS/PzhpquTv4FZ z7vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc; bh=PKFRaxD4qQqEIFqu3chPSEEEaimoA7SPNFa30ZMS/uU=; b=UkVyOS9nPCJVoWXSINvfmFccE2OywJBeScrjHNPGO/LsrEyYPo0Yu4Ihp4CECrnyUu gvexB9PKD81TTo3AOuvCK2APfu9N3zzv0nZE+v/8YOdzH4gbYCvEszeTpxiiu+jcma1z TFLhcOaS8DSRfbyrEJfFnA90sAOZOv0uUula9kTj+wknnkIFSm15yoUhaaJtMZ1NNSL1 UInh336NV+6Yy4w3B0/m8OG8BQiMlBKdf8YEll/ucIVCV3zyYam6MmsaFXtNuFTu2rx/ lX1BwfmHyZKPPto8E9bxiKPfUfPxUZgQid4lJrLmF8ZMNpNhYvlqNh5tkiY9lQ6YDlUw EK1Q== X-Gm-Message-State: ACgBeo1eDTcft9wIcT6Fj/n7ZBK2ZK3wHCInZ7QNqG8VDCK8jEu7DoWs CCmhJF5RtC/5z/yy1HirQEYKPUvgMucU9A== X-Google-Smtp-Source: AA6agR7ugDocQyFMo7I0NUqJCeOZethCdyIXG6uExhXVb1g/djDmpfI1UlZI7JLxJy/0h1pcp1JCf7rda8Xqlw== X-Received: from shakeelb.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:28b]) (user=shakeelb job=sendgmr) by 2002:a81:9b17:0:b0:335:c382:48d with SMTP id s23-20020a819b17000000b00335c382048dmr17808577ywg.244.1661127473042; Sun, 21 Aug 2022 17:17:53 -0700 (PDT) Date: Mon, 22 Aug 2022 00:17:35 +0000 In-Reply-To: <20220822001737.4120417-1-shakeelb@google.com> Message-Id: <20220822001737.4120417-2-shakeelb@google.com> Mime-Version: 1.0 References: <20220822001737.4120417-1-shakeelb@google.com> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog Subject: [PATCH 1/3] mm: page_counter: remove unneeded atomic ops for low/min From: Shakeel Butt To: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song Cc: " =?utf-8?q?Michal_Koutn=C3=BD?= " , Eric Dumazet , Soheil Hassas Yeganeh , Feng Tang , Oliver Sang , Andrew Morton , lkp@lists.01.org, cgroups@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Shakeel Butt ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661127473; a=rsa-sha256; cv=none; b=4alt99/+FCLuBsED4nDZ5enjuUfjtHlLLM7DoCNKISJoWO1KLQ4SfO35sjkSfqJNsLzlNe vGV6v0qEANii/wbHV0UPRFrk9bwHtOcDj8SusW/XEdEfU8IrY7R3pDIqMGoTbHRTU2/YSn fhlFf//Vw9MXzFUBvsDPQ4F3OV67RIs= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=HizZvrBQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3McsCYwgKCPAkZScWWdTYggYdW.Ugedafmp-eecnSUc.gjY@flex--shakeelb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3McsCYwgKCPAkZScWWdTYggYdW.Ugedafmp-eecnSUc.gjY@flex--shakeelb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661127473; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PKFRaxD4qQqEIFqu3chPSEEEaimoA7SPNFa30ZMS/uU=; b=yshSedbrovoAf6FwVeRLbJuRgaX8b/w7YZYfGADMm+YisC/0oaFXFuI2SQmaEFBSmePUKM kduaV495+YJk5XwH0Bik+K2fASloQYwwPQT7eknC7fCpXtMwhbxMnVElPQR+EzZHbvSSZh btqrcj31ZyDEowO1lrPtylhf8hHwtq0= Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=HizZvrBQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3McsCYwgKCPAkZScWWdTYggYdW.Ugedafmp-eecnSUc.gjY@flex--shakeelb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3McsCYwgKCPAkZScWWdTYggYdW.Ugedafmp-eecnSUc.gjY@flex--shakeelb.bounces.google.com X-Rspam-User: X-Stat-Signature: xyc5wihjhik8ji8qz67uujxqo5sahrcw X-Rspamd-Queue-Id: B547D4006A X-Rspamd-Server: rspam06 X-HE-Tag: 1661127473-813547 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For cgroups using low or min protections, the function propagate_protected_usage() was doing an atomic xchg() operation irrespectively. It only needs to do that operation if the new value of protection is different from older one. This patch does that. To evaluate the impact of this optimization, on a 72 CPUs machine, we ran the following workload in a three level of cgroup hierarchy with top level having min and low setup appropriately. More specifically memory.min equal to size of netperf binary and memory.low double of that. $ netserver -6 # 36 instances of netperf with following params $ netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K Results (average throughput of netperf): Without (6.0-rc1) 10482.7 Mbps With patch 14542.5 Mbps (38.7% improvement) With the patch, the throughput improved by 38.7% Signed-off-by: Shakeel Butt Reported-by: kernel test robot Acked-by: Soheil Hassas Yeganeh Reviewed-by: Feng Tang Acked-by: Roman Gushchin --- mm/page_counter.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/page_counter.c b/mm/page_counter.c index eb156ff5d603..47711aa28161 100644 --- a/mm/page_counter.c +++ b/mm/page_counter.c @@ -17,24 +17,23 @@ static void propagate_protected_usage(struct page_counter *c, unsigned long usage) { unsigned long protected, old_protected; - unsigned long low, min; long delta; if (!c->parent) return; - min = READ_ONCE(c->min); - if (min || atomic_long_read(&c->min_usage)) { - protected = min(usage, min); + protected = min(usage, READ_ONCE(c->min)); + old_protected = atomic_long_read(&c->min_usage); + if (protected != old_protected) { old_protected = atomic_long_xchg(&c->min_usage, protected); delta = protected - old_protected; if (delta) atomic_long_add(delta, &c->parent->children_min_usage); } - low = READ_ONCE(c->low); - if (low || atomic_long_read(&c->low_usage)) { - protected = min(usage, low); + protected = min(usage, READ_ONCE(c->low)); + old_protected = atomic_long_read(&c->low_usage); + if (protected != old_protected) { old_protected = atomic_long_xchg(&c->low_usage, protected); delta = protected - old_protected; if (delta) From patchwork Mon Aug 22 00:17:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 12950092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37DBCC00140 for ; Mon, 22 Aug 2022 00:18:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A5D498E0001; Sun, 21 Aug 2022 20:18:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A0DC76B0074; Sun, 21 Aug 2022 20:18:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FC408E0001; Sun, 21 Aug 2022 20:18:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7EECF6B0073 for ; Sun, 21 Aug 2022 20:18:02 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 521EC80129 for ; Mon, 22 Aug 2022 00:18:02 +0000 (UTC) X-FDA: 79825316004.18.6154438 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf09.hostedemail.com (Postfix) with ESMTP id 1A4891400A8 for ; Mon, 22 Aug 2022 00:18:01 +0000 (UTC) Received: by mail-pl1-f202.google.com with SMTP id j11-20020a170902da8b00b0016f17813479so6260345plx.5 for ; Sun, 21 Aug 2022 17:18:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc; bh=V/ZZIFSPKFcCAZIO5J88CNRgR4nImf4CDqSEwHgZtc8=; b=VRt8PiD5az0Dl+WRznAYAHoEFU2MambiOUqw9hJIF1ucU60TS0sVFR1bUuaJD8nPng xGG21VhjqfptsMV9A8m6NtDIcNUxnu3moTzE7NDuS0132TTFDbZUQdoq7PUgYhDiEc88 VWXGevOtBTC071FhOwLWbdC2RbpbtXjJBP30qVmnMrkSyaRORF9MkIgIOE44YT8fPxhx MXMroQq5cbkgX/PlRdMgvF1WbQYDo+Czb0BC6hLgVyoEUZLM+nIZRg1Iz+BkUSiS19+u NDrsfq1Tm8UcqdVd/B6tTxgxOaUMNvPwSrhSYvXv0b64wiY5pKs+NN5viMa0qiHMezJn gzSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc; bh=V/ZZIFSPKFcCAZIO5J88CNRgR4nImf4CDqSEwHgZtc8=; b=Y57x/WJ7AewxchKwWcDIVRu6lhOmeLonz8PKLiYueqeM5ooQfkZ2gAx895gDPNuGzu nxwSnnl87Z3dD7lU/KMTJeLKSzhFejCdRTHi9wXWeYBt3NtgCEnwnyD4M66ld9CYVagR X08MNRvpOqovbpDnu4ixQXcRLjbOM1lawKMGHHeXRaIaWId/WlADstJ9cxLK20nw+cya rmETEESG879LemQzt3a0xJHEYgtxiwNUvkp2CPvR8Yn8L2PzD0yV10iRXfdhH6CWUFOk WT7uttuebNjZbQPxapHtNBgrh2BQ7NW6ed7iuwxRamGJQJs9XVI3/+z2LU5cTrtEEW3N 2Ezg== X-Gm-Message-State: ACgBeo1PaHcblLRnG4jez6XXZjU4hmo0nTG6I3gN6+dpjVifAOgQca/F gtMNym/71FnexR2tBv6ezsUhYNkwTG7G4Q== X-Google-Smtp-Source: AA6agR4ZgdLQRLm8RNOfyL+FjO9Haoqcxi6ZB51K8Zqax6/eLDbW7KkNV8E35m3mv6YMj7ylkoz8NrwpsF9kng== X-Received: from shakeelb.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:28b]) (user=shakeelb job=sendgmr) by 2002:a17:902:da8e:b0:172:9f0b:3778 with SMTP id j14-20020a170902da8e00b001729f0b3778mr17761139plx.166.1661127480895; Sun, 21 Aug 2022 17:18:00 -0700 (PDT) Date: Mon, 22 Aug 2022 00:17:36 +0000 In-Reply-To: <20220822001737.4120417-1-shakeelb@google.com> Message-Id: <20220822001737.4120417-3-shakeelb@google.com> Mime-Version: 1.0 References: <20220822001737.4120417-1-shakeelb@google.com> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog Subject: [PATCH 2/3] mm: page_counter: rearrange struct page_counter fields From: Shakeel Butt To: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song Cc: " =?utf-8?q?Michal_Koutn=C3=BD?= " , Eric Dumazet , Soheil Hassas Yeganeh , Feng Tang , Oliver Sang , Andrew Morton , lkp@lists.01.org, cgroups@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Shakeel Butt ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661127482; a=rsa-sha256; cv=none; b=gAFGkwlbN/nHuZIB/Z/ty7odfoSiKZ17iHl0yp1IJ02jolnsxgCXL+CngOEPnwj+DIB2bM LV112K87oeCMkd4FUPSUBWvFMiwJ7d3Szyznaq5/vdjgZd8+eLYpL8CrboXHSfzSphFgbW tVFaBdKRrEMYpY8dHO0HNTRrZH2318k= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=VRt8PiD5; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3OMsCYwgKCPcrgZjddkafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--shakeelb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3OMsCYwgKCPcrgZjddkafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--shakeelb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661127482; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V/ZZIFSPKFcCAZIO5J88CNRgR4nImf4CDqSEwHgZtc8=; b=7MzVkFBa4/wxCzYV3GqAiAeTrPSEhTpapmB3ixNB8B+8CYUIn0bsHRjHRw3hbntPiWU5ZT tpTGoDRn8iCqAmwjmIf3QkjIz8bmYw+6spaHHqh4HEW4K7qnQTH4b0twec0857bdLNw4EO JLJcs8rgcFrrtyZfybCSWi6Hxr17gCk= X-Rspamd-Queue-Id: 1A4891400A8 X-Stat-Signature: gpxjw6po1ge6t6tgksexnkf6k43u6d5s Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=VRt8PiD5; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3OMsCYwgKCPcrgZjddkafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--shakeelb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3OMsCYwgKCPcrgZjddkafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--shakeelb.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1661127481-452831 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With memcg v2 enabled, memcg->memory.usage is a very hot member for the workloads doing memcg charging on multiple CPUs concurrently. Particularly the network intensive workloads. In addition, there is a false cache sharing between memory.usage and memory.high on the charge path. This patch moves the usage into a separate cacheline and move all the read most fields into separate cacheline. To evaluate the impact of this optimization, on a 72 CPUs machine, we ran the following workload in a three level of cgroup hierarchy with top level having min and low setup appropriately. More specifically memory.min equal to size of netperf binary and memory.low double of that. $ netserver -6 # 36 instances of netperf with following params $ netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K Results (average throughput of netperf): Without (6.0-rc1) 10482.7 Mbps With patch 12413.7 Mbps (18.4% improvement) With the patch, the throughput improved by 18.4%. One side-effect of this patch is the increase in the size of struct mem_cgroup. However for the performance improvement, this additional size is worth it. In addition there are opportunities to reduce the size of struct mem_cgroup like deprecation of kmem and tcpmem page counters and better packing. Signed-off-by: Shakeel Butt Reported-by: kernel test robot Reviewed-by: Feng Tang Acked-by: Roman Gushchin --- include/linux/page_counter.h | 34 +++++++++++++++++++++++----------- 1 file changed, 23 insertions(+), 11 deletions(-) diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h index 679591301994..8ce99bde645f 100644 --- a/include/linux/page_counter.h +++ b/include/linux/page_counter.h @@ -3,15 +3,27 @@ #define _LINUX_PAGE_COUNTER_H #include +#include #include #include +#if defined(CONFIG_SMP) +struct pc_padding { + char x[0]; +} ____cacheline_internodealigned_in_smp; +#define PC_PADDING(name) struct pc_padding name +#else +#define PC_PADDING(name) +#endif + struct page_counter { + /* + * Make sure 'usage' does not share cacheline with any other field. The + * memcg->memory.usage is a hot member of struct mem_cgroup. + */ + PC_PADDING(_pad1_); atomic_long_t usage; - unsigned long min; - unsigned long low; - unsigned long high; - unsigned long max; + PC_PADDING(_pad2_); /* effective memory.min and memory.min usage tracking */ unsigned long emin; @@ -23,16 +35,16 @@ struct page_counter { atomic_long_t low_usage; atomic_long_t children_low_usage; - /* legacy */ unsigned long watermark; unsigned long failcnt; - /* - * 'parent' is placed here to be far from 'usage' to reduce - * cache false sharing, as 'usage' is written mostly while - * parent is frequently read for cgroup's hierarchical - * counting nature. - */ + /* Keep all the read most fields in a separete cacheline. */ + PC_PADDING(_pad3_); + + unsigned long min; + unsigned long low; + unsigned long high; + unsigned long max; struct page_counter *parent; }; From patchwork Mon Aug 22 00:17:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 12950093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A4F6C32772 for ; Mon, 22 Aug 2022 00:18:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E4E38E0002; Sun, 21 Aug 2022 20:18:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 293CA6B0074; Sun, 21 Aug 2022 20:18:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 15BEB8E0002; Sun, 21 Aug 2022 20:18:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 08A806B0073 for ; Sun, 21 Aug 2022 20:18:11 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D21441C5D44 for ; Mon, 22 Aug 2022 00:18:10 +0000 (UTC) X-FDA: 79825316340.11.8653EC2 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 7D3A3180051 for ; Mon, 22 Aug 2022 00:18:10 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-333f0d49585so159567387b3.9 for ; Sun, 21 Aug 2022 17:18:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc; bh=DTW77A/Vjl+d5bcQDIprq1tGD6UgniZYOTpaR8iNZIc=; b=Dg9yytO9IkPtQr0hKr1opAsmaHFl6Ti0qmk9dgNkQKa6EeZnhuamJ8wFuDDpvnSucO 3jnI66hn0RZluaprAxAC9vZCxkQXXdYA0KwOtMgEey7a9lQkqmoQ767sSys3ES+TUduk 8r9futG6QW922rVpAUoBuyvHCBDkXpj6Z6uIr5bt4pi4JjwABrXPfF7E2R9JF1EvrU3W 3HTuZx9jrCVbRGBo1EjBv/Hu+VbfEktqbj+TpV4cjusZQtsDY5OAzcQ1W3tgUYcjFPCH ebqBDDU8/54Qiztpi0pJPbzqNXn5hxGVXBNshftViCKjFt6eCv7LnIrPsIrM9eVLCdKl WwwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc; bh=DTW77A/Vjl+d5bcQDIprq1tGD6UgniZYOTpaR8iNZIc=; b=D/3rikxLvmpuwkR+2pT8egIcWyg67GAAenWb1TrQQtfszUMmUwlsfpEz86R8OZb4pE YsAb2pdFe1uzCpmQtclDdv7hgVKsSxhuy2UmmMIjkesULNEQxi/nsVGFfoctUP2kBg8O clfqOvIsLAgeWLrc42OoyoRo54RE91IS01X7x0E5Kw2M8ASOiS8jwVVccWW80NZTw0br IEXbniX1AdTZPLGyD0gaPr+pzgMBgK22xiY/+tB7BOmFw/58zOB88jvQghSlzGDh1fc4 raKEOZU9wOTU0Bp+TD+ugO02oiJFuybyMmYzfZKWCWeUpJbZqxA/ZaTyv3Hq5yGbQytt hLoA== X-Gm-Message-State: ACgBeo08FEUPM6bprL3Zzkpy9LHq8hgmLHvFndqQZtuzmUi8v2NXDB2X qnq1zb6AIAuaOzKDP7A2oh9llRsrECbiGw== X-Google-Smtp-Source: AA6agR6tW1MO8bdpOOCAdPjlI2KgLdp2empQdKN2rFu+N7JEzRBKSYp9P+zG59MACSGNvPbZ7uCstsPP+839Yw== X-Received: from shakeelb.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:28b]) (user=shakeelb job=sendgmr) by 2002:a25:9f8d:0:b0:686:9a3d:6f85 with SMTP id u13-20020a259f8d000000b006869a3d6f85mr16587811ybq.400.1661127489865; Sun, 21 Aug 2022 17:18:09 -0700 (PDT) Date: Mon, 22 Aug 2022 00:17:37 +0000 In-Reply-To: <20220822001737.4120417-1-shakeelb@google.com> Message-Id: <20220822001737.4120417-4-shakeelb@google.com> Mime-Version: 1.0 References: <20220822001737.4120417-1-shakeelb@google.com> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog Subject: [PATCH 3/3] memcg: increase MEMCG_CHARGE_BATCH to 64 From: Shakeel Butt To: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song Cc: " =?utf-8?q?Michal_Koutn=C3=BD?= " , Eric Dumazet , Soheil Hassas Yeganeh , Feng Tang , Oliver Sang , Andrew Morton , lkp@lists.01.org, cgroups@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Shakeel Butt ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661127490; a=rsa-sha256; cv=none; b=H6V/hD66mM1vQUiS67zOGiTrpF2iMckgvPbKDR+hYw7lj33/4A377Ysg/5+Oy3APD3G6Dh AaTl9rW4jtD49V5LsvV4xGLC6FikRNuW1HPZiQOVYCOflNylT0VmWV5QrcRMshWj141ZVt 7CDOHycEhZnvu1Z5ql7g7OQi90+eysI= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Dg9yytO9; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3QcsCYwgKCAIujcmggndiqqing.eqonkpwz-oomxcem.qti@flex--shakeelb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3QcsCYwgKCAIujcmggndiqqing.eqonkpwz-oomxcem.qti@flex--shakeelb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661127490; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DTW77A/Vjl+d5bcQDIprq1tGD6UgniZYOTpaR8iNZIc=; b=stF0efqVU8BvBhEpR1vPQdzMusAxM0dBAkg+aLMj2i05NotbIPis6kdNRYV7Pvd6Yrj17a stwkFfqlYnCITSgJPNwuK4MtXdZR2Wa92FhvDKtjBn9Mtjeohf7SoDpTMi3m6Xrxgwg+r/ agSLtWv/GmgstMYhQYDlFyeXJOW6u1s= X-Stat-Signature: uf7ahomqqzj9i89qx4hckqfxsbpoqb9c Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Dg9yytO9; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3QcsCYwgKCAIujcmggndiqqing.eqonkpwz-oomxcem.qti@flex--shakeelb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3QcsCYwgKCAIujcmggndiqqing.eqonkpwz-oomxcem.qti@flex--shakeelb.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 7D3A3180051 X-HE-Tag: 1661127490-613066 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For several years, MEMCG_CHARGE_BATCH was kept at 32 but with bigger machines and the network intensive workloads requiring througput in Gbps, 32 is too small and makes the memcg charging path a bottleneck. For now, increase it to 64 for easy acceptance to 6.0. We will need to revisit this in future for ever increasing demand of higher performance. Please note that the memcg charge path drain the per-cpu memcg charge stock, so there should not be any oom behavior change. To evaluate the impact of this optimization, on a 72 CPUs machine, we ran the following workload in a three level of cgroup hierarchy with top level having min and low setup appropriately. More specifically memory.min equal to size of netperf binary and memory.low double of that. $ netserver -6 # 36 instances of netperf with following params $ netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K Results (average throughput of netperf): Without (6.0-rc1) 10482.7 Mbps With patch 17064.7 Mbps (62.7% improvement) With the patch, the throughput improved by 62.7%. Signed-off-by: Shakeel Butt Reported-by: kernel test robot Acked-by: Soheil Hassas Yeganeh Reviewed-by: Feng Tang Acked-by: Michal Hocko Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4d31ce55b1c0..70ae91188e16 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -354,10 +354,11 @@ struct mem_cgroup { }; /* - * size of first charge trial. "32" comes from vmscan.c's magic value. - * TODO: maybe necessary to use big numbers in big irons. + * size of first charge trial. + * TODO: maybe necessary to use big numbers in big irons or dynamic based of the + * workload. */ -#define MEMCG_CHARGE_BATCH 32U +#define MEMCG_CHARGE_BATCH 64U extern struct mem_cgroup *root_mem_cgroup;