From patchwork Tue Aug 30 05:59:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 12958779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B72AECAAD5 for ; Tue, 30 Aug 2022 06:00:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7744D6B0074; Tue, 30 Aug 2022 02:00:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D660940007; Tue, 30 Aug 2022 02:00:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 529386B0078; Tue, 30 Aug 2022 02:00:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3AD576B0074 for ; Tue, 30 Aug 2022 02:00:10 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 1635DA0C5A for ; Tue, 30 Aug 2022 06:00:10 +0000 (UTC) X-FDA: 79855208580.11.0D29650 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf05.hostedemail.com (Postfix) with ESMTP id B2CF410002A for ; Tue, 30 Aug 2022 06:00:09 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id d12so10100349plr.6 for ; Mon, 29 Aug 2022 23:00:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc; bh=Q/bOSV74zP/orRWcu6mpaXasvBllKooATbpIY/UJY9Y=; b=iaukGuuSL1/LSoa2a8zvNDakcBpOAs7MfnIJteuVEJ5NQBob0p1ommjQMhEd5LlEoE 3htxWDIFRvRg9I/rJJYiCqOKiqhZQuESXYz630zXko5IAx1WbGVcshFimeZXWGNB/SnY 5mK/DZ0y9jYAiJ9QBGv+phsOfZG9PcdxwMV469eWeGhIZ95oTwQdHIbM4Dt1LKaGt6FH 881oKRZvjgD5NcinenMX6woRhwbU5swnfG93gJTk69hAqwQ/f1WpTdYSuaXVBIuVuLGx 8pSlJRwM8GFYQHytfTX0l4ArbkcaUnvJiJX1wDG4/wmZc6NG52kB5JqK3ebs3dyjcAr2 kMBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc; bh=Q/bOSV74zP/orRWcu6mpaXasvBllKooATbpIY/UJY9Y=; b=KSnNfOitzP+CbR7ZhrZGzdz4clHs5xkglv87+2zlo/8bk7Z+wo+V3eQlusf4GM43If /xAQiZrRHwo/lbNxy/+EUSSrR1qSRvoMme8XEPtWQiIp7vKJ7+0eyxa4FJ/5FqUGPzAd NA4B3Kkmudsf20yrbEDwKOlphb2B/oUXvlbzhkM3XOK0eVL97/mdX/mRetDuWYmvOg+z mXKonv9GGqxEDffK5QWO5Ml9uBU08zyNy19XE//t+KLs0QQ4lyTr43QvAW59s+ySiGNE zrTZzs9AFtluK66XiSiqGxuj3DitmYHyZbjcGPPUCA1ljdrBSFDzezmXDIdIUMoCEKNc wtlA== X-Gm-Message-State: ACgBeo0EoOPgsUFISoC8fmiD0YDfguIKTO7y7IEvnZxAsct2m7XvAy15 aLvRMt1jZUbPw99iBwrNGZg= X-Google-Smtp-Source: AA6agR7PlXIXR4sqyokZ+7EQDmI6EE9outPPHOElmTSglOYEtJUS2Ps5iwrspbimzGdWpiFrjO8XcA== X-Received: by 2002:a17:90b:3ec9:b0:1f5:7e38:1fd1 with SMTP id rm9-20020a17090b3ec900b001f57e381fd1mr21952406pjb.101.1661839208721; Mon, 29 Aug 2022 23:00:08 -0700 (PDT) Received: from KASONG-MB0.tencent.com ([103.7.29.31]) by smtp.gmail.com with ESMTPSA id m24-20020a17090b069800b001fd962747d4sm5311546pjz.30.2022.08.29.23.00.05 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 29 Aug 2022 23:00:08 -0700 (PDT) From: Kairui Song To: cgroups@vger.kernel.org, linux-mm@kvack.org Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 1/2] mm: memcontrol: remove mem_cgroup_kmem_disabled Date: Tue, 30 Aug 2022 13:59:48 +0800 Message-Id: <20220830055949.12640-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220830055949.12640-1-ryncsn@gmail.com> References: <20220830055949.12640-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661839209; a=rsa-sha256; cv=none; b=Z8C+LHjF7C5cMEirknh5sgdQkc0k3lQ1Li/cQb0bjcUYD/Srj/A3q6Gp7+a6pmRnavSMH9 tdY13eYWyJqBwKg1jAvHWyzaG7jsxR9I/fDUyrP0AbFThslyLJh/TjDRkGe7LGHQqhbXbC yA7oohR4jDNEZqn8qdspGWKd+inWSOw= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=iaukGuuS; spf=pass (imf05.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661839209; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q/bOSV74zP/orRWcu6mpaXasvBllKooATbpIY/UJY9Y=; b=cr6kCrxZydZccBx+ky3zaZzkkieJSkdaXQhV7DDFIqDdBYC83ykKacIPhKUriwX1Xa8rUK OLfdUG9wfgWvENo9JNFjCwnCd3UvCtb47MNCPga+9Dgw6BHRobXXsBsw+E8EciWUcOnfXh pGvS7XM/XMIJobmzX7Hpx+2LuIOK7x4= Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=iaukGuuS; spf=pass (imf05.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam05 X-Stat-Signature: f7tq4y73t8ujicxyakfzsd7fh6egnq6z X-Rspamd-Queue-Id: B2CF410002A X-Rspam-User: X-HE-Tag: 1661839209-855904 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kairui Song There are currently two helpers for checking if cgroup kmem accounting is enabled: - mem_cgroup_kmem_disabled - memcg_kmem_enabled mem_cgroup_kmem_disabled is a simple helper that returns true if cgroup.memory=nokmem is specified, otherwise returns false. memcg_kmem_enabled is a bit different, it returns true if cgroup.memory=nokmem is not specified and there is at least one non-root cgroup ever created. And once there is any non-root memcg created, it won't go back to return false again. This may help improve performance for some corner use cases where the user enables memory cgroup and kmem accounting globally but never create any cgroup. Considering that corner case is rare, especially nowadays cgroup is widely used as a standard way to organize services. And the "once enabled never disable" behavior is kind of strange. This commit simplifies the behavior of memcg_kmem_enabled, making it simply the opposite of mem_cgroup_kmem_disabled, always true if cgroup.memory=nokmem is not specified. So mem_cgroup_kmem_disabled can be dropped. This simplifies the code, and besides, memcg_kmem_enabled makes use of static key so it has a lower overhead. Signed-off-by: Kairui Song Reported-by: kernel test robot --- include/linux/memcontrol.h | 8 +------- mm/memcontrol.c | 17 +++++++---------- mm/percpu.c | 2 +- mm/slab_common.c | 2 +- 4 files changed, 10 insertions(+), 19 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 6257867fbf953..9c08464ed6b46 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1731,7 +1731,6 @@ static inline void set_shrinker_bit(struct mem_cgroup *memcg, #endif #ifdef CONFIG_MEMCG_KMEM -bool mem_cgroup_kmem_disabled(void); int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge_page(struct page *page, int order); @@ -1779,7 +1778,7 @@ static inline void count_objcg_event(struct obj_cgroup *objcg, { struct mem_cgroup *memcg; - if (mem_cgroup_kmem_disabled()) + if (!memcg_kmem_enabled()) return; rcu_read_lock(); @@ -1825,11 +1824,6 @@ static inline struct mem_cgroup *mem_cgroup_or_root(struct mem_cgroup *memcg) return memcg ? memcg : root_mem_cgroup; } #else -static inline bool mem_cgroup_kmem_disabled(void) -{ - return true; -} - static inline int memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b69979c9ced5c..20e26ccd7dddc 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -86,7 +86,7 @@ EXPORT_PER_CPU_SYMBOL_GPL(int_active_memcg); static bool cgroup_memory_nosocket __ro_after_init; /* Kernel memory accounting disabled? */ -static bool cgroup_memory_nokmem __ro_after_init; +static bool cgroup_memory_nokmem __initdata; /* Whether the swap controller is active */ #ifdef CONFIG_MEMCG_SWAP @@ -255,11 +255,6 @@ struct mem_cgroup *vmpressure_to_memcg(struct vmpressure *vmpr) #ifdef CONFIG_MEMCG_KMEM static DEFINE_SPINLOCK(objcg_lock); -bool mem_cgroup_kmem_disabled(void) -{ - return cgroup_memory_nokmem; -} - static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, unsigned int nr_pages); @@ -3667,7 +3662,7 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) { struct obj_cgroup *objcg; - if (mem_cgroup_kmem_disabled()) + if (!memcg_kmem_enabled()) return 0; if (unlikely(mem_cgroup_is_root(memcg))) @@ -3680,8 +3675,6 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) objcg->memcg = memcg; rcu_assign_pointer(memcg->objcg, objcg); - static_branch_enable(&memcg_kmem_enabled_key); - memcg->kmemcg_id = memcg->id.id; return 0; @@ -3691,7 +3684,7 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) { struct mem_cgroup *parent; - if (mem_cgroup_kmem_disabled()) + if (!memcg_kmem_enabled()) return; if (unlikely(mem_cgroup_is_root(memcg))) @@ -7153,6 +7146,10 @@ static int __init cgroup_memory(char *s) if (!strcmp(token, "nokmem")) cgroup_memory_nokmem = true; } + + if (!cgroup_memory_nokmem) + static_branch_enable(&memcg_kmem_enabled_key); + return 1; } __setup("cgroup.memory=", cgroup_memory); diff --git a/mm/percpu.c b/mm/percpu.c index 27697b2429c2e..c62d6e98f7d20 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1467,7 +1467,7 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) goto md_blocks_fail; #ifdef CONFIG_MEMCG_KMEM - if (!mem_cgroup_kmem_disabled()) { + if (memcg_kmem_enabled()) { chunk->obj_cgroups = pcpu_mem_zalloc(pcpu_chunk_map_bits(chunk) * sizeof(struct obj_cgroup *), gfp); diff --git a/mm/slab_common.c b/mm/slab_common.c index 17996649cfe3e..bbdc0fe3c5e34 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -829,7 +829,7 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags) if (type == KMALLOC_RECLAIM) { flags |= SLAB_RECLAIM_ACCOUNT; } else if (IS_ENABLED(CONFIG_MEMCG_KMEM) && (type == KMALLOC_CGROUP)) { - if (mem_cgroup_kmem_disabled()) { + if (!memcg_kmem_enabled()) { kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx]; return; } From patchwork Tue Aug 30 05:59:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 12958778 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F11CECAAD1 for ; Tue, 30 Aug 2022 06:00:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96FF96B0075; Tue, 30 Aug 2022 02:00:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F9D2940007; Tue, 30 Aug 2022 02:00:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 772246B007B; Tue, 30 Aug 2022 02:00:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 65CC56B0075 for ; Tue, 30 Aug 2022 02:00:13 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 0F7301A0BE4 for ; Tue, 30 Aug 2022 06:00:13 +0000 (UTC) X-FDA: 79855208706.13.E6E54C8 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by imf01.hostedemail.com (Postfix) with ESMTP id B82324000D for ; Tue, 30 Aug 2022 06:00:12 +0000 (UTC) Received: by mail-pg1-f181.google.com with SMTP id q63so9736199pga.9 for ; Mon, 29 Aug 2022 23:00:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc; bh=+u8MsnXVLuPRXhuPrr3jHb9wYGXAHxn2UH8zEXx8loM=; b=WRDjKb7fg8Tsj3l+ddy3zDk90jir/6RXBcU4yHYfsGG62r8O4snu+Wj3ld1oCPOLrI dQwhSqGyI0L68HyBUz5dJXZuEAFPv2/1YywtYnnNlJVqZ7fyQvf/xWot6Gd32Uu54COH qlza/MtKuW+craGIfFi8rOSJsELzaJpPy+bKavkFcqN53gYK35drpsntz99ESHveiF2w yMkkTBQLTbDGPVTy517HM+mKWwnLlRVmMF3M/LsJunBt5gfQx8XHJ3p6GHcHq5prStXT EOBSSsqA9w9pZWwHBWllzf4gm09jt+fGobriEMn7+JAwQ3xD1nsZPLLjDCmhboRsRVHn ovXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc; bh=+u8MsnXVLuPRXhuPrr3jHb9wYGXAHxn2UH8zEXx8loM=; b=6ofMPhjldPUzYeDQOHweEyqmA4Crvuz+t1I843Avmaqp8YI/Iwjts0yFmG7x0gHaYx UFSU6BtI1XPcKdSVMbO9RlAOTGsT6buawGpdNu9X13DPb5CcDQBOsSo7rjIRv0A2+Asf e/fsionBuE28FnMiiib8Z0yNAYGwoEzs2s+iC3mu6LthnTs/EyK+w8aM9NVQKdEWudWT bxGy+pKPQsJ/J5j5XIgNDZYypZ4vHQRE2SliLUAoh52CzV48gH7caPLdMMy8vDFvYXIJ R6D+LV8EKEZtzeuR1wIbyX9NKUj20KzjTlhW7y1iHIoPMWCEeCfIeNPh5K5K9ysnrEy6 s4IQ== X-Gm-Message-State: ACgBeo2zZUZcROMSTQ9a4z7z6bB72PtF9AJoLx8tsZU8aIs0CBW7HtrK 1mIxLeLRHhps4rWeI/8qFEfaJgSwR4lDGsI7 X-Google-Smtp-Source: AA6agR5ihjHu4632B9Qdv1JRLYurWHsyNOOp5mcpUyDOk0w8f09JiUu5xJ4j/A/OqHhnf3Kza6U52g== X-Received: by 2002:a05:6a00:1681:b0:52e:d9b1:8596 with SMTP id k1-20020a056a00168100b0052ed9b18596mr19969441pfc.56.1661839211830; Mon, 29 Aug 2022 23:00:11 -0700 (PDT) Received: from KASONG-MB0.tencent.com ([103.7.29.31]) by smtp.gmail.com with ESMTPSA id m24-20020a17090b069800b001fd962747d4sm5311546pjz.30.2022.08.29.23.00.09 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 29 Aug 2022 23:00:11 -0700 (PDT) From: Kairui Song To: cgroups@vger.kernel.org, linux-mm@kvack.org Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 2/2] mm: memcontrol: make cgroup_memory_noswap a static key Date: Tue, 30 Aug 2022 13:59:49 +0800 Message-Id: <20220830055949.12640-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220830055949.12640-1-ryncsn@gmail.com> References: <20220830055949.12640-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661839212; a=rsa-sha256; cv=none; b=QExyymUvgiCqbwJ/LtRM4hNN/CFNOtAQsTgLj4LCU7OCMN1LahL7VRPjFHlS+v/0Z6of8F p0If5J0I7FL2zi2XwOkiv1kW7yOtF226xONSRMfN9fqPQLNXyYu0mSFMXQZcqrVxi2BpOC CGdWJL3UM8c1hUBi+BOXNf8RIJRe4dY= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=WRDjKb7f; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661839212; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+u8MsnXVLuPRXhuPrr3jHb9wYGXAHxn2UH8zEXx8loM=; b=de3NmTOh1Qa56MaPZDtUVN/uTG63nz8lZhg0hsJuO3+Q8YX1Kh5eHCM7soLyKzdeC9nbKk TNM+le71lkDY1VPbc0M1hUEePQYinHINthNr1v+KedZI3tbWSdUWAtgEP4A6btXY7lIDtX oWoPZj4c72yi+901r2qg5K4Ybbko65I= Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=WRDjKb7f; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Stat-Signature: f514gkwfazgd9cw33may53njc4e4gxct X-Rspamd-Queue-Id: B82324000D X-HE-Tag: 1661839212-15870 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kairui Song cgroup_memory_noswap is used in many hot path, so make it a static key to lower the kernel overhead. Using 8G of ZRAM as SWAP, benchmark using `perf stat -d -d -d --repeat 100` with the following code snip in a non-root cgroup: #include #include #include #include #define MB 1024UL * 1024UL int main(int argc, char **argv){ void *p = mmap(NULL, 8000 * MB, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); memset(p, 0xff, 8000 * MB); madvise(p, 8000 * MB, MADV_PAGEOUT); memset(p, 0xff, 8000 * MB); return 0; } Before: 7,021.43 msec task-clock # 0.967 CPUs utilized ( +- 0.03% ) 4,010 context-switches # 573.853 /sec ( +- 0.01% ) 0 cpu-migrations # 0.000 /sec 2,052,057 page-faults # 293.661 K/sec ( +- 0.00% ) 12,616,546,027 cycles # 1.805 GHz ( +- 0.06% ) (39.92%) 156,823,666 stalled-cycles-frontend # 1.25% frontend cycles idle ( +- 0.10% ) (40.25%) 310,130,812 stalled-cycles-backend # 2.47% backend cycles idle ( +- 4.39% ) (40.73%) 18,692,516,591 instructions # 1.49 insn per cycle # 0.01 stalled cycles per insn ( +- 0.04% ) (40.75%) 4,907,447,976 branches # 702.283 M/sec ( +- 0.05% ) (40.30%) 13,002,578 branch-misses # 0.26% of all branches ( +- 0.08% ) (40.48%) 7,069,786,296 L1-dcache-loads # 1.012 G/sec ( +- 0.03% ) (40.32%) 649,385,847 L1-dcache-load-misses # 9.13% of all L1-dcache accesses ( +- 0.07% ) (40.10%) 1,485,448,688 L1-icache-loads # 212.576 M/sec ( +- 0.15% ) (39.49%) 31,628,457 L1-icache-load-misses # 2.13% of all L1-icache accesses ( +- 0.40% ) (39.57%) 6,667,311 dTLB-loads # 954.129 K/sec ( +- 0.21% ) (39.50%) 5,668,555 dTLB-load-misses # 86.40% of all dTLB cache accesses ( +- 0.12% ) (39.03%) 765 iTLB-loads # 109.476 /sec ( +- 21.81% ) (39.44%) 4,370,351 iTLB-load-misses # 214320.09% of all iTLB cache accesses ( +- 1.44% ) (39.86%) 149,207,254 L1-dcache-prefetches # 21.352 M/sec ( +- 0.13% ) (40.27%) 7.25869 +- 0.00203 seconds time elapsed ( +- 0.03% ) After: 6,576.16 msec task-clock # 0.953 CPUs utilized ( +- 0.10% ) 4,020 context-switches # 605.595 /sec ( +- 0.01% ) 0 cpu-migrations # 0.000 /sec 2,052,056 page-faults # 309.133 K/sec ( +- 0.00% ) 11,967,619,180 cycles # 1.803 GHz ( +- 0.36% ) (38.76%) 161,259,240 stalled-cycles-frontend # 1.38% frontend cycles idle ( +- 0.27% ) (36.58%) 253,605,302 stalled-cycles-backend # 2.16% backend cycles idle ( +- 4.45% ) (34.78%) 19,328,171,892 instructions # 1.65 insn per cycle # 0.01 stalled cycles per insn ( +- 0.10% ) (31.46%) 5,213,967,902 branches # 785.461 M/sec ( +- 0.18% ) (30.68%) 12,385,170 branch-misses # 0.24% of all branches ( +- 0.26% ) (34.13%) 7,271,687,822 L1-dcache-loads # 1.095 G/sec ( +- 0.12% ) (35.29%) 649,873,045 L1-dcache-load-misses # 8.93% of all L1-dcache accesses ( +- 0.11% ) (41.41%) 1,950,037,608 L1-icache-loads # 293.764 M/sec ( +- 0.33% ) (43.11%) 31,365,566 L1-icache-load-misses # 1.62% of all L1-icache accesses ( +- 0.39% ) (45.89%) 6,767,809 dTLB-loads # 1.020 M/sec ( +- 0.47% ) (48.42%) 6,339,590 dTLB-load-misses # 95.43% of all dTLB cache accesses ( +- 0.50% ) (46.60%) 736 iTLB-loads # 110.875 /sec ( +- 1.79% ) (48.60%) 4,314,836 iTLB-load-misses # 518653.73% of all iTLB cache accesses ( +- 0.63% ) (42.91%) 144,950,156 L1-dcache-prefetches # 21.836 M/sec ( +- 0.37% ) (41.39%) 6.89935 +- 0.00703 seconds time elapsed ( +- 0.10% ) The performance is clearly better. Signed-off-by: Kairui Song Acked-by: Michal Hocko --- mm/memcontrol.c | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 20e26ccd7dddc..8ea5589345a14 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -90,9 +90,18 @@ static bool cgroup_memory_nokmem __initdata; /* Whether the swap controller is active */ #ifdef CONFIG_MEMCG_SWAP -static bool cgroup_memory_noswap __ro_after_init; +static bool cgroup_memory_noswap __initdata; + +static DEFINE_STATIC_KEY_FALSE(memcg_swap_enabled_key); +static inline bool memcg_swap_enabled(void) +{ + return static_branch_likely(&memcg_swap_enabled_key); +} #else -#define cgroup_memory_noswap 1 +static inline bool memcg_swap_enabled(void) +{ + return false; +} #endif #ifdef CONFIG_CGROUP_WRITEBACK @@ -102,7 +111,7 @@ static DECLARE_WAIT_QUEUE_HEAD(memcg_cgwb_frn_waitq); /* Whether legacy memory+swap accounting is active */ static bool do_memsw_account(void) { - return !cgroup_subsys_on_dfl(memory_cgrp_subsys) && !cgroup_memory_noswap; + return !cgroup_subsys_on_dfl(memory_cgrp_subsys) && memcg_swap_enabled(); } #define THRESHOLDS_EVENTS_TARGET 128 @@ -7264,7 +7273,7 @@ void mem_cgroup_swapout(struct folio *folio, swp_entry_t entry) if (!mem_cgroup_is_root(memcg)) page_counter_uncharge(&memcg->memory, nr_entries); - if (!cgroup_memory_noswap && memcg != swap_memcg) { + if (memcg_swap_enabled() && memcg != swap_memcg) { if (!mem_cgroup_is_root(swap_memcg)) page_counter_charge(&swap_memcg->memsw, nr_entries); page_counter_uncharge(&memcg->memsw, nr_entries); @@ -7316,7 +7325,7 @@ int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry) memcg = mem_cgroup_id_get_online(memcg); - if (!cgroup_memory_noswap && !mem_cgroup_is_root(memcg) && + if (memcg_swap_enabled() && !mem_cgroup_is_root(memcg) && !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) { memcg_memory_event(memcg, MEMCG_SWAP_MAX); memcg_memory_event(memcg, MEMCG_SWAP_FAIL); @@ -7348,7 +7357,7 @@ void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages) rcu_read_lock(); memcg = mem_cgroup_from_id(id); if (memcg) { - if (!cgroup_memory_noswap && !mem_cgroup_is_root(memcg)) { + if (memcg_swap_enabled() && !mem_cgroup_is_root(memcg)) { if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) page_counter_uncharge(&memcg->swap, nr_pages); else @@ -7364,7 +7373,7 @@ long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg) { long nr_swap_pages = get_nr_swap_pages(); - if (cgroup_memory_noswap || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) + if (!memcg_swap_enabled() || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) return nr_swap_pages; for (; memcg != root_mem_cgroup; memcg = parent_mem_cgroup(memcg)) nr_swap_pages = min_t(long, nr_swap_pages, @@ -7381,7 +7390,7 @@ bool mem_cgroup_swap_full(struct page *page) if (vm_swap_full()) return true; - if (cgroup_memory_noswap || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) + if (!memcg_swap_enabled() || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) return false; memcg = page_memcg(page); @@ -7689,6 +7698,8 @@ static int __init mem_cgroup_swap_init(void) if (cgroup_memory_noswap) return 0; + static_branch_enable(&memcg_swap_enabled_key); + WARN_ON(cgroup_add_dfl_cftypes(&memory_cgrp_subsys, swap_files)); WARN_ON(cgroup_add_legacy_cftypes(&memory_cgrp_subsys, memsw_files)); #if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP)