From patchwork Tue Jul 30 15:01:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13747518 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3910C3DA70 for ; Tue, 30 Jul 2024 15:02:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C7BF6B009A; Tue, 30 Jul 2024 11:02:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3505C6B00A1; Tue, 30 Jul 2024 11:02:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CAA06B00A2; Tue, 30 Jul 2024 11:02:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EC1F86B00A1 for ; Tue, 30 Jul 2024 11:02:37 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 89F84A43F7 for ; Tue, 30 Jul 2024 15:02:37 +0000 (UTC) X-FDA: 82396735554.17.B6EBB0E Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by imf25.hostedemail.com (Postfix) with ESMTP id 5F912A0037 for ; Tue, 30 Jul 2024 15:02:33 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=xGXw0ucR; spf=pass (imf25.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.173 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722351725; a=rsa-sha256; cv=none; b=V2KTg/6YVMqejpg2XIFeDy7Xgxa2Axlk0pTtlc5UF2i8FTYUK4ibDLXX2bjyQyEDWuxkDw CFbz9ryyNEx+jrKtOynfSojL0eiazeOvKlQtuDI7UGNQGKgEJ+DYTuaxxI7Ji3j7r5bKza WHnz/2pVGguhl5MD/N0GACi+CxEFG+w= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=xGXw0ucR; spf=pass (imf25.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.173 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722351725; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qK7WOu+hv7tl+cxtg7CXPtDPtjLG3K7kUlaYAaWdes0=; b=KaL9s6wjaKEgtJ2liIK2XT9XyZhGPsfXN/yfOIMQsZiuqcv8Px4HRxKsXPmbelfamyYjTW f4Tw6GhThDcG/Jn9UQ86O4vgZH3194huemJECJusBRFj7TqswDx4TLPYshA6+jBOtDkaR4 GKn5zNX0vEhss4tVnzOhNKfe3vk6E70= Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-7a1d0ad7113so319575385a.2 for ; Tue, 30 Jul 2024 08:02:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1722351752; x=1722956552; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=qK7WOu+hv7tl+cxtg7CXPtDPtjLG3K7kUlaYAaWdes0=; b=xGXw0ucRoSX3KJKbrlrXuxlkXqRPIekaaCR9PFcR5B87lYQeOyTKIUH2ZZQeHOSP9V LO6AjiZ1DTE2Bh1Xz32DE3SHkAyb1qL4qiooYK/h2dnVQYF9YCC1R3ql7X+Y3klnvGyu hUSe2d5DgPo0abtbciV74cVxeBlItdxktdi32Kjxk3OIWp4PuuaGII33IWFCxzhzVeMA RTH1yhl5JIXy/xA0zeeuH6lE3bJeaGiD3qKSvScaDIcQwnAWVOxEjAU6U1uUDhOzOZjI mc/kTm3BW1YQmJR5TTDjIeUZ2KMoXL3jy2ZrELWz8b0zPZh2VH6r9ZxPLaHNwm7nDXa8 cY1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722351752; x=1722956552; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qK7WOu+hv7tl+cxtg7CXPtDPtjLG3K7kUlaYAaWdes0=; b=wjOf086WSjbicmRAxRVR4/m4hox1IEcy2Gdtzrp7Be7IfQOY24cmycr/mArCbHgTDb 6JkHCcXir/TKh9A4OPA19jBX8BcozQMWNwQCTIzAdJpxAojEiaEmzEdQHdfCUBuIXxKd djcFZ14r846TiBVXjvp3wEzdTXBatKl/tW8zfiZe7/1m4NL3wLY36BHwEEZWJZUlFa7x UYEyO2dg84m2uA16m0RSiP5/hKH2kraeU5GsJDXPRk6QQdowp+Dffh07QXQytFzIzhIs XAvu31UjxtYJ7/Y9WB+FC9UiK8aD4Rc9MF5TvLCNRujlrfEeeWrztUuUeRJBJarYz6Qa 7mBg== X-Forwarded-Encrypted: i=1; AJvYcCVQYulriWIXsKHb/qyXUmyZKrrZYXGMdfbFA6vhIGTHmMjqlnTcm84aHls3jeucJEuP5RWxsq+PFFnN9GJEFDM/kWE= X-Gm-Message-State: AOJu0Yz+tCQsv9lBSkJaXt2dG6GbSYwKY87BdPcH/49enGnY/fmWe83p pnkrq1sNl6QsfgVgjo9BxeC9D09+d6cmPA41UD7pq6kTwvxn4Q6AqQJNxoPo2/o= X-Google-Smtp-Source: AGHT+IGuBFGt6MK7hI5v2XIYYwSP41dPGObuYRlJA5D+J3MWpue7Ez/s4WblQus1adMdjSSsNHR4iQ== X-Received: by 2002:a05:620a:179f:b0:79c:e7d:22b6 with SMTP id af79cd13be357-7a1e525fafbmr1390496785a.39.1722351722274; Tue, 30 Jul 2024 08:02:02 -0700 (PDT) Received: from soleen.c.googlers.com.com (197.5.86.34.bc.googleusercontent.com. [34.86.5.197]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a1d73efffdsm645934285a.69.2024.07.30.08.02.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jul 2024 08:02:01 -0700 (PDT) From: Pasha Tatashin To: akpm@linux-foundation.org, jpoimboe@kernel.org, pasha.tatashin@soleen.com, kent.overstreet@linux.dev, peterz@infradead.org, nphamcs@gmail.com, cerasuolodomenico@gmail.com, surenb@google.com, lizhijian@fujitsu.com, willy@infradead.org, shakeel.butt@linux.dev, vbabka@suse.cz, ziy@nvidia.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, yosryahmed@google.com Subject: [PATCH v6 1/3] memcg: increase the valid index range for memcg stats Date: Tue, 30 Jul 2024 15:01:56 +0000 Message-ID: <20240730150158.832783-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.46.0.rc1.232.g9752f9e123-goog In-Reply-To: <20240730150158.832783-1-pasha.tatashin@soleen.com> References: <20240730150158.832783-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Stat-Signature: 97pomtgkoyxm11h9g8m3gxi5iifxswry X-Rspamd-Queue-Id: 5F912A0037 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1722351753-869746 X-HE-Meta: U2FsdGVkX1+OYp/94hLI3CPHGH7FztXWONee81jNyty0+OZKFM/wzqlAqrS/u0cT+yUVdDmzj3q1ksagBJFfYkLV55HQbVY/hUgmuYRlPcqqgqWNmVmXyMLS0ksieD2tWVraANRW4RvHPWBADbO785Mq5TmQJXZc2Luu11pvYEKT2KXJAxaQD/0HhcBbJ1kptaArrOdRs/20EdvBstlExZV+fMVnDKDg/hlzgFt4sJ3WGq4d97AUQS56rzfaP8AWBjH7rz/jNTyZFRWF4eiRd6Z/gUeBwla6pG0RZZUQ7crbN7ZwJ/lkrLTuZ7GodV6qKDE3X2r9JzAtdIqc5T23BknIQWLuVAhNE1se++6LvPpS8zk4BihrlaQ5R01ocR5QBtIEepM5VvSnheNOc77qWJA3gvMjbJFF6zHdqo3tzotaFpyeP1T7Kj1KGXYuwEXnfhhMg5CQTfKz712cm9KOneV1jgR1PH2lmfWyrblNLyegL7giLYvofaaqO+C9aPhVccfapGEWxHfweGg355IpdFMNT8KiUeQr+y7FaUGkTiVI3zVijIwl+Xs1U+ssxO9qKbKMKcAm5RJoYKDHyLaLosrJdaiCtJxqvUOrqWy1G0YTiGVNX0A/7/WjJTsAU50b+Rnm0DlG53ls2QCa5QyT3LjMt6EUOYNz5MwWdDRlK3tYThG/85xFWMGtCWUbCDykeKdWeaB5GAuOqoD0AZf/z3rggxHOtB3+cR726xRWFhgm0jHW6nH+hxKr1qBFeWFEGU2cvxZPpGcxWPA32EDjHNpNggEAdKiQwczeD0d8IYdTyP88mnd8Ht60Jp3op5Y94ITcv3Zkfv5hA9tirQYi3LFENpZ0nvB0oGZh5234uR/1o13aYUo9w+BQ+j7gLUG83ceIPZqAFN+LylaZ8Kgtx9DsnR061ZYGjNry7VO71hfyn5dwnmgmXNEHmzAGqtowyAIXH5zPL0nFz3Qq6wg vHQu+GQP 8IheaI/8D87CQG2ifzutQDzrdCO+2aDDvOw87dDc9FDbG+IBbXSOHtyztXac7pvXPjMsETb68VW02y6Mr+j2bUKhHLq3sMMDXIdBhno4SrkVLpEgs6MYz7gmF5++K69XwAjZVeHTaa34rRFtKK5VL8AlxCpfSnwpbt2jAzk2pyaTg3KD6y2bZUJI5uwJeb3QDDYa26DxECNCvnmHVvcsbLtOplxZVpok1w1N7p9HWSXh54pYAyxTBGQyFN7flGc0+JZ2cF63FeOd7bgnwdLtFRj5HJXHmJ6twoX2yUSUptYtbJWSn0xFAgl6LObXUvgnE/Z2V3a6XF49HtHLV5dqN26Ujv8lyM5LiH+LhxV4uIkthrto= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000011, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Shakeel Butt At the moment the valid index for the indirection tables for memcg stats and events is < S8_MAX. These indirection tables are used in performance critical codepaths. With the latest addition to the vm_events, the NR_VM_EVENT_ITEMS has gone over S8_MAX. One way to resolve is to increase the entry size of the indirection table from int8_t to int16_t but this will increase the potential number of cachelines needed to access the indirection table. This patch took a different approach and make the valid index < U8_MAX. In this way the size of the indirection tables will remain same and we only need to invalid index check from less than 0 to equal to U8_MAX. In this approach we have also removed a subtraction from the performance critical codepaths. Signed-off-by: Shakeel Butt Co-developed-by: Pasha Tatashin Signed-off-by: Pasha Tatashin Reviewed-by: Yosry Ahmed --- mm/memcontrol.c | 50 +++++++++++++++++++++++++++---------------------- 1 file changed, 28 insertions(+), 22 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 960371788687..84f383952d32 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -320,24 +320,27 @@ static const unsigned int memcg_stat_items[] = { #define NR_MEMCG_NODE_STAT_ITEMS ARRAY_SIZE(memcg_node_stat_items) #define MEMCG_VMSTAT_SIZE (NR_MEMCG_NODE_STAT_ITEMS + \ ARRAY_SIZE(memcg_stat_items)) -static int8_t mem_cgroup_stats_index[MEMCG_NR_STAT] __read_mostly; +#define BAD_STAT_IDX(index) ((u32)(index) >= U8_MAX) +static u8 mem_cgroup_stats_index[MEMCG_NR_STAT] __read_mostly; static void init_memcg_stats(void) { - int8_t i, j = 0; + u8 i, j = 0; - BUILD_BUG_ON(MEMCG_NR_STAT >= S8_MAX); + BUILD_BUG_ON(MEMCG_NR_STAT >= U8_MAX); - for (i = 0; i < NR_MEMCG_NODE_STAT_ITEMS; ++i) - mem_cgroup_stats_index[memcg_node_stat_items[i]] = ++j; + memset(mem_cgroup_stats_index, U8_MAX, sizeof(mem_cgroup_stats_index)); - for (i = 0; i < ARRAY_SIZE(memcg_stat_items); ++i) - mem_cgroup_stats_index[memcg_stat_items[i]] = ++j; + for (i = 0; i < NR_MEMCG_NODE_STAT_ITEMS; ++i, ++j) + mem_cgroup_stats_index[memcg_node_stat_items[i]] = j; + + for (i = 0; i < ARRAY_SIZE(memcg_stat_items); ++i, ++j) + mem_cgroup_stats_index[memcg_stat_items[i]] = j; } static inline int memcg_stats_index(int idx) { - return mem_cgroup_stats_index[idx] - 1; + return mem_cgroup_stats_index[idx]; } struct lruvec_stats_percpu { @@ -369,7 +372,7 @@ unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx) return node_page_state(lruvec_pgdat(lruvec), idx); i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return 0; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); @@ -392,7 +395,7 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return 0; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); @@ -435,21 +438,24 @@ static const unsigned int memcg_vm_event_stat[] = { }; #define NR_MEMCG_EVENTS ARRAY_SIZE(memcg_vm_event_stat) -static int8_t mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly; +static u8 mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly; static void init_memcg_events(void) { - int8_t i; + u8 i; + + BUILD_BUG_ON(NR_VM_EVENT_ITEMS >= U8_MAX); - BUILD_BUG_ON(NR_VM_EVENT_ITEMS >= S8_MAX); + memset(mem_cgroup_events_index, U8_MAX, + sizeof(mem_cgroup_events_index)); for (i = 0; i < NR_MEMCG_EVENTS; ++i) - mem_cgroup_events_index[memcg_vm_event_stat[i]] = i + 1; + mem_cgroup_events_index[memcg_vm_event_stat[i]] = i; } static inline int memcg_events_index(enum vm_event_item idx) { - return mem_cgroup_events_index[idx] - 1; + return mem_cgroup_events_index[idx]; } struct memcg_vmstats_percpu { @@ -621,7 +627,7 @@ unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) long x; int i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return 0; x = READ_ONCE(memcg->vmstats->state[i]); @@ -662,7 +668,7 @@ void __mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx, if (mem_cgroup_disabled()) return; - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return; __this_cpu_add(memcg->vmstats_percpu->state[i], val); @@ -675,7 +681,7 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) long x; int i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return 0; x = READ_ONCE(memcg->vmstats->state_local[i]); @@ -694,7 +700,7 @@ static void __mod_memcg_lruvec_state(struct lruvec *lruvec, struct mem_cgroup *memcg; int i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); @@ -810,7 +816,7 @@ void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, if (mem_cgroup_disabled()) return; - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return; memcg_stats_lock(); @@ -823,7 +829,7 @@ unsigned long memcg_events(struct mem_cgroup *memcg, int event) { int i = memcg_events_index(event); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, event)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, event)) return 0; return READ_ONCE(memcg->vmstats->events[i]); @@ -833,7 +839,7 @@ unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) { int i = memcg_events_index(event); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, event)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, event)) return 0; return READ_ONCE(memcg->vmstats->events_local[i]);