From patchwork Wed Nov 27 02:57:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13886507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4096FD66BAD for ; Wed, 27 Nov 2024 02:57:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AEF9B6B009D; Tue, 26 Nov 2024 21:57:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A78F96B009E; Tue, 26 Nov 2024 21:57:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A44F6B00A0; Tue, 26 Nov 2024 21:57:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6760E6B009D for ; Tue, 26 Nov 2024 21:57:48 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 122E7814B4 for ; Wed, 27 Nov 2024 02:57:48 +0000 (UTC) X-FDA: 82830364848.22.388111A Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf26.hostedemail.com (Postfix) with ESMTP id 1E4B3140019 for ; Wed, 27 Nov 2024 02:57:41 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=p1DNpvqT; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3qIpGZwcKCBkNJzC16J5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--yuanchu.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3qIpGZwcKCBkNJzC16J5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--yuanchu.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732676263; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YMfZReVH6Y6OSe0PATfKlnZ+c8+zk2ht0ykfo5UOUYo=; b=qoR2YT6csg4cZlraj4XakbEYF/rG55afL30YM0iWfRF1tYzP3NtTuO/1hHa8TDy0/psXOT GprXlyW5hxu4+OZbnr3xQLJFu3QuSMK8b2FXw2pZFzR/5qbxbs2ai3WsO/WrA/I2uaWAhU d8B8UizBweB+TC/7JvaROrhnxrv5pDQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732676263; a=rsa-sha256; cv=none; b=RiWrw8pKheCvDFhD6dgL7POHGIlfLJOKFg3tllbumtz5VC8xpITzPCKEUs6memcUYKAjdv 6Y1u8+PU+oRE/my+F7PZ1jG0IiC8lAvLSGXMU8WGJ2WXnGrmK14XlLcKQfC+Tlr+j0Aydo q8TmrpFkP7ZrRjc6JJCRH4z8W8vRXI0= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=p1DNpvqT; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3qIpGZwcKCBkNJzC16J5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--yuanchu.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3qIpGZwcKCBkNJzC16J5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--yuanchu.bounces.google.com Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-724f5009c7dso3593958b3a.2 for ; Tue, 26 Nov 2024 18:57:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1732676265; x=1733281065; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YMfZReVH6Y6OSe0PATfKlnZ+c8+zk2ht0ykfo5UOUYo=; b=p1DNpvqTG9ZMPAY3uLqfEBZWFFN0Sxqxk3aQZpIn2o5KlCNrID4T1RLzcieiU+afkq FZGGFlILyrzyHRuP3HceQKQ8f+dNtHwBBx7nHeyNAnWlVXty6zpd7BAeNdmi59P5A8C7 UqM7XZ4huyOJg9u2NP9f7k3Q53qZ+gjhS1hQMy3ztwgdA2ptBU7SRBnrs91mmNUX9MH4 5+N+zva+S9kPHrDRuud/8s0tdPG5YBFAnOk5QjqaLgOtrHr+AvkmnntlXgRw6LTqO3Pp u1wsuRgvkVa5p7ct23LaslEHM7+0dCXChRHwtY557LbOhBZ3ZPqS+So+Gd2imTDBL2TU XrRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732676265; x=1733281065; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YMfZReVH6Y6OSe0PATfKlnZ+c8+zk2ht0ykfo5UOUYo=; b=CUs0oyEK2gr03+ODnY6QXuRATJqKuFJKd7ywhVQStrQayRElXy8alvsGZsDO9UxIGZ ShhRnC0ozYvfUxCh9oaff0l0LPnIQw6/gvvY4YV1ADKNKd++5ev0yamH9w+VWRe0wXKI Ul3BDSI6noDrsx8eW0YNdh4zO69QU4tNkrT8qmBK3OERE6iNq2gAlzS91Wyq9WqfSiBE 7j4F1+EweBsW/7KkLb/pCj+39LggaiAwPol7NJHoRL533BoT7fHSI4whQYi5FVmEWcRY 7/+16armay3zV4qp6wHpl/vIJdvZxSD5ZSRYvrNE8CuHJ7GL8LQn2KRDW+5tTs/lg2AP Kzvg== X-Forwarded-Encrypted: i=1; AJvYcCX311UImaMiQ8IpiVcYqD9QpR+vrKkI4BtRNAix5kO/x6c5UlmfJclEGyiakWR7UgYZ9J43pulPOQ==@kvack.org X-Gm-Message-State: AOJu0YwQyWOVeSCmbNLj6c676b0KyRyEfEerAO+O0ooG+StrfCKeMTD7 aPhfOtAkgNZSgQwcVj9eVMDtUehjFiJU44j6iPvp/9lcEgcttCTagl8vHuiaMmATrWvdC+5BLPK LQFo5gQ== X-Google-Smtp-Source: AGHT+IGS3z2xtRaD82Eg6lpZpx9W4L7i5UELBQBxL0/M15PzUsyV5kTYfTLGRUtmYRU55oTDmRTrliUGszQh X-Received: from pjtd4.prod.google.com ([2002:a17:90b:44:b0:2e2:8d64:6213]) (user=yuanchu job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2b47:b0:2ea:49a8:917b with SMTP id 98e67ed59e1d1-2ee08dab307mr2422606a91.0.1732676264733; Tue, 26 Nov 2024 18:57:44 -0800 (PST) Date: Tue, 26 Nov 2024 18:57:20 -0800 In-Reply-To: <20241127025728.3689245-1-yuanchu@google.com> Mime-Version: 1.0 References: <20241127025728.3689245-1-yuanchu@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241127025728.3689245-2-yuanchu@google.com> Subject: [PATCH v4 1/9] mm: aggregate workingset information into histograms From: Yuanchu Xie To: Andrew Morton , David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Tejun Heo , Johannes Weiner , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , " =?utf-8?q?Eugenio_P=C3=A9rez?= " , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Mike Rapoport , Shuah Khan , Christian Brauner , Daniel Watson , Yuanchu Xie , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, linux-mm@kvack.org, linux-kselftest@vger.kernel.org X-Stat-Signature: uqtimremymihufu5h4pchfr14497ix6d X-Rspamd-Queue-Id: 1E4B3140019 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1732676261-263983 X-HE-Meta: U2FsdGVkX18EnKD9nEgzvRbPNAYHSop2M7NfWDXiG95vuK7bLvDJNdWWKNuRIP/e0tM8EC1S5xOPr2ZwPQMpvCstwGpro4zKtlI41NWUi0qShD5ETpcROaQA90/bG8iOL3Rw5K14KYIvNasANucdtIl4b4UKYsxmwtWFJPnhWbEbPU2qCkgM7W1audgt0QFPGWjGujxFqJO1OHluIfO4htwCgLawRRb1OQqY7Hv8TY/KM9mDSK5MPFlH2vUIzs8ttA6CB50UO5Yg3r0bx2GPSVxf+MTuJdxvXZtmGHSfd1dqY1xVVjOm1Iip1dSw29Uws0RpErr4LgmSM9Z8fH3mmAu3DctIhGRxyip8aq/qFbmVmJG5MUVZKrHYHJSmega87VUmi5PGjXcKWponyQAXxZnUQMyq3429mzrnqx/LS3fh+MHgq/bVCk3g4Jmy8swjOVIRXOX9hP3E1529rCJSkb787zwrL7amwpYczwEvosaC4bpCRB0d8lZw1IAD72XQUzG+hG5Sky1ucP6vbKqueKqqtYmgCaGJYn8vV1QKvnx+KDrRRv0OJatcKmGbDhp6eJju2xFgWamkls36ZbtCdbDaywT6/ioQR+Bx1W4TdWYnfCK1Ev1UbILjdFk9VIzy6MUTZmAsmGSs+vkD29sVXBx6vbycCAmv9eowbsJCKnw5MBFErZMVCQ318H99HWIrRLmgKSbnL+ETNj47KSeIJ8iU9/cZt0LvsFQtZgZJJZgHv7fP8S+PakOn5LGBwKW4GYMK5o5qJI/axzHH3EnB8V9jOBFHoaQImppo9euyqv6txVRd2FEjyZl9emrcIO8c+IProW0uiDccqfF5AERI+U36ePPbuOBBbIv/j3R+hd0eRh742v2YNBK4qiSTxCckswZTQ3gjbIckQ0tQDs0d304QwnGoQIEGASXxXZBx7DWtdtpKzCvNCplzvI/ToFPc8kJyBW7r6b69OxhWO9U Ki09aPXw uZ4gQ+Q1DzPo4J0WTlyd6N22k16nHU1y98ddHjC3lE2q4dISIaE4Wc3Ja0Kqwixw390qMIo25NEG+J9TGQ6xGJC0RY45mprd9+pkZ9mkkTPtOTOkfFYSQm9uQl9brFjaTylmMEIgH+zRrUEs4MiIuHb05/GLaYXsd1iKwsu1gZNHf0Fo7m6SZ5U/1ffx3Lqv6Dg4UVpat1wjEbYR02yt3O73QY9ctuyL0uQCej3WxBXJDhT726ivBj+nJ9WWzgUSTybfVVHgZUwLlFTgaL2z+36Lu6vwB+X7nwq/GQlMqqiwm0TuE102thocrssewO8U8/Dfjj0QtS6pdJvmf8qGJCRagOP4LBOD73t2ebFuvM65YAfvz/tvG9JXLX2Kdhaxt6BCCql387Pkg9i54t+97HONzYEjNoiSNMr3hC5yr1rRMc12i9LUrOT/qg3Nag5qNC8qt+dvdsp4DYbcAuywdU79nUHTkLSwkER//KkjaIX+mTKK+ijiWGiXXQwn/jnk6xJHH/gkmJ2Y7/xdZDpUZplk8Ja0JxVFv94At1Yr8xM2njtE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hierarchically aggregate all memcgs' MGLRU generations and their page counts into working set page age histograms. The histograms break down the system's workingset per-node, per-anon/file. The sysfs interfaces are as follows: /sys/devices/system/node/nodeX/workingset_report/page_age A per-node page age histogram, showing an aggregate of the node's lruvecs. The information is extracted from MGLRU's per-generation page counters. Reading this file causes a hierarchical aging of all lruvecs, scanning pages and creates a new generation in each lruvec. For example: 1000 anon=0 file=0 2000 anon=0 file=0 100000 anon=5533696 file=5566464 18446744073709551615 anon=0 file=0 /sys/devices/system/node/nodeX/workingset_report/page_age_interval A comma separated list of time in milliseconds that configures what the page age histogram uses for aggregation. Signed-off-by: Yuanchu Xie --- drivers/base/node.c | 6 + include/linux/mmzone.h | 9 + include/linux/workingset_report.h | 79 ++++++ mm/Kconfig | 9 + mm/Makefile | 1 + mm/internal.h | 5 + mm/memcontrol.c | 2 + mm/mm_init.c | 2 + mm/mmzone.c | 2 + mm/vmscan.c | 10 +- mm/workingset_report.c | 451 ++++++++++++++++++++++++++++++ 11 files changed, 572 insertions(+), 4 deletions(-) create mode 100644 include/linux/workingset_report.h create mode 100644 mm/workingset_report.c diff --git a/drivers/base/node.c b/drivers/base/node.c index eb72580288e6..ba5b8720dbfa 100644 --- a/drivers/base/node.c +++ b/drivers/base/node.c @@ -20,6 +20,8 @@ #include #include #include +#include +#include static const struct bus_type node_subsys = { .name = "node", @@ -626,6 +628,7 @@ static int register_node(struct node *node, int num) } else { hugetlb_register_node(node); compaction_register_node(node); + wsr_init_sysfs(node); } return error; @@ -642,6 +645,9 @@ void unregister_node(struct node *node) { hugetlb_unregister_node(node); compaction_unregister_node(node); + wsr_remove_sysfs(node); + wsr_destroy_lruvec(mem_cgroup_lruvec(NULL, NODE_DATA(node->dev.id))); + wsr_destroy_pgdat(NODE_DATA(node->dev.id)); node_remove_accesses(node); node_remove_caches(node); device_unregister(&node->dev); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 80bc5640bb60..ee728c0c5a3b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -24,6 +24,7 @@ #include #include #include +#include /* Free memory management - zoned buddy allocator. */ #ifndef CONFIG_ARCH_FORCE_MAX_ORDER @@ -630,6 +631,9 @@ struct lruvec { struct lru_gen_mm_state mm_state; #endif #endif /* CONFIG_LRU_GEN */ +#ifdef CONFIG_WORKINGSET_REPORT + struct wsr_state wsr; +#endif /* CONFIG_WORKINGSET_REPORT */ #ifdef CONFIG_MEMCG struct pglist_data *pgdat; #endif @@ -1424,6 +1428,11 @@ typedef struct pglist_data { struct lru_gen_memcg memcg_lru; #endif +#ifdef CONFIG_WORKINGSET_REPORT + struct mutex wsr_update_mutex; + struct wsr_report_bins __rcu *wsr_page_age_bins; +#endif + CACHELINE_PADDING(_pad2_); /* Per-node vmstats */ diff --git a/include/linux/workingset_report.h b/include/linux/workingset_report.h new file mode 100644 index 000000000000..d7c2ee14ec87 --- /dev/null +++ b/include/linux/workingset_report.h @@ -0,0 +1,79 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_WORKINGSET_REPORT_H +#define _LINUX_WORKINGSET_REPORT_H + +#include +#include + +struct mem_cgroup; +struct pglist_data; +struct node; +struct lruvec; + +#ifdef CONFIG_WORKINGSET_REPORT + +#define WORKINGSET_REPORT_MIN_NR_BINS 2 +#define WORKINGSET_REPORT_MAX_NR_BINS 32 + +#define WORKINGSET_INTERVAL_MAX ((unsigned long)-1) +#define ANON_AND_FILE 2 + +struct wsr_report_bin { + unsigned long idle_age; + unsigned long nr_pages[ANON_AND_FILE]; +}; + +struct wsr_report_bins { + /* excludes the WORKINGSET_INTERVAL_MAX bin */ + unsigned long nr_bins; + /* last bin contains WORKINGSET_INTERVAL_MAX */ + unsigned long idle_age[WORKINGSET_REPORT_MAX_NR_BINS]; + struct rcu_head rcu; +}; + +struct wsr_page_age_histo { + unsigned long timestamp; + struct wsr_report_bin bins[WORKINGSET_REPORT_MAX_NR_BINS]; +}; + +struct wsr_state { + /* breakdown of workingset by page age */ + struct mutex page_age_lock; + struct wsr_page_age_histo *page_age; +}; + +void wsr_init_lruvec(struct lruvec *lruvec); +void wsr_destroy_lruvec(struct lruvec *lruvec); +void wsr_init_pgdat(struct pglist_data *pgdat); +void wsr_destroy_pgdat(struct pglist_data *pgdat); +void wsr_init_sysfs(struct node *node); +void wsr_remove_sysfs(struct node *node); + +/* + * Returns true if the wsr is configured to be refreshed. + * The next refresh time is stored in refresh_time. + */ +bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, + struct pglist_data *pgdat); +#else +static inline void wsr_init_lruvec(struct lruvec *lruvec) +{ +} +static inline void wsr_destroy_lruvec(struct lruvec *lruvec) +{ +} +static inline void wsr_init_pgdat(struct pglist_data *pgdat) +{ +} +static inline void wsr_destroy_pgdat(struct pglist_data *pgdat) +{ +} +static inline void wsr_init_sysfs(struct node *node) +{ +} +static inline void wsr_remove_sysfs(struct node *node) +{ +} +#endif /* CONFIG_WORKINGSET_REPORT */ + +#endif /* _LINUX_WORKINGSET_REPORT_H */ diff --git a/mm/Kconfig b/mm/Kconfig index 84000b016808..be949786796d 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1301,6 +1301,15 @@ config ARCH_HAS_USER_SHADOW_STACK The architecture has hardware support for userspace shadow call stacks (eg, x86 CET, arm64 GCS or RISC-V Zicfiss). +config WORKINGSET_REPORT + bool "Working set reporting" + depends on LRU_GEN && SYSFS + help + Report system and per-memcg working set to userspace. + + This option exports stats and events giving the user more insight + into its memory working set. + source "mm/damon/Kconfig" endmenu diff --git a/mm/Makefile b/mm/Makefile index d5639b036166..f5ef0768253a 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -98,6 +98,7 @@ obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o +obj-$(CONFIG_WORKINGSET_REPORT) += workingset_report.o ifdef CONFIG_SWAP obj-$(CONFIG_MEMCG) += swap_cgroup.o endif diff --git a/mm/internal.h b/mm/internal.h index 64c2eb0b160e..bbd3c1501bac 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -470,9 +470,14 @@ extern unsigned long highest_memmap_pfn; /* * in mm/vmscan.c: */ +struct scan_control; +bool isolate_lru_page(struct page *page); bool folio_isolate_lru(struct folio *folio); void folio_putback_lru(struct folio *folio); extern void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason); +bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, bool can_swap, + bool force_scan); +void set_task_reclaim_state(struct task_struct *task, struct reclaim_state *rs); /* * in mm/rmap.c: diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 53db98d2c4a1..096856b35fbc 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -63,6 +63,7 @@ #include #include #include +#include #include "internal.h" #include #include @@ -3453,6 +3454,7 @@ static void free_mem_cgroup_per_node_info(struct mem_cgroup *memcg, int node) if (!pn) return; + wsr_destroy_lruvec(&pn->lruvec); free_percpu(pn->lruvec_stats_percpu); kfree(pn->lruvec_stats); kfree(pn); diff --git a/mm/mm_init.c b/mm/mm_init.c index 4ba5607aaf19..b4f7c904ce33 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -30,6 +30,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -1378,6 +1379,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat) pgdat_page_ext_init(pgdat); lruvec_init(&pgdat->__lruvec); + wsr_init_pgdat(pgdat); } static void __meminit zone_init_internals(struct zone *zone, enum zone_type idx, int nid, diff --git a/mm/mmzone.c b/mm/mmzone.c index f9baa8882fbf..0352a2018067 100644 --- a/mm/mmzone.c +++ b/mm/mmzone.c @@ -90,6 +90,8 @@ void lruvec_init(struct lruvec *lruvec) */ list_del(&lruvec->lists[LRU_UNEVICTABLE]); + wsr_init_lruvec(lruvec); + lru_gen_init_lruvec(lruvec); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 28ba2b06fc7d..89da4d8dfb5f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -271,8 +272,7 @@ static int sc_swappiness(struct scan_control *sc, struct mem_cgroup *memcg) } #endif -static void set_task_reclaim_state(struct task_struct *task, - struct reclaim_state *rs) +void set_task_reclaim_state(struct task_struct *task, struct reclaim_state *rs) { /* Check for an overwrite */ WARN_ON_ONCE(rs && task->reclaim_state); @@ -3861,8 +3861,8 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, return success; } -static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, - bool can_swap, bool force_scan) +bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, bool can_swap, + bool force_scan) { bool success; struct lru_gen_mm_walk *walk; @@ -5640,6 +5640,8 @@ static int __init init_lru_gen(void) if (sysfs_create_group(mm_kobj, &lru_gen_attr_group)) pr_err("lru_gen: failed to create sysfs group\n"); + wsr_init_sysfs(NULL); + debugfs_create_file("lru_gen", 0644, NULL, NULL, &lru_gen_rw_fops); debugfs_create_file("lru_gen_full", 0444, NULL, NULL, &lru_gen_ro_fops); diff --git a/mm/workingset_report.c b/mm/workingset_report.c new file mode 100644 index 000000000000..a4dcf62fcd96 --- /dev/null +++ b/mm/workingset_report.c @@ -0,0 +1,451 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "internal.h" + +void wsr_init_pgdat(struct pglist_data *pgdat) +{ + mutex_init(&pgdat->wsr_update_mutex); + RCU_INIT_POINTER(pgdat->wsr_page_age_bins, NULL); +} + +void wsr_destroy_pgdat(struct pglist_data *pgdat) +{ + struct wsr_report_bins __rcu *bins; + + mutex_lock(&pgdat->wsr_update_mutex); + bins = rcu_replace_pointer(pgdat->wsr_page_age_bins, NULL, + lockdep_is_held(&pgdat->wsr_update_mutex)); + kfree_rcu(bins, rcu); + mutex_unlock(&pgdat->wsr_update_mutex); + mutex_destroy(&pgdat->wsr_update_mutex); +} + +void wsr_init_lruvec(struct lruvec *lruvec) +{ + struct wsr_state *wsr = &lruvec->wsr; + + memset(wsr, 0, sizeof(*wsr)); + mutex_init(&wsr->page_age_lock); +} + +void wsr_destroy_lruvec(struct lruvec *lruvec) +{ + struct wsr_state *wsr = &lruvec->wsr; + + mutex_destroy(&wsr->page_age_lock); + kfree(wsr->page_age); + memset(wsr, 0, sizeof(*wsr)); +} + +static int workingset_report_intervals_parse(char *src, + struct wsr_report_bins *bins) +{ + int err = 0, i = 0; + char *cur, *next = strim(src); + + if (*next == '\0') + return 0; + + while ((cur = strsep(&next, ","))) { + unsigned int interval; + + err = kstrtouint(cur, 0, &interval); + if (err) + goto out; + + bins->idle_age[i] = msecs_to_jiffies(interval); + if (i > 0 && bins->idle_age[i] <= bins->idle_age[i - 1]) { + err = -EINVAL; + goto out; + } + + if (++i == WORKINGSET_REPORT_MAX_NR_BINS) { + err = -ERANGE; + goto out; + } + } + + if (i && i < WORKINGSET_REPORT_MIN_NR_BINS - 1) { + err = -ERANGE; + goto out; + } + + bins->nr_bins = i; + bins->idle_age[i] = WORKINGSET_INTERVAL_MAX; +out: + return err ?: i; +} + +static unsigned long get_gen_start_time(const struct lru_gen_folio *lrugen, + unsigned long seq, + unsigned long max_seq, + unsigned long curr_timestamp) +{ + int younger_gen; + + if (seq == max_seq) + return curr_timestamp; + younger_gen = lru_gen_from_seq(seq + 1); + return READ_ONCE(lrugen->timestamps[younger_gen]); +} + +static void collect_page_age_type(const struct lru_gen_folio *lrugen, + struct wsr_report_bin *bin, + unsigned long max_seq, unsigned long min_seq, + unsigned long curr_timestamp, int type) +{ + unsigned long seq; + + for (seq = max_seq; seq + 1 > min_seq; seq--) { + int gen, zone; + unsigned long gen_end, gen_start, size = 0; + + gen = lru_gen_from_seq(seq); + + for (zone = 0; zone < MAX_NR_ZONES; zone++) + size += max( + READ_ONCE(lrugen->nr_pages[gen][type][zone]), + 0L); + + gen_start = get_gen_start_time(lrugen, seq, max_seq, + curr_timestamp); + gen_end = READ_ONCE(lrugen->timestamps[gen]); + + while (bin->idle_age != WORKINGSET_INTERVAL_MAX && + time_before(gen_end + bin->idle_age, curr_timestamp)) { + unsigned long gen_in_bin = (long)gen_start - + (long)curr_timestamp + + (long)bin->idle_age; + unsigned long gen_len = (long)gen_start - (long)gen_end; + + if (!gen_len) + break; + if (gen_in_bin) { + unsigned long split_bin = + size / gen_len * gen_in_bin; + + bin->nr_pages[type] += split_bin; + size -= split_bin; + } + gen_start = curr_timestamp - bin->idle_age; + bin++; + } + bin->nr_pages[type] += size; + } +} + +/* + * proportionally aggregate Multi-gen LRU bins into a working set report + * MGLRU generations: + * current time + * | max_seq timestamp + * | | max_seq - 1 timestamp + * | | | unbounded + * | | | | + * -------------------------------- + * | max_seq | ... | ... | min_seq + * -------------------------------- + * + * Bins: + * + * current time + * | current - idle_age[0] + * | | current - idle_age[1] + * | | | unbounded + * | | | | + * ------------------------------ + * | bin 0 | ... | ... | bin n-1 + * ------------------------------ + * + * Assume the heuristic that pages are in the MGLRU generation + * through uniform accesses, so we can aggregate them + * proportionally into bins. + */ +static void collect_page_age(struct wsr_page_age_histo *page_age, + const struct lruvec *lruvec) +{ + int type; + const struct lru_gen_folio *lrugen = &lruvec->lrugen; + unsigned long curr_timestamp = jiffies; + unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq); + unsigned long min_seq[ANON_AND_FILE] = { + READ_ONCE(lruvec->lrugen.min_seq[LRU_GEN_ANON]), + READ_ONCE(lruvec->lrugen.min_seq[LRU_GEN_FILE]), + }; + struct wsr_report_bin *bin = &page_age->bins[0]; + + for (type = 0; type < ANON_AND_FILE; type++) + collect_page_age_type(lrugen, bin, max_seq, min_seq[type], + curr_timestamp, type); +} + +/* First step: hierarchically scan child memcgs. */ +static void refresh_scan(struct wsr_state *wsr, struct mem_cgroup *root, + struct pglist_data *pgdat) +{ + struct mem_cgroup *memcg; + unsigned int flags; + struct reclaim_state rs = { 0 }; + + set_task_reclaim_state(current, &rs); + flags = memalloc_noreclaim_save(); + + memcg = mem_cgroup_iter(root, NULL, NULL); + do { + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); + unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq); + + /* + * setting can_swap=true and force_scan=true ensures + * proper workingset stats when the system cannot swap. + */ + try_to_inc_max_seq(lruvec, max_seq, true, true); + cond_resched(); + } while ((memcg = mem_cgroup_iter(root, memcg, NULL))); + + memalloc_noreclaim_restore(flags); + set_task_reclaim_state(current, NULL); +} + +/* Second step: aggregate child memcgs into the page age histogram. */ +static void refresh_aggregate(struct wsr_page_age_histo *page_age, + struct mem_cgroup *root, + struct pglist_data *pgdat) +{ + struct mem_cgroup *memcg; + struct wsr_report_bin *bin; + + for (bin = page_age->bins; + bin->idle_age != WORKINGSET_INTERVAL_MAX; bin++) { + bin->nr_pages[0] = 0; + bin->nr_pages[1] = 0; + } + /* the last used bin has idle_age == WORKINGSET_INTERVAL_MAX. */ + bin->nr_pages[0] = 0; + bin->nr_pages[1] = 0; + + memcg = mem_cgroup_iter(root, NULL, NULL); + do { + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); + + collect_page_age(page_age, lruvec); + cond_resched(); + } while ((memcg = mem_cgroup_iter(root, memcg, NULL))); + WRITE_ONCE(page_age->timestamp, jiffies); +} + +static void copy_node_bins(struct pglist_data *pgdat, + struct wsr_page_age_histo *page_age) +{ + struct wsr_report_bins *node_page_age_bins; + int i = 0; + + rcu_read_lock(); + node_page_age_bins = rcu_dereference(pgdat->wsr_page_age_bins); + if (!node_page_age_bins) + goto nocopy; + for (i = 0; i < node_page_age_bins->nr_bins; ++i) + page_age->bins[i].idle_age = node_page_age_bins->idle_age[i]; + +nocopy: + page_age->bins[i].idle_age = WORKINGSET_INTERVAL_MAX; + rcu_read_unlock(); +} + +bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, + struct pglist_data *pgdat) +{ + struct wsr_page_age_histo *page_age; + + if (!READ_ONCE(wsr->page_age)) + return false; + + refresh_scan(wsr, root, pgdat); + mutex_lock(&wsr->page_age_lock); + page_age = READ_ONCE(wsr->page_age); + if (page_age) { + copy_node_bins(pgdat, page_age); + refresh_aggregate(page_age, root, pgdat); + } + mutex_unlock(&wsr->page_age_lock); + return !!page_age; +} +EXPORT_SYMBOL_GPL(wsr_refresh_report); + +static struct pglist_data *kobj_to_pgdat(struct kobject *kobj) +{ + int nid = IS_ENABLED(CONFIG_NUMA) ? kobj_to_dev(kobj)->id : + first_memory_node; + + return NODE_DATA(nid); +} + +static struct wsr_state *kobj_to_wsr(struct kobject *kobj) +{ + return &mem_cgroup_lruvec(NULL, kobj_to_pgdat(kobj))->wsr; +} + +static ssize_t page_age_intervals_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct wsr_report_bins *bins; + int len = 0; + struct pglist_data *pgdat = kobj_to_pgdat(kobj); + + rcu_read_lock(); + bins = rcu_dereference(pgdat->wsr_page_age_bins); + if (bins) { + int i; + int nr_bins = bins->nr_bins; + + for (i = 0; i < bins->nr_bins; ++i) { + len += sysfs_emit_at( + buf, len, "%u", + jiffies_to_msecs(bins->idle_age[i])); + if (i + 1 < nr_bins) + len += sysfs_emit_at(buf, len, ","); + } + } + len += sysfs_emit_at(buf, len, "\n"); + rcu_read_unlock(); + + return len; +} + +static ssize_t page_age_intervals_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *src, size_t len) +{ + struct wsr_report_bins *bins = NULL, __rcu *old; + char *buf = NULL; + int err = 0; + struct pglist_data *pgdat = kobj_to_pgdat(kobj); + + buf = kstrdup(src, GFP_KERNEL); + if (!buf) { + err = -ENOMEM; + goto failed; + } + + bins = + kzalloc(sizeof(struct wsr_report_bins), GFP_KERNEL); + + if (!bins) { + err = -ENOMEM; + goto failed; + } + + err = workingset_report_intervals_parse(buf, bins); + if (err < 0) + goto failed; + + if (err == 0) { + kfree(bins); + bins = NULL; + } + + mutex_lock(&pgdat->wsr_update_mutex); + old = rcu_replace_pointer(pgdat->wsr_page_age_bins, bins, + lockdep_is_held(&pgdat->wsr_update_mutex)); + mutex_unlock(&pgdat->wsr_update_mutex); + kfree_rcu(old, rcu); + kfree(buf); + return len; +failed: + kfree(bins); + kfree(buf); + + return err; +} + +static struct kobj_attribute page_age_intervals_attr = + __ATTR_RW(page_age_intervals); + +static ssize_t page_age_show(struct kobject *kobj, struct kobj_attribute *attr, + char *buf) +{ + struct wsr_report_bin *bin; + int ret = 0; + struct wsr_state *wsr = kobj_to_wsr(kobj); + + + mutex_lock(&wsr->page_age_lock); + if (!wsr->page_age) + wsr->page_age = + kzalloc(sizeof(struct wsr_page_age_histo), GFP_KERNEL); + mutex_unlock(&wsr->page_age_lock); + + wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj)); + + mutex_lock(&wsr->page_age_lock); + if (!wsr->page_age) + goto unlock; + for (bin = wsr->page_age->bins; + bin->idle_age != WORKINGSET_INTERVAL_MAX; bin++) + ret += sysfs_emit_at(buf, ret, "%u anon=%lu file=%lu\n", + jiffies_to_msecs(bin->idle_age), + bin->nr_pages[0] * PAGE_SIZE, + bin->nr_pages[1] * PAGE_SIZE); + + ret += sysfs_emit_at(buf, ret, "%lu anon=%lu file=%lu\n", + WORKINGSET_INTERVAL_MAX, + bin->nr_pages[0] * PAGE_SIZE, + bin->nr_pages[1] * PAGE_SIZE); + +unlock: + mutex_unlock(&wsr->page_age_lock); + return ret; +} + +static struct kobj_attribute page_age_attr = __ATTR_RO(page_age); + +static struct attribute *workingset_report_attrs[] = { + &page_age_intervals_attr.attr, &page_age_attr.attr, NULL +}; + +static const struct attribute_group workingset_report_attr_group = { + .name = "workingset_report", + .attrs = workingset_report_attrs, +}; + +void wsr_init_sysfs(struct node *node) +{ + struct kobject *kobj = node ? &node->dev.kobj : mm_kobj; + struct wsr_state *wsr; + + if (IS_ENABLED(CONFIG_NUMA) && !node) + return; + + wsr = kobj_to_wsr(kobj); + + if (sysfs_create_group(kobj, &workingset_report_attr_group)) + pr_warn("Workingset report failed to create sysfs files\n"); +} +EXPORT_SYMBOL_GPL(wsr_init_sysfs); + +void wsr_remove_sysfs(struct node *node) +{ + struct kobject *kobj = &node->dev.kobj; + struct wsr_state *wsr; + + if (IS_ENABLED(CONFIG_NUMA) && !node) + return; + + wsr = kobj_to_wsr(kobj); + sysfs_remove_group(kobj, &workingset_report_attr_group); +} +EXPORT_SYMBOL_GPL(wsr_remove_sysfs); From patchwork Wed Nov 27 02:57:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13886508 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D38CAD66BAE for ; Wed, 27 Nov 2024 02:57:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4FE106B00A0; Tue, 26 Nov 2024 21:57:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4AFE88D0002; Tue, 26 Nov 2024 21:57:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 326708D0001; Tue, 26 Nov 2024 21:57:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 12DFA6B00A0 for ; Tue, 26 Nov 2024 21:57:50 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id BBBA64167C for ; Wed, 27 Nov 2024 02:57:49 +0000 (UTC) X-FDA: 82830364806.03.7DDB76B Received: from mail-oa1-f74.google.com (mail-oa1-f74.google.com [209.85.160.74]) by imf25.hostedemail.com (Postfix) with ESMTP id 24BCCA0011 for ; Wed, 27 Nov 2024 02:57:43 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VURwKIuT; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3qopGZwcKCBsPL1E38L7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yuanchu.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3qopGZwcKCBsPL1E38L7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yuanchu.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732676265; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5lhbaogLbZu3E/Vkjc13k2hWVq6P2Y+49H1lvwk45sc=; b=W8Rj7Wba7sWU6TYpjsWFSK5WR9wGceIp5QwQvNbF0gRxC7WLGijXNAG2E21i+klCOYtCdj A6X7B32/UWBbQhapGhWfroeDij5uTyP0duMztK8YK3STqrJLWWBBouRSdZnR67laKN0V8z za2a7kx+YhrMai4MXXe3VU0QD4fCtKg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732676265; a=rsa-sha256; cv=none; b=bo3iEX2/XvhUzuhPDcoB09/QFXRrvfNVsmM423djJbc7bURveVpLHCTsdygPHlyVzg1SnI mJy1uWa2SMp7W6TvzYZkGEZwVofoW627u4qC7nkRM0xzEZk6GpCBC4yubG3orF7g564XAn pFa63PtKZAHkEJyF81WKExcC8TYgQ+U= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VURwKIuT; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3qopGZwcKCBsPL1E38L7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yuanchu.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3qopGZwcKCBsPL1E38L7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yuanchu.bounces.google.com Received: by mail-oa1-f74.google.com with SMTP id 586e51a60fabf-2975e172b3bso2353302fac.1 for ; Tue, 26 Nov 2024 18:57:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1732676267; x=1733281067; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5lhbaogLbZu3E/Vkjc13k2hWVq6P2Y+49H1lvwk45sc=; b=VURwKIuTZH5BbQre9hEUI+eZmIAmetlPkchXhi/nvzy+QGcLq1BCTPSK2peIjI4ExX neZ2TKzgmUTqDyzefjs6MgpKriRh/1tA1/LnwyoQcdz+qzuun2uNGWPhZAg87ImrbE8e Uplb0gkM2pPxje2Cb80XN/dNhFxikYDhWhOIjLX+fGYo4SKECF6JCD8tsMqXANmiO0jB frcki8tj8b537WXU8f2sUH2PgdI5AV3n5/vqe9tu7KewwwTddcamC49XGEh83zQirfbk pktV2geQqf3+Pxpp8OEF1N/N1rWPDEZwPwDhdWA2ieRyHM6AmNMtTLYC58VYv2N28q7N bezg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732676267; x=1733281067; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5lhbaogLbZu3E/Vkjc13k2hWVq6P2Y+49H1lvwk45sc=; b=s0y6jxYTkbVkY/P6UpMcxFKljL+XAlxwZGdb5J74qg5SnI3y+Y4U2jFhVgNiDhHx2E tM8TCGCQycaiXVhzSpitMqV6dFc0OCDHf4/rH7k3+e3hCEYrnpiu01rdxNpcgmwlHOaS V6n3uPrlbsOyVBd3kVP0GgRQdBdCAAyQR5hr4jpw8Wuq7HXF4NMxOj4NMiUHyoL4hBzt aRmjrdf7kW2XxjJ6LinqMOcoCdCv5X2PzwDJ1tc4CzdRctujocWMHKR2xvXBCdYOOZnP aF/Yzri63ZHCUShNbgI6+CqQh/SRqqdU5XjnjogAfitjLJUgW0ezBtw+Dsx4wete83ac rjqA== X-Forwarded-Encrypted: i=1; AJvYcCXdC/87eUpcupxN9Lb93GZBJrA2AtGh9NZvw/kIa6Iy4bUB3qD7mmh6zkLj55qrpNwsosh660OfBQ==@kvack.org X-Gm-Message-State: AOJu0YzxcDBk+CAAd8fSdHYc/Qi0Zf1uW4Mutxibl6lUWSgt2z03e/ZX eQ+xOkRkGPShp/8LhkhgL92texEoDV7sd2u5m0Hc4HQW9zzRwG+x6VbZnN3N1luWsugyRu0CO/r p61cRuw== X-Google-Smtp-Source: AGHT+IEx9U6wR1hCMpTboiP7q8J5ECqe/TnGoNjy6O4ISdQoe4PI0qJfC/k0oduc+Ab8ibmO8UdzD0uyW92D X-Received: from oabqq6.prod.google.com ([2002:a05:6870:ef06:b0:296:e2da:79c8]) (user=yuanchu job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6871:a002:b0:296:a7c7:6a08 with SMTP id 586e51a60fabf-29dc4012ca6mr1026883fac.15.1732676266733; Tue, 26 Nov 2024 18:57:46 -0800 (PST) Date: Tue, 26 Nov 2024 18:57:21 -0800 In-Reply-To: <20241127025728.3689245-1-yuanchu@google.com> Mime-Version: 1.0 References: <20241127025728.3689245-1-yuanchu@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241127025728.3689245-3-yuanchu@google.com> Subject: [PATCH v4 2/9] mm: use refresh interval to rate-limit workingset report aggregation From: Yuanchu Xie To: Andrew Morton , David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Tejun Heo , Johannes Weiner , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , " =?utf-8?q?Eugenio_P=C3=A9rez?= " , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Mike Rapoport , Shuah Khan , Christian Brauner , Daniel Watson , Yuanchu Xie , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, linux-mm@kvack.org, linux-kselftest@vger.kernel.org X-Stat-Signature: wnfri1ajwi4fbk8s34te9573brrxtf59 X-Rspamd-Queue-Id: 24BCCA0011 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1732676263-677679 X-HE-Meta: U2FsdGVkX1+V5pingA1CJPmr/jBj/rQEG95KPjgeeThumoIhF7fFypoBkcCFILmGxU9sVmtSIm649LmewPty/TJtVdxEchd3zYyyIA447uM7beZxRcMIQhyd2bfbjwDdoT0xD5AOvPLZ6veubM88RyEY/n+fHRD0bnoK4FlakGlfpsn+ldJ6VZjWlwwhH4iYWpxMdAo8eusVwngmIpxngCsO323D7Wk5v+ytuawWz+sbllA0SIEH4HeGonSq1hcGBJnVNu/rmQFORiGwXCt3lXU/CYu/g3hUYgpp/s1EwzOuqAxgNOszO5Es+4azGBnHLTCSSiw044DNvbr2RQELGU66CrXMPcgJtHmUzxexjUKTBW7Ge3bDH3clnq0SLiKTK70n1wO6t+zDV/qdKVjlSrIjiP1jetQbKu99ArGHqXH2kbCt+mZZCvFhhKByUhiFuR3vl79VcBnIuYzneGhoS1op7qqQmGX03dXShdJdXB+SLqXlfDs5sQ+fxI1qSDHFVCbBS/meg7ECOriJsW+1lGFAAOP9DN8R1s4i+3JhJiDWrXTIUvo7j/l1yUXAtdUfhk3bbzO4rjCjGPfIhn8gY6RawwwQpZLhUY1dv/qQrCwb0M7LUBmibgIaN6JhKlxXDcr2YnLqmwOOdEmrfJtK+4AxSahAIIr9JFCa6vQYSk51u5VRPyQQqypObpqEdqrHPorZDySzNYv1DAvbHTIzGrWdb8ZSfdXO4N73sz3Y4N233Ej/G1sowTGgImxDgaHijhMSMnDfrSk2/wf7oMXureAjqK0RpvebBCb4ORAzM7AeG3GUrd1TO3UdHYEXGNDZ9rce4puRL4/k3wZ73uBydYp88BI8EP5555upWlfkxfY3K+gl532LB+EbUTGTaetLo2TlkYliJVC295prXq7fGBmVIAc0X11Z8RGbr/OUW1ndf4mHQ8IUk8sCLItxGTyLgR5lwUqNzzZZ41oZTsQ oGdJ0R9U WddNFCH5rA1YBGuR5bA/CuUMPhPlTVK+MGFAIWjI6Kh78U3evLaCJ2uSxC9jw3tMk1gMYMdU/Bru0Jz2TbvJEBAM9IKxI6JkDQvm2a2rCV8m95Em5sQB0zLclheuVWfvEFT7ULNbug3ZlBJORon74JIetDjczKVlQVFTyXUCtWFUil89xJkXOLYv0qx5CdKwZAQerg8oeZTjugLOm125jeTk1L6bx1MjtzCUIjw1biyBp8c8GRIbvEF3WBfFioUSwn+BbsiNVvuQisB7kH3cpgciKZND5hhWdB/tqokd+JnhlEOKgHS0aIKjrr8fp1M72TFrYj5hmHARZB+5m9x5/FvkqaU+jMy2FRbEwADOWYRHvD63X3BMNFHl1nIHQOd/sUutBwtGG8UCcjjsFrNIZ6pBFOrG0mAWvWcAANj8gthY5DTk/l245yNREMyP5auiAOHpeA5mNRKILtRzk/J8QEzE1ShFzXy1CFMJFa9LrqTHzxR43oZP3i7p8nNUIEOWCT6oN01LxpGUJ+p6y35K9lDbA4WR+qBlb+e3rKiwg1ar1544m5th/dh5D8w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The refresh interval is a rate limiting factor to workingset page age histogram reads. When a workingset report is generated, the oldest timestamp of all the lruvecs is stored as the timestamp of the report. The same report will be read until the report expires beyond the refresh interval, at which point a new report is generated. Sysfs interface /sys/devices/system/node/nodeX/workingset_report/refresh_interval time in milliseconds specifying how long the report is valid for Signed-off-by: Yuanchu Xie --- include/linux/workingset_report.h | 1 + mm/workingset_report.c | 101 ++++++++++++++++++++++++------ 2 files changed, 83 insertions(+), 19 deletions(-) diff --git a/include/linux/workingset_report.h b/include/linux/workingset_report.h index d7c2ee14ec87..8bae6a600410 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -37,6 +37,7 @@ struct wsr_page_age_histo { }; struct wsr_state { + unsigned long refresh_interval; /* breakdown of workingset by page age */ struct mutex page_age_lock; struct wsr_page_age_histo *page_age; diff --git a/mm/workingset_report.c b/mm/workingset_report.c index a4dcf62fcd96..8678536ccfc7 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -174,9 +174,11 @@ static void collect_page_age_type(const struct lru_gen_folio *lrugen, * Assume the heuristic that pages are in the MGLRU generation * through uniform accesses, so we can aggregate them * proportionally into bins. + * + * Returns the timestamp of the youngest gen in this lruvec. */ -static void collect_page_age(struct wsr_page_age_histo *page_age, - const struct lruvec *lruvec) +static unsigned long collect_page_age(struct wsr_page_age_histo *page_age, + const struct lruvec *lruvec) { int type; const struct lru_gen_folio *lrugen = &lruvec->lrugen; @@ -191,11 +193,14 @@ static void collect_page_age(struct wsr_page_age_histo *page_age, for (type = 0; type < ANON_AND_FILE; type++) collect_page_age_type(lrugen, bin, max_seq, min_seq[type], curr_timestamp, type); + + return READ_ONCE(lruvec->lrugen.timestamps[lru_gen_from_seq(max_seq)]); } /* First step: hierarchically scan child memcgs. */ static void refresh_scan(struct wsr_state *wsr, struct mem_cgroup *root, - struct pglist_data *pgdat) + struct pglist_data *pgdat, + unsigned long refresh_interval) { struct mem_cgroup *memcg; unsigned int flags; @@ -208,12 +213,15 @@ static void refresh_scan(struct wsr_state *wsr, struct mem_cgroup *root, do { struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq); + int gen = lru_gen_from_seq(max_seq); + unsigned long birth = READ_ONCE(lruvec->lrugen.timestamps[gen]); /* * setting can_swap=true and force_scan=true ensures * proper workingset stats when the system cannot swap. */ - try_to_inc_max_seq(lruvec, max_seq, true, true); + if (time_is_before_jiffies(birth + refresh_interval)) + try_to_inc_max_seq(lruvec, max_seq, true, true); cond_resched(); } while ((memcg = mem_cgroup_iter(root, memcg, NULL))); @@ -228,6 +236,7 @@ static void refresh_aggregate(struct wsr_page_age_histo *page_age, { struct mem_cgroup *memcg; struct wsr_report_bin *bin; + unsigned long oldest_lruvec_time = jiffies; for (bin = page_age->bins; bin->idle_age != WORKINGSET_INTERVAL_MAX; bin++) { @@ -241,11 +250,15 @@ static void refresh_aggregate(struct wsr_page_age_histo *page_age, memcg = mem_cgroup_iter(root, NULL, NULL); do { struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); + unsigned long lruvec_time = + collect_page_age(page_age, lruvec); + + if (time_before(lruvec_time, oldest_lruvec_time)) + oldest_lruvec_time = lruvec_time; - collect_page_age(page_age, lruvec); cond_resched(); } while ((memcg = mem_cgroup_iter(root, memcg, NULL))); - WRITE_ONCE(page_age->timestamp, jiffies); + WRITE_ONCE(page_age->timestamp, oldest_lruvec_time); } static void copy_node_bins(struct pglist_data *pgdat, @@ -270,17 +283,25 @@ bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, struct pglist_data *pgdat) { struct wsr_page_age_histo *page_age; + unsigned long refresh_interval = READ_ONCE(wsr->refresh_interval); if (!READ_ONCE(wsr->page_age)) return false; - refresh_scan(wsr, root, pgdat); + if (!refresh_interval) + return false; + mutex_lock(&wsr->page_age_lock); page_age = READ_ONCE(wsr->page_age); - if (page_age) { - copy_node_bins(pgdat, page_age); - refresh_aggregate(page_age, root, pgdat); - } + if (!page_age) + goto unlock; + if (page_age->timestamp && + time_is_after_jiffies(page_age->timestamp + refresh_interval)) + goto unlock; + refresh_scan(wsr, root, pgdat, refresh_interval); + copy_node_bins(pgdat, page_age); + refresh_aggregate(page_age, root, pgdat); +unlock: mutex_unlock(&wsr->page_age_lock); return !!page_age; } @@ -299,6 +320,52 @@ static struct wsr_state *kobj_to_wsr(struct kobject *kobj) return &mem_cgroup_lruvec(NULL, kobj_to_pgdat(kobj))->wsr; } +static ssize_t refresh_interval_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct wsr_state *wsr = kobj_to_wsr(kobj); + unsigned int interval = READ_ONCE(wsr->refresh_interval); + + return sysfs_emit(buf, "%u\n", jiffies_to_msecs(interval)); +} + +static ssize_t refresh_interval_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t len) +{ + unsigned int interval; + int err; + struct wsr_state *wsr = kobj_to_wsr(kobj); + + err = kstrtouint(buf, 0, &interval); + if (err) + return err; + + mutex_lock(&wsr->page_age_lock); + if (interval && !wsr->page_age) { + struct wsr_page_age_histo *page_age = + kzalloc(sizeof(struct wsr_page_age_histo), GFP_KERNEL); + + if (!page_age) { + err = -ENOMEM; + goto unlock; + } + wsr->page_age = page_age; + } + if (!interval && wsr->page_age) { + kfree(wsr->page_age); + wsr->page_age = NULL; + } + + WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(interval)); +unlock: + mutex_unlock(&wsr->page_age_lock); + return err ?: len; +} + +static struct kobj_attribute refresh_interval_attr = + __ATTR_RW(refresh_interval); + static ssize_t page_age_intervals_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) { @@ -382,13 +449,6 @@ static ssize_t page_age_show(struct kobject *kobj, struct kobj_attribute *attr, int ret = 0; struct wsr_state *wsr = kobj_to_wsr(kobj); - - mutex_lock(&wsr->page_age_lock); - if (!wsr->page_age) - wsr->page_age = - kzalloc(sizeof(struct wsr_page_age_histo), GFP_KERNEL); - mutex_unlock(&wsr->page_age_lock); - wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj)); mutex_lock(&wsr->page_age_lock); @@ -414,7 +474,10 @@ static ssize_t page_age_show(struct kobject *kobj, struct kobj_attribute *attr, static struct kobj_attribute page_age_attr = __ATTR_RO(page_age); static struct attribute *workingset_report_attrs[] = { - &page_age_intervals_attr.attr, &page_age_attr.attr, NULL + &refresh_interval_attr.attr, + &page_age_intervals_attr.attr, + &page_age_attr.attr, + NULL }; static const struct attribute_group workingset_report_attr_group = { From patchwork Wed Nov 27 02:57:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13886509 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 318C3D66BAD for ; Wed, 27 Nov 2024 02:57:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 153418D0002; Tue, 26 Nov 2024 21:57:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 104FE8D0001; Tue, 26 Nov 2024 21:57:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E52068D0002; Tue, 26 Nov 2024 21:57:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C4B608D0001 for ; Tue, 26 Nov 2024 21:57:51 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 582DA4164E for ; Wed, 27 Nov 2024 02:57:51 +0000 (UTC) X-FDA: 82830365100.28.52A4C8B Received: from mail-oa1-f74.google.com (mail-oa1-f74.google.com [209.85.160.74]) by imf04.hostedemail.com (Postfix) with ESMTP id 34D7340016 for ; Wed, 27 Nov 2024 02:57:43 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nNovH3Di; spf=pass (imf04.hostedemail.com: domain of 3rIpGZwcKCB0RN3G5AN9HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--yuanchu.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3rIpGZwcKCB0RN3G5AN9HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732676264; a=rsa-sha256; cv=none; b=UKdPufD5YhXHsU3pUlHT5s83lXOaz2dPydlV5aloPyv9pubIp1pDMcM/Rl5doUiuejbSxG MaJXiSYxGuqIRl04mZ77ZYvp3d3rFVnJp6xkqgGeE/yqMa3YL1wrnr2by4dTMuNrLhYY5P 37vimVsoS8myEtkegCfFVr6Yu0FMkqI= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nNovH3Di; spf=pass (imf04.hostedemail.com: domain of 3rIpGZwcKCB0RN3G5AN9HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--yuanchu.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3rIpGZwcKCB0RN3G5AN9HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732676264; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jJsIXJ522dse8Oa1czRmZ0i3siB+gYc3GobTJqrZlLM=; b=L5zkmeyexU5nTThlBl7AFOByUX6x+23+wa1DLDbs0jw9lA83PvObPRLrKCw6Q4dGL8YZvw Cce9QZkDCOBOmeLHE4eisHGUwwjYB757jzWzVAz0BtH3yIs7Y07XvjuHSiTF/B9uSym8Pa 8sRn8f/wu8LPkkup3X3ZvRmVgshPE48= Received: by mail-oa1-f74.google.com with SMTP id 586e51a60fabf-2969ecd4315so4690290fac.3 for ; Tue, 26 Nov 2024 18:57:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1732676268; x=1733281068; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jJsIXJ522dse8Oa1czRmZ0i3siB+gYc3GobTJqrZlLM=; b=nNovH3DiiOVid7okcpjgzoNdRrDAnJWRBgfmvcGz7n4/aheQJoE6CC+98jsVnWtVDk MilySeJ3iFvIBfs/vx8SPsdcCGHbRThzlqMzA5AD1lssK4V7QNpa8bWnoMyXilm9iquR sN0Wn2Ds6A2IKZTqOjonhLLawgLBbprqmy/0blsn2E5AyuXLvaaV6tlOIcMFj8y2wCH3 gB3OSaJiaC3PceMWIBHMjN+eC9okWA5byW2+B3uf3CeJaoElLSCSG17kQFOlCl5zifc/ wyDcEgtUzJFfDrHBX9H4gcoq7u+G/rgDVh/m18YlwW3HuyM3KeSBqVoN70wMeLTFEbEd X+rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732676268; x=1733281068; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jJsIXJ522dse8Oa1czRmZ0i3siB+gYc3GobTJqrZlLM=; b=BgzSu237neeFvzbFushElCm7w4sOrqDtW8K/xLCC/rZbFtaJR503NK0qjbUxxomgA9 HtlB2O5KcwNlUOkdSLLW/Xp7IEI/WLpIp6b1Z/bABCDvGgMaCYo0JF8MSjxLuhbVopvA f7gXJuVvdsgoNlsdN6NmVzJewoVFw7vulnei9FGU7uOXQph8B4qhKUJceMpobB1Resdc W4Sb/jgHAtiKsKADGx7SSEigAxPmUACkCPu/H78gn8DAd9nA4whWK5orvls/SFYAa5fQ nXJUPEnPlDlUH3Gkcto2TxmRjT2D8uxrZp9wkOJ1vfHYythYSrFFs6IxTEvZBuSjwGlN Ppyg== X-Forwarded-Encrypted: i=1; AJvYcCW0t9WLuNb7Zc7bCcqywAeaYWgFSMYLAe0k1H3SfpLLiKe8ICMBmTCGDPFmzxHjfIYyCQq9eHoIoQ==@kvack.org X-Gm-Message-State: AOJu0Yz6MqSqnHaywe4YDvYNreUgJlz9cIbdXVdcqHeu6F0yr/HdBPJ6 I+UQtgmk9MrZXPEdtWfUBOk0XjF6OoV50wau4HY7aupL46vs3rccD7uGGLnD7eCOYGFloW3Nlm0 xhiTJTA== X-Google-Smtp-Source: AGHT+IHpxh9NC9fR/Y+VVMw56gHiKhQb3M845RK+LO+TsitSupQfuVS+RxLJBMoZkNtQHfRh61ZaI/968chM X-Received: from oabxa1.prod.google.com ([2002:a05:6870:7f01:b0:295:f4c4:2bdd]) (user=yuanchu job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6870:9a98:b0:297:27c6:d50f with SMTP id 586e51a60fabf-29dc3f9ee1emr1306226fac.2.1732676268691; Tue, 26 Nov 2024 18:57:48 -0800 (PST) Date: Tue, 26 Nov 2024 18:57:22 -0800 In-Reply-To: <20241127025728.3689245-1-yuanchu@google.com> Mime-Version: 1.0 References: <20241127025728.3689245-1-yuanchu@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241127025728.3689245-4-yuanchu@google.com> Subject: [PATCH v4 3/9] mm: report workingset during memory pressure driven scanning From: Yuanchu Xie To: Andrew Morton , David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Tejun Heo , Johannes Weiner , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , " =?utf-8?q?Eugenio_P=C3=A9rez?= " , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Mike Rapoport , Shuah Khan , Christian Brauner , Daniel Watson , Yuanchu Xie , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, linux-mm@kvack.org, linux-kselftest@vger.kernel.org X-Rspamd-Queue-Id: 34D7340016 X-Stat-Signature: 5bmip8kqdqhe4zctj3zaoiz4fs4wugdc X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1732676263-942823 X-HE-Meta: U2FsdGVkX19vy1B7hcuclx2cbbKrrNwmfFR6voVs2yF/hlJXBw5Ab0deng6kvPHpHwGq/nmceZaw71NbfxOdi6GjnvL1s1QgrzsiMsOPuWQ9PApjj/XIXdNadgITaIikmqNvBeLlhD2dcgB00Hm8fzV1SMmcBtSfrvcucDP0oRrXvOZCMN57iKkqITyUyz877Ev0Gw+aRLCFW9Qou2gMdI9gHHr7B1mt3OYO5aNm4KX2c7pc7eN6ZagyBrazjfqiuKQVJYdoV9V8SLpaDpkLE7HWjVIbUqrUIZbNx2Y1O1gYDw3EWVSQwlg7oLAGhaD0c2GKW2kQTOMau1/6oHwSzURfm877IziVz3tuL80ee1XiZ0ZkYn9NY97xjcO0dL9+Y18IeV7chWAphZAlitytoKv/7TJnDdQeYP17RmNP7wt+rQIRWYrigDbmqW+oDJTsUmG4OTX+GF/t4+oueVU7k4Qx6J+xiEtZpl6hB+smAMcKccJLdGUFPgMzKo8a6Ls6W9YnOWS6lHnO2oUjNGBAmKyr6Q4A01zU3BEFd+iy/7Hhm0W6xTu1/AukHqTq2iJ52mHor7fWlc3zfnajN/D58JY022y3RlKVfwibGIG/40DipPRsHfGLfG+XIXdtOjrZVcQWupDJzvJx8BLmkTBKCxEFkk+Q8/3ihrYvm0QeB5B6V7oHDF6Os3K/uutYWZJozEl2CmwdGKm1nMBbtBr1suhYiJbYO2hvgKN938/G4GOKjQrhv+pdL3Iyd5VtAz8mXWR8ZGQz+TiGnyNQoxaTn82/foRyR7TaGWdYzHGI+Y4KNpicVjs3/UgrdaVYNRtve1PL1ZVOIvT6jYzdhOFcHWj11XeMaNurYkO2u/ypPb7ukU82QyEXDQUuQMczooChCUWzYKsrLQ0TedltiBLlaxAv1mflzCu3Qb/PKFGsUfs/VL3J+cs0m/rq/Q+fEnpDGyDEAFCyz6XXv6tJX6o xSmOD7M+ aGwWgJQaI4Xow1Uh2uztIRTt0/+542oTqKBlS81HcsJS/Y58FqRG7M5HDV0oXL3r0VQbUGoiF/xH6HXzPL3E9jTj00UggTApauqj4Erb2qSqTjZkg/WmMqvaA6axVbf7xOpZdDz8mvHLjMHUKovBb9PJS2enSHmwAcsMiQhcnFuuM5bwvTPq1iIlfmYQrwtzosBQZuzEOgCR2hqiQl+SQ6tShnPfIKE6Euwu+HQcXrG0M7eYEtSy2UofuC457c3pQyIpp6Vezgjhk/UNecAyAJf50aWURavo6v4lkIWrSnb3o5dWiFr7A1hbIAMR4cdH2DPzD8DOwgHgOz479K7JBDGNTr0DoqdmVUtp9t5CYt14Zgaizc4DDm097gkpnNilDpLu68MWljq0EWOQ+IJME8ZtbNBDeFX595rq9t8eRnpcsEUGcEN7kqmW2XacfB7sUAV1cEg46jboewY9lyMWtBEWcZK3/0EB210rjmFDp+pR+pCElQlWZVZ2t5Xb/v7XrVOYAloyLQQn50GUcojbZ2o+rVRetUIHfFKAzC9ijrFinGOU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a node reaches its low watermarks and wakes up kswapd, notify all userspace programs waiting on the workingset page age histogram of the memory pressure, so a userspace agent can read the workingset report in time and make policy decisions, such as logging, oom-killing, or migration. Sysfs interface: /sys/devices/system/node/nodeX/workingset_report/report_threshold time in milliseconds that specifies how often the userspace agent can be notified for node memory pressure. Signed-off-by: Yuanchu Xie --- include/linux/workingset_report.h | 4 +++ mm/internal.h | 12 ++++++++ mm/vmscan.c | 46 +++++++++++++++++++++++++++++++ mm/workingset_report.c | 43 ++++++++++++++++++++++++++++- 4 files changed, 104 insertions(+), 1 deletion(-) diff --git a/include/linux/workingset_report.h b/include/linux/workingset_report.h index 8bae6a600410..2ec8b927b200 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -37,7 +37,11 @@ struct wsr_page_age_histo { }; struct wsr_state { + unsigned long report_threshold; unsigned long refresh_interval; + + struct kernfs_node *page_age_sys_file; + /* breakdown of workingset by page age */ struct mutex page_age_lock; struct wsr_page_age_histo *page_age; diff --git a/mm/internal.h b/mm/internal.h index bbd3c1501bac..508b7d9937d6 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -479,6 +479,18 @@ bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, bool can_swap, bool force_scan); void set_task_reclaim_state(struct task_struct *task, struct reclaim_state *rs); +#ifdef CONFIG_WORKINGSET_REPORT +/* + * in mm/wsr.c + */ +void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat); +#else +static inline void notify_workingset(struct mem_cgroup *memcg, + struct pglist_data *pgdat) +{ +} +#endif + /* * in mm/rmap.c: */ diff --git a/mm/vmscan.c b/mm/vmscan.c index 89da4d8dfb5f..2bca81271d15 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2578,6 +2578,15 @@ static bool can_age_anon_pages(struct pglist_data *pgdat, return can_demote(pgdat->node_id, sc); } +#ifdef CONFIG_WORKINGSET_REPORT +static void try_to_report_workingset(struct pglist_data *pgdat, struct scan_control *sc); +#else +static inline void try_to_report_workingset(struct pglist_data *pgdat, + struct scan_control *sc) +{ +} +#endif + #ifdef CONFIG_LRU_GEN #ifdef CONFIG_LRU_GEN_ENABLED @@ -4004,6 +4013,8 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) set_initial_priority(pgdat, sc); + try_to_report_workingset(pgdat, sc); + memcg = mem_cgroup_iter(NULL, NULL, NULL); do { struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); @@ -5649,6 +5660,38 @@ static int __init init_lru_gen(void) }; late_initcall(init_lru_gen); +#ifdef CONFIG_WORKINGSET_REPORT +static void try_to_report_workingset(struct pglist_data *pgdat, + struct scan_control *sc) +{ + struct mem_cgroup *memcg = sc->target_mem_cgroup; + struct wsr_state *wsr = &mem_cgroup_lruvec(memcg, pgdat)->wsr; + unsigned long threshold = READ_ONCE(wsr->report_threshold); + + if (sc->priority == DEF_PRIORITY) + return; + + if (!threshold) + return; + + if (!mutex_trylock(&wsr->page_age_lock)) + return; + + if (!wsr->page_age) { + mutex_unlock(&wsr->page_age_lock); + return; + } + + if (time_is_after_jiffies(wsr->page_age->timestamp + threshold)) { + mutex_unlock(&wsr->page_age_lock); + return; + } + + mutex_unlock(&wsr->page_age_lock); + notify_workingset(memcg, pgdat); +} +#endif /* CONFIG_WORKINGSET_REPORT */ + #else /* !CONFIG_LRU_GEN */ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) @@ -6200,6 +6243,9 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc) if (zone->zone_pgdat == last_pgdat) continue; last_pgdat = zone->zone_pgdat; + + if (!sc->proactive) + try_to_report_workingset(zone->zone_pgdat, sc); shrink_node(zone->zone_pgdat, sc); } diff --git a/mm/workingset_report.c b/mm/workingset_report.c index 8678536ccfc7..bbefb0046669 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -320,6 +320,33 @@ static struct wsr_state *kobj_to_wsr(struct kobject *kobj) return &mem_cgroup_lruvec(NULL, kobj_to_pgdat(kobj))->wsr; } +static ssize_t report_threshold_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct wsr_state *wsr = kobj_to_wsr(kobj); + unsigned int threshold = READ_ONCE(wsr->report_threshold); + + return sysfs_emit(buf, "%u\n", jiffies_to_msecs(threshold)); +} + +static ssize_t report_threshold_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t len) +{ + unsigned int threshold; + struct wsr_state *wsr = kobj_to_wsr(kobj); + + if (kstrtouint(buf, 0, &threshold)) + return -EINVAL; + + WRITE_ONCE(wsr->report_threshold, msecs_to_jiffies(threshold)); + + return len; +} + +static struct kobj_attribute report_threshold_attr = + __ATTR_RW(report_threshold); + static ssize_t refresh_interval_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) { @@ -474,6 +501,7 @@ static ssize_t page_age_show(struct kobject *kobj, struct kobj_attribute *attr, static struct kobj_attribute page_age_attr = __ATTR_RO(page_age); static struct attribute *workingset_report_attrs[] = { + &report_threshold_attr.attr, &refresh_interval_attr.attr, &page_age_intervals_attr.attr, &page_age_attr.attr, @@ -495,8 +523,13 @@ void wsr_init_sysfs(struct node *node) wsr = kobj_to_wsr(kobj); - if (sysfs_create_group(kobj, &workingset_report_attr_group)) + if (sysfs_create_group(kobj, &workingset_report_attr_group)) { pr_warn("Workingset report failed to create sysfs files\n"); + return; + } + + wsr->page_age_sys_file = + kernfs_walk_and_get(kobj->sd, "workingset_report/page_age"); } EXPORT_SYMBOL_GPL(wsr_init_sysfs); @@ -509,6 +542,14 @@ void wsr_remove_sysfs(struct node *node) return; wsr = kobj_to_wsr(kobj); + kernfs_put(wsr->page_age_sys_file); sysfs_remove_group(kobj, &workingset_report_attr_group); } EXPORT_SYMBOL_GPL(wsr_remove_sysfs); + +void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat) +{ + struct wsr_state *wsr = &mem_cgroup_lruvec(memcg, pgdat)->wsr; + + kernfs_notify(wsr->page_age_sys_file); +} From patchwork Wed Nov 27 02:57:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13886510 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86E06D66BAC for ; Wed, 27 Nov 2024 02:57:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 524F38D0003; Tue, 26 Nov 2024 21:57:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 45F268D0001; Tue, 26 Nov 2024 21:57:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D9808D0003; Tue, 26 Nov 2024 21:57:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 079508D0001 for ; Tue, 26 Nov 2024 21:57:54 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B0AAC120DBA for ; Wed, 27 Nov 2024 02:57:53 +0000 (UTC) X-FDA: 82830365100.15.B0839C5 Received: from mail-ot1-f73.google.com (mail-ot1-f73.google.com [209.85.210.73]) by imf04.hostedemail.com (Postfix) with ESMTP id 84EF240012 for ; Wed, 27 Nov 2024 02:57:45 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=smvHYIUM; spf=pass (imf04.hostedemail.com: domain of 3ropGZwcKCB8TP5I7CPBJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--yuanchu.bounces.google.com designates 209.85.210.73 as permitted sender) smtp.mailfrom=3ropGZwcKCB8TP5I7CPBJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732676269; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nc9u6Gr3t/eex7ob4WWy8qTSgpDGth3PA2eslkY0ahg=; b=fv0NGqY62dMzqyku+ZQ1qmShcLoxGP46ZYsH/VC9eJ4tb/xzMkUw8uRshKDf4uPtfFcypz sK59UWgwk8Gb1RIUjDHEHm74SFz0HJaVOKXoPa+2D5ApY6lmGH86EjssPxBkdjz4O3oYDZ KcRVGjSBTMp6PzhN8Vfk0pzRAQlPlCc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732676269; a=rsa-sha256; cv=none; b=DHzpk/zP4ZTiAYOle8LMIvYzQ1M7CR1O2XUHXJZh/UQepRKTOKH1KKJjW0krAwdP9bNvMG RAgYlYcXwWS4Z3zc0AhyzBZndgH+nQtKrs6NdLvzOkhkLLpoZUlNtOs9fSBtDegk5PxH8o 1Scjg7EcCV1oPMTIE0syctAAeWUapnQ= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=smvHYIUM; spf=pass (imf04.hostedemail.com: domain of 3ropGZwcKCB8TP5I7CPBJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--yuanchu.bounces.google.com designates 209.85.210.73 as permitted sender) smtp.mailfrom=3ropGZwcKCB8TP5I7CPBJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-ot1-f73.google.com with SMTP id 46e09a7af769-71d557437e0so1959463a34.3 for ; Tue, 26 Nov 2024 18:57:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1732676271; x=1733281071; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nc9u6Gr3t/eex7ob4WWy8qTSgpDGth3PA2eslkY0ahg=; b=smvHYIUM4wHBpMM+/9JXfp4R+PFVKtRvFT48UuF4L9MKY+7uQB+pCvcpj+wePh1/9E I2gbzjejVxBGyQY2TsnfqgbC6BjY0SheRgEUPUh7LSG/N148WO89DMdxv7G09aQbllYg GtaC1ZYKPtuEB4+EQETM8MirEn4wHYxeGheBDxxHwhs6jisiwh1nh0Ru9XilOjD2Hs7P 1dmOmhnIBWvQFKGEpuKT262Ri15WvprM8oh1iivi2zPYAMk3vTLxZLiRhlWh2VRssOeK 16UIRr7oyah5E7xTYjsQX9cDbzI58+1MJaNZ5sa7zSocbDr/UTndnwxB4Kt4NKOMyXrD z7Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732676271; x=1733281071; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nc9u6Gr3t/eex7ob4WWy8qTSgpDGth3PA2eslkY0ahg=; b=ZegN7oDURx1kpNrQuRu7A4Lnmdqnp8JfLj7V9P9zX2kMhO3xwjmd7MMlG/mfcPFlYx aR9UcuEYkMQE0XU7FFRXEqRQj1YNCvJ2SDTtrk+dkKV4dt2+sW4DkQ7lSX+3VVxiz/mS cdYe2eVAOLWe1qh93g8v0lxvYcQ/WEZsQ28ioH6cDAdZsU9UrBklLVQU7nIW0BIch2sn Szi1Ji6qFhTKQHJHSW73zUz8DS8WdHVIiXHU8HgdPeoQPmiRYi52p5sCbwfsjsaV29g9 d9T5g16a+L50Bf8McvaRHpoky7WCLLGIOa8cl8zuVgCIS1V3EAq1zMfdY8bJ7hlJ0vA2 XkkQ== X-Forwarded-Encrypted: i=1; AJvYcCWXXj6S1pJ1L+l8JXbaAWqJxDRdWAx8+bOEYcSJFaO0lp3B3ZOnVTDYWxfzZ71+BLMHhk0oARJhtg==@kvack.org X-Gm-Message-State: AOJu0YzDRGJh5r3KK0NYhC0GTwrVPIw6tVLH5bUMuY3Ah9hAL0TFm6+b R2UowvmvVRff+JgsxGkAYmDXTYPHlRIR+s5hWufOh/sNp38v5tWlR2Apnbngp8QuL12q1QfYIi4 FmhcDMw== X-Google-Smtp-Source: AGHT+IHgj0Am4IyS+M4OwpOyz4MtxJ9EoZ2BKJ1SYuM2EnZPBNIt4U8MU/D28Yi7V0kFKazRJu4A5wWgVUd3 X-Received: from pgbcz4.prod.google.com ([2002:a05:6a02:2304:b0:7fc:26c4:ff6c]) (user=yuanchu job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6359:4ca0:b0:1ca:9e4a:d06 with SMTP id e5c5f4694b2df-1cab15f4bd5mr150435955d.13.1732676270571; Tue, 26 Nov 2024 18:57:50 -0800 (PST) Date: Tue, 26 Nov 2024 18:57:23 -0800 In-Reply-To: <20241127025728.3689245-1-yuanchu@google.com> Mime-Version: 1.0 References: <20241127025728.3689245-1-yuanchu@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241127025728.3689245-5-yuanchu@google.com> Subject: [PATCH v4 4/9] mm: extend workingset reporting to memcgs From: Yuanchu Xie To: Andrew Morton , David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Tejun Heo , Johannes Weiner , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , " =?utf-8?q?Eugenio_P=C3=A9rez?= " , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Mike Rapoport , Shuah Khan , Christian Brauner , Daniel Watson , Yuanchu Xie , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, linux-mm@kvack.org, linux-kselftest@vger.kernel.org X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 84EF240012 X-Stat-Signature: 7ggsngdqk8recq7t7gg9d7zymh33y5hs X-Rspam-User: X-HE-Tag: 1732676265-541682 X-HE-Meta: U2FsdGVkX19KOo0bP4XQSvHy8xi875Cymoj11aCkEitIoL4yhQGWDJaiSlOF+jc2uXVkku5rJ9rw0fIjy2HcsfOZ/DV1K1mRR8u8hU7vjTpiK0KWIZ2PSu8arakWDuyeQKsde5E/rjBo06p3cOZQYpzUMityqYJGm8Cq6J1ypuVIJsDTpqUn8Xb9q/5sXPLQ+xa+jcrIW3rVHNr6BVSVaskHuflFgTe91KfomCZnwJE7jaBJ9sW3sii43dcRLyAlvZuTr+H2y8To8hixqss7j6bfGZ+4jFPwtXwRFjxEX1QC41fJPR0duFsjqyEQxC5XzVGgymcwdK969Nx6PVhddHZHYBdnBaQnkKDI7AOYjXKXFX9VnCUmLfXryIRBQ1O+wYgL55yvHXj1f3IOrwi3b/VD6ec61QFlmEM5opYshYioDfHsV6fwsw2G9AXLiJWVFZPw91gWU3ABOE7eSAQcsM2EWxjHBkKl0cWsgQsDIx8iOf3FdoU7YW82zyoJjCyE0rgxqFgbpa+8dt3A4DhUSZb1bWUWhu1AiWmUd2Lap3jp07ucLe/K94eOxX1s5LAisXy0PW3sqT+oLRYmMcAvGKf6BKBvqYoGOnW+yrEFEED/CIZJ7J2c7c/9LJsz8qo4GX+SbzmrA33vO4gsH44nWD17O+blqIK+/mVaTI70UYQZ0T5bka31akvSPsAm5vhxqVPooqQLZ4b2aqkQ2y9HVt9ImMZxvZlU7fiMI5FRL+1II4uQ/aD9Yz1JpHalZd3QFSBLgPnZd2hqml/8MO2Wp5Jc1b+WSrwxtLkZS/0n/rfdNEzEXOBbEaAMJ5+O1nI2A2othCNaHVL7U5T9H+L+oBPoVDM2bnv0yaNmkCvGl/AUuVnPfhyZXYkKGXlQ6QMoe6Yuvzsd7NLkVZOs/jAu+LXOHSwY5PbAHlltnuUT9MHsoDjAF754voiurXVxTRpUN9YiNyoBzR5zOEBBZ9p LHGg0rF4 5Y19R4i1nB33LeEfhK/kycQ9RR3bVEQOWUKsJYw78RBQZCECF0VdmEkzPHN6ptQnzxoVufDeyw3nBOlufezQ8Y7TbOA6BpuFWJ6vBFuZHUOzxGvDBlqrdw3xCqCr510nQPHVFOd9N7wGSTWkACzu7uPe05yQqoG2wFAUrU2wx+Xouu0qm+b6vwpox0/sDlASTsbB4qKlMvOcN6RKlk/CXhzn29kMNL9r7vtCGVMHxaPPBWKdmpT7m5kTcTs1ySfeAqCbRwjxN6fiFCwn6CbNGUiviQx1+5Ad3FOzA0EExYkXm738iuAlgRreKEalfAYG0TiDM9vz8Ol7es42wwqq6JL1CPtDHjaWrqze3hbuXGgMYAAvOLnrnSlG/IpnJ5EjevHmTByLFQBGmTm1xLZbQtbz2TPpNNGK1kCjr9KvzJEhT17Y4YqyEU0U1LIbI+XF6H/0xRe69qhAVmBAKieZpBdA4HcoOOrmSzBzRklmpIENxCmiYqg5k4m8pkNvCdXjj8jIppKdvLqbu44/EaJgOmGo7x5F4xRZ8YVyFy8YrmxqpWZBJQFQoPHaGRQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Break down the system-wide workingset report into per-memcg reports, which aggregages its children hierarchically. The per-node workingset reporting histograms and refresh/report threshold files are presented as memcg files, showing a report containing all the nodes. The per-node page age interval is configurable in sysfs and not available per-memcg, while the refresh interval and report threshold are configured per-memcg. Memcg interface: /sys/fs/cgroup/.../memory.workingset.page_age The memcg equivalent of the sysfs workingset page age histogram breaks down the workingset of this memcg and its children into page age intervals. Each node is prefixed with a node header and a newline. Non-proactive direct reclaim on this memcg can also wake up userspace agents that are waiting on this file. e.g. N0 1000 anon=0 file=0 2000 anon=0 file=0 3000 anon=0 file=0 4000 anon=0 file=0 5000 anon=0 file=0 18446744073709551615 anon=0 file=0 /sys/fs/cgroup/.../memory.workingset.refresh_interval The memcg equivalent of the sysfs refresh interval. A per-node number of how much time a page age histogram is valid for, in milliseconds. e.g. echo N0=2000 > memory.workingset.refresh_interval /sys/fs/cgroup/.../memory.workingset.report_threshold The memcg equivalent of the sysfs report threshold. A per-node number of how often userspace agent waiting on the page age histogram can be woken up, in milliseconds. e.g. echo N0=1000 > memory.workingset.report_threshold Signed-off-by: Yuanchu Xie --- include/linux/memcontrol.h | 21 ++++ include/linux/workingset_report.h | 15 ++- mm/internal.h | 2 + mm/memcontrol.c | 160 +++++++++++++++++++++++++++++- mm/workingset_report.c | 50 +++++++--- 5 files changed, 230 insertions(+), 18 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e1b41554a5fb..fd595b33a54d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -323,6 +323,11 @@ struct mem_cgroup { spinlock_t event_list_lock; #endif /* CONFIG_MEMCG_V1 */ +#ifdef CONFIG_WORKINGSET_REPORT + /* memory.workingset.page_age file */ + struct cgroup_file workingset_page_age_file; +#endif + struct mem_cgroup_per_node *nodeinfo[]; }; @@ -1094,6 +1099,16 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm, void split_page_memcg(struct page *head, int old_order, int new_order); +static inline struct cgroup_file * +mem_cgroup_page_age_file(struct mem_cgroup *memcg) +{ +#ifdef CONFIG_WORKINGSET_REPORT + return &memcg->workingset_page_age_file; +#else + return NULL; +#endif +} + #else /* CONFIG_MEMCG */ #define MEM_CGROUP_ID_SHIFT 0 @@ -1511,6 +1526,12 @@ void count_memcg_event_mm(struct mm_struct *mm, enum vm_event_item idx) static inline void split_page_memcg(struct page *head, int old_order, int new_order) { } + +static inline struct cgroup_file * +mem_cgroup_page_age_file(struct mem_cgroup *memcg) +{ + return NULL; +} #endif /* CONFIG_MEMCG */ /* diff --git a/include/linux/workingset_report.h b/include/linux/workingset_report.h index 2ec8b927b200..616be6469768 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -9,6 +9,8 @@ struct mem_cgroup; struct pglist_data; struct node; struct lruvec; +struct cgroup_file; +struct wsr_state; #ifdef CONFIG_WORKINGSET_REPORT @@ -40,7 +42,10 @@ struct wsr_state { unsigned long report_threshold; unsigned long refresh_interval; - struct kernfs_node *page_age_sys_file; + union { + struct kernfs_node *page_age_sys_file; + struct cgroup_file *page_age_cgroup_file; + }; /* breakdown of workingset by page age */ struct mutex page_age_lock; @@ -60,6 +65,9 @@ void wsr_remove_sysfs(struct node *node); */ bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, struct pglist_data *pgdat); + +int wsr_set_refresh_interval(struct wsr_state *wsr, + unsigned long refresh_interval); #else static inline void wsr_init_lruvec(struct lruvec *lruvec) { @@ -79,6 +87,11 @@ static inline void wsr_init_sysfs(struct node *node) static inline void wsr_remove_sysfs(struct node *node) { } +static inline int wsr_set_refresh_interval(struct wsr_state *wsr, + unsigned long refresh_interval) +{ + return 0; +} #endif /* CONFIG_WORKINGSET_REPORT */ #endif /* _LINUX_WORKINGSET_REPORT_H */ diff --git a/mm/internal.h b/mm/internal.h index 508b7d9937d6..50ca0c6e651c 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -484,6 +484,8 @@ void set_task_reclaim_state(struct task_struct *task, struct reclaim_state *rs); * in mm/wsr.c */ void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat); +int workingset_report_intervals_parse(char *src, + struct wsr_report_bins *bins); #else static inline void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 096856b35fbc..d1032c6efc66 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4348,6 +4348,144 @@ static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf, return nbytes; } +#ifdef CONFIG_WORKINGSET_REPORT +static int memory_ws_refresh_interval_show(struct seq_file *m, void *v) +{ + int nid; + struct mem_cgroup *memcg = mem_cgroup_from_seq(m); + + for_each_node_state(nid, N_MEMORY) { + struct wsr_state *wsr = + &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + + seq_printf(m, "N%d=%u ", nid, + jiffies_to_msecs(READ_ONCE(wsr->refresh_interval))); + } + seq_putc(m, '\n'); + + return 0; +} + +static ssize_t memory_wsr_threshold_parse(char *buf, size_t nbytes, + unsigned int *nid_out, + unsigned int *msecs) +{ + char *node, *threshold; + unsigned int nid; + int err; + + buf = strstrip(buf); + threshold = buf; + node = strsep(&threshold, "="); + + if (*node != 'N') + return -EINVAL; + + err = kstrtouint(node + 1, 0, &nid); + if (err) + return err; + + if (nid >= nr_node_ids || !node_state(nid, N_MEMORY)) + return -EINVAL; + + err = kstrtouint(threshold, 0, msecs); + if (err) + return err; + + *nid_out = nid; + + return nbytes; +} + +static ssize_t memory_ws_refresh_interval_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, + loff_t off) +{ + unsigned int nid, msecs; + struct wsr_state *wsr; + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); + ssize_t ret = memory_wsr_threshold_parse(buf, nbytes, &nid, &msecs); + + if (ret < 0) + return ret; + + wsr = &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + + ret = wsr_set_refresh_interval(wsr, msecs); + return ret ?: nbytes; +} + +static int memory_ws_report_threshold_show(struct seq_file *m, void *v) +{ + int nid; + struct mem_cgroup *memcg = mem_cgroup_from_seq(m); + + for_each_node_state(nid, N_MEMORY) { + struct wsr_state *wsr = + &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + + seq_printf(m, "N%d=%u ", nid, + jiffies_to_msecs(READ_ONCE(wsr->report_threshold))); + } + seq_putc(m, '\n'); + + return 0; +} + +static ssize_t memory_ws_report_threshold_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, + loff_t off) +{ + unsigned int nid, msecs; + struct wsr_state *wsr; + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); + ssize_t ret = memory_wsr_threshold_parse(buf, nbytes, &nid, &msecs); + + if (ret < 0) + return ret; + + wsr = &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + WRITE_ONCE(wsr->report_threshold, msecs_to_jiffies(msecs)); + return ret; +} + +static int memory_ws_page_age_show(struct seq_file *m, void *v) +{ + int nid; + struct mem_cgroup *memcg = mem_cgroup_from_seq(m); + + for_each_node_state(nid, N_MEMORY) { + struct wsr_state *wsr = + &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + struct wsr_report_bin *bin; + + if (!READ_ONCE(wsr->page_age)) + continue; + + wsr_refresh_report(wsr, memcg, NODE_DATA(nid)); + mutex_lock(&wsr->page_age_lock); + if (!wsr->page_age) + goto unlock; + seq_printf(m, "N%d\n", nid); + for (bin = wsr->page_age->bins; + bin->idle_age != WORKINGSET_INTERVAL_MAX; bin++) + seq_printf(m, "%u anon=%lu file=%lu\n", + jiffies_to_msecs(bin->idle_age), + bin->nr_pages[0] * PAGE_SIZE, + bin->nr_pages[1] * PAGE_SIZE); + + seq_printf(m, "%lu anon=%lu file=%lu\n", WORKINGSET_INTERVAL_MAX, + bin->nr_pages[0] * PAGE_SIZE, + bin->nr_pages[1] * PAGE_SIZE); + +unlock: + mutex_unlock(&wsr->page_age_lock); + } + + return 0; +} +#endif + static struct cftype memory_files[] = { { .name = "current", @@ -4419,7 +4557,27 @@ static struct cftype memory_files[] = { .flags = CFTYPE_NS_DELEGATABLE, .write = memory_reclaim, }, - { } /* terminate */ +#ifdef CONFIG_WORKINGSET_REPORT + { + .name = "workingset.refresh_interval", + .flags = CFTYPE_NOT_ON_ROOT | CFTYPE_NS_DELEGATABLE, + .seq_show = memory_ws_refresh_interval_show, + .write = memory_ws_refresh_interval_write, + }, + { + .name = "workingset.report_threshold", + .flags = CFTYPE_NOT_ON_ROOT | CFTYPE_NS_DELEGATABLE, + .seq_show = memory_ws_report_threshold_show, + .write = memory_ws_report_threshold_write, + }, + { + .name = "workingset.page_age", + .flags = CFTYPE_NOT_ON_ROOT | CFTYPE_NS_DELEGATABLE, + .file_offset = offsetof(struct mem_cgroup, workingset_page_age_file), + .seq_show = memory_ws_page_age_show, + }, +#endif + {} /* terminate */ }; struct cgroup_subsys memory_cgrp_subsys = { diff --git a/mm/workingset_report.c b/mm/workingset_report.c index bbefb0046669..1e1bdb8bf75b 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -37,9 +37,12 @@ void wsr_destroy_pgdat(struct pglist_data *pgdat) void wsr_init_lruvec(struct lruvec *lruvec) { struct wsr_state *wsr = &lruvec->wsr; + struct mem_cgroup *memcg = lruvec_memcg(lruvec); memset(wsr, 0, sizeof(*wsr)); mutex_init(&wsr->page_age_lock); + if (memcg && !mem_cgroup_is_root(memcg)) + wsr->page_age_cgroup_file = mem_cgroup_page_age_file(memcg); } void wsr_destroy_lruvec(struct lruvec *lruvec) @@ -51,8 +54,8 @@ void wsr_destroy_lruvec(struct lruvec *lruvec) memset(wsr, 0, sizeof(*wsr)); } -static int workingset_report_intervals_parse(char *src, - struct wsr_report_bins *bins) +int workingset_report_intervals_parse(char *src, + struct wsr_report_bins *bins) { int err = 0, i = 0; char *cur, *next = strim(src); @@ -356,20 +359,14 @@ static ssize_t refresh_interval_show(struct kobject *kobj, return sysfs_emit(buf, "%u\n", jiffies_to_msecs(interval)); } -static ssize_t refresh_interval_store(struct kobject *kobj, - struct kobj_attribute *attr, - const char *buf, size_t len) +int wsr_set_refresh_interval(struct wsr_state *wsr, + unsigned long refresh_interval) { - unsigned int interval; - int err; - struct wsr_state *wsr = kobj_to_wsr(kobj); - - err = kstrtouint(buf, 0, &interval); - if (err) - return err; + int err = 0; + unsigned long old_interval = 0; mutex_lock(&wsr->page_age_lock); - if (interval && !wsr->page_age) { + if (refresh_interval && !wsr->page_age) { struct wsr_page_age_histo *page_age = kzalloc(sizeof(struct wsr_page_age_histo), GFP_KERNEL); @@ -377,16 +374,34 @@ static ssize_t refresh_interval_store(struct kobject *kobj, err = -ENOMEM; goto unlock; } + page_age->bins[0].idle_age = WORKINGSET_INTERVAL_MAX; wsr->page_age = page_age; } - if (!interval && wsr->page_age) { + if (!refresh_interval && wsr->page_age) { kfree(wsr->page_age); wsr->page_age = NULL; } - WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(interval)); + old_interval = READ_ONCE(wsr->refresh_interval); + WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(refresh_interval)); unlock: mutex_unlock(&wsr->page_age_lock); + return err; +} + +static ssize_t refresh_interval_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t len) +{ + unsigned int interval; + int err; + struct wsr_state *wsr = kobj_to_wsr(kobj); + + err = kstrtouint(buf, 0, &interval); + if (err) + return err; + + err = wsr_set_refresh_interval(wsr, interval); return err ?: len; } @@ -551,5 +566,8 @@ void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat) { struct wsr_state *wsr = &mem_cgroup_lruvec(memcg, pgdat)->wsr; - kernfs_notify(wsr->page_age_sys_file); + if (mem_cgroup_is_root(memcg)) + kernfs_notify(wsr->page_age_sys_file); + else + cgroup_file_notify(wsr->page_age_cgroup_file); } From patchwork Wed Nov 27 02:57:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13886511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AE02D66BAE for ; Wed, 27 Nov 2024 02:57:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01B908D0005; Tue, 26 Nov 2024 21:57:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F0DCA8D0001; Tue, 26 Nov 2024 21:57:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D3B8F8D0005; Tue, 26 Nov 2024 21:57:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AEEAC8D0001 for ; Tue, 26 Nov 2024 21:57:55 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3D301AEE5E for ; Wed, 27 Nov 2024 02:57:55 +0000 (UTC) X-FDA: 82830364932.20.A4CCC8F Received: from mail-oa1-f74.google.com (mail-oa1-f74.google.com [209.85.160.74]) by imf23.hostedemail.com (Postfix) with ESMTP id 049B8140015 for ; Wed, 27 Nov 2024 02:57:49 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=RFI5M5MS; spf=pass (imf23.hostedemail.com: domain of 3sIpGZwcKCCEVR7K9ERDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuanchu.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3sIpGZwcKCCEVR7K9ERDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732676269; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t/WPTXcvcabaljBBrVkxMdtvNE4T0NFrHkMqJHgXQGk=; b=qnH4FmI5vKA59U/dwRgY61vaVoFfnBdAfTGlljI1DJ6SY/ZEVV5dvhjYpYZZKRButC48tR Ir2mNV8ynoR+tG3Eyw+PuiIpP7Mh1ZVilZzbIMVrcl7i4LaMklUlAl6hDyxGWgYZ32XF61 I2oBHJkHslFsnrMgIkgkswZ3wKiQjrQ= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=RFI5M5MS; spf=pass (imf23.hostedemail.com: domain of 3sIpGZwcKCCEVR7K9ERDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuanchu.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3sIpGZwcKCCEVR7K9ERDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732676269; a=rsa-sha256; cv=none; b=WoTIl80nx1xgKrCQFsO21h5Bt6yxuU0pKKez6wttTCuXkaQQvHrnBcALfH9Qtwu77k7uAc nzshwGN3KzDxBfbqXUSbpL6WVlabPFL6SvSvR3F1Xp2/e8tsrGnN3gBG4ztcv9eaqK+MRl +Fysmrnqxov+oZ+Xh4xxw+QTy8M4Nlg= Received: by mail-oa1-f74.google.com with SMTP id 586e51a60fabf-29678d451b1so3433414fac.3 for ; Tue, 26 Nov 2024 18:57:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1732676272; x=1733281072; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=t/WPTXcvcabaljBBrVkxMdtvNE4T0NFrHkMqJHgXQGk=; b=RFI5M5MSy2fKdSRxGP37cLehQ6cg2M0GWzC7M/shZXpIMJ/oO+sNLwohXDacwBkh5D eLFGP7oqW7OJWNvBtE3Esc05ce3Dx/FYX0FfsenktjP0N1jJZL2Om/H7pj+3rWnBwXDt OrNWtCvNQcxCL13AEX1zVub+BePxALobp89tRU16DikVyjPzRp6iWGw4Yy6n/piBSIQ1 5qk7qAB3hachoWCzPKcuZJ7ieGIWxQtUalFPh1vsOIH9zgqGqUtSvOnyjLlJzJ6oZCjN jjfH5PgkbanElc3V8Wckiiw4Sx5Pwi//haLXcBJ02fnL9fzqRny89l7nI8PvJcz+aOHo EoCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732676272; x=1733281072; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=t/WPTXcvcabaljBBrVkxMdtvNE4T0NFrHkMqJHgXQGk=; b=ny9WqoUVrXDk14enT0lCPsSR+xkXLwKiRf7mIBKY/xlXgEeRGNcfXsnRAF5+CKK9jK HSPjOoY/KEWveMFTMmj4xDgt4s4F6Voa6x8P3ah3W4ZHDXFkRXxDCyOBVOPexLyrJsRw bUI7BNw74vUUMus8fe4ISmGnStpllC1Pyh31YJ/EqX62tuMx22naCbuoZ6GIRq0Sbr1h GGmW79UlamN3tXEQjci40PDBSCGxnWuUKoCLamUN1CLSDjp/d5hamOe2QIccCrSk4ogO yOMbr4eOGm/+qMj9ctbISVdMWM2FnKSFmik3YC95g6cH7Jm/3IVrf323ELWtMzJ9Scb0 oKkg== X-Forwarded-Encrypted: i=1; AJvYcCWuewfwxSpwN80JMcxxrD472oC6Bwl2fnRLW1XX73MVCzyxT+jTS0clvzUeWQFD1nqFt6DaDtKK6w==@kvack.org X-Gm-Message-State: AOJu0Yw34RXkF+69sy+MsiTDxOW+rmFisLopvmaQwuYV+WFlcV6BvOvw MiI5fNHg+ueYvQVWJLAn0kczqfdxCkBGguAJrFfHOeVSgy8YhU1swtnYgxkAJYZ7ZqrQM6a4nkE aU+2xLA== X-Google-Smtp-Source: AGHT+IFRcRWDlCpSpvonVfz6gqukD5+iS1Nh+k6upjKjqm+Gv9S2zOic8Epnn/W/99f5T4mKjm5ZX+88GSwn X-Received: from oacnq1.prod.google.com ([2002:a05:6871:3781:b0:296:523f:4d02]) (user=yuanchu job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6870:4996:b0:297:2376:9b07 with SMTP id 586e51a60fabf-29dc400538dmr1178830fac.10.1732676272567; Tue, 26 Nov 2024 18:57:52 -0800 (PST) Date: Tue, 26 Nov 2024 18:57:24 -0800 In-Reply-To: <20241127025728.3689245-1-yuanchu@google.com> Mime-Version: 1.0 References: <20241127025728.3689245-1-yuanchu@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241127025728.3689245-6-yuanchu@google.com> Subject: [PATCH v4 5/9] mm: add kernel aging thread for workingset reporting From: Yuanchu Xie To: Andrew Morton , David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Tejun Heo , Johannes Weiner , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , " =?utf-8?q?Eugenio_P=C3=A9rez?= " , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Mike Rapoport , Shuah Khan , Christian Brauner , Daniel Watson , Yuanchu Xie , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, linux-mm@kvack.org, linux-kselftest@vger.kernel.org X-Rspamd-Queue-Id: 049B8140015 X-Rspamd-Server: rspam12 X-Stat-Signature: adwg83gd65jrqsnbcnbyst4qbfkf5oyh X-Rspam-User: X-HE-Tag: 1732676269-942221 X-HE-Meta: U2FsdGVkX1+tVgnbOHOCuMXsJOFFYl+eR2YLfXzZQOvv2A/dxnPBBYCn+3L31KsFja24LHoi2PTKYS7BJpQFxnnPsYXSfIhTErZUs0xXJhX0eIalztFx4BSQs97b4Z5kQOhAdnuS5H9waSL71jsjq06ccgSEHJGizxjszcDGGvKIwOJY3mgPhsW0iQqL3fWJyoe6mIduJGIAUA10dMCq4yJGfAYojNwR1DSqOECC+m1PJXRMl+6r3pO7fswAq1K4XIdlthiO8i5kLyus1BfiPV09DGlPD3FUvDZwPCVg8pUPzRXAmqLqG3vf6/1WrIOPhc3hatVzhBhvDZoxoqAXcVxIiL68eaMSGx2I8sgE6NNv+Bz+Q8d7ZXJj/yIMZLiCJ3Pynp+9M7NJQjGyF91cBiTrug33zKPwxIjBBDaPqCxb5raHYu7YOmvCTUv1U3bcvxV55+T8jtu3V5u4iB6J3e8ZCbohiT8ti2mO6A5RRUO1Z7adl2eDMJqvTj8akI4LAxmv/GpaEjmHRe7Y3tjZ3a+DlQs143g3l1gTDgjYEnjGuw5zGGRjGfPzCPMyesyI4k5ZPmfe8BGk/VW30yfsEAnEg1LWzSI8daLJF0HyFFb1osnZ0B2DzwK8S+m5sV84zhmyoNBgNeb1I8J/yNsWshm3yDUtEXuGmBWznlvQK2hoUgtmRyEkm7us66rNumcS8ARkHNX5Nv6pPZtP1oxVCAmZAiKBVxtwEdgV2GA4nL3P4QasAvXb+deeVnMzdsfSf4oRe269eJMi2X7+wZ6moAfNZxXfRma2DouK9lyOwVN9g4aaGSc6ul4TdlNusPYT5SV/eEFh0FUoOX1cIx9KBbPXxQEftd/mzsa2RapPY/Yu1BaZnzHcERb2mhrfAvim5VA+GrW5QigVQP8J2yyyy8ybKnHPNNCAzMvxSGIv35AJ5M2bIBK6YsfI8koGo0giYqjUuIkkfrwxU8WhjpN /y/qngPM jXW9Q9cqUE7sD1tUX6cC7jAZtCGcRMGFfyUZu4ALjg6vGdvFPe3DMkfyIEc949FBkR9Lmj3vcC0/a072i6+FgwwW6sadoQFHqltiqj/gk28yXHTlpRks3QU4BMNT1pvm92rfdcZZiylaMnuYfcu39jG9YADxG3SrOOdw7WE1x0BtCEZ8uvcw9Gm5PcUV6TDjUwBMZ2VwyiQhaQbIH1sjypRfq1xucI5sQU5ZDiPzrWXYSnNwm1vyoIsIc/E73f+HqEHlNdsgW/ncaESwjAEVa9LlVjQUw02VT1bxTflJ+XnUpbf1GQrE4DNGhJRWrNPypk/oLEyYL10QDlb6YMu4IixQ3ZMoC6ejn0oz3TyqspJtaraq+zISq3JlLk3CPe6l3yQdWkmt+iuTmHodxuFyJqd9y2oxH3crXlSAViae/e1OF2RyhFAqdG8TFTavkEEyy9ja4Zb0Ra8MWv2zNN/XxLcDFfU4AgwfIRLIeN/sEsB7nPmmBSvMTsNdJOn1csQ1fvLLVTtt4py43G+hG6eorfgiHLD2XMmWXS9DvFA35qA8Ptmo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For reliable and timely aging on memcgs, one has to read the page age histograms on time. A kernel thread makes it easier by aging memcgs with valid refresh_interval when they can be refreshed, and also reduces the latency of any userspace consumers of the page age histogram. The kerne aging thread is gated behind CONFIG_WORKINGSET_REPORT_AGING. Debugging stats may be added in the future for when aging cannot keep up with the configured refresh_interval. Signed-off-by: Yuanchu Xie --- include/linux/workingset_report.h | 10 ++- mm/Kconfig | 6 ++ mm/Makefile | 1 + mm/memcontrol.c | 2 +- mm/workingset_report.c | 13 ++- mm/workingset_report_aging.c | 127 ++++++++++++++++++++++++++++++ 6 files changed, 154 insertions(+), 5 deletions(-) create mode 100644 mm/workingset_report_aging.c diff --git a/include/linux/workingset_report.h b/include/linux/workingset_report.h index 616be6469768..f6bbde2a04c3 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -64,7 +64,15 @@ void wsr_remove_sysfs(struct node *node); * The next refresh time is stored in refresh_time. */ bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, - struct pglist_data *pgdat); + struct pglist_data *pgdat, unsigned long *refresh_time); + +#ifdef CONFIG_WORKINGSET_REPORT_AGING +void wsr_wakeup_aging_thread(void); +#else /* CONFIG_WORKINGSET_REPORT_AGING */ +static inline void wsr_wakeup_aging_thread(void) +{ +} +#endif /* CONFIG_WORKINGSET_REPORT_AGING */ int wsr_set_refresh_interval(struct wsr_state *wsr, unsigned long refresh_interval); diff --git a/mm/Kconfig b/mm/Kconfig index be949786796d..a8def8c65610 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1310,6 +1310,12 @@ config WORKINGSET_REPORT This option exports stats and events giving the user more insight into its memory working set. +config WORKINGSET_REPORT_AGING + bool "Workingset report kernel aging thread" + depends on WORKINGSET_REPORT + help + Performs aging on memcgs with their configured refresh intervals. + source "mm/damon/Kconfig" endmenu diff --git a/mm/Makefile b/mm/Makefile index f5ef0768253a..3a282510f960 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -99,6 +99,7 @@ obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o obj-$(CONFIG_WORKINGSET_REPORT) += workingset_report.o +obj-$(CONFIG_WORKINGSET_REPORT_AGING) += workingset_report_aging.o ifdef CONFIG_SWAP obj-$(CONFIG_MEMCG) += swap_cgroup.o endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d1032c6efc66..ea83f10b22a1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4462,7 +4462,7 @@ static int memory_ws_page_age_show(struct seq_file *m, void *v) if (!READ_ONCE(wsr->page_age)) continue; - wsr_refresh_report(wsr, memcg, NODE_DATA(nid)); + wsr_refresh_report(wsr, memcg, NODE_DATA(nid), NULL); mutex_lock(&wsr->page_age_lock); if (!wsr->page_age) goto unlock; diff --git a/mm/workingset_report.c b/mm/workingset_report.c index 1e1bdb8bf75b..dad539e602bb 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -283,7 +283,7 @@ static void copy_node_bins(struct pglist_data *pgdat, } bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, - struct pglist_data *pgdat) + struct pglist_data *pgdat, unsigned long *refresh_time) { struct wsr_page_age_histo *page_age; unsigned long refresh_interval = READ_ONCE(wsr->refresh_interval); @@ -300,10 +300,14 @@ bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, goto unlock; if (page_age->timestamp && time_is_after_jiffies(page_age->timestamp + refresh_interval)) - goto unlock; + goto time; refresh_scan(wsr, root, pgdat, refresh_interval); copy_node_bins(pgdat, page_age); refresh_aggregate(page_age, root, pgdat); + +time: + if (refresh_time) + *refresh_time = page_age->timestamp + refresh_interval; unlock: mutex_unlock(&wsr->page_age_lock); return !!page_age; @@ -386,6 +390,9 @@ int wsr_set_refresh_interval(struct wsr_state *wsr, WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(refresh_interval)); unlock: mutex_unlock(&wsr->page_age_lock); + if (!err && refresh_interval && + (!old_interval || jiffies_to_msecs(old_interval) > refresh_interval)) + wsr_wakeup_aging_thread(); return err; } @@ -491,7 +498,7 @@ static ssize_t page_age_show(struct kobject *kobj, struct kobj_attribute *attr, int ret = 0; struct wsr_state *wsr = kobj_to_wsr(kobj); - wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj)); + wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj), NULL); mutex_lock(&wsr->page_age_lock); if (!wsr->page_age) diff --git a/mm/workingset_report_aging.c b/mm/workingset_report_aging.c new file mode 100644 index 000000000000..91ad5020778a --- /dev/null +++ b/mm/workingset_report_aging.c @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Workingset report kernel aging thread + * + * Performs aging on behalf of memcgs with their configured refresh interval. + * While a userspace program can periodically read the page age breakdown + * per-memcg and trigger aging, the kernel performing aging is less overhead, + * more consistent, and more reliable for the use case where every memcg should + * be aged according to their refresh interval. + */ +#define pr_fmt(fmt) "workingset report aging: " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static DECLARE_WAIT_QUEUE_HEAD(aging_wait); +static bool refresh_pending; + +static bool do_aging_node(int nid, unsigned long *next_wake_time) +{ + struct mem_cgroup *memcg; + bool should_wait = true; + struct pglist_data *pgdat = NODE_DATA(nid); + + memcg = mem_cgroup_iter(NULL, NULL, NULL); + do { + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); + struct wsr_state *wsr = &lruvec->wsr; + unsigned long refresh_time; + + /* use returned time to decide when to wake up next */ + if (wsr_refresh_report(wsr, memcg, pgdat, &refresh_time)) { + if (should_wait) { + should_wait = false; + *next_wake_time = refresh_time; + } else if (time_before(refresh_time, *next_wake_time)) { + *next_wake_time = refresh_time; + } + } + + cond_resched(); + } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL))); + + return should_wait; +} + +static int do_aging(void *unused) +{ + while (!kthread_should_stop()) { + int nid; + long timeout_ticks; + unsigned long next_wake_time; + bool should_wait = true; + + WRITE_ONCE(refresh_pending, false); + for_each_node_state(nid, N_MEMORY) { + unsigned long node_next_wake_time; + + if (do_aging_node(nid, &node_next_wake_time)) + continue; + if (should_wait) { + should_wait = false; + next_wake_time = node_next_wake_time; + } else if (time_before(node_next_wake_time, + next_wake_time)) { + next_wake_time = node_next_wake_time; + } + } + + if (should_wait) { + wait_event_interruptible(aging_wait, refresh_pending); + continue; + } + + /* sleep until next aging */ + timeout_ticks = next_wake_time - jiffies; + if (timeout_ticks > 0 && + timeout_ticks != MAX_SCHEDULE_TIMEOUT) { + schedule_timeout_idle(timeout_ticks); + continue; + } + } + return 0; +} + +/* Invoked when refresh_interval shortens or changes to a non-zero value. */ +void wsr_wakeup_aging_thread(void) +{ + WRITE_ONCE(refresh_pending, true); + wake_up_interruptible(&aging_wait); +} + +static struct task_struct *aging_thread; + +static int aging_init(void) +{ + struct task_struct *task; + + task = kthread_run(do_aging, NULL, "kagingd"); + + if (IS_ERR(task)) { + pr_err("Failed to create aging kthread\n"); + return PTR_ERR(task); + } + + aging_thread = task; + pr_info("module loaded\n"); + return 0; +} + +static void aging_exit(void) +{ + kthread_stop(aging_thread); + aging_thread = NULL; + pr_info("module unloaded\n"); +} + +module_init(aging_init); +module_exit(aging_exit); From patchwork Wed Nov 27 02:57:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13886512 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C749D66BAC for ; Wed, 27 Nov 2024 02:58:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9EA18D0006; Tue, 26 Nov 2024 21:57:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C521B8D0001; Tue, 26 Nov 2024 21:57:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AEEC08D0006; Tue, 26 Nov 2024 21:57:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8821F8D0001 for ; Tue, 26 Nov 2024 21:57:57 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3F06DAED75 for ; Wed, 27 Nov 2024 02:57:57 +0000 (UTC) X-FDA: 82830365268.05.55FC25E Received: from mail-oa1-f74.google.com (mail-oa1-f74.google.com [209.85.160.74]) by imf22.hostedemail.com (Postfix) with ESMTP id 0685DC000A for ; Wed, 27 Nov 2024 02:57:48 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=I21KE1fw; spf=pass (imf22.hostedemail.com: domain of 3sopGZwcKCCMXT9MBGTFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--yuanchu.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3sopGZwcKCCMXT9MBGTFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732676269; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uPMcP9d5tfxIgvPgFZ20QTJyKk7eKxs4Pbg7D460d7w=; b=hvmRh+LKQi2Ac+kTBy+n+yXIR3rHk8isvBTEv26HHu6yWghWQ2QiBAnw8/Q3TvdC+36/Ug +HFShniGCq7xnMy0Q3/rNsyyn+0vG+Wmmcf5CHMPk/CeKxJfU9FE+a5FQmkWv+1wMbK7Mz tnq/tQM+XKeljAhRK/+FuxXB5qW4D24= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=I21KE1fw; spf=pass (imf22.hostedemail.com: domain of 3sopGZwcKCCMXT9MBGTFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--yuanchu.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3sopGZwcKCCMXT9MBGTFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732676269; a=rsa-sha256; cv=none; b=EkkaA/id00w+y99vbaZBG5fr1hrNPUSUgqpXaEhF90NLDVvR+K96Ma0ZaI/4Yy4Hu9viR3 89s8mHtXdsBX5W8zyB1fqrLQKGekn1pehPLdoIJyy+IO7sCqT9HQkzaxZBHISvubo2VmxG mzBVe+4tKK0OG7VWPgWCCvHHcLvKoY0= Received: by mail-oa1-f74.google.com with SMTP id 586e51a60fabf-29678d451b1so3433421fac.3 for ; Tue, 26 Nov 2024 18:57:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1732676274; x=1733281074; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uPMcP9d5tfxIgvPgFZ20QTJyKk7eKxs4Pbg7D460d7w=; b=I21KE1fw15stkEBeNsaZtZG4KOnfHjOIQWWGNt+JfT4z4cUYl+YNE/NUOcv1Zt1JGt dq4JD8EC2AyYDGhV2QnBG4IO4tNIynDcJn6aeuCHng+7UBGspVr5yTWnYxmUtA/6JdIJ cXYGZDSoTIhHZKFIDm1iN16/p7p5psQ3nDJvMTVWZ3OQi//6aHRZQAVvpVQc4gmy4P5+ HwvlYNbB62OF4GCeXi72I4Z7H/c7i+l1NCJ8kIP1ofUkQ01eR8SieuATlM0YyNXfTt3G OQzVUBFR4NBVZtGUsSlBUqpr9TBaraZ20lIJIKfjVXi41WKk15HSu451pJUUK7Tvgvn+ HY2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732676274; x=1733281074; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uPMcP9d5tfxIgvPgFZ20QTJyKk7eKxs4Pbg7D460d7w=; b=iiQoxpAHXitmBhALelooSrRboSpEnEfVXI7q+rDs6D6eyUxojXdhsHlX4lu2kr4pkU vqMx5MLANjt0r7uWEgQ4hmHuiQ3IbvRHt4fnpG+bseiV49gbQT8VeTyxqXwWHY+SwE3F R+hNrhADMwcTk8uVIqx7xR7Feb5NnhIqyOxyKt7w+lFTjAZoGdvywwkVH6a81PVTlY6q DsBN5CG6WSVjZ/Z3VsJYMepwdu6n2w4yXR3niAhLb3EyCiratnJV/u2yIhjCaBEIRBc9 rLPlw7DZrqXQHHtQFmYmQoim15RC74dWPbOlYwX6E24eJBRglCeaXfTp75baoXFx3U9C yD7w== X-Forwarded-Encrypted: i=1; AJvYcCV+4UbveOnvaZScoUwVwT+mthvHO1yIT498EnrxGVDd6G77im9w+S0SC6ls2hjZ8nwrVJHfrNBxXA==@kvack.org X-Gm-Message-State: AOJu0Yy2gCrvWJCGqIqcIf4tZx4xFTViR9QbPQxklr2B4+4Tlj35RhDB YapeWAzI3JHGmOT5EpLkYy2wXuV8ZrXStEHk1SdgjYQZjSB3YnFQWLo9wcUAYjKrG4JPZFlVx9A CDbFXZw== X-Google-Smtp-Source: AGHT+IF4oOUQg1XlpZfeQrxfk9coJrlSFrVoytICljsVfJIJL3Y9yL1lfE+LHwmmhU8AIRAlVCgQWKW2HBQ5 X-Received: from oablh7.prod.google.com ([2002:a05:6870:b07:b0:296:c3ce:6155]) (user=yuanchu job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6870:b30a:b0:296:80aa:a9b7 with SMTP id 586e51a60fabf-29dc418cde7mr1259622fac.21.1732676274537; Tue, 26 Nov 2024 18:57:54 -0800 (PST) Date: Tue, 26 Nov 2024 18:57:25 -0800 In-Reply-To: <20241127025728.3689245-1-yuanchu@google.com> Mime-Version: 1.0 References: <20241127025728.3689245-1-yuanchu@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241127025728.3689245-7-yuanchu@google.com> Subject: [PATCH v4 6/9] selftest: test system-wide workingset reporting From: Yuanchu Xie To: Andrew Morton , David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Tejun Heo , Johannes Weiner , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , " =?utf-8?q?Eugenio_P=C3=A9rez?= " , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Mike Rapoport , Shuah Khan , Christian Brauner , Daniel Watson , Yuanchu Xie , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, linux-mm@kvack.org, linux-kselftest@vger.kernel.org X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 0685DC000A X-Rspam-User: X-Stat-Signature: mwm7b5ih864dbcn6rtuhs5m9ddew44fj X-HE-Tag: 1732676268-581284 X-HE-Meta: U2FsdGVkX1+1RScZMa6OlyX75qv7ao+fDLCcM/Y3JKJNCnekSlbCU/+lkDTw7f4tLe+QWtx52oTInM5VdO7SoIuIalInAP2RKjfHBuLyNQ8QsPdzk3d3IKxq78FeFmqyWESJNkPmK3D1Xzha5D7jOsGL0SvzBcTkrg1enmBmTKEct0mCjYa0Wau8sjaoxEyBu+SshKuUVjhiuGM1PCXDw1RokQVzO0AZ2J9ltc8Nkia4HJou+sz88DPv3kOgKVDJGd6lszCjsJ1SCd5jaF4v/uSPntH0kILSZxd538RZI8760At+9rCRtVhXsgFagTQqeaTT0eHybtl7eAQqOiJA6Uv/Z9ES+jI9jtZfPITn+K9zt4+G+zxUYswCIazIRv9bGPsKWuJ5NGLKzTVyO6o/ZcBBJck+TGJXpyXoubokW6pU8HGhUETpj6AfcuCd4/z/JP1kbg81WgTWY87RJQqBWgocyrj+f/su9W346ugfYRXcquVK7cOWKmbK5XPJcKXvWscE2qkTFxpsn0G2mc7VcTiUj6yzQOO+nWhQWN4goeSvjDlOdb4Wsuwojg7wff06XlA/Fy47YAX60uICJ2GAqKj0c62bbAsPb9aftRSgumwQo93k3ic0+3Yxx9Ar1/CSIc/uziswEDUNQFqPige1M6Jt/JRgnLUUEvUJ2y5A/2uiddtBqiDJ+RN8BHMnPUDz849b9N/6jVr9BR3pEPSW2JuC/FRQrsa0pZe2wGADJTPFiYw4rh+FnFIZ5onj4785RKVXiQ86ossBfH9ekiylGKDW4MjZrN9R0z4JsrFALMvYGSLf4Uq5eXayJM1y+w06ZKMokAHpUoTwhQFGmLKgErG3afVfJNaQcsxHyfl4gV+9BI6pndIgYearMW91XM9fd3bAfiRN7E3AFeURN1naSGGxFJ0/zwQFKu/QAkVJO1o8MAQ8Vm1tu5/mJPHgUdit2dPwWdBwIdfM948bliL 8csK0zpM emWKhhAc6+PitcQ3dkNQB12V9BGZsHeDH0/CClKavMOK8b8x/cux7124SLvHkGqzsAocn+Zxwlo/x47DDNZcrPN8G3ztyAWi54Qy8uN5snfcVglIarUfSblVXDSO1LRZe7qAPcw+r6LClZW2wvrYp8Z++l3iPS62WK8bhMbkdR0Uf1zz0ziBWXIFW2iMZ+CfqkhIbnDgtPl6UNb48Lj3lqRQKyswwDTDWmAnPf1rT8R70Dpp4/TzSTAExX4OtxzR0Vh7T/XbIb/b8UgWliEi0ZhbbrPq59VUncPCsvKgJo3ApO8B3RKHlCXzPTIyKerNH/I3KxzRcuXKr2qpbjxRzM54PDzLSfkBHY8eYV6Qpylpq8fJE90DLDlHIk07OPsLJUUZUa1hg+GyUfOc5Kz+h0BaWZVnu43lhEOlwnj50m8GCs4HP2bmaLCiOJ37z1wOCtA0cC+GwPvZIssP/TcM14rSwTQx2NLaEVYvza43Ndljptg0Kjx65KeZNJ+nCFjR7TQdltZwpHoOuX7iKHqoE/bO1i7Q4t8/995F/BEOPPEDQ/O8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A basic test that verifies the working set size of a simple memory accessor. It should work with or without the aging thread. When running tests with run_vmtests.sh, file workingset report testing requires an environment variable WORKINGSET_REPORT_TEST_FILE_PATH to store a temporary file, which is passed into the test invocation as a parameter. Signed-off-by: Yuanchu Xie --- tools/testing/selftests/mm/.gitignore | 1 + tools/testing/selftests/mm/Makefile | 3 + tools/testing/selftests/mm/run_vmtests.sh | 5 + .../testing/selftests/mm/workingset_report.c | 306 ++++++++++++++++ .../testing/selftests/mm/workingset_report.h | 39 +++ .../selftests/mm/workingset_report_test.c | 330 ++++++++++++++++++ 6 files changed, 684 insertions(+) create mode 100644 tools/testing/selftests/mm/workingset_report.c create mode 100644 tools/testing/selftests/mm/workingset_report.h create mode 100644 tools/testing/selftests/mm/workingset_report_test.c diff --git a/tools/testing/selftests/mm/.gitignore b/tools/testing/selftests/mm/.gitignore index da030b43e43b..e5cd0085ab74 100644 --- a/tools/testing/selftests/mm/.gitignore +++ b/tools/testing/selftests/mm/.gitignore @@ -51,3 +51,4 @@ hugetlb_madv_vs_map mseal_test seal_elf droppable +workingset_report_test diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile index 0f8c110e0805..5c6a7464da6e 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -79,6 +79,7 @@ TEST_GEN_FILES += hugetlb_fault_after_madv TEST_GEN_FILES += hugetlb_madv_vs_map TEST_GEN_FILES += hugetlb_dio TEST_GEN_FILES += droppable +TEST_GEN_FILES += workingset_report_test ifneq ($(ARCH),arm64) TEST_GEN_FILES += soft-dirty @@ -138,6 +139,8 @@ $(TEST_GEN_FILES): vm_util.c thp_settings.c $(OUTPUT)/uffd-stress: uffd-common.c $(OUTPUT)/uffd-unit-tests: uffd-common.c +$(OUTPUT)/workingset_report_test: workingset_report.c + ifeq ($(ARCH),x86_64) BINARIES_32 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_32)) BINARIES_64 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_64)) diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index c5797ad1d37b..63782667381a 100755 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -75,6 +75,8 @@ separated by spaces: read-only VMAs - mdwe test prctl(PR_SET_MDWE, ...) +- workingset_report + test workingset reporting example: ./run_vmtests.sh -t "hmm mmap ksm" EOF @@ -456,6 +458,9 @@ CATEGORY="mkdirty" run_test ./mkdirty CATEGORY="mdwe" run_test ./mdwe_test +CATEGORY="workingset_report" run_test ./workingset_report_test \ + "${WORKINGSET_REPORT_TEST_FILE_PATH}" + echo "SUMMARY: PASS=${count_pass} SKIP=${count_skip} FAIL=${count_fail}" | tap_prefix echo "1..${count_total}" | tap_output diff --git a/tools/testing/selftests/mm/workingset_report.c b/tools/testing/selftests/mm/workingset_report.c new file mode 100644 index 000000000000..ee4dda5c371d --- /dev/null +++ b/tools/testing/selftests/mm/workingset_report.c @@ -0,0 +1,306 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "workingset_report.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../kselftest.h" + +#define SYSFS_NODE_ONLINE "/sys/devices/system/node/online" +#define PROC_DROP_CACHES "/proc/sys/vm/drop_caches" + +/* Returns read len on success, or -errno on failure. */ +static ssize_t read_text(const char *path, char *buf, size_t max_len) +{ + ssize_t len; + int fd, err; + size_t bytes_read = 0; + + if (!max_len) + return -EINVAL; + + fd = open(path, O_RDONLY); + if (fd < 0) + return -errno; + + while (bytes_read < max_len - 1) { + len = read(fd, buf + bytes_read, max_len - 1 - bytes_read); + + if (len <= 0) + break; + bytes_read += len; + } + + buf[bytes_read] = '\0'; + + err = -errno; + close(fd); + return len < 0 ? err : bytes_read; +} + +/* Returns written len on success, or -errno on failure. */ +static ssize_t write_text(const char *path, const char *buf, ssize_t max_len) +{ + int fd, len, err; + size_t bytes_written = 0; + + fd = open(path, O_WRONLY | O_APPEND); + if (fd < 0) + return -errno; + + while (bytes_written < max_len) { + len = write(fd, buf + bytes_written, max_len - bytes_written); + + if (len < 0) + break; + bytes_written += len; + } + + err = -errno; + close(fd); + return len < 0 ? err : bytes_written; +} + +static long read_num(const char *path) +{ + char buf[21]; + + if (read_text(path, buf, sizeof(buf)) <= 0) + return -1; + return (long)strtoul(buf, NULL, 10); +} + +static int write_num(const char *path, unsigned long n) +{ + char buf[21]; + + sprintf(buf, "%lu", n); + if (write_text(path, buf, strlen(buf)) < 0) + return -1; + return 0; +} + +long sysfs_get_refresh_interval(int nid) +{ + char file[128]; + + snprintf(file, sizeof(file), + "/sys/devices/system/node/node%d/workingset_report/refresh_interval", + nid); + return read_num(file); +} + +int sysfs_set_refresh_interval(int nid, long interval) +{ + char file[128]; + + snprintf(file, sizeof(file), + "/sys/devices/system/node/node%d/workingset_report/refresh_interval", + nid); + return write_num(file, interval); +} + +int sysfs_get_page_age_intervals_str(int nid, char *buf, int len) +{ + char path[128]; + + snprintf(path, sizeof(path), + "/sys/devices/system/node/node%d/workingset_report/page_age_intervals", + nid); + return read_text(path, buf, len); + +} + +int sysfs_set_page_age_intervals_str(int nid, const char *buf, int len) +{ + char path[128]; + + snprintf(path, sizeof(path), + "/sys/devices/system/node/node%d/workingset_report/page_age_intervals", + nid); + return write_text(path, buf, len); +} + +int sysfs_set_page_age_intervals(int nid, const char *const intervals[], + int nr_intervals) +{ + char file[128]; + char buf[1024]; + int i; + int err, len = 0; + + for (i = 0; i < nr_intervals; ++i) { + err = snprintf(buf + len, sizeof(buf) - len, "%s", intervals[i]); + + if (err < 0) + return err; + len += err; + + if (i < nr_intervals - 1) { + err = snprintf(buf + len, sizeof(buf) - len, ","); + if (err < 0) + return err; + len += err; + } + } + + snprintf(file, sizeof(file), + "/sys/devices/system/node/node%d/workingset_report/page_age_intervals", + nid); + return write_text(file, buf, len); +} + +int get_nr_nodes(void) +{ + char buf[22]; + char *found; + + if (read_text(SYSFS_NODE_ONLINE, buf, sizeof(buf)) <= 0) + return -1; + found = strstr(buf, "-"); + if (found) + return (int)strtoul(found + 1, NULL, 10) + 1; + return (long)strtoul(buf, NULL, 10) + 1; +} + +int drop_pagecache(void) +{ + return write_num(PROC_DROP_CACHES, 1); +} + +ssize_t sysfs_page_age_read(int nid, char *buf, size_t len) + +{ + char file[128]; + + snprintf(file, sizeof(file), + "/sys/devices/system/node/node%d/workingset_report/page_age", + nid); + return read_text(file, buf, len); +} + +/* + * Finds the first occurrence of "N\n" + * Modifies buf to terminate before the next occurrence of "N". + * Returns a substring of buf starting after "N\n" + */ +char *page_age_split_node(char *buf, int nid, char **next) +{ + char node_str[5]; + char *found; + int node_str_len; + + node_str_len = snprintf(node_str, sizeof(node_str), "N%u\n", nid); + + /* find the node prefix first */ + found = strstr(buf, node_str); + if (!found) { + ksft_print_msg("cannot find '%s' in page_idle_age", node_str); + return NULL; + } + found += node_str_len; + + *next = strchr(found, 'N'); + if (*next) + *(*next - 1) = '\0'; + + return found; +} + +ssize_t page_age_read(const char *buf, const char *interval, int pagetype) +{ + static const char * const type[ANON_AND_FILE] = { "anon=", "file=" }; + char *found; + + found = strstr(buf, interval); + if (!found) { + ksft_print_msg("cannot find %s in page_age", interval); + return -1; + } + found = strstr(found, type[pagetype]); + if (!found) { + ksft_print_msg("cannot find %s in page_age", type[pagetype]); + return -1; + } + found += strlen(type[pagetype]); + return (long)strtoul(found, NULL, 10); +} + +static const char *TEMP_FILE = "/tmp/workingset_selftest"; +void cleanup_file_workingset(void) +{ + remove(TEMP_FILE); +} + +int alloc_file_workingset(void *arg) +{ + int err = 0; + char *ptr; + int fd; + int ppid; + char *mapped; + size_t size = (size_t)arg; + size_t page_size = getpagesize(); + + ppid = getppid(); + + fd = open(TEMP_FILE, O_RDWR | O_CREAT); + if (fd < 0) { + err = -errno; + ksft_perror("failed to open temp file\n"); + goto cleanup; + } + + if (fallocate(fd, 0, 0, size) < 0) { + err = -errno; + ksft_perror("fallocate"); + goto cleanup; + } + + mapped = (char *)mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, + fd, 0); + if (mapped == NULL) { + err = -errno; + ksft_perror("mmap"); + goto cleanup; + } + + while (getppid() == ppid) { + sync(); + for (ptr = mapped; ptr < mapped + size; ptr += page_size) + *ptr = *ptr ^ 0xFF; + } + +cleanup: + cleanup_file_workingset(); + return err; +} + +int alloc_anon_workingset(void *arg) +{ + char *buf, *ptr; + int ppid = getppid(); + size_t size = (size_t)arg; + size_t page_size = getpagesize(); + + buf = malloc(size); + + if (!buf) { + ksft_print_msg("cannot allocate anon workingset"); + exit(1); + } + + while (getppid() == ppid) { + for (ptr = buf; ptr < buf + size; ptr += page_size) + *ptr = *ptr ^ 0xFF; + } + + free(buf); + return 0; +} diff --git a/tools/testing/selftests/mm/workingset_report.h b/tools/testing/selftests/mm/workingset_report.h new file mode 100644 index 000000000000..c5c281e4069b --- /dev/null +++ b/tools/testing/selftests/mm/workingset_report.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef WORKINGSET_REPORT_H_ +#define WORKINGSET_REPORT_H_ + +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif + +#include +#include +#include +#include +#include + +#define PAGETYPE_ANON 0 +#define PAGETYPE_FILE 1 +#define ANON_AND_FILE 2 + +int get_nr_nodes(void); +int drop_pagecache(void); + +long sysfs_get_refresh_interval(int nid); +int sysfs_set_refresh_interval(int nid, long interval); + +int sysfs_get_page_age_intervals_str(int nid, char *buf, int len); +int sysfs_set_page_age_intervals_str(int nid, const char *buf, int len); + +int sysfs_set_page_age_intervals(int nid, const char *const intervals[], + int nr_intervals); + +char *page_age_split_node(char *buf, int nid, char **next); +ssize_t sysfs_page_age_read(int nid, char *buf, size_t len); +ssize_t page_age_read(const char *buf, const char *interval, int pagetype); + +int alloc_file_workingset(void *arg); +void cleanup_file_workingset(void); +int alloc_anon_workingset(void *arg); + +#endif /* WORKINGSET_REPORT_H_ */ diff --git a/tools/testing/selftests/mm/workingset_report_test.c b/tools/testing/selftests/mm/workingset_report_test.c new file mode 100644 index 000000000000..89ff4e9d746e --- /dev/null +++ b/tools/testing/selftests/mm/workingset_report_test.c @@ -0,0 +1,330 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "workingset_report.h" + +#include +#include +#include +#include + +#include "../clone3/clone3_selftests.h" + +#define REFRESH_INTERVAL 5000 +#define MB(x) (x << 20) + +static void sleep_ms(int milliseconds) +{ + struct timespec ts; + + ts.tv_sec = milliseconds / 1000; + ts.tv_nsec = (milliseconds % 1000) * 1000000; + nanosleep(&ts, NULL); +} + +/* + * Checks if two given values differ by less than err% of their sum. + */ +static inline int values_close(long a, long b, int err) +{ + return labs(a - b) <= (a + b) / 100 * err; +} + +static const char * const PAGE_AGE_INTERVALS[] = { + "6000", "10000", "15000", "18446744073709551615", +}; +#define NR_PAGE_AGE_INTERVALS (ARRAY_SIZE(PAGE_AGE_INTERVALS)) + +static int set_page_age_intervals_all_nodes(const char *intervals, int nr_nodes) +{ + int i; + + for (i = 0; i < nr_nodes; ++i) { + int err = sysfs_set_page_age_intervals_str( + i, &intervals[i * 1024], strlen(&intervals[i * 1024])); + + if (err < 0) + return err; + } + return 0; +} + +static int get_page_age_intervals_all_nodes(char *intervals, int nr_nodes) +{ + int i; + + for (i = 0; i < nr_nodes; ++i) { + int err = sysfs_get_page_age_intervals_str( + i, &intervals[i * 1024], 1024); + + if (err < 0) + return err; + } + return 0; +} + +static int set_refresh_interval_all_nodes(const long *interval, int nr_nodes) +{ + int i; + + for (i = 0; i < nr_nodes; ++i) { + int err = sysfs_set_refresh_interval(i, interval[i]); + + if (err < 0) + return err; + } + return 0; +} + +static int get_refresh_interval_all_nodes(long *interval, int nr_nodes) +{ + int i; + + for (i = 0; i < nr_nodes; ++i) { + long val = sysfs_get_refresh_interval(i); + + if (val < 0) + return val; + interval[i] = val; + } + return 0; +} + +static pid_t clone_and_run(int fn(void *arg), void *arg) +{ + pid_t pid; + + struct __clone_args args = { + .exit_signal = SIGCHLD, + }; + + pid = sys_clone3(&args, sizeof(struct __clone_args)); + + if (pid == 0) + exit(fn(arg)); + + return pid; +} + +static int read_workingset(int pagetype, int nid, + unsigned long page_age[NR_PAGE_AGE_INTERVALS]) +{ + int i, err; + char buf[4096]; + + err = sysfs_page_age_read(nid, buf, sizeof(buf)); + if (err < 0) + return err; + + for (i = 0; i < NR_PAGE_AGE_INTERVALS; ++i) { + err = page_age_read(buf, PAGE_AGE_INTERVALS[i], pagetype); + if (err < 0) + return err; + page_age[i] = err; + } + + return 0; +} + +static ssize_t read_interval_all_nodes(int pagetype, int interval) +{ + int i, err; + unsigned long page_age[NR_PAGE_AGE_INTERVALS]; + ssize_t ret = 0; + int nr_nodes = get_nr_nodes(); + + for (i = 0; i < nr_nodes; ++i) { + err = read_workingset(pagetype, i, page_age); + if (err < 0) + return err; + + ret += page_age[interval]; + } + + return ret; +} + +#define TEST_SIZE MB(500l) + +static int run_test(int f(void)) +{ + int i, err, test_result; + long *old_refresh_intervals; + long *new_refresh_intervals; + char *old_page_age_intervals; + int nr_nodes = get_nr_nodes(); + + if (nr_nodes <= 0) { + ksft_print_msg("failed to get nr_nodes\n"); + return KSFT_FAIL; + } + + old_refresh_intervals = calloc(nr_nodes, sizeof(long)); + new_refresh_intervals = calloc(nr_nodes, sizeof(long)); + old_page_age_intervals = calloc(nr_nodes, 1024); + + if (!(old_refresh_intervals && new_refresh_intervals && + old_page_age_intervals)) { + ksft_print_msg("failed to allocate memory for intervals\n"); + return KSFT_FAIL; + } + + err = get_refresh_interval_all_nodes(old_refresh_intervals, nr_nodes); + if (err < 0) { + ksft_print_msg("failed to read refresh interval\n"); + return KSFT_FAIL; + } + + err = get_page_age_intervals_all_nodes(old_page_age_intervals, nr_nodes); + if (err < 0) { + ksft_print_msg("failed to read page age interval\n"); + return KSFT_FAIL; + } + + for (i = 0; i < nr_nodes; ++i) + new_refresh_intervals[i] = REFRESH_INTERVAL; + + for (i = 0; i < nr_nodes; ++i) { + err = sysfs_set_page_age_intervals(i, PAGE_AGE_INTERVALS, + NR_PAGE_AGE_INTERVALS - 1); + if (err < 0) { + ksft_print_msg("failed to set page age interval\n"); + test_result = KSFT_FAIL; + goto fail; + } + } + + err = set_refresh_interval_all_nodes(new_refresh_intervals, nr_nodes); + if (err < 0) { + ksft_print_msg("failed to set refresh interval\n"); + test_result = KSFT_FAIL; + goto fail; + } + + sync(); + drop_pagecache(); + + test_result = f(); + +fail: + err = set_refresh_interval_all_nodes(old_refresh_intervals, nr_nodes); + if (err < 0) { + ksft_print_msg("failed to restore refresh interval\n"); + test_result = KSFT_FAIL; + } + err = set_page_age_intervals_all_nodes(old_page_age_intervals, nr_nodes); + if (err < 0) { + ksft_print_msg("failed to restore page age interval\n"); + test_result = KSFT_FAIL; + } + return test_result; +} + +static char *file_test_path; +static int test_file(void) +{ + ssize_t ws_size_ref, ws_size_test; + int ret = KSFT_FAIL, i; + pid_t pid = 0; + + if (!file_test_path) { + ksft_print_msg("Set a path to test file workingset\n"); + return KSFT_SKIP; + } + + ws_size_ref = read_interval_all_nodes(PAGETYPE_FILE, 0); + if (ws_size_ref < 0) + goto cleanup; + + pid = clone_and_run(alloc_file_workingset, (void *)TEST_SIZE); + if (pid < 0) + goto cleanup; + + read_interval_all_nodes(PAGETYPE_FILE, 0); + sleep_ms(REFRESH_INTERVAL); + + for (i = 0; i < 3; ++i) { + sleep_ms(REFRESH_INTERVAL); + ws_size_test = read_interval_all_nodes(PAGETYPE_FILE, 0); + ws_size_test += read_interval_all_nodes(PAGETYPE_FILE, 1); + if (ws_size_test < 0) + goto cleanup; + + if (!values_close(ws_size_test - ws_size_ref, TEST_SIZE, 10)) { + ksft_print_msg( + "file working set size difference too large: actual=%ld, expected=%ld\n", + ws_size_test - ws_size_ref, TEST_SIZE); + goto cleanup; + } + } + ret = KSFT_PASS; + +cleanup: + if (pid > 0) + kill(pid, SIGKILL); + cleanup_file_workingset(); + return ret; +} + +static int test_anon(void) +{ + ssize_t ws_size_ref, ws_size_test; + pid_t pid = 0; + int ret = KSFT_FAIL, i; + + ws_size_ref = read_interval_all_nodes(PAGETYPE_ANON, 0); + ws_size_ref += read_interval_all_nodes(PAGETYPE_ANON, 1); + if (ws_size_ref < 0) + goto cleanup; + + pid = clone_and_run(alloc_anon_workingset, (void *)TEST_SIZE); + if (pid < 0) + goto cleanup; + + sleep_ms(REFRESH_INTERVAL); + read_interval_all_nodes(PAGETYPE_ANON, 0); + + for (i = 0; i < 5; ++i) { + sleep_ms(REFRESH_INTERVAL); + ws_size_test = read_interval_all_nodes(PAGETYPE_ANON, 0); + ws_size_test += read_interval_all_nodes(PAGETYPE_ANON, 1); + if (ws_size_test < 0) + goto cleanup; + + if (!values_close(ws_size_test - ws_size_ref, TEST_SIZE, 10)) { + ksft_print_msg( + "anon working set size difference too large: actual=%ld, expected=%ld\n", + ws_size_test - ws_size_ref, TEST_SIZE); + goto cleanup; + } + } + ret = KSFT_PASS; + +cleanup: + if (pid > 0) + kill(pid, SIGKILL); + return ret; +} + + +#define T(x) { x, #x } +struct workingset_test { + int (*fn)(void); + const char *name; +} tests[] = { + T(test_anon), + T(test_file), +}; +#undef T + +int main(int argc, char **argv) +{ + int i, err; + + if (argc > 1) + file_test_path = argv[1]; + + for (i = 0; i < ARRAY_SIZE(tests); i++) { + err = run_test(tests[i].fn); + ksft_test_result_code(err, tests[i].name, NULL); + } + return 0; +} From patchwork Wed Nov 27 02:57:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13886513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39002D66BAD for ; Wed, 27 Nov 2024 02:58:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2B3FC8D0007; Tue, 26 Nov 2024 21:58:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 264228D0001; Tue, 26 Nov 2024 21:58:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 08FEC8D0007; Tue, 26 Nov 2024 21:57:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D63708D0001 for ; Tue, 26 Nov 2024 21:57:59 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 896101C7810 for ; Wed, 27 Nov 2024 02:57:59 +0000 (UTC) X-FDA: 82830365436.28.BC8A701 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf14.hostedemail.com (Postfix) with ESMTP id E69C2100002 for ; Wed, 27 Nov 2024 02:57:51 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fLDcEwrA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3tIpGZwcKCCUZVBODIVHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--yuanchu.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3tIpGZwcKCCUZVBODIVHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--yuanchu.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732676275; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FdWbI+s0p0wBgh+VsWAydrlLqfgeB+x4CgOEMGbe8TM=; b=vMYkhtZS2kmUorq7l6CJpldSBzwr4muvLIA917xjFZX0NInWBGQL2pXeezq50iamRMBJhf gM5G8KVQNmiUfzThkTQ+Qi1IIFrezgWH7luKaeaBCOvkOkSTcNGpxchtF3iEoHjXs69s9a lgHracjIZpgmuxjfM58lT6G3AxTP3PQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732676275; a=rsa-sha256; cv=none; b=tDj8jrQcOkODjbA7ECb+6DwlqXMGPKO+wB9QJ8upAApVPtfIBditOQhzcpJLXG/Tm/SVXz hf63nTfKbqSwg7YQQvcbYrooyqqr+8q4rYy9e5wjvTyaNbW0eKsBvsiYo//S5YPgS5BBNW 9DlwzA6qU8+GKeD9d1gcBdiHhmTzU+M= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fLDcEwrA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3tIpGZwcKCCUZVBODIVHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--yuanchu.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3tIpGZwcKCCUZVBODIVHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--yuanchu.bounces.google.com Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ee0c9962daso342528a91.2 for ; Tue, 26 Nov 2024 18:57:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1732676276; x=1733281076; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FdWbI+s0p0wBgh+VsWAydrlLqfgeB+x4CgOEMGbe8TM=; b=fLDcEwrAgcenfq5q/jklvHDttU78q4n0Y3zlJCDdX8hbScFAre4BpFX6OFPinL9qwe RQln4hB+ou9w+fw7tAt06LdDFc51KiYJFJvXRH/+tDTwaqctK8Y4PhRv5YtNimO7ORLV nFO84G2qLuATki8x5eHTdLagI+t/bmFrVwvGjvhL5+3oMsc8oPpmtnlMmC/em1J5jz+H Mg5BoeQKk+mkiQM9iYRp3PJn9r2tRgtpnfr/A4tdYSzG4tP7Hj/O3A4uXGazEfg8sGbu WrXN/mOzw+XoX00yO7bjmfo26xe6n9tlGdu1kpqwdNZf5zmR7r2lJnNF6kAVNb2i/dHd O6WQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732676276; x=1733281076; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FdWbI+s0p0wBgh+VsWAydrlLqfgeB+x4CgOEMGbe8TM=; b=IaLBSsQZ38jlFHYME9YDpGySQQ6BlLJO9zOOclSVjfBUwvrbhUe4tTvPlo2FLI33Yk MHhDip0RSv4u7gQkVpXxGm3awAO8xu8iH+vyd/r6rGzh4DFKvlz5zAsH0dApFi7FWJ4d XwUHR/UtsQyXp3FEpcy/MroCHCIlTO2iFafFLN9+Dn8q3pcIESjOfwMiwBgneZWVri+n 64vJCKd9CDS961jGMfrhq7duDtNDPzbEcNv+yrYYvxAS0AsWREnFjYre2EzL3WxIJHB8 9IwsaSylmH68zwcDNYVaaON5zjbYPkQKtPqOhWBkOG9XnvObD4qF4BDiW2HQnkjLCJ6U zEaw== X-Forwarded-Encrypted: i=1; AJvYcCXG0WEssSBxDum/+66TZDnFmdPexQsqGesPLkTwU6I+aSudh5eBvKf+IOtQ20EqvRkObH4kmlYcFg==@kvack.org X-Gm-Message-State: AOJu0YxOuYaUfCv+QuC75UcLBrlGifU/ELIc/4nb+ILV9eJXnu/8EgNf t2KkDaGgRHTHG9rdO+3Ke/r/Ct1SXqsd8DKBKWGWXQUnDgOzVUkcMrDKtxUP4/dFGT93CBrQBJ5 8JqXLvw== X-Google-Smtp-Source: AGHT+IGwhfii5+MtQdbRihMKL41ITlobJS3ldyPZjP/ITZaiG2+mPbpCQb3KZbCSJSGpnWkssvQZr9R3iWRd X-Received: from pjbok13.prod.google.com ([2002:a17:90b:1d4d:b0:2da:ac73:93e5]) (user=yuanchu job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:1d10:b0:2ea:aa56:4b0 with SMTP id 98e67ed59e1d1-2ee08e99941mr2057809a91.3.1732676276518; Tue, 26 Nov 2024 18:57:56 -0800 (PST) Date: Tue, 26 Nov 2024 18:57:26 -0800 In-Reply-To: <20241127025728.3689245-1-yuanchu@google.com> Mime-Version: 1.0 References: <20241127025728.3689245-1-yuanchu@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241127025728.3689245-8-yuanchu@google.com> Subject: [PATCH v4 7/9] Docs/admin-guide/mm/workingset_report: document sysfs and memcg interfaces From: Yuanchu Xie To: Andrew Morton , David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Tejun Heo , Johannes Weiner , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , " =?utf-8?q?Eugenio_P=C3=A9rez?= " , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Mike Rapoport , Shuah Khan , Christian Brauner , Daniel Watson , Yuanchu Xie , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, linux-mm@kvack.org, linux-kselftest@vger.kernel.org X-Stat-Signature: 1sr5ywf3zdswir3bq16o5jk3towbdonu X-Rspamd-Queue-Id: E69C2100002 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1732676271-112392 X-HE-Meta: U2FsdGVkX19vaZfFo7dJLXUX8gyzIH8EJEJEKJ/CxDHisutYbwkK3gRCeGpCP9wkCqoUD6PaKKte0hhcf2KhDcr65TMEHGjPZ+OxX7AVlNZUrXIvGvp+NqA1olS4rPMK4vTaglht7oo9x9UWi2paJg5L0FvApVqR478Kf437qejcCTasD81NAfwDNKxUW3ww7Az3r+TJzeYwJlUWe3QHCoy9IhLQdqkTemGcpAto1fv1lsPYgOTRNi2Fts/vglfpiHqTWCyqUAaBL8tbB2OKCg6m/blfs7f0xdY2ldKLgtYt4v8cfWjljDzHnmgVU10utD0MO8zSYWbUDHrWZYd1OTNeY6tMPV+KWXBDYC/lTKBIUj7XxXsvkL4z6G/mgSVCy7ifU5CCGbqgAzrYGQ6hM0wr3gADIrrpLQo5Iha731X4iJW3Y3nV43v9AQDGZJFDqOoSzBMwDdZfbIw1RtHYjTOvq3laUjb8WwkuFuvsNaN6fZlWu4S5n2jOm8JeVuLxavlwoeQB2QZy5f9kaScW5QDgTEHo//SXaptcaAmQ/J+oSzO6Vapt8sor5mBZzJvx02WgeRWzAh6GycpMaNGoI6HcCvJa7777VuCw+Cbz8Sm8PanJlQk1xxQZbeGdUUBC5NZY9ZdsorueDCk5gd/JWRxhZj5tgqHU4l+mpY1v+xbQM3mIKviVbiNrO6XI09e7x7EBFXxkJYJbeuGnLj23ClsTWvkCEd3KmTjPuSgwIY0F56QkUuQjcwU42zoH59vbSXeTLUOmgNfOH7PocOxJwVmI5vmuECoZNjUm2KC3yazcYFoH0Rvzrvto/5xn6m8G8+QNkQ2tvHBFnKpP+mreOMO2fPAPkCTxzGwCmF1WIye/MTt0R/MciAKka8Df1O2YTCkE5hSgS5qX02wJl7AqSG9hddmIk5vMETRcEHF7CtBKrJYZELScBOtYwc4b1hCIxZjnAgCj0VZK67xDeVZ efESxiOP f1Krd6Yp2hVuH2evlgpz/6+6HeM5rPAPLILkG4Jq7h0kZEDqCpLeYgkJAODPCpsxFN0f1PicUbaMVUTkgFEqUUhpt0xdLDsdSgcxB4X702W2TU9veYK7/RWwSMcaMyAPOWcFI5gp9G+dXY32NGqwbQifAd7VJMb73LGsSkX4oscHxrbamo7FA/uiSPXHytWRE2RkdNeYVfWmHfZegfnjfS/uJFLqyLvfkS3RGULlfZMO/1VbtTSeEvgIHhk8V+pLsx2d+RnDhiXLDBbevr6hnI+cgSs6hbOvAUvd6G+njzPO0uduaGIUexlc47ZHF2peTGJ9U/qm8ms8+/WHjeB+daMKuCIXZHzC47Z5trEfq+w15wGpCnF2ua7KU6i8oOYE1/XhvZOItISZ/CZ5njOQUjpWZ0ZDWaRo5JVimvY5IovT3To+rK2ZvTUBhMdtS6Vw+EIixhs/hlkJKBy5k+0HvNH0cbuoTUcsfWpupwvlkpwl8oQFdpGKyTNXp3GQBAAKZj8W8awOvzmMQuIzUZ2fYTSG5K9EwTy4hJjCIQDo2mH0SPh9AQpdkK45tApmCGt6Dl7R++NvCzjUMqtm0+uvC7IRFMo/MRd3eLY0f X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add workingset reporting documentation for better discoverability of its sysfs and memcg interfaces. Also document the required kernel config to enable workingset reporting. Signed-off-by: Yuanchu Xie --- Documentation/admin-guide/mm/index.rst | 1 + .../admin-guide/mm/workingset_report.rst | 105 ++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 Documentation/admin-guide/mm/workingset_report.rst diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-guide/mm/index.rst index 8b35795b664b..61a2a347fc91 100644 --- a/Documentation/admin-guide/mm/index.rst +++ b/Documentation/admin-guide/mm/index.rst @@ -41,4 +41,5 @@ the Linux memory management. swap_numa transhuge userfaultfd + workingset_report zswap diff --git a/Documentation/admin-guide/mm/workingset_report.rst b/Documentation/admin-guide/mm/workingset_report.rst new file mode 100644 index 000000000000..0969513705c4 --- /dev/null +++ b/Documentation/admin-guide/mm/workingset_report.rst @@ -0,0 +1,105 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +Workingset Report +================= +Workingset report provides a view of memory coldness in user-defined +time intervals, e.g. X bytes are Y milliseconds cold. It breaks down +the user pages in the system per-NUMA node, per-memcg, for both +anonymous and file pages into histograms that look like: +:: + + 1000 anon=137368 file=24530 + 20000 anon=34342 file=0 + 30000 anon=353232 file=333608 + 40000 anon=407198 file=206052 + 9223372036854775807 anon=4925624 file=892892 + +The workingset reports can be used to drive proactive reclaim, by +identifying the number of cold bytes in a memcg, then writing to +``memory.reclaim``. + +Quick start +=========== +Build the kernel with the following configurations. The report relies +on Multi-gen LRU for page coldness. + +* ``CONFIG_LRU_GEN=y`` +* ``CONFIG_LRU_GEN_ENABLED=y`` +* ``CONFIG_WORKINGSET_REPORT=y`` + +Optionally, the aging kernel daemon can be enabled with the following +configuration. +* ``CONFIG_WORKINGSET_REPORT_AGING=y`` + +Sysfs interfaces +================ +``/sys/devices/system/node/nodeX/workingset_report/page_age`` provides +a per-node page age histogram, showing an aggregate of the node's lruvecs. +Reading this file causes a hierarchical aging of all lruvecs, scanning +pages and creates a new Multi-gen LRU generation in each lruvec. +For example: +:: + + 1000 anon=0 file=0 + 2000 anon=0 file=0 + 100000 anon=5533696 file=5566464 + 18446744073709551615 anon=0 file=0 + +``/sys/devices/system/node/nodeX/workingset_report/page_age_intervals`` +is a comma-separated list of time in milliseconds that configures what +the page age histogram uses for aggregation. For the above histogram, +the intervals are:: + + 1000,2000,100000 + +``/sys/devices/system/node/nodeX/workingset_report/refresh_interval`` +defines the amount of time the report is valid for in milliseconds. +When a report is still valid, reading the ``page_age`` file shows +the existing valid report, instead of generating a new one. + +``/sys/devices/system/node/nodeX/workingset_report/report_threshold`` +specifies how often the userspace agent can be notified for node +memory pressure, in milliseconds. When a node reaches its low +watermarks and wakes up kswapd, programs waiting on ``page_age`` are +woken up so they can read the histogram and make policy decisions. + +Memcg interface +=============== +While ``page_age_interval`` is defined per-node in sysfs, ``page_age``, +``refresh_interval`` and ``report_threshold`` are available per-memcg. + +``/sys/fs/cgroup/.../memory.workingset.page_age`` +The memcg equivalent of the sysfs workingset page age histogram +breaks down the workingset of this memcg and its children into +page age intervals. Each node is prefixed with a node header and +a newline. Non-proactive direct reclaim on this memcg can also +wake up userspace agents that are waiting on this file. +E.g. +:: + + N0 + 1000 anon=0 file=0 + 2000 anon=0 file=0 + 3000 anon=0 file=0 + 4000 anon=0 file=0 + 5000 anon=0 file=0 + 18446744073709551615 anon=0 file=0 + +``/sys/fs/cgroup/.../memory.workingset.refresh_interval`` +The memcg equivalent of the sysfs refresh interval. A per-node +number of how much time a page age histogram is valid for, in +milliseconds. +E.g. +:: + + echo N0=2000 > memory.workingset.refresh_interval + +``/sys/fs/cgroup/.../memory.workingset.report_threshold`` +The memcg equivalent of the sysfs report threshold. A per-node +number of how often userspace agent waiting on the page age +histogram can be woken up, in milliseconds. +E.g. +:: + + echo N0=1000 > memory.workingset.report_threshold From patchwork Wed Nov 27 02:57:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13886514 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A6D6D66BAC for ; Wed, 27 Nov 2024 02:58:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD7428D0008; Tue, 26 Nov 2024 21:58:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D5FC88D0001; Tue, 26 Nov 2024 21:58:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C00188D0008; Tue, 26 Nov 2024 21:58:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9ED418D0001 for ; Tue, 26 Nov 2024 21:58:01 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 475311615B1 for ; Wed, 27 Nov 2024 02:58:01 +0000 (UTC) X-FDA: 82830365436.15.56DBBC2 Received: from mail-oa1-f73.google.com (mail-oa1-f73.google.com [209.85.160.73]) by imf29.hostedemail.com (Postfix) with ESMTP id DB12212000C for ; Wed, 27 Nov 2024 02:57:51 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ry9A6Yye; spf=pass (imf29.hostedemail.com: domain of 3topGZwcKCCcbXDQFKXJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--yuanchu.bounces.google.com designates 209.85.160.73 as permitted sender) smtp.mailfrom=3topGZwcKCCcbXDQFKXJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732676275; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PZpBMHL8IdsZHMO87vdTflNT6N6icPZfEjrZz1n0oFE=; b=ZZ+45ApPZ8yR/DCG2qW/HjlWFu7yaI01aOj72P+s7F9/QfroWpaKH41rUd6CK3Aud+aaJy +gK9Tna18Jkr6LQT84oJ5w/i5iL31J3CGMpx//KUZFLWGLMeMoM/Hd/f4z8fXxbtjHfgBc yRlN5AratyKO80G0qerOXcZ/i38wWNI= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ry9A6Yye; spf=pass (imf29.hostedemail.com: domain of 3topGZwcKCCcbXDQFKXJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--yuanchu.bounces.google.com designates 209.85.160.73 as permitted sender) smtp.mailfrom=3topGZwcKCCcbXDQFKXJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732676275; a=rsa-sha256; cv=none; b=yWzAzsLbCVZEPbkK9OmgSPz9d4DzQZQgp8JUIbLpGXdCYGNVOJnmgj/ril5v6CCklrwX4K RJDad01lfimyKN8xVDlOIS25ruyofC0+TS05jbxPQq9H3BIO8YsHc0goPi9q5ePVUueJ8d k0/SbdA9fZ2TJEp9QecxIjuglahtjbE= Received: by mail-oa1-f73.google.com with SMTP id 586e51a60fabf-29678d451b1so3433448fac.3 for ; Tue, 26 Nov 2024 18:57:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1732676278; x=1733281078; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PZpBMHL8IdsZHMO87vdTflNT6N6icPZfEjrZz1n0oFE=; b=ry9A6YyeQPQpex0mNem+Vah0XgQYL2TuRKppX2v5QieR22nbebQfARGOhblYGdWteb mVQQQNcpLd6Qs6cpRc2ws4fy/CWiKpQsHxcK//tSFTcyDTuvPniRUkjZ4n6nmCKzhL7x xSDA3zDzbIojE6kuFyDSNMrboMn8S4QKkMhtLB2TF5+qwELvDO6UmDH6mbvK8XQCFCUx zt/LRDjmZLPHP/72ylDU+k1NmRka/rQBWQZ1V0PRbvJJJpZZRuhuucmNE/ffUX8zDIPI 1jdXVufE/NIFfIEr38WrzKH/LhCpxF824I/kWiGYgYR6CAAnJFDDrFMX8RKorEkvOW6H jG9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732676278; x=1733281078; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PZpBMHL8IdsZHMO87vdTflNT6N6icPZfEjrZz1n0oFE=; b=FhFPZjyVxcmZQLbVSlUxUPBBpOrEEhx6R200KaQ+I3UjEgXIiLLI83PtlxQGj9BMAs dB61rcT/tcnxuW21bxppts3fTFCQY6TQqAxEv3QGFzK+LxMyAsNCzGiSVR66AmPoa/uD XpFMf/ewQJVzU/safxE7h/xpEHnZ6Cr06LUl7WC3vA28MSEvcaNJQal1Vhlno23+xLbh siQ5ol84E/qMQZprmdwxWuslc5TMUmJ6B2SCZSiBrK4+7JratL9/4dgyQ4m0qUYfkZes l2AGqHWIVz2qRLaN/zTPQHVQDLJiUblr1Uw/VAD2VYEuswXVpTawHDiq9pVzCmhGMyod Ia6g== X-Forwarded-Encrypted: i=1; AJvYcCVIWqmex5HB3BOu7p5yIs7tu4sEPVz9y5tsMseuRWRnVkjAbfHF7R6+swUPBfcMMqR6YZ6lhr2jIg==@kvack.org X-Gm-Message-State: AOJu0YwiqBWsKsJKcCULv/fBRb5klcLy2XlngipXGDF4pCFEqMBiP3R7 KOeHbrAUpck/6J3Gbyj/iwta5Qk/UqBLmOuMJZXVxzoBfJneGD9rtmLcsO4vhb21dq1vEnb7/cI X9FwUkQ== X-Google-Smtp-Source: AGHT+IFKHf2rAZNzRLIYi5pyoUxBdmbZ99C2meBLNg4KMzGEhTQ6C34gvUg4FAtRwQPJnnGabMdsTk0k6Rq/ X-Received: from oabuz16.prod.google.com ([2002:a05:6870:af90:b0:295:ebc2:deb2]) (user=yuanchu job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6870:5d8f:b0:297:270c:575 with SMTP id 586e51a60fabf-29dc40097dbmr1170968fac.16.1732676278453; Tue, 26 Nov 2024 18:57:58 -0800 (PST) Date: Tue, 26 Nov 2024 18:57:27 -0800 In-Reply-To: <20241127025728.3689245-1-yuanchu@google.com> Mime-Version: 1.0 References: <20241127025728.3689245-1-yuanchu@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241127025728.3689245-9-yuanchu@google.com> Subject: [PATCH v4 8/9] Docs/admin-guide/cgroup-v2: document workingset reporting From: Yuanchu Xie To: Andrew Morton , David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Tejun Heo , Johannes Weiner , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , " =?utf-8?q?Eugenio_P=C3=A9rez?= " , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Mike Rapoport , Shuah Khan , Christian Brauner , Daniel Watson , Yuanchu Xie , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, linux-mm@kvack.org, linux-kselftest@vger.kernel.org X-Rspamd-Server: rspam05 X-Stat-Signature: btwgkz8ccnanrtq4tx5n6jde6gabcgop X-Rspamd-Queue-Id: DB12212000C X-Rspam-User: X-HE-Tag: 1732676271-530389 X-HE-Meta: U2FsdGVkX1/zvtK3exkbP2EWPYLrVaagdJhspjKLkDbxVgCGI49UFs51+3j6vxTHQcztsN+FRDdquV/leQkesYn60GQlyMaf4Ak7Ml3eQadhHLYZ4ZSRYXrtSpnMUTRUpjHgcBgRqFxO2de0BZhsS3f9EmXfoSkOj3RMm7hJvlirjUgVsFhz/CEm9RHl1+Dt80B35y9sav9uBhUyhj/P+4GCtBVEXFsA3XfPm5G2yl5u60GONo+zGyUFl8V633Ez2jVaC8nbcuJmHkWLVFZ/BNHgVJMtBWjZR409d+hoXOMXKYrYMVnWhamz9h4/Wd6LSgHlNlnPWgzpqn9j3WIxlz5gEV/E1BAbRVvMQd4S3/P2cLoFGKV1YlQ0lzBROTM4FdIw5SHsP3vnsO664/LycWY2jK/DhrkYLGVur+rEUQdlP3deeLe75UxZYUY8zWLjPUSXgjvRLijuNgKHsbFQvOhyoWgB2C5yz84iTuvSv+ztkPArpn3GyaEvfJHbMColb6ATsE5k5Seeqp51fW3R5f6mOxnfAFSLz+cdSU54CosxbXSMu2owfNLGBBOFlPd3kcpUgcDC9IH3bPjwzBgFF0oxro+ilH74kcqM0PrhUdlaHSMSh8diEzXvBC9REtYsjc6VYxBFU2FVsoqLKMpxtHYk/ifuMWcPAnpqzH10eIdda7qlOLUsOdIAIJDL25Tmxs6fXgKcR/OZoFGm4OHb/7hBt+XFc2vsJNhDffaCcBwrt8wzCNJjnff+8rw89B7olnjCxHVcPp2wmVgldT+5RJ0XIj6D85ZcIzjjRwuJ61zCaha6BmOdmiKcI+JW1I+kyXe8kcn0mX77h31GVN2OWv5I8oco4CsoJ63P4PabZ3/dQnxKrdQJLDiWdD/1rr/LRpTHa7c8l904QFNBQR80hPA1OY2Sn5VZCBXfVrH3V8XMfRPn5WB95mqQc2a7kbfFiqMNvMMvoPAnaDazFLJ kf9tm7NT RMjCFzHC4zpiDwF0QsSDs1msJcwvc4ME4fbBLnikrPyzO5XLpzNEuJOTbYv8qVlXiTXe7H691g4zy8xBZxvfcTarKpMK7LYAxXcPZAEQGQe4pYSn6axKzIHdY1wANdElQJl4BpOx/EDhVrq11AV83hH34Xt+gCjaQhrzmty5qFEkTm5cKVp7IV39PmLHoVWc7LZ0SHgOTq5doW2lc6Ywzj42F4NFFYxxApWcsjzyn1OB3yGPP73MpYMGe/FmhsG/EQWUT/EE7vp8Bp2OFpQFJuD0cUgA5aNJYTak1/F63w20SWzaOEpmQkvO00dEo6rqbjmCT+ZIAH6yTTLast+QIFNGlegYiOuo6WAejcYjOQZLPNTBgLYrpcMmgkbTo87Dhzvt0v+ikhF+6WhLBFoFxpKp14M/HEawtScncgPDCEqM3hQCbLFcjj4dbEQg3H4bJWxPiqiQLuyk4a1v/YkB9SC4GNtfGxu7+cX+bMTl1Ofb2DHfIPlDQmrN2ljg9EjPEQtW/3toJdiPZnykbrkvfPamCKh/Jjjp7oWZCrO6+hDS26mMrcTtH1DszcAtiU+nvpweeCbR3pTk9dBfRVt1h75GYKQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add workingset reporting documentation for better discoverability of its memcg interfaces. Point the memcg documentation to Documentation/admin-guide/mm/workingset_report.rst for more details. Signed-off-by: Yuanchu Xie --- Documentation/admin-guide/cgroup-v2.rst | 35 +++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 2cb58daf3089..67a183f08245 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -1784,6 +1784,41 @@ The following nested keys are defined. Shows pressure stall information for memory. See :ref:`Documentation/accounting/psi.rst ` for details. + memory.workingset.page_age + A read-only histogram which exists on non-root cgroups. + + This breaks down the cgroup's memory footprint into different + types of memory and groups them per-node into user-defined coldness + bins. + + The output format of memory.workingset.page_age is:: + + N0 + type= + type= + ... + 18446744073709551615 type= + + The type of memory can be anon, file, or new types added later. + Don't rely on the types remaining fixed. See + :ref:`Documentation/admin-guide/mm/workingset_report.rst ` + for details. + + memory.workingset.refresh_interval + A read-write nested-keyed file which exists on non-root cgroups. + + Setting it to a non-zero value for any node enables working set + reporting for that node. The default is 0 for each node. See + :ref:`Documentation/admin-guide/mm/workingset_report.rst ` + for details. + + memory.workingset.report_threshold + A read-write nested-keyed file which exists on non-root cgroups. + + The amount of milliseconds to wait before reporting the working + set again. The default is 0 for each node. See + :ref:`Documentation/admin-guide/mm/workingset_report.rst ` + for details. Usage Guidelines ~~~~~~~~~~~~~~~~ From patchwork Wed Nov 27 02:57:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13886515 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2D1FD66BAC for ; Wed, 27 Nov 2024 02:58:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D61108D0009; Tue, 26 Nov 2024 21:58:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D120F8D0001; Tue, 26 Nov 2024 21:58:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AEEBF8D0009; Tue, 26 Nov 2024 21:58:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 86E628D0001 for ; Tue, 26 Nov 2024 21:58:03 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 33CA281512 for ; Wed, 27 Nov 2024 02:58:03 +0000 (UTC) X-FDA: 82830365520.04.44345AF Received: from mail-oa1-f74.google.com (mail-oa1-f74.google.com [209.85.160.74]) by imf19.hostedemail.com (Postfix) with ESMTP id DDDB91A000B for ; Wed, 27 Nov 2024 02:57:54 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nwLAAgwn; spf=pass (imf19.hostedemail.com: domain of 3uIpGZwcKCCkdZFSHMZLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yuanchu.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3uIpGZwcKCCkdZFSHMZLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732676276; a=rsa-sha256; cv=none; b=eidfgwoWOaqXg20Avb2YSIFv6n6uLT3DIjPnEvSwgKOgmM94S9/bOk702jNO9HMCnUqGYd zz3R2+5/tQxj3LNYnfxnw8cjDpoR0H1TSq96f3gYQSpNWBKv03+JrVDhdo7pcM+Z/HMX/r sd14Mk+4ZCXYvYqXY8bXgeozm2uP3L0= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nwLAAgwn; spf=pass (imf19.hostedemail.com: domain of 3uIpGZwcKCCkdZFSHMZLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yuanchu.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3uIpGZwcKCCkdZFSHMZLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732676276; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=J230iZRfwuVra4Pz4XMka7O3vESUEZ4tRN3Joobiohw=; b=2K4oVeNsolRwNvkb9hcVg71tf5RRKTlgSy7j1o6pMreorG1C3AFii/eG0T/lQsnWVcH9in pj3lPMu13cuXpl4ipqbAs49IdsIxYOniehZun2skOp+I0poyt6GIZG4v1V12M0rOfnPyIB oYYTXFKy3ISFkJckNdZtIqarSxIS5sA= Received: by mail-oa1-f74.google.com with SMTP id 586e51a60fabf-297066ae069so6879187fac.3 for ; Tue, 26 Nov 2024 18:58:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1732676280; x=1733281080; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=J230iZRfwuVra4Pz4XMka7O3vESUEZ4tRN3Joobiohw=; b=nwLAAgwnVUi13T+6z394bTrApUc+fXqrB/K+N+AMX71x3KW9EkmQvdp0XqjmL3Sxlj l0Ci0n/OekMIy79dvUDm5g4MWka3xDhTox+dsIYvZ9H6HtnJCTBULwckQl8ubhKoVzF+ AM7hjLi1qfIqm5DUcfH15X1n8h9q2xnHnYjP0BdvOyVSfVyaUZVYRntQdir3NfdU6qvK xyXoHVONnL8uf8fDOUI7apcLXsAfhtTr2K6N9k4uU7eDWPTI72Jz5WDzG1XSrypIbrJI jZvkzrxhS+9TpqVAnQckrEfDsRYNGc+YYKrJDlHETE6YbKW5QbsfCavpmIWtAKa4qHkB 0bQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732676280; x=1733281080; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=J230iZRfwuVra4Pz4XMka7O3vESUEZ4tRN3Joobiohw=; b=fZwwtsJR3oxsX22HWnzVvWCkNN7atHhN5rupCup7ei8t8bccaN8Lho6IrDdtgi2XCK aPqIKsZVvd9Of8jqvzn/q+TCrFn2b8vLtP1m4U7ttmththzgk2Jns2UXqB2cBUqnWZyO eye+mJkhHG3sp8h9nRhIjmkMMz4RTAN8uE4mQte6hokYo/izdOu9jLCXf9oTBU+hrMyR VToMhrl7iE6cugDxpYwTTEC3JZqtIu5xmvVJcYqMUXMxQu+5dqGBTwL1g+Qqan2+fzAR C6gv8J74ZcJqwfxtTt1R6XVJTbEo2RZPyIzi9ymQSEH8ype2ABlaStyF20gaHbjuv8bu NEnA== X-Forwarded-Encrypted: i=1; AJvYcCWnzTkL02x+d5+r1rTNfcd78GCWZmePCRgimoyni0FAXYvGxVkUIrXx/HWlh7tsmzidbQZBiOKbDg==@kvack.org X-Gm-Message-State: AOJu0Yz6FAy4qTAyNelXZ3L4CWcdiFkqOTjVQ3FieMv7KyJSWc/WeD2v xcinDRi70RcXi5zQW2CJBERctDLdnZ2i4shJEx9D9LuYY4p4rYyuJjKTmnAJaXro3CS0YjPoan7 9npZREg== X-Google-Smtp-Source: AGHT+IFIJL8h5DmsRNBM5ZiX5QfIqL9o1tI1OHXYDHS0S9zx9IEBE7CHBXLRc6dJk0foy6FX7GvuZ5iRwwn6 X-Received: from oabvr9.prod.google.com ([2002:a05:6871:a0c9:b0:295:f44d:8dfa]) (user=yuanchu job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6870:3b1b:b0:296:cdee:f7de with SMTP id 586e51a60fabf-29dc417fa5fmr1273477fac.21.1732676280422; Tue, 26 Nov 2024 18:58:00 -0800 (PST) Date: Tue, 26 Nov 2024 18:57:28 -0800 In-Reply-To: <20241127025728.3689245-1-yuanchu@google.com> Mime-Version: 1.0 References: <20241127025728.3689245-1-yuanchu@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241127025728.3689245-10-yuanchu@google.com> Subject: [PATCH v4 9/9] virtio-balloon: add workingset reporting From: Yuanchu Xie To: Andrew Morton , David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Tejun Heo , Johannes Weiner , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , " =?utf-8?q?Eugenio_P=C3=A9rez?= " , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Mike Rapoport , Shuah Khan , Christian Brauner , Daniel Watson , Yuanchu Xie , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, linux-mm@kvack.org, linux-kselftest@vger.kernel.org X-Rspamd-Queue-Id: DDDB91A000B X-Stat-Signature: waftjtkdjjpkzkhzmewqxzpmz884u57q X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1732676274-260603 X-HE-Meta: U2FsdGVkX19fQLQE2jSfR4BcK7o/0ceUEUJzclIsmQS94OJCzE6VNJjg9r/9EjWQpBfV4+WzsR9+vrspfHx8KSuIVn47OROZSG/67FthY8soa51rY6IdM6jpayqpF5NiWhugGi5W9HVnNaY4fp0ayuSGxsV1H0/49wRUE/3CtTfw58MYqwGRw1d4NhNarCkJoQhbbm3CFPJt+i/UXzI8ZyuiCZcWKEoyYaWgSf7ko6/Fo6u9cTeivPPlaSF/mUiFHdwMqlVZe+zdbSzqLVZxFhOhnrfiadnDPzicSMy/NmfWOWZBJG5QJookJ+VcZ4urmZqMEJ5hJpcRSbReesgUV5ewCmXNQF+t9Ko5kpiRd6x1QqhT1tCH7bH/7sATf2SYF8MgmRwE0sRvavzgfMAe5qUrksR07n3NbbDpFSOzPKu2N6r4RJhJvXRE9q9iaqBbabrnOXj9watyLO0O/x+ayp1lbj0HTKlzIfM1Ir2KcxPVTV6RfvrJSGn6jzMi4vVzx4xVe4vdIQ8BJJBG+SQoU8cJVmZKDItdMFEYUwnVRSthgukH8oCVwtVNZtV3DfDyysWTfDHNztqNQGWNOhYxZWOd4f3ywV/VNncnxObYku1Dob7Ky7gX5NKkRmroArGkXJWHPhxCUUZ2d9XiyZ/vjFd4DHo7EjIBxH23Zfz3PqQvNa3lSJcXlgrNUX+UPuwMwbS7FoQ/1di94NAPBXo6e6gzw1RV4aVt6B6zjDcLHnd6mW99Qln8eU1S2y7+Ut3J9aqgATA2HC/ohg7bf0jhS15R4PB974+yT0ssJOOjbjbsAD4h5OQkZEY32EO1ocpgYPoCJ6uFxwmU5W0ncv0M6oQSf8J/wuXdLfXKBaDZVBCFxgC5C/FK9uRGH8yebygp4rvN9IXSkgi+F+TaVaWFomBr2zlre8IjN0zjMARftqdvvjZhkMgitWwsYiANRZ2g67Zgi5MVUOUeoedhY8e ahceSx2p BitgnizOMtVp3cPz6oG7yRzeKSEeJqy5g3xBIbXBhgOGM2YoAjsOAx+gZlu1ttAc1dwq+3sY2OBKja/f1BO77/bg7LlWNosFfv/arbMJSG6O2Kz0BIsjTYp6z4uAS/J49T64KwIRS5lWoO1zOPb0Rr2AVMcktAxUUQ1bNz8fh8lQblt+khrHBXiBlG+7+FcOs9cN9AFA8n4VFQe+xliBqxJdkK+NZFlrkepQvqa67+vTlP0YTR3zZenTVFDAwG7Wdom8PPgx1oO2zVXqCLC0kd/HqYM4jrQ9cXOz1mTEz3k/6Ulo1IflcaG6MIMBM48ZeUaQpBfBKWCRHlsNlFQLgw7J8mImurMOARtD1sLmNAc5dI2BNT9TNY8BQPS0e6t8shTyUUoZM8YzHBDqN8T58scS9mrs8KU05fisWARcCw2B2zE2AjA1HKCiAQ2dnu8KY3YpgLJIydmOl44kS4DJESZzF2xInWYzZkrmdnWjBfqruwyXZ8FVVXyfU7ytiAf1oELCMdzX0aFtg+/kTKWjHf6af8BH05p5jmVrYVEFLAx+FnkGZH2mlLO3GXZE6TYLPtvKr0BySm4745R0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ballooning is a way to dynamically size a VM, and it requires guest collaboration. The amount to balloon without adversely affecting guest performance is hard to compute without clear metrics from the guest. Workingset reporting can provide guidance to the host to allow better collaborative ballooning, such that the host balloon controller can properly gauge the amount of memory the guest is actively using, i.e., the working set. A draft QEMU series [1] is being worked on. Currently it is able to configure the workingset reporting bins, refresh_interval, and report threshold. Through QMP or HMP, a balloon controller can request a workingset report. There is also a script [2] exercising the QMP interface with a visual breakdown of the guest's workingset size. According to the OASIS VIRTIO v1.3, there's a new balloon device in the works and this one I'm adding to is the "traditional" balloon. If the existing balloon device is not the right place for new features. I'm more than happy to add it to the new one as well. For technical details, this patch adds the a generic mechanism into workingset reporting infrastructure to allow other parts of the kernel to receive workingset reports. Two virtqueues are added to the virtio-balloon device, notification_vq and report_vq. The notification virtqueue allows the host to configure the guest workingset reporting parameters and request a report. The report virtqueue sends a working set report to the host when one is requested or due to memory pressure. The workingset reporting feature is gated by the compilation flag CONFIG_WORKINGSET_REPORT and the balloon feature flag VIRTIO_BALLOON_F_WS_REPORTING. [1] https://github.com/Dummyc0m/qemu/tree/wsr [2] https://gist.github.com/Dummyc0m/d45b4e1b0dda8f2bc6cd8cfb37cc7e34 Signed-off-by: Yuanchu Xie --- drivers/virtio/virtio_balloon.c | 390 +++++++++++++++++++++++++++- include/linux/balloon_compaction.h | 1 + include/linux/mmzone.h | 4 + include/linux/workingset_report.h | 66 ++++- include/uapi/linux/virtio_balloon.h | 30 +++ mm/workingset_report.c | 89 ++++++- 6 files changed, 566 insertions(+), 14 deletions(-) diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c index b36d2803674e..8eb300653dd8 100644 --- a/drivers/virtio/virtio_balloon.c +++ b/drivers/virtio/virtio_balloon.c @@ -18,6 +18,7 @@ #include #include #include +#include /* * Balloon device works in 4K page units. So each page is pointed to by @@ -45,6 +46,8 @@ enum virtio_balloon_vq { VIRTIO_BALLOON_VQ_STATS, VIRTIO_BALLOON_VQ_FREE_PAGE, VIRTIO_BALLOON_VQ_REPORTING, + VIRTIO_BALLOON_VQ_WORKING_SET, + VIRTIO_BALLOON_VQ_NOTIFY, VIRTIO_BALLOON_VQ_MAX }; @@ -124,6 +127,23 @@ struct virtio_balloon { spinlock_t wakeup_lock; bool processing_wakeup_event; u32 wakeup_signal_mask; + +#ifdef CONFIG_WORKINGSET_REPORT + struct virtqueue *working_set_vq, *notification_vq; + + /* Protects node_id, wsr_receiver, and report_buf */ + struct mutex wsr_report_lock; + int wsr_node_id; + struct wsr_receiver wsr_receiver; + /* Buffer to report to host */ + struct virtio_balloon_working_set_report *report_buf; + + /* Buffer to hold incoming notification from the host. */ + struct virtio_balloon_working_set_notify *notify_buf; + + struct work_struct update_balloon_working_set_work; + struct work_struct update_balloon_notification_work; +#endif }; #define VIRTIO_BALLOON_WAKEUP_SIGNAL_ADJUST (1 << 0) @@ -339,8 +359,352 @@ static unsigned int leak_balloon(struct virtio_balloon *vb, size_t num) return num_freed_pages; } -static inline void update_stat(struct virtio_balloon *vb, int idx, - u16 tag, u64 val) +#ifdef CONFIG_WORKINGSET_REPORT +static bool wsr_is_configured(struct virtio_balloon *vb) +{ + if (node_online(READ_ONCE(vb->wsr_node_id)) && + READ_ONCE(vb->wsr_receiver.wsr.refresh_interval) > 0 && + READ_ONCE(vb->wsr_receiver.wsr.page_age) != NULL) + return true; + return false; +} + +/* wsr_receiver callback */ +static void wsr_receiver_notify(struct wsr_receiver *receiver) +{ + int bin; + struct virtio_balloon *vb = + container_of(receiver, struct virtio_balloon, wsr_receiver); + + /* if we fail to acquire the locks, send stale report */ + if (!mutex_trylock(&vb->wsr_report_lock)) + goto out; + if (!mutex_trylock(&receiver->wsr.page_age_lock)) + goto out_unlock_report_buf; + if (!READ_ONCE(receiver->wsr.page_age)) + goto out_unlock_page_age; + + vb->report_buf->error = cpu_to_le32(0); + vb->report_buf->node_id = cpu_to_le32(vb->wsr_node_id); + for (bin = 0; bin < WORKINGSET_REPORT_MAX_NR_BINS; ++bin) { + struct virtio_balloon_working_set_report_bin *dest = + &vb->report_buf->bins[bin]; + struct wsr_report_bin *src = &receiver->wsr.page_age->bins[bin]; + + dest->anon_bytes = + cpu_to_le64(src->nr_pages[LRU_GEN_ANON] * PAGE_SIZE); + dest->file_bytes = + cpu_to_le64(src->nr_pages[LRU_GEN_FILE] * PAGE_SIZE); + if (src->idle_age == WORKINGSET_INTERVAL_MAX) { + dest->idle_age = cpu_to_le64(WORKINGSET_INTERVAL_MAX); + break; + } + dest->idle_age = cpu_to_le64(jiffies_to_msecs(src->idle_age)); + } + +out_unlock_page_age: + mutex_unlock(&receiver->wsr.page_age_lock); +out_unlock_report_buf: + mutex_unlock(&vb->wsr_report_lock); +out: + /* Send the working set report to the device. */ + spin_lock(&vb->stop_update_lock); + if (!vb->stop_update) + queue_work(system_freezable_wq, &vb->update_balloon_working_set_work); + spin_unlock(&vb->stop_update_lock); +} + +static void virtio_balloon_working_set_request(struct virtio_balloon *vb, + int nid) +{ + int err = 0; + + if (!node_online(nid)) { + err = -EINVAL; + goto error; + } + + err = wsr_refresh_receiver_report(NODE_DATA(nid)); + if (err) + goto error; + + return; +error: + mutex_lock(&vb->wsr_report_lock); + vb->report_buf->error = cpu_to_le16(err); + vb->report_buf->node_id = cpu_to_le32(nid); + mutex_unlock(&vb->wsr_report_lock); + spin_lock(&vb->stop_update_lock); + if (!vb->stop_update) + queue_work(system_freezable_wq, + &vb->update_balloon_working_set_work); + spin_unlock(&vb->stop_update_lock); +} + +static void notification_receive(struct virtqueue *vq) +{ + struct virtio_balloon *vb = vq->vdev->priv; + + spin_lock(&vb->stop_update_lock); + if (!vb->stop_update) + queue_work(system_freezable_wq, &vb->update_balloon_notification_work); + spin_unlock(&vb->stop_update_lock); +} + +static int +virtio_balloon_register_working_set_receiver(struct virtio_balloon *vb) +{ + struct pglist_data *pgdat; + struct wsr_report_bins *bins = NULL, __rcu *old; + int nid, bin, err = 0, old_nid = vb->wsr_node_id; + struct virtio_balloon_working_set_notify *notify = vb->notify_buf; + + nid = le16_to_cpu(notify->node_id); + if (!node_online(nid)) { + dev_warn(&vb->vdev->dev, "node not online %d\n", nid); + return -EINVAL; + } + + pgdat = NODE_DATA(nid); + bins = kzalloc(sizeof(struct wsr_report_bins), GFP_KERNEL); + + if (!bins) + return -ENOMEM; + + for (bin = 0; bin < WORKINGSET_REPORT_MAX_NR_BINS; ++bin) { + u32 age_msecs = le32_to_cpu(notify->idle_age[bin]); + unsigned long age = msecs_to_jiffies(age_msecs); + + /* + * A correct idle_age array should end in + * WORKINGSET_INTERVAL_MAX. + */ + if (age_msecs == (u32)WORKINGSET_INTERVAL_MAX) { + bins->idle_age[bin] = WORKINGSET_INTERVAL_MAX; + break; + } + bins->idle_age[bin] = age; + if (bin > 0 && bins->idle_age[bin] <= bins->idle_age[bin - 1]) { + dev_warn(&vb->vdev->dev, "bins not increasing\n"); + err = -EINVAL; + goto error; + } + } + if (bin < WORKINGSET_REPORT_MIN_NR_BINS - 1 || + bin == WORKINGSET_REPORT_MAX_NR_BINS) { + err = -ERANGE; + goto error; + } + bins->nr_bins = bin; + + mutex_lock(&vb->wsr_report_lock); + err = wsr_set_refresh_interval( + &vb->wsr_receiver.wsr, + le32_to_cpu(notify->refresh_interval)); + if (err) { + mutex_unlock(&vb->wsr_report_lock); + goto error; + } + if (old_nid != NUMA_NO_NODE) + wsr_remove_receiver(&vb->wsr_receiver, NODE_DATA(old_nid)); + WRITE_ONCE(vb->wsr_node_id, nid); + WRITE_ONCE(vb->wsr_receiver.wsr.report_threshold, + msecs_to_jiffies(le32_to_cpu(notify->report_threshold))); + WRITE_ONCE(vb->wsr_receiver.notify, wsr_receiver_notify); + mutex_unlock(&vb->wsr_report_lock); + + /* update the bins for target node */ + mutex_lock(&pgdat->wsr_update_mutex); + old = rcu_replace_pointer(pgdat->wsr_page_age_bins, bins, + lockdep_is_held(&pgdat->wsr_update_mutex)); + mutex_unlock(&pgdat->wsr_update_mutex); + kfree_rcu(old, rcu); + + wsr_register_receiver(&vb->wsr_receiver, pgdat); + + return 0; +error: + kfree(bins); + return err; +} + +static void update_balloon_notification_func(struct work_struct *work) +{ + unsigned int len, op; + int err; + struct virtio_balloon *vb; + struct scatterlist sg_in; + + vb = container_of(work, struct virtio_balloon, + update_balloon_notification_work); + op = le16_to_cpu(vb->notify_buf->op); + + switch (op) { + case VIRTIO_BALLOON_WS_OP_REQUEST: + virtio_balloon_working_set_request(vb, + READ_ONCE(vb->wsr_node_id)); + break; + case VIRTIO_BALLOON_WS_OP_CONFIG: + err = virtio_balloon_register_working_set_receiver(vb); + if (err) + dev_warn(&vb->vdev->dev, + "Error configuring working set, %d\n", err); + break; + default: + dev_warn(&vb->vdev->dev, "Received invalid notification, %u\n", + op); + break; + } + + /* Detach all the used buffers from the vq */ + while (virtqueue_get_buf(vb->notification_vq, &len)) + ; + /* Add a new notification buffer for device to fill. */ + sg_init_one(&sg_in, vb->notify_buf, sizeof(*vb->notify_buf)); + virtqueue_add_inbuf(vb->notification_vq, &sg_in, 1, vb, GFP_KERNEL); + virtqueue_kick(vb->notification_vq); +} + +static void update_balloon_ws_func(struct work_struct *work) +{ + struct virtio_balloon *vb; + + vb = container_of(work, struct virtio_balloon, + update_balloon_working_set_work); + + if (wsr_is_configured(vb)) { + struct scatterlist sg_out; + int unused; + int err; + + /* Detach all the used buffers from the vq */ + while (virtqueue_get_buf(vb->working_set_vq, &unused)) + ; + sg_init_one(&sg_out, vb->report_buf, sizeof(*vb->report_buf)); + err = virtqueue_add_outbuf(vb->working_set_vq, &sg_out, 1, vb, GFP_KERNEL); + if (unlikely(err)) + dev_err(&vb->vdev->dev, + "Failed to send working set report err = %d\n", + err); + else + virtqueue_kick(vb->working_set_vq); + + } else { + dev_warn(&vb->vdev->dev, "Working Set not initialized."); + } +} + +static void wsr_init_vqs_info(struct virtio_balloon *vb, + struct virtqueue_info vqs_info[]) +{ + if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_WS_REPORTING)) { + vqs_info[VIRTIO_BALLOON_VQ_WORKING_SET].name = "ws"; + vqs_info[VIRTIO_BALLOON_VQ_WORKING_SET].callback = NULL; + vqs_info[VIRTIO_BALLOON_VQ_NOTIFY].name = "notify"; + vqs_info[VIRTIO_BALLOON_VQ_NOTIFY].callback = notification_receive; + } +} + +static int wsr_init_vq(struct virtio_balloon *vb, struct virtqueue *vqs[]) +{ + if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_WS_REPORTING)) { + struct scatterlist sg; + int err; + + vb->working_set_vq = vqs[VIRTIO_BALLOON_VQ_WORKING_SET]; + vb->notification_vq = vqs[VIRTIO_BALLOON_VQ_NOTIFY]; + + /* Prime the notification virtqueue for the device to fill. */ + sg_init_one(&sg, vb->notify_buf, sizeof(*vb->notify_buf)); + err = virtqueue_add_inbuf(vb->notification_vq, &sg, 1, vb, GFP_KERNEL); + if (unlikely(err)) { + dev_err(&vb->vdev->dev, + "Failed to prepare notifications, err = %d\n", err); + return err; + } + virtqueue_kick(vb->notification_vq); + } + return 0; +} + +static void wsr_init_work(struct virtio_balloon *vb) +{ + if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_WS_REPORTING)) { + INIT_WORK(&vb->update_balloon_working_set_work, + update_balloon_ws_func); + INIT_WORK(&vb->update_balloon_notification_work, + update_balloon_notification_func); + } +} + +static int wsr_init(struct virtio_balloon *vb) +{ + if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_WS_REPORTING)) { + vb->report_buf = kzalloc(sizeof(*vb->report_buf), GFP_KERNEL); + if (!vb->report_buf) + return -ENOMEM; + + vb->notify_buf = kzalloc(sizeof(*vb->notify_buf), GFP_KERNEL); + if (!vb->notify_buf) { + kfree(vb->report_buf); + vb->report_buf = NULL; + return -ENOMEM; + } + + wsr_init_state(&vb->wsr_receiver.wsr); + vb->wsr_node_id = NUMA_NO_NODE; + vb->report_buf->bins[0].idle_age = WORKINGSET_INTERVAL_MAX; + } + return 0; +} + +static void wsr_remove(struct virtio_balloon *vb) +{ + if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_WS_REPORTING) && + vb->wsr_node_id != NUMA_NO_NODE) { + wsr_remove_receiver(&vb->wsr_receiver, NODE_DATA(vb->wsr_node_id)); + wsr_destroy_state(&vb->wsr_receiver.wsr); + } + + kfree(vb->report_buf); + kfree(vb->notify_buf); + mutex_destroy(&vb->wsr_report_lock); +} + +static void wsr_cancel_work(struct virtio_balloon *vb) +{ + if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_WS_REPORTING)) { + cancel_work_sync(&vb->update_balloon_working_set_work); + cancel_work_sync(&vb->update_balloon_notification_work); + } +} +#else +static inline void wsr_init_vqs_info(struct virtio_balloon *vb, + struct virtqueue_info vqs_info[]) +{ +} +static inline int wsr_init_vq(struct virtio_balloon *vb, + struct virtqueue *vqs[]) +{ + return 0; +} +static inline void wsr_init_work(struct virtio_balloon *vb) +{ +} +static inline int wsr_init(struct virtio_balloon *vb) +{ + return 0; +} +static inline void wsr_remove(struct virtio_balloon *vb) +{ +} +static inline void wsr_cancel_work(struct virtio_balloon *vb) +{ +} +#endif + +static inline void update_stat(struct virtio_balloon *vb, int idx, u16 tag, + u64 val) { BUG_ON(idx >= VIRTIO_BALLOON_S_NR); vb->stats[idx].tag = cpu_to_virtio16(vb->vdev, tag); @@ -605,6 +969,8 @@ static int init_vqs(struct virtio_balloon *vb) vqs_info[VIRTIO_BALLOON_VQ_REPORTING].callback = balloon_ack; } + wsr_init_vqs_info(vb, vqs_info); + err = virtio_find_vqs(vb->vdev, VIRTIO_BALLOON_VQ_MAX, vqs, vqs_info, NULL); if (err) @@ -615,6 +981,7 @@ static int init_vqs(struct virtio_balloon *vb) if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ)) { struct scatterlist sg; unsigned int num_stats; + vb->stats_vq = vqs[VIRTIO_BALLOON_VQ_STATS]; /* @@ -640,6 +1007,11 @@ static int init_vqs(struct virtio_balloon *vb) if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_REPORTING)) vb->reporting_vq = vqs[VIRTIO_BALLOON_VQ_REPORTING]; + err = wsr_init_vq(vb, vqs); + + if (err) + return err; + return 0; } @@ -961,15 +1333,21 @@ static int virtballoon_probe(struct virtio_device *vdev) goto out; } + vb->vdev = vdev; + INIT_WORK(&vb->update_balloon_stats_work, update_balloon_stats_func); INIT_WORK(&vb->update_balloon_size_work, update_balloon_size_func); + wsr_init_work(vb); spin_lock_init(&vb->stop_update_lock); mutex_init(&vb->balloon_lock); init_waitqueue_head(&vb->acked); - vb->vdev = vdev; balloon_devinfo_init(&vb->vb_dev_info); + err = wsr_init(vb); + if (err) + goto out_remove_wsr; + err = init_vqs(vb); if (err) goto out_free_vb; @@ -1085,7 +1463,6 @@ static int virtballoon_probe(struct virtio_device *vdev) if (towards_target(vb)) virtballoon_changed(vdev); return 0; - out_unregister_oom: if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) unregister_oom_notifier(&vb->oom_nb); @@ -1099,6 +1476,8 @@ static int virtballoon_probe(struct virtio_device *vdev) vdev->config->del_vqs(vdev); out_free_vb: kfree(vb); +out_remove_wsr: + wsr_remove(vb); out: return err; } @@ -1130,11 +1509,13 @@ static void virtballoon_remove(struct virtio_device *vdev) unregister_oom_notifier(&vb->oom_nb); if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) virtio_balloon_unregister_shrinker(vb); + wsr_remove(vb); spin_lock_irq(&vb->stop_update_lock); vb->stop_update = true; spin_unlock_irq(&vb->stop_update_lock); cancel_work_sync(&vb->update_balloon_size_work); cancel_work_sync(&vb->update_balloon_stats_work); + wsr_cancel_work(vb); if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) { cancel_work_sync(&vb->report_free_page_work); @@ -1200,6 +1581,7 @@ static unsigned int features[] = { VIRTIO_BALLOON_F_FREE_PAGE_HINT, VIRTIO_BALLOON_F_PAGE_POISON, VIRTIO_BALLOON_F_REPORTING, + VIRTIO_BALLOON_F_WS_REPORTING, }; static struct virtio_driver virtio_balloon_driver = { diff --git a/include/linux/balloon_compaction.h b/include/linux/balloon_compaction.h index 5ca2d5699620..d92b8337dbcf 100644 --- a/include/linux/balloon_compaction.h +++ b/include/linux/balloon_compaction.h @@ -43,6 +43,7 @@ #include #include #include +#include /* * Balloon device information descriptor. diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index ee728c0c5a3b..9a2dc506779d 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1429,8 +1429,12 @@ typedef struct pglist_data { #endif #ifdef CONFIG_WORKINGSET_REPORT + /* protects wsr_page_age_bins */ struct mutex wsr_update_mutex; struct wsr_report_bins __rcu *wsr_page_age_bins; + /* protects wsr_receiver_lost */ + struct mutex wsr_receiver_mutex; + struct list_head wsr_receiver_list; #endif CACHELINE_PADDING(_pad2_); diff --git a/include/linux/workingset_report.h b/include/linux/workingset_report.h index f6bbde2a04c3..1074b89035e9 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -11,13 +11,14 @@ struct node; struct lruvec; struct cgroup_file; struct wsr_state; - -#ifdef CONFIG_WORKINGSET_REPORT +struct wsr_receiver; #define WORKINGSET_REPORT_MIN_NR_BINS 2 #define WORKINGSET_REPORT_MAX_NR_BINS 32 #define WORKINGSET_INTERVAL_MAX ((unsigned long)-1) + +#ifdef CONFIG_WORKINGSET_REPORT #define ANON_AND_FILE 2 struct wsr_report_bin { @@ -52,6 +53,8 @@ struct wsr_state { struct wsr_page_age_histo *page_age; }; +void wsr_init_state(struct wsr_state *wsr); +void wsr_destroy_state(struct wsr_state *wsr); void wsr_init_lruvec(struct lruvec *lruvec); void wsr_destroy_lruvec(struct lruvec *lruvec); void wsr_init_pgdat(struct pglist_data *pgdat); @@ -66,6 +69,47 @@ void wsr_remove_sysfs(struct node *node); bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, struct pglist_data *pgdat, unsigned long *refresh_time); +/* + * If refresh_interval > 0, enable working set reporting and kick + * the aging thread (if configured). + * If refresh_interval = 0, disable working set reporting and free + * the bookkeeping resources. + * + * @param refresh_interval milliseconds. + */ +int wsr_set_refresh_interval(struct wsr_state *wsr, + unsigned long refresh_interval); + +struct wsr_receiver { + /* + * Working set reporting ensures that two notify calls to + * the same receivercannot interleave one another. + * + * Must be set before calling wsr_register_receiver. + */ + void (*notify)(struct wsr_receiver *receiver); + struct wsr_state wsr; + struct list_head list; +}; + +/* + * Register a per-node receiver + * report_threshold and refresh_interval are configured + * by the caller in struct wsr_state and contain valid values. + * page_age is allocated. + */ +void wsr_register_receiver(struct wsr_receiver *receiver, + struct pglist_data *pgdat); + +void wsr_remove_receiver(struct wsr_receiver *receiver, + struct pglist_data *pgdat); + +/* + * Refresh the report for the specified node, unless a refresh is already + * in progress or the parameters are being updated. + */ +int wsr_refresh_receiver_report(struct pglist_data *pgdat); + #ifdef CONFIG_WORKINGSET_REPORT_AGING void wsr_wakeup_aging_thread(void); #else /* CONFIG_WORKINGSET_REPORT_AGING */ @@ -77,6 +121,12 @@ static inline void wsr_wakeup_aging_thread(void) int wsr_set_refresh_interval(struct wsr_state *wsr, unsigned long refresh_interval); #else +static inline void wsr_init_state(struct wsr_state *wsr) +{ +} +static inline void wsr_destroy_state(struct wsr_state *wsr) +{ +} static inline void wsr_init_lruvec(struct lruvec *lruvec) { } @@ -100,6 +150,18 @@ static inline int wsr_set_refresh_interval(struct wsr_state *wsr, { return 0; } +static inline int wsr_register_receiver(struct wsr_receiver *receiver, + struct pglist_data *pgdat) +{ + return -ENODEV; +} +static inline void wsr_remove_receiver(struct wsr_receiver *receiver, + struct pglist_data *pgdat) +{ +} +static inline void wsr_refresh_receiver_report(struct pglist_data *pgdat) +{ +} #endif /* CONFIG_WORKINGSET_REPORT */ #endif /* _LINUX_WORKINGSET_REPORT_H */ diff --git a/include/uapi/linux/virtio_balloon.h b/include/uapi/linux/virtio_balloon.h index ee35a372805d..668eaa39c85b 100644 --- a/include/uapi/linux/virtio_balloon.h +++ b/include/uapi/linux/virtio_balloon.h @@ -25,6 +25,7 @@ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ +#include "linux/workingset_report.h" #include #include #include @@ -37,6 +38,7 @@ #define VIRTIO_BALLOON_F_FREE_PAGE_HINT 3 /* VQ to report free pages */ #define VIRTIO_BALLOON_F_PAGE_POISON 4 /* Guest is using page poisoning */ #define VIRTIO_BALLOON_F_REPORTING 5 /* Page reporting virtqueue */ +#define VIRTIO_BALLOON_F_WS_REPORTING 6 /* Working Set Size reporting */ /* Size of a PFN in the balloon interface. */ #define VIRTIO_BALLOON_PFN_SHIFT 12 @@ -128,4 +130,32 @@ struct virtio_balloon_stat { __virtio64 val; } __attribute__((packed)); +/* Operations from the device */ +#define VIRTIO_BALLOON_WS_OP_REQUEST 1 +#define VIRTIO_BALLOON_WS_OP_CONFIG 2 + +struct virtio_balloon_working_set_notify { + /* REQUEST or CONFIG */ + __le16 op; + __le16 node_id; + /* the following fields valid iff op=CONFIG */ + __le32 report_threshold; + __le32 refresh_interval; + __le32 idle_age[WORKINGSET_REPORT_MAX_NR_BINS]; +}; + +struct virtio_balloon_working_set_report_bin { + __le64 idle_age; + /* bytes in this bucket for anon and file */ + __le64 anon_bytes; + __le64 file_bytes; +}; + +struct virtio_balloon_working_set_report { + __le32 error; + __le32 node_id; + struct virtio_balloon_working_set_report_bin + bins[WORKINGSET_REPORT_MAX_NR_BINS]; +}; + #endif /* _LINUX_VIRTIO_BALLOON_H */ diff --git a/mm/workingset_report.c b/mm/workingset_report.c index dad539e602bb..4b3397ebdbd0 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -20,27 +20,51 @@ void wsr_init_pgdat(struct pglist_data *pgdat) { mutex_init(&pgdat->wsr_update_mutex); RCU_INIT_POINTER(pgdat->wsr_page_age_bins, NULL); + INIT_LIST_HEAD(&pgdat->wsr_receiver_list); } void wsr_destroy_pgdat(struct pglist_data *pgdat) { struct wsr_report_bins __rcu *bins; + struct list_head *cursor, *next; mutex_lock(&pgdat->wsr_update_mutex); bins = rcu_replace_pointer(pgdat->wsr_page_age_bins, NULL, lockdep_is_held(&pgdat->wsr_update_mutex)); - kfree_rcu(bins, rcu); mutex_unlock(&pgdat->wsr_update_mutex); + kfree_rcu(bins, rcu); + mutex_lock(&pgdat->wsr_receiver_mutex); + list_for_each_safe(cursor, next, &pgdat->wsr_receiver_list) { + /* pgdat does not own the receiver, so it's not free'd here */ + list_del(cursor); + } + mutex_unlock(&pgdat->wsr_receiver_mutex); + mutex_destroy(&pgdat->wsr_update_mutex); + mutex_destroy(&pgdat->wsr_receiver_mutex); +} + +void wsr_init_state(struct wsr_state *wsr) +{ + memset(wsr, 0, sizeof(*wsr)); + mutex_init(&wsr->page_age_lock); +} +EXPORT_SYMBOL_GPL(wsr_init_state); + +void wsr_destroy_state(struct wsr_state *wsr) +{ + kfree(wsr->page_age); + mutex_destroy(&wsr->page_age_lock); + memset(wsr, 0, sizeof(*wsr)); } +EXPORT_SYMBOL_GPL(wsr_destroy_state); void wsr_init_lruvec(struct lruvec *lruvec) { struct wsr_state *wsr = &lruvec->wsr; struct mem_cgroup *memcg = lruvec_memcg(lruvec); - memset(wsr, 0, sizeof(*wsr)); - mutex_init(&wsr->page_age_lock); + wsr_init_state(wsr); if (memcg && !mem_cgroup_is_root(memcg)) wsr->page_age_cgroup_file = mem_cgroup_page_age_file(memcg); } @@ -49,9 +73,7 @@ void wsr_destroy_lruvec(struct lruvec *lruvec) { struct wsr_state *wsr = &lruvec->wsr; - mutex_destroy(&wsr->page_age_lock); - kfree(wsr->page_age); - memset(wsr, 0, sizeof(*wsr)); + wsr_destroy_state(wsr); } int workingset_report_intervals_parse(char *src, @@ -395,6 +417,7 @@ int wsr_set_refresh_interval(struct wsr_state *wsr, wsr_wakeup_aging_thread(); return err; } +EXPORT_SYMBOL_GPL(wsr_set_refresh_interval); static ssize_t refresh_interval_store(struct kobject *kobj, struct kobj_attribute *attr, @@ -569,12 +592,62 @@ void wsr_remove_sysfs(struct node *node) } EXPORT_SYMBOL_GPL(wsr_remove_sysfs); +/* wsr belongs to the root memcg or memcg is disabled */ +static int notify_receiver(struct wsr_state *wsr, struct pglist_data *pgdat) +{ + struct list_head *cursor; + + if (!mutex_trylock(&pgdat->wsr_receiver_mutex)) + return -EAGAIN; + list_for_each(cursor, &pgdat->wsr_receiver_list) { + struct wsr_receiver *entry = + list_entry(cursor, struct wsr_receiver, list); + + wsr_refresh_report(&entry->wsr, NULL, pgdat, NULL); + entry->notify(entry); + } + mutex_unlock(&pgdat->wsr_receiver_mutex); + return 0; +} + +int wsr_refresh_receiver_report(struct pglist_data *pgdat) +{ + struct wsr_state *wsr = &mem_cgroup_lruvec(NULL, pgdat)->wsr; + + return notify_receiver(wsr, pgdat); +} +EXPORT_SYMBOL_GPL(wsr_refresh_receiver_report); + void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat) { struct wsr_state *wsr = &mem_cgroup_lruvec(memcg, pgdat)->wsr; - if (mem_cgroup_is_root(memcg)) + if (mem_cgroup_is_root(memcg)) { kernfs_notify(wsr->page_age_sys_file); - else + notify_receiver(wsr, pgdat); + } else cgroup_file_notify(wsr->page_age_cgroup_file); } + +void wsr_register_receiver(struct wsr_receiver *receiver, + struct pglist_data *pgdat) +{ + struct wsr_state *wsr = &receiver->wsr; + + mutex_lock(&pgdat->wsr_receiver_mutex); + list_add_tail(&receiver->list, &pgdat->wsr_receiver_list); + mutex_unlock(&pgdat->wsr_receiver_mutex); + + if (!!wsr->page_age && READ_ONCE(wsr->refresh_interval)) + wsr_wakeup_aging_thread(); +} +EXPORT_SYMBOL(wsr_register_receiver); + +void wsr_remove_receiver(struct wsr_receiver *receiver, + struct pglist_data *pgdat) +{ + mutex_lock(&pgdat->wsr_receiver_mutex); + list_del(&receiver->list); + mutex_unlock(&pgdat->wsr_receiver_mutex); +} +EXPORT_SYMBOL(wsr_remove_receiver);