From patchwork Mon Feb 17 10:25:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 11386025 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 490721395 for ; Mon, 17 Feb 2020 10:26:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ED5962067D for ; Mon, 17 Feb 2020 10:26:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="VFvV+UP6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ED5962067D Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 21A166B0006; Mon, 17 Feb 2020 05:26:31 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1F1466B0007; Mon, 17 Feb 2020 05:26:31 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E0456B0008; Mon, 17 Feb 2020 05:26:31 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0071.hostedemail.com [216.40.44.71]) by kanga.kvack.org (Postfix) with ESMTP id E7AD36B0006 for ; Mon, 17 Feb 2020 05:26:30 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 8D591181AEF07 for ; Mon, 17 Feb 2020 10:26:30 +0000 (UTC) X-FDA: 76499239740.02.suit27_1babd5f5fb335 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=309bfddc3=sjpark@amazon.com,:akpm@linux-foundation.org:sjpark@amazon.de:acme@kernel.org:alexander.shishkin@linux.intel.com:amit@kernel.org:brendan.d.gregg@gmail.com:brendanhiggins@google.com:cai@lca.pw:colin.king@canonical.com:corbet@lwn.net:dwmw@amazon.com:jolsa@redhat.com:kirill@shutemov.name:mark.rutland@arm.com:mgorman@suse.de:minchan@kernel.org:mingo@redhat.com:namhyung@kernel.org:peterz@infradead.org:rdunlap@infradead.org:rostedt@goodmis.org:shuah@kernel.org:sj38.park@gmail.com:vdavydov.dev@gmail.com::linux-doc@vger.kernel.org:linux-kernel@vger.kernel.org,RULES_HIT:2539:30003:30034:30041:30051:30054:30064:30067:30075,0,RBL:52.95.49.90:@amazon.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: suit27_1babd5f5fb335 X-Filterd-Recvd-Size: 12941 Received: from smtp-fw-6002.amazon.com (smtp-fw-6002.amazon.com [52.95.49.90]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Mon, 17 Feb 2020 10:26:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1581935190; x=1613471190; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=cQ0LHqoaBUPla1G15rrmXAWWwxHmqoTLU/9P2Rx46Xk=; b=VFvV+UP661dknFdFO0CXtLTAzYtkWLj20uWnKXaTHlPh8zI2BYJcO2UP eVOzF+nCaI8JfuFlrcwB7nFW8X0etmo5X/JaIi2/u2Qt2GyFe/7K1DE23 Vmp4ppOFXuWb3Afl5/DPr23N+vPnxWWQwnivL6ok4SUaGXpOusSQks4kh Y=; IronPort-SDR: uLurQsy5a+OKWydMW/0ewzmkZMq4Pd5bu/PN547oUSrRHExUXgzLUmNIlUbtWuSFJo8h3sShjb nBrxBEr8fAJg== X-IronPort-AV: E=Sophos;i="5.70,452,1574121600"; d="scan'208";a="16610486" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2c-4e7c8266.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 17 Feb 2020 10:26:27 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2c-4e7c8266.us-west-2.amazon.com (Postfix) with ESMTPS id 5EE9BA2E1C; Mon, 17 Feb 2020 10:26:25 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 17 Feb 2020 10:26:24 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.161.214) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 17 Feb 2020 10:26:14 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH v5 01/14] mm: Introduce Data Access MONitor (DAMON) Date: Mon, 17 Feb 2020 11:25:31 +0100 Message-ID: <20200217102544.29012-2-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200217102544.29012-1-sjpark@amazon.com> References: <20200217102544.29012-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.214] X-ClientProxiedBy: EX13D30UWB004.ant.amazon.com (10.43.161.51) To EX13D31EUA001.ant.amazon.com (10.43.165.15) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park This commit introduces a kernel module named DAMON. Note that this commit is implementing only the stub for the module load/unload, basic data structures, and simple manipulation functions of the structures to keep the size of commit small. The core mechanisms of DAMON will be implemented one by one by following commits. Brief Introduction ================== Memory management decisions can be improved if finer data access information is available. However, because such finer information usually comes with higher overhead, most systems including Linux forgives the potential improvement and rely on only coarse information or some light-weight heuristics. The pseudo-LRU and the aggressive THP promotions are such examples. A number of experimental data access pattern awared memory management optimizations say the sacrifices are huge. However, none of those has successfully adopted to Linux kernel mainly due to the absence of a scalable and efficient data access monitoring mechanism. DAMON is a data access monitoring solution for the problem. It is 1) accurate enough for the DRAM level memory management, 2) light-weight enough to be applied online, and 3) keeps predefined upper-bound overhead regardless of the size of target workloads (thus scalable). DAMON is implemented as a standalone kernel module and provides several simple interfaces. Owing to that, though it has mainly designed for the kernel's memory management mechanisms, it can be also used for a wide range of user space programs and people. Frequently Asked Questions ========================== Q: Why not integrated with perf? A: From the perspective of perf like profilers, DAMON can be thought of as a data source in kernel, like tracepoints, pressure stall information (psi), or idle page tracking. Thus, it can be easily integrated with those. However, this patchset doesn't provide a fancy perf integration because current step of DAMON development is focused on its core logic only. That said, DAMON already provides two interfaces for user space programs, which based on debugfs and tracepoint, respectively. Using the tracepoint interface, you can use DAMON with perf. This patchset also provides the debugfs interface based user space tool for DAMON. It can be used to record, visualize, and analyze data access pattern of target processes in a convenient way. Q: Why a new module, instead of extending perf or other tools? A: First, DAMON aims to be used by other programs including the kernel. Therefore, having dependency to specific tools like perf is not desirable. Second, because it need to be lightweight as much as possible so that it can be used online, any unnecessary overhead such as kernel - user space context switching cost should be avoided. These are the two most biggest reasons why DAMON is implemented in the kernel space. The idle page tracking subsystem would be the kernel module that most seems similar to DAMON. However, it's own interface is not compatible with DAMON. Also, the internal implementation of it has no common part to be reused by DAMON. Q: Can 'perf mem' provide the data required for DAMON? A: On the systems supporting 'perf mem', yes. DAMON is using the PTE Accessed bits in low level. Other H/W or S/W features that can be used for the purpose could be used. However, as explained with above question, DAMON need to be implemented in the kernel space. Signed-off-by: SeongJae Park --- mm/Kconfig | 12 +++ mm/Makefile | 1 + mm/damon.c | 224 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 237 insertions(+) create mode 100644 mm/damon.c diff --git a/mm/Kconfig b/mm/Kconfig index ab80933be65f..387d469f40ec 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -739,4 +739,16 @@ config ARCH_HAS_HUGEPD config MAPPING_DIRTY_HELPERS bool +config DAMON + tristate "Data Access Monitor" + depends on MMU + default n + help + Provides data access monitoring. + + DAMON is a kernel module that allows users to monitor the actual + memory access pattern of specific user-space processes. It aims to + be 1) accurate enough to be useful for performance-centric domains, + and 2) sufficiently light-weight so that it can be applied online. + endmenu diff --git a/mm/Makefile b/mm/Makefile index 1937cc251883..2911b3832c90 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -108,3 +108,4 @@ obj-$(CONFIG_ZONE_DEVICE) += memremap.o obj-$(CONFIG_HMM_MIRROR) += hmm.o obj-$(CONFIG_MEMFD_CREATE) += memfd.o obj-$(CONFIG_MAPPING_DIRTY_HELPERS) += mapping_dirty_helpers.o +obj-$(CONFIG_DAMON) += damon.o diff --git a/mm/damon.c b/mm/damon.c new file mode 100644 index 000000000000..aafdca35b7b8 --- /dev/null +++ b/mm/damon.c @@ -0,0 +1,224 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Data Access Monitor + * + * Copyright 2019 Amazon.com, Inc. or its affiliates. All rights reserved. + * + * Author: SeongJae Park + */ + +#define pr_fmt(fmt) "damon: " fmt + +#include +#include +#include +#include + +#define damon_get_task_struct(t) \ + (get_pid_task(find_vpid(t->pid), PIDTYPE_PID)) + +#define damon_next_region(r) \ + (container_of(r->list.next, struct damon_region, list)) + +#define damon_prev_region(r) \ + (container_of(r->list.prev, struct damon_region, list)) + +#define damon_for_each_region(r, t) \ + list_for_each_entry(r, &t->regions_list, list) + +#define damon_for_each_region_safe(r, next, t) \ + list_for_each_entry_safe(r, next, &t->regions_list, list) + +#define damon_for_each_task(ctx, t) \ + list_for_each_entry(t, &(ctx)->tasks_list, list) + +#define damon_for_each_task_safe(ctx, t, next) \ + list_for_each_entry_safe(t, next, &(ctx)->tasks_list, list) + +/* Represents a monitoring target region on the virtual address space */ +struct damon_region { + unsigned long vm_start; + unsigned long vm_end; + unsigned long sampling_addr; + unsigned int nr_accesses; + struct list_head list; +}; + +/* Represents a monitoring target task */ +struct damon_task { + unsigned long pid; + struct list_head regions_list; + struct list_head list; +}; + +struct damon_ctx { + struct rnd_state rndseed; + + struct list_head tasks_list; /* 'damon_task' objects */ +}; + +/* Get a random number in [l, r) */ +#define damon_rand(ctx, l, r) (l + prandom_u32_state(&ctx->rndseed) % (r - l)) + +/* + * Construct a damon_region struct + * + * Returns the pointer to the new struct if success, or NULL otherwise + */ +static struct damon_region *damon_new_region(struct damon_ctx *ctx, + unsigned long vm_start, unsigned long vm_end) +{ + struct damon_region *ret; + + ret = kmalloc(sizeof(struct damon_region), GFP_KERNEL); + if (!ret) + return NULL; + ret->vm_start = vm_start; + ret->vm_end = vm_end; + ret->nr_accesses = 0; + ret->sampling_addr = damon_rand(ctx, vm_start, vm_end); + INIT_LIST_HEAD(&ret->list); + + return ret; +} + +/* + * Add a region between two other regions + */ +static inline void damon_add_region(struct damon_region *r, + struct damon_region *prev, struct damon_region *next) +{ + __list_add(&r->list, &prev->list, &next->list); +} + +/* + * Append a region to a task's list of regions + */ +static void damon_add_region_tail(struct damon_region *r, struct damon_task *t) +{ + list_add_tail(&r->list, &t->regions_list); +} + +/* + * Delete a region from its list + */ +static void damon_del_region(struct damon_region *r) +{ + list_del(&r->list); +} + +/* + * De-allocate a region + */ +static void damon_free_region(struct damon_region *r) +{ + kfree(r); +} + +static void damon_destroy_region(struct damon_region *r) +{ + damon_del_region(r); + damon_free_region(r); +} + +/* + * Construct a damon_task struct + * + * Returns the pointer to the new struct if success, or NULL otherwise + */ +static struct damon_task *damon_new_task(unsigned long pid) +{ + struct damon_task *t; + + t = kmalloc(sizeof(struct damon_task), GFP_KERNEL); + if (!t) + return NULL; + t->pid = pid; + INIT_LIST_HEAD(&t->regions_list); + + return t; +} + +/* Returns n-th damon_region of the given task */ +struct damon_region *damon_nth_region_of(struct damon_task *t, unsigned int n) +{ + struct damon_region *r; + unsigned int i; + + i = 0; + damon_for_each_region(r, t) { + if (i++ == n) + return r; + } + return NULL; +} + +static void damon_add_task_tail(struct damon_ctx *ctx, struct damon_task *t) +{ + list_add_tail(&t->list, &ctx->tasks_list); +} + +static void damon_del_task(struct damon_task *t) +{ + list_del(&t->list); +} + +static void damon_free_task(struct damon_task *t) +{ + struct damon_region *r, *next; + + damon_for_each_region_safe(r, next, t) + damon_free_region(r); + kfree(t); +} + +static void damon_destroy_task(struct damon_task *t) +{ + damon_del_task(t); + damon_free_task(t); +} + +/* + * Returns number of monitoring target tasks + */ +static unsigned int nr_damon_tasks(struct damon_ctx *ctx) +{ + struct damon_task *t; + unsigned int ret = 0; + + damon_for_each_task(ctx, t) + ret++; + return ret; +} + +/* + * Returns the number of target regions for a given target task + */ +static unsigned int nr_damon_regions(struct damon_task *t) +{ + struct damon_region *r; + unsigned int ret = 0; + + damon_for_each_region(r, t) + ret++; + return ret; +} + +static int __init damon_init(void) +{ + pr_info("init\n"); + + return 0; +} + +static void __exit damon_exit(void) +{ + pr_info("exit\n"); +} + +module_init(damon_init); +module_exit(damon_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("SeongJae Park "); +MODULE_DESCRIPTION("DAMON: Data Access MONitor"); From patchwork Mon Feb 17 10:25:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 11386029 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 38EDF159A for ; Mon, 17 Feb 2020 10:26:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DE3D420725 for ; Mon, 17 Feb 2020 10:26:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="EXhBNAKA" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DE3D420725 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BDCC16B0008; Mon, 17 Feb 2020 05:26:55 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B8F206B000A; Mon, 17 Feb 2020 05:26:55 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A55EE6B000C; Mon, 17 Feb 2020 05:26:55 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0053.hostedemail.com [216.40.44.53]) by kanga.kvack.org (Postfix) with ESMTP id 830366B0008 for ; Mon, 17 Feb 2020 05:26:55 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 3E1A38248047 for ; Mon, 17 Feb 2020 10:26:55 +0000 (UTC) X-FDA: 76499240790.14.wood59_1f3ae8569b942 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=309bfddc3=sjpark@amazon.com,:akpm@linux-foundation.org:sjpark@amazon.de:acme@kernel.org:alexander.shishkin@linux.intel.com:amit@kernel.org:brendan.d.gregg@gmail.com:brendanhiggins@google.com:cai@lca.pw:colin.king@canonical.com:corbet@lwn.net:dwmw@amazon.com:jolsa@redhat.com:kirill@shutemov.name:mark.rutland@arm.com:mgorman@suse.de:minchan@kernel.org:mingo@redhat.com:namhyung@kernel.org:peterz@infradead.org:rdunlap@infradead.org:rostedt@goodmis.org:shuah@kernel.org:sj38.park@gmail.com:vdavydov.dev@gmail.com::linux-doc@vger.kernel.org:linux-kernel@vger.kernel.org,RULES_HIT:30003:30012:30034:30051:30054:30056:30064:30070:30075:30090,0,RBL:72.21.198.25:@amazon.com:.lbl8.mailshell.net-62.18.0.100 66.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: wood59_1f3ae8569b942 X-Filterd-Recvd-Size: 20157 Received: from smtp-fw-4101.amazon.com (smtp-fw-4101.amazon.com [72.21.198.25]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Mon, 17 Feb 2020 10:26:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1581935215; x=1613471215; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=B6/QBiKcbLLT6n1pLhuWZIuWTpvozLtdsrSESZyAb6g=; b=EXhBNAKAo7Ay8ty0v8IpQ+ADhIosAI56Unc3M4HudUk6GrMv16yWaPEv tlRhZFQiqE1hsQjlHukWcxnk9hBx+KrwG8zig+BHHFJ7tZSHK0Hu60jZi RnuaIg8WAXV6sSZB27bvVZwOS9WVOR5xmQDfOFgPl+RY5qAvxYy/08tft M=; IronPort-SDR: +CQel/n+xy7QBZXBZvOE6eFLEjk2QwXESLUDPJJ10/Kaxe586r4O/wPIZm5eppZZjQgWC7mvl/ Y62cZrQqQr+g== X-IronPort-AV: E=Sophos;i="5.70,452,1574121600"; d="scan'208";a="17329264" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2c-4e7c8266.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 17 Feb 2020 10:26:39 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2c-4e7c8266.us-west-2.amazon.com (Postfix) with ESMTPS id E04F4A2E1A; Mon, 17 Feb 2020 10:26:36 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 17 Feb 2020 10:26:36 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.161.214) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 17 Feb 2020 10:26:26 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH v5 02/14] mm/damon: Implement region based sampling Date: Mon, 17 Feb 2020 11:25:32 +0100 Message-ID: <20200217102544.29012-3-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200217102544.29012-1-sjpark@amazon.com> References: <20200217102544.29012-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.214] X-ClientProxiedBy: EX13D30UWB004.ant.amazon.com (10.43.161.51) To EX13D31EUA001.ant.amazon.com (10.43.165.15) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park This commit implements DAMON's basic access check and region based sampling mechanisms. This change would seems make no sense, mainly because it is only a part of the DAMON's logics. Following two commits will make more sense. This commit also exports `lookup_page_ext()` to GPL modules because DAMON uses the function but also supports the module build. Basic Access Check ------------------ DAMON basically reports what pages are how frequently accessed. Note that the frequency is not an absolute number of accesses, but a relative frequency among the pages of the target workloads. Users can control the resolution of the reports by setting two time intervals, ``sampling interval`` and ``aggregation interval``. In detail, DAMON checks access to each page per ``sampling interval``, aggregates the results (counts the number of the accesses to each page), and reports the aggregated results per ``aggregation interval``. For the access check of each page, DAMON uses the Accessed bits of PTEs. This is thus similar to common periodic access checks based access tracking mechanisms, which overhead is increasing as the size of the target process grows. Region Based Sampling --------------------- To avoid the unbounded increase of the overhead, DAMON groups a number of adjacent pages that assumed to have same access frequencies into a region. As long as the assumption (pages in a region have same access frequencies) is kept, only one page in the region is required to be checked. Thus, for each ``sampling interval``, DAMON randomly picks one page in each region and clears its Accessed bit. After one more ``sampling interval``, DAMON reads the Accessed bit of the page and increases the access frequency of the region if the bit has set meanwhile. Therefore, the monitoring overhead is controllable by setting the number of regions. Nonetheless, this scheme cannot preserve the quality of the output if the assumption is not kept. Following commit will introduce how we can make the guarantee with best effort. Signed-off-by: SeongJae Park --- mm/damon.c | 509 ++++++++++++++++++++++++++++++++++++++++++++++++++ mm/page_ext.c | 1 + 2 files changed, 510 insertions(+) diff --git a/mm/damon.c b/mm/damon.c index aafdca35b7b8..ca21a617b5ff 100644 --- a/mm/damon.c +++ b/mm/damon.c @@ -9,9 +9,14 @@ #define pr_fmt(fmt) "damon: " fmt +#include +#include #include #include +#include #include +#include +#include #include #define damon_get_task_struct(t) \ @@ -51,7 +56,24 @@ struct damon_task { struct list_head list; }; +/* + * For each 'sample_interval', DAMON checks whether each region is accessed or + * not. It aggregates and keeps the access information (number of accesses to + * each region) for each 'aggr_interval' time. + * + * All time intervals are in micro-seconds. + */ struct damon_ctx { + unsigned long sample_interval; + unsigned long aggr_interval; + unsigned long min_nr_regions; + + struct timespec64 last_aggregation; + + struct task_struct *kdamond; + bool kdamond_stop; + spinlock_t kdamond_lock; + struct rnd_state rndseed; struct list_head tasks_list; /* 'damon_task' objects */ @@ -204,6 +226,493 @@ static unsigned int nr_damon_regions(struct damon_task *t) return ret; } +/* + * Get the mm_struct of the given task + * + * Callser should put the mm_struct after use, unless it is NULL. + * + * Returns the mm_struct of the task on success, NULL on failure + */ +static struct mm_struct *damon_get_mm(struct damon_task *t) +{ + struct task_struct *task; + struct mm_struct *mm; + + task = damon_get_task_struct(t); + if (!task) + return NULL; + + mm = get_task_mm(task); + put_task_struct(task); + return mm; +} + +/* + * Size-evenly split a region into 'nr_pieces' small regions + * + * Returns 0 on success, or negative error code otherwise. + */ +static int damon_split_region_evenly(struct damon_ctx *ctx, + struct damon_region *r, unsigned int nr_pieces) +{ + unsigned long sz_orig, sz_piece, orig_end; + struct damon_region *piece = NULL, *next; + unsigned long start; + + if (!r || !nr_pieces) + return -EINVAL; + + orig_end = r->vm_end; + sz_orig = r->vm_end - r->vm_start; + sz_piece = sz_orig / nr_pieces; + + if (!sz_piece) + return -EINVAL; + + r->vm_end = r->vm_start + sz_piece; + next = damon_next_region(r); + for (start = r->vm_end; start + sz_piece <= orig_end; + start += sz_piece) { + piece = damon_new_region(ctx, start, start + sz_piece); + damon_add_region(piece, r, next); + r = piece; + } + if (piece) + piece->vm_end = orig_end; + return 0; +} + +struct region { + unsigned long start; + unsigned long end; +}; + +static unsigned long sz_region(struct region *r) +{ + return r->end - r->start; +} + +static void swap_regions(struct region *r1, struct region *r2) +{ + struct region tmp; + + tmp = *r1; + *r1 = *r2; + *r2 = tmp; +} + +/* + * Find the three regions in an address space + * + * vma the head vma of the target address space + * regions an array of three 'struct region's that results will be saved + * + * This function receives an address space and finds three regions in it which + * separated by the two biggest unmapped regions in the space. Please refer to + * below comments of 'damon_init_regions_of()' function to know why this is + * necessary. + * + * Returns 0 if success, or negative error code otherwise. + */ +static int damon_three_regions_in_vmas(struct vm_area_struct *vma, + struct region regions[3]) +{ + struct region gap = {0,}, first_gap = {0,}, second_gap = {0,}; + struct vm_area_struct *last_vma = NULL; + unsigned long start = 0; + + /* Find two biggest gaps so that first_gap > second_gap > others */ + for (; vma; vma = vma->vm_next) { + if (!last_vma) { + start = vma->vm_start; + last_vma = vma; + continue; + } + gap.start = last_vma->vm_end; + gap.end = vma->vm_start; + if (sz_region(&gap) > sz_region(&second_gap)) { + swap_regions(&gap, &second_gap); + if (sz_region(&second_gap) > sz_region(&first_gap)) + swap_regions(&second_gap, &first_gap); + } + last_vma = vma; + } + + if (!sz_region(&second_gap) || !sz_region(&first_gap)) + return -EINVAL; + + /* Sort the two biggest gaps by address */ + if (first_gap.start > second_gap.start) + swap_regions(&first_gap, &second_gap); + + /* Store the result */ + regions[0].start = start; + regions[0].end = first_gap.start; + regions[1].start = first_gap.end; + regions[1].end = second_gap.start; + regions[2].start = second_gap.end; + regions[2].end = last_vma->vm_end; + + return 0; +} + +/* + * Get the three regions in the given task + * + * Returns 0 on success, negative error code otherwise. + */ +static int damon_three_regions_of(struct damon_task *t, + struct region regions[3]) +{ + struct mm_struct *mm; + int ret; + + mm = damon_get_mm(t); + if (!mm) + return -EINVAL; + + down_read(&mm->mmap_sem); + ret = damon_three_regions_in_vmas(mm->mmap, regions); + up_read(&mm->mmap_sem); + + mmput(mm); + return ret; +} + +/* + * Initialize the monitoring target regions for the given task + * + * t the given target task + * + * Because only a number of small portions of the entire address space + * is acutally mapped to the memory and accessed, monitoring the unmapped + * regions is wasteful. That said, because we can deal with small noises, + * tracking every mapping is not strictly required but could even incur a high + * overhead if the mapping frequently changes or the number of mappings is + * high. Nonetheless, this may seems very weird. DAMON's dynamic regions + * adjustment mechanism, which will be implemented with following commit will + * make this more sense. + * + * For the reason, we convert the complex mappings to three distinct regions + * that cover every mapped areas of the address space. Also the two gaps + * between the three regions are the two biggest unmapped areas in the given + * address space. In detail, this function first identifies the start and the + * end of the mappings and the two biggest unmapped areas of the address space. + * Then, it constructs the three regions as below: + * + * [mappings[0]->start, big_two_unmapped_areas[0]->start) + * [big_two_unmapped_areas[0]->end, big_two_unmapped_areas[1]->start) + * [big_two_unmapped_areas[1]->end, mappings[nr_mappings - 1]->end) + * + * As usual memory map of processes is as below, the gap between the heap and + * the uppermost mmap()-ed region, and the gap between the lowermost mmap()-ed + * region and the stack will be two biggest unmapped regions. Because these + * gaps are exceptionally huge areas in usual address space, excluding these + * two biggest unmapped regions will be sufficient to make a trade-off. + * + * + * + * + * (other mmap()-ed regions and small unmapped regions) + * + * + * + */ +static void damon_init_regions_of(struct damon_ctx *c, struct damon_task *t) +{ + struct damon_region *r; + struct region regions[3]; + int i; + + if (damon_three_regions_of(t, regions)) { + pr_err("Failed to get three regions of task %lu\n", t->pid); + return; + } + + /* Set the initial three regions of the task */ + for (i = 0; i < 3; i++) { + r = damon_new_region(c, regions[i].start, regions[i].end); + damon_add_region_tail(r, t); + } + + /* Split the middle region into 'min_nr_regions - 2' regions */ + r = damon_nth_region_of(t, 1); + if (damon_split_region_evenly(c, r, c->min_nr_regions - 2)) + pr_warn("Init middle region failed to be split\n"); +} + +/* Initialize '->regions_list' of every task */ +static void kdamond_init_regions(struct damon_ctx *ctx) +{ + struct damon_task *t; + + damon_for_each_task(ctx, t) + damon_init_regions_of(ctx, t); +} + +/* + * Check whether the given region has accessed since the last check + * + * mm 'mm_struct' for the given virtual address space + * r the region to be checked + */ +static void kdamond_check_access(struct damon_ctx *ctx, + struct mm_struct *mm, struct damon_region *r) +{ + pte_t *pte = NULL; + pmd_t *pmd = NULL; + spinlock_t *ptl; + + if (follow_pte_pmd(mm, r->sampling_addr, NULL, &pte, &pmd, &ptl)) + goto mkold; + + /* Read the page table access bit of the page */ + if (pte && pte_young(*pte)) + r->nr_accesses++; +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + else if (pmd && pmd_young(*pmd)) + r->nr_accesses++; +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + + spin_unlock(ptl); + +mkold: + /* mkold next target */ + r->sampling_addr = damon_rand(ctx, r->vm_start, r->vm_end); + + if (follow_pte_pmd(mm, r->sampling_addr, NULL, &pte, &pmd, &ptl)) + return; + + if (pte) { + if (pte_young(*pte)) { + clear_page_idle(pte_page(*pte)); + set_page_young(pte_page(*pte)); + } + *pte = pte_mkold(*pte); + } +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + else if (pmd) { + if (pmd_young(*pmd)) { + clear_page_idle(pmd_page(*pmd)); + set_page_young(pte_page(*pte)); + } + *pmd = pmd_mkold(*pmd); + } +#endif + + spin_unlock(ptl); +} + +/* + * Check whether a time interval is elapsed + * + * baseline the time to check whether the interval has elapsed since + * interval the time interval (microseconds) + * + * See whether the given time interval has passed since the given baseline + * time. If so, it also updates the baseline to current time for next check. + * + * Returns true if the time interval has passed, or false otherwise. + */ +static bool damon_check_reset_time_interval(struct timespec64 *baseline, + unsigned long interval) +{ + struct timespec64 now; + + ktime_get_coarse_ts64(&now); + if ((timespec64_to_ns(&now) - timespec64_to_ns(baseline)) < + interval * 1000) + return false; + *baseline = now; + return true; +} + +/* + * Check whether it is time to flush the aggregated information + */ +static bool kdamond_aggregate_interval_passed(struct damon_ctx *ctx) +{ + return damon_check_reset_time_interval(&ctx->last_aggregation, + ctx->aggr_interval); +} + +/* + * Reset the aggregated monitoring results + */ +static void kdamond_flush_aggregated(struct damon_ctx *c) +{ + struct damon_task *t; + struct damon_region *r; + + damon_for_each_task(c, t) { + damon_for_each_region(r, t) + r->nr_accesses = 0; + } +} + +/* + * Check whether current monitoring should be stopped + * + * If users asked to stop, need stop. Even though no user has asked to stop, + * need stop if every target task has dead. + * + * Returns true if need to stop current monitoring. + */ +static bool kdamond_need_stop(struct damon_ctx *ctx) +{ + struct damon_task *t; + struct task_struct *task; + bool stop; + + spin_lock(&ctx->kdamond_lock); + stop = ctx->kdamond_stop; + spin_unlock(&ctx->kdamond_lock); + if (stop) + return true; + + damon_for_each_task(ctx, t) { + task = damon_get_task_struct(t); + if (task) { + put_task_struct(task); + return false; + } + } + + return true; +} + +/* + * The monitoring daemon that runs as a kernel thread + */ +static int kdamond_fn(void *data) +{ + struct damon_ctx *ctx = (struct damon_ctx *)data; + struct damon_task *t; + struct damon_region *r, *next; + struct mm_struct *mm; + + pr_info("kdamond (%d) starts\n", ctx->kdamond->pid); + kdamond_init_regions(ctx); + while (!kdamond_need_stop(ctx)) { + damon_for_each_task(ctx, t) { + mm = damon_get_mm(t); + if (!mm) + continue; + damon_for_each_region(r, t) + kdamond_check_access(ctx, mm, r); + mmput(mm); + } + + if (kdamond_aggregate_interval_passed(ctx)) + kdamond_flush_aggregated(ctx); + + usleep_range(ctx->sample_interval, ctx->sample_interval + 1); + } + damon_for_each_task(ctx, t) { + damon_for_each_region_safe(r, next, t) + damon_destroy_region(r); + } + pr_info("kdamond (%d) finishes\n", ctx->kdamond->pid); + spin_lock(&ctx->kdamond_lock); + ctx->kdamond = NULL; + spin_unlock(&ctx->kdamond_lock); + return 0; +} + +/* + * Controller functions + */ + +/* + * Start or stop the kdamond + * + * Returns 0 if success, negative error code otherwise. + */ +static int damon_turn_kdamond(struct damon_ctx *ctx, bool on) +{ + spin_lock(&ctx->kdamond_lock); + ctx->kdamond_stop = !on; + if (!ctx->kdamond && on) { + ctx->kdamond = kthread_run(kdamond_fn, ctx, "kdamond"); + if (!ctx->kdamond) + goto fail; + goto success; + } + if (ctx->kdamond && !on) { + spin_unlock(&ctx->kdamond_lock); + while (true) { + spin_lock(&ctx->kdamond_lock); + if (!ctx->kdamond) + goto success; + spin_unlock(&ctx->kdamond_lock); + + usleep_range(ctx->sample_interval, + ctx->sample_interval * 2); + } + } + + /* tried to turn on while turned on, or turn off while turned off */ + +fail: + spin_unlock(&ctx->kdamond_lock); + return -EINVAL; + +success: + spin_unlock(&ctx->kdamond_lock); + return 0; +} + +/* + * This function should not be called while the kdamond is running. + */ +static int damon_set_pids(struct damon_ctx *ctx, + unsigned long *pids, ssize_t nr_pids) +{ + ssize_t i; + struct damon_task *t, *next; + + damon_for_each_task_safe(ctx, t, next) + damon_destroy_task(t); + + for (i = 0; i < nr_pids; i++) { + t = damon_new_task(pids[i]); + if (!t) { + pr_err("Failed to alloc damon_task\n"); + return -ENOMEM; + } + damon_add_task_tail(ctx, t); + } + + return 0; +} + +/* + * Set attributes for the monitoring + * + * sample_int time interval between samplings + * aggr_int time interval between aggregations + * min_nr_reg minimal number of regions + * + * This function should not be called while the kdamond is running. + * Every time interval is in micro-seconds. + * + * Returns 0 on success, negative error code otherwise. + */ +static int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int, + unsigned long aggr_int, unsigned long min_nr_reg) +{ + if (min_nr_reg < 3) { + pr_err("min_nr_regions (%lu) should be bigger than 2\n", + min_nr_reg); + return -EINVAL; + } + + ctx->sample_interval = sample_int; + ctx->aggr_interval = aggr_int; + ctx->min_nr_regions = min_nr_reg; + return 0; +} + static int __init damon_init(void) { pr_info("init\n"); diff --git a/mm/page_ext.c b/mm/page_ext.c index 4ade843ff588..71169b45bba9 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -131,6 +131,7 @@ struct page_ext *lookup_page_ext(const struct page *page) MAX_ORDER_NR_PAGES); return get_entry(base, index); } +EXPORT_SYMBOL_GPL(lookup_page_ext); static int __init alloc_node_page_ext(int nid) { From patchwork Mon Feb 17 10:25:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 11386027 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 73676159A for ; Mon, 17 Feb 2020 10:26:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 253C520836 for ; Mon, 17 Feb 2020 10:26:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="tm1NvLOh" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 253C520836 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4FF846B0007; Mon, 17 Feb 2020 05:26:53 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4AF766B0008; Mon, 17 Feb 2020 05:26:53 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C5426B000A; Mon, 17 Feb 2020 05:26:53 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0233.hostedemail.com [216.40.44.233]) by kanga.kvack.org (Postfix) with ESMTP id 273C76B0007 for ; Mon, 17 Feb 2020 05:26:53 -0500 (EST) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A31712DFA for ; Mon, 17 Feb 2020 10:26:52 +0000 (UTC) X-FDA: 76499240664.15.road04_1ed8d6fc3fe3d X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=309bfddc3=sjpark@amazon.com,:akpm@linux-foundation.org:sjpark@amazon.de:acme@kernel.org:alexander.shishkin@linux.intel.com:amit@kernel.org:brendan.d.gregg@gmail.com:brendanhiggins@google.com:cai@lca.pw:colin.king@canonical.com:corbet@lwn.net:dwmw@amazon.com:jolsa@redhat.com:kirill@shutemov.name:mark.rutland@arm.com:mgorman@suse.de:minchan@kernel.org:mingo@redhat.com:namhyung@kernel.org:peterz@infradead.org:rdunlap@infradead.org:rostedt@goodmis.org:shuah@kernel.org:sj38.park@gmail.com:vdavydov.dev@gmail.com::linux-doc@vger.kernel.org:linux-kernel@vger.kernel.org,RULES_HIT:30003:30012:30034:30045:30054:30056:30064:30070:30090,0,RBL:207.171.190.10:@amazon.com:.lbl8.mailshell.net-62.18.0.100 66.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: road04_1ed8d6fc3fe3d X-Filterd-Recvd-Size: 11451 Received: from smtp-fw-33001.amazon.com (smtp-fw-33001.amazon.com [207.171.190.10]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Mon, 17 Feb 2020 10:26:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1581935212; x=1613471212; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=5tSdLQU1csS1GDh89l5Yxz9zW7p2GIVbHfo3C6tuh8Q=; b=tm1NvLOhxm60+Fq02qwa32dzvxJVI6xTn5Tk7kzGF26G0mT7MYfa88eM etIlm2PIM0wMTTkTHOSURseOSVZMkBmPd7KgyctmNur9JaGTDLcAbI+yA 35ELEOo0bb1vaxGF6ukZ0+NswbYXNfGy22LR6O3Vfz9OSWJbMeJ+QDMzD k=; IronPort-SDR: ZZaFFuDrcAo8fWHKb9rNWL4CAY4wN1mm55cWhGK024H104CUiPgP8Aa22HjfGEo5naPuig9Ov4 /sL0ZX62lkXA== X-IronPort-AV: E=Sophos;i="5.70,452,1574121600"; d="scan'208";a="26833756" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2b-4ff6265a.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 17 Feb 2020 10:26:50 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2b-4ff6265a.us-west-2.amazon.com (Postfix) with ESMTPS id 17096A26D0; Mon, 17 Feb 2020 10:26:49 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 17 Feb 2020 10:26:48 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.161.214) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 17 Feb 2020 10:26:38 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH v5 03/14] mm/damon: Adaptively adjust regions Date: Mon, 17 Feb 2020 11:25:33 +0100 Message-ID: <20200217102544.29012-4-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200217102544.29012-1-sjpark@amazon.com> References: <20200217102544.29012-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.214] X-ClientProxiedBy: EX13D30UWB004.ant.amazon.com (10.43.161.51) To EX13D31EUA001.ant.amazon.com (10.43.165.15) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park At the beginning of the monitoring, DAMON constructs the initial regions by evenly splitting the memory mapped address space of the process into the user-specified minimal number of regions. In this initial state, the assumption of the regions (pages in same region have similar access frequencies) is normally not kept and thus the monitoring quality could be low. To keep the assumption as much as possible, DAMON adaptively merges and splits each region. For each ``aggregation interval``, it compares the access frequencies of adjacent regions and merges those if the frequency difference is small. Then, after it reports and clears the aggregated access frequency of each region, it splits each region into two regions if the total number of regions is smaller than the half of the user-specified maximum number of regions. In this way, DAMON provides its best-effort quality and minimal overhead while keeping the bounds users set for their trade-off. Signed-off-by: SeongJae Park --- mm/damon.c | 151 ++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 144 insertions(+), 7 deletions(-) diff --git a/mm/damon.c b/mm/damon.c index ca21a617b5ff..8bd9b989aeb2 100644 --- a/mm/damon.c +++ b/mm/damon.c @@ -67,6 +67,7 @@ struct damon_ctx { unsigned long sample_interval; unsigned long aggr_interval; unsigned long min_nr_regions; + unsigned long max_nr_regions; struct timespec64 last_aggregation; @@ -389,9 +390,12 @@ static int damon_three_regions_of(struct damon_task *t, * regions is wasteful. That said, because we can deal with small noises, * tracking every mapping is not strictly required but could even incur a high * overhead if the mapping frequently changes or the number of mappings is - * high. Nonetheless, this may seems very weird. DAMON's dynamic regions - * adjustment mechanism, which will be implemented with following commit will - * make this more sense. + * high. The adaptive regions adjustment mechanism will further help to deal + * with the noises by simply identifying the unmapped areas as a region that + * has no access. Moreover, applying the real mappings that would have many + * unmapped areas inside will make the adaptive mechanism quite complex. That + * said, too huge unmapped areas inside the monitoring target should be removed + * to not take the time for the adaptive mechanism. * * For the reason, we convert the complex mappings to three distinct regions * that cover every mapped areas of the address space. Also the two gaps @@ -550,6 +554,123 @@ static void kdamond_flush_aggregated(struct damon_ctx *c) } } +#define sz_damon_region(r) (r->vm_end - r->vm_start) + +/* + * Merge two adjacent regions into one region + */ +static void damon_merge_two_regions(struct damon_region *l, + struct damon_region *r) +{ + l->nr_accesses = (l->nr_accesses * sz_damon_region(l) + + r->nr_accesses * sz_damon_region(r)) / + (sz_damon_region(l) + sz_damon_region(r)); + l->vm_end = r->vm_end; + damon_destroy_region(r); +} + +#define diff_of(a, b) (a > b ? a - b : b - a) + +/* + * Merge adjacent regions having similar access frequencies + * + * t task that merge operation will make change + * thres merge regions having '->nr_accesses' diff smaller than this + */ +static void damon_merge_regions_of(struct damon_task *t, unsigned int thres) +{ + struct damon_region *r, *prev = NULL, *next; + + damon_for_each_region_safe(r, next, t) { + if (!prev || prev->vm_end != r->vm_start) + goto next; + if (diff_of(prev->nr_accesses, r->nr_accesses) > thres) + goto next; + damon_merge_two_regions(prev, r); + continue; +next: + prev = r; + } +} + +/* + * Merge adjacent regions having similar access frequencies + * + * threshold merge regions havind nr_accesses diff larger than this + * + * This function merges monitoring target regions which are adjacent and their + * access frequencies are similar. This is for minimizing the monitoring + * overhead under the dynamically changeable access pattern. If a merge was + * unnecessarily made, later 'kdamond_split_regions()' will revert it. + */ +static void kdamond_merge_regions(struct damon_ctx *c, unsigned int threshold) +{ + struct damon_task *t; + + damon_for_each_task(c, t) + damon_merge_regions_of(t, threshold); +} + +/* + * Split a region into two small regions + * + * r the region to be split + * sz_r size of the first sub-region that will be made + */ +static void damon_split_region_at(struct damon_ctx *ctx, + struct damon_region *r, unsigned long sz_r) +{ + struct damon_region *new; + + new = damon_new_region(ctx, r->vm_start + sz_r, r->vm_end); + r->vm_end = new->vm_start; + + damon_add_region(new, r, damon_next_region(r)); +} + +static void damon_split_regions_of(struct damon_ctx *ctx, struct damon_task *t) +{ + struct damon_region *r, *next; + unsigned long sz_left_region; + + damon_for_each_region_safe(r, next, t) { + /* + * Randomly select size of left sub-region to be at least + * 10 percent and at most 90% of original region + */ + sz_left_region = (prandom_u32_state(&ctx->rndseed) % 9 + 1) * + (r->vm_end - r->vm_start) / 10; + /* Do not allow blank region */ + if (sz_left_region == 0) + continue; + damon_split_region_at(ctx, r, sz_left_region); + } +} + +/* + * splits every target regions into two randomly-sized regions + * + * This function splits every target regions into two random-sized regions if + * current total number of the regions is smaller than the half of the + * user-specified maximum number of regions. This is for maximizing the + * monitoring accuracy under the dynamically changeable access patterns. If a + * split was unnecessarily made, later 'kdamond_merge_regions()' will revert + * it. + */ +static void kdamond_split_regions(struct damon_ctx *ctx) +{ + struct damon_task *t; + unsigned int nr_regions = 0; + + damon_for_each_task(ctx, t) + nr_regions += nr_damon_regions(t); + if (nr_regions > ctx->max_nr_regions / 2) + return; + + damon_for_each_task(ctx, t) + damon_split_regions_of(ctx, t); +} + /* * Check whether current monitoring should be stopped * @@ -590,21 +711,29 @@ static int kdamond_fn(void *data) struct damon_task *t; struct damon_region *r, *next; struct mm_struct *mm; + unsigned long max_nr_accesses; pr_info("kdamond (%d) starts\n", ctx->kdamond->pid); kdamond_init_regions(ctx); while (!kdamond_need_stop(ctx)) { + max_nr_accesses = 0; damon_for_each_task(ctx, t) { mm = damon_get_mm(t); if (!mm) continue; - damon_for_each_region(r, t) + damon_for_each_region(r, t) { kdamond_check_access(ctx, mm, r); + if (r->nr_accesses > max_nr_accesses) + max_nr_accesses = r->nr_accesses; + } mmput(mm); } - if (kdamond_aggregate_interval_passed(ctx)) + if (kdamond_aggregate_interval_passed(ctx)) { + kdamond_merge_regions(ctx, max_nr_accesses / 10); kdamond_flush_aggregated(ctx); + kdamond_split_regions(ctx); + } usleep_range(ctx->sample_interval, ctx->sample_interval + 1); } @@ -692,24 +821,32 @@ static int damon_set_pids(struct damon_ctx *ctx, * sample_int time interval between samplings * aggr_int time interval between aggregations * min_nr_reg minimal number of regions + * max_nr_reg maximum number of regions * * This function should not be called while the kdamond is running. * Every time interval is in micro-seconds. * * Returns 0 on success, negative error code otherwise. */ -static int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int, - unsigned long aggr_int, unsigned long min_nr_reg) +static int damon_set_attrs(struct damon_ctx *ctx, + unsigned long sample_int, unsigned long aggr_int, + unsigned long min_nr_reg, unsigned long max_nr_reg) { if (min_nr_reg < 3) { pr_err("min_nr_regions (%lu) should be bigger than 2\n", min_nr_reg); return -EINVAL; } + if (min_nr_reg >= ctx->max_nr_regions) { + pr_err("invalid nr_regions. min (%lu) >= max (%lu)\n", + min_nr_reg, max_nr_reg); + return -EINVAL; + } ctx->sample_interval = sample_int; ctx->aggr_interval = aggr_int; ctx->min_nr_regions = min_nr_reg; + ctx->max_nr_regions = max_nr_reg; return 0; } From patchwork Mon Feb 17 10:25:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 11386033 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A3738159A for ; Mon, 17 Feb 2020 10:27:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6637920725 for ; Mon, 17 Feb 2020 10:27:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="PhuShPu3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6637920725 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A2F2D6B000A; Mon, 17 Feb 2020 05:27:04 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9DFCA6B000C; Mon, 17 Feb 2020 05:27:04 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F63D6B000D; Mon, 17 Feb 2020 05:27:04 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0021.hostedemail.com [216.40.44.21]) by kanga.kvack.org (Postfix) with ESMTP id 74B326B000A for ; Mon, 17 Feb 2020 05:27:04 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 378908248047 for ; Mon, 17 Feb 2020 10:27:04 +0000 (UTC) X-FDA: 76499241168.21.pigs01_2088719aa1c28 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=309bfddc3=sjpark@amazon.com,:akpm@linux-foundation.org:sjpark@amazon.de:acme@kernel.org:alexander.shishkin@linux.intel.com:amit@kernel.org:brendan.d.gregg@gmail.com:brendanhiggins@google.com:cai@lca.pw:colin.king@canonical.com:corbet@lwn.net:dwmw@amazon.com:jolsa@redhat.com:kirill@shutemov.name:mark.rutland@arm.com:mgorman@suse.de:minchan@kernel.org:mingo@redhat.com:namhyung@kernel.org:peterz@infradead.org:rdunlap@infradead.org:rostedt@goodmis.org:shuah@kernel.org:sj38.park@gmail.com:vdavydov.dev@gmail.com::linux-doc@vger.kernel.org:linux-kernel@vger.kernel.org,RULES_HIT:30003:30012:30034:30054:30056:30064:30070:30075,0,RBL:207.171.184.25:@amazon.com:.lbl8.mailshell.net-62.18.0.100 66.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: pigs01_2088719aa1c28 X-Filterd-Recvd-Size: 8420 Received: from smtp-fw-9101.amazon.com (smtp-fw-9101.amazon.com [207.171.184.25]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Mon, 17 Feb 2020 10:27:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1581935224; x=1613471224; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=HQ9SNscXPAG5ex+O+AHC+jdfEtu9KpmX1gBzL7Va3Gg=; b=PhuShPu34TUX0+eFs7wdH4sm9Fq4Zq1wwF+La+LhNbdMEDFEfqamlPwB vffHp55LWhe77cyjcp3LQIY9shLS/EjbrXce8KL8WCtHiG4waIP3Nx5XZ qUUlT3/ym9rcML038nLVOrBKsrqr7LHI2QF9UalU5wt9Uty1skUucT0Ao s=; IronPort-SDR: 6uVMgtTsacghzMRPbMFygm86vD98fXJCL6QzPhLfXQn8wRN9Yvesr5VdDhSRqm3loW5sBBXNrP pWkUTZGR2enw== X-IronPort-AV: E=Sophos;i="5.70,452,1574121600"; d="scan'208";a="17073296" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2c-6f38efd9.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 17 Feb 2020 10:27:01 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2c-6f38efd9.us-west-2.amazon.com (Postfix) with ESMTPS id 8721BA06AC; Mon, 17 Feb 2020 10:26:59 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 17 Feb 2020 10:26:59 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.161.214) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 17 Feb 2020 10:26:49 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH v5 04/14] mm/damon: Apply dynamic memory mapping changes Date: Mon, 17 Feb 2020 11:25:34 +0100 Message-ID: <20200217102544.29012-5-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200217102544.29012-1-sjpark@amazon.com> References: <20200217102544.29012-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.214] X-ClientProxiedBy: EX13D30UWB004.ant.amazon.com (10.43.161.51) To EX13D31EUA001.ant.amazon.com (10.43.165.15) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park Only a number of parts in the virtual address space of the processes is mapped to physical memory and accessed. Thus, tracking the unmapped address regions is just wasteful. However, tracking every memory mapping change might incur an overhead. For the reason, DAMON applies the dynamic memory mapping changes to the tracking regions only for each of a user-specified time interval (``regions update interval``). Signed-off-by: SeongJae Park --- mm/damon.c | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 95 insertions(+), 4 deletions(-) diff --git a/mm/damon.c b/mm/damon.c index 8bd9b989aeb2..56725faca51f 100644 --- a/mm/damon.c +++ b/mm/damon.c @@ -59,17 +59,22 @@ struct damon_task { /* * For each 'sample_interval', DAMON checks whether each region is accessed or * not. It aggregates and keeps the access information (number of accesses to - * each region) for each 'aggr_interval' time. + * each region) for each 'aggr_interval' time. And for each + * 'regions_update_interval', damon checks whether the memory mapping of the + * target tasks has changed (e.g., by mmap() calls from the applications) and + * applies the changes. * * All time intervals are in micro-seconds. */ struct damon_ctx { unsigned long sample_interval; unsigned long aggr_interval; + unsigned long regions_update_interval; unsigned long min_nr_regions; unsigned long max_nr_regions; struct timespec64 last_aggregation; + struct timespec64 last_regions_update; struct task_struct *kdamond; bool kdamond_stop; @@ -671,6 +676,87 @@ static void kdamond_split_regions(struct damon_ctx *ctx) damon_split_regions_of(ctx, t); } +/* + * Check whether it is time to check and apply the dynamic mmap changes + * + * Returns true if it is. + */ +static bool kdamond_need_update_regions(struct damon_ctx *ctx) +{ + return damon_check_reset_time_interval(&ctx->last_regions_update, + ctx->regions_update_interval); +} + +static bool damon_intersect(struct damon_region *r, struct region *re) +{ + return !(r->vm_end <= re->start || re->end <= r->vm_start); +} + +/* + * Update damon regions for the three big regions of the given task + * + * t the given task + * bregions the three big regions of the task + */ +static void damon_apply_three_regions(struct damon_ctx *ctx, + struct damon_task *t, struct region bregions[3]) +{ + struct damon_region *r, *next; + unsigned int i = 0; + + /* Remove regions which isn't in the three big regions now */ + damon_for_each_region_safe(r, next, t) { + for (i = 0; i < 3; i++) { + if (damon_intersect(r, &bregions[i])) + break; + } + if (i == 3) + damon_destroy_region(r); + } + + /* Adjust intersecting regions to fit with the threee big regions */ + for (i = 0; i < 3; i++) { + struct damon_region *first = NULL, *last; + struct damon_region *newr; + struct region *br; + + br = &bregions[i]; + /* Get the first and last regions which intersects with br */ + damon_for_each_region(r, t) { + if (damon_intersect(r, br)) { + if (!first) + first = r; + last = r; + } + if (r->vm_start >= br->end) + break; + } + if (!first) { + /* no damon_region intersects with this big region */ + newr = damon_new_region(ctx, br->start, br->end); + damon_add_region(newr, damon_prev_region(r), r); + } else { + first->vm_start = br->start; + last->vm_end = br->end; + } + } +} + +/* + * Update regions for current memory mappings + */ +static void kdamond_update_regions(struct damon_ctx *ctx) +{ + struct region three_regions[3]; + struct damon_task *t; + + damon_for_each_task(ctx, t) { + if (damon_three_regions_of(t, three_regions)) + continue; + damon_apply_three_regions(ctx, t, three_regions); + } +} + /* * Check whether current monitoring should be stopped * @@ -735,6 +821,9 @@ static int kdamond_fn(void *data) kdamond_split_regions(ctx); } + if (kdamond_need_update_regions(ctx)) + kdamond_update_regions(ctx); + usleep_range(ctx->sample_interval, ctx->sample_interval + 1); } damon_for_each_task(ctx, t) { @@ -820,6 +909,7 @@ static int damon_set_pids(struct damon_ctx *ctx, * * sample_int time interval between samplings * aggr_int time interval between aggregations + * regions_update_int time interval between vma update checks * min_nr_reg minimal number of regions * max_nr_reg maximum number of regions * @@ -828,9 +918,9 @@ static int damon_set_pids(struct damon_ctx *ctx, * * Returns 0 on success, negative error code otherwise. */ -static int damon_set_attrs(struct damon_ctx *ctx, - unsigned long sample_int, unsigned long aggr_int, - unsigned long min_nr_reg, unsigned long max_nr_reg) +static int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int, + unsigned long aggr_int, unsigned long regions_update_int, + unsigned long min_nr_reg, unsigned long max_nr_reg) { if (min_nr_reg < 3) { pr_err("min_nr_regions (%lu) should be bigger than 2\n", @@ -845,6 +935,7 @@ static int damon_set_attrs(struct damon_ctx *ctx, ctx->sample_interval = sample_int; ctx->aggr_interval = aggr_int; + ctx->regions_update_interval = regions_update_int; ctx->min_nr_regions = min_nr_reg; ctx->max_nr_regions = max_nr_reg; return 0; From patchwork Mon Feb 17 10:25:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 11386035 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2208817F0 for ; Mon, 17 Feb 2020 10:27:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E0647208C4 for ; Mon, 17 Feb 2020 10:27:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="PcW8jZ7w" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E0647208C4 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2B5EB6B0005; Mon, 17 Feb 2020 05:27:31 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 264FB6B000D; Mon, 17 Feb 2020 05:27:31 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1547C6B000E; Mon, 17 Feb 2020 05:27:31 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0202.hostedemail.com [216.40.44.202]) by kanga.kvack.org (Postfix) with ESMTP id F21416B0005 for ; Mon, 17 Feb 2020 05:27:30 -0500 (EST) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id AF83D2464 for ; Mon, 17 Feb 2020 10:27:30 +0000 (UTC) X-FDA: 76499242260.08.sun50_246d1139bc716 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=309bfddc3=sjpark@amazon.com,:akpm@linux-foundation.org:sjpark@amazon.de:acme@kernel.org:alexander.shishkin@linux.intel.com:amit@kernel.org:brendan.d.gregg@gmail.com:brendanhiggins@google.com:cai@lca.pw:colin.king@canonical.com:corbet@lwn.net:dwmw@amazon.com:jolsa@redhat.com:kirill@shutemov.name:mark.rutland@arm.com:mgorman@suse.de:minchan@kernel.org:mingo@redhat.com:namhyung@kernel.org:peterz@infradead.org:rdunlap@infradead.org:rostedt@goodmis.org:shuah@kernel.org:sj38.park@gmail.com:vdavydov.dev@gmail.com::linux-doc@vger.kernel.org:linux-kernel@vger.kernel.org,RULES_HIT:30034:30054:30064,0,RBL:72.21.196.25:@amazon.com:.lbl8.mailshell.net-62.18.0.100 66.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: sun50_246d1139bc716 X-Filterd-Recvd-Size: 4311 Received: from smtp-fw-2101.amazon.com (smtp-fw-2101.amazon.com [72.21.196.25]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Mon, 17 Feb 2020 10:27:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1581935251; x=1613471251; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=noSLYb1CP/9HIx9FUVJwc4ZRGLjE8MhCwuZnuaCR6MU=; b=PcW8jZ7wwV6c3vlBjB3stHTnMKn95169ZeMdqPRAnrB+VC7puQKFfBdU Yi6SGYApSgg/gmVNzt93jQa+0jqUFxprnjqz1lXsfjALWY4ne1duS/ocw el2aKCS9NYOsKE6TCS458ArJXF1vpiudmRiiN2dOfIlL1lvPUsL1YZmuM Y=; IronPort-SDR: sTRa+7B9JfDyukr3WOLLnr9UI7EJR8Bgv7YWdTMMr0A8NzioSXgbBIHCBq6xL5+Jl5Nqg1ViH2 mtnDEN7ORsGQ== X-IronPort-AV: E=Sophos;i="5.70,452,1574121600"; d="scan'208";a="17477633" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-2c-2225282c.us-west-2.amazon.com) ([10.43.8.2]) by smtp-border-fw-out-2101.iad2.amazon.com with ESMTP; 17 Feb 2020 10:27:16 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2c-2225282c.us-west-2.amazon.com (Postfix) with ESMTPS id 8D46DA33CA; Mon, 17 Feb 2020 10:27:13 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 17 Feb 2020 10:27:13 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.161.214) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 17 Feb 2020 10:27:03 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH v5 05/14] mm/damon: Implement callbacks Date: Mon, 17 Feb 2020 11:25:35 +0100 Message-ID: <20200217102544.29012-6-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200217102544.29012-1-sjpark@amazon.com> References: <20200217102544.29012-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.214] X-ClientProxiedBy: EX13D30UWB004.ant.amazon.com (10.43.161.51) To EX13D31EUA001.ant.amazon.com (10.43.165.15) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park This commit implements callbacks for DAMON. Using this, DAMON users can install their callbacks for each step of the access monitoring so that they can do something interesting with the monitored access pattrns online. For example, callbacks can report the monitored patterns to users or do some access pattern based memory management such as proactive reclamations or access pattern based THP promotions/demotions decision makings. Signed-off-by: SeongJae Park --- mm/damon.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/mm/damon.c b/mm/damon.c index 56725faca51f..90f61e378a61 100644 --- a/mm/damon.c +++ b/mm/damon.c @@ -83,6 +83,10 @@ struct damon_ctx { struct rnd_state rndseed; struct list_head tasks_list; /* 'damon_task' objects */ + + /* callbacks */ + void (*sample_cb)(struct damon_ctx *context); + void (*aggregate_cb)(struct damon_ctx *context); }; /* Get a random number in [l, r) */ @@ -814,9 +818,13 @@ static int kdamond_fn(void *data) } mmput(mm); } + if (ctx->sample_cb) + ctx->sample_cb(ctx); if (kdamond_aggregate_interval_passed(ctx)) { kdamond_merge_regions(ctx, max_nr_accesses / 10); + if (ctx->aggregate_cb) + ctx->aggregate_cb(ctx); kdamond_flush_aggregated(ctx); kdamond_split_regions(ctx); } From patchwork Mon Feb 17 10:25:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 11386041 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A132717F0 for ; Mon, 17 Feb 2020 10:27:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 643FB20836 for ; Mon, 17 Feb 2020 10:27:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="DFxW2z4U" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 643FB20836 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 501826B000D; Mon, 17 Feb 2020 05:27:44 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4B2436B000E; Mon, 17 Feb 2020 05:27:44 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 379046B0010; Mon, 17 Feb 2020 05:27:44 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0089.hostedemail.com [216.40.44.89]) by kanga.kvack.org (Postfix) with ESMTP id 1EED76B000D for ; Mon, 17 Feb 2020 05:27:44 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id C16FE181AEF07 for ; Mon, 17 Feb 2020 10:27:43 +0000 (UTC) X-FDA: 76499242806.11.pest71_264d760a1f936 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=309bfddc3=sjpark@amazon.com,:akpm@linux-foundation.org:sjpark@amazon.de:acme@kernel.org:alexander.shishkin@linux.intel.com:amit@kernel.org:brendan.d.gregg@gmail.com:brendanhiggins@google.com:cai@lca.pw:colin.king@canonical.com:corbet@lwn.net:dwmw@amazon.com:jolsa@redhat.com:kirill@shutemov.name:mark.rutland@arm.com:mgorman@suse.de:minchan@kernel.org:mingo@redhat.com:namhyung@kernel.org:peterz@infradead.org:rdunlap@infradead.org:rostedt@goodmis.org:shuah@kernel.org:sj38.park@gmail.com:vdavydov.dev@gmail.com::linux-doc@vger.kernel.org:linux-kernel@vger.kernel.org,RULES_HIT:30003:30012:30034:30051:30054:30064:30070:30075,0,RBL:52.95.48.154:@amazon.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: pest71_264d760a1f936 X-Filterd-Recvd-Size: 8216 Received: from smtp-fw-6001.amazon.com (smtp-fw-6001.amazon.com [52.95.48.154]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Mon, 17 Feb 2020 10:27:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1581935264; x=1613471264; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=oZnJcj23ZgAKIikbswlN6tGEpYxRqIysLxZZhWCaknk=; b=DFxW2z4UvYyK2WYwGvW+zWqTMYbzdf1HWrdTkyaYRMgNkLCSFNwFfz97 IKBy1/xw0QAJwykpLCg49LZ1hz7wQeYGXnx9zw4cktCMAh3RBXAnFlcwu GYfhMq1kFLYodtcgYZF/3If04JXXYNCWHB9tpJpyDIIRm23Qch7rF05kW M=; IronPort-SDR: GazxHAAmHUukeku1pj7SYzlT6iaU8QitX8fXT+jZrrLOabChlTSrfYQ19t85UQf3YqkvjujREx 2c/TMU8m8D9w== X-IronPort-AV: E=Sophos;i="5.70,452,1574121600"; d="scan'208";a="18057870" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2c-4e7c8266.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6001.iad6.amazon.com with ESMTP; 17 Feb 2020 10:27:28 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2c-4e7c8266.us-west-2.amazon.com (Postfix) with ESMTPS id C7E50A2E1B; Mon, 17 Feb 2020 10:27:25 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 17 Feb 2020 10:27:25 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.161.214) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 17 Feb 2020 10:27:15 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH v5 06/14] mm/damon: Implement access pattern recording Date: Mon, 17 Feb 2020 11:25:36 +0100 Message-ID: <20200217102544.29012-7-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200217102544.29012-1-sjpark@amazon.com> References: <20200217102544.29012-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.214] X-ClientProxiedBy: EX13D30UWB004.ant.amazon.com (10.43.161.51) To EX13D31EUA001.ant.amazon.com (10.43.165.15) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park This commit implements the recording feature of DAMON. If this feature is enabled, DAMON writes the monitored access patterns in its binary format into a file which specified by the user. This is already able to be implemented by each user using the callbacks. However, as the recording is expected to be used widely, this commit implements the feature in the DAMON, for more convenience and efficiency. Signed-off-by: SeongJae Park --- mm/damon.c | 124 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 121 insertions(+), 3 deletions(-) diff --git a/mm/damon.c b/mm/damon.c index 90f61e378a61..3d46bf5e353a 100644 --- a/mm/damon.c +++ b/mm/damon.c @@ -76,6 +76,11 @@ struct damon_ctx { struct timespec64 last_aggregation; struct timespec64 last_regions_update; + unsigned char *rbuf; + unsigned int rbuf_len; + unsigned int rbuf_offset; + char *rfile_path; + struct task_struct *kdamond; bool kdamond_stop; spinlock_t kdamond_lock; @@ -89,6 +94,8 @@ struct damon_ctx { void (*aggregate_cb)(struct damon_ctx *context); }; +#define MAX_RFILE_PATH_LEN 256 + /* Get a random number in [l, r) */ #define damon_rand(ctx, l, r) (l + prandom_u32_state(&ctx->rndseed) % (r - l)) @@ -550,16 +557,81 @@ static bool kdamond_aggregate_interval_passed(struct damon_ctx *ctx) } /* - * Reset the aggregated monitoring results + * Flush the content in the result buffer to the result file + */ +static void damon_flush_rbuffer(struct damon_ctx *ctx) +{ + ssize_t sz; + loff_t pos; + struct file *rfile; + + while (ctx->rbuf_offset) { + pos = 0; + rfile = filp_open(ctx->rfile_path, O_CREAT | O_RDWR | O_APPEND, + 0644); + if (IS_ERR(rfile)) { + pr_err("Cannot open the result file %s\n", + ctx->rfile_path); + return; + } + + sz = kernel_write(rfile, ctx->rbuf, ctx->rbuf_offset, &pos); + filp_close(rfile, NULL); + + ctx->rbuf_offset -= sz; + } +} + +/* + * Write a data into the result buffer + */ +static void damon_write_rbuf(struct damon_ctx *ctx, void *data, ssize_t size) +{ + if (!ctx->rbuf_len || !ctx->rbuf) + return; + if (ctx->rbuf_offset + size > ctx->rbuf_len) + damon_flush_rbuffer(ctx); + + memcpy(&ctx->rbuf[ctx->rbuf_offset], data, size); + ctx->rbuf_offset += size; +} + +/* + * Flush the aggregated monitoring results to the result buffer + * + * Stores current tracking results to the result buffer and reset 'nr_accesses' + * of each regions. The format for the result buffer is as below: + * + *