From patchwork Tue Apr 7 10:00:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 11477845 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B801292A for ; Tue, 7 Apr 2020 10:02:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 69D332078C for ; Tue, 7 Apr 2020 10:02:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="LKnmKUqF" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 69D332078C Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 836138E001D; Tue, 7 Apr 2020 06:02:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7E6968E0001; Tue, 7 Apr 2020 06:02:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6FD8A8E001D; Tue, 7 Apr 2020 06:02:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0191.hostedemail.com [216.40.44.191]) by kanga.kvack.org (Postfix) with ESMTP id 585888E0001 for ; Tue, 7 Apr 2020 06:02:23 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 20642180AD806 for ; Tue, 7 Apr 2020 10:02:23 +0000 (UTC) X-FDA: 76680618966.19.song07_6c06f2f7e6d37 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=359b079f7=sjpark@amazon.com,,RULES_HIT:30004:30012:30034:30054:30064:30070:30075,0,RBL:207.171.184.25:@amazon.com:.lbl8.mailshell.net-62.18.0.100 66.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: song07_6c06f2f7e6d37 X-Filterd-Recvd-Size: 11597 Received: from smtp-fw-9101.amazon.com (smtp-fw-9101.amazon.com [207.171.184.25]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Tue, 7 Apr 2020 10:02:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1586253742; x=1617789742; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=bqyWO94kOUuvdW2B5n79dpmnjV7Acaw0GdH8vXda4E8=; b=LKnmKUqFGYPHodfP7ZSio6Bz04p41alndWckd2YSbrolQ0zYOe0SJ3Cl gvyOXiq1bmQ/JxM+9T01IFEYtsle2VCCvItoTcNOcXf8ksUNnNF3vNmCs zylZpJXrJ2FfpiRU+/5BgWAjjS2fsxO+7sFjjI8YpwPJtv+chKgotp+t5 I=; IronPort-SDR: wj6kxm553tygYfzaCNOEPfZj2tee8FULjvoIHK+Kun+duTab3SjOugm0Aa6I9/uae1Fc4PM3xQ 9L/d+vEDak7w== X-IronPort-AV: E=Sophos;i="5.72,353,1580774400"; d="scan'208";a="27348609" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2b-5bdc5131.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 07 Apr 2020 10:02:19 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2b-5bdc5131.us-west-2.amazon.com (Postfix) with ESMTPS id 43D3FA291C; Tue, 7 Apr 2020 10:02:17 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 7 Apr 2020 10:02:16 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.162.171) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 7 Apr 2020 10:02:02 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC v6 3/7] mm/damon: Implement data access monitoring-based operation schemes Date: Tue, 7 Apr 2020 12:00:02 +0200 Message-ID: <20200407100007.3894-4-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200407100007.3894-1-sjpark@amazon.com> References: <20200407100007.3894-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.162.171] X-ClientProxiedBy: EX13D34UWA002.ant.amazon.com (10.43.160.245) To EX13D31EUA001.ant.amazon.com (10.43.165.15) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park In many cases, users might use DAMON for simple data access awared memory management optimizations such as applying an operation scheme to a memory region of a specific size having a specific access frequency for a specific time. For example, "page out a memory region larger than 100 MiB but having a low access frequency more than 10 minutes", or "Use THP for a memory region larger than 2 MiB having a high access frequency for more than 2 seconds". To minimize users from spending their time for implementation of such simple data access monitoring-based operation schemes, this commit makes DAMON to handle such schemes directly. With this commit, users can simply specify their desired schemes to DAMON. Each of the schemes is composed with conditions for filtering of the target memory regions and desired memory management action for the target. In specific, the format is:: The filtering conditions are size of memory region, number of accesses to the region monitored by DAMON, and the age of the region. The age of region is incremented periodically but reset when its addresses or access frequency has significanly changed or the action of a scheme has applied. For the action, current implementation supports only a few of madvise() hints, ``MADV_WILLNEED``, ``MADV_COLD``, ``MADV_PAGEOUT``, ``MADV_HUGEPAGE``, and ``MADV_NOHUGEPAGE``. Signed-off-by: SeongJae Park --- include/linux/damon.h | 24 +++++++ mm/damon.c | 149 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 173 insertions(+) diff --git a/include/linux/damon.h b/include/linux/damon.h index 49205c71c63d..b0fa898ed6d8 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -36,6 +36,27 @@ struct damon_task { struct list_head list; }; +/* Data Access Monitoring-based Operation Scheme */ +enum damos_action { + DAMOS_WILLNEED, + DAMOS_COLD, + DAMOS_PAGEOUT, + DAMOS_HUGEPAGE, + DAMOS_NOHUGEPAGE, + DAMOS_ACTION_LEN, +}; + +struct damos { + unsigned int min_sz_region; + unsigned int max_sz_region; + unsigned int min_nr_accesses; + unsigned int max_nr_accesses; + unsigned int min_age_region; + unsigned int max_age_region; + enum damos_action action; + struct list_head list; +}; + /* * For each 'sample_interval', DAMON checks whether each region is accessed or * not. It aggregates and keeps the access information (number of accesses to @@ -66,6 +87,7 @@ struct damon_ctx { struct rnd_state rndseed; struct list_head tasks_list; /* 'damon_task' objects */ + struct list_head schemes_list; /* 'damos' objects */ /* callbacks */ void (*sample_cb)(struct damon_ctx *context); @@ -76,6 +98,8 @@ int damon_set_pids(struct damon_ctx *ctx, unsigned long *pids, ssize_t nr_pids); int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int, unsigned long aggr_int, unsigned long regions_update_int, unsigned long min_nr_reg, unsigned long max_nr_reg); +int damon_set_schemes(struct damon_ctx *ctx, + struct damos **schemes, ssize_t nr_schemes); int damon_set_recording(struct damon_ctx *ctx, unsigned int rbuf_len, char *rfile_path); int damon_start(struct damon_ctx *ctx); diff --git a/mm/damon.c b/mm/damon.c index 7e56011b9b5f..3c10ff3a487c 100644 --- a/mm/damon.c +++ b/mm/damon.c @@ -11,6 +11,7 @@ #define CREATE_TRACE_POINTS +#include #include #include #include @@ -45,6 +46,12 @@ #define damon_for_each_task_safe(ctx, t, next) \ list_for_each_entry_safe(t, next, &(ctx)->tasks_list, list) +#define damon_for_each_schemes(ctx, r) \ + list_for_each_entry(r, &(ctx)->schemes_list, list) + +#define damon_for_each_schemes_safe(ctx, s, next) \ + list_for_each_entry_safe(s, next, &(ctx)->schemes_list, list) + #define MAX_RFILE_PATH_LEN 256 /* Get a random number in [l, r) */ @@ -173,6 +180,27 @@ static void damon_destroy_task(struct damon_task *t) damon_free_task(t); } +static void damon_add_scheme(struct damon_ctx *ctx, struct damos *s) +{ + list_add_tail(&s->list, &ctx->schemes_list); +} + +static void damon_del_scheme(struct damos *s) +{ + list_del(&s->list); +} + +static void damon_free_scheme(struct damos *s) +{ + kfree(s); +} + +static void damon_destroy_scheme(struct damos *s) +{ + damon_del_scheme(s); + damon_free_scheme(s); +} + static unsigned int nr_damon_tasks(struct damon_ctx *ctx) { struct damon_task *t; @@ -701,6 +729,101 @@ static void kdamond_count_age(struct damon_ctx *c, unsigned int threshold) } } +#ifndef CONFIG_ADVISE_SYSCALLS +static int damos_madvise(struct damon_task *task, struct damon_region *r, + int behavior) +{ + return -EINVAL; +} +#else +static int damos_madvise(struct damon_task *task, struct damon_region *r, + int behavior) +{ + struct task_struct *t; + struct mm_struct *mm; + int ret = -ENOMEM; + + t = damon_get_task_struct(task); + if (!t) + goto out; + mm = damon_get_mm(task); + if (!mm) + goto put_task_out; + + ret = do_madvise(t, mm, PAGE_ALIGN(r->vm_start), + PAGE_ALIGN(r->vm_end - r->vm_start), behavior); + mmput(mm); +put_task_out: + put_task_struct(t); +out: + return ret; +} +#endif /* CONFIG_ADVISE_SYSCALLS */ + +static int damos_do_action(struct damon_task *task, struct damon_region *r, + enum damos_action action) +{ + int madv_action; + + switch (action) { + case DAMOS_WILLNEED: + madv_action = MADV_WILLNEED; + break; + case DAMOS_COLD: + madv_action = MADV_COLD; + break; + case DAMOS_PAGEOUT: + madv_action = MADV_PAGEOUT; + break; + case DAMOS_HUGEPAGE: + madv_action = MADV_HUGEPAGE; + break; + case DAMOS_NOHUGEPAGE: + madv_action = MADV_NOHUGEPAGE; + break; + default: + pr_warn("Wrong action %d\n", action); + return -EINVAL; + } + + return damos_madvise(task, r, madv_action); +} + +static void damon_do_apply_schemes(struct damon_ctx *c, struct damon_task *t, + struct damon_region *r) +{ + struct damos *s; + unsigned long sz; + + damon_for_each_schemes(c, s) { + sz = r->vm_end - r->vm_start; + if ((s->min_sz_region && sz < s->min_sz_region) || + (s->max_sz_region && s->max_sz_region < sz)) + continue; + if ((s->min_nr_accesses && r->nr_accesses < s->min_nr_accesses) + || (s->max_nr_accesses && + s->max_nr_accesses < r->nr_accesses)) + continue; + if ((s->min_age_region && r->age < s->min_age_region) || + (s->max_age_region && + s->max_age_region < r->age)) + continue; + damos_do_action(t, r, s->action); + r->age = 0; + } +} + +static void kdamond_apply_schemes(struct damon_ctx *c) +{ + struct damon_task *t; + struct damon_region *r; + + damon_for_each_task(c, t) { + damon_for_each_region(r, t) + damon_do_apply_schemes(c, t, r); + } +} + #define sz_damon_region(r) (r->vm_end - r->vm_start) /* @@ -1011,6 +1134,7 @@ static int kdamond_fn(void *data) kdamond_count_age(ctx, max_nr_accesses / 10); if (ctx->aggregate_cb) ctx->aggregate_cb(ctx); + kdamond_apply_schemes(ctx); kdamond_reset_aggregated(ctx); kdamond_split_regions(ctx); } @@ -1090,6 +1214,30 @@ int damon_stop(struct damon_ctx *ctx) return -EBUSY; } +/** + * damon_set_schemes() - Set data access monitoring based operation schemes. + * @ctx: monitoring context + * @schemes: array of the schemes + * @nr_schemes: number of entries in @schemes + * + * This function should not be called while the kdamond of the context is + * running. + * + * Return: 0 if success, or negative error code otherwise. + */ +int damon_set_schemes(struct damon_ctx *ctx, struct damos **schemes, + ssize_t nr_schemes) +{ + struct damos *s, *next; + ssize_t i; + + damon_for_each_schemes_safe(ctx, s, next) + damon_destroy_scheme(s); + for (i = 0; i < nr_schemes; i++) + damon_add_scheme(ctx, schemes[i]); + return 0; +} + /** * damon_set_pids() - Set monitoring target processes. * @ctx: monitoring context @@ -1526,6 +1674,7 @@ static int __init damon_init_user_ctx(void) prandom_seed_state(&ctx->rndseed, 42); INIT_LIST_HEAD(&ctx->tasks_list); + INIT_LIST_HEAD(&ctx->schemes_list); return 0; }