From patchwork Mon Jun 15 16:19:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 11605397 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 059FF92A for ; Mon, 15 Jun 2020 16:21:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B79D92078E for ; Mon, 15 Jun 2020 16:21:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="hG6Y0kim" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B79D92078E Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EF0E66B000A; Mon, 15 Jun 2020 12:21:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EA0086B000C; Mon, 15 Jun 2020 12:21:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D425D6B000D; Mon, 15 Jun 2020 12:21:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0085.hostedemail.com [216.40.44.85]) by kanga.kvack.org (Postfix) with ESMTP id BB26D6B000A for ; Mon, 15 Jun 2020 12:21:19 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 71D03181AC9BF for ; Mon, 15 Jun 2020 16:21:19 +0000 (UTC) X-FDA: 76931961078.10.feast42_4006b6726df7 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id 3D38AE3499 for ; Mon, 15 Jun 2020 16:21:19 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=4280d6d30=sjpark@amazon.com,,RULES_HIT:30012:30034:30054:30056:30064:30070:30075,0,RBL:72.21.198.25:@amazon.com:.lbl8.mailshell.net-62.18.0.100 66.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: feast42_4006b6726df7 X-Filterd-Recvd-Size: 13215 Received: from smtp-fw-4101.amazon.com (smtp-fw-4101.amazon.com [72.21.198.25]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Mon, 15 Jun 2020 16:21:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1592238079; x=1623774079; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=kqIab3xglKx0fy5qTGilfaA/67p8jUh4Ymldbzg99T0=; b=hG6Y0kimZe9u34ielpgWqsgHc/LP3iUkQFd2AtEyNzjPw9cMOyyYwRad gIWdkZtHMfuEF6OkT/5zX8MBXE9tliQDCJ4yiW5WTgRDAfmj+QWSHTcyW I98zIvvnDaKDJPgm+nA72eNx3tZv/7Md1AHU+zRpNhao2Oq30svcPL7Wz 4=; IronPort-SDR: if8CRx50ytdWNmeZ8p379M7T1pLT8XK3UA+aUfYBVLvh8s5XPwzbah2SFvhAPBndVevi87PC9/ n7ADZ62Yl7WA== X-IronPort-AV: E=Sophos;i="5.73,515,1583193600"; d="scan'208";a="36385423" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2b-a7fdc47a.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 15 Jun 2020 16:21:15 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-a7fdc47a.us-west-2.amazon.com (Postfix) with ESMTPS id 3F795C05E2; Mon, 15 Jun 2020 16:21:13 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 15 Jun 2020 16:21:12 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.161.145) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 15 Jun 2020 16:20:55 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH v16 04/14] mm/damon: Adaptively adjust regions Date: Mon, 15 Jun 2020 18:19:17 +0200 Message-ID: <20200615161927.12637-5-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200615161927.12637-1-sjpark@amazon.com> References: <20200615161927.12637-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.145] X-ClientProxiedBy: EX13D23UWA004.ant.amazon.com (10.43.160.72) To EX13D31EUA001.ant.amazon.com (10.43.165.15) X-Rspamd-Queue-Id: 3D38AE3499 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park Even somehow the initial monitoring target regions are well constructed to fulfill the assumption (pages in same region have similar access frequencies), the data access pattern can be dynamically changed. This will result in low monitoring quality. To keep the assumption as much as possible, DAMON adaptively merges and splits each region. For each ``aggregation interval``, it compares the access frequencies of adjacent regions and merges those if the frequency difference is small. Then, after it reports and clears the aggregated access frequency of each region, it splits each region into two or third regions of random size, if the total number of regions after the splits wouldn't exceed the user-specified maximum number of regions. In this way, DAMON provides its best-effort quality and minimal overhead while keeping the overhead bound. Signed-off-by: SeongJae Park Reviewed-by: Leonard Foerster --- include/linux/damon.h | 7 +- mm/damon.c | 181 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 182 insertions(+), 6 deletions(-) diff --git a/include/linux/damon.h b/include/linux/damon.h index 649fb8b6209f..9588bc162c3a 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -65,6 +65,7 @@ struct damon_task { * @sample_interval: The time between access samplings. * @aggr_interval: The time between monitor results aggregations. * @min_nr_regions: The number of initial monitoring regions. + * @max_nr_regions: The maximum number of monitoring regions. * * For each @sample_interval, DAMON checks whether each region is accessed or * not. It aggregates and keeps the access information (number of accesses to @@ -115,6 +116,7 @@ struct damon_ctx { unsigned long sample_interval; unsigned long aggr_interval; unsigned long min_nr_regions; + unsigned long max_nr_regions; struct timespec64 last_aggregation; @@ -133,8 +135,9 @@ struct damon_ctx { }; int damon_set_pids(struct damon_ctx *ctx, int *pids, ssize_t nr_pids); -int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int, - unsigned long aggr_int, unsigned long min_nr_reg); +int damon_set_attrs(struct damon_ctx *ctx, + unsigned long sample_int, unsigned long aggr_int, + unsigned long min_nr_reg, unsigned long max_nr_reg); int damon_start(struct damon_ctx *ctx); int damon_stop(struct damon_ctx *ctx); diff --git a/mm/damon.c b/mm/damon.c index aba02c652b51..5cf39b3ad222 100644 --- a/mm/damon.c +++ b/mm/damon.c @@ -176,6 +176,26 @@ static unsigned int nr_damon_regions(struct damon_task *t) return nr_regions; } +/* Returns the size upper limit for each monitoring region */ +static unsigned long damon_region_sz_limit(struct damon_ctx *ctx) +{ + struct damon_task *t; + struct damon_region *r; + unsigned long sz = 0; + + damon_for_each_task(t, ctx) { + damon_for_each_region(r, t) + sz += r->ar.end - r->ar.start; + } + + if (ctx->min_nr_regions) + sz /= ctx->min_nr_regions; + if (sz < MIN_REGION) + sz = MIN_REGION; + + return sz; +} + /* * Functions for DAMON core logics and features */ @@ -226,6 +246,145 @@ static void kdamond_reset_aggregated(struct damon_ctx *c) } } +#define sz_damon_region(r) (r->ar.end - r->ar.start) + +/* + * Merge two adjacent regions into one region + */ +static void damon_merge_two_regions(struct damon_region *l, + struct damon_region *r) +{ + l->nr_accesses = (l->nr_accesses * sz_damon_region(l) + + r->nr_accesses * sz_damon_region(r)) / + (sz_damon_region(l) + sz_damon_region(r)); + l->ar.end = r->ar.end; + damon_destroy_region(r); +} + +#define diff_of(a, b) (a > b ? a - b : b - a) + +/* + * Merge adjacent regions having similar access frequencies + * + * t task affected by merge operation + * thres '->nr_accesses' diff threshold for the merge + * sz_limit size upper limit of each region + */ +static void damon_merge_regions_of(struct damon_task *t, unsigned int thres, + unsigned long sz_limit) +{ + struct damon_region *r, *prev = NULL, *next; + + damon_for_each_region_safe(r, next, t) { + if (prev && prev->ar.end == r->ar.start && + diff_of(prev->nr_accesses, r->nr_accesses) <= thres && + sz_damon_region(prev) + sz_damon_region(r) <= sz_limit) + damon_merge_two_regions(prev, r); + else + prev = r; + } +} + +/* + * Merge adjacent regions having similar access frequencies + * + * threshold '->nr_accesses' diff threshold for the merge + * sz_limit size upper limit of each region + * + * This function merges monitoring target regions which are adjacent and their + * access frequencies are similar. This is for minimizing the monitoring + * overhead under the dynamically changeable access pattern. If a merge was + * unnecessarily made, later 'kdamond_split_regions()' will revert it. + */ +static void kdamond_merge_regions(struct damon_ctx *c, unsigned int threshold, + unsigned long sz_limit) +{ + struct damon_task *t; + + damon_for_each_task(t, c) + damon_merge_regions_of(t, threshold, sz_limit); +} + +/* + * Split a region in two + * + * r the region to be split + * sz_r size of the first sub-region that will be made + */ +static void damon_split_region_at(struct damon_ctx *ctx, + struct damon_region *r, unsigned long sz_r) +{ + struct damon_region *new; + + new = damon_new_region(ctx, r->ar.start + sz_r, r->ar.end); + r->ar.end = new->ar.start; + + damon_insert_region(new, r, damon_next_region(r)); +} + +/* Split every region in the given task into 'nr_subs' regions */ +static void damon_split_regions_of(struct damon_ctx *ctx, + struct damon_task *t, int nr_subs) +{ + struct damon_region *r, *next; + unsigned long sz_region, sz_sub = 0; + int i; + + damon_for_each_region_safe(r, next, t) { + sz_region = r->ar.end - r->ar.start; + + for (i = 0; i < nr_subs - 1 && + sz_region > 2 * MIN_REGION; i++) { + /* + * Randomly select size of left sub-region to be at + * least 10 percent and at most 90% of original region + */ + sz_sub = ALIGN_DOWN(damon_rand(1, 10) * + sz_region / 10, MIN_REGION); + /* Do not allow blank region */ + if (sz_sub == 0 || sz_sub >= sz_region) + continue; + + damon_split_region_at(ctx, r, sz_sub); + sz_region = sz_sub; + } + } +} + +/* + * splits every target region into two randomly-sized regions + * + * This function splits every target region into two random-sized regions if + * current total number of the regions is equal or smaller than half of the + * user-specified maximum number of regions. This is for maximizing the + * monitoring accuracy under the dynamically changeable access patterns. If a + * split was unnecessarily made, later 'kdamond_merge_regions()' will revert + * it. + */ +static void kdamond_split_regions(struct damon_ctx *ctx) +{ + struct damon_task *t; + unsigned int nr_regions = 0; + static unsigned int last_nr_regions; + int nr_subregions = 2; + + damon_for_each_task(t, ctx) + nr_regions += nr_damon_regions(t); + + if (nr_regions > ctx->max_nr_regions / 2) + return; + + /* If number of regions is not changed, we are maybe in corner case */ + if (last_nr_regions == nr_regions && + nr_regions < ctx->max_nr_regions / 3) + nr_subregions = 3; + + damon_for_each_task(t, ctx) + damon_split_regions_of(ctx, t, nr_subregions); + + last_nr_regions = nr_regions; +} + /* * Check whether current monitoring should be stopped * @@ -269,10 +428,14 @@ static int kdamond_fn(void *data) struct damon_ctx *ctx = (struct damon_ctx *)data; struct damon_task *t; struct damon_region *r, *next; + unsigned int max_nr_accesses = 0; + unsigned long sz_limit = 0; pr_info("kdamond (%d) starts\n", ctx->kdamond->pid); if (ctx->init_target_regions) ctx->init_target_regions(ctx); + sz_limit = damon_region_sz_limit(ctx); + while (!kdamond_need_stop(ctx)) { if (ctx->prepare_access_checks) ctx->prepare_access_checks(ctx); @@ -282,14 +445,16 @@ static int kdamond_fn(void *data) usleep_range(ctx->sample_interval, ctx->sample_interval + 1); if (ctx->check_accesses) - ctx->check_accesses(ctx); + max_nr_accesses = ctx->check_accesses(ctx); if (kdamond_aggregate_interval_passed(ctx)) { if (ctx->aggregate_cb) ctx->aggregate_cb(ctx); + kdamond_merge_regions(ctx, max_nr_accesses / 10, + sz_limit); kdamond_reset_aggregated(ctx); + kdamond_split_regions(ctx); } - } damon_for_each_task(t, ctx) { damon_for_each_region_safe(r, next, t) @@ -399,24 +564,32 @@ int damon_set_pids(struct damon_ctx *ctx, int *pids, ssize_t nr_pids) * @sample_int: time interval between samplings * @aggr_int: time interval between aggregations * @min_nr_reg: minimal number of regions + * @max_nr_reg: maximum number of regions * * This function should not be called while the kdamond is running. * Every time interval is in micro-seconds. * * Return: 0 on success, negative error code otherwise. */ -int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int, - unsigned long aggr_int, unsigned long min_nr_reg) +int damon_set_attrs(struct damon_ctx *ctx, + unsigned long sample_int, unsigned long aggr_int, + unsigned long min_nr_reg, unsigned long max_nr_reg) { if (min_nr_reg < 3) { pr_err("min_nr_regions (%lu) must be at least 3\n", min_nr_reg); return -EINVAL; } + if (min_nr_reg > max_nr_reg) { + pr_err("invalid nr_regions. min (%lu) > max (%lu)\n", + min_nr_reg, max_nr_reg); + return -EINVAL; + } ctx->sample_interval = sample_int; ctx->aggr_interval = aggr_int; ctx->min_nr_regions = min_nr_reg; + ctx->max_nr_regions = max_nr_reg; return 0; }