From patchwork Sun May 12 19:36:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 13662845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6FCEC25B5F for ; Sun, 12 May 2024 19:37:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D70386B01F5; Sun, 12 May 2024 15:37:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CF74D6B01F7; Sun, 12 May 2024 15:37:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD4156B01F8; Sun, 12 May 2024 15:37:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 909466B01F5 for ; Sun, 12 May 2024 15:37:10 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 43F27C0A91 for ; Sun, 12 May 2024 19:37:10 +0000 (UTC) X-FDA: 82110752220.05.E8C1AFF Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf28.hostedemail.com (Postfix) with ESMTP id 09EB6C0013 for ; Sun, 12 May 2024 19:37:07 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=HgPPQHOu; spf=pass (imf28.hostedemail.com: domain of sj@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715542628; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RMZ78f86iXATK3e9FW2bU7qfqiML1ZuX822lcGBmhTg=; b=oPuoZcTILfFbzs4zPV2XEc6ISMYxfrBv6XPAvS4N0/NiaGRTeTR9iOEfkeoqpj3N6VKIEc h67sNCCiI3dIElp6TTopEcC0Xk1gtdIrWvtj+wZPXkhN1VyzIr+HhFtt8GlXWjs0mWEntl qW00mlbjxZTHAmCgPfVTV3OHukU2RCY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715542628; a=rsa-sha256; cv=none; b=jMmaDo8lG7atYh2EzuNBJfKg0zTf84Kg/5lQIZWSah1Z8fZA4hRNC5at7kpAMZNB+sr721 yDX7DRGGz/9BNdS+Wx82ugvKifp1ZZwW3J7CIuQTs/hrG2kYVGY5+qbdIJOiDQGHcEjq1M z3rPtwdfPk289soSY1HxNzIkGpuc8h4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=HgPPQHOu; spf=pass (imf28.hostedemail.com: domain of sj@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 2D21CCE0B2F for ; Sun, 12 May 2024 19:37:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 82D63C32781; Sun, 12 May 2024 19:37:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715542624; bh=t7+2aKGmVybNoTsNs3UjmFvYLVfl4dPfhLTblMrn/KY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HgPPQHOuTXT6FE4SuvaFzlVnrDo3qPzoUsiD7CoL2rElMRvR+osrfWPN6jSENu3/N S/FQHQk2Rr+NjPrG3a6KPW/EOJJlpItNzFgLnfdOyt8YPV8zlLA7YEME7w8J29SvOd xtAl3JLWQp9TD3ZmaoFxgS30+sh780Kf4SBcjCCsmEyI03JuMEjE2ScLQ6q4X0Kn/D ned+SNZshIE9bsE+wZ7PY3mJz/1uEfojxUiHQj1hKwNpmei3xixA7O3lrv7E0mmM3H 4fLoi2EPD8RswWtisl8TDrhMmAcLhdKPIKZdr/0z/0Mk22fJ03z8TLbbB4TfGMa/pM jxFu0+NhhMvkQ== From: SeongJae Park To: Cc: SeongJae Park , damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC IDEA v2 5/6] mm/damon/acma: implement scale up feature Date: Sun, 12 May 2024 12:36:56 -0700 Message-Id: <20240512193657.79298-6-sj@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240512193657.79298-1-sj@kernel.org> References: <20240512193657.79298-1-sj@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: fs9wraza4wazkrezuhyf85z44a7agwdp X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 09EB6C0013 X-HE-Tag: 1715542627-202674 X-HE-Meta: U2FsdGVkX18ml1cWVOEFmZtUTiQJhLh0I/Ym1g3T9z2un9EgokgYu8AjwytGdFXXC45A/2VmIetdJn5ef9CxO1L4nTYGmE5oVpmDZDXeLQ05T0jsm75W8j70qps248P3nUuPvUPmldaUThJ9BYH+J2DpAHjI/2z764MImp8v3yD1MVrhkUKiuIk9nG7p/cRFFe8OB6Z0aAo7LVW/CeBevfAJREOiD4yzMM30IZI7gTEHVzO4JQroZnF/WwnJ60oreJHp4mpjJXKVF/0AwyZSFut/B4HuyLCAAGcEhI3c0XfBV65BM4PKMHfvlRh/0fKkBx9eDbpqmj3rAcvp69cRcev0MollRL3peJl0cPq4IbQfE2GfutWQA8e3t1b6R/dQw+kKpcZsbYCDQunV5PxXtXJjduICI4RqHzMQzhgd5M0w2eZrwlG2Qru4IpHvT3ZMXp2t5d454SgtNg5Imv7cbxb+4x81IVgfq/iP7mhITmvH77ZPErdJYjOlt3hR1S+qtA40h4wKMqNHuSu5IXNPQpPG2Zsd3zSZ/WFkiv/O9garemWSlfxASa4HIVKS2wqKx5Zlzd6P5ojVeIvEhKLWUUujl4TJtJjNEpS/WOHtcySlusj8xYMcRu5ckgoX90pMswQug7REskYvh4e9yTEREmtJx4HzEsdQPe+QKrP2v8fRUuWj2t+vXjTnC0L2Pgb5/uZgHWF6AcqoXkXLliiDjOShHjHWcETC7+LE/x+ydI55ga4bIUOEXbxc1EVmaGzSqkl92tlT2Y6FAOAurtMq2vW3V2PttZ/mVuMdPGNBbbxBX9koJD+h1JbxopsofrnAWK51f9+dZHTMd80Tw6eOV8Dgo1V0yaQkU/y/g1KPYL1iqcR1yBn7ReBJa4iLPJdFoL2V3FC0xwr9ceLv530Ej+q7pvubufY3EpoTuRmL4L1tok1eJjrmPQH+k76SbDbq8pURoIxwiMWv1TFCAO8 hMQeUKVt Yq0M57HZ3mzD+E9W3UWKhax/pk/J8UsQ0ytofgjMAByqIRWXEub6A13PtYF+IUgdzFplj3S78szNuVsQoAk27VjMr1Wmkc0CppslTlSI4HJHSyYhjxFs8Ny1fNaepUFu/l9QU2IyFmojPisOT49NIOcjBPTUygHsqsa7rZGa+JI3VK74FvHyd/6EeI9FokWLh1Zh2M4q+Ld/JEJETSoMFUZ1bm43xS5kwBBqeE+Spi8JqzZE3iAaxY8aCmVA3gfdLxEmsriS5rNPx/6qlCrmlbqIgygbmZ3fsEOjI6OtZP5LAevr+G+bGt1UV0uamGHP7LjgnYJXcZcigiAozC3C1g9WkbQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement the memory scale up feature of Access/Contiguity-aware Memory Auto-scaling module. It runs DAMOS_FREE action scheme with user-acceptable level of memory pressure stall as its target, in 512 contig pages base granularity. For minimizing DAMON-internal DAMOS_ALLOC-ed region management overhead, the scheme is applied to only 128 MiB not-completely-allocated contiguous memory region of lowest address. Note that it does nothing with free_callback() since the host will notify in-guest use of the page via page fault and return it on demand. Signed-off-by: SeongJae Park --- mm/damon/acma.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 90 insertions(+), 4 deletions(-) diff --git a/mm/damon/acma.c b/mm/damon/acma.c index b093b90471dd..32827cbf2fa7 100644 --- a/mm/damon/acma.c +++ b/mm/damon/acma.c @@ -54,6 +54,13 @@ module_param(commit_inputs, bool, 0600); static unsigned long min_mem_kb __read_mostly; module_param(min_mem, ulong, 0600); +/* + * Maximum amount of memory to be guaranteed to the system. In other words, + * the upper limit of the scaling. + */ +static unsigned long max_mem_kb __read_mostly; +module_param(max_mem, ulong, 0600); + /* * Desired level of memory pressure-stall time in microseconds. * @@ -144,6 +151,11 @@ DEFINE_DAMON_MODULES_DAMOS_STATS_PARAMS(damon_acma_scale_down_stat, acma_scale_down_tried_regions, acma_scale_down_succ_regions, acma_scale_down_quota_exceeds); +static struct damos_stat damon_acma_scale_up_stat; +DEFINE_DAMON_MODULES_DAMOS_STATS_PARAMS(damon_acma_scale_up_stat, + acma_scale_up_tried_regions, acma_scale_up_succ_regions, + acma_scale_up_quota_exceeds); + static struct damos_access_pattern damon_acma_stub_pattern = { /* Find regions having PAGE_SIZE or larger size */ .min_sz_region = PAGE_SIZE, @@ -164,8 +176,8 @@ static struct damos *damon_acma_new_scheme( { struct damos_quota quota = damon_acma_quota; - /* Use 1/2 of total quota for hot/cold pages sorting */ - quota.ms = quota.ms / 2; + /* Use 1/3 of total quota for hot/cold pages sorting */ + quota.ms = quota.ms / 3; return damon_new_scheme( pattern, @@ -246,6 +258,36 @@ static int damon_acma_set_scale_down_region_filter(struct damos *scheme) return 0; } +/* + * Similar to damon_acma_set_scale_down_region_filter() but for scaling up. + */ +static int damon_acma_set_scale_up_region_filter(struct damos *scheme) +{ + struct damos_filter *filter = damos_new_filter( + DAMOS_FILTER_TYPE_ADDR, false); + unsigned long start; + unsigned long start_limit, end_limit; + + if (!filter) + return -ENOMEM; + + /* scale up no above max_mem_kb */ + start_limit = monitor_region_start; + end_limit = start_limit + max_mem_kb * KB; + + /* not-completely-free-ed SCALE_WINDOW region of lowest address */ + for (start = start_limit; start <= end_limit - SCALE_WINDOW; + start += SCALE_WINDOW) { + if (damon_alloced_bytes(end - SCALE_WINDOW)) + break; + } + filter->addr_range.start = start; + filter->addr_range.end = min(end_limit, start + SCALE_WINDOW); + + damos_add_filter(scheme, filter); + return 0; +} + /* * Called back from DAMOS for every damos->alloc_order contig pages that * just successfully DAMOS_ALLOC-ed. @@ -307,11 +349,40 @@ static struct damos *damon_acma_new_scale_down_scheme(struct damos *old) return scheme; } +/* + * Scale up scheme + */ +static struct damos *damon_acma_new_scale_up_scheme(void) +{ + struct damos_access_pattern pattern = damon_acma_stub_pattern; + struct damos *scheme; + int err; + + scheme = damon_acma_new_scheme(&pattern, DAMOS_FREE); + if (!scheme) + return NULL; + err = damon_acma_set_scheme_quota(scheme, old, + DAMOS_QUOTA_SOME_MEM_PUSI_US); + if (err) { + damon_destroy_scheme(scheme); + return NULL; + } + scheme->alloc_order = scale_pg_order; + scheme->alloc_callback = NULL; + err = damon_acma_set_scale_up_region_filter(scale_up_scheme); + if (err) { + damon_destroy_scheme(scale_down_scheme); + return NULL; + } + return scheme; +} + static int damon_acma_apply_parameters(void) { struct damos *scheme, *reclaim_scheme; - struct damos *scale_down_scheme; + struct damos *scale_down_scheme, *scale_up_scheme; struct damos *old_reclaim_scheme = NULL, *old_scale_down_scheme = NULL; + struct damos *old_scale_up_scheme = NULL; struct damos_quota_goal *goal; int err = 0; @@ -324,7 +395,11 @@ static int damon_acma_apply_parameters(void) old_reclaim_scheme = scheme; continue; } - old_scale_down_scheme = scheme; + if (!old_scale_down_scheme) { + old_scale_down_scheme = scheme; + continue; + } + old_scale_up_scheme = scheme; } reclaim_scheme = damon_acma_new_reclaim_scheme(old_reclaim_scheme); @@ -340,6 +415,14 @@ static int damon_acma_apply_parameters(void) } damon_add_scheme(ctx, scale_down_scheme); + scale_up_scheme = damon_acma_new_scale_up_scheme(old_scale_up_scheme); + if (!scale_up_scheme) { + damon_destroy_scheme(scale_down_scheme); + damon_destroy_scheme(reclaim_scheme); + return -ENOMEM; + } + damon_add_scheme(ctx, scale_up_scheme); + return damon_set_region_biggest_system_ram_default(target, &monitor_region_start, &monitor_region_end); @@ -428,6 +511,9 @@ static int damon_acma_after_aggregation(struct damon_ctx *c) case DAMOS_ALLOC: damon_acma_scale_down_stat = s->stat; break; + case DAMOS_FREE: + damon_acma_scale_up_stat = s->stat; + break; default: break; }