From patchwork Sun May 12 19:36:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 13662840 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A998C25B77 for ; Sun, 12 May 2024 19:37:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B5C156B01F0; Sun, 12 May 2024 15:37:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A8B6B6B01F3; Sun, 12 May 2024 15:37:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 862CF6B01F1; Sun, 12 May 2024 15:37:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 594DA6B01B3 for ; Sun, 12 May 2024 15:37:07 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id EB6EC1C0583 for ; Sun, 12 May 2024 19:37:06 +0000 (UTC) X-FDA: 82110752052.01.C6BC937 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf22.hostedemail.com (Postfix) with ESMTP id 58A7BC000E for ; Sun, 12 May 2024 19:37:05 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=TnT+ejCW; spf=pass (imf22.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715542625; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g+AjUyZNCIDu9N3jTwSpZb1GlERAOdp6NP0QfDvG0AU=; b=ff55A8NJgLoGPNl61hKhccHSTJ8Md78iBRQIOAoDD5aybwXfRPD9VYd6z4LkyJN1yxWN/i HBm/ozYz4HS2Zqs+OQ/UJ7thDQs5TLdrRNE4BREuXBcFhi/ZsIGDuzPB8vgIE71VCMqlH2 WoBkxMlsD1odYPo02eyuIHJoX4lcpd4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715542625; a=rsa-sha256; cv=none; b=ouyarR3QXzo7Q5X0dNqPCjwdWpfkU1lZLXoPtURuxni6M65KusJoBwdHkys56+WWGIo1SH /LB1AOL9P0r1Ovizn8hHbCzWq9fMjRDj7nd6tkNEpiHA7vd9fMgZou1usHNnlwHabjZI+7 A1W9/eIMamHz8FPUxVFs9j56eOM+A3Y= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=TnT+ejCW; spf=pass (imf22.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 7C42560B99 for ; Sun, 12 May 2024 19:37:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 08855C4AF0B; Sun, 12 May 2024 19:37:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715542624; bh=tWKq7K6XmSUSldcREZ1dFiZ70aiSWF0Pc/jUVnK6YyM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TnT+ejCW/rIvYMQFgfYDh/RgVbKdYfxAURsgsZMDmHalrJ0Yth5mnC1e+08nyM6hf 0fEbQnHb7Kv6iXI/YIiPXkXlIuTvt+Wh0XgYG6uQyLUl8kv4XAV6voDMIIUDOlM8+t tz+cjrWmCiWVJin0IHHRkzBTx3ih/uBG8HEgYtsy4odEvm47bgDSbAWdNF8+hVsT5Y fgFLwwlbWYnbw5PhmnH+lSrTlZAGkYz80u0gsp5i1Ks0fqvvtJj4QP8HKMuS/qsdAy gqP6loR91ecl68XNi7JsjXq4sqW7hqLY/pU3z2qnCiYIdhq1Hex8UKCKLBWsyt2Pw2 mFszbSNsaWI1Q== From: SeongJae Park To: Cc: SeongJae Park , damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC IDEA v2 4/6] mm/damon/acma: implement scale down feature Date: Sun, 12 May 2024 12:36:55 -0700 Message-Id: <20240512193657.79298-5-sj@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240512193657.79298-1-sj@kernel.org> References: <20240512193657.79298-1-sj@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: hoqji567e1e4sjuzcukykzpyndn1mnw7 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 58A7BC000E X-HE-Tag: 1715542625-873658 X-HE-Meta: U2FsdGVkX1/65lfwF0XXHq3by4APmbBcRCT6o0V0pNkGMxWyPgYcA3mPmn89kNzS5uw/NpSzAX9WA037WdKgdLupRSS8ubEa1Ju89KU1zieycG3Dmqw5zlIGhJ4xBbf+NxO2XXw53oRlUjoelhUiC8hX6p8mZ2XZIbOatfzzzMizEZI3Do+jny3qCCsB+qaC8V06Y4//Sw18YDKKHKMOglPH9CaWLzl/ZIR7s4IKcZJLkIJpUfV5Esv7G2ezerpPAyzgOka+5afG4CIJA6M5EvG1H6JV2K/lzyqtneQSyy0VOOyFEWl+I0wUCHpNwyU/PavTTQwqPH1e1U5D6lbQ9sfqKMET+QTRv5P44v35GQFTv5DvoEkqOY8UvfVZFRkXeYFfmy8/fFcTB+PqpKppIywnxRO2bYGGJulFfpWuLphyE9QA2RIcnJguB4howun889HXzbj12VKD4RpHhtWhxWwOQjNQN4ELTmmRQPivBtSc8az16JjMGz/gCBDpipeyY55X3OsibWV5RU/OFcrEtJ/Gdcs/JJTeqhFl7Dowh/WQab29NpKWZZE7NMNSK9W2bsVCLJyn01o0kZnjsrJzcQnYgSlTMd4vjsaCOcZu4iNmED5S6R9tSDdhv8juyUbtOYHtKqisr8sAGL/C3RFhes0VS133+F7E+1aWJ5jmemx00ol/dXmsfT5U9dTz93acu4ngzXR9goEohk1raSt5A7iMISnOD1/kAUogYdbiy/kwsgXLefyBRgTuWcRU+mhzxpkGJcxj/KTpDkZl1wrCBUy7Sui6ShATPXX+PpRhqL5u8cWOZk8LSkWSMFlD1kRjCFJhhABfnamFysyBR3rPuR5Ai59EhLWrbzzSBdNyDMCL/mkUdGPgjB8sR2Kb0xoLXMCIM9ptZfjLnPyS+AP5KhXMNRFzQymo9OBdT19L4N1y+4meAI95Ifr6LWlbgtTeSvRyIp5ngpH6xtLYv3H oL0cEprS 20Z/nwMOfEx0jQM1dLZ42LhLTXWzlXSvp/TXIn4wTJPcep08DWBU+S/1ZDMRSS6aY24bUmElYQwF3rinhRgcOZycmDyQas862oSSNCtdcBco0eukVy2BZHn+DipAhpzEnbssNuF6gKzI622pe8/L8uZmH5WVQk/SSiH/x4W2lbNTkeS+85lJXjkFzxwUXEtB4kkbShBFxGUENbFj5hbldnfquPAefcm49/ea/0AoDhx27sDsyWxaITlL/rB3SMTawBMtyPCPW7DvW8+CLFNSzbSmosDTey9dwTZwUw6Ex9HHg5FfgAqRpBbJA9R57luf93GWwSfH35NRlMJw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement the memory scale down feature of Access/Contiguity-aware Memory Auto-scaling module. It runs DMAOS_ALLOC action scheme with user-acceptable level of memory pressure stall as its target, in 512 contig pages base granularity, and report the 512 contig successfully DAMOS_ALLOC-ed regions to the host for reuse. For minimizing DAMON-internal DAMOS_ALLOC-ed region management overhead, the scheme is applied to only 128 MiB not-completely-allocated contiguous memory region of highest address. Signed-off-by: SeongJae Park --- mm/damon/acma.c | 137 +++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 131 insertions(+), 6 deletions(-) diff --git a/mm/damon/acma.c b/mm/damon/acma.c index 276b61fd4e26..b093b90471dd 100644 --- a/mm/damon/acma.c +++ b/mm/damon/acma.c @@ -8,9 +8,6 @@ * and report it to the host when the system is having memory pressure level * under the threshold, and vice versa, respectively. * - * At this moment, the scaling is not implemented, hence this is just a memory - * pressure-aware proactive reclamation module. - * * Author: SeongJae Park */ @@ -50,6 +47,13 @@ static bool enabled __read_mostly; static bool commit_inputs __read_mostly; module_param(commit_inputs, bool, 0600); +/* + * Minimum amount of memory to be guaranteed to the system. In other words, + * the lower limit of the scaling. + */ +static unsigned long min_mem_kb __read_mostly; +module_param(min_mem, ulong, 0600); + /* * Desired level of memory pressure-stall time in microseconds. * @@ -66,6 +70,18 @@ module_param(commit_inputs, bool, 0600); static unsigned long quota_mem_pressure_us __read_mostly = 1000; module_param(quota_mem_pressure_us, ulong, 0600); +/* + * Basic scale down/up granularity. ACMA will allocate and report contiguous + * pages of this size at once. 512 pages (2 MiB for 4 KiB page setup) by + * default. + * + * To minimize DAMON-internal ALLOC-ed memory management overhead, we further + * apply SCALE_WINDOW. Refer to damon_acma_set_scale_down_region_filter() for + * more detail about it. + */ +static unsigned int scale_pg_order __read_mostly = 9; +module_param(scale_pg_order, uint, 0600); + static struct damos_quota damon_acma_quota = { /* Use up to 15 ms per 1 sec for scaling, by default */ .ms = 15, @@ -123,6 +139,11 @@ DEFINE_DAMON_MODULES_DAMOS_STATS_PARAMS(damon_acma_reclaim_stat, acma_reclaim_tried_regions, acma_reclaim_succ_regions, acma_reclaim_quota_exceeds); +static struct damos_stat damon_acma_scale_down_stat; +DEFINE_DAMON_MODULES_DAMOS_STATS_PARAMS(damon_acma_scale_down_stat, + acma_scale_down_tried_regions, acma_scale_down_succ_regions, + acma_scale_down_quota_exceeds); + static struct damos_access_pattern damon_acma_stub_pattern = { /* Find regions having PAGE_SIZE or larger size */ .min_sz_region = PAGE_SIZE, @@ -143,6 +164,9 @@ static struct damos *damon_acma_new_scheme( { struct damos_quota quota = damon_acma_quota; + /* Use 1/2 of total quota for hot/cold pages sorting */ + quota.ms = quota.ms / 2; + return damon_new_scheme( pattern, action, @@ -177,6 +201,61 @@ static int damon_acma_set_scheme_quota(struct damos *scheme, struct damos *old, return 0; } +/* + * scale_pg_order is for basic scaling granularity. Have a larger granularity + * to limit DAMON-internal alloc-ed pages management overhead. + */ +#define SCALE_WINDOW (128 * MB) + +/* + * Set scale_down scheme's address range type filter to apply scaling down to + * only current scaling window. Scaling window is SCALE_WINDOW size contiguous + * memory region of highest address that not yet completely DAMOS_ALLOC-ed and + * reported. + * + * TODO: Apply 'struct page' reduction in SCALE_WINDOW or lower granularity. + * E.g., hot-unplug the memory block, or apply vmemmap remapping-based approach + * like hugetlb vmemmap optimization + * (https://docs.kernel.org/mm/vmemmap_dedup.html). + */ +static int damon_acma_set_scale_down_region_filter(struct damos *scheme) +{ + struct damos_filter *filter = damos_new_filter( + DAMOS_FILTER_TYPE_ADDR, false); + unsigned long end; + unsigned long start_limit, end_limit; + + if (!filter) + return -ENOMEM; + + /* scale down no below min_mem_kb */ + end_limit = monitor_region_end; + start_limit = monitor_region_start + min_mem_kb * KB; + + /* not-completely-alloc-ed SCALE_WINDOW region of highest address */ + for (end = end_limit; end >= start_limit + SCALE_WINDOW; + end -= SCALE_WINDOW) { + if (damon_alloced_bytes(end, end - SCALE_WINDOW) + != SCALE_WINDOW) + break; + } + filter->addr_range.start = max(start_limit, end - SCALE_WINDOW); + filter->addr_range.end = end; + + damos_add_filter(scheme, filter); + return 0; +} + +/* + * Called back from DAMOS for every damos->alloc_order contig pages that + * just successfully DAMOS_ALLOC-ed. + */ +static int damon_acma_alloc_callback(unsigned long start_addr) +{ + /* For non-zero return value, DAMOS free the pages. */ + return page_report(PHYS_PFN(addr), 1 << scale_pg_order); +} + /* * Reclaim cold pages on entire physical address space */ @@ -199,10 +278,40 @@ static struct damos *damon_acma_new_reclaim_scheme(struct damos *old) return scheme; } +/* + * Scale down scheme + */ +static struct damos *damon_acma_new_scale_down_scheme(struct damos *old) +{ + struct damos_access_pattern pattern = damon_acma_stub_pattern; + struct damos *scheme; + int err; + + scheme = damon_acma_new_scheme(&pattern, DAMOS_ALLOC); + if (!scheme) + return NULL; + err = damon_acma_set_scheme_quota(scheme, old, + DAMOS_QUOTA_SOME_MEM_PSI_US); + if (err) { + damon_destroy_scheme(scheme); + return NULL; + } + /* alloc in 512 pages granularity */ + scheme->alloc_order = scale_pg_order; + scheme->alloc_callback = damon_acma_alloc_callback; + err = damon_acma_set_scale_down_region_filter(scale_down_scheme); + if (err) { + damon_destroy_scheme(scheme); + return NULL; + } + return scheme; +} + static int damon_acma_apply_parameters(void) { struct damos *scheme, *reclaim_scheme; - struct damos *old_reclaim_scheme = NULL; + struct damos *scale_down_scheme; + struct damos *old_reclaim_scheme = NULL, *old_scale_down_scheme = NULL; struct damos_quota_goal *goal; int err = 0; @@ -210,14 +319,27 @@ static int damon_acma_apply_parameters(void) if (err) return err; - damon_for_each_scheme(scheme, ctx) - old_reclaim_scheme = scheme; + damon_for_each_scheme(scheme, ctx) { + if (!old_reclaim_scheme) { + old_reclaim_scheme = scheme; + continue; + } + old_scale_down_scheme = scheme; + } reclaim_scheme = damon_acma_new_reclaim_scheme(old_reclaim_scheme); if (!reclaim_scheme) return -ENOMEM; damon_set_schemes(ctx, &reclaim_scheme, 1); + scale_down_scheme = damon_acma_new_scale_down_scheme( + old_scale_down_scheme); + if (!scale_down_scheme) { + damon_destroy_scheme(reclaim_scheme); + return -ENOMEM; + } + damon_add_scheme(ctx, scale_down_scheme); + return damon_set_region_biggest_system_ram_default(target, &monitor_region_start, &monitor_region_end); @@ -303,6 +425,9 @@ static int damon_acma_after_aggregation(struct damon_ctx *c) case DAMOS_LRU_RECLAIM: damon_acma_reclaim_stat = s->stat; break; + case DAMOS_ALLOC: + damon_acma_scale_down_stat = s->stat; + break; default: break; }