From patchwork Sun May 12 19:36:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 13662839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AC40C25B4F for ; Sun, 12 May 2024 19:37:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 974106B01B3; Sun, 12 May 2024 15:37:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8FF436B01F0; Sun, 12 May 2024 15:37:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C6BD6B01F2; Sun, 12 May 2024 15:37:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5A1F16B01F0 for ; Sun, 12 May 2024 15:37:07 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D94AE1206DE for ; Sun, 12 May 2024 19:37:06 +0000 (UTC) X-FDA: 82110752052.06.6F19470 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf08.hostedemail.com (Postfix) with ESMTP id 318C6160009 for ; Sun, 12 May 2024 19:37:04 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=cOw5DjkQ; spf=pass (imf08.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715542624; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3Q6gqoJXRhcCdmSXjVP58FIntI93Kua5+37iE3msd6E=; b=ddGpl6zFSdUVmGnKgvSLPQmqOgnjns5cea3F3VTZNxBilSx2krz1H+XjND5E4Hk3CjLgCG W9gcnbThWKbZGD61A3CxHafaDRQ+Jj4K3POPuDAFGJM1WH9XPGSaN6J8UD9Ya+fayygvG5 wkDuOYkSJpRuetc2L0SCApiw4tzRC/A= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=cOw5DjkQ; spf=pass (imf08.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715542624; a=rsa-sha256; cv=none; b=dmEW97Goe8AZwULQwo1kC5m1Kz9Mt5pNi/lXVRv7tJ4OYYgjFrAAEm0Iv3c/45ZUkQRt8T IoSx404kY147KEP5MatYSvW9yRgRIKxdXTHloSaThoY1MxSPORtC0ohRM+wRII+loalNPU UZ05opCcPldJqwHwoKV6GL2n1Zn04Fg= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 1A21F60A72 for ; Sun, 12 May 2024 19:37:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 748F6C32781; Sun, 12 May 2024 19:37:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715542622; bh=CqpUnQlsaA0hxF2A02w+ec82+zmgruebmlsoPX119nQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cOw5DjkQimingooso9+sMyhUkYjNrrGwHTOQzKKRi9slOdfRzmBV4kv15dQtm9n+L ntn0NyHd/3Odq6AAio3VWqBxu+XxFZM+zH8EDolJr5ljMQlge3osuqPtAaF0XA7m+V kNQV8xGZQtIa1BzslABKrJ/U6jmURBT4kDcbAxhcWpTU0WX9SFeoe6E+8VObLOqr3L cOENEqMwjc1kHFXboPh1TdciQO2gNDieBnTXvMH8uBZAA8fQq6Ie4RpWuuYWWTkQnZ SGvWbLMe2qfIvfoTqfNaKvQrZQgsShfh42TmHSqpdsfptCIwQoXlx+9plOBnkZjnkP 9/QIdYHVe3Mpg== From: SeongJae Park To: Cc: SeongJae Park , damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC IDEA v2 1/6] mm/damon: implement DAMOS actions for access-aware contiguous memory allocation Date: Sun, 12 May 2024 12:36:52 -0700 Message-Id: <20240512193657.79298-2-sj@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240512193657.79298-1-sj@kernel.org> References: <20240512193657.79298-1-sj@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 318C6160009 X-Stat-Signature: tqxq61cmp1twxq9j53wb3anu1oefaorn X-HE-Tag: 1715542624-729051 X-HE-Meta: U2FsdGVkX1+RRoySiWFuEviw/X3kzelt1xplbEo5DZqqH/b1xab0yWpkw66LQK1hthjjfJHtx8Q9FL507wnhQ2FFjgBdCPJAk6LNF/csLmA1DyrbeQVK6uJq/NaZXoaG+9o2jRS2N/nC3cakC+vlGF/J7pK6U8VRMCvvUfz8+P7nFZscqGFzfzYt+BH40kRHonIwBrp+Vt4fnBd3XPoNO5zYA7lHWKLZu8vX+pKTilCutbygxea3IxuYKjUFzvBqK2RR9RinmB11nZs1GwDtAll0osPj8FKRkD5zUvZSuB6WzS+HC4mAck9/ep1Jkkgcq9vdAvndOnwjc/mr/UwBXyD8p7lFyy2vBtkAgdkcKrkRa0uDFUA6qWuMdtMHR17pEH8Q4/K2bhwwbGROu79JO5pEac1ZxZrUlXuvzvmW7kjsIFZiC2VdNJ29woh2PDekRP+alQKVqCikXuao52d3d0Y7vyHCtuf02NEuNvhYmBNW1xoXzTws7v9AAIqqrqgR6jHwqL2Yrp0/YjzSOsRrtRlHrk8R7xfpLu0dhV6M4BWmuB0fdPnznaTiTg1FVXvKWX744Ox/BJ0sB3lV7vfGtNanoHpdRt2o31LkfI31Ok/1dI0v8WjOCDN18xeQATvAAARkXxqgZmNcCyDPp5t/AMdszpeer+cbSmCjONZE/Fo8FwmCh/MhWBoy2XliGQxYEuAYqStY8OejyouCb63RlfFxacwQhfWQEO+7eYSpV3v/PMfDEDipQsB41EF7lBelB07HeHe9sC8u576TznceM+31ZSg3/uyBGSzB9EyUEHAkcIYwN37S/9i0WVGy3WlStDq+kvoF8a/RglDExXEtIAeCorXIf6S+Gqstc+OQ9ygTdpQ1n3O21/YHs14g0tHRwAl5S5Zr5elMmEu/i0+zDkdIrqD2p6aSrFrdiPDVy4lCjAsZ/t5rqxK6+Si1a4bKyR0eIWUO6UW9EovJK+9 54xjyE8f Bb+asun+oshpEtI/U2vdDHmUfswUg65JHc8WWRHMMKtPOSjpwEzX54yF30ZYyjBHsKS0Bs8m+4zjk10hqVVa2ZzQDWicgCzLwjea3KjZq6sQX4IbdFTeLJ7yjNlQaYOwzhw1uxX0r3F+7AmNdKOegix+XyiW/MLuSs+uZ902euo54pRhW0g3loEbn90jIKoTpRu6RwlB61Jxki7wolAN8c55RNrmAB9Ex2pt+BSCS+ROZGq/XO7JLoFMsL+0IRrtxMnleoXafe3g7LQ+y+1wL38pTf0LhCN8Z4qo/tsUKJ9y65+sFDMG9Wn/iPw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement two DAMOS actions, namely DAMOS_ALLOC and DAMOS_FREE. As the name says, the actions allocate/de-allocate given DAMOS target memory in user-defined base granularity. Signed-off-by: SeongJae Park --- include/linux/damon.h | 37 ++++++++++++++++ mm/damon/paddr.c | 93 ++++++++++++++++++++++++++++++++++++++++ mm/damon/sysfs-schemes.c | 4 ++ 3 files changed, 134 insertions(+) diff --git a/include/linux/damon.h b/include/linux/damon.h index 0c3f93374e8d..933bc7777f2d 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -107,6 +107,11 @@ struct damon_target { * @DAMOS_LRU_DEPRIO: Deprioritize the region on its LRU lists. * @DAMOS_MIGRATE_HOT: Migrate the regions prioritizing warmer regions. * @DAMOS_MIGRATE_COLD: Migrate the regions prioritizing colder regions. +#ifdef CONFIG_ACMA + * @DAMOS_ALLOC: Allocate pages in the region, + * &struct damos->alloc_order pages at once. + * @DAMOS_FREE: Return DAMOS_ALLOC-ed pages back to the system. +#endif * @DAMOS_STAT: Do nothing but count the stat. * @NR_DAMOS_ACTIONS: Total number of DAMOS actions * @@ -126,6 +131,10 @@ enum damos_action { DAMOS_LRU_DEPRIO, DAMOS_MIGRATE_HOT, DAMOS_MIGRATE_COLD, +#ifdef CONFIG_ACMA + DAMOS_ALLOC, + DAMOS_FREE, +#endif DAMOS_STAT, /* Do nothing but only record the stat */ NR_DAMOS_ACTIONS, }; @@ -375,6 +384,11 @@ struct damos_access_pattern { * struct damos - Represents a Data Access Monitoring-based Operation Scheme. * @pattern: Access pattern of target regions. * @action: &damo_action to be applied to the target regions. +#ifdef CONFIG_ACMA + * @alloc_order: DAMOS_ALLOC/FREE applying granularity. + * @alloc_callback: DAMOS_ALLOC success callback. + * @free_callback: DAMOS_FREE callback. +#endif * @apply_interval_us: The time between applying the @action. * @quota: Control the aggressiveness of this scheme. * @wmarks: Watermarks for automated (in)activation of this scheme. @@ -388,6 +402,18 @@ struct damos_access_pattern { * CPU time or IO resources for the &action, "a is used. * * If @apply_interval_us is zero, &damon_attrs->aggr_interval is used instead. +#ifdef CONFIG_ACMA + * + * If @action is CONFIG_ALLOC or CONFIG_FREE, the action is applied to + * @alloc_order pages of the region at once. For example, if the region has + * 1024 pages, and @alloc_order is 9, DAMOS tries to allocate or free first 512 + * (2^9) contiguous pages at once, and then next 512 pages. + * + * For each success of such allocation attemp, @alloc_callback is called back. + * For each attempt of deallocation, @free_callback is called back first, + * before trying the deallocation. If @free_callback returns non-zero, the + * deallocation attempt is aborted. +#endif * * To do the work only when needed, schemes can be activated for specific * system situations using &wmarks. If all schemes that registered to the @@ -409,6 +435,11 @@ struct damos_access_pattern { struct damos { struct damos_access_pattern pattern; enum damos_action action; +#ifdef CONFIG_ACMA + unsigned int alloc_order; + int (*alloc_callback)(unsigned long start_addr); + int (*free_callback)(unsigned long start_addr); +#endif unsigned long apply_interval_us; /* private: internal use only */ /* @@ -784,6 +815,12 @@ int damon_stop(struct damon_ctx **ctxs, int nr_ctxs); int damon_set_region_biggest_system_ram_default(struct damon_target *t, unsigned long *start, unsigned long *end); +#ifdef CONFIG_ACMA + +unsigned long damon_alloced_bytes(void); + +#endif + #endif /* CONFIG_DAMON */ #endif /* _DAMON_H */ diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index 81163206e70c..f66bd032c523 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -475,6 +475,93 @@ static unsigned long damon_pa_migrate(struct damon_region *r, struct damos *s) } +#ifdef CONFIG_ACMA + +static bool damon_pa_preempted(unsigned long pfn) +{ + /* todo: implement */ +} + +/* always success for preempted=false */ +static int damon_pa_set_preempted(unsigned long pfn, bool preempted) +{ + /* todo: implement */ +} + +/* + * Return ownership of the memory to the system. At the moment, only user of + * this function is virtio-balloon. They could use page fault-based mechanisms + * to catch returned ownership. Therefore this function doesn't notify this + * event to the report subscribers. In future, we could add some notification + * system of this event for more users such as contig memory allocator. + */ +static int damon_pa_free(unsigned long pfn, struct damos *scheme) +{ + if (!damon_pa_preemted(pfn)) + return -EINVAL; + + free_contig_range(pfn, DAMON_MEM_PREEMPT_PAGES); + damon_pa_set_preempted(pfn, false); + /* + * We intentionally do not report this event to the preempted memory + * report subscriber. They could use page fault handler like + * mechanisms. + */ + return 0; +} + +/* + * Pass ownership of the memory to page reporting subscribers. The subscribers + * can use the reported memory for their purpose, e.g., letting Host + * re-allocate it to other guest, or use as contig allocation memory pool. + */ +static int damon_pa_alloc(unsigned long pfn, struct damos *scheme) +{ + int err; + + if (damon_pa_preempted(pfn)) + return -EINVAL; + if (alloc_contig_range(pfn, pfn + DAMON_MEM_PREEMPT_PAGES, + MIGRATE_MOVABLE, GFP_KERNEL)) + return -ENOMEM; + err = damon_pa_set_preempted(pfn, true); + if (err) { + free_contig_range(pfn, DAMON_MEM_PREEMPT_PAGES); + return err; + } + if (!scheme->alloc_callback) + return 0; + err = scheme->alloc_callback(PFN_PHYS(pfn)); + if (err) { + damon_pa_free(pfn); + return err; + } + return 0; +} + +/* Preempt or yield memory regions from system */ +static unsigned long damon_pa_alloc_or_free( + struct damon_region *r, struct damos *s, bool alloc) +{ + unsigned long pfn; + unsigned long applied = 0; + + for (pfn = PHYS_PFN(r->start); pfn < PHYS_PFN(r->end); + pfn += DAMON_MEM_PREEMPT_PAGES) { + if (alloc) { + if (damon_pa_alloc(pfn, s)) + continue; + } else { + if (damon_pa_free(pfn, s)) + continue; + } + applied += 1; + } + return applied * PAGE_SIZE * DAMON_MEM_PREEMPT_PAGES; +} + +#endif + static unsigned long damon_pa_apply_scheme(struct damon_ctx *ctx, struct damon_target *t, struct damon_region *r, struct damos *scheme) @@ -489,6 +576,12 @@ static unsigned long damon_pa_apply_scheme(struct damon_ctx *ctx, case DAMOS_MIGRATE_HOT: case DAMOS_MIGRATE_COLD: return damon_pa_migrate(r, scheme); +#ifdef CONFIG_ACMA + case DAMOS_ALLOC: + return damon_pa_alloc_or_free(r, scheme, true); + case DAMOS_FREE: + return damon_pa_alloc_or_free(r, scheme, false); +#endif case DAMOS_STAT: break; default: diff --git a/mm/damon/sysfs-schemes.c b/mm/damon/sysfs-schemes.c index 66fccfa776d7..54be4d661881 100644 --- a/mm/damon/sysfs-schemes.c +++ b/mm/damon/sysfs-schemes.c @@ -1460,6 +1460,10 @@ static const char * const damon_sysfs_damos_action_strs[] = { "lru_deprio", "migrate_hot", "migrate_cold", +#ifdef CONFIG_ACMA + "damos_alloc", + "damos_free", +#endif "stat", }; From patchwork Sun May 12 19:36:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 13662843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9805BC25B4F for ; Sun, 12 May 2024 19:37:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 50F1A6B01F4; Sun, 12 May 2024 15:37:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 385356B01F8; Sun, 12 May 2024 15:37:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F41816B01F5; Sun, 12 May 2024 15:37:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CCEF16B01F3 for ; Sun, 12 May 2024 15:37:09 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 803931605F4 for ; Sun, 12 May 2024 19:37:09 +0000 (UTC) X-FDA: 82110752178.10.F20C57F Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf10.hostedemail.com (Postfix) with ESMTP id 38A37C0004 for ; Sun, 12 May 2024 19:37:06 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dHtVn72x; spf=pass (imf10.hostedemail.com: domain of sj@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715542627; a=rsa-sha256; cv=none; b=nLIqT3TyHSBBs5v8dOUCz1ua/YFiDBm20snt0tFIC1Ai3V0LE7oLzvmOaKs1AAW/nYP625 GRdz5dTQfNP2KYxrZkyCYbmgrGQiz7mAjb7nfqeXLqU6p6rlLPSnAZSIjbAuXufj1yttoH NVsdh3JvdgwjmCg6wu7AakvOmu4US/s= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dHtVn72x; spf=pass (imf10.hostedemail.com: domain of sj@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715542627; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3VsJsFTggLmvzlXDXdyYvLznX3mEVqACGaEl38+jCp8=; b=K7BeSCWJbkmJ9JOef4xn3kQAbrYhSutRoI8ZR9zK78QtTl7Rcmbls+h6QxiuB9iMODGK8s jn3u5BWgG/L35vIPGW4ADZn5oiUg8znqN/X109lKGjyjdDP+U50h5p4ttNoCKdM/q5g1r1 /P0RbOiXx05pWfp9hyRkFnYh4QItzAQ= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 1472CCE0B28 for ; Sun, 12 May 2024 19:37:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 041D4C4AF07; Sun, 12 May 2024 19:37:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715542623; bh=eA97GqCkk4n26tmC1Bwl8xISrSNwe8VkWXSlnv6HYfA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dHtVn72xYanh37wq6PWLMmhIrGmLcFVVTyjKYs678JRv84j1w/tfkahirKvFO1q5x KpzfcFr+27uNWoBMN8YdM80ZeBJ5ntYkDpqAErYH7phfT6V590NZUktHP8hsr43shj 11SHgTqjq0kCs7enVwWeRpyBVqFmTOriugipY96/eVvnDk9EZ+Z+6r+lB9pPMTAmtX BN0DAcW7f2aFzZbfqdjP11FAyLFauYphcj627BL3rQ9IXTqplr7+t6vFjcGIXhSeQr YDgGuvsR+uZ3BXoeBY8JWh3eRCppv3gXv1YppMLMiExGwYaA9AqtUUWTyPPrEVZfNP 5A4fMn01yEfdg== From: SeongJae Park To: Cc: SeongJae Park , linux-kernel@vger.kernel.org, damon@lists.linux.dev, linux-mm@kvack.org Subject: [RFC IDEA v2 2/6] mm/damon: add the initial part of access/contiguity-aware memory auto-scaling module Date: Sun, 12 May 2024 12:36:53 -0700 Message-Id: <20240512193657.79298-3-sj@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240512193657.79298-1-sj@kernel.org> References: <20240512193657.79298-1-sj@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 38A37C0004 X-Stat-Signature: jtste499h4ca8zsqxwzfap9yknn9d4p9 X-HE-Tag: 1715542626-946240 X-HE-Meta: U2FsdGVkX1+0BrvFlgNlc5B2IcqrE/s6ZqXzFFD4G0MXSFPZ3hcBBa+k/IPcMF+Dx3RQNgDdSlNxOnko4X+u19jsmEF2q6h5n3PccHXLROhHyaK6NEbr3i0xqdjSN2s2wRKaLg5kpi4wffBewA/BXZiZLEVmeof1wZrNBe4R+Ydl0jB/yE8TdOGO+8UD8+uSkEdMKf2IoRdiJnP9EhAV5BTokLIDna8+GzX6avQI2luQHBAhdkfanUGnJLWyos7dqWqHOyFLMaXstWhvPrtZjkgKaCbclUBrv8DJVwoWLCLPwV58/Wx5OpCmI3QscoalwUCuXWuqA5/z4zqoMtw/P85w+XIjm27mzo5OoL+oU/ry4m4sPYaA7BYPVmhvCM5sRFse0XdT0louDDA33eZmG+8NSf9lZlH+4xFS+/gDYPoDt4Lvgd2f83zHo3DTQLz8X61nzRU/tbs/r1qC0Ey5w+CaG6Si963xARybS8K6EdQsceu5OjKzm0qLib6TvBUc4aDOUSgvdyPICSCteRoPmV8WFWbG8hP09T2RC4W/ql9pHVccD8yHPCRKeVzRn9CZ4+Im3gCmfcIMgqZObl3zKWx9f+S91EFZSVZGvOjTcEMKV7oLvtfaMhQ/4+LxasyyQFLIQTvk51GtO3YWSDPkyTW+ZNszsqfqAw2VYnMXUAGmLnxscgM3nqaTZPP33udplcYGBKDWACJc0apufwVGaqYlL6d2ImnGeyCwCYI5tEkZ5BBIh2zLDgzOnnx6QUX7sIncXqVZ3WoYjw5Tmzom2Pwmy+XCPH+1wdK8+yUbEUPEjHdToszNVPbuWtqGZWWCJ/KuOSBYYWlIQj1Ap4Q8Zq6n2L2uGtgeV73HpAbv/Y1HoSPxEk2Pb81ujb8LVxH+vHbbtXKpQcxr6TjFIsQdorqCk3xsG3Ar/jOEy4KxnH+RZYdqtxcBu0gS11O3+B+jV/+XTLcPprQ/VzTTkCG mI/dq0FJ X3fSLsoSQnLI/e0qprwBIW75/DJwRb/ghDNwLqByTSPWR6AN5ZbDAzRkd2uMePfGFRLOkJGcvdI38XzrhJH15QU5+8k45wiEhmh49Xhhw8rUTZjV5zZz59pIMOHaRFF+lbQ4fuED7hSm1erLUnXKt+0UXGmYhTPtOvmvZh7u3qlXMXgIKEM3E1CVSX0iAjGTVP1VGOh6KMwecBUAUodiaH/Xj/WZ4HzyOl5Adbru7nGlgIXgiH7ttB+av5487/ulzev3/H+OCaERomlmeidzWSGZ+zhPoc8atPTu6hvcbsrwUxm63Fp+LXsm+FVW4E6nC0k1y50SeBucpJ8tFiVXN8feJXYEBgyAoFMjmGgmxRMMl6F4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Start adding a DAMON application module for access/contiguity-aware memory auto-scaling. The module does proactive reclamation, scale-down, and scale-up of memory under user-defined min/max memory and acceptable level of memory pressure using three DAMOS schemes each designed for each of the three main operations. Nonetheless, this is only the initial part of the implementation. Hence this commit implements only the memory pressure-aware auto-tuning proactive reclamation feature. Following commits will implement scale down and up, respectively. Signed-off-by: SeongJae Park --- mm/damon/Kconfig | 10 ++ mm/damon/Makefile | 1 + mm/damon/acma.c | 335 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 346 insertions(+) create mode 100644 mm/damon/acma.c diff --git a/mm/damon/Kconfig b/mm/damon/Kconfig index fecb8172410c..4fe7520601dd 100644 --- a/mm/damon/Kconfig +++ b/mm/damon/Kconfig @@ -121,4 +121,14 @@ config DAMON_LRU_SORT protect frequently accessed (hot) pages while rarely accessed (cold) pages reclaimed first under memory pressure. +config DAMON_ACMA + bool "Build Access/Contiguity-aware Memory Auto-scaling (DAMON_ACMA)" + depends on DAMON_PADDR + help + This builds the DAMON-based Access/Contiguity-aware Memory + Auto-scaling subsystem. It preempts unnecessary memory from the + system and report it to the host while respecting user-specified + min/max memory for the system and maximum memory pressure stall time + ratio. + endmenu diff --git a/mm/damon/Makefile b/mm/damon/Makefile index f7add3f4aa79..814c8da3081b 100644 --- a/mm/damon/Makefile +++ b/mm/damon/Makefile @@ -7,3 +7,4 @@ obj-$(CONFIG_DAMON_SYSFS) += sysfs-common.o sysfs-schemes.o sysfs.o obj-$(CONFIG_DAMON_DBGFS) += dbgfs.o obj-$(CONFIG_DAMON_RECLAIM) += modules-common.o reclaim.o obj-$(CONFIG_DAMON_LRU_SORT) += modules-common.o lru_sort.o +obj-$(CONFIG_DAMON_ACMA) += modules-common.o acma.o diff --git a/mm/damon/acma.c b/mm/damon/acma.c new file mode 100644 index 000000000000..276b61fd4e26 --- /dev/null +++ b/mm/damon/acma.c @@ -0,0 +1,335 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * DAMON-based Access/Contiguity-aware Memory Auto-scaling + * + * Let user specifies min/max memory of the system and acceptable level of + * memory pressure stall level. While respecting those, automatically scale + * the memory of the system up and down by scale_downing memory from the system + * and report it to the host when the system is having memory pressure level + * under the threshold, and vice versa, respectively. + * + * At this moment, the scaling is not implemented, hence this is just a memory + * pressure-aware proactive reclamation module. + * + * Author: SeongJae Park + */ + +#define pr_fmt(fmt) "damon-acma: " fmt + +#include +#include +#include + +#include "modules-common.h" + +#ifdef MODULE_PARAM_PREFIX +#undef MODULE_PARAM_PREFIX +#endif +#define MODULE_PARAM_PREFIX "damon_acma." + +/* + * Enable or disable DAMON_ACMA. + * + * You can enable DAMON_ACMA by setting the value of this parameter as ``Y``. + * Setting it as ``N`` disables DAMON_ACMA. Note that DAMON_ACMA could do no + * real monitoring and memory auto-scaling due to the watermarks-based + * activation condition. Refer to below descriptions for the watermarks + * parameter for this. + */ +static bool enabled __read_mostly; + +/* + * Make DAMON_ACMA reads the input parameters again, except ``enabled``. + * + * Input parameters that updated while DAMON_ACMA is running are not + * applied by default. Once this parameter is set as ``Y``, DAMON_ACMA + * reads values of parametrs except ``enabled`` again. Once the re-reading is + * done, this parameter is set as ``N``. If invalid parameters are found while + * the re-reading, DAMON_ACMA will be disabled. + */ +static bool commit_inputs __read_mostly; +module_param(commit_inputs, bool, 0600); + +/* + * Desired level of memory pressure-stall time in microseconds. + * + * While keeping the caps that set by other quotas, DAMON_RECLAIM automatically + * increases and decreases the effective level of the quota aiming this level of + * memory pressure is incurred. System-wide ``some`` memory PSI in microseconds + * per quota reset interval (``quota_reset_interval_ms``) is collected and + * compared to this value to see if the aim is satisfied. Value zero means + * disabling this auto-tuning feature. + * + * 1 ms/ 1 second (0.1%) by default. Inspired by the PSI threshold of TMO + * (https://dl.acm.org/doi/10.1145/3503222.3507731). + */ +static unsigned long quota_mem_pressure_us __read_mostly = 1000; +module_param(quota_mem_pressure_us, ulong, 0600); + +static struct damos_quota damon_acma_quota = { + /* Use up to 15 ms per 1 sec for scaling, by default */ + .ms = 15, + .sz = 0, + .reset_interval = 1000, + /* Within the quota, mark hotter regions accessed first. */ + .weight_sz = 0, + .weight_nr_accesses = 1, + .weight_age = 0, +}; +DEFINE_DAMON_MODULES_DAMOS_TIME_QUOTA(damon_acma_quota); + +static struct damos_watermarks damon_acma_wmarks = { + .metric = DAMOS_WMARK_NONE, +}; + +static struct damon_attrs damon_acma_mon_attrs = { + .sample_interval = 1000000, /* 1 second */ + .aggr_interval = 20000000, /* 20 seconds */ + .ops_update_interval = 0, + .min_nr_regions = 10, + .max_nr_regions = 1000, +}; +DEFINE_DAMON_MODULES_MON_ATTRS_PARAMS(damon_acma_mon_attrs); + +/* + * Start of the target memory region in physical address. + * + * The start physical address of memory region that DAMON_ACMA will do work + * against. By default, biggest System RAM is used as the region. + */ +static unsigned long monitor_region_start __read_mostly; +module_param(monitor_region_start, ulong, 0600); + +/* + * End of the target memory region in physical address. + * + * The end physical address of memory region that DAMON_ACMA will do work + * against. By default, biggest System RAM is used as the region. + */ +static unsigned long monitor_region_end __read_mostly; +module_param(monitor_region_end, ulong, 0600); + +/* + * PID of the DAMON thread + * + * If DAMON_ACMA is enabled, this becomes the PID of the worker thread. + * Else, -1. + */ +static int kdamond_pid __read_mostly = -1; +module_param(kdamond_pid, int, 0400); + +static struct damos_stat damon_acma_reclaim_stat; +DEFINE_DAMON_MODULES_DAMOS_STATS_PARAMS(damon_acma_reclaim_stat, + acma_reclaim_tried_regions, acma_reclaim_succ_regions, + acma_reclaim_quota_exceeds); + +static struct damos_access_pattern damon_acma_stub_pattern = { + /* Find regions having PAGE_SIZE or larger size */ + .min_sz_region = PAGE_SIZE, + .max_sz_region = ULONG_MAX, + /* no matter its access frequency */ + .min_nr_accesses = 0, + .max_nr_accesses = UINT_MAX, + /* no matter its age */ + .min_age_region = 0, + .max_age_region = UINT_MAX, +}; + +static struct damon_ctx *ctx; +static struct damon_target *target; + +static struct damos *damon_acma_new_scheme( + struct damos_access_pattern *pattern, enum damos_action action) +{ + struct damos_quota quota = damon_acma_quota; + + return damon_new_scheme( + pattern, + action, + /* work for every second */ + 1000000, + /* under the quota. */ + "a, + /* (De)activate this according to the watermarks. */ + &damon_acma_wmarks); +} + +static void damon_acma_copy_quota_status(struct damos_quota *dst, + struct damos_quota *src) +{ + dst->total_charged_sz = src->total_charged_sz; + dst->total_charged_ns = src->total_charged_ns; + dst->charged_sz = src->charged_sz; + dst->charged_from = src->charged_from; + dst->charge_target_from = src->charge_target_from; + dst->charge_addr_from = src->charge_addr_from; +} + +static int damon_acma_set_scheme_quota(struct damos *scheme, struct damos *old, + damos_quota_goal_metric goal_metric) +{ + if (old) + damon_acma_copy_quota_status(&scheme->quota, &old->quota); + goal = damos_new_quota_goal(goal_metric, quota_mem_pressure_us); + if (!goal) + return -ENOMEM; + damos_add_quota_goal(&scheme->quota, goal); + return 0; +} + +/* + * Reclaim cold pages on entire physical address space + */ +static struct damos *damon_acma_new_reclaim_scheme(struct damos *old) +{ + struct damos_access_pattern pattern = damon_acma_stub_pattern; + struct damos *scheme; + int err; + + pattern.max_nr_accesses = 0; + scheme = damon_acma_new_scheme(&pattern, DAMOS_PAGEOUT); + if (!scheme) + return NULL; + err = damon_acma_set_scheme_quota(scheme, old, + DAMOS_QUOTA_SOME_MEM_PSI_US); + if (err) { + damon_destroy_scheme(scheme); + return NULL; + } + return scheme; +} + +static int damon_acma_apply_parameters(void) +{ + struct damos *scheme, *reclaim_scheme; + struct damos *old_reclaim_scheme = NULL; + struct damos_quota_goal *goal; + int err = 0; + + err = damon_set_attrs(ctx, &damon_acma_mon_attrs); + if (err) + return err; + + damon_for_each_scheme(scheme, ctx) + old_reclaim_scheme = scheme; + + reclaim_scheme = damon_acma_new_reclaim_scheme(old_reclaim_scheme); + if (!reclaim_scheme) + return -ENOMEM; + damon_set_schemes(ctx, &reclaim_scheme, 1); + + return damon_set_region_biggest_system_ram_default(target, + &monitor_region_start, + &monitor_region_end); +} + +static int damon_acma_turn(bool on) +{ + int err; + + if (!on) { + err = damon_stop(&ctx, 1); + if (!err) + kdamond_pid = -1; + return err; + } + + err = damon_acma_apply_parameters(); + if (err) + return err; + + err = damon_start(&ctx, 1, true); + if (err) + return err; + kdamond_pid = ctx->kdamond->pid; + return 0; +} + +static int damon_acma_enabled_store(const char *val, + const struct kernel_param *kp) +{ + bool is_enabled = enabled; + bool enable; + int err; + + err = kstrtobool(val, &enable); + if (err) + return err; + + if (is_enabled == enable) + return 0; + + /* Called before init function. The function will handle this. */ + if (!ctx) + goto set_param_out; + + err = damon_acma_turn(enable); + if (err) + return err; + +set_param_out: + enabled = enable; + return err; +} + +static const struct kernel_param_ops enabled_param_ops = { + .set = damon_acma_enabled_store, + .get = param_get_bool, +}; + +module_param_cb(enabled, &enabled_param_ops, &enabled, 0600); +MODULE_PARM_DESC(enabled, + "Enable or disable DAMON_ACMA (default: disabled)"); + +static int damon_acma_handle_commit_inputs(void) +{ + int err; + + if (!commit_inputs) + return 0; + + err = damon_acma_apply_parameters(); + commit_inputs = false; + return err; +} + +static int damon_acma_after_aggregation(struct damon_ctx *c) +{ + struct damos *s; + + /* update the stats parameter */ + damon_for_each_scheme(s, c) { + switch (s->action) { + case DAMOS_LRU_RECLAIM: + damon_acma_reclaim_stat = s->stat; + break; + default: + break; + } + + return damon_acma_handle_commit_inputs(); +} + +static int damon_acma_after_wmarks_check(struct damon_ctx *c) +{ + return damon_acma_handle_commit_inputs(); +} + +static int __init damon_acma_init(void) +{ + int err = damon_modules_new_paddr_ctx_target(&ctx, &target); + + if (err) + return err; + + ctx->callback.after_wmarks_check = damon_acma_after_wmarks_check; + ctx->callback.after_aggregation = damon_acma_after_aggregation; + + /* 'enabled' has set before this function, probably via command line */ + if (enabled) + err = damon_acma_turn(true); + + return err; +} + +module_init(damon_acma_init); From patchwork Sun May 12 19:36:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 13662844 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80A9BC25B5F for ; Sun, 12 May 2024 19:37:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 780EC6B01F2; Sun, 12 May 2024 15:37:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 55F916B01F7; Sun, 12 May 2024 15:37:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13A576B01F2; Sun, 12 May 2024 15:37:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D278D6B01F4 for ; Sun, 12 May 2024 15:37:09 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7FF71140852 for ; Sun, 12 May 2024 19:37:09 +0000 (UTC) X-FDA: 82110752178.09.662C650 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf17.hostedemail.com (Postfix) with ESMTP id 7967E40013 for ; Sun, 12 May 2024 19:37:07 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=u6l3DmWi; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf17.hostedemail.com: domain of sj@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=sj@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715542628; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0Ww0A3VA+M/2XjNLA0vOA5Bik7Ld5ECnqE/YNkhERXs=; b=PPVU3Lwvt8n2m1jfkxoYJzIfLllL/NGkEXzzx3GA0d9iIiTbyY3okKKDYDTz+Z/QECTBR+ ktTT7q3XruSB/3Sd12Z8TGkBXlUIDmHLF0bUw+jeoZHYmfs6SnoHL+RQNzhdxYbxT/ORBo wy9Ur3Yhx9rw/DDlFAEkmvJt441w3pg= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=u6l3DmWi; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf17.hostedemail.com: domain of sj@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=sj@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715542628; a=rsa-sha256; cv=none; b=QlLtNZA14vWrDxswU5hUO0TkREOjUHBoP/Sc09clwzsGFTtdGrfu+vT+4IT52+66e1ejYl D7zL/a6qby4B+Qla4flzM4zb3lZZAxFGj+4G1VfHNRPCYYO/F09+JtQtoVJmr9RmILqqov CSKtUN3d1p/xhT7oAKPbksJ9mllR8mM= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 8D0B6CE0B15 for ; Sun, 12 May 2024 19:37:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 87BA2C116B1; Sun, 12 May 2024 19:37:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715542623; bh=uU0i+7VRHwnLzmsgFart/FdYz1FyWcfTrlJwgj1wLXw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u6l3DmWiydgP0NCLJj1UuD9gUii+CJo1Lz8Nttns5L737tG96vubosSPx07CxVI5D LHL2N3WpDvMNSRhitK0IUhJGLqFT8TQr93XSR72I1rg+c9hIxAv1x4Xc9RomlhFG96 M2BK0wmkH5SAVuxj8dxYuLcpV92z0Sf0579l8x2xcgPEZErxYf/uBD58dYVEcgYDEe HLmCLsbGhvRBV/pJXARjMT6uxcIfKIUTJK/BSBCr01drUp/RCnku/RO/kpKv6lpdF1 gd3aw7wFgDE2rHdr4jWd6zLyknqG0ExTPfkTNEKWNTQVnve326rH6M4G2IEqANuk09 wRn+E1JBQfGpg== From: SeongJae Park To: Cc: SeongJae Park , damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC IDEA v2 3/6] mm/page_reporting: implement a function for reporting specific pfn range Date: Sun, 12 May 2024 12:36:54 -0700 Message-Id: <20240512193657.79298-4-sj@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240512193657.79298-1-sj@kernel.org> References: <20240512193657.79298-1-sj@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7967E40013 X-Stat-Signature: dmgrtncfjsbm3bw13h97cg5n6n8pb4ic X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1715542627-854673 X-HE-Meta: U2FsdGVkX1/xCWMKKhojZwUzswKXXvP5lb6ObzCAP3HVvgoRHDk32i25s4gQLarUkJyjv1QgYf8Bh2Rti4CuJ41cvhAlCwGf280NwtBMkCufmMEkzeAnKmOyu9BpTDSnnEqb4h7x92bIScKOTLSkemcokECV0GwGD+Us0TJO5pqiMBbMVBVfhteVic8nskn6MAqyexI4CpUUbd2PjhKtekKKMElSV8lfjqUjHTgNLSrkZospIDJuHIm5yM32v7p1O2eH4Kle5a0JZHbiMFhArLRsHHfwV7ko535XVVkFlZy7DsXJNU3TMJX0cw9hm3YGiZ4j8FSrZDklyx9gsgqr+Xs3UX+vbpdGS9Tw9+xVi/6ZhE4JOLpbVI0yamu4a3j0UmJHtXF3mJw/zuv7c/DxSblpSw9ns3GQSuIlYFWo643xoXXcP86azAV5bK+pVm19vRXONkTCRqbcmrMtI2ezBqExVZfcg442U9kSVejcyvt1ditoHJ3nEcZKxxuebJCTfztFPtSh+9T6nT0iSvf55QBHwEWXCsVdW/XUpkHmZYk+j6E59JtS8TYrpuO2RV49gzW/Cdk+9YlMsMIpDa4fWc5PW6Zo8Sq7Ii8EH6SB+oI+ZygmHlLHA8xaJNEwyfjvm2nYJDCGAgFknKvW+GaUEKCXVe4mw7vZQMAScKPsq1CPHmU/3sLWdjhwL3HqwjxFMXgQFJSCFTHekxKDVa2PtF8bce6sc4vBg1cDvW06iLWaGJh/DQBTx9R0CC3pOEBXd7iYvGiJ/wHa2NGsEvdeAu1lTCgJ8ZhcHgEALuslKmxdRgUKVj9BjvRW7CLjsfdknCQYw+ha2It2R3L6OFrqM5ZqwfKq2ZzLFMW2Q57w5jNJByt3wjkxKRwbdboWgFG2BZHGhlYmV/3k/rFApwjtVk9Ukdg2dYkSnccYXE9Am9ZAGhi3p7MEvczNaW3G4KfJSfu4UbKveKY0dgRzHLc Wv3V46fw tYvlcP0ThVkPQumnxBJ34JTPXi/NfQsFuY7tZvYT70b+1XCUx0YfUQP5N81wyyCvxuFu0O4zZS236FojqszCjgjiorpTTacO4iNUbzUUHkthyZpexEjN/mZRssaiRtrefcJ6Lp3iUIszhrC4F8LcNdyAe2SU3QcDE0aarcfUAvoKEEtZXbehFvbZliXwfSnJ+jIyA9M2FTzmbC15MKGnHgCSvJI2UwjMRHLbtYiRPU/fPrOMORFc3ZT25xashexeLP0+RDB2bylanAhhITKyw0RG2N3m5r9e/SwiLlt6fD5eVgn7nifzjOn8KsDfQkuk+Wgz3xltBKORr4SuVnJMTrMc2LxT7j/GZMuDr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement a function for reporting pages of specific pfn range, for non-free pages reporting use case. The use case will be implemented by following commits. Signed-off-by: SeongJae Park --- mm/page_reporting.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/mm/page_reporting.c b/mm/page_reporting.c index e4c428e61d8c..e14f2e979f16 100644 --- a/mm/page_reporting.c +++ b/mm/page_reporting.c @@ -349,6 +349,33 @@ static void page_reporting_process(struct work_struct *work) static DEFINE_MUTEX(page_reporting_mutex); DEFINE_STATIC_KEY_FALSE(page_reporting_enabled); +#ifdef CONFIG_ACMA + +int page_report(unsigned long pfn, unsigned long nr_pages) +{ + struct page_reporting_dev_info *prdev; + struct scatterlist sgl; + int err; + + rcu_read_lock(); + + prdev = rcu_dereference(pr_dev_info); + if (!prdev || !prdev->report) { + rcu_read_unlock(); + return -ENOENT; + } + + sg_init_table(&sgl, 1); + sg_set_page(&sgl, NULL, nr_pages << PAGE_SHIFT, 0); + sgl.dma_address = PFN_PHYS(pfn); + + err = prdev->report(prdev, sgl, 1); + rcu_read_unlock(); + return err; +} + +#endif + int page_reporting_register(struct page_reporting_dev_info *prdev) { int err = 0; From patchwork Sun May 12 19:36:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 13662840 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A998C25B77 for ; Sun, 12 May 2024 19:37:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B5C156B01F0; Sun, 12 May 2024 15:37:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A8B6B6B01F3; Sun, 12 May 2024 15:37:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 862CF6B01F1; Sun, 12 May 2024 15:37:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 594DA6B01B3 for ; Sun, 12 May 2024 15:37:07 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id EB6EC1C0583 for ; Sun, 12 May 2024 19:37:06 +0000 (UTC) X-FDA: 82110752052.01.C6BC937 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf22.hostedemail.com (Postfix) with ESMTP id 58A7BC000E for ; Sun, 12 May 2024 19:37:05 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=TnT+ejCW; spf=pass (imf22.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715542625; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g+AjUyZNCIDu9N3jTwSpZb1GlERAOdp6NP0QfDvG0AU=; b=ff55A8NJgLoGPNl61hKhccHSTJ8Md78iBRQIOAoDD5aybwXfRPD9VYd6z4LkyJN1yxWN/i HBm/ozYz4HS2Zqs+OQ/UJ7thDQs5TLdrRNE4BREuXBcFhi/ZsIGDuzPB8vgIE71VCMqlH2 WoBkxMlsD1odYPo02eyuIHJoX4lcpd4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715542625; a=rsa-sha256; cv=none; b=ouyarR3QXzo7Q5X0dNqPCjwdWpfkU1lZLXoPtURuxni6M65KusJoBwdHkys56+WWGIo1SH /LB1AOL9P0r1Ovizn8hHbCzWq9fMjRDj7nd6tkNEpiHA7vd9fMgZou1usHNnlwHabjZI+7 A1W9/eIMamHz8FPUxVFs9j56eOM+A3Y= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=TnT+ejCW; spf=pass (imf22.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 7C42560B99 for ; Sun, 12 May 2024 19:37:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 08855C4AF0B; Sun, 12 May 2024 19:37:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715542624; bh=tWKq7K6XmSUSldcREZ1dFiZ70aiSWF0Pc/jUVnK6YyM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TnT+ejCW/rIvYMQFgfYDh/RgVbKdYfxAURsgsZMDmHalrJ0Yth5mnC1e+08nyM6hf 0fEbQnHb7Kv6iXI/YIiPXkXlIuTvt+Wh0XgYG6uQyLUl8kv4XAV6voDMIIUDOlM8+t tz+cjrWmCiWVJin0IHHRkzBTx3ih/uBG8HEgYtsy4odEvm47bgDSbAWdNF8+hVsT5Y fgFLwwlbWYnbw5PhmnH+lSrTlZAGkYz80u0gsp5i1Ks0fqvvtJj4QP8HKMuS/qsdAy gqP6loR91ecl68XNi7JsjXq4sqW7hqLY/pU3z2qnCiYIdhq1Hex8UKCKLBWsyt2Pw2 mFszbSNsaWI1Q== From: SeongJae Park To: Cc: SeongJae Park , damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC IDEA v2 4/6] mm/damon/acma: implement scale down feature Date: Sun, 12 May 2024 12:36:55 -0700 Message-Id: <20240512193657.79298-5-sj@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240512193657.79298-1-sj@kernel.org> References: <20240512193657.79298-1-sj@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: hoqji567e1e4sjuzcukykzpyndn1mnw7 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 58A7BC000E X-HE-Tag: 1715542625-873658 X-HE-Meta: U2FsdGVkX1/65lfwF0XXHq3by4APmbBcRCT6o0V0pNkGMxWyPgYcA3mPmn89kNzS5uw/NpSzAX9WA037WdKgdLupRSS8ubEa1Ju89KU1zieycG3Dmqw5zlIGhJ4xBbf+NxO2XXw53oRlUjoelhUiC8hX6p8mZ2XZIbOatfzzzMizEZI3Do+jny3qCCsB+qaC8V06Y4//Sw18YDKKHKMOglPH9CaWLzl/ZIR7s4IKcZJLkIJpUfV5Esv7G2ezerpPAyzgOka+5afG4CIJA6M5EvG1H6JV2K/lzyqtneQSyy0VOOyFEWl+I0wUCHpNwyU/PavTTQwqPH1e1U5D6lbQ9sfqKMET+QTRv5P44v35GQFTv5DvoEkqOY8UvfVZFRkXeYFfmy8/fFcTB+PqpKppIywnxRO2bYGGJulFfpWuLphyE9QA2RIcnJguB4howun889HXzbj12VKD4RpHhtWhxWwOQjNQN4ELTmmRQPivBtSc8az16JjMGz/gCBDpipeyY55X3OsibWV5RU/OFcrEtJ/Gdcs/JJTeqhFl7Dowh/WQab29NpKWZZE7NMNSK9W2bsVCLJyn01o0kZnjsrJzcQnYgSlTMd4vjsaCOcZu4iNmED5S6R9tSDdhv8juyUbtOYHtKqisr8sAGL/C3RFhes0VS133+F7E+1aWJ5jmemx00ol/dXmsfT5U9dTz93acu4ngzXR9goEohk1raSt5A7iMISnOD1/kAUogYdbiy/kwsgXLefyBRgTuWcRU+mhzxpkGJcxj/KTpDkZl1wrCBUy7Sui6ShATPXX+PpRhqL5u8cWOZk8LSkWSMFlD1kRjCFJhhABfnamFysyBR3rPuR5Ai59EhLWrbzzSBdNyDMCL/mkUdGPgjB8sR2Kb0xoLXMCIM9ptZfjLnPyS+AP5KhXMNRFzQymo9OBdT19L4N1y+4meAI95Ifr6LWlbgtTeSvRyIp5ngpH6xtLYv3H oL0cEprS 20Z/nwMOfEx0jQM1dLZ42LhLTXWzlXSvp/TXIn4wTJPcep08DWBU+S/1ZDMRSS6aY24bUmElYQwF3rinhRgcOZycmDyQas862oSSNCtdcBco0eukVy2BZHn+DipAhpzEnbssNuF6gKzI622pe8/L8uZmH5WVQk/SSiH/x4W2lbNTkeS+85lJXjkFzxwUXEtB4kkbShBFxGUENbFj5hbldnfquPAefcm49/ea/0AoDhx27sDsyWxaITlL/rB3SMTawBMtyPCPW7DvW8+CLFNSzbSmosDTey9dwTZwUw6Ex9HHg5FfgAqRpBbJA9R57luf93GWwSfH35NRlMJw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement the memory scale down feature of Access/Contiguity-aware Memory Auto-scaling module. It runs DMAOS_ALLOC action scheme with user-acceptable level of memory pressure stall as its target, in 512 contig pages base granularity, and report the 512 contig successfully DAMOS_ALLOC-ed regions to the host for reuse. For minimizing DAMON-internal DAMOS_ALLOC-ed region management overhead, the scheme is applied to only 128 MiB not-completely-allocated contiguous memory region of highest address. Signed-off-by: SeongJae Park --- mm/damon/acma.c | 137 +++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 131 insertions(+), 6 deletions(-) diff --git a/mm/damon/acma.c b/mm/damon/acma.c index 276b61fd4e26..b093b90471dd 100644 --- a/mm/damon/acma.c +++ b/mm/damon/acma.c @@ -8,9 +8,6 @@ * and report it to the host when the system is having memory pressure level * under the threshold, and vice versa, respectively. * - * At this moment, the scaling is not implemented, hence this is just a memory - * pressure-aware proactive reclamation module. - * * Author: SeongJae Park */ @@ -50,6 +47,13 @@ static bool enabled __read_mostly; static bool commit_inputs __read_mostly; module_param(commit_inputs, bool, 0600); +/* + * Minimum amount of memory to be guaranteed to the system. In other words, + * the lower limit of the scaling. + */ +static unsigned long min_mem_kb __read_mostly; +module_param(min_mem, ulong, 0600); + /* * Desired level of memory pressure-stall time in microseconds. * @@ -66,6 +70,18 @@ module_param(commit_inputs, bool, 0600); static unsigned long quota_mem_pressure_us __read_mostly = 1000; module_param(quota_mem_pressure_us, ulong, 0600); +/* + * Basic scale down/up granularity. ACMA will allocate and report contiguous + * pages of this size at once. 512 pages (2 MiB for 4 KiB page setup) by + * default. + * + * To minimize DAMON-internal ALLOC-ed memory management overhead, we further + * apply SCALE_WINDOW. Refer to damon_acma_set_scale_down_region_filter() for + * more detail about it. + */ +static unsigned int scale_pg_order __read_mostly = 9; +module_param(scale_pg_order, uint, 0600); + static struct damos_quota damon_acma_quota = { /* Use up to 15 ms per 1 sec for scaling, by default */ .ms = 15, @@ -123,6 +139,11 @@ DEFINE_DAMON_MODULES_DAMOS_STATS_PARAMS(damon_acma_reclaim_stat, acma_reclaim_tried_regions, acma_reclaim_succ_regions, acma_reclaim_quota_exceeds); +static struct damos_stat damon_acma_scale_down_stat; +DEFINE_DAMON_MODULES_DAMOS_STATS_PARAMS(damon_acma_scale_down_stat, + acma_scale_down_tried_regions, acma_scale_down_succ_regions, + acma_scale_down_quota_exceeds); + static struct damos_access_pattern damon_acma_stub_pattern = { /* Find regions having PAGE_SIZE or larger size */ .min_sz_region = PAGE_SIZE, @@ -143,6 +164,9 @@ static struct damos *damon_acma_new_scheme( { struct damos_quota quota = damon_acma_quota; + /* Use 1/2 of total quota for hot/cold pages sorting */ + quota.ms = quota.ms / 2; + return damon_new_scheme( pattern, action, @@ -177,6 +201,61 @@ static int damon_acma_set_scheme_quota(struct damos *scheme, struct damos *old, return 0; } +/* + * scale_pg_order is for basic scaling granularity. Have a larger granularity + * to limit DAMON-internal alloc-ed pages management overhead. + */ +#define SCALE_WINDOW (128 * MB) + +/* + * Set scale_down scheme's address range type filter to apply scaling down to + * only current scaling window. Scaling window is SCALE_WINDOW size contiguous + * memory region of highest address that not yet completely DAMOS_ALLOC-ed and + * reported. + * + * TODO: Apply 'struct page' reduction in SCALE_WINDOW or lower granularity. + * E.g., hot-unplug the memory block, or apply vmemmap remapping-based approach + * like hugetlb vmemmap optimization + * (https://docs.kernel.org/mm/vmemmap_dedup.html). + */ +static int damon_acma_set_scale_down_region_filter(struct damos *scheme) +{ + struct damos_filter *filter = damos_new_filter( + DAMOS_FILTER_TYPE_ADDR, false); + unsigned long end; + unsigned long start_limit, end_limit; + + if (!filter) + return -ENOMEM; + + /* scale down no below min_mem_kb */ + end_limit = monitor_region_end; + start_limit = monitor_region_start + min_mem_kb * KB; + + /* not-completely-alloc-ed SCALE_WINDOW region of highest address */ + for (end = end_limit; end >= start_limit + SCALE_WINDOW; + end -= SCALE_WINDOW) { + if (damon_alloced_bytes(end, end - SCALE_WINDOW) + != SCALE_WINDOW) + break; + } + filter->addr_range.start = max(start_limit, end - SCALE_WINDOW); + filter->addr_range.end = end; + + damos_add_filter(scheme, filter); + return 0; +} + +/* + * Called back from DAMOS for every damos->alloc_order contig pages that + * just successfully DAMOS_ALLOC-ed. + */ +static int damon_acma_alloc_callback(unsigned long start_addr) +{ + /* For non-zero return value, DAMOS free the pages. */ + return page_report(PHYS_PFN(addr), 1 << scale_pg_order); +} + /* * Reclaim cold pages on entire physical address space */ @@ -199,10 +278,40 @@ static struct damos *damon_acma_new_reclaim_scheme(struct damos *old) return scheme; } +/* + * Scale down scheme + */ +static struct damos *damon_acma_new_scale_down_scheme(struct damos *old) +{ + struct damos_access_pattern pattern = damon_acma_stub_pattern; + struct damos *scheme; + int err; + + scheme = damon_acma_new_scheme(&pattern, DAMOS_ALLOC); + if (!scheme) + return NULL; + err = damon_acma_set_scheme_quota(scheme, old, + DAMOS_QUOTA_SOME_MEM_PSI_US); + if (err) { + damon_destroy_scheme(scheme); + return NULL; + } + /* alloc in 512 pages granularity */ + scheme->alloc_order = scale_pg_order; + scheme->alloc_callback = damon_acma_alloc_callback; + err = damon_acma_set_scale_down_region_filter(scale_down_scheme); + if (err) { + damon_destroy_scheme(scheme); + return NULL; + } + return scheme; +} + static int damon_acma_apply_parameters(void) { struct damos *scheme, *reclaim_scheme; - struct damos *old_reclaim_scheme = NULL; + struct damos *scale_down_scheme; + struct damos *old_reclaim_scheme = NULL, *old_scale_down_scheme = NULL; struct damos_quota_goal *goal; int err = 0; @@ -210,14 +319,27 @@ static int damon_acma_apply_parameters(void) if (err) return err; - damon_for_each_scheme(scheme, ctx) - old_reclaim_scheme = scheme; + damon_for_each_scheme(scheme, ctx) { + if (!old_reclaim_scheme) { + old_reclaim_scheme = scheme; + continue; + } + old_scale_down_scheme = scheme; + } reclaim_scheme = damon_acma_new_reclaim_scheme(old_reclaim_scheme); if (!reclaim_scheme) return -ENOMEM; damon_set_schemes(ctx, &reclaim_scheme, 1); + scale_down_scheme = damon_acma_new_scale_down_scheme( + old_scale_down_scheme); + if (!scale_down_scheme) { + damon_destroy_scheme(reclaim_scheme); + return -ENOMEM; + } + damon_add_scheme(ctx, scale_down_scheme); + return damon_set_region_biggest_system_ram_default(target, &monitor_region_start, &monitor_region_end); @@ -303,6 +425,9 @@ static int damon_acma_after_aggregation(struct damon_ctx *c) case DAMOS_LRU_RECLAIM: damon_acma_reclaim_stat = s->stat; break; + case DAMOS_ALLOC: + damon_acma_scale_down_stat = s->stat; + break; default: break; } From patchwork Sun May 12 19:36:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 13662845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6FCEC25B5F for ; Sun, 12 May 2024 19:37:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D70386B01F5; Sun, 12 May 2024 15:37:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CF74D6B01F7; Sun, 12 May 2024 15:37:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD4156B01F8; Sun, 12 May 2024 15:37:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 909466B01F5 for ; Sun, 12 May 2024 15:37:10 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 43F27C0A91 for ; Sun, 12 May 2024 19:37:10 +0000 (UTC) X-FDA: 82110752220.05.E8C1AFF Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf28.hostedemail.com (Postfix) with ESMTP id 09EB6C0013 for ; Sun, 12 May 2024 19:37:07 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=HgPPQHOu; spf=pass (imf28.hostedemail.com: domain of sj@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715542628; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RMZ78f86iXATK3e9FW2bU7qfqiML1ZuX822lcGBmhTg=; b=oPuoZcTILfFbzs4zPV2XEc6ISMYxfrBv6XPAvS4N0/NiaGRTeTR9iOEfkeoqpj3N6VKIEc h67sNCCiI3dIElp6TTopEcC0Xk1gtdIrWvtj+wZPXkhN1VyzIr+HhFtt8GlXWjs0mWEntl qW00mlbjxZTHAmCgPfVTV3OHukU2RCY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715542628; a=rsa-sha256; cv=none; b=jMmaDo8lG7atYh2EzuNBJfKg0zTf84Kg/5lQIZWSah1Z8fZA4hRNC5at7kpAMZNB+sr721 yDX7DRGGz/9BNdS+Wx82ugvKifp1ZZwW3J7CIuQTs/hrG2kYVGY5+qbdIJOiDQGHcEjq1M z3rPtwdfPk289soSY1HxNzIkGpuc8h4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=HgPPQHOu; spf=pass (imf28.hostedemail.com: domain of sj@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 2D21CCE0B2F for ; Sun, 12 May 2024 19:37:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 82D63C32781; Sun, 12 May 2024 19:37:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715542624; bh=t7+2aKGmVybNoTsNs3UjmFvYLVfl4dPfhLTblMrn/KY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HgPPQHOuTXT6FE4SuvaFzlVnrDo3qPzoUsiD7CoL2rElMRvR+osrfWPN6jSENu3/N S/FQHQk2Rr+NjPrG3a6KPW/EOJJlpItNzFgLnfdOyt8YPV8zlLA7YEME7w8J29SvOd xtAl3JLWQp9TD3ZmaoFxgS30+sh780Kf4SBcjCCsmEyI03JuMEjE2ScLQ6q4X0Kn/D ned+SNZshIE9bsE+wZ7PY3mJz/1uEfojxUiHQj1hKwNpmei3xixA7O3lrv7E0mmM3H 4fLoi2EPD8RswWtisl8TDrhMmAcLhdKPIKZdr/0z/0Mk22fJ03z8TLbbB4TfGMa/pM jxFu0+NhhMvkQ== From: SeongJae Park To: Cc: SeongJae Park , damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC IDEA v2 5/6] mm/damon/acma: implement scale up feature Date: Sun, 12 May 2024 12:36:56 -0700 Message-Id: <20240512193657.79298-6-sj@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240512193657.79298-1-sj@kernel.org> References: <20240512193657.79298-1-sj@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: fs9wraza4wazkrezuhyf85z44a7agwdp X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 09EB6C0013 X-HE-Tag: 1715542627-202674 X-HE-Meta: U2FsdGVkX18ml1cWVOEFmZtUTiQJhLh0I/Ym1g3T9z2un9EgokgYu8AjwytGdFXXC45A/2VmIetdJn5ef9CxO1L4nTYGmE5oVpmDZDXeLQ05T0jsm75W8j70qps248P3nUuPvUPmldaUThJ9BYH+J2DpAHjI/2z764MImp8v3yD1MVrhkUKiuIk9nG7p/cRFFe8OB6Z0aAo7LVW/CeBevfAJREOiD4yzMM30IZI7gTEHVzO4JQroZnF/WwnJ60oreJHp4mpjJXKVF/0AwyZSFut/B4HuyLCAAGcEhI3c0XfBV65BM4PKMHfvlRh/0fKkBx9eDbpqmj3rAcvp69cRcev0MollRL3peJl0cPq4IbQfE2GfutWQA8e3t1b6R/dQw+kKpcZsbYCDQunV5PxXtXJjduICI4RqHzMQzhgd5M0w2eZrwlG2Qru4IpHvT3ZMXp2t5d454SgtNg5Imv7cbxb+4x81IVgfq/iP7mhITmvH77ZPErdJYjOlt3hR1S+qtA40h4wKMqNHuSu5IXNPQpPG2Zsd3zSZ/WFkiv/O9garemWSlfxASa4HIVKS2wqKx5Zlzd6P5ojVeIvEhKLWUUujl4TJtJjNEpS/WOHtcySlusj8xYMcRu5ckgoX90pMswQug7REskYvh4e9yTEREmtJx4HzEsdQPe+QKrP2v8fRUuWj2t+vXjTnC0L2Pgb5/uZgHWF6AcqoXkXLliiDjOShHjHWcETC7+LE/x+ydI55ga4bIUOEXbxc1EVmaGzSqkl92tlT2Y6FAOAurtMq2vW3V2PttZ/mVuMdPGNBbbxBX9koJD+h1JbxopsofrnAWK51f9+dZHTMd80Tw6eOV8Dgo1V0yaQkU/y/g1KPYL1iqcR1yBn7ReBJa4iLPJdFoL2V3FC0xwr9ceLv530Ej+q7pvubufY3EpoTuRmL4L1tok1eJjrmPQH+k76SbDbq8pURoIxwiMWv1TFCAO8 hMQeUKVt Yq0M57HZ3mzD+E9W3UWKhax/pk/J8UsQ0ytofgjMAByqIRWXEub6A13PtYF+IUgdzFplj3S78szNuVsQoAk27VjMr1Wmkc0CppslTlSI4HJHSyYhjxFs8Ny1fNaepUFu/l9QU2IyFmojPisOT49NIOcjBPTUygHsqsa7rZGa+JI3VK74FvHyd/6EeI9FokWLh1Zh2M4q+Ld/JEJETSoMFUZ1bm43xS5kwBBqeE+Spi8JqzZE3iAaxY8aCmVA3gfdLxEmsriS5rNPx/6qlCrmlbqIgygbmZ3fsEOjI6OtZP5LAevr+G+bGt1UV0uamGHP7LjgnYJXcZcigiAozC3C1g9WkbQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement the memory scale up feature of Access/Contiguity-aware Memory Auto-scaling module. It runs DAMOS_FREE action scheme with user-acceptable level of memory pressure stall as its target, in 512 contig pages base granularity. For minimizing DAMON-internal DAMOS_ALLOC-ed region management overhead, the scheme is applied to only 128 MiB not-completely-allocated contiguous memory region of lowest address. Note that it does nothing with free_callback() since the host will notify in-guest use of the page via page fault and return it on demand. Signed-off-by: SeongJae Park --- mm/damon/acma.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 90 insertions(+), 4 deletions(-) diff --git a/mm/damon/acma.c b/mm/damon/acma.c index b093b90471dd..32827cbf2fa7 100644 --- a/mm/damon/acma.c +++ b/mm/damon/acma.c @@ -54,6 +54,13 @@ module_param(commit_inputs, bool, 0600); static unsigned long min_mem_kb __read_mostly; module_param(min_mem, ulong, 0600); +/* + * Maximum amount of memory to be guaranteed to the system. In other words, + * the upper limit of the scaling. + */ +static unsigned long max_mem_kb __read_mostly; +module_param(max_mem, ulong, 0600); + /* * Desired level of memory pressure-stall time in microseconds. * @@ -144,6 +151,11 @@ DEFINE_DAMON_MODULES_DAMOS_STATS_PARAMS(damon_acma_scale_down_stat, acma_scale_down_tried_regions, acma_scale_down_succ_regions, acma_scale_down_quota_exceeds); +static struct damos_stat damon_acma_scale_up_stat; +DEFINE_DAMON_MODULES_DAMOS_STATS_PARAMS(damon_acma_scale_up_stat, + acma_scale_up_tried_regions, acma_scale_up_succ_regions, + acma_scale_up_quota_exceeds); + static struct damos_access_pattern damon_acma_stub_pattern = { /* Find regions having PAGE_SIZE or larger size */ .min_sz_region = PAGE_SIZE, @@ -164,8 +176,8 @@ static struct damos *damon_acma_new_scheme( { struct damos_quota quota = damon_acma_quota; - /* Use 1/2 of total quota for hot/cold pages sorting */ - quota.ms = quota.ms / 2; + /* Use 1/3 of total quota for hot/cold pages sorting */ + quota.ms = quota.ms / 3; return damon_new_scheme( pattern, @@ -246,6 +258,36 @@ static int damon_acma_set_scale_down_region_filter(struct damos *scheme) return 0; } +/* + * Similar to damon_acma_set_scale_down_region_filter() but for scaling up. + */ +static int damon_acma_set_scale_up_region_filter(struct damos *scheme) +{ + struct damos_filter *filter = damos_new_filter( + DAMOS_FILTER_TYPE_ADDR, false); + unsigned long start; + unsigned long start_limit, end_limit; + + if (!filter) + return -ENOMEM; + + /* scale up no above max_mem_kb */ + start_limit = monitor_region_start; + end_limit = start_limit + max_mem_kb * KB; + + /* not-completely-free-ed SCALE_WINDOW region of lowest address */ + for (start = start_limit; start <= end_limit - SCALE_WINDOW; + start += SCALE_WINDOW) { + if (damon_alloced_bytes(end - SCALE_WINDOW)) + break; + } + filter->addr_range.start = start; + filter->addr_range.end = min(end_limit, start + SCALE_WINDOW); + + damos_add_filter(scheme, filter); + return 0; +} + /* * Called back from DAMOS for every damos->alloc_order contig pages that * just successfully DAMOS_ALLOC-ed. @@ -307,11 +349,40 @@ static struct damos *damon_acma_new_scale_down_scheme(struct damos *old) return scheme; } +/* + * Scale up scheme + */ +static struct damos *damon_acma_new_scale_up_scheme(void) +{ + struct damos_access_pattern pattern = damon_acma_stub_pattern; + struct damos *scheme; + int err; + + scheme = damon_acma_new_scheme(&pattern, DAMOS_FREE); + if (!scheme) + return NULL; + err = damon_acma_set_scheme_quota(scheme, old, + DAMOS_QUOTA_SOME_MEM_PUSI_US); + if (err) { + damon_destroy_scheme(scheme); + return NULL; + } + scheme->alloc_order = scale_pg_order; + scheme->alloc_callback = NULL; + err = damon_acma_set_scale_up_region_filter(scale_up_scheme); + if (err) { + damon_destroy_scheme(scale_down_scheme); + return NULL; + } + return scheme; +} + static int damon_acma_apply_parameters(void) { struct damos *scheme, *reclaim_scheme; - struct damos *scale_down_scheme; + struct damos *scale_down_scheme, *scale_up_scheme; struct damos *old_reclaim_scheme = NULL, *old_scale_down_scheme = NULL; + struct damos *old_scale_up_scheme = NULL; struct damos_quota_goal *goal; int err = 0; @@ -324,7 +395,11 @@ static int damon_acma_apply_parameters(void) old_reclaim_scheme = scheme; continue; } - old_scale_down_scheme = scheme; + if (!old_scale_down_scheme) { + old_scale_down_scheme = scheme; + continue; + } + old_scale_up_scheme = scheme; } reclaim_scheme = damon_acma_new_reclaim_scheme(old_reclaim_scheme); @@ -340,6 +415,14 @@ static int damon_acma_apply_parameters(void) } damon_add_scheme(ctx, scale_down_scheme); + scale_up_scheme = damon_acma_new_scale_up_scheme(old_scale_up_scheme); + if (!scale_up_scheme) { + damon_destroy_scheme(scale_down_scheme); + damon_destroy_scheme(reclaim_scheme); + return -ENOMEM; + } + damon_add_scheme(ctx, scale_up_scheme); + return damon_set_region_biggest_system_ram_default(target, &monitor_region_start, &monitor_region_end); @@ -428,6 +511,9 @@ static int damon_acma_after_aggregation(struct damon_ctx *c) case DAMOS_ALLOC: damon_acma_scale_down_stat = s->stat; break; + case DAMOS_FREE: + damon_acma_scale_up_stat = s->stat; + break; default: break; } From patchwork Sun May 12 19:36:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 13662841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2335FC25B5F for ; Sun, 12 May 2024 19:37:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0A046B01F1; Sun, 12 May 2024 15:37:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D93816B01F2; Sun, 12 May 2024 15:37:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B97E76B01F3; Sun, 12 May 2024 15:37:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9AB766B01F1 for ; Sun, 12 May 2024 15:37:08 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 43B961605F4 for ; Sun, 12 May 2024 19:37:08 +0000 (UTC) X-FDA: 82110752136.15.091E58A Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf24.hostedemail.com (Postfix) with ESMTP id 9389518000E for ; Sun, 12 May 2024 19:37:06 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="e+J22fH/"; spf=pass (imf24.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715542626; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NkNPmUrhueWtYL66O2BZcuzr0aj9MLsurBwxaoHW6ao=; b=KTkwB4VnuZS++9wvwAtqhyJrjE6l+rd7+0lRxFguaW5sxxNIEeqsq9f3kOtDckCaQ6O5Oi ZAPrOO79XQt6MUjWitZk24wqJFX3JnZPyKtVzAF27HM2l3cw0hRh/BgTUS9qI/ri0Kr3/v yyhPEftgzwp1cMy6S+LXlYFGkyF0p+M= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="e+J22fH/"; spf=pass (imf24.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715542626; a=rsa-sha256; cv=none; b=pSyyzze5cOEGcelpB4CNdx5oBe9KdKQ+9kYgdbgfNZ3obMkUt4VNcYB/upKt3nyhNL+jSw F7pueE/SJlrrkLujKdwoVGz8HBw6ebbPg4hbKqbojsAePF6gzuRG7keM0sDKwd7OmA7gtj xoYOkTL+9VodU5z7Tv1nAtTQ6PjlVLs= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id C010A60B97; Sun, 12 May 2024 19:37:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 081B8C32783; Sun, 12 May 2024 19:37:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715542625; bh=qXYTRSugt7rwl/3A7n+cY924/FHphURhvhYWI9AMXzs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=e+J22fH/4YnyP8x5FrAEpo47lsNbj13oV2jS+1bItSJLZScow+ISz0erPzyq51tDX omRw3NzuQiO1RBpqmHWxZkNdG/C3JpIBRCJRCbdigcjcQA99HuS1zHzIBjLEjyvLM/ kwq3wZLGlKZW9tGsCgDmoFB0XF9ZVchI0zsqHmkv96mGBGcdzYb+wpO4wWv7zIZLUT e0crt2XhymhE1BsA7nONdVsfepGmV7Jk3FLLgaOEUqReDaQPf3eJt2SsfUKoYypJLN FXf9Xgi9xcMVJdrj3qSs3ai2k7adxPQmN4NKNo1MKgwJi/TTmrrNUkoh3eJ0gORigB +mWlKI7cGWgbg== From: SeongJae Park To: Cc: SeongJae Park , "Michael S. Tsirkin" , David Hildenbrand , Jason Wang , Xuan Zhuo , virtualization@lists.linux.dev, linux-kernel@vger.kernel.org, damon@lists.linux.dev, linux-mm@kvack.org Subject: [RFC IDEA v2 6/6] drivers/virtio/virtio_balloon: integrate ACMA and ballooning Date: Sun, 12 May 2024 12:36:57 -0700 Message-Id: <20240512193657.79298-7-sj@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240512193657.79298-1-sj@kernel.org> References: <20240512193657.79298-1-sj@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 9389518000E X-Stat-Signature: mheuebkniwiiwys78tx3xek7u6ggbf5i X-Rspam-User: X-HE-Tag: 1715542626-976232 X-HE-Meta: U2FsdGVkX19VWBJNZXOR7dKc5/25jdHQk0iHEkGRFs7IyjUSK7rTWmqwoE2lGSrj0Ny0ZL2W5avA8YsPP+FutykHvjGZza2es7w8HwWTrrA+u0xeg1+/qCfwnIMr3h0sFC8kYloAyz/B+Ot23j/VqwyI7QiUHuH3HhCZXpRIh8WjzLJ1aULUi4BfR/Hd9K0IIwNG9D/a2D491upXm2894VBKn2f5uIjyxKZxO2EygRqLVQjaYYOirduaVkhWIKVVSo5mTlZztcJdvAgu2exDsMe2XDRcPomol6uJOr/Qg91gwRKAp2G7elOR6wa0Pv8uM3jcoMgQZ3BedtAFnY3ThBGMtIlsG+OuxjWSJJIP0Fef1NVATotzWuJzB/wORKD8d/oiiH/1Hly4yHqcZPshT3o7s9yPCRwDE9buhYlc7CApCf314eHSdG0XrwcwNMTUgEKSp2c32PjAZrbwlpP9rUemxfO54Dn/0JitrIuCk9APqPK4regj3z9hIwexgyrHcG8QHmQCm54069EikDR26C6DuqrA5nM9KchWBdRa6Us+7OxNKoazA8GWM5UVTs9PUJrLbulUNybxHud17qHkfSDqoKtEKmP9UFuLNYXgixGQQe5ATKTTYNABngrxFE43g/fXf1OVVD5b+BORbi48MiGjobwINqxGfz3gxVWunNc8s1PMcL94hV2Bw2oW24i5zysyJlCONqJQ2u8cdB1aNlWdXjr+yxLEq1QSJ+ApiXbdn3Nm8+pA3Kwz6XWxD0jj09SKh7JzOqdOemm9iNBzaiLLYMzR02357vnJDcZtCUxBFnPgcI3KAnmRXNSx5synWJGhAeovrhlyaTEAi1W1unyG9pcaxv/OvdrRsSdCTkOL6hylZ9ZnrKCgBxF0mUpXHvCZS6hwKwZyxXRTTo4CrJTNuHMseT7SRlg8eLhkPoGZaNxYO4hvrqqCWJSDPCjEt8ENBWU12PNuRJB6RVX 731nlhAH XIXyFK6rd8J8Dy24m2mH4V/h2el1yjubi/jkFga5uTfLQ5yadKB/3hwk6FUOWbxy8WMyoXYG4nesKa0C4yf3tAL7jc/mY46L/LbkaQC/rorAo9pPEJO/aBtlbi5sVZ5M00Zp8M+K6OwP0yeaIdHRy1KsnrhsbaJPyqcl5/Y90JMQqSh46uRDL3jgywjwdjIndr9kIFQ8PrlSFsBZON74ypv1oNiRiiBByWON2xkdkBTL1wH4w/K2o0pxpf46moWXFzzASCDesWOZ+a06bgw+43x+Z0tKwGVY8Ms+l X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let the host effectively inflate the balloon in access/contiguity-aware way when the guest kernel is compiled with specific kernel config. When the config is enabled and the host requests balloon size change, virtio-balloon adjusts ACMA's max-mem parameter instead of allocating guest pages and put it into the balloon. As a result, the host can use the requested amount of guest memory, so from the host's perspective, the ballooning just works, but in transparent and access/contiguity-aware way. Signed-off-by: SeongJae Park --- drivers/virtio/virtio_balloon.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c index 1f5b3dd31fcf..a954d75789ae 100644 --- a/drivers/virtio/virtio_balloon.c +++ b/drivers/virtio/virtio_balloon.c @@ -472,6 +472,32 @@ static void virtballoon_changed(struct virtio_device *vdev) struct virtio_balloon *vb = vdev->priv; unsigned long flags; +#ifdef CONFIG_ACMA_BALLOON + s64 target; + u32 num_pages; + + + /* Legacy balloon config space is LE, unlike all other devices. */ + virtio_cread_le(vb->vdev, struct virtio_balloon_config, num_pages, + &num_pages); + + /* + * Aligned up to guest page size to avoid inflating and deflating + * balloon endlessly. + */ + target = ALIGN(num_pages, VIRTIO_BALLOON_PAGES_PER_PAGE); + + /* + * If the given new max mem size is larger than current acma's max mem + * size, same to normal max mem adjustment. + * If the given new max mem size is smaller than current acma's max mem + * size, strong aggressiveness is applied while memory for meeting the + * new max mem is met is stolen. + */ + acma_set_max_mem_aggressive(totalram_pages() - target); + return; +#endif + spin_lock_irqsave(&vb->stop_update_lock, flags); if (!vb->stop_update) { start_update_balloon_size(vb);