From patchwork Tue Oct 6 12:39:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 11818613 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DE09413B2 for ; Tue, 6 Oct 2020 12:43:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 788E22087E for ; Tue, 6 Oct 2020 12:43:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="JliAFBY7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 788E22087E Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9BC026B0071; Tue, 6 Oct 2020 08:43:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 947576B0075; Tue, 6 Oct 2020 08:43:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80F5E900002; Tue, 6 Oct 2020 08:43:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0064.hostedemail.com [216.40.44.64]) by kanga.kvack.org (Postfix) with ESMTP id 56A2C6B0071 for ; Tue, 6 Oct 2020 08:43:28 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C42073640 for ; Tue, 6 Oct 2020 12:43:27 +0000 (UTC) X-FDA: 77341466454.17.toe46_220977c271c6 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 73512180D0181 for ; Tue, 6 Oct 2020 12:43:27 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=54163b472=sjpark@amazon.com,,RULES_HIT:30004:30012:30029:30030:30034:30051:30054:30064:30070:30091,0,RBL:72.21.196.25:@amazon.com:.lbl8.mailshell.net-62.18.0.100 66.10.201.10;04y89eoxn1xnpgpka3gb6y8rse6btop3cse3wgci9tctd88axt8dbi6bncehxjt.t437yneik9864e7mne8kdo6ks8jay7m7mb7armskmqgyuoy8naefqrw9gw3z3ju.r-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: toe46_220977c271c6 X-Filterd-Recvd-Size: 14901 Received: from smtp-fw-2101.amazon.com (smtp-fw-2101.amazon.com [72.21.196.25]) by imf29.hostedemail.com (Postfix) with ESMTP for ; Tue, 6 Oct 2020 12:43:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1601988207; x=1633524207; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=XEjhfcWOE6sCpVcYRXwQLFfPVP/yv/3hCxGv89889MM=; b=JliAFBY7Odd4er7EvMsfWzafksBwdEzTIGrvfJQoYY3IdZcgm1JwPabW EyDKl3vCvBYAMMmvZnz1fExpvG/XSkrMxWmZ58wC7y2iYmJLOlaQoSQST JT8Yh1ebLjenwggE/p7VSyaU8J00VHHHVw11buENUOT79+oOImKP+mUnW k=; X-IronPort-AV: E=Sophos;i="5.77,343,1596499200"; d="scan'208";a="58084739" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-2a-e7be2041.us-west-2.amazon.com) ([10.43.8.2]) by smtp-border-fw-out-2101.iad2.amazon.com with ESMTP; 06 Oct 2020 12:43:23 +0000 Received: from EX13D31EUA004.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2a-e7be2041.us-west-2.amazon.com (Postfix) with ESMTPS id 11581A1C99; Tue, 6 Oct 2020 12:43:20 +0000 (UTC) Received: from u3f2cd687b01c55.ant.amazon.com (10.43.161.237) by EX13D31EUA004.ant.amazon.com (10.43.165.161) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 6 Oct 2020 12:43:01 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC v15 8/8] Docs/admin-guide/mm/damon: Document DAMON-based Operation Schemes Date: Tue, 6 Oct 2020 14:39:31 +0200 Message-ID: <20201006123931.5847-9-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201006123931.5847-1-sjpark@amazon.com> References: <20201006123931.5847-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.237] X-ClientProxiedBy: EX13D18UWA002.ant.amazon.com (10.43.160.199) To EX13D31EUA004.ant.amazon.com (10.43.165.161) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park This commit add description of DAMON-based operation schemes in the DAMON documents. Signed-off-by: SeongJae Park --- Documentation/admin-guide/mm/damon/guide.rst | 41 ++++++- Documentation/admin-guide/mm/damon/start.rst | 11 ++ Documentation/admin-guide/mm/damon/usage.rst | 109 ++++++++++++++++++- Documentation/vm/damon/index.rst | 1 - 4 files changed, 156 insertions(+), 6 deletions(-) diff --git a/Documentation/admin-guide/mm/damon/guide.rst b/Documentation/admin-guide/mm/damon/guide.rst index c51fb843efaa..1f9aa2ebbdb6 100644 --- a/Documentation/admin-guide/mm/damon/guide.rst +++ b/Documentation/admin-guide/mm/damon/guide.rst @@ -53,6 +53,11 @@ heats``. If it shows a simple pattern consists of a small number of memory regions having high contrast of access temperature, you could consider manual `Program Modification`_. +If the access pattern is very frequently changing so that you cannot figure out +what is the performance important region using your human eye, `Automated +DAMON-based Memory Operations`_ might help the case owing to its machine-level +microscope view. + If you still want to absorb more benefits, you should develop `Personalized DAMON Application`_ for your special case. @@ -120,6 +125,36 @@ shows the visualized access patterns of streamcluster workload in PARSEC3 benchmark suite. We can easily identify the 100 MiB sized hot object. +Automated DAMON-based Memory Operations +--------------------------------------- + +Though `Manual Program Optimization` works well in many cases and DAMON can +help it, modifying the source code is not a good option in many cases. First +of all, the source code could be too old or unavailable. And, many workloads +will have complex data access patterns that even hard to distinguish hot memory +objects and cold memory objects with the human eye. Finding the mapping from +the visualized access pattern to the source code and injecting the hinting +system calls inside the code will also be quite challenging. + +By using DAMON-based operation schemes (DAMOS) via ``damo schemes``, you will +be able to easily optimize your workload in such a case. Our example schemes +called 'efficient THP' and 'proactive reclamation' achieved significant speedup +and memory space saves against 25 realistic workloads [2]_. + +That said, note that you need careful tune of the schemes (e.g., target region +size and age) and monitoring attributes for the successful use of this +approach. Because the optimal values of the parameters will be dependent on +each system and workload, misconfiguring the parameters could result in worse +memory management. + +For the tuning, you could measure the performance metrics such as IPC, TLB +misses, and swap in/out events and adjusts the parameters based on their +changes. The total number and the total size of the regions that each scheme +is applied, which are provided via the debugfs interface and the programming +interface can also be useful. Writing a program automating this optimal +parameter could be an option. + + Personalized DAMON Application ------------------------------ @@ -146,9 +181,9 @@ Referencing previously done successful practices could help you getting the sense for this kind of optimizations. There is an academic paper [1]_ reporting the visualized access pattern and manual `Program Modification`_ results for a number of realistic workloads. You can also get -the visualized access patterns [3]_ [4]_ [5]_ and automated DAMON-based memory -operations results for other realistic workloads that collected with latest -version of DAMON [2]_ . +the visualized access patterns [3]_ [4]_ [5]_ and +`Automated DAMON-based Memory Operations`_ results for other realistic +workloads that collected with latest version of DAMON [2]_ . .. [1] https://dl.acm.org/doi/10.1145/3366626.3368125 .. [2] https://damonitor.github.io/test/result/perf/latest/html/ diff --git a/Documentation/admin-guide/mm/damon/start.rst b/Documentation/admin-guide/mm/damon/start.rst index deed2ea2321e..35cf4e4ca6aa 100644 --- a/Documentation/admin-guide/mm/damon/start.rst +++ b/Documentation/admin-guide/mm/damon/start.rst @@ -90,6 +90,17 @@ image files. :: You can show the images in a web page [1]_ . Those made with other realistic workloads are also available [2]_ [3]_ [4]_. + +Data Access Pattern Aware Memory Management +=========================================== + +Below three commands make every memory region of size >=4K that doesn't +accessed for >=60 seconds in your workload to be swapped out. :: + + $ echo "#min-size max-size min-acc max-acc min-age max-age action" > scheme + $ echo "4K max 0 0 60s max pageout" >> scheme + $ damo schemes -c my_thp_scheme + .. [1] https://damonitor.github.io/doc/html/v17/admin-guide/mm/damon/start.html#visualizing-recorded-patterns .. [2] https://damonitor.github.io/test/result/visual/latest/rec.heatmap.1.png.html .. [3] https://damonitor.github.io/test/result/visual/latest/rec.wss_sz.png.html diff --git a/Documentation/admin-guide/mm/damon/usage.rst b/Documentation/admin-guide/mm/damon/usage.rst index a6606d27a559..96278227f925 100644 --- a/Documentation/admin-guide/mm/damon/usage.rst +++ b/Documentation/admin-guide/mm/damon/usage.rst @@ -219,11 +219,70 @@ Similar to that of ``heats --heatmap``, it also supports 'gnuplot' based simple visualization of the distribution via ``--plot`` option. +DAMON-based Operation Schemes +----------------------------- + +The ``schemes`` subcommand allows users to do DAMON-based memory management +optimizations in a few seconds. Similar to ``record``, it receives monitoring +attributes and target. However, in addition to those, ``schemes`` receives +data access pattern-based memory operation schemes, which describes what memory +operation action should be applied to memory regions showing specific data +access pattern. Then, it starts the data access monitoring and automatically +applies the schemes to the targets. + +The operation schemes should be saved in a text file in below format and passed +to ``schemes`` subcommand via ``--schemes`` option. :: + + min-size max-size min-acc max-acc min-age max-age action + +The format also supports comments, several units for size and age of regions, +and human readable action names. Currently supported operation actions are +``willneed``, ``cold``, ``pageout``, ``hugepage`` and ``nohugepage``. Each of +the actions works same to the madvise() system call hints having the name. +Please also note that the range is inclusive (closed interval), and ``0`` for +max values means infinite. Below example schemes are possible. :: + + # format is: + # + # + # B/K/M/G/T for Bytes/KiB/MiB/GiB/TiB + # us/ms/s/m/h/d for micro-seconds/milli-seconds/seconds/minutes/hours/days + # 'min/max' for possible min/max value. + + # if a region keeps a high access frequency for >=100ms, put the region on + # the head of the LRU list (call madvise() with MADV_WILLNEED). + min max 80 max 100ms max willneed + + # if a region keeps a low access frequency for >=200ms and <=one hour, put + # the region on the tail of the LRU list (call madvise() with MADV_COLD). + min max 10 20 200ms 1h cold + + # if a region keeps a very low access frequency for >=60 seconds, swap out + # the region immediately (call madvise() with MADV_PAGEOUT). + min max 0 10 60s max pageout + + # if a region of a size >=2MiB keeps a very high access frequency for + # >=100ms, let the region to use huge pages (call madvise() with + # MADV_HUGEPAGE). + 2M max 90 100 100ms max hugepage + + # If a regions of a size >=2MiB keeps small access frequency for >=100ms, + # avoid the region using huge pages (call madvise() with MADV_NOHUGEPAGE). + 2M max 0 25 100ms max nohugepage + +For example, you can make a running process named 'foo' to use huge pages for +memory regions keeping 2MB or larger size and having very high access frequency +for at least 100 milliseconds using below commands:: + + $ echo "2M max 90 max 100ms max hugepage" > my_thp_scheme + $ ./damo schemes --schemes my_thp_scheme `pidof foo` + + debugfs Interface ================= -DAMON exports four files, ``attrs``, ``target_ids``, ``record``, and -``monitor_on`` under its debugfs directory, ``/damon/``. +DAMON exports five files, ``attrs``, ``target_ids``, ``record``, ``schemes`` +and ``monitor_on`` under its debugfs directory, ``/damon/``. Attributes @@ -280,6 +339,52 @@ saved in ``/damon.data``. :: The recording can be disabled by setting the buffer size zero. +Schemes +------- + +For usual DAMON-based data access aware memory management optimizations, users +would simply want the system to apply a memory management action to a memory +region of a specific size having a specific access frequency for a specific +time. DAMON receives such formalized operation schemes from the user and +applies those to the target processes. It also counts the total number and +size of regions that each scheme is applied. This statistics can be used for +online analysis or tuning of the schemes. + +Users can get and set the schemes by reading from and writing to ``schemes`` +debugfs file. Reading the file also shows the statistics of each scheme. To +the file, each of the schemes should be represented in each line in below form: + + min-size max-size min-acc max-acc min-age max-age action + +Note that the ranges are closed interval. Bytes for the size of regions +(``min-size`` and ``max-size``), number of monitored accesses per aggregate +interval for access frequency (``min-acc`` and ``max-acc``), number of +aggregate intervals for the age of regions (``min-age`` and ``max-age``), and a +predefined integer for memory management actions should be used. The supported +numbers and their meanings are as below. + + - 0: Call ``madvise()`` for the region with ``MADV_WILLNEED`` + - 1: Call ``madvise()`` for the region with ``MADV_COLD`` + - 2: Call ``madvise()`` for the region with ``MADV_PAGEOUT`` + - 3: Call ``madvise()`` for the region with ``MADV_HUGEPAGE`` + - 4: Call ``madvise()`` for the region with ``MADV_NOHUGEPAGE`` + - 5: Do nothing but count the statistics + +You can disable schemes by simply writing an empty string to the file. For +example, below commands applies a scheme saying "If a memory region of size in +[4KiB, 8KiB] is showing accesses per aggregate interval in [0, 5] for aggregate +interval in [10, 20], page out the region", check the entered scheme again, and +finally remove the scheme. :: + + # cd /damon + # echo "4096 8192 0 5 10 20 2" > schemes + # cat schemes + 4096 8192 0 5 10 20 2 0 0 + # echo > schemes + +The last two integers in the 4th line of above example is the total number and +the total size of the regions that the scheme is applied. + Turning On/Off -------------- diff --git a/Documentation/vm/damon/index.rst b/Documentation/vm/damon/index.rst index 17dca3c12aad..69aec1287aaf 100644 --- a/Documentation/vm/damon/index.rst +++ b/Documentation/vm/damon/index.rst @@ -28,4 +28,3 @@ workloads and systems. design eval api - plans