From patchwork Mon Jul 6 11:53:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 11645539 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DCCD01398 for ; Mon, 6 Jul 2020 11:57:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 66FCE20C56 for ; Mon, 6 Jul 2020 11:57:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="MJgP8sTU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 66FCE20C56 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8E1CD6B0023; Mon, 6 Jul 2020 07:57:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8B7896B0024; Mon, 6 Jul 2020 07:57:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 75CAE6B0025; Mon, 6 Jul 2020 07:57:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0223.hostedemail.com [216.40.44.223]) by kanga.kvack.org (Postfix) with ESMTP id 584AF6B0023 for ; Mon, 6 Jul 2020 07:57:16 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 15159180AD802 for ; Mon, 6 Jul 2020 11:57:16 +0000 (UTC) X-FDA: 77007500472.15.beds25_2e0613d26eab Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id D55F71814B0C9 for ; Mon, 6 Jul 2020 11:57:15 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=4497ae310=sjpark@amazon.com,,RULES_HIT:30003:30029:30034:30054:30064:30070:30075:30090,0,RBL:72.21.198.25:@amazon.com:.lbl8.mailshell.net-62.18.0.100 66.10.201.10;04ygctmhij56hyz6buo8t1pg3bimqocxxzkxr4j76k44t73wctgfzqnqst85zar.rdbcae8igxszk6qtp5e1cecze6jkzpakci4jzrq8ufoxarq7naapkt3uz3pjykb.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: beds25_2e0613d26eab X-Filterd-Recvd-Size: 39791 Received: from smtp-fw-4101.amazon.com (smtp-fw-4101.amazon.com [72.21.198.25]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Mon, 6 Jul 2020 11:57:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1594036636; x=1625572636; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=UsisRMf/wmr9WtPXvRpmUVPBbxX+DidK88B7f4yymVc=; b=MJgP8sTUD2nVvITreKVsdp+5QqQkkZtEkw564k2VWibk8A0uNyGw0CMi zSNdRyyywHw5dNobpBXVRU3e1Y9JVXjQtB3cT8K4MruDqPljxV+6pczQ4 AzX2BRyYvYE3JWqO6BCn/e6nc11beyACY5NwehTPRwoHvuEYe5d4z9b7Z E=; IronPort-SDR: V9Mkv60XvoVqPwuwzrGziduELnM7xt6P8UEGuuJeYYmz+sJGBEUH6uAq/m35chI/0EJZZwuOsm n3Bz6yqZYVFQ== X-IronPort-AV: E=Sophos;i="5.75,318,1589241600"; d="scan'208";a="40364215" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-1c1b5cdd.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 06 Jul 2020 11:57:12 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2a-1c1b5cdd.us-west-2.amazon.com (Postfix) with ESMTPS id BF80DA243E; Mon, 6 Jul 2020 11:57:08 +0000 (UTC) Received: from EX13D31EUA004.ant.amazon.com (10.43.165.161) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 6 Jul 2020 11:57:08 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.160.100) by EX13D31EUA004.ant.amazon.com (10.43.165.161) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 6 Jul 2020 11:56:51 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH v17 10/15] tools: Introduce a minimal user-space tool for DAMON Date: Mon, 6 Jul 2020 13:53:17 +0200 Message-ID: <20200706115322.29598-11-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200706115322.29598-1-sjpark@amazon.com> References: <20200706115322.29598-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.100] X-ClientProxiedBy: EX13D21UWA003.ant.amazon.com (10.43.160.184) To EX13D31EUA004.ant.amazon.com (10.43.165.161) X-Rspamd-Queue-Id: D55F71814B0C9 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park This commit imtroduces a shallow wrapper python script, ``/tools/damon/damo`` that provides more convenient interface. Note that it is only aimed to be used for minimal reference of the DAMON's debugfs interfaces and for debugging of the DAMON itself. Signed-off-by: SeongJae Park --- tools/damon/.gitignore | 1 + tools/damon/_damon.py | 129 ++++++++++++++ tools/damon/_dist.py | 36 ++++ tools/damon/_recfile.py | 23 +++ tools/damon/bin2txt.py | 67 +++++++ tools/damon/damo | 37 ++++ tools/damon/heats.py | 362 ++++++++++++++++++++++++++++++++++++++ tools/damon/nr_regions.py | 91 ++++++++++ tools/damon/record.py | 106 +++++++++++ tools/damon/report.py | 45 +++++ tools/damon/wss.py | 97 ++++++++++ 11 files changed, 994 insertions(+) create mode 100644 tools/damon/.gitignore create mode 100644 tools/damon/_damon.py create mode 100644 tools/damon/_dist.py create mode 100644 tools/damon/_recfile.py create mode 100644 tools/damon/bin2txt.py create mode 100755 tools/damon/damo create mode 100644 tools/damon/heats.py create mode 100644 tools/damon/nr_regions.py create mode 100644 tools/damon/record.py create mode 100644 tools/damon/report.py create mode 100644 tools/damon/wss.py diff --git a/tools/damon/.gitignore b/tools/damon/.gitignore new file mode 100644 index 000000000000..96403d36ff93 --- /dev/null +++ b/tools/damon/.gitignore @@ -0,0 +1 @@ +__pycache__/* diff --git a/tools/damon/_damon.py b/tools/damon/_damon.py new file mode 100644 index 000000000000..2a08468ad27e --- /dev/null +++ b/tools/damon/_damon.py @@ -0,0 +1,129 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +""" +Contains core functions for DAMON debugfs control. +""" + +import os +import subprocess + +debugfs_attrs = None +debugfs_record = None +debugfs_pids = None +debugfs_monitor_on = None + +def set_target_pid(pid): + return subprocess.call('echo %s > %s' % (pid, debugfs_pids), shell=True, + executable='/bin/bash') + +def turn_damon(on_off): + return subprocess.call("echo %s > %s" % (on_off, debugfs_monitor_on), + shell=True, executable="/bin/bash") + +def is_damon_running(): + with open(debugfs_monitor_on, 'r') as f: + return f.read().strip() == 'on' + +class Attrs: + sample_interval = None + aggr_interval = None + regions_update_interval = None + min_nr_regions = None + max_nr_regions = None + rbuf_len = None + rfile_path = None + + def __init__(self, s, a, r, n, x, l, f): + self.sample_interval = s + self.aggr_interval = a + self.regions_update_interval = r + self.min_nr_regions = n + self.max_nr_regions = x + self.rbuf_len = l + self.rfile_path = f + + def __str__(self): + return "%s %s %s %s %s %s %s" % (self.sample_interval, + self.aggr_interval, self.regions_update_interval, + self.min_nr_regions, self.max_nr_regions, self.rbuf_len, + self.rfile_path) + + def attr_str(self): + return "%s %s %s %s %s " % (self.sample_interval, self.aggr_interval, + self.regions_update_interval, self.min_nr_regions, + self.max_nr_regions) + + def record_str(self): + return '%s %s ' % (self.rbuf_len, self.rfile_path) + + def apply(self): + ret = subprocess.call('echo %s > %s' % (self.attr_str(), debugfs_attrs), + shell=True, executable='/bin/bash') + if ret: + return ret + ret = subprocess.call('echo %s > %s' % (self.record_str(), + debugfs_record), shell=True, executable='/bin/bash') + if ret: + return ret + +def current_attrs(): + with open(debugfs_attrs, 'r') as f: + attrs = f.read().split() + attrs = [int(x) for x in attrs] + + with open(debugfs_record, 'r') as f: + rattrs = f.read().split() + attrs.append(int(rattrs[0])) + attrs.append(rattrs[1]) + + return Attrs(*attrs) + +def chk_update_debugfs(debugfs): + global debugfs_attrs + global debugfs_record + global debugfs_pids + global debugfs_monitor_on + + debugfs_damon = os.path.join(debugfs, 'damon') + debugfs_attrs = os.path.join(debugfs_damon, 'attrs') + debugfs_record = os.path.join(debugfs_damon, 'record') + debugfs_pids = os.path.join(debugfs_damon, 'pids') + debugfs_monitor_on = os.path.join(debugfs_damon, 'monitor_on') + + if not os.path.isdir(debugfs_damon): + print("damon debugfs dir (%s) not found", debugfs_damon) + exit(1) + + for f in [debugfs_attrs, debugfs_record, debugfs_pids, debugfs_monitor_on]: + if not os.path.isfile(f): + print("damon debugfs file (%s) not found" % f) + exit(1) + +def cmd_args_to_attrs(args): + "Generate attributes with specified arguments" + sample_interval = args.sample + aggr_interval = args.aggr + regions_update_interval = args.updr + min_nr_regions = args.minr + max_nr_regions = args.maxr + rbuf_len = args.rbuf + if not os.path.isabs(args.out): + args.out = os.path.join(os.getcwd(), args.out) + rfile_path = args.out + return Attrs(sample_interval, aggr_interval, regions_update_interval, + min_nr_regions, max_nr_regions, rbuf_len, rfile_path) + +def set_attrs_argparser(parser): + parser.add_argument('-d', '--debugfs', metavar='', type=str, + default='/sys/kernel/debug', help='debugfs mounted path') + parser.add_argument('-s', '--sample', metavar='', type=int, + default=5000, help='sampling interval') + parser.add_argument('-a', '--aggr', metavar='', type=int, + default=100000, help='aggregate interval') + parser.add_argument('-u', '--updr', metavar='', type=int, + default=1000000, help='regions update interval') + parser.add_argument('-n', '--minr', metavar='<# regions>', type=int, + default=10, help='minimal number of regions') + parser.add_argument('-m', '--maxr', metavar='<# regions>', type=int, + default=1000, help='maximum number of regions') diff --git a/tools/damon/_dist.py b/tools/damon/_dist.py new file mode 100644 index 000000000000..9851ec964e5c --- /dev/null +++ b/tools/damon/_dist.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import os +import struct +import subprocess + +def access_patterns(f): + nr_regions = struct.unpack('I', f.read(4))[0] + + patterns = [] + for r in range(nr_regions): + saddr = struct.unpack('L', f.read(8))[0] + eaddr = struct.unpack('L', f.read(8))[0] + nr_accesses = struct.unpack('I', f.read(4))[0] + patterns.append([eaddr - saddr, nr_accesses]) + return patterns + +def plot_dist(data_file, output_file, xlabel, ylabel): + terminal = output_file.split('.')[-1] + if not terminal in ['pdf', 'jpeg', 'png', 'svg']: + os.remove(data_file) + print("Unsupported plot output type.") + exit(-1) + + gnuplot_cmd = """ + set term %s; + set output '%s'; + set key off; + set xlabel '%s'; + set ylabel '%s'; + plot '%s' with linespoints;""" % (terminal, output_file, xlabel, ylabel, + data_file) + subprocess.call(['gnuplot', '-e', gnuplot_cmd]) + os.remove(data_file) + diff --git a/tools/damon/_recfile.py b/tools/damon/_recfile.py new file mode 100644 index 000000000000..331b4d8165d8 --- /dev/null +++ b/tools/damon/_recfile.py @@ -0,0 +1,23 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import struct + +fmt_version = 0 + +def set_fmt_version(f): + global fmt_version + + mark = f.read(16) + if mark == b'damon_recfmt_ver': + fmt_version = struct.unpack('i', f.read(4))[0] + else: + fmt_version = 0 + f.seek(0) + return fmt_version + +def pid(f): + if fmt_version == 0: + return struct.unpack('L', f.read(8))[0] + else: + return struct.unpack('i', f.read(4))[0] diff --git a/tools/damon/bin2txt.py b/tools/damon/bin2txt.py new file mode 100644 index 000000000000..8b9b57a0d727 --- /dev/null +++ b/tools/damon/bin2txt.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import argparse +import os +import struct +import sys + +import _recfile + +def parse_time(bindat): + "bindat should be 16 bytes" + sec = struct.unpack('l', bindat[0:8])[0] + nsec = struct.unpack('l', bindat[8:16])[0] + return sec * 1000000000 + nsec; + +def pr_region(f): + saddr = struct.unpack('L', f.read(8))[0] + eaddr = struct.unpack('L', f.read(8))[0] + nr_accesses = struct.unpack('I', f.read(4))[0] + print("%012x-%012x(%10d):\t%d" % + (saddr, eaddr, eaddr - saddr, nr_accesses)) + +def pr_task_info(f): + pid = _recfile.pid(f) + print("pid: ", pid) + nr_regions = struct.unpack('I', f.read(4))[0] + print("nr_regions: ", nr_regions) + for r in range(nr_regions): + pr_region(f) + +def set_argparser(parser): + parser.add_argument('--input', '-i', type=str, metavar='', + default='damon.data', help='input file name') + +def main(args=None): + if not args: + parser = argparse.ArgumentParser() + set_argparser(parser) + args = parser.parse_args() + + file_path = args.input + + if not os.path.isfile(file_path): + print('input file (%s) is not exist' % file_path) + exit(1) + + with open(file_path, 'rb') as f: + _recfile.set_fmt_version(f) + start_time = None + while True: + timebin = f.read(16) + if len(timebin) != 16: + break + time = parse_time(timebin) + if not start_time: + start_time = time + print("start_time: ", start_time) + print("rel time: %16d" % (time - start_time)) + nr_tasks = struct.unpack('I', f.read(4))[0] + print("nr_tasks: ", nr_tasks) + for t in range(nr_tasks): + pr_task_info(f) + print("") + +if __name__ == '__main__': + main() diff --git a/tools/damon/damo b/tools/damon/damo new file mode 100755 index 000000000000..58e1099ae5fc --- /dev/null +++ b/tools/damon/damo @@ -0,0 +1,37 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import argparse + +import record +import report + +class SubCmdHelpFormatter(argparse.RawDescriptionHelpFormatter): + def _format_action(self, action): + parts = super(argparse.RawDescriptionHelpFormatter, + self)._format_action(action) + # skip sub parsers help + if action.nargs == argparse.PARSER: + parts = '\n'.join(parts.split('\n')[1:]) + return parts + +parser = argparse.ArgumentParser(formatter_class=SubCmdHelpFormatter) + +subparser = parser.add_subparsers(title='command', dest='command', + metavar='') +subparser.required = True + +parser_record = subparser.add_parser('record', + help='record data accesses of the given target processes') +record.set_argparser(parser_record) + +parser_report = subparser.add_parser('report', + help='report the recorded data accesses in the specified form') +report.set_argparser(parser_report) + +args = parser.parse_args() + +if args.command == 'record': + record.main(args) +elif args.command == 'report': + report.main(args) diff --git a/tools/damon/heats.py b/tools/damon/heats.py new file mode 100644 index 000000000000..99837083874e --- /dev/null +++ b/tools/damon/heats.py @@ -0,0 +1,362 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +""" +Transform binary trace data into human readable text that can be used for +heatmap drawing, or directly plot the data in a heatmap format. + +Format of the text is: + +