From patchwork Mon Oct 5 10:55:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 11816463 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3F735139A for ; Mon, 5 Oct 2020 11:00:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B8E4920795 for ; Mon, 5 Oct 2020 11:00:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="cH5LaJR+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B8E4920795 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E1B966B0075; Mon, 5 Oct 2020 07:00:38 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DCA876B0078; Mon, 5 Oct 2020 07:00:38 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C92CF6B007B; Mon, 5 Oct 2020 07:00:38 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0087.hostedemail.com [216.40.44.87]) by kanga.kvack.org (Postfix) with ESMTP id 88F6E6B0075 for ; Mon, 5 Oct 2020 07:00:38 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 21267181AC550 for ; Mon, 5 Oct 2020 11:00:38 +0000 (UTC) X-FDA: 77337578556.07.metal37_1d06995271bd Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id E88381803F9AC for ; Mon, 5 Oct 2020 11:00:37 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=5409d9be5=sjpark@amazon.com,,RULES_HIT:30003:30029:30034:30051:30054:30064:30070:30075:30080:30090,0,RBL:52.95.49.90:@amazon.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10;04yruayphroq19jk69n81sizfbqtpyc6jeodbrntsk44t73wctfrbqb3bedhu8c.s4wnkchmm1xcjtffp5e1ceczehffrdwoqnd1em9kq7a5qxmqwugju61nkkbk9gx.w-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: metal37_1d06995271bd X-Filterd-Recvd-Size: 41351 Received: from smtp-fw-6002.amazon.com (smtp-fw-6002.amazon.com [52.95.49.90]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Mon, 5 Oct 2020 11:00:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1601895637; x=1633431637; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=w601Bo9VNwpxMW2d5Fj37aHP7rXY9b49Q1CByIrNGSU=; b=cH5LaJR+V/X+LHSxT495jqs5ZRmw+j+XX3O/cu6umZ5bT/z13kfK/01b Qek5LVNG9uQJMvNMWOqYJw3ZP5gcMdFGkmHwRvQGj/XTvjFvs3juRtnSr wJ+7K+WniktCLuK0JMKlrlnuIlJu98U03FdaU2erTf4lSvPomWDUQwsht U=; X-IronPort-AV: E=Sophos;i="5.77,338,1596499200"; d="scan'208";a="58015942" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-1e-c7c08562.us-east-1.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 05 Oct 2020 11:00:36 +0000 Received: from EX13D31EUA001.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan3.iad.amazon.com [10.40.163.38]) by email-inbound-relay-1e-c7c08562.us-east-1.amazon.com (Postfix) with ESMTPS id 064B12411CC; Mon, 5 Oct 2020 11:00:24 +0000 (UTC) Received: from u3f2cd687b01c55.ant.amazon.com (10.43.160.146) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 5 Oct 2020 11:00:07 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH v21 14/18] tools: Introduce a minimal user-space tool for DAMON Date: Mon, 5 Oct 2020 12:55:18 +0200 Message-ID: <20201005105522.23841-15-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201005105522.23841-1-sjpark@amazon.com> References: <20201005105522.23841-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.146] X-ClientProxiedBy: EX13d09UWC002.ant.amazon.com (10.43.162.102) To EX13D31EUA001.ant.amazon.com (10.43.165.15) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park 'damon-dbgfs' provides simple user space interface for DAMON, but using the interface for complex usages could require annoying repetitive works. Writing a user space data access monitoring applications on top of the debugfs interface and using the application would be better for such complex cases. This commit introduces a reference implementation of such user space application built on top of the debugfs interface, namely 'DAMon Operator' (DAMO). It contains a shallow wrapper python script of the debugfs interface and various visualization of the monitoring results convenient user interface. Note that it is initially aimed to be used for minimal reference of the 'damon-dbgfs' interface and for debugging of the DAMON itself. Signed-off-by: SeongJae Park --- tools/damon/.gitignore | 1 + tools/damon/_damon.py | 130 ++++++++++++++ tools/damon/_dist.py | 35 ++++ tools/damon/_recfile.py | 23 +++ tools/damon/bin2txt.py | 67 +++++++ tools/damon/damo | 37 ++++ tools/damon/heats.py | 362 ++++++++++++++++++++++++++++++++++++++ tools/damon/nr_regions.py | 91 ++++++++++ tools/damon/record.py | 135 ++++++++++++++ tools/damon/report.py | 45 +++++ tools/damon/wss.py | 100 +++++++++++ 11 files changed, 1026 insertions(+) create mode 100644 tools/damon/.gitignore create mode 100644 tools/damon/_damon.py create mode 100644 tools/damon/_dist.py create mode 100644 tools/damon/_recfile.py create mode 100644 tools/damon/bin2txt.py create mode 100755 tools/damon/damo create mode 100644 tools/damon/heats.py create mode 100644 tools/damon/nr_regions.py create mode 100644 tools/damon/record.py create mode 100644 tools/damon/report.py create mode 100644 tools/damon/wss.py diff --git a/tools/damon/.gitignore b/tools/damon/.gitignore new file mode 100644 index 000000000000..96403d36ff93 --- /dev/null +++ b/tools/damon/.gitignore @@ -0,0 +1 @@ +__pycache__/* diff --git a/tools/damon/_damon.py b/tools/damon/_damon.py new file mode 100644 index 000000000000..1f6a292e8c25 --- /dev/null +++ b/tools/damon/_damon.py @@ -0,0 +1,130 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +""" +Contains core functions for DAMON debugfs control. +""" + +import os +import subprocess + +debugfs_attrs = None +debugfs_record = None +debugfs_target_ids = None +debugfs_monitor_on = None + +def set_target_id(tid): + with open(debugfs_target_ids, 'w') as f: + f.write('%s\n' % tid) + +def turn_damon(on_off): + return subprocess.call("echo %s > %s" % (on_off, debugfs_monitor_on), + shell=True, executable="/bin/bash") + +def is_damon_running(): + with open(debugfs_monitor_on, 'r') as f: + return f.read().strip() == 'on' + +class Attrs: + sample_interval = None + aggr_interval = None + regions_update_interval = None + min_nr_regions = None + max_nr_regions = None + rbuf_len = None + rfile_path = None + + def __init__(self, s, a, r, n, x, l, f): + self.sample_interval = s + self.aggr_interval = a + self.regions_update_interval = r + self.min_nr_regions = n + self.max_nr_regions = x + self.rbuf_len = l + self.rfile_path = f + + def __str__(self): + return "%s %s %s %s %s %s %s" % (self.sample_interval, + self.aggr_interval, self.regions_update_interval, + self.min_nr_regions, self.max_nr_regions, self.rbuf_len, + self.rfile_path) + + def attr_str(self): + return "%s %s %s %s %s " % (self.sample_interval, self.aggr_interval, + self.regions_update_interval, self.min_nr_regions, + self.max_nr_regions) + + def record_str(self): + return '%s %s ' % (self.rbuf_len, self.rfile_path) + + def apply(self): + ret = subprocess.call('echo %s > %s' % (self.attr_str(), debugfs_attrs), + shell=True, executable='/bin/bash') + if ret: + return ret + ret = subprocess.call('echo %s > %s' % (self.record_str(), + debugfs_record), shell=True, executable='/bin/bash') + if ret: + return ret + +def current_attrs(): + with open(debugfs_attrs, 'r') as f: + attrs = f.read().split() + attrs = [int(x) for x in attrs] + + with open(debugfs_record, 'r') as f: + rattrs = f.read().split() + attrs.append(int(rattrs[0])) + attrs.append(rattrs[1]) + + return Attrs(*attrs) + +def chk_update_debugfs(debugfs): + global debugfs_attrs + global debugfs_record + global debugfs_target_ids + global debugfs_monitor_on + + debugfs_damon = os.path.join(debugfs, 'damon') + debugfs_attrs = os.path.join(debugfs_damon, 'attrs') + debugfs_record = os.path.join(debugfs_damon, 'record') + debugfs_target_ids = os.path.join(debugfs_damon, 'target_ids') + debugfs_monitor_on = os.path.join(debugfs_damon, 'monitor_on') + + if not os.path.isdir(debugfs_damon): + print("damon debugfs dir (%s) not found", debugfs_damon) + exit(1) + + for f in [debugfs_attrs, debugfs_record, debugfs_target_ids, + debugfs_monitor_on]: + if not os.path.isfile(f): + print("damon debugfs file (%s) not found" % f) + exit(1) + +def cmd_args_to_attrs(args): + "Generate attributes with specified arguments" + sample_interval = args.sample + aggr_interval = args.aggr + regions_update_interval = args.updr + min_nr_regions = args.minr + max_nr_regions = args.maxr + rbuf_len = args.rbuf + if not os.path.isabs(args.out): + args.out = os.path.join(os.getcwd(), args.out) + rfile_path = args.out + return Attrs(sample_interval, aggr_interval, regions_update_interval, + min_nr_regions, max_nr_regions, rbuf_len, rfile_path) + +def set_attrs_argparser(parser): + parser.add_argument('-d', '--debugfs', metavar='', type=str, + default='/sys/kernel/debug', help='debugfs mounted path') + parser.add_argument('-s', '--sample', metavar='', type=int, + default=5000, help='sampling interval') + parser.add_argument('-a', '--aggr', metavar='', type=int, + default=100000, help='aggregate interval') + parser.add_argument('-u', '--updr', metavar='', type=int, + default=1000000, help='regions update interval') + parser.add_argument('-n', '--minr', metavar='<# regions>', type=int, + default=10, help='minimal number of regions') + parser.add_argument('-m', '--maxr', metavar='<# regions>', type=int, + default=1000, help='maximum number of regions') diff --git a/tools/damon/_dist.py b/tools/damon/_dist.py new file mode 100644 index 000000000000..6435f98f4275 --- /dev/null +++ b/tools/damon/_dist.py @@ -0,0 +1,35 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import os +import struct +import subprocess + +def access_patterns(f): + nr_regions = struct.unpack('I', f.read(4))[0] + + patterns = [] + for r in range(nr_regions): + saddr = struct.unpack('L', f.read(8))[0] + eaddr = struct.unpack('L', f.read(8))[0] + nr_accesses = struct.unpack('I', f.read(4))[0] + patterns.append([eaddr - saddr, nr_accesses]) + return patterns + +def plot_dist(data_file, output_file, xlabel, ylabel): + terminal = output_file.split('.')[-1] + if not terminal in ['pdf', 'jpeg', 'png', 'svg']: + os.remove(data_file) + print("Unsupported plot output type.") + exit(-1) + + gnuplot_cmd = """ + set term %s; + set output '%s'; + set key off; + set xlabel '%s'; + set ylabel '%s'; + plot '%s' with linespoints;""" % (terminal, output_file, xlabel, ylabel, + data_file) + subprocess.call(['gnuplot', '-e', gnuplot_cmd]) + os.remove(data_file) diff --git a/tools/damon/_recfile.py b/tools/damon/_recfile.py new file mode 100644 index 000000000000..45dd8ffdb5ae --- /dev/null +++ b/tools/damon/_recfile.py @@ -0,0 +1,23 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import struct + +fmt_version = 0 + +def set_fmt_version(f): + global fmt_version + + mark = f.read(16) + if mark == b'damon_recfmt_ver': + fmt_version = struct.unpack('i', f.read(4))[0] + else: + fmt_version = 0 + f.seek(0) + return fmt_version + +def target_id(f): + if fmt_version == 1: + return struct.unpack('i', f.read(4))[0] + else: + return struct.unpack('L', f.read(8))[0] diff --git a/tools/damon/bin2txt.py b/tools/damon/bin2txt.py new file mode 100644 index 000000000000..79516c72f449 --- /dev/null +++ b/tools/damon/bin2txt.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import argparse +import os +import struct +import sys + +import _recfile + +def parse_time(bindat): + "bindat should be 16 bytes" + sec = struct.unpack('l', bindat[0:8])[0] + nsec = struct.unpack('l', bindat[8:16])[0] + return sec * 1000000000 + nsec; + +def pr_region(f): + saddr = struct.unpack('L', f.read(8))[0] + eaddr = struct.unpack('L', f.read(8))[0] + nr_accesses = struct.unpack('I', f.read(4))[0] + print("%012x-%012x(%10d):\t%d" % + (saddr, eaddr, eaddr - saddr, nr_accesses)) + +def pr_task_info(f): + target_id = _recfile.target_id(f) + print("target_id: ", target_id) + nr_regions = struct.unpack('I', f.read(4))[0] + print("nr_regions: ", nr_regions) + for r in range(nr_regions): + pr_region(f) + +def set_argparser(parser): + parser.add_argument('--input', '-i', type=str, metavar='', + default='damon.data', help='input file name') + +def main(args=None): + if not args: + parser = argparse.ArgumentParser() + set_argparser(parser) + args = parser.parse_args() + + file_path = args.input + + if not os.path.isfile(file_path): + print('input file (%s) is not exist' % file_path) + exit(1) + + with open(file_path, 'rb') as f: + _recfile.set_fmt_version(f) + start_time = None + while True: + timebin = f.read(16) + if len(timebin) != 16: + break + time = parse_time(timebin) + if not start_time: + start_time = time + print("start_time: ", start_time) + print("rel time: %16d" % (time - start_time)) + nr_tasks = struct.unpack('I', f.read(4))[0] + print("nr_tasks: ", nr_tasks) + for t in range(nr_tasks): + pr_task_info(f) + print("") + +if __name__ == '__main__': + main() diff --git a/tools/damon/damo b/tools/damon/damo new file mode 100755 index 000000000000..58e1099ae5fc --- /dev/null +++ b/tools/damon/damo @@ -0,0 +1,37 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import argparse + +import record +import report + +class SubCmdHelpFormatter(argparse.RawDescriptionHelpFormatter): + def _format_action(self, action): + parts = super(argparse.RawDescriptionHelpFormatter, + self)._format_action(action) + # skip sub parsers help + if action.nargs == argparse.PARSER: + parts = '\n'.join(parts.split('\n')[1:]) + return parts + +parser = argparse.ArgumentParser(formatter_class=SubCmdHelpFormatter) + +subparser = parser.add_subparsers(title='command', dest='command', + metavar='') +subparser.required = True + +parser_record = subparser.add_parser('record', + help='record data accesses of the given target processes') +record.set_argparser(parser_record) + +parser_report = subparser.add_parser('report', + help='report the recorded data accesses in the specified form') +report.set_argparser(parser_report) + +args = parser.parse_args() + +if args.command == 'record': + record.main(args) +elif args.command == 'report': + report.main(args) diff --git a/tools/damon/heats.py b/tools/damon/heats.py new file mode 100644 index 000000000000..78a2a793f50e --- /dev/null +++ b/tools/damon/heats.py @@ -0,0 +1,362 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +""" +Transform binary trace data into human readable text that can be used for +heatmap drawing, or directly plot the data in a heatmap format. + +Format of the text is: + +