From patchwork Wed Mar 27 15:57:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kemeng Shi X-Patchwork-Id: 13605701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E26AC47DD9 for ; Wed, 27 Mar 2024 07:02:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9BE466B009D; Wed, 27 Mar 2024 03:01:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A6DD6B009F; Wed, 27 Mar 2024 03:01:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 343A46B00A4; Wed, 27 Mar 2024 03:01:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 03BA66B00A3 for ; Wed, 27 Mar 2024 03:01:57 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D17181409DE for ; Wed, 27 Mar 2024 07:01:56 +0000 (UTC) X-FDA: 81941924232.17.D74ADFE Received: from dggsgout11.his.huawei.com (unknown [45.249.212.51]) by imf21.hostedemail.com (Postfix) with ESMTP id DFB491C0004 for ; Wed, 27 Mar 2024 07:01:53 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of shikemeng@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=shikemeng@huaweicloud.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711522914; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1oNGHV3l3pEb2H6bSXX4d7sGy1z0hmhBCBFfC5+ZDzI=; b=quHOrPEpuESXWdNaeJAYJ+/Ds8G9LdHZMTDVPdBJEiD9tE9IxVDJQ1yYvmvvoecSUJ5n/i G9p9O2JnuLt5Yezor9LyxUJfIat82e7gFsjsVdr3xqhiy2v7v/wY8SqXNe/Q2Yjaw18tl2 E1C05a+u57XBy5otVbY+bXhnNzZ0u7U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711522914; a=rsa-sha256; cv=none; b=XFO3KZkbe7hhvlAIzSWM/Zj76SYhN3Awc9+ciF8TLNiAa53WiXfMz6W0jzYQPcrW6kX3Er N7iboB57rP47bbAFAub/aSQ1aCSrWV/ssXLQfb04ULY/mn8gHiqAqOv2/ZOZF+Sko7MLxQ 4Tv9kMiMm1MrNnSNvpNtB0qvQFwBsPU= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of shikemeng@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=shikemeng@huaweicloud.com; dmarc=none Received: from mail.maildlp.com (unknown [172.19.163.216]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4V4Hf05nc3z4f3kkK for ; Wed, 27 Mar 2024 15:01:44 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.75]) by mail.maildlp.com (Postfix) with ESMTP id D5CC51A017A for ; Wed, 27 Mar 2024 15:01:48 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.101.6]) by APP2 (Coremail) with SMTP id Syh0CgAnSQxYxANmi6+LIQ--.50310S6; Wed, 27 Mar 2024 15:01:48 +0800 (CST) From: Kemeng Shi To: akpm@linux-foundation.org, willy@infradead.org, jack@suse.cz, bfoster@redhat.com, tj@kernel.org Cc: dsterba@suse.com, mjguzik@gmail.com, dhowells@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v2 4/6] writeback: add wb_monitor.py script to monitor writeback info on bdi Date: Wed, 27 Mar 2024 23:57:49 +0800 Message-Id: <20240327155751.3536-5-shikemeng@huaweicloud.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20240327155751.3536-1-shikemeng@huaweicloud.com> References: <20240327155751.3536-1-shikemeng@huaweicloud.com> MIME-Version: 1.0 X-CM-TRANSID: Syh0CgAnSQxYxANmi6+LIQ--.50310S6 X-Coremail-Antispam: 1UD129KBjvJXoW3Xr1UCrWrWFWxXw1DJFy7Jrb_yoWxJr1fpF s8Aw13Ar1xZa4xJrnY9a40vry5Cws5Cr17XrW7ArWakan8Wa4FyryrCFyUAry3Cr9rJ39x X3ya93y8KFWjgFJanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUPmb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M280x2IEY4vEnII2IxkI6r1a6r45M2 8IrcIa0xkI8VA2jI8067AKxVWUAVCq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAv FVAK0II2c7xJM28CjxkF64kEwVA0rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVW7JVWDJw A2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE 3s1l84ACjcxK6I8E87Iv6xkF7I0E14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr2 1l5I8CrVACY4xI64kE6c02F40Ex7xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv 67AKxVWUJVW8JwAm72CE4IkC6x0Yz7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7 CjxVA2Y2ka0xkIwI1l42xK82IYc2Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2Iq xVAqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r 1q6r43MIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY 6xkF7I0E14v26F4j6r4UJwCI42IY6xAIw20EY4v20xvaj40_Jr0_JF4lIxAIcVC2z280aV AFwI0_Jr0_Gr1lIxAIcVC2z280aVCY1x0267AKxVW8JVW8JrUvcSsGvfC2KfnxnUUI43ZE Xa7IU0TqcUUUUUU== X-CM-SenderInfo: 5vklyvpphqwq5kxd4v5lfo033gof0z/ X-Stat-Signature: j5nr9cm7jw7fuiaizjy97hofnquizhoe X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: DFB491C0004 X-Rspam-User: X-HE-Tag: 1711522913-650321 X-HE-Meta: U2FsdGVkX19T/HNCvzsOcGejyASYmR1N9jsvz7dxy8RejJOfyRc3LMnzGio4zy7OsCYNTTBi2KfMT5HXhmDPvgFXpFVjbUH/xpYG8PynNFjGnbdZ1xADZU3jli438KtGiMtJpK8p69DPxr2v4fRZGFt8RsCnCZ0ee3nRINVNfObnKOk63vWzq/PfD05HD7i74UGW8xYasOBhrw7Gl3+sG57zS6kHU8/fxxvehbfnO5Hx1KQm2UIJ/1QYle1j7BUwpUj9ZNwmGD5tsiQHI23WtHD5A4dKX7Kw5nhN1MhnDmhvw/uY/PgfG89ot6NO5chjNqhXx9laXpWop0EdDopEcsUh+OJ9Fd/S19ArixAznhGKjfFfANw9sNgCaZdFGj9OGBPeTr4akAdORQSgP6F8Qp9rLg0vM5LariNabiozwNxFZLZIsVmGSh95K+m3IqcZavM/7EA/SvreeP49MTw6ba7RB+KJ8ITZuxwgzx82xhmyOP78A6tGDxCN4x52J33iwBGws35iihLtTJMd7mUjmGSgr6c7J87gQEpIYIZZDXiC59pwiRyTheVEyCCLqBQlTbcU4mv035mzW8ZuE/6Kla0Yjk0JI6COw0LcsN+f+iyRxN/NI+1OXCPwSLsrQmraGib9TrTgLT4yZAvUjoVEG2CwYMiNVdwgbhq1ExTS4Y7B7hfp25idAgTeJPo+u4k+wn9W0IclwAkbKGg3+oBoCqz0ZoMEmhCinvrwhoQT4vi8uKcCT9gdo8bPUDfmDlV4JNENmtDX2xTiekSVpHxXWiWdIphJf/kBKYD7SWddoccnmV5Ax/G/Zoz17r4b9WLOjScyAsDYd0pnBKxriEnyEICWbCK2xZzei/stLHBGuU1wPorRiRJs+7u7p09HgDSC/WNBT/zvyW8vw+LZvlaYkO658SWfDjQXRRkY/PCtc6bDWVtg3WMngUR0kO//glo1o9qk3LgF3OnRxzKvve0 QvLBwn2t vbeW29eys34AZ8B6RKuqHbh3b6TFHmkMAI5TwZ40QKa24U/WriqbM0gLVRU+fvkI8yz0blBcC2+YQuljnri/IzSshXQv2DWIv55SWolNRxMRWNXWpfQ0vQxt/pMWa+LEBbTHH6Umm7wH+0B8YCqKppAjEvbROSGMpujFIjTyPgmhB8oRMsnqkC59xG7oHjL0y/cAUhfgKK5uVdHBaNgpVY/uaK4PVDigNmcJ99PzHwDMbh2gQ72v1SVOknWuPsuvAvOyQ2I18CPf0RRdhvR6FpgBrzokvwF+soGNJvjrUZMUjkQf5lm2yvRhcYLyhPdgoQV7BYjiYUWCjWpU3as9wugANDEBcY2ezkAayAQDsFCLTg91WANdXcI5bWkmktpIPiiLI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000091, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add wb_monitor.py script to monitor writeback information on backing dev which makes it easier and more convenient to observe writeback behaviors of running system. The wb_monitor.py script is written based on wq_monitor.py. Following domain hierarchy is tested: global domain (320G) / \ cgroup domain1(10G) cgroup domain2(10G) | | bdi wb1 wb2 The wb_monitor.py script output is as following: ./wb_monitor.py 252:16 -c writeback reclaimable dirtied written avg_bw 252:16_1 0 0 0 0 102400 252:16_4284 672 820064 9230368 8410304 685612 252:16_4325 896 819840 10491264 9671648 652348 252:16 1568 1639904 19721632 18081952 1440360 writeback reclaimable dirtied written avg_bw 252:16_1 0 0 0 0 102400 252:16_4284 672 820064 9230368 8410304 685612 252:16_4325 896 819840 10491264 9671648 652348 252:16 1568 1639904 19721632 18081952 1440360 ... Signed-off-by: Kemeng Shi Suggested-by: Tejun Heo --- tools/writeback/wb_monitor.py | 172 ++++++++++++++++++++++++++++++++++ 1 file changed, 172 insertions(+) create mode 100644 tools/writeback/wb_monitor.py diff --git a/tools/writeback/wb_monitor.py b/tools/writeback/wb_monitor.py new file mode 100644 index 000000000000..5e3591f1f9a9 --- /dev/null +++ b/tools/writeback/wb_monitor.py @@ -0,0 +1,172 @@ +#!/usr/bin/env drgn +# +# Copyright (C) 2024 Kemeng Shi +# Copyright (C) 2024 Huawei Inc + +desc = """ +This is a drgn script based on wq_monitor.py to monitor writeback info on +backing dev. For more info on drgn, visit https://github.com/osandov/drgn. + + writeback(kB) Amount of dirty pages are currently being written back to + disk. + + reclaimable(kB) Amount of pages are currently reclaimable. + + dirtied(kB) Amount of pages have been dirtied. + + wrttien(kB) Amount of dirty pages have been written back to disk. + + avg_wb(kBps) Smoothly estimated write bandwidth of writing dirty pages + back to disk. +""" + +import signal +import re +import time +import json + +import drgn +from drgn.helpers.linux.list import list_for_each_entry + +import argparse +parser = argparse.ArgumentParser(description=desc, + formatter_class=argparse.RawTextHelpFormatter) +parser.add_argument('bdi', metavar='REGEX', nargs='*', + help='Target backing device name patterns (all if empty)') +parser.add_argument('-i', '--interval', metavar='SECS', type=float, default=1, + help='Monitoring interval (0 to print once and exit)') +parser.add_argument('-j', '--json', action='store_true', + help='Output in json') +parser.add_argument('-c', '--cgroup', action='store_true', + help='show writeback of bdi in cgroup') +args = parser.parse_args() + +bdi_list = prog['bdi_list'] + +WB_RECLAIMABLE = prog['WB_RECLAIMABLE'] +WB_WRITEBACK = prog['WB_WRITEBACK'] +WB_DIRTIED = prog['WB_DIRTIED'] +WB_WRITTEN = prog['WB_WRITTEN'] +NR_WB_STAT_ITEMS = prog['NR_WB_STAT_ITEMS'] + +PAGE_SHIFT = prog['PAGE_SHIFT'] + +def K(x): + return x << (PAGE_SHIFT - 10) + +class Stats: + def dict(self, now): + return { 'timestamp' : now, + 'name' : self.name, + 'writeback' : self.stats[WB_WRITEBACK], + 'reclaimable' : self.stats[WB_RECLAIMABLE], + 'dirtied' : self.stats[WB_DIRTIED], + 'written' : self.stats[WB_WRITTEN], + 'avg_wb' : self.avg_bw, } + + def table_header_str(): + return f'{"":>16} {"writeback":>10} {"reclaimable":>12} ' \ + f'{"dirtied":>9} {"written":>9} {"avg_bw":>9}' + + def table_row_str(self): + out = f'{self.name[-16:]:16} ' \ + f'{self.stats[WB_WRITEBACK]:10} ' \ + f'{self.stats[WB_RECLAIMABLE]:12} ' \ + f'{self.stats[WB_DIRTIED]:9} ' \ + f'{self.stats[WB_WRITTEN]:9} ' \ + f'{self.avg_bw:9} ' + return out + + def show_header(): + if Stats.table_fmt: + print() + print(Stats.table_header_str()) + + def show_stats(self): + if Stats.table_fmt: + print(self.table_row_str()) + else: + print(self.dict(Stats.now)) + +class WbStats(Stats): + def __init__(self, wb): + bdi_name = wb.bdi.dev_name.string_().decode() + # avoid to use bdi.wb.memcg_css which is only defined when + # CONFIG_CGROUP_WRITEBACK is enabled + if wb == wb.bdi.wb.address_of_(): + ino = "1" + else: + ino = str(wb.memcg_css.cgroup.kn.id.value_()) + self.name = bdi_name + '_' + ino + + self.stats = [0] * NR_WB_STAT_ITEMS + for i in range(NR_WB_STAT_ITEMS): + if wb.stat[i].count >= 0: + self.stats[i] = int(K(wb.stat[i].count)) + else: + self.stats[i] = 0 + + self.avg_bw = int(K(wb.avg_write_bandwidth)) + +class BdiStats(Stats): + def __init__(self, bdi): + self.name = bdi.dev_name.string_().decode() + self.stats = [0] * NR_WB_STAT_ITEMS + self.avg_bw = 0 + + def collectStats(self, wb_stats): + for i in range(NR_WB_STAT_ITEMS): + self.stats[i] += wb_stats.stats[i] + + self.avg_bw += wb_stats.avg_bw + +exit_req = False + +def sigint_handler(signr, frame): + global exit_req + exit_req = True + +def main(): + # handle args + Stats.table_fmt = not args.json + interval = args.interval + cgroup = args.cgroup + + re_str = None + if args.bdi: + for r in args.bdi: + if re_str is None: + re_str = r + else: + re_str += '|' + r + + filter_re = re.compile(re_str) if re_str else None + + # monitoring loop + signal.signal(signal.SIGINT, sigint_handler) + + while not exit_req: + Stats.now = time.time() + + Stats.show_header() + for bdi in list_for_each_entry('struct backing_dev_info', bdi_list.address_of_(), 'bdi_list'): + bdi_stats = BdiStats(bdi) + if filter_re and not filter_re.search(bdi_stats.name): + continue + + for wb in list_for_each_entry('struct bdi_writeback', bdi.wb_list.address_of_(), 'bdi_node'): + wb_stats = WbStats(wb) + bdi_stats.collectStats(wb_stats) + if cgroup: + wb_stats.show_stats() + + bdi_stats.show_stats() + if cgroup and Stats.table_fmt: + print() + + if interval == 0: + break + time.sleep(interval) + +if __name__ == "__main__": + main()