From patchwork Mon Mar 13 20:46:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 13173254 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EECD2C6FD19 for ; Mon, 13 Mar 2023 20:46:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 03F376B0072; Mon, 13 Mar 2023 16:46:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F31616B0074; Mon, 13 Mar 2023 16:46:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF9E46B0075; Mon, 13 Mar 2023 16:46:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D0B2C6B0072 for ; Mon, 13 Mar 2023 16:46:24 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 983A7120E25 for ; Mon, 13 Mar 2023 20:46:24 +0000 (UTC) X-FDA: 80565057888.29.D759671 Received: from 66-220-144-178.mail-mxout.facebook.com (66-220-144-178.mail-mxout.facebook.com [66.220.144.178]) by imf03.hostedemail.com (Postfix) with ESMTP id E87E22001D for ; Mon, 13 Mar 2023 20:46:21 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; dmarc=none; spf=neutral (imf03.hostedemail.com: 66.220.144.178 is neither permitted nor denied by domain of shr@devkernel.io) smtp.mailfrom=shr@devkernel.io ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678740382; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=bo1W4RId4rcCN7xwGzNozWhikgxmlXylFBS07dVXnso=; b=NrGjNPoaTgfIsQbJSS0q38P405/OODGRIAffq6wR3o1EmzyseyOkcuR6+LFQfcyHIg15gf ZlQGvvwvWdCEHQuZB1bY9ZuWy/42wpENjivBQUoP3+XofBEztzK21uswIpD0PHXKeKWVaK xJDbd81VTBDDjoJZajxe/G8N7mtSdfs= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; dmarc=none; spf=neutral (imf03.hostedemail.com: 66.220.144.178 is neither permitted nor denied by domain of shr@devkernel.io) smtp.mailfrom=shr@devkernel.io ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678740382; a=rsa-sha256; cv=none; b=le16YzHLMFYT6Xsz+hqZCsFjTabXM7+e7RvLMNJcmbvecBcBPwHa0wCgVat/B9ApCOmuSj 5Q/jH12aLROj2Lsu53Hze0jf7KQZo49fBj9YwsC0Qk1LPGCmhqLBSIsZ/NlmHzMespcxQf 5YYN1sAQs8qfPeR4/wHLRtLYLMEvkzw= Received: by dev0134.prn3.facebook.com (Postfix, from userid 425415) id B030890C527E; Mon, 13 Mar 2023 13:46:08 -0700 (PDT) From: Stefan Roesch To: kernel-team@fb.com Cc: shr@devkernel.io, linux-mm@kvack.org, rostedt@goodmis.org, linux-trace-kernel@vger.kernel.org, akpm@linux-foundation.org Subject: [PATCH v2] mm: add tracepoints to ksm Date: Mon, 13 Mar 2023 13:46:06 -0700 Message-Id: <20230313204606.3581988-1-shr@devkernel.io> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: E87E22001D X-Stat-Signature: dtzmgth5we5kgeiz5p1dayxbegf3j7a3 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678740381-372073 X-HE-Meta: U2FsdGVkX1+jaghDPrpVLz1KJY/RRe6XVxeAI3t+PpJ17I8hqz54v3nhSUIKp8sp6Kj5jZkfz6TfESlsmPWK+JnMZO7QaWNgLc8kwVSMIMSGndXDBMxJ2+VU3uOsi4DUR4xxMS8I6OjfMNlEoJEsIQ0CMn3KzhKsdsekXjJ4SIpQCZ6Spv8VbI7vsQGO7lHck85YiGyXuaWY3AKbxnaOR/GtMarO4wC7TIl9gQU7o4/lSYn0+fhFCct1zFp6dykhxwm569u506Dm2Or4E2NEMcjLBcjYiqXEvkx4vJG9T1aOO66nhvpTTAdK8SF0gtwc4YYYr6Rf2m0WjrYpOiduJAiV6GIseytBGtSQWxV68m+bBxE4tOfew0lryRAX41Uv+PTOCgoZvL8czv/RmxYz1dQhsXFcqLksK7/hWOfMTClLflh736wRvDw12yvLeTx/ERlUxeliXZCK38weCNWdRaTnfaW8afsOF+RyKlpzgnQHMzvhld5FwjA8tkALxS/kmcNLrB7ZSDs7MMWsDlVcLyts+1e25qIrqibA8g4wUPH6OQhPKCppBjIwY7u6+Q0mDmnd+3Bvdr+UGQugPfWA6z7Atc8/RoFHBpaTjfIW+2wROtPab8sl8aXjiK+zKeiFWZRqHujMIpUHTh6CDA2T90BYkEtu3ZZpTQTt6ps043l0Cf4N+SrHt1u7ZrZTvdWHVESMhubj1xxO9iYTeWRUb7gbmiVBeDKRilZNngSUkDW3GPc3xt6qkvuzwJZk7S7DjsU6eZGQBzBqThNtvhJpQcwU5/Wx5ooLrZ0FNlXGc50UMq6a1z+XqnfZzs5c4lShfyukJwqZ/IZuXjg7/lNPPF9ggHm4V2+xIsFcQseBrWOqvJf9M9Qzcfd/DRtuedik8C7ARTRvzCKnR0aGQz9aTfZxvtdJIYKYRKuTSgUHoynZav3MoE/dBIrTuKu2ZkmxQyQ1ZbiLXOXj0Vp1G+T cpNefZKn RX4VtT7hVObaAx/ynUxv/UASVzrXX7MkxFqPdXAZgP3+wlA2Tb9Y4e9tjdquaD9rdWXWxErkUFir1Zn+yOu79Jjrk7kv6lTUJ1/7sKOu37EFAkfx/ZvDs/7ZWivbcYnCmGY973XVmbxw5yISbLO4CNHGcue+HpvqUBCPjBJKFvNzMf6mOU35AJJ6VYiJ38rLf75KVBooMpWawtgvTso1GJKzHFtDlmvSTJ1rhB+Ht5xATX1G/zY2NKXz6lSOOyqNYIWKC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This adds the following tracepoints to ksm: - start / stop scan - ksm enter / exit - merge a page - merge a page with ksm - remove a page - remove a rmap item This patch has been split off from the RFC patch series "mm: process/cgroup ksm support". Link: https://lkml.kernel.org/r/20230210214645.2720847-1-shr@devkernel.io Signed-off-by: Stefan Roesch Reviewed-by: Steven Rostedt (Google) Signed-off-by: Andrew Morton --- mm/Makefile | 1 + mm/ksm.c | 62 ++++-------- mm/ksm.h | 45 +++++++++ mm/ksm_trace.h | 258 +++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 324 insertions(+), 42 deletions(-) create mode 100644 mm/ksm.h create mode 100644 mm/ksm_trace.h diff --git a/mm/Makefile b/mm/Makefile index 3a08f5d7b178..1cb6ae4404f2 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -35,6 +35,7 @@ KCOV_INSTRUMENT_failslab.o := n CFLAGS_init-mm.o += $(call cc-disable-warning, override-init) CFLAGS_init-mm.o += $(call cc-disable-warning, initializer-overrides) +CFLAGS_ksm.o := -I$(src) mmu-y := nommu.o mmu-$(CONFIG_MMU) := highmem.o memory.o mincore.o \ diff --git a/mm/ksm.c b/mm/ksm.c index 192f29fcc93c..6effc4e0a79f 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -44,6 +44,10 @@ #include #include "internal.h" #include "mm_slot.h" +#include "ksm.h" + +#define CREATE_TRACE_POINTS +#include "ksm_trace.h" #ifdef CONFIG_NUMA #define NUMA(x) (x) @@ -139,46 +143,6 @@ struct ksm_scan { unsigned long seqnr; }; -/** - * struct ksm_stable_node - node of the stable rbtree - * @node: rb node of this ksm page in the stable tree - * @head: (overlaying parent) &migrate_nodes indicates temporarily on that list - * @hlist_dup: linked into the stable_node->hlist with a stable_node chain - * @list: linked into migrate_nodes, pending placement in the proper node tree - * @hlist: hlist head of rmap_items using this ksm page - * @kpfn: page frame number of this ksm page (perhaps temporarily on wrong nid) - * @chain_prune_time: time of the last full garbage collection - * @rmap_hlist_len: number of rmap_item entries in hlist or STABLE_NODE_CHAIN - * @nid: NUMA node id of stable tree in which linked (may not match kpfn) - */ -struct ksm_stable_node { - union { - struct rb_node node; /* when node of stable tree */ - struct { /* when listed for migration */ - struct list_head *head; - struct { - struct hlist_node hlist_dup; - struct list_head list; - }; - }; - }; - struct hlist_head hlist; - union { - unsigned long kpfn; - unsigned long chain_prune_time; - }; - /* - * STABLE_NODE_CHAIN can be any negative number in - * rmap_hlist_len negative range, but better not -1 to be able - * to reliably detect underflows. - */ -#define STABLE_NODE_CHAIN -1024 - int rmap_hlist_len; -#ifdef CONFIG_NUMA - int nid; -#endif -}; - /** * struct ksm_rmap_item - reverse mapping item for virtual addresses * @rmap_list: next rmap_item in mm_slot's singly-linked rmap_list @@ -655,10 +619,12 @@ static void remove_node_from_stable_tree(struct ksm_stable_node *stable_node) BUG_ON(stable_node->rmap_hlist_len < 0); hlist_for_each_entry(rmap_item, &stable_node->hlist, hlist) { - if (rmap_item->hlist.next) + if (rmap_item->hlist.next) { ksm_pages_sharing--; - else + trace_ksm_remove_rmap_item(stable_node, rmap_item, rmap_item->mm); + } else { ksm_pages_shared--; + } rmap_item->mm->ksm_merging_pages--; @@ -679,6 +645,7 @@ static void remove_node_from_stable_tree(struct ksm_stable_node *stable_node) BUILD_BUG_ON(STABLE_NODE_DUP_HEAD <= &migrate_nodes); BUILD_BUG_ON(STABLE_NODE_DUP_HEAD >= &migrate_nodes + 1); + trace_ksm_remove_ksm_page(stable_node); if (stable_node->head == &migrate_nodes) list_del(&stable_node->list); else @@ -1375,6 +1342,8 @@ static int try_to_merge_with_ksm_page(struct ksm_rmap_item *rmap_item, get_anon_vma(vma->anon_vma); out: mmap_read_unlock(mm); + trace_ksm_merge_with_ksm_page(kpage, page_to_pfn(kpage ? kpage : page), + rmap_item, mm, err); return err; } @@ -2122,6 +2091,9 @@ static int try_to_merge_with_kernel_zero_page(struct ksm_rmap_item *rmap_item, if (vma) { err = try_to_merge_one_page(vma, page, ZERO_PAGE(rmap_item->address)); + trace_ksm_merge_one_page( + page_to_pfn(ZERO_PAGE(rmap_item->address)), + rmap_item, mm, err); if (!err) { rmap_item->address |= ZERO_PAGE_FLAG; ksm_zero_pages_sharing++; @@ -2365,6 +2337,8 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page) mm_slot = ksm_scan.mm_slot; if (mm_slot == &ksm_mm_head) { + trace_ksm_start_scan(ksm_scan.seqnr, ksm_rmap_items); + /* * A number of pages can hang around indefinitely on per-cpu * pagevecs, raised page count preventing write_protect_page @@ -2544,6 +2518,7 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page) if (mm_slot != &ksm_mm_head) goto next_mm; + trace_ksm_stop_scan(ksm_scan.seqnr, ksm_rmap_items); ksm_scan.seqnr++; return NULL; } @@ -2696,6 +2671,7 @@ int __ksm_enter(struct mm_struct *mm, int flag) if (needs_wakeup) wake_up_interruptible(&ksm_thread_wait); + trace_ksm_enter(mm); return 0; } @@ -2764,6 +2740,8 @@ void __ksm_exit(struct mm_struct *mm, int flag) mmap_write_lock(mm); mmap_write_unlock(mm); } + + trace_ksm_exit(mm); } struct page *ksm_might_need_to_copy(struct page *page, diff --git a/mm/ksm.h b/mm/ksm.h new file mode 100644 index 000000000000..b5855ac996f7 --- /dev/null +++ b/mm/ksm.h @@ -0,0 +1,45 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __MM_KSM_H__ +#define __MM_KSM_H__ + +/** + * struct ksm_stable_node - node of the stable rbtree + * @node: rb node of this ksm page in the stable tree + * @head: (overlaying parent) &migrate_nodes indicates temporarily on that list + * @hlist_dup: linked into the stable_node->hlist with a stable_node chain + * @list: linked into migrate_nodes, pending placement in the proper node tree + * @hlist: hlist head of rmap_items using this ksm page + * @kpfn: page frame number of this ksm page (perhaps temporarily on wrong nid) + * @chain_prune_time: time of the last full garbage collection + * @rmap_hlist_len: number of rmap_item entries in hlist or STABLE_NODE_CHAIN + * @nid: NUMA node id of stable tree in which linked (may not match kpfn) + */ +struct ksm_stable_node { + union { + struct rb_node node; /* when node of stable tree */ + struct { /* when listed for migration */ + struct list_head *head; + struct { + struct hlist_node hlist_dup; + struct list_head list; + }; + }; + }; + struct hlist_head hlist; + union { + unsigned long kpfn; + unsigned long chain_prune_time; + }; + /* + * STABLE_NODE_CHAIN can be any negative number in + * rmap_hlist_len negative range, but better not -1 to be able + * to reliably detect underflows. + */ +#define STABLE_NODE_CHAIN -1024 + int rmap_hlist_len; +#ifdef CONFIG_NUMA + int nid; +#endif +}; + +#endif diff --git a/mm/ksm_trace.h b/mm/ksm_trace.h new file mode 100644 index 000000000000..3cf8c36e08c2 --- /dev/null +++ b/mm/ksm_trace.h @@ -0,0 +1,258 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM ksm + +#if !defined(_TRACE_KSM_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_KSM_H + +#include +#include "ksm.h" + +/** + * ksm_scan_template - called for start / stop scan + * + * @seq: sequence number of scan + * @rmap_entries: actual number of rmap entries + * + * Allows to trace the start / stop of a ksm scan. + */ +DECLARE_EVENT_CLASS(ksm_scan_template, + + TP_PROTO(int seq, u32 rmap_entries), + + TP_ARGS(seq, rmap_entries), + + TP_STRUCT__entry( + __field(int, seq) + __field(u32, rmap_entries) + ), + + TP_fast_assign( + __entry->seq = seq; + __entry->rmap_entries = rmap_entries; + ), + + TP_printk("seq %d rmap size %d", + __entry->seq, __entry->rmap_entries) +); + +/** + * ksm_start_scan - called after a new ksm scan is started + * + * @seq: sequence number of scan + * @rmap_entries: actual number of rmap entries + * + * Allows to trace the start of a ksm scan. + */ +DEFINE_EVENT(ksm_scan_template, ksm_start_scan, + + TP_PROTO(int seq, u32 rmap_entries), + + TP_ARGS(seq, rmap_entries) +); + +/** + * ksm_stop_scan - called after a new ksm scan has completed + * + * @seq: sequence number of scan + * @rmap_entries: actual number of rmap entries + * + * Allows to trace the completion of a ksm scan. + */ +DEFINE_EVENT(ksm_scan_template, ksm_stop_scan, + + TP_PROTO(int seq, u32 rmap_entries), + + TP_ARGS(seq, rmap_entries) +); + +/** + * ksm_enter - called after a new process has been added / removed from ksm + * + * @mm: address of the mm object of the process + * + * Allows to trace the when a process has been added or removed from ksm. + */ +DECLARE_EVENT_CLASS(ksm_enter_exit_template, + + TP_PROTO(void *mm), + + TP_ARGS(mm), + + TP_STRUCT__entry( + __field(void *, mm) + ), + + TP_fast_assign( + __entry->mm = mm; + ), + + TP_printk("mm %p", __entry->mm) +); + +/** + * ksm_enter - called after a new process has been added to ksm + * + * @mm: address of the mm object of the process + * + * Allows to trace the when a process has been added to ksm. + */ +DEFINE_EVENT(ksm_enter_exit_template, ksm_enter, + + TP_PROTO(void *mm), + + TP_ARGS(mm) +); + +/** + * ksm_exit - called after a new process has been removed from ksm + * + * @mm: address of the mm object of the process + * + * Allows to trace the when a process has been removed from ksm. + */ +DEFINE_EVENT(ksm_enter_exit_template, ksm_exit, + + TP_PROTO(void *mm), + + TP_ARGS(mm) +); + +/** + * ksm_merge_one_page - called after a page has been merged + * + * @pfn: page frame number of ksm page + * @rmap_item: address of rmap_item object + * @mm: address of the process mm struct + * @err: success + * + * Allows to trace the ksm merging of individual pages. + */ +TRACE_EVENT(ksm_merge_one_page, + + TP_PROTO(unsigned long pfn, void *rmap_item, void *mm, int err), + + TP_ARGS(pfn, rmap_item, mm, err), + + TP_STRUCT__entry( + __field(unsigned long, pfn) + __field(void *, rmap_item) + __field(void *, mm) + __field(int, err) + ), + + TP_fast_assign( + __entry->pfn = pfn; + __entry->rmap_item = rmap_item; + __entry->mm = mm; + __entry->err = err; + ), + + TP_printk("ksm pfn %lu rmap_item %p mm %p error %d", + __entry->pfn, __entry->rmap_item, __entry->mm, __entry->err) +); + +/** + * ksm_merge_with_ksm_page - called after a page has been merged with a ksm page + * + * @ksm_page: address ksm page + * @pfn: page frame number of ksm page + * @rmap_item: address of rmap_item object + * @mm: address of the mm object of the process + * @err: success + * + * Allows to trace the merging of a page with a ksm page. + */ +TRACE_EVENT(ksm_merge_with_ksm_page, + + TP_PROTO(void *ksm_page, unsigned long pfn, void *rmap_item, void *mm, int err), + + TP_ARGS(ksm_page, pfn, rmap_item, mm, err), + + TP_STRUCT__entry( + __field(void *, ksm_page) + __field(unsigned long, pfn) + __field(void *, rmap_item) + __field(void *, mm) + __field(int, err) + ), + + TP_fast_assign( + __entry->ksm_page = ksm_page; + __entry->pfn = pfn; + __entry->rmap_item = rmap_item; + __entry->mm = mm; + __entry->err = err; + ), + + TP_printk("%spfn %lu rmap_item %p mm %p error %d", + (__entry->ksm_page ? "ksm " : ""), + __entry->pfn, __entry->rmap_item, __entry->mm, __entry->err) +); + +/** + * ksm_remove_ksm_page - called after a ksm page has been removed + * + * @stable_node: pointer to stable node + * + * Allows to trace the removing of stable ksm pages. + */ +TRACE_EVENT(ksm_remove_ksm_page, + + TP_PROTO(struct ksm_stable_node *stable_node), + + TP_ARGS(stable_node), + + TP_STRUCT__entry( + __field(unsigned long, pfn) + ), + + TP_fast_assign( + __entry->pfn = stable_node->kpfn; + ), + + TP_printk("pfn %lu", __entry->pfn) +); + +/** + * ksm_remove_rmap_item - called after a rmap_item has been removed from the + * stable tree + * + * @stable_node: pointer to stable node + * @rmap_item: address of rmap_item object + * @mm: address of the process mm struct + * + * Allows to trace the removal of pages from the stable tree list. + */ +TRACE_EVENT(ksm_remove_rmap_item, + + TP_PROTO(struct ksm_stable_node *stable_node, void *rmap_item, void *mm), + + TP_ARGS(stable_node, rmap_item, mm), + + TP_STRUCT__entry( + __field(unsigned long, pfn) + __field(void *, rmap_item) + __field(void *, mm) + ), + + TP_fast_assign( + __entry->pfn = stable_node->kpfn; + __entry->rmap_item = rmap_item; + __entry->mm = mm; + ), + + TP_printk("pfn %lu rmap_item %p mm %p", + __entry->pfn, __entry->rmap_item, __entry->mm) +); + +#endif /* _TRACE_KSM_H */ + +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH . + +#undef TRACE_INCLUDE_FILE +#define TRACE_INCLUDE_FILE ksm_trace + +/* This part must be outside protection */ +#include