| Message ID | 20220719195325.402745-10-gpiccoli@igalia.com (mailing list archive) |
|---|---|
| State | Not Applicable |
| Headers | show |
| Series | The panic notifiers refactor strikes back - fixes/clean-ups | expand |
| Context | Check | Description |
|---|---|---|
| netdev/tree_selection | success | Not a local patch, async |
On 7/19/2022 12:53 PM, Guilherme G. Piccoli wrote: > Currently we have a debug infrastructure in the notifiers file, but > it's very simple/limited. Extend it by: > > (a) Showing all registered/unregistered notifiers' callback names; I'm not yet convinced that this is the right direction. The original intent for this "debug" feature was to be lightweight enough that it could run in production, since at the time, rootkits liked to clobber/hijack notifiers and there were also some other signs of corruption at the time. By making something print (even at pr_info) for what are probably frequent non-error operations, you turn something that is light into something that's a lot more heavy and generally that's not a great idea... it'll be a performance surprise.
On 19/07/2022 17:33, Arjan van de Ven wrote: > On 7/19/2022 12:53 PM, Guilherme G. Piccoli wrote: >> Currently we have a debug infrastructure in the notifiers file, but >> it's very simple/limited. Extend it by: >> >> (a) Showing all registered/unregistered notifiers' callback names; > > > I'm not yet convinced that this is the right direction. > The original intent for this "debug" feature was to be lightweight enough that it could run in production, since at the time, rootkits > liked to clobber/hijack notifiers and there were also some other signs of corruption at the time. > > By making something print (even at pr_info) for what are probably frequent non-error operations, you turn something that is light > into something that's a lot more heavy and generally that's not a great idea... it'll be a performance surprise. > > Is registering/un-registering notifiers a hot path, or performance sensitive usually? For me, this patch proved to be very useful, and once enabled, shows relatively few entries in dmesg, these operations aren't so common thing it seems. Also, if this Kconfig option was meant to run in production, maybe the first thing would be have some sysfs tuning or anything able to turn it on - I've worked with a variety of customers and the most terrifying thing in servers is to install a new kernel and reboot heh My understanding is that this debug infrastructure would be useful for notifiers writers and people playing with the core notifiers code...tracing would be much more useful in the context of checking if some specific notifier got registered/executed in production environment I guess. Cheers, Guilherme
On 7/19/2022 1:44 PM, Guilherme G. Piccoli wrote: > On 19/07/2022 17:33, Arjan van de Ven wrote: >> On 7/19/2022 12:53 PM, Guilherme G. Piccoli wrote: >>> Currently we have a debug infrastructure in the notifiers file, but >>> it's very simple/limited. Extend it by: >>> >>> (a) Showing all registered/unregistered notifiers' callback names; >> >> >> I'm not yet convinced that this is the right direction. >> The original intent for this "debug" feature was to be lightweight enough that it could run in production, since at the time, rootkits >> liked to clobber/hijack notifiers and there were also some other signs of corruption at the time. >> >> By making something print (even at pr_info) for what are probably frequent non-error operations, you turn something that is light >> into something that's a lot more heavy and generally that's not a great idea... it'll be a performance surprise. >> >> > > Is registering/un-registering notifiers a hot path, or performance > sensitive usually? For me, this patch proved to be very useful, and once > enabled, shows relatively few entries in dmesg, these operations aren't > so common thing it seems. > > Also, if this Kconfig option was meant to run in production, maybe the > first thing would be have some sysfs tuning or anything able to turn it > on - I've worked with a variety of customers and the most terrifying > thing in servers is to install a new kernel and reboot heh > > My understanding is that this debug infrastructure would be useful for > notifiers writers and people playing with the core notifiers > code...tracing would be much more useful in the context of checking if > some specific notifier got registered/executed in production environment > I guess. I would totally support an approach where instead of pr_info, there's a tracepoint for these events (and that shouldnt' need to be conditional on a config option) that's not what the patch does though.
On 19/07/2022 17:48, Arjan van de Ven wrote: > [...] > I would totally support an approach where instead of pr_info, there's a tracepoint > for these events (and that shouldnt' need to be conditional on a config option) > > that's not what the patch does though. This is a good idea Arjan! We could use trace events or pr_debug() - which one do you prefer? With that, we could maybe remove this Kconfig option, having it always enabled, what do you think ?
On 7/19/2022 2:00 PM, Guilherme G. Piccoli wrote: > On 19/07/2022 17:48, Arjan van de Ven wrote: >> [...] >> I would totally support an approach where instead of pr_info, there's a tracepoint >> for these events (and that shouldnt' need to be conditional on a config option) >> >> that's not what the patch does though. > > This is a good idea Arjan! We could use trace events or pr_debug() - > which one do you prefer? > I'd go for a trace point to be honest
On 19/07/2022 19:04, Arjan van de Ven wrote: > On 7/19/2022 2:00 PM, Guilherme G. Piccoli wrote: >> On 19/07/2022 17:48, Arjan van de Ven wrote: >>> [...] >>> I would totally support an approach where instead of pr_info, there's a tracepoint >>> for these events (and that shouldnt' need to be conditional on a config option) >>> >>> that's not what the patch does though. >> >> This is a good idea Arjan! We could use trace events or pr_debug() - >> which one do you prefer? >> > > I'd go for a trace point to be honest > ACK, I'll re-work it as a trace event then. Cheers, Guilherme
diff --git a/kernel/notifier.c b/kernel/notifier.c index 0d5bd62c480e..350761b34f8a 100644 --- a/kernel/notifier.c +++ b/kernel/notifier.c @@ -37,6 +37,10 @@ static int notifier_chain_register(struct notifier_block **nl, } n->next = *nl; rcu_assign_pointer(*nl, n); + + if (IS_ENABLED(CONFIG_DEBUG_NOTIFIERS)) + pr_info("notifiers: registered %ps\n", n->notifier_call); + return 0; } @@ -46,6 +50,11 @@ static int notifier_chain_unregister(struct notifier_block **nl, while ((*nl) != NULL) { if ((*nl) == n) { rcu_assign_pointer(*nl, n->next); + + if (IS_ENABLED(CONFIG_DEBUG_NOTIFIERS)) + pr_info("notifiers: unregistered %ps\n", + n->notifier_call); + return 0; } nl = &((*nl)->next); @@ -77,13 +86,14 @@ static int notifier_call_chain(struct notifier_block **nl, while (nb && nr_to_call) { next_nb = rcu_dereference_raw(nb->next); -#ifdef CONFIG_DEBUG_NOTIFIERS - if (unlikely(!func_ptr_is_kernel_text(nb->notifier_call))) { - WARN(1, "Invalid notifier called!"); - nb = next_nb; - continue; + if (IS_ENABLED(CONFIG_DEBUG_NOTIFIERS)) { + if (!func_ptr_is_kernel_text(nb->notifier_call)) { + WARN(1, "Invalid notifier called!"); + nb = next_nb; + continue; + } + pr_debug("notifiers: calling %ps\n", nb->notifier_call); } -#endif ret = nb->notifier_call(nb, val, v); if (nr_calls)
Currently we have a debug infrastructure in the notifiers file, but it's very simple/limited. Extend it by: (a) Showing all registered/unregistered notifiers' callback names; (b) Adding a dynamic debug tuning to allow showing called notifiers' function names. Notice that this should be guarded as a tunable since it can flood the kernel log buffer. Cc: Arjan van de Ven <arjan@linux.intel.com> Cc: Cong Wang <xiyou.wangcong@gmail.com> Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Valentin Schneider <valentin.schneider@arm.com> Cc: Xiaoming Ni <nixiaoming@huawei.com> Signed-off-by: Guilherme G. Piccoli <gpiccoli@igalia.com> --- V2: - Major improvement thanks to the great idea from Xiaoming - changed all the ksym wheel reinvention to printk %ps modifier; - Instead of ifdefs, using IS_ENABLED() - thanks Steven. - Removed an unlikely() hint on debug path. kernel/notifier.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-)