[0/4] perf lock contention: Improve aggr x filter combination (v1)

Message

Namhyung Kim Feb. 2, 2023, 5:04 a.m. UTC

Hello,

The callstack filter can be useful to debug lock issues but it has a
limitation that it only works with caller aggregation mode (which is the
default setting).  IOW it cannot filter by callstack when showing tasks
or lock addresses/names.

But sometimes users want to use the filter for other aggregation mode.
Like "show me lock addresses/names from this caller only" or "show me
tasks having these callers".

When it's using tracepoint events from the data file, the situation is
good since the tracepoints have all the necessary info.  But when using
BPF it needs to extend the key of lock stat BPF map to have more than
one info like 'pid + stack_id' or 'lock_addr + stack_id'.  As callstack
filter works in userspace, it should save the both info.

With this change we can now use the -S/--callstack-filter with the
-t/--threads option or -l/--lock-addr option.  It's also possible to use
it with other filter options.

The following example shows the top 5 tasks that have contention
somewhere in the epoll handling.

  $ sudo perf lock con -abt -S epoll -E5 -- sleep 1
   contended   total wait     max wait     avg wait          pid   comm

           2     58.64 us     32.38 us     29.32 us      1514752   Chrome_IOThread
           3     29.31 us     12.65 us      9.77 us         3773   Xorg
           1     17.45 us     17.45 us     17.45 us      1514906   Chrome_ChildIOT
           1     15.41 us     15.41 us     15.41 us      1515382   Chrome_ChildIOT
           1     12.52 us     12.52 us     12.52 us       293878   IPC I/O Parent

You get get the code at 'perf/lock-filter-v1' branch in

  git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Thanks,
Namhyung

Namhyung Kim (4):
  perf lock contention: Factor out lock_contention_get_name()
  perf lock contention: Use lock_stat_find{,new}
  perf lock contention: Support filters for different aggregation
  perf test: Add more test cases for perf lock contention

 tools/perf/builtin-lock.c                     |  79 ++++----
 tools/perf/tests/shell/lock_contention.sh     |  66 ++++++-
 tools/perf/util/bpf_lock_contention.c         | 178 +++++++++++-------
 .../perf/util/bpf_skel/lock_contention.bpf.c  |  15 +-
 tools/perf/util/bpf_skel/lock_data.h          |   4 +-
 tools/perf/util/lock-contention.h             |   5 +
 6 files changed, 234 insertions(+), 113 deletions(-)


base-commit: 7cfa9f5e440054db7c7e28e83a045d36993ff958

Comments

Arnaldo Carvalho de Melo Feb. 2, 2023, 12:54 p.m. UTC | #1

Em Wed, Feb 01, 2023 at 09:04:51PM -0800, Namhyung Kim escreveu:
> Hello,
> 
> The callstack filter can be useful to debug lock issues but it has a
> limitation that it only works with caller aggregation mode (which is the
> default setting).  IOW it cannot filter by callstack when showing tasks
> or lock addresses/names.

Thanks, applied, the only nit was for a preexisting problem I only
caught now in a refactoring, please consider fixing those.

- Arnaldo
 
> But sometimes users want to use the filter for other aggregation mode.
> Like "show me lock addresses/names from this caller only" or "show me
> tasks having these callers".
> 
> When it's using tracepoint events from the data file, the situation is
> good since the tracepoints have all the necessary info.  But when using
> BPF it needs to extend the key of lock stat BPF map to have more than
> one info like 'pid + stack_id' or 'lock_addr + stack_id'.  As callstack
> filter works in userspace, it should save the both info.
> 
> With this change we can now use the -S/--callstack-filter with the
> -t/--threads option or -l/--lock-addr option.  It's also possible to use
> it with other filter options.
> 
> The following example shows the top 5 tasks that have contention
> somewhere in the epoll handling.
> 
>   $ sudo perf lock con -abt -S epoll -E5 -- sleep 1
>    contended   total wait     max wait     avg wait          pid   comm
> 
>            2     58.64 us     32.38 us     29.32 us      1514752   Chrome_IOThread
>            3     29.31 us     12.65 us      9.77 us         3773   Xorg
>            1     17.45 us     17.45 us     17.45 us      1514906   Chrome_ChildIOT
>            1     15.41 us     15.41 us     15.41 us      1515382   Chrome_ChildIOT
>            1     12.52 us     12.52 us     12.52 us       293878   IPC I/O Parent
> 
> You get get the code at 'perf/lock-filter-v1' branch in
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git
> 
> Thanks,
> Namhyung
> 
> Namhyung Kim (4):
>   perf lock contention: Factor out lock_contention_get_name()
>   perf lock contention: Use lock_stat_find{,new}
>   perf lock contention: Support filters for different aggregation
>   perf test: Add more test cases for perf lock contention
> 
>  tools/perf/builtin-lock.c                     |  79 ++++----
>  tools/perf/tests/shell/lock_contention.sh     |  66 ++++++-
>  tools/perf/util/bpf_lock_contention.c         | 178 +++++++++++-------
>  .../perf/util/bpf_skel/lock_contention.bpf.c  |  15 +-
>  tools/perf/util/bpf_skel/lock_data.h          |   4 +-
>  tools/perf/util/lock-contention.h             |   5 +
>  6 files changed, 234 insertions(+), 113 deletions(-)
> 
> 
> base-commit: 7cfa9f5e440054db7c7e28e83a045d36993ff958
> -- 
> 2.39.1.456.gfc5497dd1b-goog
>