| Message ID | 20240620125703.3297-1-Dhananjay.Ugwekar@amd.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Add per-core RAPL energy counter support for AMD CPUs | expand |
Hello Dhananjay, On 6/20/2024 6:26 PM, Dhananjay Ugwekar wrote: > Currently the energy-cores event in the power PMU aggregates energy > consumption data at a package level. On the other hand the core energy > RAPL counter in AMD CPUs has a core scope (which means the energy > consumption is recorded separately for each core). Earlier efforts to add > the core event in the power PMU had failed [1], due to the difference in > the scope of these two events. Hence, there is a need for a new core scope > PMU. > > This patchset adds a new "power_per_core" PMU alongside the existing > "power" PMU, which will be responsible for collecting the new > "energy-per-core" event. > > Tested the package level and core level PMU counters with workloads > pinned to different CPUs. > > Results with workload pinned to CPU 1 in Core 1 on an AMD Zen4 Genoa > machine: > > $ perf stat -a --per-core -e power_per_core/energy-per-core/ sleep 1 When testing this on a 2P 3rd Generation EPYC System (2 x 64/128T), I ran into an issue where it seems like the energy reporting for the system is coming from the second socket. Following are the CPUs on each socket of the system: Node 0: 0-63, 128-191 Node 1: 64-127, 192-255 Following are the experiments I ran: $ # Run a busy loop on each thread of the first socket $ for i in `seq 0 63` `seq 128 191`; do taskset -c $i ~/scripts/loop & done $ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5 S0-D0-C0 1 0.00 Joules power_per_core/energy-per-core/ S0-D0-C1 1 0.00 Joules power_per_core/energy-per-core/ S0-D0-C2 1 0.00 Joules power_per_core/energy-per-core/ S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/ ... S0-D0-C63 1 0.00 Joules power_per_core/energy-per-core/ S1-D1-C0 1 0.00 Joules power_per_core/energy-per-core/ S1-D1-C1 1 0.00 Joules power_per_core/energy-per-core/ S1-D1-C2 1 0.00 Joules power_per_core/energy-per-core/ S1-D1-C3 1 0.00 Joules power_per_core/energy-per-core/ ... S1-D1-C63 1 0.00 Joules power_per_core/energy-per-core/ From the energy data, it looks as if the system is entirely idle. If I repeat the same, pinning the running busy loop on the threads of second socket, I see the following: $ # Run a busy loop on each thread of the second socket $ for i in `seq 64 127` `seq 192 255`; do taskset -c $i ~/scripts/loop & done $ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5 S0-D0-C0 1 11.79 Joules power_per_core/energy-per-core/ S0-D0-C1 1 11.80 Joules power_per_core/energy-per-core/ S0-D0-C2 1 11.90 Joules power_per_core/energy-per-core/ S0-D0-C3 1 11.88 Joules power_per_core/energy-per-core/ ... S0-D0-C63 1 11.76 Joules power_per_core/energy-per-core/ S1-D1-C0 1 11.81 Joules power_per_core/energy-per-core/ S1-D1-C1 1 11.80 Joules power_per_core/energy-per-core/ S1-D1-C2 1 11.90 Joules power_per_core/energy-per-core/ S1-D1-C3 1 11.88 Joules power_per_core/energy-per-core/ ... S1-D1-C63 1 11.76 Joules power_per_core/energy-per-core/ The whole system seems to be busy this time around. I've verified that only half the system is busy using htop in either case. Running some more experiments, I see the following: $ taskset -c 1 ~/scripts/loop& # First thread from Core 1, Socket $ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5 S0-D0-C0 1 0.02 Joules power_per_core/energy-per-core/ S0-D0-C1 1 0.21 Joules power_per_core/energy-per-core/ S0-D0-C2 1 0.20 Joules power_per_core/energy-per-core/ S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/ ... (Seemingly idle system) $ taskset -c 65 ~/scripts/loop& $ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5 S0-D0-C0 1 0.01 Joules power_per_core/energy-per-core/ S0-D0-C1 1 16.73 Joules power_per_core/energy-per-core/ S0-D0-C2 1 0.00 Joules power_per_core/energy-per-core/ S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/ ... S0-D0-C63 1 0.00 Joules power_per_core/energy-per-core/ S1-D1-C0 1 0.01 Joules power_per_core/energy-per-core/ S1-D1-C1 1 16.73 Joules power_per_core/energy-per-core/ S1-D1-C2 1 0.00 Joules power_per_core/energy-per-core/ S1-D1-C3 1 0.00 Joules power_per_core/energy-per-core/ ... S1-D1-C63 1 0.00 Joules power_per_core/energy-per-core/ (Core 1 from both sockets look busy reporting identical energy values) Hope it helps narrow down the issue. > > Performance counter stats for 'system wide': > > S0-D0-C0 1 0.02 Joules power_per_core/energy-per-core/ > S0-D0-C1 1 5.72 Joules power_per_core/energy-per-core/ > S0-D0-C2 1 0.02 Joules power_per_core/energy-per-core/ > S0-D0-C3 1 0.02 Joules power_per_core/energy-per-core/ > S0-D0-C4 1 0.02 Joules power_per_core/energy-per-core/ > S0-D0-C5 1 0.02 Joules power_per_core/energy-per-core/ > S0-D0-C6 1 0.02 Joules power_per_core/energy-per-core/ > S0-D0-C7 1 0.02 Joules power_per_core/energy-per-core/ > S0-D0-C8 1 0.02 Joules power_per_core/energy-per-core/ > S0-D0-C9 1 0.02 Joules power_per_core/energy-per-core/ > S0-D0-C10 1 0.02 Joules power_per_core/energy-per-core/ > > [1]: https://lore.kernel.org/lkml/3e766f0e-37d4-0f82-3868-31b14228868d@linux.intel.com/ > > This patchset applies cleanly on top of v6.10-rc4 as well as latest > tip/master. P.S. I tested these changes on top of tip:perf/core > > [..snip..] >
Hello Prateek, On 6/21/2024 1:54 PM, K Prateek Nayak wrote: > Hello Dhananjay, > > On 6/20/2024 6:26 PM, Dhananjay Ugwekar wrote: >> Currently the energy-cores event in the power PMU aggregates energy >> consumption data at a package level. On the other hand the core energy >> RAPL counter in AMD CPUs has a core scope (which means the energy >> consumption is recorded separately for each core). Earlier efforts to add >> the core event in the power PMU had failed [1], due to the difference in >> the scope of these two events. Hence, there is a need for a new core scope >> PMU. >> >> This patchset adds a new "power_per_core" PMU alongside the existing >> "power" PMU, which will be responsible for collecting the new >> "energy-per-core" event. >> >> Tested the package level and core level PMU counters with workloads >> pinned to different CPUs. >> >> Results with workload pinned to CPU 1 in Core 1 on an AMD Zen4 Genoa >> machine: >> >> $ perf stat -a --per-core -e power_per_core/energy-per-core/ sleep 1 > > When testing this on a 2P 3rd Generation EPYC System (2 x 64/128T), I > ran into an issue where it seems like the energy reporting for the > system is coming from the second socket. Following are the CPUs on each > socket of the system: > > Node 0: 0-63, 128-191 > Node 1: 64-127, 192-255 > > Following are the experiments I ran: > > $ # Run a busy loop on each thread of the first socket > $ for i in `seq 0 63` `seq 128 191`; do taskset -c $i ~/scripts/loop & done > $ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5 > > S0-D0-C0 1 0.00 Joules power_per_core/energy-per-core/ > S0-D0-C1 1 0.00 Joules power_per_core/energy-per-core/ > S0-D0-C2 1 0.00 Joules power_per_core/energy-per-core/ > S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/ > ... > S0-D0-C63 1 0.00 Joules power_per_core/energy-per-core/ > S1-D1-C0 1 0.00 Joules power_per_core/energy-per-core/ > S1-D1-C1 1 0.00 Joules power_per_core/energy-per-core/ > S1-D1-C2 1 0.00 Joules power_per_core/energy-per-core/ > S1-D1-C3 1 0.00 Joules power_per_core/energy-per-core/ > ... > S1-D1-C63 1 0.00 Joules power_per_core/energy-per-core/ > > From the energy data, it looks as if the system is entirely idle. > > If I repeat the same, pinning the running busy loop on the threads of > second socket, I see the following: > > $ # Run a busy loop on each thread of the second socket > $ for i in `seq 64 127` `seq 192 255`; do taskset -c $i ~/scripts/loop & done > $ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5 > > S0-D0-C0 1 11.79 Joules power_per_core/energy-per-core/ > S0-D0-C1 1 11.80 Joules power_per_core/energy-per-core/ > S0-D0-C2 1 11.90 Joules power_per_core/energy-per-core/ > S0-D0-C3 1 11.88 Joules power_per_core/energy-per-core/ > ... > S0-D0-C63 1 11.76 Joules power_per_core/energy-per-core/ > S1-D1-C0 1 11.81 Joules power_per_core/energy-per-core/ > S1-D1-C1 1 11.80 Joules power_per_core/energy-per-core/ > S1-D1-C2 1 11.90 Joules power_per_core/energy-per-core/ > S1-D1-C3 1 11.88 Joules power_per_core/energy-per-core/ > ... > S1-D1-C63 1 11.76 Joules power_per_core/energy-per-core/ > > The whole system seems to be busy this time around. I've verified that > only half the system is busy using htop in either case. > > Running some more experiments, I see the following: > > $ taskset -c 1 ~/scripts/loop& # First thread from Core 1, Socket > $ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5 > > S0-D0-C0 1 0.02 Joules power_per_core/energy-per-core/ > S0-D0-C1 1 0.21 Joules power_per_core/energy-per-core/ > S0-D0-C2 1 0.20 Joules power_per_core/energy-per-core/ > S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/ > ... > (Seemingly idle system) > > > $ taskset -c 65 ~/scripts/loop& > $ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5 > > S0-D0-C0 1 0.01 Joules power_per_core/energy-per-core/ > S0-D0-C1 1 16.73 Joules power_per_core/energy-per-core/ > S0-D0-C2 1 0.00 Joules power_per_core/energy-per-core/ > S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/ > ... > S0-D0-C63 1 0.00 Joules power_per_core/energy-per-core/ > S1-D1-C0 1 0.01 Joules power_per_core/energy-per-core/ > S1-D1-C1 1 16.73 Joules power_per_core/energy-per-core/ > S1-D1-C2 1 0.00 Joules power_per_core/energy-per-core/ > S1-D1-C3 1 0.00 Joules power_per_core/energy-per-core/ > ... > S1-D1-C63 1 0.00 Joules power_per_core/energy-per-core/ > > (Core 1 from both sockets look busy reporting identical energy > values) > > Hope it helps narrow down the issue. I think my assumption that topology_core_id() will return a unique core ID across the system might not be correct. It seems the core ID is unique only within a package, will fix this in the next version. Thanks a lot for testing and helping narrow down the issue! Regards, Dhananjay > >> >> Performance counter stats for 'system wide': >> >> S0-D0-C0 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C1 1 5.72 Joules power_per_core/energy-per-core/ >> S0-D0-C2 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C3 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C4 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C5 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C6 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C7 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C8 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C9 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C10 1 0.02 Joules power_per_core/energy-per-core/ >> >> [1]: https://lore.kernel.org/lkml/3e766f0e-37d4-0f82-3868-31b14228868d@linux.intel.com/ >> >> This patchset applies cleanly on top of v6.10-rc4 as well as latest >> tip/master. > > P.S. I tested these changes on top of tip:perf/core > >> >> [..snip..] >> >