| Message ID | 20230512095743.3393563-1-lukasz.luba@arm.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Introduce runtime modifiable Energy Model | expand |
Hi Lukasz, On Fri, May 12, 2023 at 11:58 AM Lukasz Luba <lukasz.luba@arm.com> wrote: > > Hi all, > > This patch set adds a new feature which allows to modify Energy Model (EM) > power values at runtime. It will allow to better reflect power model of > a recent SoCs and silicon. Different characteristics of the power usage > can be leveraged and thus better decisions made during task placement in EAS. > > It's part of feature set know as Dynamic Energy Model. It has been presented > and discussed recently at OSPM2023 [3]. This patch set implements the 1st > improvement for the EM. > > The concepts: > 1. The CPU power usage can vary due to the workload that it's running or due > to the temperature of the SoC. The same workload can use more power when the > temperature of the silicon has increased (e.g. due to hot GPU or ISP). > In such situation or EM can be adjusted and reflect the fact of increased > power usage. That power increase is due to a factor called static power > (sometimes called simply: leakage). The CPUs in recent SoCs are different. > We have heterogeneous SoCs with 3 (or even 4) different microarchitectures. > They are also built differently with High Performance (HP) cells or > Low Power (LP) cells. They are affected by the temperature increase > differently: HP cells have bigger leakage. The SW model can leverage that > knowledge. > 2. It is also possible to change the EM to better reflect the currently > running workload. Usually the EM is derived from some average power values > taken from experiments with benchmark (e.g. Dhrystone). The model derived > from such scenario might not represent properly the workloads usually running > on the device. Therefore, runtime modification of the EM allows to switch to > a different model, when there is a need. > 3. The EM can be adjusted after boot, when all the modules are loaded and > more information about the SoC is available e.g. chip binning. This would help > to better reflect the silicon characteristics. Thus, this EM modification > API allows it now. It wasn't possible in the past and the EM had to be > 'set in stone'. > > Some design details: > The internal mechanisms for the memory allocation are handled internally in the > EM. Kernel modules can just call the new API to update the EM data and the > new memory would be provided and owned by the EM. The EM memory is used by > EAS, which impacts those design decisions. The EM writers are protected by > a mutex. This new runtime modified EM table is protected using RCU mechanism, > which fits the current EAS hot path (which already uses RCU read lock). > The unregister API handles only non-CPU (e.g. GPU, ISP) devices and uses the > same mutex as EM modifiers to make sure the memory is safely freed. > > More detailed explanation and background can be found in presentations > during LPC2022 [1][2] or in the documentation patches. > > Changelog: > v2: > - solved build warning of unused variable in patch 13/17 when EM is > not compiled in, e.g. on Intel platform for this cpufreq_cooling > - re-based on top of v6.4-rc1 > v1: > - implementation can be found here [4] > > [1] https://lpc.events/event/16/contributions/1341/attachments/955/1873/Dynamic_Energy_Model_to_handle_leakage_power.pdf > [2] https://lpc.events/event/16/contributions/1194/attachments/1114/2139/LPC2022_Energy_model_accuracy.pdf > [3] https://www.youtube.com/watch?v=2C-5uikSbtM&list=PL0fKordpLTjKsBOUcZqnzlHShri4YBL1H > [4] https://lore.kernel.org/lkml/20230314103357.26010-1-lukasz.luba@arm.com/ > > Lukasz Luba (17): > PM: EM: Refactor em_cpufreq_update_efficiencies() arguments > PM: EM: Find first CPU online while updating OPP efficiency > PM: EM: Refactor em_pd_get_efficient_state() to be more flexible > PM: EM: Create a new function em_compute_costs() > trace: energy_model: Add trace event for EM runtime modifications > PM: EM: Add update_power() callback for runtime modifications > PM: EM: Check if the get_cost() callback is present in > em_compute_costs() > PM: EM: Introduce runtime modifiable table > PM: EM: Add RCU mechanism which safely cleans the old data > PM: EM: Add runtime update interface to modify EM power > PM: EM: Use runtime modified EM for CPUs energy estimation in EAS > PM: EM: Add argument to get_cost() for runtime modification > PM: EM: Refactor struct em_perf_domain and add default_table > Documentation: EM: Add a new section about the design > Documentation: EM: Add a runtime modifiable EM design description > Documentation: EM: Add example with driver modifying the EM > Documentation: EM: Describe the API of runtime modifications I haven't seen any responses from anyone having a vested interest in the Energy Model code. I'm not sure what this means, but I surely can't do much about it myself without any input from the potentially interested parties. Thanks!
On 12/05/2023 11:57, Lukasz Luba wrote: > Hi all, > > This patch set adds a new feature which allows to modify Energy Model (EM) > power values at runtime. It will allow to better reflect power model of > a recent SoCs and silicon. Different characteristics of the power usage > can be leveraged and thus better decisions made during task placement in EAS. > > It's part of feature set know as Dynamic Energy Model. It has been presented > and discussed recently at OSPM2023 [3]. This patch set implements the 1st > improvement for the EM. Why is the feature set called Dynamic Energy Model? Dynamic Energy Model: Runtime modifiable EM Proper CPU performance state evaluation CPU idle wakeup costs CPU capacity as new EM data Didn't this `Dynamic` stand for the modifiability of the EM only? > The concepts: > 1. The CPU power usage can vary due to the workload that it's running or due > to the temperature of the SoC. The same workload can use more power when the > temperature of the silicon has increased (e.g. due to hot GPU or ISP). > In such situation or EM can be adjusted and reflect the fact of increased > power usage. That power increase is due to a factor called static power > (sometimes called simply: leakage). The CPUs in recent SoCs are different. > We have heterogeneous SoCs with 3 (or even 4) different microarchitectures. > They are also built differently with High Performance (HP) cells or > Low Power (LP) cells. They are affected by the temperature increase > differently: HP cells have bigger leakage. The SW model can leverage that > knowledge. > 2. It is also possible to change the EM to better reflect the currently > running workload. Usually the EM is derived from some average power values > taken from experiments with benchmark (e.g. Dhrystone). The model derived > from such scenario might not represent properly the workloads usually running > on the device. Therefore, runtime modification of the EM allows to switch to > a different model, when there is a need. > 3. The EM can be adjusted after boot, when all the modules are loaded and > more information about the SoC is available e.g. chip binning. This would help > to better reflect the silicon characteristics. Thus, this EM modification > API allows it now. It wasn't possible in the past and the EM had to be > 'set in stone'. > > Some design details: > The internal mechanisms for the memory allocation are handled internally in the > EM. Kernel modules can just call the new API to update the EM data and the > new memory would be provided and owned by the EM. The EM memory is used by > EAS, which impacts those design decisions. The EM writers are protected by > a mutex. This new runtime modified EM table is protected using RCU mechanism, > which fits the current EAS hot path (which already uses RCU read lock). > The unregister API handles only non-CPU (e.g. GPU, ISP) devices and uses the > same mutex as EM modifiers to make sure the memory is safely freed. This only mentions memory allocation and locking? A global design overview, containing e.g. Why 2 tables, modifiable (a) and default (b)? Why does only EAS use (a)? (a) and (b) being the same performance state table until first call to modify (a) () as an introduction into the patches would be more helpful here. > More detailed explanation and background can be found in presentations > during LPC2022 [1][2] or in the documentation patches. I checked 15/17 as well but could find any of this information there either. [...]
Hi Rafael, On 5/24/23 18:25, Rafael J. Wysocki wrote: > Hi Lukasz, > > On Fri, May 12, 2023 at 11:58 AM Lukasz Luba <lukasz.luba@arm.com> wrote: >> >> Hi all, >> >> This patch set adds a new feature which allows to modify Energy Model (EM) >> power values at runtime. It will allow to better reflect power model of >> a recent SoCs and silicon. Different characteristics of the power usage >> can be leveraged and thus better decisions made during task placement in EAS. >> >> It's part of feature set know as Dynamic Energy Model. It has been presented >> and discussed recently at OSPM2023 [3]. This patch set implements the 1st >> improvement for the EM. >> >> The concepts: >> 1. The CPU power usage can vary due to the workload that it's running or due >> to the temperature of the SoC. The same workload can use more power when the >> temperature of the silicon has increased (e.g. due to hot GPU or ISP). >> In such situation or EM can be adjusted and reflect the fact of increased >> power usage. That power increase is due to a factor called static power >> (sometimes called simply: leakage). The CPUs in recent SoCs are different. >> We have heterogeneous SoCs with 3 (or even 4) different microarchitectures. >> They are also built differently with High Performance (HP) cells or >> Low Power (LP) cells. They are affected by the temperature increase >> differently: HP cells have bigger leakage. The SW model can leverage that >> knowledge. >> 2. It is also possible to change the EM to better reflect the currently >> running workload. Usually the EM is derived from some average power values >> taken from experiments with benchmark (e.g. Dhrystone). The model derived >> from such scenario might not represent properly the workloads usually running >> on the device. Therefore, runtime modification of the EM allows to switch to >> a different model, when there is a need. >> 3. The EM can be adjusted after boot, when all the modules are loaded and >> more information about the SoC is available e.g. chip binning. This would help >> to better reflect the silicon characteristics. Thus, this EM modification >> API allows it now. It wasn't possible in the past and the EM had to be >> 'set in stone'. >> >> Some design details: >> The internal mechanisms for the memory allocation are handled internally in the >> EM. Kernel modules can just call the new API to update the EM data and the >> new memory would be provided and owned by the EM. The EM memory is used by >> EAS, which impacts those design decisions. The EM writers are protected by >> a mutex. This new runtime modified EM table is protected using RCU mechanism, >> which fits the current EAS hot path (which already uses RCU read lock). >> The unregister API handles only non-CPU (e.g. GPU, ISP) devices and uses the >> same mutex as EM modifiers to make sure the memory is safely freed. >> >> More detailed explanation and background can be found in presentations >> during LPC2022 [1][2] or in the documentation patches. >> >> Changelog: >> v2: >> - solved build warning of unused variable in patch 13/17 when EM is >> not compiled in, e.g. on Intel platform for this cpufreq_cooling >> - re-based on top of v6.4-rc1 >> v1: >> - implementation can be found here [4] >> >> [1] https://lpc.events/event/16/contributions/1341/attachments/955/1873/Dynamic_Energy_Model_to_handle_leakage_power.pdf >> [2] https://lpc.events/event/16/contributions/1194/attachments/1114/2139/LPC2022_Energy_model_accuracy.pdf >> [3] https://www.youtube.com/watch?v=2C-5uikSbtM&list=PL0fKordpLTjKsBOUcZqnzlHShri4YBL1H >> [4] https://lore.kernel.org/lkml/20230314103357.26010-1-lukasz.luba@arm.com/ >> >> Lukasz Luba (17): >> PM: EM: Refactor em_cpufreq_update_efficiencies() arguments >> PM: EM: Find first CPU online while updating OPP efficiency >> PM: EM: Refactor em_pd_get_efficient_state() to be more flexible >> PM: EM: Create a new function em_compute_costs() >> trace: energy_model: Add trace event for EM runtime modifications >> PM: EM: Add update_power() callback for runtime modifications >> PM: EM: Check if the get_cost() callback is present in >> em_compute_costs() >> PM: EM: Introduce runtime modifiable table >> PM: EM: Add RCU mechanism which safely cleans the old data >> PM: EM: Add runtime update interface to modify EM power >> PM: EM: Use runtime modified EM for CPUs energy estimation in EAS >> PM: EM: Add argument to get_cost() for runtime modification >> PM: EM: Refactor struct em_perf_domain and add default_table >> Documentation: EM: Add a new section about the design >> Documentation: EM: Add a runtime modifiable EM design description >> Documentation: EM: Add example with driver modifying the EM >> Documentation: EM: Describe the API of runtime modifications > > I haven't seen any responses from anyone having a vested interest in > the Energy Model code. > > I'm not sure what this means, but I surely can't do much about it > myself without any input from the potentially interested parties. My apologies for the delay. Correct, it has been missing attention, but now Dietmar is reviewing the stuff. He commented a few things and I'm going to respond and address them in v3. Thanks for having a look into this thread! Lukasz
On 5/30/23 12:07, Dietmar Eggemann wrote: > On 12/05/2023 11:57, Lukasz Luba wrote: >> Hi all, >> >> This patch set adds a new feature which allows to modify Energy Model (EM) >> power values at runtime. It will allow to better reflect power model of >> a recent SoCs and silicon. Different characteristics of the power usage >> can be leveraged and thus better decisions made during task placement in EAS. >> >> It's part of feature set know as Dynamic Energy Model. It has been presented >> and discussed recently at OSPM2023 [3]. This patch set implements the 1st >> improvement for the EM. > > Why is the feature set called Dynamic Energy Model? > > Dynamic Energy Model: > > Runtime modifiable EM > > Proper CPU performance state evaluation > > CPU idle wakeup costs > > CPU capacity as new EM data > > Didn't this `Dynamic` stand for the modifiability of the EM only? The 'modifiability' is the main feature, but not the last one. The 2nd: 'Proper CPU performance state evaluation' is also a 'dynamic' thing, which is related to the currently set CPU frequency (for Mid or Big) and the dependent Little's and L3 cache frequency. As you know that frequency can change in time, so it's 'dynamic' situation and will lands to the 'Dynamic EM'. (The 2 below need more thinking and experiments) The 3rd can be also quite dynamic. You might change the wake-up cost for in the EM if you want to avoid waking up big cores in some workloads. You might pay penalty for bigger latency, because tasks would be more queued on Mids/Littles, but that's for power saving scenario. The 4th is about CPU capacity. We still have to conduct more investigations, but it might be useful to change the EM and provide a new CPU capacity from it to the OS. In the 3-gear SoC the CPUs might have quite different capacity in different scenarios (workloads). This could be also a 'dynamic' thing, e.g. triggered by middle-ware for a long running video call. > >> The concepts: >> 1. The CPU power usage can vary due to the workload that it's running or due >> to the temperature of the SoC. The same workload can use more power when the >> temperature of the silicon has increased (e.g. due to hot GPU or ISP). >> In such situation or EM can be adjusted and reflect the fact of increased >> power usage. That power increase is due to a factor called static power >> (sometimes called simply: leakage). The CPUs in recent SoCs are different. >> We have heterogeneous SoCs with 3 (or even 4) different microarchitectures. >> They are also built differently with High Performance (HP) cells or >> Low Power (LP) cells. They are affected by the temperature increase >> differently: HP cells have bigger leakage. The SW model can leverage that >> knowledge. >> 2. It is also possible to change the EM to better reflect the currently >> running workload. Usually the EM is derived from some average power values >> taken from experiments with benchmark (e.g. Dhrystone). The model derived >> from such scenario might not represent properly the workloads usually running >> on the device. Therefore, runtime modification of the EM allows to switch to >> a different model, when there is a need. >> 3. The EM can be adjusted after boot, when all the modules are loaded and >> more information about the SoC is available e.g. chip binning. This would help >> to better reflect the silicon characteristics. Thus, this EM modification >> API allows it now. It wasn't possible in the past and the EM had to be >> 'set in stone'. >> >> Some design details: >> The internal mechanisms for the memory allocation are handled internally in the >> EM. Kernel modules can just call the new API to update the EM data and the >> new memory would be provided and owned by the EM. The EM memory is used by >> EAS, which impacts those design decisions. The EM writers are protected by >> a mutex. This new runtime modified EM table is protected using RCU mechanism, >> which fits the current EAS hot path (which already uses RCU read lock). >> The unregister API handles only non-CPU (e.g. GPU, ISP) devices and uses the >> same mutex as EM modifiers to make sure the memory is safely freed. > > This only mentions memory allocation and locking? A global design > overview, containing e.g. > > Why 2 tables, modifiable (a) and default (b)? > > Why does only EAS use (a)? > > (a) and (b) being the same performance state table until first call to > modify (a) () > > as an introduction into the patches would be more helpful here. > >> More detailed explanation and background can be found in presentations >> during LPC2022 [1][2] or in the documentation patches. > > I checked 15/17 as well but could find any of this information there either. > > [...] For the above two comments: Yes, I see your point and will address that in the next v3. Thank you fro your valuable comments and time for reviewing it! Lukasz
Hi all, This patch set adds a new feature which allows to modify Energy Model (EM) power values at runtime. It will allow to better reflect power model of a recent SoCs and silicon. Different characteristics of the power usage can be leveraged and thus better decisions made during task placement in EAS. It's part of feature set know as Dynamic Energy Model. It has been presented and discussed recently at OSPM2023 [3]. This patch set implements the 1st improvement for the EM. The concepts: 1. The CPU power usage can vary due to the workload that it's running or due to the temperature of the SoC. The same workload can use more power when the temperature of the silicon has increased (e.g. due to hot GPU or ISP). In such situation or EM can be adjusted and reflect the fact of increased power usage. That power increase is due to a factor called static power (sometimes called simply: leakage). The CPUs in recent SoCs are different. We have heterogeneous SoCs with 3 (or even 4) different microarchitectures. They are also built differently with High Performance (HP) cells or Low Power (LP) cells. They are affected by the temperature increase differently: HP cells have bigger leakage. The SW model can leverage that knowledge. 2. It is also possible to change the EM to better reflect the currently running workload. Usually the EM is derived from some average power values taken from experiments with benchmark (e.g. Dhrystone). The model derived from such scenario might not represent properly the workloads usually running on the device. Therefore, runtime modification of the EM allows to switch to a different model, when there is a need. 3. The EM can be adjusted after boot, when all the modules are loaded and more information about the SoC is available e.g. chip binning. This would help to better reflect the silicon characteristics. Thus, this EM modification API allows it now. It wasn't possible in the past and the EM had to be 'set in stone'. Some design details: The internal mechanisms for the memory allocation are handled internally in the EM. Kernel modules can just call the new API to update the EM data and the new memory would be provided and owned by the EM. The EM memory is used by EAS, which impacts those design decisions. The EM writers are protected by a mutex. This new runtime modified EM table is protected using RCU mechanism, which fits the current EAS hot path (which already uses RCU read lock). The unregister API handles only non-CPU (e.g. GPU, ISP) devices and uses the same mutex as EM modifiers to make sure the memory is safely freed. More detailed explanation and background can be found in presentations during LPC2022 [1][2] or in the documentation patches. Changelog: v2: - solved build warning of unused variable in patch 13/17 when EM is not compiled in, e.g. on Intel platform for this cpufreq_cooling - re-based on top of v6.4-rc1 v1: - implementation can be found here [4] Regards, Lukasz Luba [1] https://lpc.events/event/16/contributions/1341/attachments/955/1873/Dynamic_Energy_Model_to_handle_leakage_power.pdf [2] https://lpc.events/event/16/contributions/1194/attachments/1114/2139/LPC2022_Energy_model_accuracy.pdf [3] https://www.youtube.com/watch?v=2C-5uikSbtM&list=PL0fKordpLTjKsBOUcZqnzlHShri4YBL1H [4] https://lore.kernel.org/lkml/20230314103357.26010-1-lukasz.luba@arm.com/ Lukasz Luba (17): PM: EM: Refactor em_cpufreq_update_efficiencies() arguments PM: EM: Find first CPU online while updating OPP efficiency PM: EM: Refactor em_pd_get_efficient_state() to be more flexible PM: EM: Create a new function em_compute_costs() trace: energy_model: Add trace event for EM runtime modifications PM: EM: Add update_power() callback for runtime modifications PM: EM: Check if the get_cost() callback is present in em_compute_costs() PM: EM: Introduce runtime modifiable table PM: EM: Add RCU mechanism which safely cleans the old data PM: EM: Add runtime update interface to modify EM power PM: EM: Use runtime modified EM for CPUs energy estimation in EAS PM: EM: Add argument to get_cost() for runtime modification PM: EM: Refactor struct em_perf_domain and add default_table Documentation: EM: Add a new section about the design Documentation: EM: Add a runtime modifiable EM design description Documentation: EM: Add example with driver modifying the EM Documentation: EM: Describe the API of runtime modifications Documentation/power/energy-model.rst | 134 +++++++++++- drivers/cpufreq/cppc_cpufreq.c | 2 +- drivers/powercap/dtpm_cpu.c | 27 ++- drivers/powercap/dtpm_devfreq.c | 23 +- drivers/thermal/cpufreq_cooling.c | 23 +- drivers/thermal/devfreq_cooling.c | 23 +- include/linux/energy_model.h | 93 ++++++-- include/trace/events/energy_model.h | 46 ++++ kernel/power/energy_model.c | 305 +++++++++++++++++++++++---- 9 files changed, 580 insertions(+), 96 deletions(-) create mode 100644 include/trace/events/energy_model.h