[6/9] cpuidle/poll_state: poll via smp_cond_load_relaxed()

Commit Message

Ankur Arora April 30, 2024, 6:37 p.m. UTC

From: Mihai Carabas <[e-mail]>

The inner loop in poll_idle() polls up to POLL_IDLE_RELAX_COUNT times,
checking to see if the thread has the TIF_NEED_RESCHED bit set. The
loop exits once the condition is met, or if the poll time limit has
been exceeded.

The time check is done only infrequently (once in POLL_IDLE_RELAX_COUNT
iterations) so as to minimize the number of instructions executed in
each iteration. In addition, each loop iteration executes cpu_relax()
which on certain platforms provides a hint to the pipeline that the
loop is busy-waiting, thus allowing the processor to reduce power
consumption.

However, cpu_relax() is not defined optimally everywhere. In particular,
on arm64, it is implemented as a YIELD which merely serves as a hint to
prefer a different hardware thread if one is available.

arm64 exposes a better mechanism via smp_cond_load_relaxed() which uses
LDXR, WFE where the LDXR loads a memory region in exclusive state and
the WFE waits for any stores to the region.

So restructure the loop and fold both checks in smp_cond_load_relaxed().
Also, move the time check to the head of the loop so, once
TIF_NEED_RESCHED is set, we exit straight-away without doing an
unnecessary time check.

Suggested-by: Peter Zijlstra <[e-mail]>
Signed-off-by: Mihai Carabas <[e-mail]>
Signed-off-by: Ankur Arora <[e-mail]>
---
Changelog:

   - reorganized the loop to keep the original poll_idle() structure.

---
 drivers/cpuidle/poll_state.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

Patch

diff --git a/drivers/cpuidle/poll_state.c b/drivers/cpuidle/poll_state.c
index 9b6d90a72601..532e4ed19e0f 100644
--- a/drivers/cpuidle/poll_state.c
+++ b/drivers/cpuidle/poll_state.c
@@ -21,21 +21,21 @@  static int __cpuidle poll_idle(struct cpuidle_device *dev,
 
 	raw_local_irq_enable();
 	if (!current_set_polling_and_test()) {
-		unsigned int loop_count = 0;
+		unsigned int loop_count;
 		u64 limit;
 
 		limit = cpuidle_poll_time(drv, dev);
 
 		while (!need_resched()) {
-			cpu_relax();
-			if (loop_count++ < POLL_IDLE_RELAX_COUNT)
-				continue;
-
 			loop_count = 0;
 			if (local_clock_noinstr() - time_start > limit) {
 				dev->poll_time_limit = true;
 				break;
 			}
+
+			smp_cond_load_relaxed(&current_thread_info()->flags,
+					      VAL & _TIF_NEED_RESCHED ||
+					      loop_count++ >= POLL_IDLE_RELAX_COUNT);
 		}
 	}
 	raw_local_irq_disable();