arm64: smp: disable hotplug on trusted OS resident CPU

Commit Message

Sudeep Holla June 12, 2019, 12:51 p.m. UTC

The trusted OS may reject CPU_OFF calls to its resident CPU, so we must
avoid issuing those. We never migrate a Trusted OS and we already take
care to prevent CPU_OFF PSCI call. However, this is not reflected
explicitly to the userspace. Any user can attempt to hotplug trusted OS
resident CPU. The entire motion of going through the various state
transitions in the CPU hotplug state machine gets executed and the
PSCI layer finally refuses to make CPU_OFF call.

This results is unnecessary unwinding of CPU hotplug state machine in
the kernel. Instead we can mark the trusted OS resident CPU as not
available for hotplug, so that the user attempt or request to do the
same will get immediately rejected.

Cc: Mark Rutland <[e-mail]>
Cc: Catalin Marinas <[e-mail]>
Cc: Will Deacon <[e-mail]>
Signed-off-by: Sudeep Holla <[e-mail]>
---
 arch/arm64/include/asm/cpu_ops.h |  3 +++
 arch/arm64/kernel/psci.c         |  6 ++++++
 arch/arm64/kernel/setup.c        | 11 ++++++++++-
 3 files changed, 19 insertions(+), 1 deletion(-)

v1->v2:
	- Renamed cpu_is_hotpluggable to cpu_can_disable
	- Added kernel doc entry for cpu_can_disable
	- Dropped else segment as suggested

Comments

Will Deacon June 13, 2019, 9:14 a.m. UTC | #1

Hi Sudeep.

On Wed, Jun 12, 2019 at 01:51:37PM +0100, Sudeep Holla wrote:
> The trusted OS may reject CPU_OFF calls to its resident CPU, so we must
> avoid issuing those. We never migrate a Trusted OS and we already take
> care to prevent CPU_OFF PSCI call. However, this is not reflected
> explicitly to the userspace. Any user can attempt to hotplug trusted OS
> resident CPU. The entire motion of going through the various state
> transitions in the CPU hotplug state machine gets executed and the
> PSCI layer finally refuses to make CPU_OFF call.
> 
> This results is unnecessary unwinding of CPU hotplug state machine in
> the kernel. Instead we can mark the trusted OS resident CPU as not
> available for hotplug, so that the user attempt or request to do the
> same will get immediately rejected.
> 
> Cc: Mark Rutland <[e-mail]>
> Cc: Catalin Marinas <[e-mail]>
> Cc: Will Deacon <[e-mail]>
> Signed-off-by: Sudeep Holla <[e-mail]>
> ---
>  arch/arm64/include/asm/cpu_ops.h |  3 +++
>  arch/arm64/kernel/psci.c         |  6 ++++++
>  arch/arm64/kernel/setup.c        | 11 ++++++++++-
>  3 files changed, 19 insertions(+), 1 deletion(-)

I'm just trying to understand the motivation behind this. It's not a fix as
far as I can tell, but more of an optimisation for a failing CPU hotplug
case. Why is that important? I feel like I'm missing something.

Thanks,

Will

Sudeep Holla Aug. 12, 2019, 5:11 p.m. UTC | #2

Hi Will,

(sorry for responding so late, I seem to have lost your reply and
when I saw the patch today in my git, searched and saw this reply)

On Mon, Aug 12, 2019 at 06:00:52PM +0100, Sudeep Holla wrote:

>
> On 13/06/2019 10:14, Will Deacon wrote:
> > Hi Sudeep.
> >
> > On Wed, Jun 12, 2019 at 01:51:37PM +0100, Sudeep Holla wrote:
> >> The trusted OS may reject CPU_OFF calls to its resident CPU, so we must
> >> avoid issuing those. We never migrate a Trusted OS and we already take
> >> care to prevent CPU_OFF PSCI call. However, this is not reflected
> >> explicitly to the userspace. Any user can attempt to hotplug trusted OS
> >> resident CPU. The entire motion of going through the various state
> >> transitions in the CPU hotplug state machine gets executed and the
> >> PSCI layer finally refuses to make CPU_OFF call.
> >>
> >> This results is unnecessary unwinding of CPU hotplug state machine in
> >> the kernel. Instead we can mark the trusted OS resident CPU as not
> >> available for hotplug, so that the user attempt or request to do the
> >> same will get immediately rejected.
> >>
> >> Cc: Mark Rutland <[e-mail]>
> >> Cc: Catalin Marinas <[e-mail]>
> >> Cc: Will Deacon <[e-mail]>
> >> Signed-off-by: Sudeep Holla <[e-mail]>
> >> ---
> >>  arch/arm64/include/asm/cpu_ops.h |  3 +++
> >>  arch/arm64/kernel/psci.c         |  6 ++++++
> >>  arch/arm64/kernel/setup.c        | 11 ++++++++++-
> >>  3 files changed, 19 insertions(+), 1 deletion(-)
> >
> > I'm just trying to understand the motivation behind this. It's not a fix as
> > far as I can tell, but more of an optimisation for a failing CPU hotplug
> > case. Why is that important? I feel like I'm missing something.
> >

Yes it's just optimisation and not a fix. The main reasons I came up
with this was to avoid unnecessary CPU hotplug state machine unwinding
as it's generally heavy weight operation. There's no other hidden
reasons :) that you are missing.

IIRC I wrote this when I was debugging some issue with suspend-to-ram
which was broken for a different reason.

--
Regards,
Sudeep

Patch

diff --git a/arch/arm64/include/asm/cpu_ops.h b/arch/arm64/include/asm/cpu_ops.h
index 8f03446cf89f..8ce85449b502 100644
--- a/arch/arm64/include/asm/cpu_ops.h
+++ b/arch/arm64/include/asm/cpu_ops.h
@@ -34,6 +34,8 @@ 
  * @cpu_boot:	Boots a cpu into the kernel.
  * @cpu_postboot: Optionally, perform any post-boot cleanup or necesary
  *		synchronisation. Called from the cpu being booted.
+ * @cpu_can_disable: Determines whether a CPU can be disabled based on
+ *		mechanism-specific information.
  * @cpu_disable: Prepares a cpu to die. May fail for some mechanism-specific
  * 		reason, which will cause the hot unplug to be aborted. Called
  * 		from the cpu to be killed.
@@ -53,6 +55,7 @@  struct cpu_operations {
 	int		(*cpu_boot)(unsigned int);
 	void		(*cpu_postboot)(void);
 #ifdef CONFIG_HOTPLUG_CPU
+	bool		(*cpu_can_disable)(unsigned int cpu);
 	int		(*cpu_disable)(unsigned int cpu);
 	void		(*cpu_die)(unsigned int cpu);
 	int		(*cpu_kill)(unsigned int cpu);
diff --git a/arch/arm64/kernel/psci.c b/arch/arm64/kernel/psci.c
index 85ee7d07889e..97902639feb3 100644
--- a/arch/arm64/kernel/psci.c
+++ b/arch/arm64/kernel/psci.c
@@ -46,6 +46,11 @@  static int cpu_psci_cpu_boot(unsigned int cpu)
 }
 
 #ifdef CONFIG_HOTPLUG_CPU
+static bool cpu_psci_cpu_can_disable(unsigned int cpu)
+{
+	return !psci_tos_resident_on(cpu);
+}
+
 static int cpu_psci_cpu_disable(unsigned int cpu)
 {
 	/* Fail early if we don't have CPU_OFF support */
@@ -113,6 +118,7 @@  const struct cpu_operations cpu_psci_ops = {
 	.cpu_prepare	= cpu_psci_cpu_prepare,
 	.cpu_boot	= cpu_psci_cpu_boot,
 #ifdef CONFIG_HOTPLUG_CPU
+	.cpu_can_disable = cpu_psci_cpu_can_disable,
 	.cpu_disable	= cpu_psci_cpu_disable,
 	.cpu_die	= cpu_psci_cpu_die,
 	.cpu_kill	= cpu_psci_cpu_kill,
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 413d566405d1..fb9915aa250d 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -363,6 +363,15 @@  void __init setup_arch(char **cmdline_p)
 	}
 }
 
+static inline bool cpu_can_disable(unsigned int cpu)
+{
+#ifdef CONFIG_HOTPLUG_CPU
+	if (cpu_ops[cpu] && cpu_ops[cpu]->cpu_can_disable)
+		return cpu_ops[cpu]->cpu_can_disable(cpu);
+#endif
+	return false;
+}
+
 static int __init topology_init(void)
 {
 	int i;
@@ -372,7 +381,7 @@  static int __init topology_init(void)
 
 	for_each_possible_cpu(i) {
 		struct cpu *cpu = &per_cpu(cpu_data.cpu, i);
-		cpu->hotpluggable = 1;
+		cpu->hotpluggable = cpu_can_disable(i);
 		register_cpu(cpu, i);
 	}