diff mbox

[RFC/PATCH] arm: do not skip SMP init calls on SMP_ON_UP case

Message ID 20151124153305.GD8644@n2100.arm.linux.org.uk (mailing list archive)
State New, archived
Headers show

Commit Message

Russell King - ARM Linux Nov. 24, 2015, 3:33 p.m. UTC
On Tue, Nov 24, 2015 at 05:52:14PM +0300, Nikita Yushchenko wrote:
> I'm still trying to understand what is going on, and my printk()s show
> that this is not entirely true.
> 
> When smp_init() is entered on mainline om imx6s, cpu_possible_mask and
> cpu_present_mask both contain two cpus. These get initialized in
> arm_dt_init_cpu_maps() and stay unmodified since then.
> 
> But cpu_online() returns 1 for cpu0 and 0 from cpu1 - thus it is
> cpu_online() check, not possible_mask or present_mask, that prevents
> cpu1 initialization attempt.

No.  cpu_online() reports whether the CPU is currently online or not.
It's the current state of the system wrt which CPUs are running and
not running.

Initially, only the boot CPU will be marked online, and the code
which brings all CPUs online (see smp_init() in kernel/smp.c) will
check whether the CPU is already online prior to trying to bring it
up.  It will attempt it for any present CPU, up to the maximum number
of online CPUs (set by nosmp or maxcpus kernel options.)

> Not sure I understand logic behind this. With the current code,
> resulting cpu_possible_mask depends on CONFIG_SMP_ON_UP:
> - if it is set, cpu_possible_mask contains (0 1), as initialized in
> arm_dt_init_cpu_maps()
> - if it is not set, cpu_possible_mask contains (0), since
> imx_smp_init_cpus() removes 1 from there.

Right, adding debug to arch/arm/kernel/setup.c, just before the
"if (is_smp())" shows:

is_smp() 0 possible 3 present 1 online 1

which is totally wrong: if is_smp() is false, we should not be setting
up any possible CPUs.  See a patch below to fix that.

However, this doesn't matter much, because the code in setup.c won't
initialise the SMP operations struct:

        if (is_smp()) {
                if (!mdesc->smp_init || !mdesc->smp_init()) {
                        if (psci_smp_available())
                                smp_set_ops(&psci_smp_ops);
                        else if (mdesc->smp)
                                smp_set_ops(mdesc->smp);
                }
                smp_init_cpus();
                smp_build_mpidr_hash();
        }

and this in turn means that __cpu_up() will return -ENOSYS due to this
check:

        if (!smp_ops.smp_boot_secondary)
                return -ENOSYS;

That causes _cpu_up() in kernel/cpu.c to bail out, along with cpu_up().
Notice that the call to smp_init() in kernel/smp.c is silent.

Hence, kernel/smp.c will try to bring CPU 1 online (because it is
marked present), but it'll _silently_ fail with -ENOSYS.

Here's the patch to fix the DT code, which should not be setting
present CPUs when is_smp() is false.

 arch/arm/kernel/devtree.c | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

Comments

Nikita Yushchenko Nov. 28, 2015, 11:13 a.m. UTC | #1
>> Not sure I understand logic behind this. With the current code,
>> resulting cpu_possible_mask depends on CONFIG_SMP_ON_UP:
>> - if it is set, cpu_possible_mask contains (0 1), as initialized in
>> arm_dt_init_cpu_maps()
>> - if it is not set, cpu_possible_mask contains (0), since
>> imx_smp_init_cpus() removes 1 from there.
> 
> Right, adding debug to arch/arm/kernel/setup.c, just before the
> "if (is_smp())" shows:
> 
> is_smp() 0 possible 3 present 1 online 1
> 
> which is totally wrong: if is_smp() is false, we should not be setting
> up any possible CPUs.  See a patch below to fix that.
> 
> However, this doesn't matter much, because the code in setup.c won't
> initialise the SMP operations struct ...

But cpu start code is not the only place in the kernel that uses cpu_present_mask.

Are you sure that running with invalid cpu_present_mask has no side effects?

> Here's the patch to fix the DT code, which should not be setting
> present CPUs when is_smp() is false.

I see that this fixes the issue as well.

But I still don't understand rationale behind all these is_smp() checks.
This makes init sequence different with and without CONFIG_SMP_ON_UP.
Isn't kernel intended to run ok without CONFIG_SMP_ON_UP?

And if yes - then why not run the same init sequence in both cases?
Nikita Yushchenko Nov. 30, 2015, 8:25 a.m. UTC | #2
28.11.2015 14:13, Nikita Yushchenko ?????:
>>> Not sure I understand logic behind this. With the current code,
>>> resulting cpu_possible_mask depends on CONFIG_SMP_ON_UP:
>>> - if it is set, cpu_possible_mask contains (0 1), as initialized in
>>> arm_dt_init_cpu_maps()
>>> - if it is not set, cpu_possible_mask contains (0), since
>>> imx_smp_init_cpus() removes 1 from there.
>>
>> Right, adding debug to arch/arm/kernel/setup.c, just before the
>> "if (is_smp())" shows:
>>
>> is_smp() 0 possible 3 present 1 online 1
>>
>> which is totally wrong: if is_smp() is false, we should not be setting
>> up any possible CPUs.  See a patch below to fix that.
>>
>> However, this doesn't matter much, because the code in setup.c won't
>> initialise the SMP operations struct ...
> 
> But cpu start code is not the only place in the kernel that uses cpu_present_mask.
> 
> Are you sure that running with invalid cpu_present_mask has no side effects?

At least LTP suite does not like it:
while running /opt/ltp/runtest/cpuhotplug, we see things like the above

<<<test_start>>>
tag=cpuhotplug02 stime=1446628761
cmdline="cpuhotplug02.sh -c 1 -l 1"
contacts=""
analysis=exit
<<<test_output>>>
Name:   cpuhotplug02
Date:   Wed Nov  4 09:19:21 UTC 2015
Desc:   What happens to a process when its CPU is offlined?

CPU is 1
/opt/ltp/testcases/bin/cpuhotplug_hotplug.sh: line 76: echo: write error:
Function not implemented
cpuhotplug02 1 TBROK : CPU1 cannot be onlined
<<<execution_status>>>
initiation_status="ok"
duration=1 termination_type=exited termination_id=2 corefile=no
cutime=4 cstime=6
<<<test_end>>>
diff mbox

Patch

diff --git a/arch/arm/kernel/devtree.c b/arch/arm/kernel/devtree.c
index 65addcbf5b30..bd72ce91d7a2 100644
--- a/arch/arm/kernel/devtree.c
+++ b/arch/arm/kernel/devtree.c
@@ -170,15 +170,18 @@  void __init arm_dt_init_cpu_maps(void)
 		return;
 	}
 
-	/*
-	 * Since the boot CPU node contains proper data, and all nodes have
-	 * a reg property, the DT CPU list can be considered valid and the
-	 * logical map created in smp_setup_processor_id() can be overridden
-	 */
-	for (i = 0; i < cpuidx; i++) {
-		set_cpu_possible(i, true);
-		cpu_logical_map(i) = tmp_map[i];
-		pr_debug("cpu logical map 0x%x\n", cpu_logical_map(i));
+	if (is_smp()) {
+		/*
+		 * Since the boot CPU node contains proper data, and all
+		 * nodes have a reg property, the DT CPU list can be
+		 * considered valid and the logical map created in
+		 * smp_setup_processor_id() can be overridden
+		 */
+		for (i = 0; i < cpuidx; i++) {
+			set_cpu_possible(i, true);
+			cpu_logical_map(i) = tmp_map[i];
+			pr_debug("cpu logical map 0x%x\n", cpu_logical_map(i));
+		}
 	}
 }