| Message ID | 5200CBBF.1090904@jp.fujitsu.com (mailing list archive) |
|---|---|
| State | Accepted, archived |
| Headers | show |
On Tue, 2013-08-06 at 19:11 +0900, Yasuaki Ishimatsu wrote: > try_offline_node() checks that all cpus related with removed node have been > removed by using cpu_present_bits. If all cpus related with removed node have > been removed, try_offline_node() clears the node information. > > But try_offline_node() called from acpi_processor_remove() never clears > the node information. For disabling cpu_present_bits, acpi_unmap_lsapic() > need be called. But acpi_unmap_lsapic() is called after try_offline_node() > runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always > set. > > This patch moves try_offline_node() after acpi_unmap_lsapic(). > > Signed-off-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com> The change looks good to me. Acked-by: Toshi Kani <toshi.kani@hp.com> BTW, do you know why try_offline_node() has to use stop_machine()? Thanks, -Toshi -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
(2013/08/07 1:56), Toshi Kani wrote: > On Tue, 2013-08-06 at 19:11 +0900, Yasuaki Ishimatsu wrote: >> try_offline_node() checks that all cpus related with removed node have been >> removed by using cpu_present_bits. If all cpus related with removed node have >> been removed, try_offline_node() clears the node information. >> >> But try_offline_node() called from acpi_processor_remove() never clears >> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic() >> need be called. But acpi_unmap_lsapic() is called after try_offline_node() >> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always >> set. >> >> This patch moves try_offline_node() after acpi_unmap_lsapic(). >> >> Signed-off-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com> > > The change looks good to me. > > Acked-by: Toshi Kani <toshi.kani@hp.com> Thanks, Toshi! > BTW, do you know why try_offline_node() has to use stop_machine()? I don't know the reason. Thanks, Yasuaki Ishimatsu > > Thanks, > -Toshi > -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Ishimatsu-san, On 08/06/2013 06:11 PM, Yasuaki Ishimatsu wrote: > try_offline_node() checks that all cpus related with removed node have been > removed by using cpu_present_bits. If all cpus related with removed node have > been removed, try_offline_node() clears the node information. > > But try_offline_node() called from acpi_processor_remove() never clears > the node information. For disabling cpu_present_bits, acpi_unmap_lsapic() > need be called. But acpi_unmap_lsapic() is called after try_offline_node() > runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always > set. > > This patch moves try_offline_node() after acpi_unmap_lsapic(). It works well. > > Signed-off-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com> Tested-by: Gu Zheng <guz.fnst@cn.fujitsu.com> Thanks, Gu > --- > drivers/acpi/acpi_processor.c | 3 ++- > 1 files changed, 2 insertions(+), 1 deletions(-) > > diff --git a/drivers/acpi/acpi_processor.c b/drivers/acpi/acpi_processor.c > index fd6c51c..5a74a9c 100644 > --- a/drivers/acpi/acpi_processor.c > +++ b/drivers/acpi/acpi_processor.c > @@ -451,7 +451,6 @@ static void acpi_processor_remove(struct acpi_device *device) > /* Clean up. */ > per_cpu(processor_device_array, pr->id) = NULL; > per_cpu(processors, pr->id) = NULL; > - try_offline_node(cpu_to_node(pr->id)); > > /* Remove the CPU. */ > get_online_cpus(); > @@ -459,6 +458,8 @@ static void acpi_processor_remove(struct acpi_device *device) > acpi_unmap_lsapic(pr->id); > put_online_cpus(); > > + try_offline_node(cpu_to_node(pr->id)); > + > out: > free_cpumask_var(pr->throttling.shared_cpu_map); > kfree(pr); > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
(2013/08/09 11:43), Gu Zheng wrote: > Hi Ishimatsu-san, > On 08/06/2013 06:11 PM, Yasuaki Ishimatsu wrote: > >> try_offline_node() checks that all cpus related with removed node have been >> removed by using cpu_present_bits. If all cpus related with removed node have >> been removed, try_offline_node() clears the node information. >> >> But try_offline_node() called from acpi_processor_remove() never clears >> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic() >> need be called. But acpi_unmap_lsapic() is called after try_offline_node() >> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always >> set. >> >> This patch moves try_offline_node() after acpi_unmap_lsapic(). > > It works well. > >> >> Signed-off-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com> > > > Tested-by: Gu Zheng <guz.fnst@cn.fujitsu.com> Thank you for testing. Thanks, Yasuaki Ishimatsu > > Thanks, > Gu > >> --- >> drivers/acpi/acpi_processor.c | 3 ++- >> 1 files changed, 2 insertions(+), 1 deletions(-) >> >> diff --git a/drivers/acpi/acpi_processor.c b/drivers/acpi/acpi_processor.c >> index fd6c51c..5a74a9c 100644 >> --- a/drivers/acpi/acpi_processor.c >> +++ b/drivers/acpi/acpi_processor.c >> @@ -451,7 +451,6 @@ static void acpi_processor_remove(struct acpi_device *device) >> /* Clean up. */ >> per_cpu(processor_device_array, pr->id) = NULL; >> per_cpu(processors, pr->id) = NULL; >> - try_offline_node(cpu_to_node(pr->id)); >> >> /* Remove the CPU. */ >> get_online_cpus(); >> @@ -459,6 +458,8 @@ static void acpi_processor_remove(struct acpi_device *device) >> acpi_unmap_lsapic(pr->id); >> put_online_cpus(); >> >> + try_offline_node(cpu_to_node(pr->id)); >> + >> out: >> free_cpumask_var(pr->throttling.shared_cpu_map); >> kfree(pr); >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> Please read the FAQ at http://www.tux.org/lkml/ >> > > -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 08/07/2013 12:56 AM, Toshi Kani wrote: > On Tue, 2013-08-06 at 19:11 +0900, Yasuaki Ishimatsu wrote: >> try_offline_node() checks that all cpus related with removed node have been >> removed by using cpu_present_bits. If all cpus related with removed node have >> been removed, try_offline_node() clears the node information. >> >> But try_offline_node() called from acpi_processor_remove() never clears >> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic() >> need be called. But acpi_unmap_lsapic() is called after try_offline_node() >> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always >> set. >> >> This patch moves try_offline_node() after acpi_unmap_lsapic(). >> >> Signed-off-by: Yasuaki Ishimatsu<isimatu.yasuaki@jp.fujitsu.com> > > The change looks good to me. > > Acked-by: Toshi Kani<toshi.kani@hp.com> > > BTW, do you know why try_offline_node() has to use stop_machine()? try_offline_node() is used to check if the node could be hot-removed after each memory or cpu hot-remove operation. In memory hot-remove path, we have lock_memory_hotplug() to series all the memory hot-remove options. But when doing cpu hot-remove, acpi_processor_remove() |->try_offline_node() There is no lock to protect it. I think, when we are going to hot-remove a node, others should not do any memory or cpu hotplug operation. In memory hotplug path, we have lock_memory_hotplug(). But in cpu hotplug path, I didn't find any lock. So we used stop_machine() to call check_cpu_on_node(). If we find any cpu still present, we return and do not remove the node. Thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, 2013-08-09 at 15:28 +0800, Tang Chen wrote: > On 08/07/2013 12:56 AM, Toshi Kani wrote: > > On Tue, 2013-08-06 at 19:11 +0900, Yasuaki Ishimatsu wrote: > >> try_offline_node() checks that all cpus related with removed node have been > >> removed by using cpu_present_bits. If all cpus related with removed node have > >> been removed, try_offline_node() clears the node information. > >> > >> But try_offline_node() called from acpi_processor_remove() never clears > >> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic() > >> need be called. But acpi_unmap_lsapic() is called after try_offline_node() > >> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always > >> set. > >> > >> This patch moves try_offline_node() after acpi_unmap_lsapic(). > >> > >> Signed-off-by: Yasuaki Ishimatsu<isimatu.yasuaki@jp.fujitsu.com> > > > > The change looks good to me. > > > > Acked-by: Toshi Kani<toshi.kani@hp.com> > > > > BTW, do you know why try_offline_node() has to use stop_machine()? > > try_offline_node() is used to check if the node could be hot-removed > after each memory or cpu hot-remove operation. > > In memory hot-remove path, we have lock_memory_hotplug() to series all > the memory hot-remove options. > > But when doing cpu hot-remove, > > acpi_processor_remove() > |->try_offline_node() > > There is no lock to protect it. I think, when we are going to hot-remove > a node, others should not do any memory or cpu hotplug operation. In memory > hotplug path, we have lock_memory_hotplug(). But in cpu hotplug path, I > didn't find any lock. So we used stop_machine() to call check_cpu_on_node(). > If we find any cpu still present, we return and do not remove the node. CPU/Memory hotplug operations and sysfs eject are serialized with acpi_os_hotplug_execute(). CPU online/offline is protected by cpu_hotplug_[begin|done]() and [get|put]_online_cpus(). But, yes, online/offline and hotplug operations are not serialized. I tried to serialize them before, but that framework was not well received. Anyway, it looks to me that cpu_up()->mem_online_node() path can race with try_offline_node(). And I do not think stop_machine() protects this case, either. try_offline_node() stops the execution of num_online_node() during stop_machine(), but then lets it continues to run after that. I think they need to be protected by a lock. I will look further to see if I can come up with some solution. Thanks, -Toshi -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Friday, August 09, 2013 04:16:56 PM Toshi Kani wrote: > On Fri, 2013-08-09 at 15:28 +0800, Tang Chen wrote: > > On 08/07/2013 12:56 AM, Toshi Kani wrote: > > > On Tue, 2013-08-06 at 19:11 +0900, Yasuaki Ishimatsu wrote: > > >> try_offline_node() checks that all cpus related with removed node have been > > >> removed by using cpu_present_bits. If all cpus related with removed node have > > >> been removed, try_offline_node() clears the node information. > > >> > > >> But try_offline_node() called from acpi_processor_remove() never clears > > >> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic() > > >> need be called. But acpi_unmap_lsapic() is called after try_offline_node() > > >> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always > > >> set. > > >> > > >> This patch moves try_offline_node() after acpi_unmap_lsapic(). > > >> > > >> Signed-off-by: Yasuaki Ishimatsu<isimatu.yasuaki@jp.fujitsu.com> > > > > > > The change looks good to me. > > > > > > Acked-by: Toshi Kani<toshi.kani@hp.com> > > > > > > BTW, do you know why try_offline_node() has to use stop_machine()? > > > > try_offline_node() is used to check if the node could be hot-removed > > after each memory or cpu hot-remove operation. > > > > In memory hot-remove path, we have lock_memory_hotplug() to series all > > the memory hot-remove options. > > > > But when doing cpu hot-remove, > > > > acpi_processor_remove() > > |->try_offline_node() > > > > There is no lock to protect it. I think, when we are going to hot-remove > > a node, others should not do any memory or cpu hotplug operation. In memory > > hotplug path, we have lock_memory_hotplug(). But in cpu hotplug path, I > > didn't find any lock. So we used stop_machine() to call check_cpu_on_node(). > > If we find any cpu still present, we return and do not remove the node. > > CPU/Memory hotplug operations and sysfs eject are serialized with > acpi_os_hotplug_execute(). CPU online/offline is protected by > cpu_hotplug_[begin|done]() and [get|put]_online_cpus(). But, yes, > online/offline and hotplug operations are not serialized. I tried to > serialize them before, but that framework was not well received. What about lock_device_hotplug()? It is taken by both online/offline and the ACPI hotplug code, isn't it? > Anyway, it looks to me that cpu_up()->mem_online_node() path can race > with try_offline_node(). It can in principle, but I'm not sure if there's a way to trigger that race. Do you have an example? Rafael
On Sat, 2013-08-10 at 01:29 +0200, Rafael J. Wysocki wrote: > On Friday, August 09, 2013 04:16:56 PM Toshi Kani wrote: > > On Fri, 2013-08-09 at 15:28 +0800, Tang Chen wrote: > > > On 08/07/2013 12:56 AM, Toshi Kani wrote: > > > > On Tue, 2013-08-06 at 19:11 +0900, Yasuaki Ishimatsu wrote: > > > >> try_offline_node() checks that all cpus related with removed node have been > > > >> removed by using cpu_present_bits. If all cpus related with removed node have > > > >> been removed, try_offline_node() clears the node information. > > > >> > > > >> But try_offline_node() called from acpi_processor_remove() never clears > > > >> the node information. For disabling cpu_present_bits, acpi_unmap_lsapic() > > > >> need be called. But acpi_unmap_lsapic() is called after try_offline_node() > > > >> runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always > > > >> set. > > > >> > > > >> This patch moves try_offline_node() after acpi_unmap_lsapic(). > > > >> > > > >> Signed-off-by: Yasuaki Ishimatsu<isimatu.yasuaki@jp.fujitsu.com> > > > > > > > > The change looks good to me. > > > > > > > > Acked-by: Toshi Kani<toshi.kani@hp.com> > > > > > > > > BTW, do you know why try_offline_node() has to use stop_machine()? > > > > > > try_offline_node() is used to check if the node could be hot-removed > > > after each memory or cpu hot-remove operation. > > > > > > In memory hot-remove path, we have lock_memory_hotplug() to series all > > > the memory hot-remove options. > > > > > > But when doing cpu hot-remove, > > > > > > acpi_processor_remove() > > > |->try_offline_node() > > > > > > There is no lock to protect it. I think, when we are going to hot-remove > > > a node, others should not do any memory or cpu hotplug operation. In memory > > > hotplug path, we have lock_memory_hotplug(). But in cpu hotplug path, I > > > didn't find any lock. So we used stop_machine() to call check_cpu_on_node(). > > > If we find any cpu still present, we return and do not remove the node. > > > > CPU/Memory hotplug operations and sysfs eject are serialized with > > acpi_os_hotplug_execute(). CPU online/offline is protected by > > cpu_hotplug_[begin|done]() and [get|put]_online_cpus(). But, yes, > > online/offline and hotplug operations are not serialized. I tried to > > serialize them before, but that framework was not well received. > > What about lock_device_hotplug()? It is taken by both online/offline and > the ACPI hotplug code, isn't it? Oh, that's right! I forgot about this one. Yes, lock_device_hotplug() nicely protects online/offline and hotplug operations. :-) > > Anyway, it looks to me that cpu_up()->mem_online_node() path can race > > with try_offline_node(). > > It can in principle, but I'm not sure if there's a way to trigger that > race. Do you have an example? With lock_device_hotplug(), I agree that we do not have this race condition -- cpu_up() may not run while other hotplug is running. store_online() will be blocked at lock_device_hotplug() in such case. When store_online() acquired the lock, this CPU may have been deleted. So, we still need to make sure that this case is handled properly. I suppose sysfs keeps *dev valid with ref_count (Is that right?). I think cpu_up() needs to check with cpu_present(), not cpu_possible(), at the top. Otherwise, cpu_to_node(cpu) may return NUMA_NO_NODE (-1), which is probably not a good value for node_online(nid). Thanks, -Toshi -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/acpi/acpi_processor.c b/drivers/acpi/acpi_processor.c index fd6c51c..5a74a9c 100644 --- a/drivers/acpi/acpi_processor.c +++ b/drivers/acpi/acpi_processor.c @@ -451,7 +451,6 @@ static void acpi_processor_remove(struct acpi_device *device) /* Clean up. */ per_cpu(processor_device_array, pr->id) = NULL; per_cpu(processors, pr->id) = NULL; - try_offline_node(cpu_to_node(pr->id)); /* Remove the CPU. */ get_online_cpus(); @@ -459,6 +458,8 @@ static void acpi_processor_remove(struct acpi_device *device) acpi_unmap_lsapic(pr->id); put_online_cpus(); + try_offline_node(cpu_to_node(pr->id)); + out: free_cpumask_var(pr->throttling.shared_cpu_map); kfree(pr);
try_offline_node() checks that all cpus related with removed node have been removed by using cpu_present_bits. If all cpus related with removed node have been removed, try_offline_node() clears the node information. But try_offline_node() called from acpi_processor_remove() never clears the node information. For disabling cpu_present_bits, acpi_unmap_lsapic() need be called. But acpi_unmap_lsapic() is called after try_offline_node() runs. So when try_offline_node() runs, the cpu's cpu_present_bits is always set. This patch moves try_offline_node() after acpi_unmap_lsapic(). Signed-off-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com> --- drivers/acpi/acpi_processor.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html