| Message ID | 87lgbk59gs.fsf@e105922-lin.cambridge.arm.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Tue 12-06-18 16:08:03, Punit Agrawal wrote: > Michal Hocko <mhocko@kernel.org> writes: [...] > > Well, the standard way to handle memory less NUMA nodes is to simply > > fallback to the closest NUMA node. We even have an API for that > > (numa_mem_id). > > CONFIG_HAVE_MEMORYLESS node is not enabled on arm64 which means we end > up returning the original node in the fallback path. Yes this makes more sense.
Punit Agrawal <punit.agrawal@arm.com> writes: [...] > > CONFIG_HAVE_MEMORYLESS node is not enabled on arm64 which means we end > up returning the original node in the fallback path. > > Xie, does the below patch help? I can submit a proper patch if this > fixes the issue for you. > > -- >8 -- > Subject: [PATCH] arm64/numa: Enable memoryless numa nodes > > Signed-off-by: Punit Agrawal <punit.agrawal@arm.com> > --- > arch/arm64/Kconfig | 4 ++++ > arch/arm64/mm/numa.c | 2 ++ > 2 files changed, 6 insertions(+) > > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig > index eb2cf4938f6d..5317e9aa93ab 100644 > --- a/arch/arm64/Kconfig > +++ b/arch/arm64/Kconfig > @@ -756,6 +756,10 @@ config USE_PERCPU_NUMA_NODE_ID > def_bool y > depends on NUMA > > +config HAVE_MEMORYLESS_NODES > + def_bool y > + depends on NUMA > + > config HAVE_SETUP_PER_CPU_AREA > def_bool y > depends on NUMA > diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c > index dad128ba98bf..c699dcfe93de 100644 > --- a/arch/arm64/mm/numa.c > +++ b/arch/arm64/mm/numa.c > @@ -73,6 +73,8 @@ EXPORT_SYMBOL(cpumask_of_node); > static void map_cpu_to_node(unsigned int cpu, int nid) > { > set_cpu_numa_node(cpu, nid); > + set_numa_mem(local_memory_node(nid)); Argh, this should be set_cpu_numa_mem(cpu, local_memory_node(nid)); There is not guarantee that map_cpu_to_node() will be called on the local cpu. Hanjun, Xie - can you try with the update please? Thanks, Punit > + > if (nid >= 0) > cpumask_set_cpu(cpu, node_to_cpumask_map[nid]); > }
Hi Punit, On 2018/6/14 1:39, Punit Agrawal wrote: > Punit Agrawal <punit.agrawal@arm.com> writes: > > > [...] > >> >> CONFIG_HAVE_MEMORYLESS node is not enabled on arm64 which means we end >> up returning the original node in the fallback path. >> >> Xie, does the below patch help? I can submit a proper patch if this >> fixes the issue for you. >> >> -- >8 -- >> Subject: [PATCH] arm64/numa: Enable memoryless numa nodes >> >> Signed-off-by: Punit Agrawal <punit.agrawal@arm.com> >> --- >> arch/arm64/Kconfig | 4 ++++ >> arch/arm64/mm/numa.c | 2 ++ >> 2 files changed, 6 insertions(+) >> >> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig >> index eb2cf4938f6d..5317e9aa93ab 100644 >> --- a/arch/arm64/Kconfig >> +++ b/arch/arm64/Kconfig >> @@ -756,6 +756,10 @@ config USE_PERCPU_NUMA_NODE_ID >> def_bool y >> depends on NUMA >> >> +config HAVE_MEMORYLESS_NODES >> + def_bool y >> + depends on NUMA >> + >> config HAVE_SETUP_PER_CPU_AREA >> def_bool y >> depends on NUMA >> diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c >> index dad128ba98bf..c699dcfe93de 100644 >> --- a/arch/arm64/mm/numa.c >> +++ b/arch/arm64/mm/numa.c >> @@ -73,6 +73,8 @@ EXPORT_SYMBOL(cpumask_of_node); >> static void map_cpu_to_node(unsigned int cpu, int nid) >> { >> set_cpu_numa_node(cpu, nid); >> + set_numa_mem(local_memory_node(nid)); > > Argh, this should be > > set_cpu_numa_mem(cpu, local_memory_node(nid)); > > There is not guarantee that map_cpu_to_node() will be called on the > local cpu. > > Hanjun, Xie - can you try with the update please? Thanks for looking into this, we will try this tomorrow (the hardware is occupied now) and update here. Thanks Hanjun
Hi Punit, On 2018/6/14 1:39, Punit Agrawal wrote: > Punit Agrawal <punit.agrawal@arm.com> writes: > > > [...] > >> >> CONFIG_HAVE_MEMORYLESS node is not enabled on arm64 which means we end >> up returning the original node in the fallback path. >> >> Xie, does the below patch help? I can submit a proper patch if this >> fixes the issue for you. >> >> -- >8 -- >> Subject: [PATCH] arm64/numa: Enable memoryless numa nodes >> >> Signed-off-by: Punit Agrawal <punit.agrawal@arm.com> >> --- >> arch/arm64/Kconfig | 4 ++++ >> arch/arm64/mm/numa.c | 2 ++ >> 2 files changed, 6 insertions(+) >> >> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig >> index eb2cf4938f6d..5317e9aa93ab 100644 >> --- a/arch/arm64/Kconfig >> +++ b/arch/arm64/Kconfig >> @@ -756,6 +756,10 @@ config USE_PERCPU_NUMA_NODE_ID >> def_bool y >> depends on NUMA >> >> +config HAVE_MEMORYLESS_NODES >> + def_bool y >> + depends on NUMA >> + >> config HAVE_SETUP_PER_CPU_AREA >> def_bool y >> depends on NUMA >> diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c >> index dad128ba98bf..c699dcfe93de 100644 >> --- a/arch/arm64/mm/numa.c >> +++ b/arch/arm64/mm/numa.c >> @@ -73,6 +73,8 @@ EXPORT_SYMBOL(cpumask_of_node); >> static void map_cpu_to_node(unsigned int cpu, int nid) >> { >> set_cpu_numa_node(cpu, nid); >> + set_numa_mem(local_memory_node(nid)); > > Argh, this should be > > set_cpu_numa_mem(cpu, local_memory_node(nid)); > > There is not guarantee that map_cpu_to_node() will be called on the > local cpu. > > Hanjun, Xie - can you try with the update please? I've tested this patch, but it does not help. The boot message is attached. I tested on a arm board with 128 cores 4 numa nodes, but I set CONFIG_NR_CPUS=72. Then node 3 is not be created, because node 3 has no memory, and no cpu. But some pci device may related to node 3, which be set in ACPI table. 165 /* Interface called from ACPI code to setup PCI host controller */ 166 struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root) 167 { 168 int node = acpi_get_node(root->device->handle); 169 struct acpi_pci_generic_root_info *ri; 170 struct pci_bus *bus, *child; 171 struct acpi_pci_root_ops *root_ops; 172 // this node may be not created. 177 ri = kzalloc_node(sizeof(*ri), GFP_KERNEL, node); 178 if (!ri) 179 return NULL; 180 181 root_ops = kzalloc_node(sizeof(*root_ops), GFP_KERNEL, node); 182 if (!root_ops) { 183 kfree(ri); 184 return NULL; 185 } 186 187 ri->cfg = pci_acpi_setup_ecam_mapping(root); 188 if (!ri->cfg) { 189 kfree(ri); 190 kfree(root_ops); 191 return NULL; 192 } > > Thanks, > Punit > >> + >> if (nid >= 0) >> cpumask_set_cpu(cpu, node_to_cpumask_map[nid]); >> } > > . >
On Tue 19-06-18 20:03:07, Xie XiuQi wrote: [...] > I tested on a arm board with 128 cores 4 numa nodes, but I set CONFIG_NR_CPUS=72. > Then node 3 is not be created, because node 3 has no memory, and no cpu. > But some pci device may related to node 3, which be set in ACPI table. Could you double check that zonelists for node 3 are generated correctly?
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index eb2cf4938f6d..5317e9aa93ab 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -756,6 +756,10 @@ config USE_PERCPU_NUMA_NODE_ID def_bool y depends on NUMA +config HAVE_MEMORYLESS_NODES + def_bool y + depends on NUMA + config HAVE_SETUP_PER_CPU_AREA def_bool y depends on NUMA diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c index dad128ba98bf..c699dcfe93de 100644 --- a/arch/arm64/mm/numa.c +++ b/arch/arm64/mm/numa.c @@ -73,6 +73,8 @@ EXPORT_SYMBOL(cpumask_of_node); static void map_cpu_to_node(unsigned int cpu, int nid) { set_cpu_numa_node(cpu, nid); + set_numa_mem(local_memory_node(nid)); + if (nid >= 0) cpumask_set_cpu(cpu, node_to_cpumask_map[nid]); }