| Message ID | 20230317062542.61061-1-gshan@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | NUMA: Apply cluster-NUMA-node boundary for aarch64 and riscv machines | expand |
On Fri, 17 Mar 2023 14:25:39 +0800 Gavin Shan <gshan@redhat.com> wrote: > For arm64 and riscv architecture, the driver (/base/arch_topology.c) is > used to populate the CPU topology in the Linux guest. It's required that > the CPUs in one cluster can't span mutiple NUMA nodes. Otherwise, the Linux > scheduling domain can't be sorted out, as the following warning message > indicates. To avoid the unexpected confusion, this series attempts to > warn about such kind of irregular configurations. > > -smp 6,maxcpus=6,sockets=2,clusters=1,cores=3,threads=1 \ > -numa node,nodeid=0,cpus=0-1,memdev=ram0 \ > -numa node,nodeid=1,cpus=2-3,memdev=ram1 \ > -numa node,nodeid=2,cpus=4-5,memdev=ram2 \ > > ------------[ cut here ]------------ > WARNING: CPU: 0 PID: 1 at kernel/sched/topology.c:2271 build_sched_domains+0x284/0x910 > Modules linked in: > CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.14.0-268.el9.aarch64 #1 > pstate: 00400005 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) > pc : build_sched_domains+0x284/0x910 > lr : build_sched_domains+0x184/0x910 > sp : ffff80000804bd50 > x29: ffff80000804bd50 x28: 0000000000000002 x27: 0000000000000000 > x26: ffff800009cf9a80 x25: 0000000000000000 x24: ffff800009cbf840 > x23: ffff000080325000 x22: ffff0000005df800 x21: ffff80000a4ce508 > x20: 0000000000000000 x19: ffff000080324440 x18: 0000000000000014 > x17: 00000000388925c0 x16: 000000005386a066 x15: 000000009c10cc2e > x14: 00000000000001c0 x13: 0000000000000001 x12: ffff00007fffb1a0 > x11: ffff00007fffb180 x10: ffff80000a4ce508 x9 : 0000000000000041 > x8 : ffff80000a4ce500 x7 : ffff80000a4cf920 x6 : 0000000000000001 > x5 : 0000000000000001 x4 : 0000000000000007 x3 : 0000000000000002 > x2 : 0000000000001000 x1 : ffff80000a4cf928 x0 : 0000000000000001 > Call trace: > build_sched_domains+0x284/0x910 > sched_init_domains+0xac/0xe0 > sched_init_smp+0x48/0xc8 > kernel_init_freeable+0x140/0x1ac > kernel_init+0x28/0x140 > ret_from_fork+0x10/0x20 > > PATCH[1] Warn about the irregular configuration if required > PATCH[2] Enable the validation for aarch64 machines > PATCH[3] Enable the validation for riscv machines > > v3: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg01226.html > v2: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg01080.html > v1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00886.html > > Changelog > ========= > v4: > * Pick r-b and ack-b from Daniel/Philippe (Gavin) > * Replace local variable @len with possible_cpus->len in > validate_cpu_cluster_to_numa_boundary() (Philippe) > v3: > * Validate cluster-to-NUMA instead of socket-to-NUMA > boundary (Gavin) > * Move the switch from MachineState to MachineClass (Philippe) > * Warning instead of rejecting the irregular configuration (Daniel) > * Comments to mention cluster-to-NUMA is platform instead > of architectural choice (Drew) > * Drop PATCH[v2 1/4] related to qtests/numa-test (Gavin) > v2: > * Fix socket-NUMA-node boundary issues in qtests/numa-test (Gavin) > * Add helper set_numa_socket_boundary() and validate the > boundary in the generic path (Philippe) > > Gavin Shan (3): > numa: Validate cluster and NUMA node boundary if required > hw/arm: Validate cluster and NUMA node boundary > hw/riscv: Validate cluster and NUMA node boundary > > hw/arm/sbsa-ref.c | 2 ++ > hw/arm/virt.c | 2 ++ > hw/core/machine.c | 42 ++++++++++++++++++++++++++++++++++++++++++ > hw/riscv/spike.c | 2 ++ > hw/riscv/virt.c | 2 ++ > include/hw/boards.h | 1 + > 6 files changed, 51 insertions(+) > Acked-by: Igor Mammedov <imammedo@redhat.com>
Hi Peter, On 3/27/23 9:26 PM, Igor Mammedov wrote: > On Fri, 17 Mar 2023 14:25:39 +0800 > Gavin Shan <gshan@redhat.com> wrote: > >> For arm64 and riscv architecture, the driver (/base/arch_topology.c) is >> used to populate the CPU topology in the Linux guest. It's required that >> the CPUs in one cluster can't span mutiple NUMA nodes. Otherwise, the Linux >> scheduling domain can't be sorted out, as the following warning message >> indicates. To avoid the unexpected confusion, this series attempts to >> warn about such kind of irregular configurations. >> >> -smp 6,maxcpus=6,sockets=2,clusters=1,cores=3,threads=1 \ >> -numa node,nodeid=0,cpus=0-1,memdev=ram0 \ >> -numa node,nodeid=1,cpus=2-3,memdev=ram1 \ >> -numa node,nodeid=2,cpus=4-5,memdev=ram2 \ >> >> ------------[ cut here ]------------ >> WARNING: CPU: 0 PID: 1 at kernel/sched/topology.c:2271 build_sched_domains+0x284/0x910 >> Modules linked in: >> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.14.0-268.el9.aarch64 #1 >> pstate: 00400005 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) >> pc : build_sched_domains+0x284/0x910 >> lr : build_sched_domains+0x184/0x910 >> sp : ffff80000804bd50 >> x29: ffff80000804bd50 x28: 0000000000000002 x27: 0000000000000000 >> x26: ffff800009cf9a80 x25: 0000000000000000 x24: ffff800009cbf840 >> x23: ffff000080325000 x22: ffff0000005df800 x21: ffff80000a4ce508 >> x20: 0000000000000000 x19: ffff000080324440 x18: 0000000000000014 >> x17: 00000000388925c0 x16: 000000005386a066 x15: 000000009c10cc2e >> x14: 00000000000001c0 x13: 0000000000000001 x12: ffff00007fffb1a0 >> x11: ffff00007fffb180 x10: ffff80000a4ce508 x9 : 0000000000000041 >> x8 : ffff80000a4ce500 x7 : ffff80000a4cf920 x6 : 0000000000000001 >> x5 : 0000000000000001 x4 : 0000000000000007 x3 : 0000000000000002 >> x2 : 0000000000001000 x1 : ffff80000a4cf928 x0 : 0000000000000001 >> Call trace: >> build_sched_domains+0x284/0x910 >> sched_init_domains+0xac/0xe0 >> sched_init_smp+0x48/0xc8 >> kernel_init_freeable+0x140/0x1ac >> kernel_init+0x28/0x140 >> ret_from_fork+0x10/0x20 >> >> PATCH[1] Warn about the irregular configuration if required >> PATCH[2] Enable the validation for aarch64 machines >> PATCH[3] Enable the validation for riscv machines >> >> v3: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg01226.html >> v2: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg01080.html >> v1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00886.html >> >> Changelog >> ========= >> v4: >> * Pick r-b and ack-b from Daniel/Philippe (Gavin) >> * Replace local variable @len with possible_cpus->len in >> validate_cpu_cluster_to_numa_boundary() (Philippe) >> v3: >> * Validate cluster-to-NUMA instead of socket-to-NUMA >> boundary (Gavin) >> * Move the switch from MachineState to MachineClass (Philippe) >> * Warning instead of rejecting the irregular configuration (Daniel) >> * Comments to mention cluster-to-NUMA is platform instead >> of architectural choice (Drew) >> * Drop PATCH[v2 1/4] related to qtests/numa-test (Gavin) >> v2: >> * Fix socket-NUMA-node boundary issues in qtests/numa-test (Gavin) >> * Add helper set_numa_socket_boundary() and validate the >> boundary in the generic path (Philippe) >> >> Gavin Shan (3): >> numa: Validate cluster and NUMA node boundary if required >> hw/arm: Validate cluster and NUMA node boundary >> hw/riscv: Validate cluster and NUMA node boundary >> >> hw/arm/sbsa-ref.c | 2 ++ >> hw/arm/virt.c | 2 ++ >> hw/core/machine.c | 42 ++++++++++++++++++++++++++++++++++++++++++ >> hw/riscv/spike.c | 2 ++ >> hw/riscv/virt.c | 2 ++ >> include/hw/boards.h | 1 + >> 6 files changed, 51 insertions(+) >> > > Acked-by: Igor Mammedov <imammedo@redhat.com> > Not sure if QEMU v8.0 is still available to integrate this series. Otherwise, it should be something for QEMU v8.1. By the way, I'm also uncertain who needs to be merge this series. Thanks, Gavin
On Wed, 12 Apr 2023 at 02:08, Gavin Shan <gshan@redhat.com> wrote: > > Hi Peter, > > On 3/27/23 9:26 PM, Igor Mammedov wrote: > > On Fri, 17 Mar 2023 14:25:39 +0800 > > Gavin Shan <gshan@redhat.com> wrote: > > > >> For arm64 and riscv architecture, the driver (/base/arch_topology.c) is > >> used to populate the CPU topology in the Linux guest. It's required that > >> the CPUs in one cluster can't span mutiple NUMA nodes. Otherwise, the Linux > >> scheduling domain can't be sorted out, as the following warning message > >> indicates. To avoid the unexpected confusion, this series attempts to > >> warn about such kind of irregular configurations. > >> > >> -smp 6,maxcpus=6,sockets=2,clusters=1,cores=3,threads=1 \ > >> -numa node,nodeid=0,cpus=0-1,memdev=ram0 \ > >> -numa node,nodeid=1,cpus=2-3,memdev=ram1 \ > >> -numa node,nodeid=2,cpus=4-5,memdev=ram2 \ > >> > >> ------------[ cut here ]------------ > >> WARNING: CPU: 0 PID: 1 at kernel/sched/topology.c:2271 build_sched_domains+0x284/0x910 > >> Modules linked in: > >> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.14.0-268.el9.aarch64 #1 > >> pstate: 00400005 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) > >> pc : build_sched_domains+0x284/0x910 > >> lr : build_sched_domains+0x184/0x910 > >> sp : ffff80000804bd50 > >> x29: ffff80000804bd50 x28: 0000000000000002 x27: 0000000000000000 > >> x26: ffff800009cf9a80 x25: 0000000000000000 x24: ffff800009cbf840 > >> x23: ffff000080325000 x22: ffff0000005df800 x21: ffff80000a4ce508 > >> x20: 0000000000000000 x19: ffff000080324440 x18: 0000000000000014 > >> x17: 00000000388925c0 x16: 000000005386a066 x15: 000000009c10cc2e > >> x14: 00000000000001c0 x13: 0000000000000001 x12: ffff00007fffb1a0 > >> x11: ffff00007fffb180 x10: ffff80000a4ce508 x9 : 0000000000000041 > >> x8 : ffff80000a4ce500 x7 : ffff80000a4cf920 x6 : 0000000000000001 > >> x5 : 0000000000000001 x4 : 0000000000000007 x3 : 0000000000000002 > >> x2 : 0000000000001000 x1 : ffff80000a4cf928 x0 : 0000000000000001 > >> Call trace: > >> build_sched_domains+0x284/0x910 > >> sched_init_domains+0xac/0xe0 > >> sched_init_smp+0x48/0xc8 > >> kernel_init_freeable+0x140/0x1ac > >> kernel_init+0x28/0x140 > >> ret_from_fork+0x10/0x20 > >> > >> PATCH[1] Warn about the irregular configuration if required > >> PATCH[2] Enable the validation for aarch64 machines > >> PATCH[3] Enable the validation for riscv machines > >> > >> v3: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg01226.html > >> v2: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg01080.html > >> v1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00886.html > >> > >> Changelog > >> ========= > >> v4: > >> * Pick r-b and ack-b from Daniel/Philippe (Gavin) > >> * Replace local variable @len with possible_cpus->len in > >> validate_cpu_cluster_to_numa_boundary() (Philippe) > >> v3: > >> * Validate cluster-to-NUMA instead of socket-to-NUMA > >> boundary (Gavin) > >> * Move the switch from MachineState to MachineClass (Philippe) > >> * Warning instead of rejecting the irregular configuration (Daniel) > >> * Comments to mention cluster-to-NUMA is platform instead > >> of architectural choice (Drew) > >> * Drop PATCH[v2 1/4] related to qtests/numa-test (Gavin) > >> v2: > >> * Fix socket-NUMA-node boundary issues in qtests/numa-test (Gavin) > >> * Add helper set_numa_socket_boundary() and validate the > >> boundary in the generic path (Philippe) > >> > >> Gavin Shan (3): > >> numa: Validate cluster and NUMA node boundary if required > >> hw/arm: Validate cluster and NUMA node boundary > >> hw/riscv: Validate cluster and NUMA node boundary > >> > >> hw/arm/sbsa-ref.c | 2 ++ > >> hw/arm/virt.c | 2 ++ > >> hw/core/machine.c | 42 ++++++++++++++++++++++++++++++++++++++++++ > >> hw/riscv/spike.c | 2 ++ > >> hw/riscv/virt.c | 2 ++ > >> include/hw/boards.h | 1 + > >> 6 files changed, 51 insertions(+) > >> > > > > Acked-by: Igor Mammedov <imammedo@redhat.com> > > > > Not sure if QEMU v8.0 is still available to integrate this series. > Otherwise, it should be something for QEMU v8.1. By the way, I'm > also uncertain who needs to be merge this series. It barely touches arm specific boards, so I'm assuming it will be reviewed and taken by whoever handles hw/core/machine.c And yes, 8.0 is nearly out the door, this is 8.1 stuff. thanks -- PMM
On 4/12/23 7:42 PM, Peter Maydell wrote: > On Wed, 12 Apr 2023 at 02:08, Gavin Shan <gshan@redhat.com> wrote: >> On 3/27/23 9:26 PM, Igor Mammedov wrote: >>> On Fri, 17 Mar 2023 14:25:39 +0800 >>> Gavin Shan <gshan@redhat.com> wrote: >>> >>>> For arm64 and riscv architecture, the driver (/base/arch_topology.c) is >>>> used to populate the CPU topology in the Linux guest. It's required that >>>> the CPUs in one cluster can't span mutiple NUMA nodes. Otherwise, the Linux >>>> scheduling domain can't be sorted out, as the following warning message >>>> indicates. To avoid the unexpected confusion, this series attempts to >>>> warn about such kind of irregular configurations. >>>> >>>> -smp 6,maxcpus=6,sockets=2,clusters=1,cores=3,threads=1 \ >>>> -numa node,nodeid=0,cpus=0-1,memdev=ram0 \ >>>> -numa node,nodeid=1,cpus=2-3,memdev=ram1 \ >>>> -numa node,nodeid=2,cpus=4-5,memdev=ram2 \ >>>> >>>> ------------[ cut here ]------------ >>>> WARNING: CPU: 0 PID: 1 at kernel/sched/topology.c:2271 build_sched_domains+0x284/0x910 >>>> Modules linked in: >>>> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.14.0-268.el9.aarch64 #1 >>>> pstate: 00400005 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) >>>> pc : build_sched_domains+0x284/0x910 >>>> lr : build_sched_domains+0x184/0x910 >>>> sp : ffff80000804bd50 >>>> x29: ffff80000804bd50 x28: 0000000000000002 x27: 0000000000000000 >>>> x26: ffff800009cf9a80 x25: 0000000000000000 x24: ffff800009cbf840 >>>> x23: ffff000080325000 x22: ffff0000005df800 x21: ffff80000a4ce508 >>>> x20: 0000000000000000 x19: ffff000080324440 x18: 0000000000000014 >>>> x17: 00000000388925c0 x16: 000000005386a066 x15: 000000009c10cc2e >>>> x14: 00000000000001c0 x13: 0000000000000001 x12: ffff00007fffb1a0 >>>> x11: ffff00007fffb180 x10: ffff80000a4ce508 x9 : 0000000000000041 >>>> x8 : ffff80000a4ce500 x7 : ffff80000a4cf920 x6 : 0000000000000001 >>>> x5 : 0000000000000001 x4 : 0000000000000007 x3 : 0000000000000002 >>>> x2 : 0000000000001000 x1 : ffff80000a4cf928 x0 : 0000000000000001 >>>> Call trace: >>>> build_sched_domains+0x284/0x910 >>>> sched_init_domains+0xac/0xe0 >>>> sched_init_smp+0x48/0xc8 >>>> kernel_init_freeable+0x140/0x1ac >>>> kernel_init+0x28/0x140 >>>> ret_from_fork+0x10/0x20 >>>> >>>> PATCH[1] Warn about the irregular configuration if required >>>> PATCH[2] Enable the validation for aarch64 machines >>>> PATCH[3] Enable the validation for riscv machines >>>> >>>> v3: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg01226.html >>>> v2: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg01080.html >>>> v1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00886.html >>>> >>>> Changelog >>>> ========= >>>> v4: >>>> * Pick r-b and ack-b from Daniel/Philippe (Gavin) >>>> * Replace local variable @len with possible_cpus->len in >>>> validate_cpu_cluster_to_numa_boundary() (Philippe) >>>> v3: >>>> * Validate cluster-to-NUMA instead of socket-to-NUMA >>>> boundary (Gavin) >>>> * Move the switch from MachineState to MachineClass (Philippe) >>>> * Warning instead of rejecting the irregular configuration (Daniel) >>>> * Comments to mention cluster-to-NUMA is platform instead >>>> of architectural choice (Drew) >>>> * Drop PATCH[v2 1/4] related to qtests/numa-test (Gavin) >>>> v2: >>>> * Fix socket-NUMA-node boundary issues in qtests/numa-test (Gavin) >>>> * Add helper set_numa_socket_boundary() and validate the >>>> boundary in the generic path (Philippe) >>>> >>>> Gavin Shan (3): >>>> numa: Validate cluster and NUMA node boundary if required >>>> hw/arm: Validate cluster and NUMA node boundary >>>> hw/riscv: Validate cluster and NUMA node boundary >>>> >>>> hw/arm/sbsa-ref.c | 2 ++ >>>> hw/arm/virt.c | 2 ++ >>>> hw/core/machine.c | 42 ++++++++++++++++++++++++++++++++++++++++++ >>>> hw/riscv/spike.c | 2 ++ >>>> hw/riscv/virt.c | 2 ++ >>>> include/hw/boards.h | 1 + >>>> 6 files changed, 51 insertions(+) >>>> >>> >>> Acked-by: Igor Mammedov <imammedo@redhat.com> >>> >> >> Not sure if QEMU v8.0 is still available to integrate this series. >> Otherwise, it should be something for QEMU v8.1. By the way, I'm >> also uncertain who needs to be merge this series. > > It barely touches arm specific boards, so I'm assuming it will > be reviewed and taken by whoever handles hw/core/machine.c > > And yes, 8.0 is nearly out the door, this is 8.1 stuff. > Indeed. In this case, it needs to be merged via 'Machine core' tree, which is being taken care by Eduardo Habkost or Marcel Apfelbaum. Eduardo and Marcel, could you please merge this to QEMU v8.1 when it's ready? Thanks in advance. Thanks, Gavin
On Thu, 13 Apr 2023 13:50:57 +0800 Gavin Shan <gshan@redhat.com> wrote: > On 4/12/23 7:42 PM, Peter Maydell wrote: > > On Wed, 12 Apr 2023 at 02:08, Gavin Shan <gshan@redhat.com> wrote: > >> On 3/27/23 9:26 PM, Igor Mammedov wrote: > >>> On Fri, 17 Mar 2023 14:25:39 +0800 > >>> Gavin Shan <gshan@redhat.com> wrote: > >>> > >>>> For arm64 and riscv architecture, the driver (/base/arch_topology.c) is > >>>> used to populate the CPU topology in the Linux guest. It's required that > >>>> the CPUs in one cluster can't span mutiple NUMA nodes. Otherwise, the Linux > >>>> scheduling domain can't be sorted out, as the following warning message > >>>> indicates. To avoid the unexpected confusion, this series attempts to > >>>> warn about such kind of irregular configurations. > >>>> > >>>> -smp 6,maxcpus=6,sockets=2,clusters=1,cores=3,threads=1 \ > >>>> -numa node,nodeid=0,cpus=0-1,memdev=ram0 \ > >>>> -numa node,nodeid=1,cpus=2-3,memdev=ram1 \ > >>>> -numa node,nodeid=2,cpus=4-5,memdev=ram2 \ > >>>> > >>>> ------------[ cut here ]------------ > >>>> WARNING: CPU: 0 PID: 1 at kernel/sched/topology.c:2271 build_sched_domains+0x284/0x910 > >>>> Modules linked in: > >>>> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.14.0-268.el9.aarch64 #1 > >>>> pstate: 00400005 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) > >>>> pc : build_sched_domains+0x284/0x910 > >>>> lr : build_sched_domains+0x184/0x910 > >>>> sp : ffff80000804bd50 > >>>> x29: ffff80000804bd50 x28: 0000000000000002 x27: 0000000000000000 > >>>> x26: ffff800009cf9a80 x25: 0000000000000000 x24: ffff800009cbf840 > >>>> x23: ffff000080325000 x22: ffff0000005df800 x21: ffff80000a4ce508 > >>>> x20: 0000000000000000 x19: ffff000080324440 x18: 0000000000000014 > >>>> x17: 00000000388925c0 x16: 000000005386a066 x15: 000000009c10cc2e > >>>> x14: 00000000000001c0 x13: 0000000000000001 x12: ffff00007fffb1a0 > >>>> x11: ffff00007fffb180 x10: ffff80000a4ce508 x9 : 0000000000000041 > >>>> x8 : ffff80000a4ce500 x7 : ffff80000a4cf920 x6 : 0000000000000001 > >>>> x5 : 0000000000000001 x4 : 0000000000000007 x3 : 0000000000000002 > >>>> x2 : 0000000000001000 x1 : ffff80000a4cf928 x0 : 0000000000000001 > >>>> Call trace: > >>>> build_sched_domains+0x284/0x910 > >>>> sched_init_domains+0xac/0xe0 > >>>> sched_init_smp+0x48/0xc8 > >>>> kernel_init_freeable+0x140/0x1ac > >>>> kernel_init+0x28/0x140 > >>>> ret_from_fork+0x10/0x20 > >>>> > >>>> PATCH[1] Warn about the irregular configuration if required > >>>> PATCH[2] Enable the validation for aarch64 machines > >>>> PATCH[3] Enable the validation for riscv machines > >>>> > >>>> v3: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg01226.html > >>>> v2: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg01080.html > >>>> v1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00886.html > >>>> > >>>> Changelog > >>>> ========= > >>>> v4: > >>>> * Pick r-b and ack-b from Daniel/Philippe (Gavin) > >>>> * Replace local variable @len with possible_cpus->len in > >>>> validate_cpu_cluster_to_numa_boundary() (Philippe) > >>>> v3: > >>>> * Validate cluster-to-NUMA instead of socket-to-NUMA > >>>> boundary (Gavin) > >>>> * Move the switch from MachineState to MachineClass (Philippe) > >>>> * Warning instead of rejecting the irregular configuration (Daniel) > >>>> * Comments to mention cluster-to-NUMA is platform instead > >>>> of architectural choice (Drew) > >>>> * Drop PATCH[v2 1/4] related to qtests/numa-test (Gavin) > >>>> v2: > >>>> * Fix socket-NUMA-node boundary issues in qtests/numa-test (Gavin) > >>>> * Add helper set_numa_socket_boundary() and validate the > >>>> boundary in the generic path (Philippe) > >>>> > >>>> Gavin Shan (3): > >>>> numa: Validate cluster and NUMA node boundary if required > >>>> hw/arm: Validate cluster and NUMA node boundary > >>>> hw/riscv: Validate cluster and NUMA node boundary > >>>> > >>>> hw/arm/sbsa-ref.c | 2 ++ > >>>> hw/arm/virt.c | 2 ++ > >>>> hw/core/machine.c | 42 ++++++++++++++++++++++++++++++++++++++++++ > >>>> hw/riscv/spike.c | 2 ++ > >>>> hw/riscv/virt.c | 2 ++ > >>>> include/hw/boards.h | 1 + > >>>> 6 files changed, 51 insertions(+) > >>>> > >>> > >>> Acked-by: Igor Mammedov <imammedo@redhat.com> > >>> > >> > >> Not sure if QEMU v8.0 is still available to integrate this series. > >> Otherwise, it should be something for QEMU v8.1. By the way, I'm > >> also uncertain who needs to be merge this series. > > > > It barely touches arm specific boards, so I'm assuming it will > > be reviewed and taken by whoever handles hw/core/machine.c > > > > And yes, 8.0 is nearly out the door, this is 8.1 stuff. > > > > Indeed. In this case, it needs to be merged via 'Machine core' tree, > which is being taken care by Eduardo Habkost or Marcel Apfelbaum. > > Eduardo and Marcel, could you please merge this to QEMU v8.1 when it's > ready? Thanks in advance. Lately it was Paolo who taking care of generic machine queue > > Thanks, > Gavin >
Hi Igor, On 4/13/23 7:21 PM, Igor Mammedov wrote: > On Thu, 13 Apr 2023 13:50:57 +0800 > Gavin Shan <gshan@redhat.com> wrote: > >> On 4/12/23 7:42 PM, Peter Maydell wrote: >>> On Wed, 12 Apr 2023 at 02:08, Gavin Shan <gshan@redhat.com> wrote: >>>> On 3/27/23 9:26 PM, Igor Mammedov wrote: >>>>> On Fri, 17 Mar 2023 14:25:39 +0800 >>>>> Gavin Shan <gshan@redhat.com> wrote: >>>>> >>>>>> For arm64 and riscv architecture, the driver (/base/arch_topology.c) is >>>>>> used to populate the CPU topology in the Linux guest. It's required that >>>>>> the CPUs in one cluster can't span mutiple NUMA nodes. Otherwise, the Linux >>>>>> scheduling domain can't be sorted out, as the following warning message >>>>>> indicates. To avoid the unexpected confusion, this series attempts to >>>>>> warn about such kind of irregular configurations. >>>>>> >>>>>> -smp 6,maxcpus=6,sockets=2,clusters=1,cores=3,threads=1 \ >>>>>> -numa node,nodeid=0,cpus=0-1,memdev=ram0 \ >>>>>> -numa node,nodeid=1,cpus=2-3,memdev=ram1 \ >>>>>> -numa node,nodeid=2,cpus=4-5,memdev=ram2 \ >>>>>> >>>>>> ------------[ cut here ]------------ >>>>>> WARNING: CPU: 0 PID: 1 at kernel/sched/topology.c:2271 build_sched_domains+0x284/0x910 >>>>>> Modules linked in: >>>>>> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.14.0-268.el9.aarch64 #1 >>>>>> pstate: 00400005 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) >>>>>> pc : build_sched_domains+0x284/0x910 >>>>>> lr : build_sched_domains+0x184/0x910 >>>>>> sp : ffff80000804bd50 >>>>>> x29: ffff80000804bd50 x28: 0000000000000002 x27: 0000000000000000 >>>>>> x26: ffff800009cf9a80 x25: 0000000000000000 x24: ffff800009cbf840 >>>>>> x23: ffff000080325000 x22: ffff0000005df800 x21: ffff80000a4ce508 >>>>>> x20: 0000000000000000 x19: ffff000080324440 x18: 0000000000000014 >>>>>> x17: 00000000388925c0 x16: 000000005386a066 x15: 000000009c10cc2e >>>>>> x14: 00000000000001c0 x13: 0000000000000001 x12: ffff00007fffb1a0 >>>>>> x11: ffff00007fffb180 x10: ffff80000a4ce508 x9 : 0000000000000041 >>>>>> x8 : ffff80000a4ce500 x7 : ffff80000a4cf920 x6 : 0000000000000001 >>>>>> x5 : 0000000000000001 x4 : 0000000000000007 x3 : 0000000000000002 >>>>>> x2 : 0000000000001000 x1 : ffff80000a4cf928 x0 : 0000000000000001 >>>>>> Call trace: >>>>>> build_sched_domains+0x284/0x910 >>>>>> sched_init_domains+0xac/0xe0 >>>>>> sched_init_smp+0x48/0xc8 >>>>>> kernel_init_freeable+0x140/0x1ac >>>>>> kernel_init+0x28/0x140 >>>>>> ret_from_fork+0x10/0x20 >>>>>> >>>>>> PATCH[1] Warn about the irregular configuration if required >>>>>> PATCH[2] Enable the validation for aarch64 machines >>>>>> PATCH[3] Enable the validation for riscv machines >>>>>> >>>>>> v3: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg01226.html >>>>>> v2: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg01080.html >>>>>> v1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00886.html >>>>>> >>>>>> Changelog >>>>>> ========= >>>>>> v4: >>>>>> * Pick r-b and ack-b from Daniel/Philippe (Gavin) >>>>>> * Replace local variable @len with possible_cpus->len in >>>>>> validate_cpu_cluster_to_numa_boundary() (Philippe) >>>>>> v3: >>>>>> * Validate cluster-to-NUMA instead of socket-to-NUMA >>>>>> boundary (Gavin) >>>>>> * Move the switch from MachineState to MachineClass (Philippe) >>>>>> * Warning instead of rejecting the irregular configuration (Daniel) >>>>>> * Comments to mention cluster-to-NUMA is platform instead >>>>>> of architectural choice (Drew) >>>>>> * Drop PATCH[v2 1/4] related to qtests/numa-test (Gavin) >>>>>> v2: >>>>>> * Fix socket-NUMA-node boundary issues in qtests/numa-test (Gavin) >>>>>> * Add helper set_numa_socket_boundary() and validate the >>>>>> boundary in the generic path (Philippe) >>>>>> >>>>>> Gavin Shan (3): >>>>>> numa: Validate cluster and NUMA node boundary if required >>>>>> hw/arm: Validate cluster and NUMA node boundary >>>>>> hw/riscv: Validate cluster and NUMA node boundary >>>>>> >>>>>> hw/arm/sbsa-ref.c | 2 ++ >>>>>> hw/arm/virt.c | 2 ++ >>>>>> hw/core/machine.c | 42 ++++++++++++++++++++++++++++++++++++++++++ >>>>>> hw/riscv/spike.c | 2 ++ >>>>>> hw/riscv/virt.c | 2 ++ >>>>>> include/hw/boards.h | 1 + >>>>>> 6 files changed, 51 insertions(+) >>>>>> >>>>> >>>>> Acked-by: Igor Mammedov <imammedo@redhat.com> >>>>> >>>> >>>> Not sure if QEMU v8.0 is still available to integrate this series. >>>> Otherwise, it should be something for QEMU v8.1. By the way, I'm >>>> also uncertain who needs to be merge this series. >>> >>> It barely touches arm specific boards, so I'm assuming it will >>> be reviewed and taken by whoever handles hw/core/machine.c >>> >>> And yes, 8.0 is nearly out the door, this is 8.1 stuff. >>> >> >> Indeed. In this case, it needs to be merged via 'Machine core' tree, >> which is being taken care by Eduardo Habkost or Marcel Apfelbaum. >> >> Eduardo and Marcel, could you please merge this to QEMU v8.1 when it's >> ready? Thanks in advance. > > Lately it was Paolo who taking care of generic machine queue > Thanks a lot, Igor. I will ping Paolo if needed when QEMU v8.1 is ready. Thanks, Gavin
For arm64 and riscv architecture, the driver (/base/arch_topology.c) is used to populate the CPU topology in the Linux guest. It's required that the CPUs in one cluster can't span mutiple NUMA nodes. Otherwise, the Linux scheduling domain can't be sorted out, as the following warning message indicates. To avoid the unexpected confusion, this series attempts to warn about such kind of irregular configurations. -smp 6,maxcpus=6,sockets=2,clusters=1,cores=3,threads=1 \ -numa node,nodeid=0,cpus=0-1,memdev=ram0 \ -numa node,nodeid=1,cpus=2-3,memdev=ram1 \ -numa node,nodeid=2,cpus=4-5,memdev=ram2 \ ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1 at kernel/sched/topology.c:2271 build_sched_domains+0x284/0x910 Modules linked in: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.14.0-268.el9.aarch64 #1 pstate: 00400005 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : build_sched_domains+0x284/0x910 lr : build_sched_domains+0x184/0x910 sp : ffff80000804bd50 x29: ffff80000804bd50 x28: 0000000000000002 x27: 0000000000000000 x26: ffff800009cf9a80 x25: 0000000000000000 x24: ffff800009cbf840 x23: ffff000080325000 x22: ffff0000005df800 x21: ffff80000a4ce508 x20: 0000000000000000 x19: ffff000080324440 x18: 0000000000000014 x17: 00000000388925c0 x16: 000000005386a066 x15: 000000009c10cc2e x14: 00000000000001c0 x13: 0000000000000001 x12: ffff00007fffb1a0 x11: ffff00007fffb180 x10: ffff80000a4ce508 x9 : 0000000000000041 x8 : ffff80000a4ce500 x7 : ffff80000a4cf920 x6 : 0000000000000001 x5 : 0000000000000001 x4 : 0000000000000007 x3 : 0000000000000002 x2 : 0000000000001000 x1 : ffff80000a4cf928 x0 : 0000000000000001 Call trace: build_sched_domains+0x284/0x910 sched_init_domains+0xac/0xe0 sched_init_smp+0x48/0xc8 kernel_init_freeable+0x140/0x1ac kernel_init+0x28/0x140 ret_from_fork+0x10/0x20 PATCH[1] Warn about the irregular configuration if required PATCH[2] Enable the validation for aarch64 machines PATCH[3] Enable the validation for riscv machines v3: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg01226.html v2: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg01080.html v1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00886.html Changelog ========= v4: * Pick r-b and ack-b from Daniel/Philippe (Gavin) * Replace local variable @len with possible_cpus->len in validate_cpu_cluster_to_numa_boundary() (Philippe) v3: * Validate cluster-to-NUMA instead of socket-to-NUMA boundary (Gavin) * Move the switch from MachineState to MachineClass (Philippe) * Warning instead of rejecting the irregular configuration (Daniel) * Comments to mention cluster-to-NUMA is platform instead of architectural choice (Drew) * Drop PATCH[v2 1/4] related to qtests/numa-test (Gavin) v2: * Fix socket-NUMA-node boundary issues in qtests/numa-test (Gavin) * Add helper set_numa_socket_boundary() and validate the boundary in the generic path (Philippe) Gavin Shan (3): numa: Validate cluster and NUMA node boundary if required hw/arm: Validate cluster and NUMA node boundary hw/riscv: Validate cluster and NUMA node boundary hw/arm/sbsa-ref.c | 2 ++ hw/arm/virt.c | 2 ++ hw/core/machine.c | 42 ++++++++++++++++++++++++++++++++++++++++++ hw/riscv/spike.c | 2 ++ hw/riscv/virt.c | 2 ++ include/hw/boards.h | 1 + 6 files changed, 51 insertions(+)