diff mbox

[RFC,01/11] Documentation: DT: arm: define CPU topology bindings

Message ID 1365671562-2403-2-git-send-email-mark.rutland@arm.com (mailing list archive)
State New, archived
Headers show

Commit Message

Mark Rutland April 11, 2013, 9:12 a.m. UTC
From: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>

The advent of multi-cluster ARM systems requires a mechanism to describe
how in hierarchical terms CPUs are connected in ARM SoCs so that the kernel
can initialize and map resources like IRQs and memory space to specific
group(s) of CPUs.

The CPU topology is made up of multiple hierarchy levels whose bottom
layers (aka leaf nodes in device tree syntax) contain links to the HW
CPUs in the system.

The topology bindings are generic for both 32-bit and 64-bit systems and
lay the groundwork on top of which affinity schemes can be built.

This patch provides the documentation in the kernel required to define the
device tree bindings describing the CPU topology for ARM 32-bit and 64-bit
systems.

Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
---
 Documentation/devicetree/bindings/arm/topology.txt | 524 +++++++++++++++++++++
 1 file changed, 524 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/topology.txt

Comments

Rob Herring April 11, 2013, 3 p.m. UTC | #1
On 04/11/2013 04:12 AM, Mark Rutland wrote:
> From: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>
> 
> The advent of multi-cluster ARM systems requires a mechanism to describe
> how in hierarchical terms CPUs are connected in ARM SoCs so that the kernel
> can initialize and map resources like IRQs and memory space to specific
> group(s) of CPUs.
> 
> The CPU topology is made up of multiple hierarchy levels whose bottom
> layers (aka leaf nodes in device tree syntax) contain links to the HW
> CPUs in the system.
> 
> The topology bindings are generic for both 32-bit and 64-bit systems and
> lay the groundwork on top of which affinity schemes can be built.
> 
> This patch provides the documentation in the kernel required to define the
> device tree bindings describing the CPU topology for ARM 32-bit and 64-bit
> systems.

I'm now very weary of continued /cpu changes after the pain of making
the reg property reflect the mpidr value in 3.8.

> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Signed-off-by: Mark Rutland <mark.rutland@arm.com>
> ---
>  Documentation/devicetree/bindings/arm/topology.txt | 524 +++++++++++++++++++++
>  1 file changed, 524 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/arm/topology.txt
> 
> diff --git a/Documentation/devicetree/bindings/arm/topology.txt b/Documentation/devicetree/bindings/arm/topology.txt
> new file mode 100644
> index 0000000..07c4961
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/arm/topology.txt
> @@ -0,0 +1,524 @@
> +===========================================
> +ARM topology binding description
> +===========================================
> +
> +===========================================
> +1 - Introduction
> +===========================================
> +
> +In an ARM system, the hierarchy of CPUs is defined through three entities that
> +are used to describe the layout of physical CPUs in the system:
> +
> +- cluster
> +- core
> +- thread
> +
> +The cpu nodes (bindings defined in [1]) represent the devices that
> +correspond to physical CPUs and are to be mapped to the hierarchy levels.
> +
> +The bottom hierarchy level sits at core or thread level depending on whether
> +symmetric multi-threading (SMT) is supported or not.
> +
> +For instance in a system where CPUs support SMT, "cpu" nodes represent all
> +threads existing in the system and map to the hierarchy level "thread" above.
> +In systems where SMT is not supported "cpu" nodes represent all cores present
> +in the system and map to the hierarchy level "core" above.
> +
> +ARM topology bindings allow one to associate cpu nodes with hierarchical groups
> +corresponding to the system hierarchy; syntactically they are defined as device
> +tree nodes.
> +
> +The remainder of this document provides the topology bindings for ARM, based
> +on the ePAPR standard, available from:
> +
> +http://devicetree.org
> +
> +If not stated otherwise, whenever a reference to a cpu node phandle is made its
> +value must point to a cpu node compliant with the cpu node bindings as
> +documented in [1].
> +A topology description containing phandles to cpu nodes that are not compliant
> +with bindings standardized in [1] is therefore considered invalid.
> +
> +===========================================
> +2 - cpu-map node
> +===========================================
> +
> +The ARM CPU topology is defined within a container node, sitting at the top
> +level of the device tree (/), the cpu-map node.
> +
> +- cpu-map node
> +
> +	Usage: Required to define ARM CPU topology
> +
> +	Description: The cpu-map node is just a container node where its
> +		     subnodes describe the CPU topology
> +
> +	Node name must be "cpu-map".
> +
> +	A cpu-map node's child nodes can be:
> +
> +	- one or more cluster nodes
> +
> +	Any other configuration is considered invalid.
> +
> +The cpu-map node can only contain three types of child nodes:
> +
> +- cluster node
> +- core node
> +- thread node
> +

Why not put the topology in the /cpus nodes? I don't really see the
point of having a flat list of cpus and separate topology info. There is
some compatibility issue, but adding optional levels for clusters can be
handled.

> +whose bindings are described in paragraph 3.
> +
> +The nodes describing the CPU topology (cluster/core/thread) can only be
> +defined within the cpu-map node.
> +Any other configuration is consider invalid and therefore must be ignored.
> +
> +===========================================
> +2.1 - cpu-map child nodes naming convention
> +===========================================
> +
> +cpu-map child nodes must follow a naming convention where the node name
> +must be "clusterN", "coreN", "threadN" depending on the node type (ie
> +cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
> +are siblings within a single common parent node must be given a unique and
> +sequential N value, starting from 0).
> +cpu-map child nodes which do not share a common parent node can have the same
> +name (ie same number N as other cpu-map child nodes at different device tree
> +levels) since name uniqueness will be guaranteed by the device tree hierarchy.
> +
> +===========================================
> +3 - cluster/core/thread node bindings
> +===========================================
> +
> +Bindings for cluster/cpu/thread nodes are defined as follows:
> +
> +- cluster node
> +
> +	 Description: must be declared within a cpu-map node, one node
> +		      per cluster. A system can contain several layers of
> +		      clustering and cluster nodes can be contained in parent
> +		      cluster nodes.
> +
> +	The cluster node name must be "clusterN" as described in 2.1 above.
> +	A cluster node can not be a leaf node.

Follow standard conventions with "cluster@N" and a reg property with the
number.

> +
> +	A cluster node's child nodes must be:
> +
> +	- one or more cluster nodes; or
> +	- one or more core nodes
> +
> +	Any other configuration is considered invalid.
> +
> +- core node
> +
> +	Description: must be declared in a cluster node, one node per core in
> +		     the cluster. If the system does not support SMT, core
> +		     nodes are leaf nodes, otherwise they become containers of
> +		     thread nodes.
> +
> +	The core node name must be "coreN" as described in 2.1 above.
> +
> +	A core node must be a leaf node if SMT is not supported.
> +
> +	Properties for core nodes that are leaf nodes:
> +
> +	- cpu
> +		Usage: required
> +		Value type: <phandle>
> +		Definition: a phandle to the cpu node that corresponds to the
> +			    core node.
> +
> +	If a core node is not a leaf node (CPUs supporting SMT) a core node's
> +	child nodes can be:
> +
> +	- one or more thread nodes
> +
> +	Any other configuration is considered invalid.
> +
> +- thread node
> +
> +	Description: must be declared in a core node, one node per thread
> +		     in the core if the system supports SMT. Thread nodes are
> +		     always leaf nodes in the device tree.
> +
> +	The thread node name must be "threadN" as described in 2.1 above.
> +
> +	A thread node must be a leaf node.
> +
> +	A thread node must contain the following property:
> +
> +	- cpu
> +		Usage: required
> +		Value type: <phandle>
> +		Definition: a phandle to the cpu node that corresponds to
> +			    the thread node.


According to the ePAPR, threads are represented by an array of ids for
reg property, not another cpu node. Why the deviation.

Rob
Lorenzo Pieralisi April 11, 2013, 3:50 p.m. UTC | #2
On Thu, Apr 11, 2013 at 04:00:47PM +0100, Rob Herring wrote:
> On 04/11/2013 04:12 AM, Mark Rutland wrote:
> > From: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>
> > 
> > The advent of multi-cluster ARM systems requires a mechanism to describe
> > how in hierarchical terms CPUs are connected in ARM SoCs so that the kernel
> > can initialize and map resources like IRQs and memory space to specific
> > group(s) of CPUs.
> > 
> > The CPU topology is made up of multiple hierarchy levels whose bottom
> > layers (aka leaf nodes in device tree syntax) contain links to the HW
> > CPUs in the system.
> > 
> > The topology bindings are generic for both 32-bit and 64-bit systems and
> > lay the groundwork on top of which affinity schemes can be built.
> > 
> > This patch provides the documentation in the kernel required to define the
> > device tree bindings describing the CPU topology for ARM 32-bit and 64-bit
> > systems.
> 
> I'm now very weary of continued /cpu changes after the pain of making
> the reg property reflect the mpidr value in 3.8.

We won't change the reg property value, the code I am about to post
provides stricter bindings, stricter semantics and extends bindings to
cater for arm 64-bit systems.

> > Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> > Signed-off-by: Mark Rutland <mark.rutland@arm.com>
> > ---
> >  Documentation/devicetree/bindings/arm/topology.txt | 524 +++++++++++++++++++++
> >  1 file changed, 524 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/arm/topology.txt
> > 
> > diff --git a/Documentation/devicetree/bindings/arm/topology.txt b/Documentation/devicetree/bindings/arm/topology.txt
> > new file mode 100644
> > index 0000000..07c4961
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/arm/topology.txt
> > @@ -0,0 +1,524 @@
> > +===========================================
> > +ARM topology binding description
> > +===========================================
> > +
> > +===========================================
> > +1 - Introduction
> > +===========================================
> > +
> > +In an ARM system, the hierarchy of CPUs is defined through three entities that
> > +are used to describe the layout of physical CPUs in the system:
> > +
> > +- cluster
> > +- core
> > +- thread
> > +
> > +The cpu nodes (bindings defined in [1]) represent the devices that
> > +correspond to physical CPUs and are to be mapped to the hierarchy levels.
> > +
> > +The bottom hierarchy level sits at core or thread level depending on whether
> > +symmetric multi-threading (SMT) is supported or not.
> > +
> > +For instance in a system where CPUs support SMT, "cpu" nodes represent all
> > +threads existing in the system and map to the hierarchy level "thread" above.
> > +In systems where SMT is not supported "cpu" nodes represent all cores present
> > +in the system and map to the hierarchy level "core" above.
> > +
> > +ARM topology bindings allow one to associate cpu nodes with hierarchical groups
> > +corresponding to the system hierarchy; syntactically they are defined as device
> > +tree nodes.
> > +
> > +The remainder of this document provides the topology bindings for ARM, based
> > +on the ePAPR standard, available from:
> > +
> > +http://devicetree.org
> > +
> > +If not stated otherwise, whenever a reference to a cpu node phandle is made its
> > +value must point to a cpu node compliant with the cpu node bindings as
> > +documented in [1].
> > +A topology description containing phandles to cpu nodes that are not compliant
> > +with bindings standardized in [1] is therefore considered invalid.
> > +
> > +===========================================
> > +2 - cpu-map node
> > +===========================================
> > +
> > +The ARM CPU topology is defined within a container node, sitting at the top
> > +level of the device tree (/), the cpu-map node.
> > +
> > +- cpu-map node
> > +
> > +	Usage: Required to define ARM CPU topology
> > +
> > +	Description: The cpu-map node is just a container node where its
> > +		     subnodes describe the CPU topology
> > +
> > +	Node name must be "cpu-map".
> > +
> > +	A cpu-map node's child nodes can be:
> > +
> > +	- one or more cluster nodes
> > +
> > +	Any other configuration is considered invalid.
> > +
> > +The cpu-map node can only contain three types of child nodes:
> > +
> > +- cluster node
> > +- core node
> > +- thread node
> > +
> 
> Why not put the topology in the /cpus nodes? I don't really see the
> point of having a flat list of cpus and separate topology info. There is
> some compatibility issue, but adding optional levels for clusters can be
> handled.

I thought this would break all code relying on /cpu nodes being /cpus node's
children. Furthermore, I was told that the /cpus node can only have /cpu nodes
as children.

If you wish so, we can define the topology in the /cpus node, fine by me.

> > +whose bindings are described in paragraph 3.
> > +
> > +The nodes describing the CPU topology (cluster/core/thread) can only be
> > +defined within the cpu-map node.
> > +Any other configuration is consider invalid and therefore must be ignored.
> > +
> > +===========================================
> > +2.1 - cpu-map child nodes naming convention
> > +===========================================
> > +
> > +cpu-map child nodes must follow a naming convention where the node name
> > +must be "clusterN", "coreN", "threadN" depending on the node type (ie
> > +cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
> > +are siblings within a single common parent node must be given a unique and
> > +sequential N value, starting from 0).
> > +cpu-map child nodes which do not share a common parent node can have the same
> > +name (ie same number N as other cpu-map child nodes at different device tree
> > +levels) since name uniqueness will be guaranteed by the device tree hierarchy.
> > +
> > +===========================================
> > +3 - cluster/core/thread node bindings
> > +===========================================
> > +
> > +Bindings for cluster/cpu/thread nodes are defined as follows:
> > +
> > +- cluster node
> > +
> > +	 Description: must be declared within a cpu-map node, one node
> > +		      per cluster. A system can contain several layers of
> > +		      clustering and cluster nodes can be contained in parent
> > +		      cluster nodes.
> > +
> > +	The cluster node name must be "clusterN" as described in 2.1 above.
> > +	A cluster node can not be a leaf node.
> 
> Follow standard conventions with "cluster@N" and a reg property with the
> number.

We are defining the topology to decouple the cluster/core/thread concept
from the MPIDR. Having a reg property in the cluster (and core) nodes
would complicate things if that reg property must correspond to an MPIDR
bitfield. If it is meant to be just an enumeration at a given device tree
level, I am ok with changing that.

> > +
> > +	A cluster node's child nodes must be:
> > +
> > +	- one or more cluster nodes; or
> > +	- one or more core nodes
> > +
> > +	Any other configuration is considered invalid.
> > +
> > +- core node
> > +
> > +	Description: must be declared in a cluster node, one node per core in
> > +		     the cluster. If the system does not support SMT, core
> > +		     nodes are leaf nodes, otherwise they become containers of
> > +		     thread nodes.
> > +
> > +	The core node name must be "coreN" as described in 2.1 above.
> > +
> > +	A core node must be a leaf node if SMT is not supported.
> > +
> > +	Properties for core nodes that are leaf nodes:
> > +
> > +	- cpu
> > +		Usage: required
> > +		Value type: <phandle>
> > +		Definition: a phandle to the cpu node that corresponds to the
> > +			    core node.
> > +
> > +	If a core node is not a leaf node (CPUs supporting SMT) a core node's
> > +	child nodes can be:
> > +
> > +	- one or more thread nodes
> > +
> > +	Any other configuration is considered invalid.
> > +
> > +- thread node
> > +
> > +	Description: must be declared in a core node, one node per thread
> > +		     in the core if the system supports SMT. Thread nodes are
> > +		     always leaf nodes in the device tree.
> > +
> > +	The thread node name must be "threadN" as described in 2.1 above.
> > +
> > +	A thread node must be a leaf node.
> > +
> > +	A thread node must contain the following property:
> > +
> > +	- cpu
> > +		Usage: required
> > +		Value type: <phandle>
> > +		Definition: a phandle to the cpu node that corresponds to
> > +			    the thread node.
> 
> 
> According to the ePAPR, threads are represented by an array of ids for
> reg property, not another cpu node. Why the deviation.

It is not a cpu node, it is a phandle property named cpu. Can you point
me to the ePAPR section where threads bindings are described please ? I have
not managed to find these details, I am reading version 1.0.

Thanks a lot for having a look !
Lorenzo
Rob Herring April 11, 2013, 5:55 p.m. UTC | #3
On 04/11/2013 10:50 AM, Lorenzo Pieralisi wrote:
> On Thu, Apr 11, 2013 at 04:00:47PM +0100, Rob Herring wrote:
>> On 04/11/2013 04:12 AM, Mark Rutland wrote:
>>> From: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>
>>>
>>> The advent of multi-cluster ARM systems requires a mechanism to describe
>>> how in hierarchical terms CPUs are connected in ARM SoCs so that the kernel
>>> can initialize and map resources like IRQs and memory space to specific
>>> group(s) of CPUs.
>>>
>>> The CPU topology is made up of multiple hierarchy levels whose bottom
>>> layers (aka leaf nodes in device tree syntax) contain links to the HW
>>> CPUs in the system.
>>>
>>> The topology bindings are generic for both 32-bit and 64-bit systems and
>>> lay the groundwork on top of which affinity schemes can be built.
>>>
>>> This patch provides the documentation in the kernel required to define the
>>> device tree bindings describing the CPU topology for ARM 32-bit and 64-bit
>>> systems.
>>
>> I'm now very weary of continued /cpu changes after the pain of making
>> the reg property reflect the mpidr value in 3.8.
> 
> We won't change the reg property value, the code I am about to post
> provides stricter bindings, stricter semantics and extends bindings to
> cater for arm 64-bit systems.
> 
>>> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
>>> Signed-off-by: Mark Rutland <mark.rutland@arm.com>
>>> ---
>>>  Documentation/devicetree/bindings/arm/topology.txt | 524 +++++++++++++++++++++
>>>  1 file changed, 524 insertions(+)
>>>  create mode 100644 Documentation/devicetree/bindings/arm/topology.txt
>>>
>>> diff --git a/Documentation/devicetree/bindings/arm/topology.txt b/Documentation/devicetree/bindings/arm/topology.txt
>>> new file mode 100644
>>> index 0000000..07c4961
>>> --- /dev/null
>>> +++ b/Documentation/devicetree/bindings/arm/topology.txt
>>> @@ -0,0 +1,524 @@
>>> +===========================================
>>> +ARM topology binding description
>>> +===========================================
>>> +
>>> +===========================================
>>> +1 - Introduction
>>> +===========================================
>>> +
>>> +In an ARM system, the hierarchy of CPUs is defined through three entities that
>>> +are used to describe the layout of physical CPUs in the system:
>>> +
>>> +- cluster
>>> +- core
>>> +- thread
>>> +
>>> +The cpu nodes (bindings defined in [1]) represent the devices that
>>> +correspond to physical CPUs and are to be mapped to the hierarchy levels.
>>> +
>>> +The bottom hierarchy level sits at core or thread level depending on whether
>>> +symmetric multi-threading (SMT) is supported or not.
>>> +
>>> +For instance in a system where CPUs support SMT, "cpu" nodes represent all
>>> +threads existing in the system and map to the hierarchy level "thread" above.
>>> +In systems where SMT is not supported "cpu" nodes represent all cores present
>>> +in the system and map to the hierarchy level "core" above.
>>> +
>>> +ARM topology bindings allow one to associate cpu nodes with hierarchical groups
>>> +corresponding to the system hierarchy; syntactically they are defined as device
>>> +tree nodes.
>>> +
>>> +The remainder of this document provides the topology bindings for ARM, based
>>> +on the ePAPR standard, available from:
>>> +
>>> +http://devicetree.org
>>> +
>>> +If not stated otherwise, whenever a reference to a cpu node phandle is made its
>>> +value must point to a cpu node compliant with the cpu node bindings as
>>> +documented in [1].
>>> +A topology description containing phandles to cpu nodes that are not compliant
>>> +with bindings standardized in [1] is therefore considered invalid.
>>> +
>>> +===========================================
>>> +2 - cpu-map node
>>> +===========================================
>>> +
>>> +The ARM CPU topology is defined within a container node, sitting at the top
>>> +level of the device tree (/), the cpu-map node.
>>> +
>>> +- cpu-map node
>>> +
>>> +	Usage: Required to define ARM CPU topology
>>> +
>>> +	Description: The cpu-map node is just a container node where its
>>> +		     subnodes describe the CPU topology
>>> +
>>> +	Node name must be "cpu-map".
>>> +
>>> +	A cpu-map node's child nodes can be:
>>> +
>>> +	- one or more cluster nodes
>>> +
>>> +	Any other configuration is considered invalid.
>>> +
>>> +The cpu-map node can only contain three types of child nodes:
>>> +
>>> +- cluster node
>>> +- core node
>>> +- thread node
>>> +
>>
>> Why not put the topology in the /cpus nodes? I don't really see the
>> point of having a flat list of cpus and separate topology info. There is
>> some compatibility issue, but adding optional levels for clusters can be
>> handled.
> 
> I thought this would break all code relying on /cpu nodes being /cpus node's
> children. Furthermore, I was told that the /cpus node can only have /cpu nodes
> as children.

IIRC the context, that was in regards to putting things like the PMUs
under the /cpus node. Or are you referring to something else? I think
this situation is a bit different.

You will have to support existing single cluster systems without the
hierarchy.

> 
> If you wish so, we can define the topology in the /cpus node, fine by me.
> 
>>> +whose bindings are described in paragraph 3.
>>> +
>>> +The nodes describing the CPU topology (cluster/core/thread) can only be
>>> +defined within the cpu-map node.
>>> +Any other configuration is consider invalid and therefore must be ignored.
>>> +
>>> +===========================================
>>> +2.1 - cpu-map child nodes naming convention
>>> +===========================================
>>> +
>>> +cpu-map child nodes must follow a naming convention where the node name
>>> +must be "clusterN", "coreN", "threadN" depending on the node type (ie
>>> +cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
>>> +are siblings within a single common parent node must be given a unique and
>>> +sequential N value, starting from 0).
>>> +cpu-map child nodes which do not share a common parent node can have the same
>>> +name (ie same number N as other cpu-map child nodes at different device tree
>>> +levels) since name uniqueness will be guaranteed by the device tree hierarchy.
>>> +
>>> +===========================================
>>> +3 - cluster/core/thread node bindings
>>> +===========================================
>>> +
>>> +Bindings for cluster/cpu/thread nodes are defined as follows:
>>> +
>>> +- cluster node
>>> +
>>> +	 Description: must be declared within a cpu-map node, one node
>>> +		      per cluster. A system can contain several layers of
>>> +		      clustering and cluster nodes can be contained in parent
>>> +		      cluster nodes.
>>> +
>>> +	The cluster node name must be "clusterN" as described in 2.1 above.
>>> +	A cluster node can not be a leaf node.
>>
>> Follow standard conventions with "cluster@N" and a reg property with the
>> number.
> 
> We are defining the topology to decouple the cluster/core/thread concept
> from the MPIDR. Having a reg property in the cluster (and core) nodes
> would complicate things if that reg property must correspond to an MPIDR
> bitfield. If it is meant to be just an enumeration at a given device tree
> level, I am ok with changing that.

Because the cluster itself doesn't really have an id, I'm fine if its
not linked to the mpidr. Just don't change that later.

>>> +
>>> +	A cluster node's child nodes must be:
>>> +
>>> +	- one or more cluster nodes; or
>>> +	- one or more core nodes
>>> +
>>> +	Any other configuration is considered invalid.
>>> +
>>> +- core node
>>> +
>>> +	Description: must be declared in a cluster node, one node per core in
>>> +		     the cluster. If the system does not support SMT, core
>>> +		     nodes are leaf nodes, otherwise they become containers of
>>> +		     thread nodes.
>>> +
>>> +	The core node name must be "coreN" as described in 2.1 above.
>>> +
>>> +	A core node must be a leaf node if SMT is not supported.
>>> +
>>> +	Properties for core nodes that are leaf nodes:
>>> +
>>> +	- cpu
>>> +		Usage: required
>>> +		Value type: <phandle>
>>> +		Definition: a phandle to the cpu node that corresponds to the
>>> +			    core node.
>>> +
>>> +	If a core node is not a leaf node (CPUs supporting SMT) a core node's
>>> +	child nodes can be:
>>> +
>>> +	- one or more thread nodes
>>> +
>>> +	Any other configuration is considered invalid.
>>> +
>>> +- thread node
>>> +
>>> +	Description: must be declared in a core node, one node per thread
>>> +		     in the core if the system supports SMT. Thread nodes are
>>> +		     always leaf nodes in the device tree.
>>> +
>>> +	The thread node name must be "threadN" as described in 2.1 above.
>>> +
>>> +	A thread node must be a leaf node.
>>> +
>>> +	A thread node must contain the following property:
>>> +
>>> +	- cpu
>>> +		Usage: required
>>> +		Value type: <phandle>
>>> +		Definition: a phandle to the cpu node that corresponds to
>>> +			    the thread node.
>>
>>
>> According to the ePAPR, threads are represented by an array of ids for
>> reg property, not another cpu node. Why the deviation.
> 
> It is not a cpu node, it is a phandle property named cpu. Can you point
> me to the ePAPR section where threads bindings are described please ? I have
> not managed to find these details, I am reading version 1.0.
> 
You should get 1.1.

From the reg prop description:

If a CPU supports more than one thread (i.e. multiple streams of
execution) the reg property is an array with 1 element per thread. The
#address-cells on the /cpus node specifies how many cells each element
of the array takes. Software can determine the number of threads by
dividing the size of reg by the parent node's #address-cells.
If a CPU/thread can be the target of an external interrupt the "reg"
property value must be a unique CPU/thread id that is addressable by
the interrupt controller.
If a CPU/thread cannot be the target of an external interrupt, then "reg"
must be unique and out of bounds of the range addressed by the
interrupt controller
If a CPU/thread's PIR is modifiable, a client program should modify
PIR to match the "reg" property value. If PIR cannot be modified and
the PIR value is distinct from the interrupt controller numberspace, the
CPUs binding may define a binding-specific representation of PIR
values if desired.


There is also this statement:

Hardware threads that share an MMU would generally be represented under
one cpu node. If other
more complex CPU topographies are designed, the binding for the CPU must
describe the topography
(e.g. threads that don't share an MMU).

Rob
tip-bot for Dave Martin April 11, 2013, 6:01 p.m. UTC | #4
On Thu, Apr 11, 2013 at 04:50:54PM +0100, Lorenzo Pieralisi wrote:
> On Thu, Apr 11, 2013 at 04:00:47PM +0100, Rob Herring wrote:
> > On 04/11/2013 04:12 AM, Mark Rutland wrote:
> > > From: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>
> > > 
> > > The advent of multi-cluster ARM systems requires a mechanism to describe
> > > how in hierarchical terms CPUs are connected in ARM SoCs so that the kernel
> > > can initialize and map resources like IRQs and memory space to specific
> > > group(s) of CPUs.
> > > 
> > > The CPU topology is made up of multiple hierarchy levels whose bottom
> > > layers (aka leaf nodes in device tree syntax) contain links to the HW
> > > CPUs in the system.
> > > 
> > > The topology bindings are generic for both 32-bit and 64-bit systems and
> > > lay the groundwork on top of which affinity schemes can be built.
> > > 
> > > This patch provides the documentation in the kernel required to define the
> > > device tree bindings describing the CPU topology for ARM 32-bit and 64-bit
> > > systems.
> > 
> > I'm now very weary of continued /cpu changes after the pain of making
> > the reg property reflect the mpidr value in 3.8.
> 
> We won't change the reg property value, the code I am about to post
> provides stricter bindings, stricter semantics and extends bindings to
> cater for arm 64-bit systems.
> 
> > > Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> > > Signed-off-by: Mark Rutland <mark.rutland@arm.com>
> > > ---
> > >  Documentation/devicetree/bindings/arm/topology.txt | 524 +++++++++++++++++++++
> > >  1 file changed, 524 insertions(+)
> > >  create mode 100644 Documentation/devicetree/bindings/arm/topology.txt
> > > 
> > > diff --git a/Documentation/devicetree/bindings/arm/topology.txt b/Documentation/devicetree/bindings/arm/topology.txt
> > > new file mode 100644
> > > index 0000000..07c4961
> > > --- /dev/null
> > > +++ b/Documentation/devicetree/bindings/arm/topology.txt
> > > @@ -0,0 +1,524 @@
> > > +===========================================
> > > +ARM topology binding description
> > > +===========================================
> > > +
> > > +===========================================
> > > +1 - Introduction
> > > +===========================================
> > > +
> > > +In an ARM system, the hierarchy of CPUs is defined through three entities that
> > > +are used to describe the layout of physical CPUs in the system:
> > > +
> > > +- cluster
> > > +- core
> > > +- thread
> > > +
> > > +The cpu nodes (bindings defined in [1]) represent the devices that
> > > +correspond to physical CPUs and are to be mapped to the hierarchy levels.
> > > +
> > > +The bottom hierarchy level sits at core or thread level depending on whether
> > > +symmetric multi-threading (SMT) is supported or not.
> > > +
> > > +For instance in a system where CPUs support SMT, "cpu" nodes represent all
> > > +threads existing in the system and map to the hierarchy level "thread" above.
> > > +In systems where SMT is not supported "cpu" nodes represent all cores present
> > > +in the system and map to the hierarchy level "core" above.
> > > +
> > > +ARM topology bindings allow one to associate cpu nodes with hierarchical groups
> > > +corresponding to the system hierarchy; syntactically they are defined as device
> > > +tree nodes.
> > > +
> > > +The remainder of this document provides the topology bindings for ARM, based
> > > +on the ePAPR standard, available from:
> > > +
> > > +http://devicetree.org
> > > +
> > > +If not stated otherwise, whenever a reference to a cpu node phandle is made its
> > > +value must point to a cpu node compliant with the cpu node bindings as
> > > +documented in [1].
> > > +A topology description containing phandles to cpu nodes that are not compliant
> > > +with bindings standardized in [1] is therefore considered invalid.
> > > +
> > > +===========================================
> > > +2 - cpu-map node
> > > +===========================================
> > > +
> > > +The ARM CPU topology is defined within a container node, sitting at the top
> > > +level of the device tree (/), the cpu-map node.
> > > +
> > > +- cpu-map node
> > > +
> > > +	Usage: Required to define ARM CPU topology
> > > +
> > > +	Description: The cpu-map node is just a container node where its
> > > +		     subnodes describe the CPU topology
> > > +
> > > +	Node name must be "cpu-map".
> > > +
> > > +	A cpu-map node's child nodes can be:
> > > +
> > > +	- one or more cluster nodes
> > > +
> > > +	Any other configuration is considered invalid.
> > > +
> > > +The cpu-map node can only contain three types of child nodes:
> > > +
> > > +- cluster node
> > > +- core node
> > > +- thread node
> > > +
> > 
> > Why not put the topology in the /cpus nodes? I don't really see the
> > point of having a flat list of cpus and separate topology info. There is
> > some compatibility issue, but adding optional levels for clusters can be
> > handled.
> 
> I thought this would break all code relying on /cpu nodes being /cpus node's
> children. Furthermore, I was told that the /cpus node can only have /cpu nodes
> as children.
> 
> If you wish so, we can define the topology in the /cpus node, fine by me.

Can we make such extensive changes to the cpus node without violating
the ePAPR specification?

If we can, great, but I'm a but unclear on how this would be achieved.

> > > +whose bindings are described in paragraph 3.
> > > +
> > > +The nodes describing the CPU topology (cluster/core/thread) can only be
> > > +defined within the cpu-map node.
> > > +Any other configuration is consider invalid and therefore must be ignored.
> > > +
> > > +===========================================
> > > +2.1 - cpu-map child nodes naming convention
> > > +===========================================
> > > +
> > > +cpu-map child nodes must follow a naming convention where the node name
> > > +must be "clusterN", "coreN", "threadN" depending on the node type (ie
> > > +cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
> > > +are siblings within a single common parent node must be given a unique and
> > > +sequential N value, starting from 0).
> > > +cpu-map child nodes which do not share a common parent node can have the same
> > > +name (ie same number N as other cpu-map child nodes at different device tree
> > > +levels) since name uniqueness will be guaranteed by the device tree hierarchy.
> > > +
> > > +===========================================
> > > +3 - cluster/core/thread node bindings
> > > +===========================================
> > > +
> > > +Bindings for cluster/cpu/thread nodes are defined as follows:
> > > +
> > > +- cluster node
> > > +
> > > +	 Description: must be declared within a cpu-map node, one node
> > > +		      per cluster. A system can contain several layers of
> > > +		      clustering and cluster nodes can be contained in parent
> > > +		      cluster nodes.
> > > +
> > > +	The cluster node name must be "clusterN" as described in 2.1 above.
> > > +	A cluster node can not be a leaf node.
> > 
> > Follow standard conventions with "cluster@N" and a reg property with the
> > number.
> 
> We are defining the topology to decouple the cluster/core/thread concept
> from the MPIDR. Having a reg property in the cluster (and core) nodes
> would complicate things if that reg property must correspond to an MPIDR
> bitfield. If it is meant to be just an enumeration at a given device tree
> level, I am ok with changing that.

As a pure enumeration, I think that works fine.  It's more verbose
but also more conformant to DT conventions.  I'm not sure there's
another difference.

The proposed support for C preprocessing of dts files might provide a
way to help debloat this to some extent in dts source, while still
following the DT convention of using unit addresses and reg properties.
This will significantly increase the size of the FDT blob if the 
number of CPUs is large.  I don't remember offhand if we have a limit
on the size of FDT we can cope with.  Finding ways to relax the limit
is a better solution than dodging round standards, though.  We can
cross that bridge when/if we come to it.

> 
> > > +
> > > +	A cluster node's child nodes must be:
> > > +
> > > +	- one or more cluster nodes; or
> > > +	- one or more core nodes
> > > +
> > > +	Any other configuration is considered invalid.
> > > +
> > > +- core node
> > > +
> > > +	Description: must be declared in a cluster node, one node per core in
> > > +		     the cluster. If the system does not support SMT, core
> > > +		     nodes are leaf nodes, otherwise they become containers of
> > > +		     thread nodes.
> > > +
> > > +	The core node name must be "coreN" as described in 2.1 above.
> > > +
> > > +	A core node must be a leaf node if SMT is not supported.
> > > +
> > > +	Properties for core nodes that are leaf nodes:
> > > +
> > > +	- cpu
> > > +		Usage: required
> > > +		Value type: <phandle>
> > > +		Definition: a phandle to the cpu node that corresponds to the
> > > +			    core node.
> > > +
> > > +	If a core node is not a leaf node (CPUs supporting SMT) a core node's
> > > +	child nodes can be:
> > > +
> > > +	- one or more thread nodes
> > > +
> > > +	Any other configuration is considered invalid.
> > > +
> > > +- thread node
> > > +
> > > +	Description: must be declared in a core node, one node per thread
> > > +		     in the core if the system supports SMT. Thread nodes are
> > > +		     always leaf nodes in the device tree.
> > > +
> > > +	The thread node name must be "threadN" as described in 2.1 above.
> > > +
> > > +	A thread node must be a leaf node.
> > > +
> > > +	A thread node must contain the following property:
> > > +
> > > +	- cpu
> > > +		Usage: required
> > > +		Value type: <phandle>
> > > +		Definition: a phandle to the cpu node that corresponds to
> > > +			    the thread node.
> > 
> > 
> > According to the ePAPR, threads are represented by an array of ids for
> > reg property, not another cpu node. Why the deviation.
> 
> It is not a cpu node, it is a phandle property named cpu. Can you point
> me to the ePAPR section where threads bindings are described please ? I have
> not managed to find these details, I am reading version 1.0.

For cpu/reg:

[1]     If a CPU supports more than one thread (i.e. multiple streams of
        execution) the reg property is an array with 1 element per
        thread. The #address-cells on the /cpus node specifies how many
        cells each element of the array takes. Software can determine
        the number of threads by dividing the size of reg by the parent
        node's #address-cells.

I had not previously been aware of this, but I see no reason not to
follow this convention.


Also:
[2]     If other more complex CPU topographies are designed, the binding
        for the CPU must describe the topography


That's rather less helpful, but the suggestion is clear enough in that
such information should be in the cpu node and specific to that CPU's
binding.  For ARM, we can have some global extensions to the CPU node.

The problems start when you want to refer to clusters and groups of
CPUs from other nodes.  Only individual cpu nodes can be places in
the cpus node, so there is no node for a phandle to point at.

If you want to describe how other things like power, clock and
coherency domains map to clusters and larger entities, things could
get pretty awkward.

Keeping the topology description separate allows all topological entities
to appear as addressable entities in the DT; otherwise, a cryptic
convention is needed.


Hybrid approaches might be possible, putting cpu nodes into /cpus, and
giving them a "parent" property where appropriate pointing at the
relevant cluster node, which we put elsewhere in the DT.

I'm not sure whether any of these approaches is an awful lot less ugly
or more easy to handle than what it currently proposed though.

The global binding for all ARM CPUs would specify that the topology
is described by /cpu-map and its associated binding.  For my
interpretation of [2], this is a compliant approach.  ePAPR does not
specify _how_ the cpu node binding achieves a description of the
topography, just that it must achieve it.  There's no statement to
say that it must not involve other nodes or bindings.

Cheers
---Dave
tip-bot for Dave Martin April 11, 2013, 6:17 p.m. UTC | #5
On Thu, Apr 11, 2013 at 12:55:20PM -0500, Rob Herring wrote:
> On 04/11/2013 10:50 AM, Lorenzo Pieralisi wrote:
> > On Thu, Apr 11, 2013 at 04:00:47PM +0100, Rob Herring wrote:
> >> On 04/11/2013 04:12 AM, Mark Rutland wrote:
> >>> From: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>

[...]

> >>> +===========================================
> >>> +3 - cluster/core/thread node bindings
> >>> +===========================================
> >>> +
> >>> +Bindings for cluster/cpu/thread nodes are defined as follows:
> >>> +
> >>> +- cluster node
> >>> +
> >>> +	 Description: must be declared within a cpu-map node, one node
> >>> +		      per cluster. A system can contain several layers of
> >>> +		      clustering and cluster nodes can be contained in parent
> >>> +		      cluster nodes.
> >>> +
> >>> +	The cluster node name must be "clusterN" as described in 2.1 above.
> >>> +	A cluster node can not be a leaf node.
> >>
> >> Follow standard conventions with "cluster@N" and a reg property with the
> >> number.
> > 
> > We are defining the topology to decouple the cluster/core/thread concept
> > from the MPIDR. Having a reg property in the cluster (and core) nodes
> > would complicate things if that reg property must correspond to an MPIDR
> > bitfield. If it is meant to be just an enumeration at a given device tree
> > level, I am ok with changing that.
> 
> Because the cluster itself doesn't really have an id, I'm fine if its
> not linked to the mpidr. Just don't change that later.

These enumerations can follow the MPIDR for convenience if the MPIDR
allocations are sane, but otherwise it would be abstract.  The Architecture
doesn't specify any kind of an ID for a cluster (indeed, it doesn't really
specify what a cluster is at all).


However, we could require a platform or SoC binding to specify the
enumeration and it how it maps to the actual hardware -- i.e., cluster@1
on Tuesday should be the same physical cluster as cluster@1 on Monday,
to the extent that cluster@1 is present in the DT on both days and the
hardware hasn't been physically modified (i.e., cluster@1 might refer
to a fixed physical socket on a hypothetical board with physically
pluggable CPU modules).

Correspondingly, on TC2, cluster@0 would always be the A15 cluster;
cluster@1 would always be the A7 cluster.  But if the firmware disables
one of them, that cluster and the relevant CPU nodes should be removed
from the DT before passing it to the kernel.

Does this make any sense, or is it overkill?


Cheers
---Dave
Lorenzo Pieralisi April 12, 2013, 11:16 a.m. UTC | #6
On Thu, Apr 11, 2013 at 06:55:20PM +0100, Rob Herring wrote:

[...]

> >>> +The cpu-map node can only contain three types of child nodes:
> >>> +
> >>> +- cluster node
> >>> +- core node
> >>> +- thread node
> >>> +
> >>
> >> Why not put the topology in the /cpus nodes? I don't really see the
> >> point of having a flat list of cpus and separate topology info. There is
> >> some compatibility issue, but adding optional levels for clusters can be
> >> handled.
> >
> > I thought this would break all code relying on /cpu nodes being /cpus node's
> > children. Furthermore, I was told that the /cpus node can only have /cpu nodes
> > as children.
> 
> IIRC the context, that was in regards to putting things like the PMUs
> under the /cpus node. Or are you referring to something else? I think
> this situation is a bit different.

No I am referring to:

(a) PowerPC code relying on cpu nodes to be cpus node's children
(b) code that parses the DT cpus/cpu node for ARM that does require cpu
    nodes to be cpus node's children (but that has nothing to do with
    the bindings, code can be changed)

I think that our definition of cpu nodes does not comply with ePAPR 3.7,
since an MPIDR corresponding to an SMT thread cannot be represented with
one CPU node "A cpu node represents a hardware execution block that is
sufficiently independent that it is capable of running an operating
system without interfering with other CPUs possibly running other
operating systems".

This also has consequences on the cpu node bindings I am about to post
since if we go by the definition above a cpu node representing an SMT
core should contain a list of MPIDRs.

> You will have to support existing single cluster systems without the
> hierarchy.

Well yes, that's why we used another (optional) node to define the topology.

> > If you wish so, we can define the topology in the /cpus node, fine by me.
> >
> >>> +whose bindings are described in paragraph 3.
> >>> +
> >>> +The nodes describing the CPU topology (cluster/core/thread) can only be
> >>> +defined within the cpu-map node.
> >>> +Any other configuration is consider invalid and therefore must be ignored.
> >>> +
> >>> +===========================================
> >>> +2.1 - cpu-map child nodes naming convention
> >>> +===========================================
> >>> +
> >>> +cpu-map child nodes must follow a naming convention where the node name
> >>> +must be "clusterN", "coreN", "threadN" depending on the node type (ie
> >>> +cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
> >>> +are siblings within a single common parent node must be given a unique and
> >>> +sequential N value, starting from 0).
> >>> +cpu-map child nodes which do not share a common parent node can have the same
> >>> +name (ie same number N as other cpu-map child nodes at different device tree
> >>> +levels) since name uniqueness will be guaranteed by the device tree hierarchy.
> >>> +
> >>> +===========================================
> >>> +3 - cluster/core/thread node bindings
> >>> +===========================================
> >>> +
> >>> +Bindings for cluster/cpu/thread nodes are defined as follows:
> >>> +
> >>> +- cluster node
> >>> +
> >>> +    Description: must be declared within a cpu-map node, one node
> >>> +                 per cluster. A system can contain several layers of
> >>> +                 clustering and cluster nodes can be contained in parent
> >>> +                 cluster nodes.
> >>> +
> >>> +   The cluster node name must be "clusterN" as described in 2.1 above.
> >>> +   A cluster node can not be a leaf node.
> >>
> >> Follow standard conventions with "cluster@N" and a reg property with the
> >> number.
> >
> > We are defining the topology to decouple the cluster/core/thread concept
> > from the MPIDR. Having a reg property in the cluster (and core) nodes
> > would complicate things if that reg property must correspond to an MPIDR
> > bitfield. If it is meant to be just an enumeration at a given device tree
> > level, I am ok with changing that.
> 
> Because the cluster itself doesn't really have an id, I'm fine if its
> not linked to the mpidr. Just don't change that later.

Eh, the cluster does have an id, that depends on the MPIDR du jour.
I will give it more thought.

> >>> +
> >>> +   A cluster node's child nodes must be:
> >>> +
> >>> +   - one or more cluster nodes; or
> >>> +   - one or more core nodes
> >>> +
> >>> +   Any other configuration is considered invalid.
> >>> +
> >>> +- core node
> >>> +
> >>> +   Description: must be declared in a cluster node, one node per core in
> >>> +                the cluster. If the system does not support SMT, core
> >>> +                nodes are leaf nodes, otherwise they become containers of
> >>> +                thread nodes.
> >>> +
> >>> +   The core node name must be "coreN" as described in 2.1 above.
> >>> +
> >>> +   A core node must be a leaf node if SMT is not supported.
> >>> +
> >>> +   Properties for core nodes that are leaf nodes:
> >>> +
> >>> +   - cpu
> >>> +           Usage: required
> >>> +           Value type: <phandle>
> >>> +           Definition: a phandle to the cpu node that corresponds to the
> >>> +                       core node.
> >>> +
> >>> +   If a core node is not a leaf node (CPUs supporting SMT) a core node's
> >>> +   child nodes can be:
> >>> +
> >>> +   - one or more thread nodes
> >>> +
> >>> +   Any other configuration is considered invalid.
> >>> +
> >>> +- thread node
> >>> +
> >>> +   Description: must be declared in a core node, one node per thread
> >>> +                in the core if the system supports SMT. Thread nodes are
> >>> +                always leaf nodes in the device tree.
> >>> +
> >>> +   The thread node name must be "threadN" as described in 2.1 above.
> >>> +
> >>> +   A thread node must be a leaf node.
> >>> +
> >>> +   A thread node must contain the following property:
> >>> +
> >>> +   - cpu
> >>> +           Usage: required
> >>> +           Value type: <phandle>
> >>> +           Definition: a phandle to the cpu node that corresponds to
> >>> +                       the thread node.
> >>
> >>
> >> According to the ePAPR, threads are represented by an array of ids for
> >> reg property, not another cpu node. Why the deviation.
> >
> > It is not a cpu node, it is a phandle property named cpu. Can you point
> > me to the ePAPR section where threads bindings are described please ? I have
> > not managed to find these details, I am reading version 1.0.
> >
> You should get 1.1.
> 
> From the reg prop description:
> 
> If a CPU supports more than one thread (i.e. multiple streams of
> execution) the reg property is an array with 1 element per thread. The
> #address-cells on the /cpus node specifies how many cells each element
> of the array takes. Software can determine the number of threads by
> dividing the size of reg by the parent node's #address-cells.

See above, I need to investigate what this implies.

> If a CPU/thread can be the target of an external interrupt the "reg"
> property value must be a unique CPU/thread id that is addressable by
> the interrupt controller.

I am not sure what "external" means here, but basically it looks like
our DT cpu bindings are non-compliant, the interrupt controller cannot
address a cpu by using an MPIDR (yet).

> If a CPU/thread cannot be the target of an external interrupt, then "reg"
> must be unique and out of bounds of the range addressed by the
> interrupt controller
> If a CPU/thread's PIR is modifiable, a client program should modify
> PIR to match the "reg" property value. If PIR cannot be modified and
> the PIR value is distinct from the interrupt controller numberspace, the
> CPUs binding may define a binding-specific representation of PIR
> values if desired.
> 
> 
> There is also this statement:
> 
> Hardware threads that share an MMU would generally be represented under
> one cpu node. If other
> more complex CPU topographies are designed, the binding for the CPU must
> describe the topography
> (e.g. threads that don't share an MMU).

Again, see above, HW threads do share an MMU so we have to make a
decision here, either we make ARM comply with ePAPR or change the
bindings for ARM.

Lorenzo
Lorenzo Pieralisi April 12, 2013, 11:27 a.m. UTC | #7
On Thu, Apr 11, 2013 at 07:17:10PM +0100, Dave Martin wrote:
> On Thu, Apr 11, 2013 at 12:55:20PM -0500, Rob Herring wrote:
> > On 04/11/2013 10:50 AM, Lorenzo Pieralisi wrote:
> > > On Thu, Apr 11, 2013 at 04:00:47PM +0100, Rob Herring wrote:
> > >> On 04/11/2013 04:12 AM, Mark Rutland wrote:
> > >>> From: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>
> 
> [...]
> 
> > >>> +===========================================
> > >>> +3 - cluster/core/thread node bindings
> > >>> +===========================================
> > >>> +
> > >>> +Bindings for cluster/cpu/thread nodes are defined as follows:
> > >>> +
> > >>> +- cluster node
> > >>> +
> > >>> +	 Description: must be declared within a cpu-map node, one node
> > >>> +		      per cluster. A system can contain several layers of
> > >>> +		      clustering and cluster nodes can be contained in parent
> > >>> +		      cluster nodes.
> > >>> +
> > >>> +	The cluster node name must be "clusterN" as described in 2.1 above.
> > >>> +	A cluster node can not be a leaf node.
> > >>
> > >> Follow standard conventions with "cluster@N" and a reg property with the
> > >> number.
> > > 
> > > We are defining the topology to decouple the cluster/core/thread concept
> > > from the MPIDR. Having a reg property in the cluster (and core) nodes
> > > would complicate things if that reg property must correspond to an MPIDR
> > > bitfield. If it is meant to be just an enumeration at a given device tree
> > > level, I am ok with changing that.
> > 
> > Because the cluster itself doesn't really have an id, I'm fine if its
> > not linked to the mpidr. Just don't change that later.
> 
> These enumerations can follow the MPIDR for convenience if the MPIDR
> allocations are sane, but otherwise it would be abstract.  The Architecture
> doesn't specify any kind of an ID for a cluster (indeed, it doesn't really
> specify what a cluster is at all).
> 
> 
> However, we could require a platform or SoC binding to specify the
> enumeration and it how it maps to the actual hardware -- i.e., cluster@1
> on Tuesday should be the same physical cluster as cluster@1 on Monday,
> to the extent that cluster@1 is present in the DT on both days and the
> hardware hasn't been physically modified (i.e., cluster@1 might refer
> to a fixed physical socket on a hypothetical board with physically
> pluggable CPU modules).
> 
> Correspondingly, on TC2, cluster@0 would always be the A15 cluster;
> cluster@1 would always be the A7 cluster.  But if the firmware disables
> one of them, that cluster and the relevant CPU nodes should be removed
> from the DT before passing it to the kernel.
> 
> Does this make any sense, or is it overkill?

I think we can say that the reg property must correspond to a cluster
id, irrespective of the MPIDR. Whatever it is, as you said, a platform
has to have a cluster numbering scheme. How it is mapped to the MPIDR
that's platform specific. So I am happy to add a reg property but
linking it to the MPIDR goes against what we are doing here hence I
would not do it, simply because we can't say "cluster reg property must
be MPIDR[15:8]", that's wrong and there is precious little we can do
about that.

Lorenzo
Lorenzo Pieralisi April 12, 2013, 11:44 a.m. UTC | #8
On Thu, Apr 11, 2013 at 07:01:25PM +0100, Dave Martin wrote:

[...]

> > > > +===========================================
> > > > +2 - cpu-map node
> > > > +===========================================
> > > > +
> > > > +The ARM CPU topology is defined within a container node, sitting at the top
> > > > +level of the device tree (/), the cpu-map node.
> > > > +
> > > > +- cpu-map node
> > > > +
> > > > + Usage: Required to define ARM CPU topology
> > > > +
> > > > + Description: The cpu-map node is just a container node where its
> > > > +              subnodes describe the CPU topology
> > > > +
> > > > + Node name must be "cpu-map".
> > > > +
> > > > + A cpu-map node's child nodes can be:
> > > > +
> > > > + - one or more cluster nodes
> > > > +
> > > > + Any other configuration is considered invalid.
> > > > +
> > > > +The cpu-map node can only contain three types of child nodes:
> > > > +
> > > > +- cluster node
> > > > +- core node
> > > > +- thread node
> > > > +
> > >
> > > Why not put the topology in the /cpus nodes? I don't really see the
> > > point of having a flat list of cpus and separate topology info. There is
> > > some compatibility issue, but adding optional levels for clusters can be
> > > handled.
> >
> > I thought this would break all code relying on /cpu nodes being /cpus node's
> > children. Furthermore, I was told that the /cpus node can only have /cpu nodes
> > as children.
> >
> > If you wish so, we can define the topology in the /cpus node, fine by me.
> 
> Can we make such extensive changes to the cpus node without violating
> the ePAPR specification?
> 
> If we can, great, but I'm a but unclear on how this would be achieved.

+1

> > > > +whose bindings are described in paragraph 3.
> > > > +
> > > > +The nodes describing the CPU topology (cluster/core/thread) can only be
> > > > +defined within the cpu-map node.
> > > > +Any other configuration is consider invalid and therefore must be ignored.
> > > > +
> > > > +===========================================
> > > > +2.1 - cpu-map child nodes naming convention
> > > > +===========================================
> > > > +
> > > > +cpu-map child nodes must follow a naming convention where the node name
> > > > +must be "clusterN", "coreN", "threadN" depending on the node type (ie
> > > > +cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
> > > > +are siblings within a single common parent node must be given a unique and
> > > > +sequential N value, starting from 0).
> > > > +cpu-map child nodes which do not share a common parent node can have the same
> > > > +name (ie same number N as other cpu-map child nodes at different device tree
> > > > +levels) since name uniqueness will be guaranteed by the device tree hierarchy.
> > > > +
> > > > +===========================================
> > > > +3 - cluster/core/thread node bindings
> > > > +===========================================
> > > > +
> > > > +Bindings for cluster/cpu/thread nodes are defined as follows:
> > > > +
> > > > +- cluster node
> > > > +
> > > > +  Description: must be declared within a cpu-map node, one node
> > > > +               per cluster. A system can contain several layers of
> > > > +               clustering and cluster nodes can be contained in parent
> > > > +               cluster nodes.
> > > > +
> > > > + The cluster node name must be "clusterN" as described in 2.1 above.
> > > > + A cluster node can not be a leaf node.
> > >
> > > Follow standard conventions with "cluster@N" and a reg property with the
> > > number.
> >
> > We are defining the topology to decouple the cluster/core/thread concept
> > from the MPIDR. Having a reg property in the cluster (and core) nodes
> > would complicate things if that reg property must correspond to an MPIDR
> > bitfield. If it is meant to be just an enumeration at a given device tree
> > level, I am ok with changing that.
> 
> As a pure enumeration, I think that works fine.  It's more verbose
> but also more conformant to DT conventions.  I'm not sure there's
> another difference.
> 
> The proposed support for C preprocessing of dts files might provide a
> way to help debloat this to some extent in dts source, while still
> following the DT convention of using unit addresses and reg properties.
> This will significantly increase the size of the FDT blob if the
> number of CPUs is large.  I don't remember offhand if we have a limit
> on the size of FDT we can cope with.  Finding ways to relax the limit
> is a better solution than dodging round standards, though.  We can
> cross that bridge when/if we come to it.

Well, that's a problem by itself, certainly adding a reg property to
the cluster nodes will make it worse, but as you said still better that
than dodging standards.

> >
> > > > +
> > > > + A cluster node's child nodes must be:
> > > > +
> > > > + - one or more cluster nodes; or
> > > > + - one or more core nodes
> > > > +
> > > > + Any other configuration is considered invalid.
> > > > +
> > > > +- core node
> > > > +
> > > > + Description: must be declared in a cluster node, one node per core in
> > > > +              the cluster. If the system does not support SMT, core
> > > > +              nodes are leaf nodes, otherwise they become containers of
> > > > +              thread nodes.
> > > > +
> > > > + The core node name must be "coreN" as described in 2.1 above.
> > > > +
> > > > + A core node must be a leaf node if SMT is not supported.
> > > > +
> > > > + Properties for core nodes that are leaf nodes:
> > > > +
> > > > + - cpu
> > > > +         Usage: required
> > > > +         Value type: <phandle>
> > > > +         Definition: a phandle to the cpu node that corresponds to the
> > > > +                     core node.
> > > > +
> > > > + If a core node is not a leaf node (CPUs supporting SMT) a core node's
> > > > + child nodes can be:
> > > > +
> > > > + - one or more thread nodes
> > > > +
> > > > + Any other configuration is considered invalid.
> > > > +
> > > > +- thread node
> > > > +
> > > > + Description: must be declared in a core node, one node per thread
> > > > +              in the core if the system supports SMT. Thread nodes are
> > > > +              always leaf nodes in the device tree.
> > > > +
> > > > + The thread node name must be "threadN" as described in 2.1 above.
> > > > +
> > > > + A thread node must be a leaf node.
> > > > +
> > > > + A thread node must contain the following property:
> > > > +
> > > > + - cpu
> > > > +         Usage: required
> > > > +         Value type: <phandle>
> > > > +         Definition: a phandle to the cpu node that corresponds to
> > > > +                     the thread node.
> > >
> > >
> > > According to the ePAPR, threads are represented by an array of ids for
> > > reg property, not another cpu node. Why the deviation.
> >
> > It is not a cpu node, it is a phandle property named cpu. Can you point
> > me to the ePAPR section where threads bindings are described please ? I have
> > not managed to find these details, I am reading version 1.0.
> 
> For cpu/reg:
> 
> [1]     If a CPU supports more than one thread (i.e. multiple streams of
>         execution) the reg property is an array with 1 element per
>         thread. The #address-cells on the /cpus node specifies how many
>         cells each element of the array takes. Software can determine
>         the number of threads by dividing the size of reg by the parent
>         node's #address-cells.
> 
> I had not previously been aware of this, but I see no reason not to
> follow this convention.

I don't see a reason either, but this changes the current cpu node bindings
for ARM. On the upside there are no SMT ARM platforms out there, so no
backward compatibility to worry about.

This would reduce the topology problem to where cluster nodes should be
defined, either in the cpus node or a separate node (ie cpu-map :-)).

> Also:
> [2]     If other more complex CPU topographies are designed, the binding
>         for the CPU must describe the topography
> 
> 
> That's rather less helpful, but the suggestion is clear enough in that
> such information should be in the cpu node and specific to that CPU's
> binding.  For ARM, we can have some global extensions to the CPU node.
> 
> The problems start when you want to refer to clusters and groups of
> CPUs from other nodes.  Only individual cpu nodes can be places in
> the cpus node, so there is no node for a phandle to point at.
> 
> If you want to describe how other things like power, clock and
> coherency domains map to clusters and larger entities, things could
> get pretty awkward.
> 
> Keeping the topology description separate allows all topological entities
> to appear as addressable entities in the DT; otherwise, a cryptic
> convention is needed.
> 
> 
> Hybrid approaches might be possible, putting cpu nodes into /cpus, and
> giving them a "parent" property where appropriate pointing at the
> relevant cluster node, which we put elsewhere in the DT.

That's what I did, with a couple of twists:

http://lists.infradead.org/pipermail/linux-arm-kernel/2012-January/080873.html

I have no preference, time to make a decision though.

> I'm not sure whether any of these approaches is an awful lot less ugly
> or more easy to handle than what it currently proposed though.

+1

> The global binding for all ARM CPUs would specify that the topology
> is described by /cpu-map and its associated binding.  For my
> interpretation of [2], this is a compliant approach.  ePAPR does not
> specify _how_ the cpu node binding achieves a description of the
> topography, just that it must achieve it.  There's no statement to
> say that it must not involve other nodes or bindings.

Again, I think it all boils down to deciding where cluster nodes should
live.

More opinions ?

Thx,
Lorenzo
tip-bot for Dave Martin April 12, 2013, 2:36 p.m. UTC | #9
On Fri, Apr 12, 2013 at 12:44:58PM +0100, Lorenzo Pieralisi wrote:
> On Thu, Apr 11, 2013 at 07:01:25PM +0100, Dave Martin wrote:
> 
> [...]
> 
> > > > > +===========================================
> > > > > +2 - cpu-map node
> > > > > +===========================================
> > > > > +
> > > > > +The ARM CPU topology is defined within a container node, sitting at the top
> > > > > +level of the device tree (/), the cpu-map node.
> > > > > +
> > > > > +- cpu-map node
> > > > > +
> > > > > + Usage: Required to define ARM CPU topology
> > > > > +
> > > > > + Description: The cpu-map node is just a container node where its
> > > > > +              subnodes describe the CPU topology
> > > > > +
> > > > > + Node name must be "cpu-map".
> > > > > +
> > > > > + A cpu-map node's child nodes can be:
> > > > > +
> > > > > + - one or more cluster nodes
> > > > > +
> > > > > + Any other configuration is considered invalid.
> > > > > +
> > > > > +The cpu-map node can only contain three types of child nodes:
> > > > > +
> > > > > +- cluster node
> > > > > +- core node
> > > > > +- thread node
> > > > > +
> > > >
> > > > Why not put the topology in the /cpus nodes? I don't really see the
> > > > point of having a flat list of cpus and separate topology info. There is
> > > > some compatibility issue, but adding optional levels for clusters can be
> > > > handled.
> > >
> > > I thought this would break all code relying on /cpu nodes being /cpus node's
> > > children. Furthermore, I was told that the /cpus node can only have /cpu nodes
> > > as children.
> > >
> > > If you wish so, we can define the topology in the /cpus node, fine by me.
> > 
> > Can we make such extensive changes to the cpus node without violating
> > the ePAPR specification?
> > 
> > If we can, great, but I'm a but unclear on how this would be achieved.
> 
> +1
> 
> > > > > +whose bindings are described in paragraph 3.
> > > > > +
> > > > > +The nodes describing the CPU topology (cluster/core/thread) can only be
> > > > > +defined within the cpu-map node.
> > > > > +Any other configuration is consider invalid and therefore must be ignored.
> > > > > +
> > > > > +===========================================
> > > > > +2.1 - cpu-map child nodes naming convention
> > > > > +===========================================
> > > > > +
> > > > > +cpu-map child nodes must follow a naming convention where the node name
> > > > > +must be "clusterN", "coreN", "threadN" depending on the node type (ie
> > > > > +cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
> > > > > +are siblings within a single common parent node must be given a unique and
> > > > > +sequential N value, starting from 0).
> > > > > +cpu-map child nodes which do not share a common parent node can have the same
> > > > > +name (ie same number N as other cpu-map child nodes at different device tree
> > > > > +levels) since name uniqueness will be guaranteed by the device tree hierarchy.
> > > > > +
> > > > > +===========================================
> > > > > +3 - cluster/core/thread node bindings
> > > > > +===========================================
> > > > > +
> > > > > +Bindings for cluster/cpu/thread nodes are defined as follows:
> > > > > +
> > > > > +- cluster node
> > > > > +
> > > > > +  Description: must be declared within a cpu-map node, one node
> > > > > +               per cluster. A system can contain several layers of
> > > > > +               clustering and cluster nodes can be contained in parent
> > > > > +               cluster nodes.
> > > > > +
> > > > > + The cluster node name must be "clusterN" as described in 2.1 above.
> > > > > + A cluster node can not be a leaf node.
> > > >
> > > > Follow standard conventions with "cluster@N" and a reg property with the
> > > > number.
> > >
> > > We are defining the topology to decouple the cluster/core/thread concept
> > > from the MPIDR. Having a reg property in the cluster (and core) nodes
> > > would complicate things if that reg property must correspond to an MPIDR
> > > bitfield. If it is meant to be just an enumeration at a given device tree
> > > level, I am ok with changing that.
> > 
> > As a pure enumeration, I think that works fine.  It's more verbose
> > but also more conformant to DT conventions.  I'm not sure there's
> > another difference.
> > 
> > The proposed support for C preprocessing of dts files might provide a
> > way to help debloat this to some extent in dts source, while still
> > following the DT convention of using unit addresses and reg properties.
> > This will significantly increase the size of the FDT blob if the
> > number of CPUs is large.  I don't remember offhand if we have a limit
> > on the size of FDT we can cope with.  Finding ways to relax the limit
> > is a better solution than dodging round standards, though.  We can
> > cross that bridge when/if we come to it.
> 
> Well, that's a problem by itself, certainly adding a reg property to
> the cluster nodes will make it worse, but as you said still better that
> than dodging standards.
> 
> > >
> > > > > +
> > > > > + A cluster node's child nodes must be:
> > > > > +
> > > > > + - one or more cluster nodes; or
> > > > > + - one or more core nodes
> > > > > +
> > > > > + Any other configuration is considered invalid.
> > > > > +
> > > > > +- core node
> > > > > +
> > > > > + Description: must be declared in a cluster node, one node per core in
> > > > > +              the cluster. If the system does not support SMT, core
> > > > > +              nodes are leaf nodes, otherwise they become containers of
> > > > > +              thread nodes.
> > > > > +
> > > > > + The core node name must be "coreN" as described in 2.1 above.
> > > > > +
> > > > > + A core node must be a leaf node if SMT is not supported.
> > > > > +
> > > > > + Properties for core nodes that are leaf nodes:
> > > > > +
> > > > > + - cpu
> > > > > +         Usage: required
> > > > > +         Value type: <phandle>
> > > > > +         Definition: a phandle to the cpu node that corresponds to the
> > > > > +                     core node.
> > > > > +
> > > > > + If a core node is not a leaf node (CPUs supporting SMT) a core node's
> > > > > + child nodes can be:
> > > > > +
> > > > > + - one or more thread nodes
> > > > > +
> > > > > + Any other configuration is considered invalid.
> > > > > +
> > > > > +- thread node
> > > > > +
> > > > > + Description: must be declared in a core node, one node per thread
> > > > > +              in the core if the system supports SMT. Thread nodes are
> > > > > +              always leaf nodes in the device tree.
> > > > > +
> > > > > + The thread node name must be "threadN" as described in 2.1 above.
> > > > > +
> > > > > + A thread node must be a leaf node.
> > > > > +
> > > > > + A thread node must contain the following property:
> > > > > +
> > > > > + - cpu
> > > > > +         Usage: required
> > > > > +         Value type: <phandle>
> > > > > +         Definition: a phandle to the cpu node that corresponds to
> > > > > +                     the thread node.
> > > >
> > > >
> > > > According to the ePAPR, threads are represented by an array of ids for
> > > > reg property, not another cpu node. Why the deviation.
> > >
> > > It is not a cpu node, it is a phandle property named cpu. Can you point
> > > me to the ePAPR section where threads bindings are described please ? I have
> > > not managed to find these details, I am reading version 1.0.
> > 
> > For cpu/reg:
> > 
> > [1]     If a CPU supports more than one thread (i.e. multiple streams of
> >         execution) the reg property is an array with 1 element per
> >         thread. The #address-cells on the /cpus node specifies how many
> >         cells each element of the array takes. Software can determine
> >         the number of threads by dividing the size of reg by the parent
> >         node's #address-cells.
> > 
> > I had not previously been aware of this, but I see no reason not to
> > follow this convention.
> 
> I don't see a reason either, but this changes the current cpu node bindings
> for ARM. On the upside there are no SMT ARM platforms out there, so no
> backward compatibility to worry about.

Actually, I've had second thoughts about this, from discussion with Mark
et al.

The extent to which threads share stuff is not really architecturally
visible on ARM.  It will be visible in performance terms (i.e., scheduling
two threads of the same process on threads of on CPU will give better
performance than scheduling threads of different processes), but in
architectural terms still look like fully-fledged, independent CPUs.

I don't know enough about how SMT scheduling currently works in the
kernel to know how best to describe this situation to the kernel...


Anyway, for the ARM case, there is not much architectural difference
between threads within a CPU, and CPUs in a cluster.  At both topological
levels the siblings are independent.  At both levels, there is an advantage
in scheduling related threads topologically close to each other -- though
probably more so for threads in a CPU than CPUs in a cluster.

Also, threads are independent interrupt destinations.  If we want to
put flat lists of SMT threads inside CPU nodes, then we need an
extra means of describing interrupt affinities, different from the
way this is described for CPUs and clusters.  This is definitely
complexity.  I'm not sure if there is a related benefit.


> This would reduce the topology problem to where cluster nodes should be
> defined, either in the cpus node or a separate node (ie cpu-map :-)).
> 
> > Also:
> > [2]     If other more complex CPU topographies are designed, the binding
> >         for the CPU must describe the topography
> > 
> > 
> > That's rather less helpful, but the suggestion is clear enough in that
> > such information should be in the cpu node and specific to that CPU's
> > binding.  For ARM, we can have some global extensions to the CPU node.
> > 
> > The problems start when you want to refer to clusters and groups of
> > CPUs from other nodes.  Only individual cpu nodes can be places in
> > the cpus node, so there is no node for a phandle to point at.
> > 
> > If you want to describe how other things like power, clock and
> > coherency domains map to clusters and larger entities, things could
> > get pretty awkward.
> > 
> > Keeping the topology description separate allows all topological entities
> > to appear as addressable entities in the DT; otherwise, a cryptic
> > convention is needed.
> > 
> > 
> > Hybrid approaches might be possible, putting cpu nodes into /cpus, and
> > giving them a "parent" property where appropriate pointing at the
> > relevant cluster node, which we put elsewhere in the DT.
> 
> That's what I did, with a couple of twists:
> 
> http://lists.infradead.org/pipermail/linux-arm-kernel/2012-January/080873.html
> 
> I have no preference, time to make a decision though.

With the arguments above, I'm not sure this is really better than the
current proposal...

> 
> > I'm not sure whether any of these approaches is an awful lot less ugly
> > or more easy to handle than what it currently proposed though.
> 
> +1
> 
> > The global binding for all ARM CPUs would specify that the topology
> > is described by /cpu-map and its associated binding.  For my
> > interpretation of [2], this is a compliant approach.  ePAPR does not
> > specify _how_ the cpu node binding achieves a description of the
> > topography, just that it must achieve it.  There's no statement to
> > say that it must not involve other nodes or bindings.
> 
> Again, I think it all boils down to deciding where cluster nodes should
> live.

If we want to be able to describe affinities and other hardware linkages,
describing the real hardware units as nodes still feels "right".
ePAPR doesn't insist upon how this is done, so we do have choice.

The older/hybrid proposals seem to require different means of describing
linkage depending on whether the target is a topological leaf or not.

I guess the question should be "what advantage is gained from describing
this stuff in the cpus node?"

Cheers
---Dave
Lorenzo Pieralisi April 12, 2013, 4:59 p.m. UTC | #10
On Fri, Apr 12, 2013 at 03:36:44PM +0100, Dave Martin wrote:

[...]

> > > > > According to the ePAPR, threads are represented by an array of ids for
> > > > > reg property, not another cpu node. Why the deviation.
> > > >
> > > > It is not a cpu node, it is a phandle property named cpu. Can you point
> > > > me to the ePAPR section where threads bindings are described please ? I have
> > > > not managed to find these details, I am reading version 1.0.
> > >
> > > For cpu/reg:
> > >
> > > [1]     If a CPU supports more than one thread (i.e. multiple streams of
> > >         execution) the reg property is an array with 1 element per
> > >         thread. The #address-cells on the /cpus node specifies how many
> > >         cells each element of the array takes. Software can determine
> > >         the number of threads by dividing the size of reg by the parent
> > >         node's #address-cells.
> > >
> > > I had not previously been aware of this, but I see no reason not to
> > > follow this convention.
> >
> > I don't see a reason either, but this changes the current cpu node bindings
> > for ARM. On the upside there are no SMT ARM platforms out there, so no
> > backward compatibility to worry about.
> 
> Actually, I've had second thoughts about this, from discussion with Mark
> et al.
> 
> The extent to which threads share stuff is not really architecturally
> visible on ARM.  It will be visible in performance terms (i.e., scheduling
> two threads of the same process on threads of on CPU will give better
> performance than scheduling threads of different processes), but in
> architectural terms still look like fully-fledged, independent CPUs.
> 
> I don't know enough about how SMT scheduling currently works in the
> kernel to know how best to describe this situation to the kernel...
> 
> 
> Anyway, for the ARM case, there is not much architectural difference
> between threads within a CPU, and CPUs in a cluster.  At both topological
> levels the siblings are independent.  At both levels, there is an advantage
> in scheduling related threads topologically close to each other -- though
> probably more so for threads in a CPU than CPUs in a cluster.
> 
> Also, threads are independent interrupt destinations.  If we want to
> put flat lists of SMT threads inside CPU nodes, then we need an
> extra means of describing interrupt affinities, different from the
> way this is described for CPUs and clusters.  This is definitely
> complexity.  I'm not sure if there is a related benefit.

Yes, I agree, I think the bindings we came up with are neater than
having threads as multiple reg properties in cpu nodes and adding cluster
nodes (within cpus node or elsewhere). On the interrupt affinity side I think
that it should still be feasible since cpu nodes would become containers
of threads, and cluster nodes containers (or pointed at by) of cpu nodes, but
it is true that the description won't be uniform anymore.

I prefer our solution :-)

> > This would reduce the topology problem to where cluster nodes should be
> > defined, either in the cpus node or a separate node (ie cpu-map :-)).
> >
> > > Also:
> > > [2]     If other more complex CPU topographies are designed, the binding
> > >         for the CPU must describe the topography
> > >
> > >
> > > That's rather less helpful, but the suggestion is clear enough in that
> > > such information should be in the cpu node and specific to that CPU's
> > > binding.  For ARM, we can have some global extensions to the CPU node.
> > >
> > > The problems start when you want to refer to clusters and groups of
> > > CPUs from other nodes.  Only individual cpu nodes can be places in
> > > the cpus node, so there is no node for a phandle to point at.
> > >
> > > If you want to describe how other things like power, clock and
> > > coherency domains map to clusters and larger entities, things could
> > > get pretty awkward.
> > >
> > > Keeping the topology description separate allows all topological entities
> > > to appear as addressable entities in the DT; otherwise, a cryptic
> > > convention is needed.
> > >
> > >
> > > Hybrid approaches might be possible, putting cpu nodes into /cpus, and
> > > giving them a "parent" property where appropriate pointing at the
> > > relevant cluster node, which we put elsewhere in the DT.
> >
> > That's what I did, with a couple of twists:
> >
> > http://lists.infradead.org/pipermail/linux-arm-kernel/2012-January/080873.html
> >
> > I have no preference, time to make a decision though.
> 
> With the arguments above, I'm not sure this is really better than the
> current proposal...
>

I am not sure either, that's why I would like to hear other opinions as well.

> > > I'm not sure whether any of these approaches is an awful lot less ugly
> > > or more easy to handle than what it currently proposed though.
> >
> > +1
> >
> > > The global binding for all ARM CPUs would specify that the topology
> > > is described by /cpu-map and its associated binding.  For my
> > > interpretation of [2], this is a compliant approach.  ePAPR does not
> > > specify _how_ the cpu node binding achieves a description of the
> > > topography, just that it must achieve it.  There's no statement to
> > > say that it must not involve other nodes or bindings.

I agree with you, the SMT thread specification in the ePAPR would become
a bit confusing though for ARM, we must make sure that what we are doing
and the ePAPR spec evolve in concert otherwise this will become unmanageable
in the long run.

> > Again, I think it all boils down to deciding where cluster nodes should
> > live.
> 
> If we want to be able to describe affinities and other hardware linkages,
> describing the real hardware units as nodes still feels "right".
> ePAPR doesn't insist upon how this is done, so we do have choice.
> 
> The older/hybrid proposals seem to require different means of describing
> linkage depending on whether the target is a topological leaf or not.
> 
> I guess the question should be "what advantage is gained from describing
> this stuff in the cpus node?"

We do not need phandles (pointers to cpu nodes) to describe the topology,
I guess, is the only answer.
Backward compatibility is still my main worry as far as the ePAPR is
concerned, so I am still looking forward to getting feedback from powerPC
and DT people on our proposal.

Thanks a lot,
Lorenzo
diff mbox

Patch

diff --git a/Documentation/devicetree/bindings/arm/topology.txt b/Documentation/devicetree/bindings/arm/topology.txt
new file mode 100644
index 0000000..07c4961
--- /dev/null
+++ b/Documentation/devicetree/bindings/arm/topology.txt
@@ -0,0 +1,524 @@ 
+===========================================
+ARM topology binding description
+===========================================
+
+===========================================
+1 - Introduction
+===========================================
+
+In an ARM system, the hierarchy of CPUs is defined through three entities that
+are used to describe the layout of physical CPUs in the system:
+
+- cluster
+- core
+- thread
+
+The cpu nodes (bindings defined in [1]) represent the devices that
+correspond to physical CPUs and are to be mapped to the hierarchy levels.
+
+The bottom hierarchy level sits at core or thread level depending on whether
+symmetric multi-threading (SMT) is supported or not.
+
+For instance in a system where CPUs support SMT, "cpu" nodes represent all
+threads existing in the system and map to the hierarchy level "thread" above.
+In systems where SMT is not supported "cpu" nodes represent all cores present
+in the system and map to the hierarchy level "core" above.
+
+ARM topology bindings allow one to associate cpu nodes with hierarchical groups
+corresponding to the system hierarchy; syntactically they are defined as device
+tree nodes.
+
+The remainder of this document provides the topology bindings for ARM, based
+on the ePAPR standard, available from:
+
+http://devicetree.org
+
+If not stated otherwise, whenever a reference to a cpu node phandle is made its
+value must point to a cpu node compliant with the cpu node bindings as
+documented in [1].
+A topology description containing phandles to cpu nodes that are not compliant
+with bindings standardized in [1] is therefore considered invalid.
+
+===========================================
+2 - cpu-map node
+===========================================
+
+The ARM CPU topology is defined within a container node, sitting at the top
+level of the device tree (/), the cpu-map node.
+
+- cpu-map node
+
+	Usage: Required to define ARM CPU topology
+
+	Description: The cpu-map node is just a container node where its
+		     subnodes describe the CPU topology
+
+	Node name must be "cpu-map".
+
+	A cpu-map node's child nodes can be:
+
+	- one or more cluster nodes
+
+	Any other configuration is considered invalid.
+
+The cpu-map node can only contain three types of child nodes:
+
+- cluster node
+- core node
+- thread node
+
+whose bindings are described in paragraph 3.
+
+The nodes describing the CPU topology (cluster/core/thread) can only be
+defined within the cpu-map node.
+Any other configuration is consider invalid and therefore must be ignored.
+
+===========================================
+2.1 - cpu-map child nodes naming convention
+===========================================
+
+cpu-map child nodes must follow a naming convention where the node name
+must be "clusterN", "coreN", "threadN" depending on the node type (ie
+cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
+are siblings within a single common parent node must be given a unique and
+sequential N value, starting from 0).
+cpu-map child nodes which do not share a common parent node can have the same
+name (ie same number N as other cpu-map child nodes at different device tree
+levels) since name uniqueness will be guaranteed by the device tree hierarchy.
+
+===========================================
+3 - cluster/core/thread node bindings
+===========================================
+
+Bindings for cluster/cpu/thread nodes are defined as follows:
+
+- cluster node
+
+	 Description: must be declared within a cpu-map node, one node
+		      per cluster. A system can contain several layers of
+		      clustering and cluster nodes can be contained in parent
+		      cluster nodes.
+
+	The cluster node name must be "clusterN" as described in 2.1 above.
+	A cluster node can not be a leaf node.
+
+	A cluster node's child nodes must be:
+
+	- one or more cluster nodes; or
+	- one or more core nodes
+
+	Any other configuration is considered invalid.
+
+- core node
+
+	Description: must be declared in a cluster node, one node per core in
+		     the cluster. If the system does not support SMT, core
+		     nodes are leaf nodes, otherwise they become containers of
+		     thread nodes.
+
+	The core node name must be "coreN" as described in 2.1 above.
+
+	A core node must be a leaf node if SMT is not supported.
+
+	Properties for core nodes that are leaf nodes:
+
+	- cpu
+		Usage: required
+		Value type: <phandle>
+		Definition: a phandle to the cpu node that corresponds to the
+			    core node.
+
+	If a core node is not a leaf node (CPUs supporting SMT) a core node's
+	child nodes can be:
+
+	- one or more thread nodes
+
+	Any other configuration is considered invalid.
+
+- thread node
+
+	Description: must be declared in a core node, one node per thread
+		     in the core if the system supports SMT. Thread nodes are
+		     always leaf nodes in the device tree.
+
+	The thread node name must be "threadN" as described in 2.1 above.
+
+	A thread node must be a leaf node.
+
+	A thread node must contain the following property:
+
+	- cpu
+		Usage: required
+		Value type: <phandle>
+		Definition: a phandle to the cpu node that corresponds to
+			    the thread node.
+
+===========================================
+4 - Example dts
+===========================================
+
+Example 1 (ARM 64-bit, 16-cpu system, two clusters of clusters):
+
+cpu-map {
+
+	cluster0 {
+
+		cluster0 {
+
+			core0 {
+
+				thread0 {
+					cpu = <&CPU0>;
+				};
+
+				thread1 {
+					cpu = <&CPU1>;
+				};
+			};
+
+			core1 {
+
+				thread0 {
+					cpu = <&CPU2>;
+				};
+
+				thread1 {
+					cpu = <&CPU3>;
+				};
+			};
+		};
+
+		cluster1 {
+
+			core0 {
+
+				thread0 {
+					cpu = <&CPU4>;
+				};
+
+				thread1 {
+					cpu = <&CPU5>;
+				};
+			};
+
+			core1 {
+
+				thread0 {
+					cpu = <&CPU6>;
+				};
+
+				thread1 {
+					cpu = <&CPU7>;
+				};
+			};
+		};
+	};
+
+	cluster1 {
+
+		cluster0 {
+
+			core0 {
+
+				thread0 {
+					cpu = <&CPU8>;
+				};
+
+				thread1 {
+					cpu = <&CPU9>;
+				};
+			};
+
+			core1 {
+
+				thread0 {
+					cpu = <&CPU10>;
+				};
+
+				thread1 {
+					cpu = <&CPU11>;
+				};
+			};
+		};
+
+		cluster1 {
+
+			core0 {
+
+				thread0 {
+					cpu = <&CPU12>;
+				};
+
+				thread1 {
+					cpu = <&CPU13>;
+				};
+			};
+
+			core1 {
+
+				thread0 {
+					cpu = <&CPU14>;
+				};
+
+				thread1 {
+					cpu = <&CPU15>;
+				};
+			};
+		};
+	};
+};
+
+cpus {
+	#size-cells = <0>;
+	#address-cells = <2>;
+
+	CPU0: cpu@0 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x0>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU1: cpu@1 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x1>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU2: cpu@100 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x100>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU3: cpu@101 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x101>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU4: cpu@10000 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x10000>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU5: cpu@10001 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x10001>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU6: cpu@10100 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x10100>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU7: cpu@10101 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x10101>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU8: cpu@100000000 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x0>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU9: cpu@100000001 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x1>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU10: cpu@100000100 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x100>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU11: cpu@100000101 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x101>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU12: cpu@100010000 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x10000>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU13: cpu@100010001 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x10001>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU14: cpu@100010100 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x10100>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU15: cpu@100010101 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x10101>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+};
+
+Example 2 (ARM 32-bit, dual-cluster, 8-cpu system, no SMT):
+
+cpu-map {
+
+	cluster0 {
+
+		core0 {
+			cpu = <&CPU0>;
+		};
+
+		core1 {
+			cpu = <&CPU1>;
+		};
+
+		core2 {
+			cpu = <&CPU2>;
+		};
+
+		core3 {
+			cpu = <&CPU3>;
+		};
+
+	};
+
+	cluster1 {
+
+		core0 {
+			cpu = <&CPU4>;
+		};
+
+		core1 {
+			cpu = <&CPU5>;
+		};
+
+		core2 {
+			cpu = <&CPU6>;
+		};
+
+		core3 {
+			cpu = <&CPU7>;
+		};
+
+	};
+
+cpus {
+	#size-cells = <0>;
+	#address-cells = <1>;
+
+	CPU0: cpu@0 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a15";
+		reg = <0x0>;
+	};
+
+	CPU1: cpu@1 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a15";
+		reg = <0x1>;
+	};
+
+	CPU2: cpu@2 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a15";
+		reg = <0x2>;
+	};
+
+	CPU3: cpu@3 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a15";
+		reg = <0x3>;
+	};
+
+	CPU4: cpu@100 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a7";
+		reg = <0x100>;
+	};
+
+	CPU5: cpu@101 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a7";
+		reg = <0x101>;
+	};
+
+	CPU6: cpu@102 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a7";
+		reg = <0x102>;
+	};
+
+	CPU7: cpu@103 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a7";
+		reg = <0x103>;
+	};
+};
+
+Example 3 (ARM 32-bit, cortex-a8 single core):
+
+cpu-map {
+	cluster0 {
+		core0 {
+			cpu = <&CPU0>;
+		};
+	};
+};
+
+cpus {
+	#size-cells = <0>;
+	#address-cells = <1>;
+
+	CPU0: cpu@0 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a8";
+		reg = <0x0>;
+	};
+};
+
+===============================================================================
+[1] ARM Linux kernel documentation
+    Documentation/devicetree/bindings/arm/cpus.txt