[PATCHv7,10/10] doc/mm: New documentation for memory performance

Commit Message

Keith Busch Feb. 27, 2019, 10:50 p.m. UTC

Platforms may provide system memory where some physical address ranges
perform differently than others, or is side cached by the system.

Add documentation describing a high level overview of such systems and the
perforamnce and caching attributes the kernel provides for applications
wishing to query this information.

Reviewed-by: Mike Rapoport <[e-mail]>
Signed-off-by: Keith Busch <[e-mail]>
---
 Documentation/admin-guide/mm/numaperf.rst | 164 ++++++++++++++++++++++++++++++
 1 file changed, 164 insertions(+)
 create mode 100644 Documentation/admin-guide/mm/numaperf.rst

Comments

Jonathan Cameron March 11, 2019, 11:38 a.m. UTC | #1

On Wed, 27 Feb 2019 15:50:38 -0700
Keith Busch <[e-mail]> wrote:

> Platforms may provide system memory where some physical address ranges
> perform differently than others, or is side cached by the system.
The magic 'side cached' term still here in the patch description, ideally
wants cleaning up.

> 
> Add documentation describing a high level overview of such systems and the
> perforamnce and caching attributes the kernel provides for applications
performance

> wishing to query this information.
> 
> Reviewed-by: Mike Rapoport <[e-mail]>
> Signed-off-by: Keith Busch <[e-mail]>

A few comments inline. Mostly the weird corner cases that I miss understood
in one of the earlier versions of the code.

Whilst I think perhaps that one section could be tweaked a tiny bit I'm basically
happy with this if you don't want to.

Reviewed-by: Jonathan Cameron <[e-mail]>

> ---
>  Documentation/admin-guide/mm/numaperf.rst | 164 ++++++++++++++++++++++++++++++
>  1 file changed, 164 insertions(+)
>  create mode 100644 Documentation/admin-guide/mm/numaperf.rst
> 
> diff --git a/Documentation/admin-guide/mm/numaperf.rst b/Documentation/admin-guide/mm/numaperf.rst
> new file mode 100644
> index 000000000000..d32756b9be48
> --- /dev/null
> +++ b/Documentation/admin-guide/mm/numaperf.rst
> @@ -0,0 +1,164 @@
> +.. _numaperf:
> +
> +=============
> +NUMA Locality
> +=============
> +
> +Some platforms may have multiple types of memory attached to a compute
> +node. These disparate memory ranges may share some characteristics, such
> +as CPU cache coherence, but may have different performance. For example,
> +different media types and buses affect bandwidth and latency.
> +
> +A system supports such heterogeneous memory by grouping each memory type
> +under different domains, or "nodes", based on locality and performance
> +characteristics.  Some memory may share the same node as a CPU, and others
> +are provided as memory only nodes. While memory only nodes do not provide
> +CPUs, they may still be local to one or more compute nodes relative to
> +other nodes. The following diagram shows one such example of two compute
> +nodes with local memory and a memory only node for each of compute node:
> +
> + +------------------+     +------------------+
> + | Compute Node 0   +-----+ Compute Node 1   |
> + | Local Node0 Mem  |     | Local Node1 Mem  |
> + +--------+---------+     +--------+---------+
> +          |                        |
> + +--------+---------+     +--------+---------+
> + | Slower Node2 Mem |     | Slower Node3 Mem |
> + +------------------+     +--------+---------+
> +
> +A "memory initiator" is a node containing one or more devices such as
> +CPUs or separate memory I/O devices that can initiate memory requests.
> +A "memory target" is a node containing one or more physical address
> +ranges accessible from one or more memory initiators.
> +
> +When multiple memory initiators exist, they may not all have the same
> +performance when accessing a given memory target. Each initiator-target
> +pair may be organized into different ranked access classes to represent
> +this relationship. 

This concept is a bit vague at the moment. Largely because only access0
is actually defined.  We should definitely keep a close eye on any others
that are defined in future to make sure this text is still valid.

I can certainly see it being used for different ideas of 'best' rather
than simply best and second best etc.

> The highest performing initiator to a given target
> +is considered to be one of that target's local initiators, and given
> +the highest access class, 0. Any given target may have one or more
> +local initiators, and any given initiator may have multiple local
> +memory targets.
> +
> +To aid applications matching memory targets with their initiators, the
> +kernel provides symlinks to each other. The following example lists the
> +relationship for the access class "0" memory initiators and targets, which is
> +the of nodes with the highest performing access relationship::
> +
> +	# symlinks -v /sys/devices/system/node/nodeX/access0/targets/
> +	relative: /sys/devices/system/node/nodeX/access0/targets/nodeY -> ../../nodeY

So this one perhaps needs a bit more description - I would put it after initiators
which precisely fits the description you have here now.

"targets contains those nodes for which this initiator is the best possible initiator."

which is subtly different form

"targets contains those nodes to which this node has the highest
performing access characteristics."

For example in my test case:
* 4 nodes with local memory and cpu, 1 node remote and equal distant from all of the
  initiators,

targets for the compute nodes contains both themselves and the remote node, to which
the characteristics are of course worse. As you point out before, we need to look
in 
node0/access0/targets/node0/access0/initiators 
node0/access0/targets/node4/access0/initiators 
to get the relevant characteristics and work out that node0 is 'nearer' itself
(obviously this is a bit of a silly case, but we could have no memory node0 and
be talking about node4 and node5.

I am happy with the actual interface, this is just a question about whether we can tweak
this text to be slightly clearer.

> +
> +	# symlinks -v /sys/devices/system/node/nodeY/access0/initiators/
> +	relative: /sys/devices/system/node/nodeY/access0/initiators/nodeX -> ../../nodeX
> +
> +================
> +NUMA Performance
> +================
> +
> +Applications may wish to consider which node they want their memory to
> +be allocated from based on the node's performance characteristics. If
> +the system provides these attributes, the kernel exports them under the
> +node sysfs hierarchy by appending the attributes directory under the
> +memory node's access class 0 initiators as follows::
> +
> +	/sys/devices/system/node/nodeY/access0/initiators/
> +
> +These attributes apply only when accessed from nodes that have the
> +are linked under the this access's inititiators.
> +
> +The performance characteristics the kernel provides for the local initiators
> +are exported are as follows::
> +
> +	# tree -P "read*|write*" /sys/devices/system/node/nodeY/access0/initiators/
> +	/sys/devices/system/node/nodeY/access0/initiators/
> +	|-- read_bandwidth
> +	|-- read_latency
> +	|-- write_bandwidth
> +	`-- write_latency
> +
> +The bandwidth attributes are provided in MiB/second.
> +
> +The latency attributes are provided in nanoseconds.
> +
> +The values reported here correspond to the rated latency and bandwidth
> +for the platform.
> +
> +==========
> +NUMA Cache
> +==========
> +
> +System memory may be constructed in a hierarchy of elements with various
> +performance characteristics in order to provide large address space of
> +slower performing memory cached by a smaller higher performing memory. The
> +system physical addresses memory  initiators are aware of are provided
> +by the last memory level in the hierarchy. The system meanwhile uses
> +higher performing memory to transparently cache access to progressively
> +slower levels.
> +
> +The term "far memory" is used to denote the last level memory in the
> +hierarchy. Each increasing cache level provides higher performing
> +initiator access, and the term "near memory" represents the fastest
> +cache provided by the system.
> +
> +This numbering is different than CPU caches where the cache level (ex:
> +L1, L2, L3) uses the CPU-side view where each increased level is lower
> +performing. In contrast, the memory cache level is centric to the last
> +level memory, so the higher numbered cache level corresponds to  memory
> +nearer to the CPU, and further from far memory.
> +
> +The memory-side caches are not directly addressable by software. When
> +software accesses a system address, the system will return it from the
> +near memory cache if it is present. If it is not present, the system
> +accesses the next level of memory until there is either a hit in that
> +cache level, or it reaches far memory.
> +
> +An application does not need to know about caching attributes in order
> +to use the system. Software may optionally query the memory cache
> +attributes in order to maximize the performance out of such a setup.
> +If the system provides a way for the kernel to discover this information,
> +for example with ACPI HMAT (Heterogeneous Memory Attribute Table),
> +the kernel will append these attributes to the NUMA node memory target.
> +
> +When the kernel first registers a memory cache with a node, the kernel
> +will create the following directory::

Real nitpick but more precisely, "If relevant, the kernel..."  Otherwise
we say it's always there but then say it isn't below.

> +
> +	/sys/devices/system/node/nodeX/memory_side_cache/
> +
> +If that directory is not present, the system either does not not provide
> +a memory-side cache, or that information is not accessible to the kernel.
> +
> +The attributes for each level of cache is provided under its cache
> +level index::
> +
> +	/sys/devices/system/node/nodeX/memory_side_cache/indexA/
> +	/sys/devices/system/node/nodeX/memory_side_cache/indexB/
> +	/sys/devices/system/node/nodeX/memory_side_cache/indexC/
> +
> +Each cache level's directory provides its attributes. For example, the
> +following shows a single cache level and the attributes available for
> +software to query::
> +
> +	# tree sys/devices/system/node/node0/memory_side_cache/
> +	/sys/devices/system/node/node0/memory_side_cache/
> +	|-- index1
> +	|   |-- indexing
> +	|   |-- line_size
> +	|   |-- size
> +	|   `-- write_policy
> +
> +The "indexing" will be 0 if it is a direct-mapped cache, and non-zero
> +for any other indexed based, multi-way associativity.
> +
> +The "line_size" is the number of bytes accessed from the next cache
> +level on a miss.
> +
> +The "size" is the number of bytes provided by this cache level.
> +
> +The "write_policy" will be 0 for write-back, and non-zero for
> +write-through caching.
> +
> +========
> +See Also
> +========
> +.. [1] https://www.uefi.org/sites/default/files/resources/ACPI_6_2.pdf
> +       Section 5.2.27

Keith Busch March 11, 2019, 8:16 p.m. UTC | #2

On Mon, Mar 11, 2019 at 04:38:43AM -0700, Jonathan Cameron wrote:
> On Wed, 27 Feb 2019 15:50:38 -0700
> Keith Busch <[e-mail]> wrote:
> 
> > Platforms may provide system memory where some physical address ranges
> > perform differently than others, or is side cached by the system.
> The magic 'side cached' term still here in the patch description, ideally
> wants cleaning up.
> 
> > 
> > Add documentation describing a high level overview of such systems and the
> > perforamnce and caching attributes the kernel provides for applications
> performance
> 
> > wishing to query this information.
> > 
> > Reviewed-by: Mike Rapoport <[e-mail]>
> > Signed-off-by: Keith Busch <[e-mail]>
> 
> A few comments inline. Mostly the weird corner cases that I miss understood
> in one of the earlier versions of the code.
> 
> Whilst I think perhaps that one section could be tweaked a tiny bit I'm basically
> happy with this if you don't want to.
> 
> Reviewed-by: Jonathan Cameron <[e-mail]>
> 
> > ---
> >  Documentation/admin-guide/mm/numaperf.rst | 164 ++++++++++++++++++++++++++++++
> >  1 file changed, 164 insertions(+)
> >  create mode 100644 Documentation/admin-guide/mm/numaperf.rst
> > 
> > diff --git a/Documentation/admin-guide/mm/numaperf.rst b/Documentation/admin-guide/mm/numaperf.rst
> > new file mode 100644
> > index 000000000000..d32756b9be48
> > --- /dev/null
> > +++ b/Documentation/admin-guide/mm/numaperf.rst
> > @@ -0,0 +1,164 @@
> > +.. _numaperf:
> > +
> > +=============
> > +NUMA Locality
> > +=============
> > +
> > +Some platforms may have multiple types of memory attached to a compute
> > +node. These disparate memory ranges may share some characteristics, such
> > +as CPU cache coherence, but may have different performance. For example,
> > +different media types and buses affect bandwidth and latency.
> > +
> > +A system supports such heterogeneous memory by grouping each memory type
> > +under different domains, or "nodes", based on locality and performance
> > +characteristics.  Some memory may share the same node as a CPU, and others
> > +are provided as memory only nodes. While memory only nodes do not provide
> > +CPUs, they may still be local to one or more compute nodes relative to
> > +other nodes. The following diagram shows one such example of two compute
> > +nodes with local memory and a memory only node for each of compute node:
> > +
> > + +------------------+     +------------------+
> > + | Compute Node 0   +-----+ Compute Node 1   |
> > + | Local Node0 Mem  |     | Local Node1 Mem  |
> > + +--------+---------+     +--------+---------+
> > +          |                        |
> > + +--------+---------+     +--------+---------+
> > + | Slower Node2 Mem |     | Slower Node3 Mem |
> > + +------------------+     +--------+---------+
> > +
> > +A "memory initiator" is a node containing one or more devices such as
> > +CPUs or separate memory I/O devices that can initiate memory requests.
> > +A "memory target" is a node containing one or more physical address
> > +ranges accessible from one or more memory initiators.
> > +
> > +When multiple memory initiators exist, they may not all have the same
> > +performance when accessing a given memory target. Each initiator-target
> > +pair may be organized into different ranked access classes to represent
> > +this relationship. 
> 
> This concept is a bit vague at the moment. Largely because only access0
> is actually defined.  We should definitely keep a close eye on any others
> that are defined in future to make sure this text is still valid.
> 
> I can certainly see it being used for different ideas of 'best' rather
> than simply best and second best etc.

I tried to make the interface flexible to future extension, but I'm
still not sure how potential users would want to see something like
all pair-wise attributes, so I had some trouble trying to capture that
in words.
 
> > The highest performing initiator to a given target
> > +is considered to be one of that target's local initiators, and given
> > +the highest access class, 0. Any given target may have one or more
> > +local initiators, and any given initiator may have multiple local
> > +memory targets.
> > +
> > +To aid applications matching memory targets with their initiators, the
> > +kernel provides symlinks to each other. The following example lists the
> > +relationship for the access class "0" memory initiators and targets, which is
> > +the of nodes with the highest performing access relationship::
> > +
> > +	# symlinks -v /sys/devices/system/node/nodeX/access0/targets/
> > +	relative: /sys/devices/system/node/nodeX/access0/targets/nodeY -> ../../nodeY
> 
> So this one perhaps needs a bit more description - I would put it after initiators
> which precisely fits the description you have here now.
> 
> "targets contains those nodes for which this initiator is the best possible initiator."
> 
> which is subtly different form
> 
> "targets contains those nodes to which this node has the highest
> performing access characteristics."
> 
> For example in my test case:
> * 4 nodes with local memory and cpu, 1 node remote and equal distant from all of the
>   initiators,
> 
> targets for the compute nodes contains both themselves and the remote node, to which
> the characteristics are of course worse. As you point out before, we need to look
> in 
> node0/access0/targets/node0/access0/initiators 
> node0/access0/targets/node4/access0/initiators 
> to get the relevant characteristics and work out that node0 is 'nearer' itself
> (obviously this is a bit of a silly case, but we could have no memory node0 and
> be talking about node4 and node5.
> 
> I am happy with the actual interface, this is just a question about whether we can tweak
> this text to be slightly clearer.

Sure, I mention this in patch 4's commit message. Probably worth
repeating here:

    A memory initiator may have multiple memory targets in the same access
    class. The target memory's initiators in a given class indicate the
    nodes access characteristics share the same performance relative to other
    linked initiator nodes. Each target within an initiator's access class,
    though, do not necessarily perform the same as each other.

Jonathan Cameron March 12, 2019, 1:37 p.m. UTC | #3

On Mon, 11 Mar 2019 14:16:33 -0600
Keith Busch <[e-mail]> wrote:

> On Mon, Mar 11, 2019 at 04:38:43AM -0700, Jonathan Cameron wrote:
> > On Wed, 27 Feb 2019 15:50:38 -0700
> > Keith Busch <[e-mail]> wrote:
> >   
> > > Platforms may provide system memory where some physical address ranges
> > > perform differently than others, or is side cached by the system.  
> > The magic 'side cached' term still here in the patch description, ideally
> > wants cleaning up.
> >   
> > > 
> > > Add documentation describing a high level overview of such systems and the
> > > perforamnce and caching attributes the kernel provides for applications  
> > performance
> >   
> > > wishing to query this information.
> > > 
> > > Reviewed-by: Mike Rapoport <[e-mail]>
> > > Signed-off-by: Keith Busch <[e-mail]>  
> > 
> > A few comments inline. Mostly the weird corner cases that I miss understood
> > in one of the earlier versions of the code.
> > 
> > Whilst I think perhaps that one section could be tweaked a tiny bit I'm basically
> > happy with this if you don't want to.
> > 
> > Reviewed-by: Jonathan Cameron <[e-mail]>
> >   
> > > ---
> > >  Documentation/admin-guide/mm/numaperf.rst | 164 ++++++++++++++++++++++++++++++
> > >  1 file changed, 164 insertions(+)
> > >  create mode 100644 Documentation/admin-guide/mm/numaperf.rst
> > > 
> > > diff --git a/Documentation/admin-guide/mm/numaperf.rst b/Documentation/admin-guide/mm/numaperf.rst
> > > new file mode 100644
> > > index 000000000000..d32756b9be48
> > > --- /dev/null
> > > +++ b/Documentation/admin-guide/mm/numaperf.rst
> > > @@ -0,0 +1,164 @@
> > > +.. _numaperf:
> > > +
> > > +=============
> > > +NUMA Locality
> > > +=============
> > > +
> > > +Some platforms may have multiple types of memory attached to a compute
> > > +node. These disparate memory ranges may share some characteristics, such
> > > +as CPU cache coherence, but may have different performance. For example,
> > > +different media types and buses affect bandwidth and latency.
> > > +
> > > +A system supports such heterogeneous memory by grouping each memory type
> > > +under different domains, or "nodes", based on locality and performance
> > > +characteristics.  Some memory may share the same node as a CPU, and others
> > > +are provided as memory only nodes. While memory only nodes do not provide
> > > +CPUs, they may still be local to one or more compute nodes relative to
> > > +other nodes. The following diagram shows one such example of two compute
> > > +nodes with local memory and a memory only node for each of compute node:
> > > +
> > > + +------------------+     +------------------+
> > > + | Compute Node 0   +-----+ Compute Node 1   |
> > > + | Local Node0 Mem  |     | Local Node1 Mem  |
> > > + +--------+---------+     +--------+---------+
> > > +          |                        |
> > > + +--------+---------+     +--------+---------+
> > > + | Slower Node2 Mem |     | Slower Node3 Mem |
> > > + +------------------+     +--------+---------+
> > > +
> > > +A "memory initiator" is a node containing one or more devices such as
> > > +CPUs or separate memory I/O devices that can initiate memory requests.
> > > +A "memory target" is a node containing one or more physical address
> > > +ranges accessible from one or more memory initiators.
> > > +
> > > +When multiple memory initiators exist, they may not all have the same
> > > +performance when accessing a given memory target. Each initiator-target
> > > +pair may be organized into different ranked access classes to represent
> > > +this relationship.   
> > 
> > This concept is a bit vague at the moment. Largely because only access0
> > is actually defined.  We should definitely keep a close eye on any others
> > that are defined in future to make sure this text is still valid.
> > 
> > I can certainly see it being used for different ideas of 'best' rather
> > than simply best and second best etc.  
> 
> I tried to make the interface flexible to future extension, but I'm
> still not sure how potential users would want to see something like
> all pair-wise attributes, so I had some trouble trying to capture that
> in words.

Agreed, it is definitely non obvious.  We might end up with something
totally different like Jerome is proposing anyway.  Let's address
this when it happens!

>  
> > > The highest performing initiator to a given target
> > > +is considered to be one of that target's local initiators, and given
> > > +the highest access class, 0. Any given target may have one or more
> > > +local initiators, and any given initiator may have multiple local
> > > +memory targets.
> > > +
> > > +To aid applications matching memory targets with their initiators, the
> > > +kernel provides symlinks to each other. The following example lists the
> > > +relationship for the access class "0" memory initiators and targets, which is
> > > +the of nodes with the highest performing access relationship::
> > > +
> > > +	# symlinks -v /sys/devices/system/node/nodeX/access0/targets/
> > > +	relative: /sys/devices/system/node/nodeX/access0/targets/nodeY -> ../../nodeY  
> > 
> > So this one perhaps needs a bit more description - I would put it after initiators
> > which precisely fits the description you have here now.
> > 
> > "targets contains those nodes for which this initiator is the best possible initiator."
> > 
> > which is subtly different form
> > 
> > "targets contains those nodes to which this node has the highest
> > performing access characteristics."
> > 
> > For example in my test case:
> > * 4 nodes with local memory and cpu, 1 node remote and equal distant from all of the
> >   initiators,
> > 
> > targets for the compute nodes contains both themselves and the remote node, to which
> > the characteristics are of course worse. As you point out before, we need to look
> > in 
> > node0/access0/targets/node0/access0/initiators 
> > node0/access0/targets/node4/access0/initiators 
> > to get the relevant characteristics and work out that node0 is 'nearer' itself
> > (obviously this is a bit of a silly case, but we could have no memory node0 and
> > be talking about node4 and node5.
> > 
> > I am happy with the actual interface, this is just a question about whether we can tweak
> > this text to be slightly clearer.  
> 
> Sure, I mention this in patch 4's commit message. Probably worth
> repeating here:
> 
>     A memory initiator may have multiple memory targets in the same access
>     class. The target memory's initiators in a given class indicate the
>     nodes access characteristics share the same performance relative to other
>     linked initiator nodes. Each target within an initiator's access class,
>     though, do not necessarily perform the same as each other.
That sounds good to me.

Patch

diff --git a/Documentation/admin-guide/mm/numaperf.rst b/Documentation/admin-guide/mm/numaperf.rst
new file mode 100644
index 000000000000..d32756b9be48
--- /dev/null
+++ b/Documentation/admin-guide/mm/numaperf.rst
@@ -0,0 +1,164 @@ 
+.. _numaperf:
+
+=============
+NUMA Locality
+=============
+
+Some platforms may have multiple types of memory attached to a compute
+node. These disparate memory ranges may share some characteristics, such
+as CPU cache coherence, but may have different performance. For example,
+different media types and buses affect bandwidth and latency.
+
+A system supports such heterogeneous memory by grouping each memory type
+under different domains, or "nodes", based on locality and performance
+characteristics.  Some memory may share the same node as a CPU, and others
+are provided as memory only nodes. While memory only nodes do not provide
+CPUs, they may still be local to one or more compute nodes relative to
+other nodes. The following diagram shows one such example of two compute
+nodes with local memory and a memory only node for each of compute node:
+
+ +------------------+     +------------------+
+ | Compute Node 0   +-----+ Compute Node 1   |
+ | Local Node0 Mem  |     | Local Node1 Mem  |
+ +--------+---------+     +--------+---------+
+          |                        |
+ +--------+---------+     +--------+---------+
+ | Slower Node2 Mem |     | Slower Node3 Mem |
+ +------------------+     +--------+---------+
+
+A "memory initiator" is a node containing one or more devices such as
+CPUs or separate memory I/O devices that can initiate memory requests.
+A "memory target" is a node containing one or more physical address
+ranges accessible from one or more memory initiators.
+
+When multiple memory initiators exist, they may not all have the same
+performance when accessing a given memory target. Each initiator-target
+pair may be organized into different ranked access classes to represent
+this relationship. The highest performing initiator to a given target
+is considered to be one of that target's local initiators, and given
+the highest access class, 0. Any given target may have one or more
+local initiators, and any given initiator may have multiple local
+memory targets.
+
+To aid applications matching memory targets with their initiators, the
+kernel provides symlinks to each other. The following example lists the
+relationship for the access class "0" memory initiators and targets, which is
+the of nodes with the highest performing access relationship::
+
+	# symlinks -v /sys/devices/system/node/nodeX/access0/targets/
+	relative: /sys/devices/system/node/nodeX/access0/targets/nodeY -> ../../nodeY
+
+	# symlinks -v /sys/devices/system/node/nodeY/access0/initiators/
+	relative: /sys/devices/system/node/nodeY/access0/initiators/nodeX -> ../../nodeX
+
+================
+NUMA Performance
+================
+
+Applications may wish to consider which node they want their memory to
+be allocated from based on the node's performance characteristics. If
+the system provides these attributes, the kernel exports them under the
+node sysfs hierarchy by appending the attributes directory under the
+memory node's access class 0 initiators as follows::
+
+	/sys/devices/system/node/nodeY/access0/initiators/
+
+These attributes apply only when accessed from nodes that have the
+are linked under the this access's inititiators.
+
+The performance characteristics the kernel provides for the local initiators
+are exported are as follows::
+
+	# tree -P "read*|write*" /sys/devices/system/node/nodeY/access0/initiators/
+	/sys/devices/system/node/nodeY/access0/initiators/
+	|-- read_bandwidth
+	|-- read_latency
+	|-- write_bandwidth
+	`-- write_latency
+
+The bandwidth attributes are provided in MiB/second.
+
+The latency attributes are provided in nanoseconds.
+
+The values reported here correspond to the rated latency and bandwidth
+for the platform.
+
+==========
+NUMA Cache
+==========
+
+System memory may be constructed in a hierarchy of elements with various
+performance characteristics in order to provide large address space of
+slower performing memory cached by a smaller higher performing memory. The
+system physical addresses memory  initiators are aware of are provided
+by the last memory level in the hierarchy. The system meanwhile uses
+higher performing memory to transparently cache access to progressively
+slower levels.
+
+The term "far memory" is used to denote the last level memory in the
+hierarchy. Each increasing cache level provides higher performing
+initiator access, and the term "near memory" represents the fastest
+cache provided by the system.
+
+This numbering is different than CPU caches where the cache level (ex:
+L1, L2, L3) uses the CPU-side view where each increased level is lower
+performing. In contrast, the memory cache level is centric to the last
+level memory, so the higher numbered cache level corresponds to  memory
+nearer to the CPU, and further from far memory.
+
+The memory-side caches are not directly addressable by software. When
+software accesses a system address, the system will return it from the
+near memory cache if it is present. If it is not present, the system
+accesses the next level of memory until there is either a hit in that
+cache level, or it reaches far memory.
+
+An application does not need to know about caching attributes in order
+to use the system. Software may optionally query the memory cache
+attributes in order to maximize the performance out of such a setup.
+If the system provides a way for the kernel to discover this information,
+for example with ACPI HMAT (Heterogeneous Memory Attribute Table),
+the kernel will append these attributes to the NUMA node memory target.
+
+When the kernel first registers a memory cache with a node, the kernel
+will create the following directory::
+
+	/sys/devices/system/node/nodeX/memory_side_cache/
+
+If that directory is not present, the system either does not not provide
+a memory-side cache, or that information is not accessible to the kernel.
+
+The attributes for each level of cache is provided under its cache
+level index::
+
+	/sys/devices/system/node/nodeX/memory_side_cache/indexA/
+	/sys/devices/system/node/nodeX/memory_side_cache/indexB/
+	/sys/devices/system/node/nodeX/memory_side_cache/indexC/
+
+Each cache level's directory provides its attributes. For example, the
+following shows a single cache level and the attributes available for
+software to query::
+
+	# tree sys/devices/system/node/node0/memory_side_cache/
+	/sys/devices/system/node/node0/memory_side_cache/
+	|-- index1
+	|   |-- indexing
+	|   |-- line_size
+	|   |-- size
+	|   `-- write_policy
+
+The "indexing" will be 0 if it is a direct-mapped cache, and non-zero
+for any other indexed based, multi-way associativity.
+
+The "line_size" is the number of bytes accessed from the next cache
+level on a miss.
+
+The "size" is the number of bytes provided by this cache level.
+
+The "write_policy" will be 0 for write-back, and non-zero for
+write-through caching.
+
+========
+See Also
+========
+.. [1] https://www.uefi.org/sites/default/files/resources/ACPI_6_2.pdf
+       Section 5.2.27