[v2,00/15] cxl: Address translation support, part 2: Generic support and AMD Zen5 platform enablement

Message

Robert Richter Feb. 18, 2025, 1:23 p.m. UTC

This patch set adds support of address translation and enables this
for AMD Zen5 platforms. This is a new appoach in response to an
earlier attempt to implement CXL address translation [1] and the
comments on it, esp. Dan's [2]. Dan suggested to solve this by walking
the port hierarchy from the host port to the host bridge. When
crossing memory domains from one port to the other, HPA translations
are applied using a callback function to handle platform specifics.

The CXL driver currently does not implement address translation which
assumes the host physical addresses (HPA) and system physical
addresses (SPA) are equal.

Systems with different HPA and SPA addresses need address translation.
If this is the case, the hardware addresses esp. used in the HDM
decoder configurations are different to the system's or parent port
address ranges. E.g. AMD Zen5 systems may be configured to use
'Normalized addresses'. Then, CXL endpoints have their own physical
address base which is not the same as the SPA used by the CXL host
bridge. Thus, addresses need to be translated from the endpoint's to
its CXL host bridge's address range.

To enable address translation, the endpoint's HPA range must be
translated to each of the parent port's address ranges up to the root
decoder. This is implemented by traversing the decoder and port
hierarchy from the endpoint up to the root port and applying platform
specific translation functions to determine the next HPA range of the
parent port where needed:

  if (cxl_port->to_hpa)
    hpa = cxl_port->to_hpa(cxl_decoder, hpa)

A callback is introduced to translate an HPA range from a port to its
parent. Not all ports in the hierarchy must implement this function,
e.g. on Zen5 only the first root or switch port that connects the
endpoints require address translation.

The root port's HPA range is equivalent to the system's SPA range and
can then be used to find an endpoint's root port and region.  

Also, translated HPA ranges must be used to calculate the endpoint
position in the region.

Once the region was found, the decoders of all ports between the
endpoint and the root port need to be found based on the translated
HPA. Configuration checks and interleaving setup must be modified as
necessary to support address translation.

Note that only auto-discovery of decoders is supported. Thus, decoders
are locked and cannot be configured manually.

Finally, Zen5 address translation is enabled using ACPI PRMT.

This series bases on:

 [PATCH v3 00/18] cxl: Address translation support, part 1: Cleanups and refactoring

Purpose of patches:
 * Patches #1-#2: Introduction of address translation callback,
 * Patches #3-#12: Functional changes for address
   translation (common code).
 * #13: Architectural platform setup
 * Patch #15, #15: AMD Zen5 address translation.

V2:
 * rebased onto cxl/next,
 * split of v1 in two parts:
   * removed cleanups and updates from this series to post them as a
     separate series (Dave),
   * this part 2 applies on top of part 1, v3,
 * added tags to SOB chain,
 * reworked architecture, vendor and platform setup (Jonathan):
   * added patch "cxl/x86: Prepare for architectural platform setup",
   * added function arch_cxl_port_platform_setup() plus a __weak
     versions for archs other than x86,
   * moved code to core/x86,
 * added comment to cxl_to_hpa_fn (Ben),
 * updated year in copyright statement (Ben),
 * cxl_port_calc_hpa(): Removed HPA check for zero (Jonathan), return
   1 if modified,
 * cxl_port_calc_pos(): Updated description and wording (Ben),
 * added sereral patches around interleaving and SPA calculation in
   cxl_endpoint_decoder_initialize(),
 * reworked iterator in cxl_endpoint_decoder_initialize() (Gregory),
 * fixed region interleaving parameters() (Alison),
 * fixed check in cxl_region_attach() (Alison),
 * Clarified in coverletter that not all ports in a system must
   implement the to_hpa() callback (Terry).

[1] https://lore.kernel.org/linux-cxl/[e-mail]/
[2] https://lore.kernel.org/linux-cxl/[e-mail]/

Robert Richter (15):
  cxl: Modify address translation callback for generic use
  cxl: Introduce callback to translate an HPA range from a port to its
    parent
  cxl/region: Factor out code for interleaving calculations
  cxl/region: Calculate endpoint's region position during init
  cxl/region: Calculate and store the SPA range of an endpoint
  cxl/region: Use endpoint's HPA range to find the port's decoder
  cxl/region: Use translated HPA ranges to find the port's decoder
  cxl/region: Use the endpoint's SPA range to find a region
  cxl/region: Use the endpoint's SPA range to create a region
  cxl/region: Use root decoders interleaving parameters to create a
    region
  cxl/region: Use the endpoint's SPA range to check a region
  cxl/region: Lock decoders that need address translation
  cxl/x86: Prepare for architectural platform setup
  cxl/amd: Enable Zen5 address translation using ACPI PRMT
  MAINTAINERS: CXL: Add entry for AMD platform support (CXL_AMD)

 MAINTAINERS                   |   7 +
 drivers/cxl/Kconfig           |   4 +
 drivers/cxl/acpi.c            |   4 +-
 drivers/cxl/core/Makefile     |   3 +
 drivers/cxl/core/core.h       |   4 +
 drivers/cxl/core/port.c       |   4 +
 drivers/cxl/core/region.c     | 323 ++++++++++++++++++++++++++++------
 drivers/cxl/core/x86/amd.c    | 259 +++++++++++++++++++++++++++
 drivers/cxl/core/x86/common.c |  14 ++
 drivers/cxl/cxl.h             |  19 +-
 10 files changed, 583 insertions(+), 58 deletions(-)
 create mode 100644 drivers/cxl/core/x86/amd.c
 create mode 100644 drivers/cxl/core/x86/common.c


base-commit: ed50d9abb9177ba106f47e2c48e1bf1804a6956c

Comments

Gregory Price Feb. 20, 2025, 1 a.m. UTC | #1

On Tue, Feb 18, 2025 at 02:23:41PM +0100, Robert Richter wrote:
> This patch set adds support of address translation and enables this
> for AMD Zen5 platforms. This is a new appoach in response to an
> earlier attempt to implement CXL address translation [1] and the
> comments on it, esp. Dan's [2]. Dan suggested to solve this by walking
> the port hierarchy from the host port to the host bridge. When
> crossing memory domains from one port to the other, HPA translations
> are applied using a callback function to handle platform specifics.
> 
> This series bases on:
> 
>  [PATCH v3 00/18] cxl: Address translation support, part 1: Cleanups and refactoring
> 
> Purpose of patches:
>  * Patches #1-#2: Introduction of address translation callback,
>  * Patches #3-#12: Functional changes for address
>    translation (common code).
>  * #13: Architectural platform setup
>  * Patch #15, #15: AMD Zen5 address translation.
> 
> [1] https://lore.kernel.org/linux-cxl/[e-mail]/
> [2] https://lore.kernel.org/linux-cxl/[e-mail]/
> 

With the one build fix i've reported, I have tested this with Part 1 on
a Zen5 system w/ the PRM functionality.

Will review patches individually, but for the set:

Tested-by: Gregory Price <[e-mail]>