[V9,XRT,Alveo,01/14] Documentation: fpga: Add a document describing XRT Alveo drivers

Commit Message

Lizhi Hou Aug. 2, 2021, 4:05 p.m. UTC

Describe XRT driver architecture and provide basic overview of
Xilinx Alveo platform.

Signed-off-by: Sonal Santan <[e-mail]>
Signed-off-by: Max Zhen <[e-mail]>
Signed-off-by: Lizhi Hou <[e-mail]>
Reviewed-by: Tom Rix <[e-mail]>
---
 Documentation/fpga/index.rst |   1 +
 Documentation/fpga/xrt.rst   | 870 +++++++++++++++++++++++++++++++++++
 MAINTAINERS                  |  11 +
 3 files changed, 882 insertions(+)
 create mode 100644 Documentation/fpga/xrt.rst

Comments

Xu Yilun Oct. 12, 2021, 5:09 a.m. UTC | #1

On Mon, Aug 02, 2021 at 09:05:08AM -0700, Lizhi Hou wrote:
> Describe XRT driver architecture and provide basic overview of
> Xilinx Alveo platform.
> 
> Signed-off-by: Sonal Santan <[e-mail]>
> Signed-off-by: Max Zhen <[e-mail]>
> Signed-off-by: Lizhi Hou <[e-mail]>
> Reviewed-by: Tom Rix <[e-mail]>
> ---
>  Documentation/fpga/index.rst |   1 +
>  Documentation/fpga/xrt.rst   | 870 +++++++++++++++++++++++++++++++++++
>  MAINTAINERS                  |  11 +
>  3 files changed, 882 insertions(+)
>  create mode 100644 Documentation/fpga/xrt.rst
> 
> diff --git a/Documentation/fpga/index.rst b/Documentation/fpga/index.rst
> index f80f95667ca2..30134357b70d 100644
> --- a/Documentation/fpga/index.rst
> +++ b/Documentation/fpga/index.rst
> @@ -8,6 +8,7 @@ fpga
>      :maxdepth: 1
>  
>      dfl
> +    xrt
>  
>  .. only::  subproject and html
>  
> diff --git a/Documentation/fpga/xrt.rst b/Documentation/fpga/xrt.rst
> new file mode 100644
> index 000000000000..84eb41be9ac1
> --- /dev/null
> +++ b/Documentation/fpga/xrt.rst
> @@ -0,0 +1,870 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==================================
> +XRTV2 Linux Kernel Driver Overview
> +==================================
> +
> +Authors:
> +
> +* Sonal Santan <[e-mail]>
> +* Max Zhen <[e-mail]>
> +* Lizhi Hou <[e-mail]>
> +
> +XRTV2 drivers are second generation `XRT <https://github.com/Xilinx/XRT>`_
> +drivers which support `Alveo <https://www.xilinx.com/products/boards-and-kits/alveo.html>`_
> +PCIe platforms from Xilinx.
> +
> +XRTV2 drivers support *subsystem* style data driven platforms where driver's
> +configuration and behavior are determined by metadata provided by the platform
> +(in *device tree* format). Primary management physical function (MPF) driver
> +is called **xrt-mgmt**. Primary user physical function (UPF) driver is called
> +**xrt-user** and is under development. xrt_driver framework and HW subsystem
> +drivers are packaged into a library module called **xrt-lib**, which is shared
> +by **xrt-mgmt** and **xrt-user** (under development). The xrt_driver framework
> +implements a ``bus_type`` called **xrt_bus_type** which is used to discover HW
> +subsystems and facilitate inter HW subsystem interaction.
> +
> +Driver Modules
> +==============
> +
> +xrt-lib.ko
> +----------
> +
> +xrt-lib is the repository of all subsystem drivers and pure software modules that
> +can potentially be shared between xrt-mgmt and xrt-user. All these drivers are
> +structured as **xrt_driver** and are instantiated by xrt-mgmt (or xrt-user under
> +development) based on the metadata associated with the hardware. The metadata is
> +in the form of a device tree as mentioned before. Each xrt_driver statically
> +defines a subsystem node array by using a node name or a string in its ``.endpoints``
> +property. And this array is eventually translated to IOMEM resources in the
> +instantiated **xrt_device**.
> +
> +The xrt-lib infrastructure provides hooks to xrt_drivers for device node
> +management, user file operations and ioctl callbacks. The core infrastructure also
> +provides a bus functionality called **xrt_bus_type** for xrt_driver registration,
> +discovery and inter xrt_driver calls. xrt-lib does not have any dependency on PCIe
> +subsystem.
> +
> +.. note::
> +   See code in ``include/xleaf.h`` and ``include/xdevice.h``
> +
> +
> +xrt-mgmt.ko
> +------------
> +
> +The xrt-mgmt driver is a PCIe device driver driving MPF found on Xilinx's Alveo
> +PCIe device. It consists of one *root* driver, one or more *group* drivers
> +and one or more *xleaf* drivers. The group and xleaf drivers are instantiations
> +of the xrt_driver but are called group and xleaf to symbolize the logical operation
> +performed by them.
> +
> +The root driver manages the life cycle of multiple group drivers, which, in turn,
> +manages multiple xleaf drivers. This flexibility allows xrt-mgmt.ko and xrt-lib.ko
> +to support various HW subsystems exposed by different Alveo shells. The differences
> +among these Alveo shells is handled in xleaf drivers. The root and group
> +drivers are part of the infrastructure which provide common services to xleaf
> +drivers found on various Alveo shells. See :ref:`alveo_platform_overview`.
> +
> +The instantiation of specific group driver or xleaf drivers is completely data
> +driven based on metadata (mostly in device tree format) found through VSEC
> +capability and inside the firmware files, such as platform xsabin or user xclbin
> +file.
> +
> +
> +Driver Object Model
> +===================
> +
> +The driver object model looks like the following::
> +
> +                    +-----------+
> +                    |   xroot   |
> +                    +-----+-----+
> +                          |
> +              +-----------+-----------+
> +              |                       |
> +              v                       v
> +        +-----------+          +-----------+
> +        |   group   |    ...   |   group   |
> +        +-----+-----+          +------+----+
> +              |                       |
> +              |                       |
> +        +-----+----+            +-----+----+
> +        |          |            |          |
> +        v          v            v          v
> +    +-------+  +-------+    +-------+  +-------+
> +    | xleaf |..| xleaf |    | xleaf |..| xleaf |
> +    +-------+  +-------+    +-------+  +-------+
> +
> +As an example, for Xilinx Alveo U50 before user xclbin download, the tree
> +looks like the following::
> +
> +                                +-----------+
> +                                |  xrt-mgmt |
> +                                +-----+-----+
> +                                      |
> +            +-------------------------+--------------------+
> +            |                         |                    |
> +            v                         v                    v
> +       +--------+                +--------+            +--------+
> +       | group0 |                | group1 |            | group2 |
> +       +----+---+                +----+---+            +---+----+
> +            |                         |                    |
> +            |                         |                    |
> +      +-----+-----+        +----+-----+---+    +-----+-----+----+--------+
> +      |           |        |    |         |    |     |          |        |
> +      v           v        |    v         v    |     v          v        |
> + +------------+  +------+  | +------+ +------+ |  +------+ +-----------+ |
> + | xmgmt_main |  | VSEC |  | | GPIO | | QSPI | |  |  CMC | | AXI-GATE0 | |
> + +------------+  +------+  | +------+ +------+ |  +------+ +-----------+ |
> +                           | +---------+       |  +------+ +-----------+ |
> +                           +>| MAILBOX |       +->| ICAP | | AXI-GATE1 |<+
> +                             +---------+       |  +------+ +-----------+
> +                                               |  +-------+
> +                                               +->| CALIB |
> +                                                  +-------+
> +
> +After a xclbin is downloaded, group3 will be added and the tree looks like the
> +following::
> +
> +                                +-----------+
> +                                |  xrt-mgmt |
> +                                +-----+-----+
> +                                      |
> +            +-------------------------+--------------------+-----------------+
> +            |                         |                    |                 |
> +            v                         v                    v                 |
> +       +--------+                +--------+            +--------+            |
> +       | group0 |                | group1 |            | group2 |            |
> +       +----+---+                +----+---+            +---+----+            |
> +            |                         |                    |                 |
> +            |                         |                    |                 |
> +      +-----+-----+       +-----+-----+---+    +-----+-----+----+--------+   |
> +      |           |       |     |         |    |     |          |        |   |
> +      v           v       |     v         v    |     v          v        |   |
> + +------------+  +------+ | +------+ +------+  |  +------+ +-----------+ |   |
> + | xmgmt_main |  | VSEC | | | GPIO | | QSPI |  |  |  CMC | | AXI-GATE0 | |   |
> + +------------+  +------+ | +------+ +------+  |  +------+ +-----------+ |   |
> +                          | +---------+        |  +------+ +-----------+ |   |
> +                          +>| MAILBOX |        +->| ICAP | | AXI-GATE1 |<+   |
> +                            +---------+        |  +------+ +-----------+     |
> +                                               |  +-------+                  |
> +                                               +->| CALIB |                  |
> +                                                  +-------+                  |
> +                      +---+----+                                             |
> +                      | group3 |<--------------------------------------------+
> +                      +--------+
> +                          |
> +                          |
> +     +-------+--------+---+--+--------+------+-------+
> +     |       |        |      |        |      |       |
> +     v       |        v      |        v      |       v
> + +--------+  |   +--------+  |   +--------+  |    +-----+
> + | CLOCK0 |  |   | CLOCK1 |  |   | CLOCK2 |  |    | UCS |
> + +--------+  v   +--------+  v   +--------+  v    +-----+
> + +-------------+ +-------------+ +-------------+
> + | CLOCK-FREQ0 | | CLOCK-FREQ1 | | CLOCK-FREQ2 |
> + +-------------+ +-------------+ +-------------+
> +
> +
> +root
> +----
> +
> +The root driver is a PCIe device driver attached to MPF. It's part of the
> +infrastructure of the MPF driver and resides in xrt-mgmt.ko. This driver
> +
> +* manages one or more group drivers
> +* provides access to functionalities that requires pci_dev, such as PCIE config
> +  space access, to other xleaf drivers through root calls
> +* facilities inter xleaf driver calls for other xleaf drivers
> +* facilities event callbacks for other xleaf drivers
> +
> +When the root driver starts, it will explicitly create an initial group instance,
> +which contains xleaf drivers that will trigger the creation of other group
> +instances. The root driver will wait for all group and xleaf drivers to be
> +created before it returns from its probe routine and claim success of the
> +initialization of the entire xrt-mgmt driver. If any xleaf fails to initialize
> +the xrt-mgmt driver will still come online but with limited functionality.
> +
> +.. note::
> +   See code in ``lib/xroot.c`` and ``mgmt/root.c``
> +
> +
> +group
> +-----
> +
> +The group driver represents a pseudo device whose life cycle is managed by
> +root and does not have real IO mem or IRQ resources. It's part of the
> +infrastructure of the MPF driver and resides in xrt-lib.ko. This driver
> +
> +* manages one or more xleaf drivers
> +* provides access to root from xleaf drivers, so that root calls, event
> +  notifications and inter xleaf calls can happen
> +
> +In xrt-mgmt, an initial group driver instance will be created by the root. This
> +instance contains xleaf drivers that will trigger group instances to be created
> +to manage groups of xleaf drivers found on different partitions of hardware,
> +such as VSEC, Shell, and User.
> +
> +Every *fpga_region* has a group driver associated with it. The group driver is
> +created when a xclbin image is loaded on the fpga_region. The existing group
> +is destroyed when a new xclbin image is loaded. The fpga_region persists
> +across xclbin downloads.
> +
> +.. note::
> +   See code in ``lib/group.c``
> +
> +
> +xleaf
> +-----
> +
> +The xleaf driver is a xrt_driver whose life cycle is managed by
> +a group driver and may or may not have real IO mem or IRQ resources. They
> +manage HW subsystems they are attached to.
> +
> +A xleaf driver without real hardware resources manages in-memory states for
> +xrt-mgmt. These states are shareable by other xleaf drivers.
> +
> +Xleaf drivers assigned to specific hardware resources drive a specific subsystem
> +in the device. To manipulate the subsystem or carry out a task, a xleaf driver
> +may ask for help from the root via root calls and/or from other leaves via
> +inter xleaf calls.
> +
> +A xleaf can also broadcast events through infrastructure code for other leaves
> +to process. It can also receive event notification from infrastructure about
> +certain events, such as post-creation or pre-exit of a particular xleaf.
> +
> +.. note::
> +   See code in ``lib/xleaf/*.c``
> +
> +
> +xrt_bus_type
> +------------
> +
> +xrt_bus_type defines a virtual bus which handles xrt_driver probe, remove and match
> +operations. All xrt_drivers register with xrt_bus_type as part of xrt-lib driver
> +``module_init`` and un-register as part of xrt-lib driver ``module_exit``.
> +
> +.. note::
> +   See code in ``lib/lib-drv.c``
> +
> +FPGA Manager Interaction
> +========================
> +
> +fpga_manager
> +------------
> +
> +An instance of fpga_manager is created by xmgmt_main and is used for xclbin
> +image download. fpga_manager requires the full xclbin image before it can
> +start programming the FPGA configuration engine via Internal Configuration
> +Access Port (ICAP) xrt_driver.
> +
> +fpga_region
> +-----------
> +
> +For every interface exposed by the currently loaded xclbin/xsabin in the
> +*parent* fpga_region a new instance of fpga_region is created like a *child*
> +fpga_region. The device tree of the *parent* fpga_region defines the
> +resources for a new instance of fpga_bridge which isolates the parent from
> +child fpga_region. This new instance of fpga_bridge will be used when a
> +xclbin image is loaded on the child fpga_region. After the xclbin image is
> +downloaded to the fpga_region, an instance of a group is created for the
> +fpga_region using the device tree obtained as part of the xclbin. If this
> +device tree defines any child interfaces, it can trigger the creation of
> +fpga_bridge and fpga_region for the next region in the chain.
> +
> +fpga_bridge
> +-----------
> +
> +Like the fpga_region, an fpga_bridge is created by walking the device tree
> +of the parent group. The bridge is used for isolation between a parent and
> +its child.
> +
> +Driver Interfaces
> +=================
> +
> +xrt-mgmt Driver Ioctls
> +----------------------
> +
> +Ioctls exposed by the xrt-mgmt driver to user space are enumerated in the
> +following table:
> +
> +== ===================== ============================ ==========================
> +#  Functionality         ioctl request code            data format
> +== ===================== ============================ ==========================
> +1  FPGA image download   XMGMT_IOCICAPDOWNLOAD_AXLF    xmgmt_ioc_bitstream_axlf
> +== ===================== ============================ ==========================
> +
> +A user xclbin can be downloaded by using the xbmgmt tool from the XRT open source
> +suite. See example usage below::
> +
> +  xbmgmt partition --program --path /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/test/verify.xclbin --force
> +
> +xrt-mgmt Driver Sysfs
> +----------------------
> +
> +The xrt-mgmt driver exposes a rich set of sysfs interfaces. Subsystem xrt
> +drivers export sysfs node for every platform instance.
> +
> +Every partition also exports its UUIDs. See below for examples::
> +
> +  /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
> +  /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
> +
> +
> +hwmon
> +-----
> +
> +The xrt-mgmt driver exposes standard hwmon interface to report voltage, current,
> +temperature, power, etc. These can easily be viewed using *sensors* command line
> +utility.
> +
> +.. _alveo_platform_overview:
> +
> +Alveo Platform Overview
> +=======================
> +
> +Alveo platforms are architected as two physical FPGA partitions: *Shell* and
> +*User*. The Shell provides basic infrastructure for the Alveo platform like
> +PCIe connectivity, board management, Dynamic Function Exchange (DFX), sensors,
> +clocking, reset, and security. DFX, partial reconfiguration, is responsible for
> +loading the user compiled FPGA binary.
> +
> +For DFX to work properly, physical partitions require strict HW compatibility
> +with each other. Every physical partition has two interface UUIDs: the *parent*
> +UUID and the *child* UUID. For simple single stage platforms, Shell → User forms
> +the parent child relationship.
> +
> +.. note::
> +   Partition compatibility matching is a key design component of the Alveo platforms
> +   and XRT. Partitions have child and parent relationship. A loaded partition
> +   exposes child partition UUID to advertise its compatibility requirement. When
> +   loading a child partition, the xrt-mgmt driver matches the parent
> +   UUID of the child partition against the child UUID exported by the parent.
> +   The parent and child partition UUIDs are stored in the *xclbin* (for the user)
> +   and the *xsabin* (for the shell). Except for the root UUID exported by VSEC,
> +   the hardware itself does not know about the UUIDs. The UUIDs are stored in
> +   xsabin and xclbin. The image format has a special node called Partition UUIDs
> +   which define the compatibility UUIDs. See :ref:`partition_uuids`.
> +
> +
> +The physical partitions and their loading are illustrated below::
> +
> +           SHELL                               USER
> +        +-----------+                  +-------------------+
> +        |           |                  |                   |
> +        | VSEC UUID | CHILD     PARENT |    LOGIC UUID     |
> +        |           o------->|<--------o                   |
> +        |           | UUID       UUID  |                   |
> +        +-----+-----+                  +--------+----------+
> +              |                                 |
> +              .                                 .
> +              |                                 |
> +          +---+---+                      +------+--------+
> +          |  POR  |                      | USER COMPILED |
> +          | FLASH |                      |    XCLBIN     |
> +          +-------+                      +---------------+
> +
> +
> +Loading Sequence
> +----------------
> +
> +The Shell partition is loaded from flash at system boot time. It establishes the
> +PCIe link and exposes two physical functions to the BIOS. After the OS boots,
> +the xrt-mgmt driver attaches to the PCIe physical function 0 exposed by the Shell
> +and then looks for VSEC in the PCIe extended configuration space. Using VSEC, it
> +determines the logic UUID of the Shell and uses the UUID to load matching *xsabin*
> +file from Linux firmware directory. The xsabin file contains the metadata to
> +discover the peripherals that are part of the Shell and the firmware for any
> +embedded soft processors in the Shell. The xsabin file also contains Partition
> +UUIDs as described here :ref:`partition_uuids`.
> +
> +The Shell exports a child interface UUID which is used for the compatibility
> +check when loading the user compiled xclbin over the User partition as part of DFX.
> +When a user requests loading of a specific xclbin, the xrt-mgmt driver reads
> +the parent interface UUID specified in the xclbin and matches it with the child
> +interface UUID exported by the Shell to determine if the xclbin is compatible with
> +the Shell. If the match fails, loading of xclbin is denied.
> +
> +xclbin loading is requested using the ICAP_DOWNLOAD_AXLF ioctl command. When loading
> +a xclbin, the xrt-mgmt driver performs the following *logical* operations:
> +
> +1. Copy xclbin from user to kernel memory
> +2. Sanity check the xclbin contents
> +3. Isolate the User partition
> +4. Download the bitstream using the FPGA config engine (ICAP)
> +5. De-isolate the User partition
> +6. Program the clocks (ClockWiz) driving the User partition
> +7. Wait for the memory controller (MIG) calibration
> +8. Return the loading status back to the caller
> +
> +`Platform Loading Overview <https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`_
> +provides more detailed information on platform loading.
> +
> +
> +xsabin
> +------
> +
> +Each Alveo platform comes packaged with its own xsabin. The xsabin is a trusted
> +component of the platform. For format details refer to :ref:`xsabin_xclbin_container_format`
> +below. xsabin contains basic information like UUIDs, platform name and metadata in the
> +form of device tree. See :ref:`device_tree_usage` below for details and example.
> +
> +xclbin
> +------
> +
> +xclbin is compiled by end user using
> +`Vitis <https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html>`_
> +tool set from Xilinx. The xclbin contains sections describing user compiled
> +acceleration engines/kernels, memory subsystems, clocking information etc. It also
> +contains an FPGA bitstream for the user partition, UUIDs, platform name, etc.
> +
> +
> +.. _xsabin_xclbin_container_format:
> +
> +xsabin/xclbin Container Format
> +------------------------------
> +
> +xclbin/xsabin is ELF-like binary container format. It is structured as series of
> +sections. There is a file header followed by several section headers which is
> +followed by sections. A section header points to an actual section. There is an
> +optional signature at the end. The format is defined by the header file ``xclbin.h``.
> +The following figure illustrates a typical xclbin::
> +
> +
> +           +---------------------+
> +           |                     |
> +           |       HEADER        |
> +           +---------------------+
> +           |   SECTION  HEADER   |
> +           |                     |
> +           +---------------------+
> +           |         ...         |
> +           |                     |
> +           +---------------------+
> +           |   SECTION  HEADER   |
> +           |                     |
> +           +---------------------+
> +           |       SECTION       |
> +           |                     |
> +           +---------------------+
> +           |         ...         |
> +           |                     |
> +           +---------------------+
> +           |       SECTION       |
> +           |                     |
> +           +---------------------+
> +           |      SIGNATURE      |
> +           |      (OPTIONAL)     |
> +           +---------------------+
> +
> +
> +xclbin/xsabin files can be packaged, un-packaged and inspected using an XRT
> +utility called **xclbinutil**. xclbinutil is part of the XRT open source
> +software stack. The source code for xclbinutil can be found at
> +https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbinutil
> +
> +For example, to enumerate the contents of a xclbin/xsabin use the *--info* switch
> +as shown below::
> +
> +
> +  xclbinutil --info --input /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
> +  xclbinutil --info --input /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsabin
> +
> +
> +.. _device_tree_usage:
> +
> +Device Tree Usage
> +-----------------
> +
> +The xsabin file stores metadata which advertise HW subsystems present in a
> +partition. The metadata is stored in device tree format with a well defined
> +schema. XRT management driver uses this information to bind *xrt_drivers* to
> +the subsystem instantiations. The xrt_drivers are found in **xrt-lib.ko** kernel
> +module.

I'm still catching up the patchset from the very beginning, and just
finished the Documentation part. So far, I see the DT usage concern
which may impact the architecure a lot, so I should raise it ASAP.

The concern raised by the DT maintainer:
https://lore.kernel.org/linux-fpga/[e-mail]/

First of all, directly parsing FDT in device drivers is not a normal usage of DT
in linux. It is out of the current DT usage model. So it should be agreed by DT
maintainers.

Current FPGA framework modifies kernel's live tree by DT overlay, when FPGA is
dynamically reprogrammed and new HW devices appear. See
Documentation/devicetree/bindings/fpga/fpga-region.txt.

Then something less important:

  1. The bindings should be documented in Documentation/devicetree/bindings/.
  2. Are all the example DT usage conform to the exsiting bindings? I
     didn't go through all device classes, but remember like the
     interrupt-controller should have a "interrupt-controller" property, and
     the PCI properties are also different from PCI bindings.

Thanks,
Yilun

> +
> +Logic UUID
> +^^^^^^^^^^
> +A partition is identified uniquely through ``logic_uuid`` property::
> +
> +  /dts-v1/;
> +  / {
> +      logic_uuid = "0123456789abcdef0123456789abcdef";
> +      ...
> +    }
> +
> +Schema Version
> +^^^^^^^^^^^^^^
> +Schema version is defined through the ``schema_version`` node. It contains
> +``major`` and ``minor`` properties as below::
> +
> +  /dts-v1/;
> +  / {
> +       schema_version {
> +           major = <0x01>;
> +           minor = <0x00>;
> +       };
> +       ...
> +    }
> +
> +.. _partition_uuids:
> +
> +Partition UUIDs
> +^^^^^^^^^^^^^^^
> +Each partition may have parent and child UUIDs. These UUIDs are
> +defined by ``interfaces`` node and ``interface_uuid`` property::
> +
> +  /dts-v1/;
> +  / {
> +       interfaces {
> +           @0 {
> +                  interface_uuid = "0123456789abcdef0123456789abcdef";
> +           };
> +           @1 {
> +                  interface_uuid = "fedcba9876543210fedcba9876543210";
> +           };
> +           ...
> +        };
> +       ...
> +    }
> +
> +
> +Subsystem Instantiations
> +^^^^^^^^^^^^^^^^^^^^^^^^
> +Subsystem instantiations are captured as children of ``addressable_endpoints``
> +node::
> +
> +  /dts-v1/;
> +  / {
> +       addressable_endpoints {
> +           abc {
> +               ...
> +           };
> +           def {
> +               ...
> +           };
> +           ...
> +       }
> +  }
> +
> +Subnode 'abc' and 'def' are the name of subsystem nodes
> +
> +Subsystem Node
> +^^^^^^^^^^^^^^
> +Each subsystem node and its properties define a hardware instance::
> +
> +
> +  addressable_endpoints {
> +      abc {
> +          reg = <0x00 0x1f05000 0x00 0x1000>>
> +          pcie_physical_function = <0x0>;
> +          pcie_bar_mapping = <0x2>;
> +          compatible = "abc def";
> +          interrupts = <0x09 0x0c>;
> +          firmware {
> +              firmware_product_name = "abc"
> +              firmware_branch_name = "def"
> +              firmware_version_major = <1>
> +              firmware_version_minor = <2>
> +          };
> +      }
> +      ...
> +  }
> +
> +:reg:
> + Property defines an address range. `<0x00 0x1f05000 0x00 0x1000>` indicates
> + *0x00 0x1f05000* as BAR offset and *0x00 0x1000* as address length.
> +:pcie_physical_function:
> + Property specifies which PCIe physical function the subsystem node resides.
> + `<0x0>` implies physical function 0.
> +:pcie_bar_mapping:
> + Property specifies which PCIe BAR the subsystem node resides. `<0x2>` implies
> + BAR 2. A value of 0 means the property is not defined.
> +:compatible:
> + Property is a list of strings. The first string in the list specifies the exact
> + subsystem node. The following strings represent other devices that the device
> + is compatible with.
> +:interrupts:
> + Property specifies start and end interrupts for this subsystem node.
> + `<0x09 0x0c>` implies interrupts 9 to 13 are used by this subsystem.
> +:firmware:
> + Subnode defines the firmware required by this subsystem node.
> +
> +Alveo U50 Platform Example
> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> +::
> +
> +  /dts-v1/;
> +
> +  /{
> +        logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
> +
> +        schema_version {
> +                major = <0x01>;
> +                minor = <0x00>;
> +        };
> +
> +        interfaces {
> +
> +                @0 {
> +                        interface_uuid = "862c7020a250293e32036f19956669e5";
> +                };
> +        };
> +
> +        addressable_endpoints {
> +
> +                ep_blp_rom_00 {
> +                        reg = <0x00 0x1f04000 0x00 0x1000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> +                };
> +
> +                ep_card_flash_program_00 {
> +                        reg = <0x00 0x1f06000 0x00 0x1000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_quad_spi-1.0\0axi_quad_spi";
> +                        interrupts = <0x03 0x03>;
> +                };
> +
> +                ep_cmc_firmware_mem_00 {
> +                        reg = <0x00 0x1e20000 0x00 0x20000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> +
> +                        firmware {
> +                                firmware_product_name = "cmc";
> +                                firmware_branch_name = "u50";
> +                                firmware_version_major = <0x01>;
> +                                firmware_version_minor = <0x00>;
> +                        };
> +                };
> +
> +                ep_cmc_intc_00 {
> +                        reg = <0x00 0x1e03000 0x00 0x1000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> +                        interrupts = <0x04 0x04>;
> +                };
> +
> +                ep_cmc_mutex_00 {
> +                        reg = <0x00 0x1e02000 0x00 0x1000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> +                };
> +
> +                ep_cmc_regmap_00 {
> +                        reg = <0x00 0x1e08000 0x00 0x2000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> +
> +                        firmware {
> +                                firmware_product_name = "sc-fw";
> +                                firmware_branch_name = "u50";
> +                                firmware_version_major = <0x05>;
> +                        };
> +                };
> +
> +                ep_cmc_reset_00 {
> +                        reg = <0x00 0x1e01000 0x00 0x1000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> +                };
> +
> +                ep_ddr_mem_calib_00 {
> +                        reg = <0x00 0x63000 0x00 0x1000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> +                };
> +
> +                ep_debug_bscan_mgmt_00 {
> +                        reg = <0x00 0x1e90000 0x00 0x10000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-debug_bridge-1.0\0debug_bridge";
> +                };
> +
> +                ep_ert_base_address_00 {
> +                        reg = <0x00 0x21000 0x00 0x1000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> +                };
> +
> +                ep_ert_command_queue_mgmt_00 {
> +                        reg = <0x00 0x40000 0x00 0x10000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
> +                };
> +
> +                ep_ert_command_queue_user_00 {
> +                        reg = <0x00 0x40000 0x00 0x10000>;
> +                        pcie_physical_function = <0x01>;
> +                        compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
> +                };
> +
> +                ep_ert_firmware_mem_00 {
> +                        reg = <0x00 0x30000 0x00 0x8000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> +
> +                        firmware {
> +                                firmware_product_name = "ert";
> +                                firmware_branch_name = "v20";
> +                                firmware_version_major = <0x01>;
> +                        };
> +                };
> +
> +                ep_ert_intc_00 {
> +                        reg = <0x00 0x23000 0x00 0x1000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> +                        interrupts = <0x05 0x05>;
> +                };
> +
> +                ep_ert_reset_00 {
> +                        reg = <0x00 0x22000 0x00 0x1000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> +                };
> +
> +                ep_ert_sched_00 {
> +                        reg = <0x00 0x50000 0x00 0x1000>;
> +                        pcie_physical_function = <0x01>;
> +                        compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
> +                        interrupts = <0x09 0x0c>;
> +                };
> +
> +                ep_fpga_configuration_00 {
> +                        reg = <0x00 0x1e88000 0x00 0x8000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
> +                        interrupts = <0x02 0x02>;
> +                };
> +
> +                ep_icap_reset_00 {
> +                        reg = <0x00 0x1f07000 0x00 0x1000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> +                };
> +
> +                ep_msix_00 {
> +                        reg = <0x00 0x00 0x00 0x20000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
> +                        pcie_bar_mapping = <0x02>;
> +                };
> +
> +                ep_pcie_link_mon_00 {
> +                        reg = <0x00 0x1f05000 0x00 0x1000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> +                };
> +
> +                ep_pr_isolate_plp_00 {
> +                        reg = <0x00 0x1f01000 0x00 0x1000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> +                };
> +
> +                ep_pr_isolate_ulp_00 {
> +                        reg = <0x00 0x1000 0x00 0x1000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> +                };
> +
> +                ep_uuid_rom_00 {
> +                        reg = <0x00 0x64000 0x00 0x1000>;
> +                        pcie_physical_function = <0x00>;
> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> +                };
> +
> +                ep_xdma_00 {
> +                        reg = <0x00 0x00 0x00 0x10000>;
> +                        pcie_physical_function = <0x01>;
> +                        compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
> +                        pcie_bar_mapping = <0x02>;
> +                };
> +        };
> +
> +  }
> +
> +
> +
> +Deployment Models
> +=================
> +
> +Baremetal
> +---------
> +
> +In bare-metal deployments, both MPF and UPF are visible and accessible. The
> +xrt-mgmt driver binds to MPF. The xrt-mgmt driver operations are privileged and
> +available to system administrator. The full stack is illustrated below::
> +
> +                            HOST
> +
> +               [XRT-MGMT]         [XRT-USER]
> +                    |                  |
> +                    |                  |
> +                 +-----+            +-----+
> +                 | MPF |            | UPF |
> +                 |     |            |     |
> +                 | PF0 |            | PF1 |
> +                 +--+--+            +--+--+
> +          ......... ^................. ^..........
> +                    |                  |
> +                    |   PCIe DEVICE    |
> +                    |                  |
> +                 +--+------------------+--+
> +                 |         SHELL          |
> +                 |                        |
> +                 +------------------------+
> +                 |         USER           |
> +                 |                        |
> +                 |                        |
> +                 |                        |
> +                 |                        |
> +                 +------------------------+
> +
> +
> +
> +Virtualized
> +-----------
> +
> +In virtualized deployments, the privileged MPF is assigned to the host but the
> +unprivileged UPF is assigned to a guest VM via PCIe pass-through. The xrt-mgmt
> +driver in host binds to MPF. The xrt-mgmt driver operations are privileged and
> +only accessible to the MPF. The full stack is illustrated below::
> +
> +
> +                                 ..............
> +                  HOST           .    VM      .
> +                                 .            .
> +               [XRT-MGMT]        . [XRT-USER] .
> +                    |            .     |      .
> +                    |            .     |      .
> +                 +-----+         .  +-----+   .
> +                 | MPF |         .  | UPF |   .
> +                 |     |         .  |     |   .
> +                 | PF0 |         .  | PF1 |   .
> +                 +--+--+         .  +--+--+   .
> +          ......... ^................. ^..........
> +                    |                  |
> +                    |   PCIe DEVICE    |
> +                    |                  |
> +                 +--+------------------+--+
> +                 |         SHELL          |
> +                 |                        |
> +                 +------------------------+
> +                 |         USER           |
> +                 |                        |
> +                 |                        |
> +                 |                        |
> +                 |                        |
> +                 +------------------------+
> +
> +
> +
> +
> +
> +Platform Security Considerations
> +================================
> +
> +`Security of Alveo Platform <https://xilinx.github.io/XRT/master/html/security.html>`_
> +discusses the deployment options and security implications in great detail.
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 056966c9aac9..beeaf0257364 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -7274,6 +7274,17 @@ F:	Documentation/fpga/
>  F:	drivers/fpga/
>  F:	include/linux/fpga/
>  
> +FPGA XRT DRIVERS
> +M:	Lizhi Hou <[e-mail]>
> +R:	Max Zhen <[e-mail]>
> +R:	Sonal Santan <[e-mail]>
> +L:	[e-mail]
> +S:	Supported
> +W:	https://github.com/Xilinx/XRT
> +F:	Documentation/fpga/xrt.rst
> +F:	drivers/fpga/xrt/
> +F:	include/uapi/linux/xrt/
> +
>  FPU EMULATOR
>  M:	Bill Metzenthen <[e-mail]>
>  S:	Maintained
> -- 
> 2.27.0

Lizhi Hou Oct. 13, 2021, 7:13 p.m. UTC | #2

On 10/11/21 10:09 PM, Xu Yilun wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>
>
> On Mon, Aug 02, 2021 at 09:05:08AM -0700, Lizhi Hou wrote:
>> Describe XRT driver architecture and provide basic overview of
>> Xilinx Alveo platform.
>>
>> Signed-off-by: Sonal Santan <[e-mail]>
>> Signed-off-by: Max Zhen <[e-mail]>
>> Signed-off-by: Lizhi Hou <[e-mail]>
>> Reviewed-by: Tom Rix <[e-mail]>
>> ---
>>   Documentation/fpga/index.rst |   1 +
>>   Documentation/fpga/xrt.rst   | 870 +++++++++++++++++++++++++++++++++++
>>   MAINTAINERS                  |  11 +
>>   3 files changed, 882 insertions(+)
>>   create mode 100644 Documentation/fpga/xrt.rst
>>
>> diff --git a/Documentation/fpga/index.rst b/Documentation/fpga/index.rst
>> index f80f95667ca2..30134357b70d 100644
>> --- a/Documentation/fpga/index.rst
>> +++ b/Documentation/fpga/index.rst
>> @@ -8,6 +8,7 @@ fpga
>>       :maxdepth: 1
>>
>>       dfl
>> +    xrt
>>
>>   .. only::  subproject and html
>>
>> diff --git a/Documentation/fpga/xrt.rst b/Documentation/fpga/xrt.rst
>> new file mode 100644
>> index 000000000000..84eb41be9ac1
>> --- /dev/null
>> +++ b/Documentation/fpga/xrt.rst
>> @@ -0,0 +1,870 @@
>> +.. SPDX-License-Identifier: GPL-2.0
>> +
>> +==================================
>> +XRTV2 Linux Kernel Driver Overview
>> +==================================
>> +
>> +Authors:
>> +
>> +* Sonal Santan <[e-mail]>
>> +* Max Zhen <[e-mail]>
>> +* Lizhi Hou <[e-mail]>
>> +
>> +XRTV2 drivers are second generation `XRT <https://github.com/Xilinx/XRT>`_
>> +drivers which support `Alveo <https://www.xilinx.com/products/boards-and-kits/alveo.html>`_
>> +PCIe platforms from Xilinx.
>> +
>> +XRTV2 drivers support *subsystem* style data driven platforms where driver's
>> +configuration and behavior are determined by metadata provided by the platform
>> +(in *device tree* format). Primary management physical function (MPF) driver
>> +is called **xrt-mgmt**. Primary user physical function (UPF) driver is called
>> +**xrt-user** and is under development. xrt_driver framework and HW subsystem
>> +drivers are packaged into a library module called **xrt-lib**, which is shared
>> +by **xrt-mgmt** and **xrt-user** (under development). The xrt_driver framework
>> +implements a ``bus_type`` called **xrt_bus_type** which is used to discover HW
>> +subsystems and facilitate inter HW subsystem interaction.
>> +
>> +Driver Modules
>> +==============
>> +
>> +xrt-lib.ko
>> +----------
>> +
>> +xrt-lib is the repository of all subsystem drivers and pure software modules that
>> +can potentially be shared between xrt-mgmt and xrt-user. All these drivers are
>> +structured as **xrt_driver** and are instantiated by xrt-mgmt (or xrt-user under
>> +development) based on the metadata associated with the hardware. The metadata is
>> +in the form of a device tree as mentioned before. Each xrt_driver statically
>> +defines a subsystem node array by using a node name or a string in its ``.endpoints``
>> +property. And this array is eventually translated to IOMEM resources in the
>> +instantiated **xrt_device**.
>> +
>> +The xrt-lib infrastructure provides hooks to xrt_drivers for device node
>> +management, user file operations and ioctl callbacks. The core infrastructure also
>> +provides a bus functionality called **xrt_bus_type** for xrt_driver registration,
>> +discovery and inter xrt_driver calls. xrt-lib does not have any dependency on PCIe
>> +subsystem.
>> +
>> +.. note::
>> +   See code in ``include/xleaf.h`` and ``include/xdevice.h``
>> +
>> +
>> +xrt-mgmt.ko
>> +------------
>> +
>> +The xrt-mgmt driver is a PCIe device driver driving MPF found on Xilinx's Alveo
>> +PCIe device. It consists of one *root* driver, one or more *group* drivers
>> +and one or more *xleaf* drivers. The group and xleaf drivers are instantiations
>> +of the xrt_driver but are called group and xleaf to symbolize the logical operation
>> +performed by them.
>> +
>> +The root driver manages the life cycle of multiple group drivers, which, in turn,
>> +manages multiple xleaf drivers. This flexibility allows xrt-mgmt.ko and xrt-lib.ko
>> +to support various HW subsystems exposed by different Alveo shells. The differences
>> +among these Alveo shells is handled in xleaf drivers. The root and group
>> +drivers are part of the infrastructure which provide common services to xleaf
>> +drivers found on various Alveo shells. See :ref:`alveo_platform_overview`.
>> +
>> +The instantiation of specific group driver or xleaf drivers is completely data
>> +driven based on metadata (mostly in device tree format) found through VSEC
>> +capability and inside the firmware files, such as platform xsabin or user xclbin
>> +file.
>> +
>> +
>> +Driver Object Model
>> +===================
>> +
>> +The driver object model looks like the following::
>> +
>> +                    +-----------+
>> +                    |   xroot   |
>> +                    +-----+-----+
>> +                          |
>> +              +-----------+-----------+
>> +              |                       |
>> +              v                       v
>> +        +-----------+          +-----------+
>> +        |   group   |    ...   |   group   |
>> +        +-----+-----+          +------+----+
>> +              |                       |
>> +              |                       |
>> +        +-----+----+            +-----+----+
>> +        |          |            |          |
>> +        v          v            v          v
>> +    +-------+  +-------+    +-------+  +-------+
>> +    | xleaf |..| xleaf |    | xleaf |..| xleaf |
>> +    +-------+  +-------+    +-------+  +-------+
>> +
>> +As an example, for Xilinx Alveo U50 before user xclbin download, the tree
>> +looks like the following::
>> +
>> +                                +-----------+
>> +                                |  xrt-mgmt |
>> +                                +-----+-----+
>> +                                      |
>> +            +-------------------------+--------------------+
>> +            |                         |                    |
>> +            v                         v                    v
>> +       +--------+                +--------+            +--------+
>> +       | group0 |                | group1 |            | group2 |
>> +       +----+---+                +----+---+            +---+----+
>> +            |                         |                    |
>> +            |                         |                    |
>> +      +-----+-----+        +----+-----+---+    +-----+-----+----+--------+
>> +      |           |        |    |         |    |     |          |        |
>> +      v           v        |    v         v    |     v          v        |
>> + +------------+  +------+  | +------+ +------+ |  +------+ +-----------+ |
>> + | xmgmt_main |  | VSEC |  | | GPIO | | QSPI | |  |  CMC | | AXI-GATE0 | |
>> + +------------+  +------+  | +------+ +------+ |  +------+ +-----------+ |
>> +                           | +---------+       |  +------+ +-----------+ |
>> +                           +>| MAILBOX |       +->| ICAP | | AXI-GATE1 |<+
>> +                             +---------+       |  +------+ +-----------+
>> +                                               |  +-------+
>> +                                               +->| CALIB |
>> +                                                  +-------+
>> +
>> +After a xclbin is downloaded, group3 will be added and the tree looks like the
>> +following::
>> +
>> +                                +-----------+
>> +                                |  xrt-mgmt |
>> +                                +-----+-----+
>> +                                      |
>> +            +-------------------------+--------------------+-----------------+
>> +            |                         |                    |                 |
>> +            v                         v                    v                 |
>> +       +--------+                +--------+            +--------+            |
>> +       | group0 |                | group1 |            | group2 |            |
>> +       +----+---+                +----+---+            +---+----+            |
>> +            |                         |                    |                 |
>> +            |                         |                    |                 |
>> +      +-----+-----+       +-----+-----+---+    +-----+-----+----+--------+   |
>> +      |           |       |     |         |    |     |          |        |   |
>> +      v           v       |     v         v    |     v          v        |   |
>> + +------------+  +------+ | +------+ +------+  |  +------+ +-----------+ |   |
>> + | xmgmt_main |  | VSEC | | | GPIO | | QSPI |  |  |  CMC | | AXI-GATE0 | |   |
>> + +------------+  +------+ | +------+ +------+  |  +------+ +-----------+ |   |
>> +                          | +---------+        |  +------+ +-----------+ |   |
>> +                          +>| MAILBOX |        +->| ICAP | | AXI-GATE1 |<+   |
>> +                            +---------+        |  +------+ +-----------+     |
>> +                                               |  +-------+                  |
>> +                                               +->| CALIB |                  |
>> +                                                  +-------+                  |
>> +                      +---+----+                                             |
>> +                      | group3 |<--------------------------------------------+
>> +                      +--------+
>> +                          |
>> +                          |
>> +     +-------+--------+---+--+--------+------+-------+
>> +     |       |        |      |        |      |       |
>> +     v       |        v      |        v      |       v
>> + +--------+  |   +--------+  |   +--------+  |    +-----+
>> + | CLOCK0 |  |   | CLOCK1 |  |   | CLOCK2 |  |    | UCS |
>> + +--------+  v   +--------+  v   +--------+  v    +-----+
>> + +-------------+ +-------------+ +-------------+
>> + | CLOCK-FREQ0 | | CLOCK-FREQ1 | | CLOCK-FREQ2 |
>> + +-------------+ +-------------+ +-------------+
>> +
>> +
>> +root
>> +----
>> +
>> +The root driver is a PCIe device driver attached to MPF. It's part of the
>> +infrastructure of the MPF driver and resides in xrt-mgmt.ko. This driver
>> +
>> +* manages one or more group drivers
>> +* provides access to functionalities that requires pci_dev, such as PCIE config
>> +  space access, to other xleaf drivers through root calls
>> +* facilities inter xleaf driver calls for other xleaf drivers
>> +* facilities event callbacks for other xleaf drivers
>> +
>> +When the root driver starts, it will explicitly create an initial group instance,
>> +which contains xleaf drivers that will trigger the creation of other group
>> +instances. The root driver will wait for all group and xleaf drivers to be
>> +created before it returns from its probe routine and claim success of the
>> +initialization of the entire xrt-mgmt driver. If any xleaf fails to initialize
>> +the xrt-mgmt driver will still come online but with limited functionality.
>> +
>> +.. note::
>> +   See code in ``lib/xroot.c`` and ``mgmt/root.c``
>> +
>> +
>> +group
>> +-----
>> +
>> +The group driver represents a pseudo device whose life cycle is managed by
>> +root and does not have real IO mem or IRQ resources. It's part of the
>> +infrastructure of the MPF driver and resides in xrt-lib.ko. This driver
>> +
>> +* manages one or more xleaf drivers
>> +* provides access to root from xleaf drivers, so that root calls, event
>> +  notifications and inter xleaf calls can happen
>> +
>> +In xrt-mgmt, an initial group driver instance will be created by the root. This
>> +instance contains xleaf drivers that will trigger group instances to be created
>> +to manage groups of xleaf drivers found on different partitions of hardware,
>> +such as VSEC, Shell, and User.
>> +
>> +Every *fpga_region* has a group driver associated with it. The group driver is
>> +created when a xclbin image is loaded on the fpga_region. The existing group
>> +is destroyed when a new xclbin image is loaded. The fpga_region persists
>> +across xclbin downloads.
>> +
>> +.. note::
>> +   See code in ``lib/group.c``
>> +
>> +
>> +xleaf
>> +-----
>> +
>> +The xleaf driver is a xrt_driver whose life cycle is managed by
>> +a group driver and may or may not have real IO mem or IRQ resources. They
>> +manage HW subsystems they are attached to.
>> +
>> +A xleaf driver without real hardware resources manages in-memory states for
>> +xrt-mgmt. These states are shareable by other xleaf drivers.
>> +
>> +Xleaf drivers assigned to specific hardware resources drive a specific subsystem
>> +in the device. To manipulate the subsystem or carry out a task, a xleaf driver
>> +may ask for help from the root via root calls and/or from other leaves via
>> +inter xleaf calls.
>> +
>> +A xleaf can also broadcast events through infrastructure code for other leaves
>> +to process. It can also receive event notification from infrastructure about
>> +certain events, such as post-creation or pre-exit of a particular xleaf.
>> +
>> +.. note::
>> +   See code in ``lib/xleaf/*.c``
>> +
>> +
>> +xrt_bus_type
>> +------------
>> +
>> +xrt_bus_type defines a virtual bus which handles xrt_driver probe, remove and match
>> +operations. All xrt_drivers register with xrt_bus_type as part of xrt-lib driver
>> +``module_init`` and un-register as part of xrt-lib driver ``module_exit``.
>> +
>> +.. note::
>> +   See code in ``lib/lib-drv.c``
>> +
>> +FPGA Manager Interaction
>> +========================
>> +
>> +fpga_manager
>> +------------
>> +
>> +An instance of fpga_manager is created by xmgmt_main and is used for xclbin
>> +image download. fpga_manager requires the full xclbin image before it can
>> +start programming the FPGA configuration engine via Internal Configuration
>> +Access Port (ICAP) xrt_driver.
>> +
>> +fpga_region
>> +-----------
>> +
>> +For every interface exposed by the currently loaded xclbin/xsabin in the
>> +*parent* fpga_region a new instance of fpga_region is created like a *child*
>> +fpga_region. The device tree of the *parent* fpga_region defines the
>> +resources for a new instance of fpga_bridge which isolates the parent from
>> +child fpga_region. This new instance of fpga_bridge will be used when a
>> +xclbin image is loaded on the child fpga_region. After the xclbin image is
>> +downloaded to the fpga_region, an instance of a group is created for the
>> +fpga_region using the device tree obtained as part of the xclbin. If this
>> +device tree defines any child interfaces, it can trigger the creation of
>> +fpga_bridge and fpga_region for the next region in the chain.
>> +
>> +fpga_bridge
>> +-----------
>> +
>> +Like the fpga_region, an fpga_bridge is created by walking the device tree
>> +of the parent group. The bridge is used for isolation between a parent and
>> +its child.
>> +
>> +Driver Interfaces
>> +=================
>> +
>> +xrt-mgmt Driver Ioctls
>> +----------------------
>> +
>> +Ioctls exposed by the xrt-mgmt driver to user space are enumerated in the
>> +following table:
>> +
>> +== ===================== ============================ ==========================
>> +#  Functionality         ioctl request code            data format
>> +== ===================== ============================ ==========================
>> +1  FPGA image download   XMGMT_IOCICAPDOWNLOAD_AXLF    xmgmt_ioc_bitstream_axlf
>> +== ===================== ============================ ==========================
>> +
>> +A user xclbin can be downloaded by using the xbmgmt tool from the XRT open source
>> +suite. See example usage below::
>> +
>> +  xbmgmt partition --program --path /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/test/verify.xclbin --force
>> +
>> +xrt-mgmt Driver Sysfs
>> +----------------------
>> +
>> +The xrt-mgmt driver exposes a rich set of sysfs interfaces. Subsystem xrt
>> +drivers export sysfs node for every platform instance.
>> +
>> +Every partition also exports its UUIDs. See below for examples::
>> +
>> +  /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
>> +  /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
>> +
>> +
>> +hwmon
>> +-----
>> +
>> +The xrt-mgmt driver exposes standard hwmon interface to report voltage, current,
>> +temperature, power, etc. These can easily be viewed using *sensors* command line
>> +utility.
>> +
>> +.. _alveo_platform_overview:
>> +
>> +Alveo Platform Overview
>> +=======================
>> +
>> +Alveo platforms are architected as two physical FPGA partitions: *Shell* and
>> +*User*. The Shell provides basic infrastructure for the Alveo platform like
>> +PCIe connectivity, board management, Dynamic Function Exchange (DFX), sensors,
>> +clocking, reset, and security. DFX, partial reconfiguration, is responsible for
>> +loading the user compiled FPGA binary.
>> +
>> +For DFX to work properly, physical partitions require strict HW compatibility
>> +with each other. Every physical partition has two interface UUIDs: the *parent*
>> +UUID and the *child* UUID. For simple single stage platforms, Shell → User forms
>> +the parent child relationship.
>> +
>> +.. note::
>> +   Partition compatibility matching is a key design component of the Alveo platforms
>> +   and XRT. Partitions have child and parent relationship. A loaded partition
>> +   exposes child partition UUID to advertise its compatibility requirement. When
>> +   loading a child partition, the xrt-mgmt driver matches the parent
>> +   UUID of the child partition against the child UUID exported by the parent.
>> +   The parent and child partition UUIDs are stored in the *xclbin* (for the user)
>> +   and the *xsabin* (for the shell). Except for the root UUID exported by VSEC,
>> +   the hardware itself does not know about the UUIDs. The UUIDs are stored in
>> +   xsabin and xclbin. The image format has a special node called Partition UUIDs
>> +   which define the compatibility UUIDs. See :ref:`partition_uuids`.
>> +
>> +
>> +The physical partitions and their loading are illustrated below::
>> +
>> +           SHELL                               USER
>> +        +-----------+                  +-------------------+
>> +        |           |                  |                   |
>> +        | VSEC UUID | CHILD     PARENT |    LOGIC UUID     |
>> +        |           o------->|<--------o                   |
>> +        |           | UUID       UUID  |                   |
>> +        +-----+-----+                  +--------+----------+
>> +              |                                 |
>> +              .                                 .
>> +              |                                 |
>> +          +---+---+                      +------+--------+
>> +          |  POR  |                      | USER COMPILED |
>> +          | FLASH |                      |    XCLBIN     |
>> +          +-------+                      +---------------+
>> +
>> +
>> +Loading Sequence
>> +----------------
>> +
>> +The Shell partition is loaded from flash at system boot time. It establishes the
>> +PCIe link and exposes two physical functions to the BIOS. After the OS boots,
>> +the xrt-mgmt driver attaches to the PCIe physical function 0 exposed by the Shell
>> +and then looks for VSEC in the PCIe extended configuration space. Using VSEC, it
>> +determines the logic UUID of the Shell and uses the UUID to load matching *xsabin*
>> +file from Linux firmware directory. The xsabin file contains the metadata to
>> +discover the peripherals that are part of the Shell and the firmware for any
>> +embedded soft processors in the Shell. The xsabin file also contains Partition
>> +UUIDs as described here :ref:`partition_uuids`.
>> +
>> +The Shell exports a child interface UUID which is used for the compatibility
>> +check when loading the user compiled xclbin over the User partition as part of DFX.
>> +When a user requests loading of a specific xclbin, the xrt-mgmt driver reads
>> +the parent interface UUID specified in the xclbin and matches it with the child
>> +interface UUID exported by the Shell to determine if the xclbin is compatible with
>> +the Shell. If the match fails, loading of xclbin is denied.
>> +
>> +xclbin loading is requested using the ICAP_DOWNLOAD_AXLF ioctl command. When loading
>> +a xclbin, the xrt-mgmt driver performs the following *logical* operations:
>> +
>> +1. Copy xclbin from user to kernel memory
>> +2. Sanity check the xclbin contents
>> +3. Isolate the User partition
>> +4. Download the bitstream using the FPGA config engine (ICAP)
>> +5. De-isolate the User partition
>> +6. Program the clocks (ClockWiz) driving the User partition
>> +7. Wait for the memory controller (MIG) calibration
>> +8. Return the loading status back to the caller
>> +
>> +`Platform Loading Overview <https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`_
>> +provides more detailed information on platform loading.
>> +
>> +
>> +xsabin
>> +------
>> +
>> +Each Alveo platform comes packaged with its own xsabin. The xsabin is a trusted
>> +component of the platform. For format details refer to :ref:`xsabin_xclbin_container_format`
>> +below. xsabin contains basic information like UUIDs, platform name and metadata in the
>> +form of device tree. See :ref:`device_tree_usage` below for details and example.
>> +
>> +xclbin
>> +------
>> +
>> +xclbin is compiled by end user using
>> +`Vitis <https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html>`_
>> +tool set from Xilinx. The xclbin contains sections describing user compiled
>> +acceleration engines/kernels, memory subsystems, clocking information etc. It also
>> +contains an FPGA bitstream for the user partition, UUIDs, platform name, etc.
>> +
>> +
>> +.. _xsabin_xclbin_container_format:
>> +
>> +xsabin/xclbin Container Format
>> +------------------------------
>> +
>> +xclbin/xsabin is ELF-like binary container format. It is structured as series of
>> +sections. There is a file header followed by several section headers which is
>> +followed by sections. A section header points to an actual section. There is an
>> +optional signature at the end. The format is defined by the header file ``xclbin.h``.
>> +The following figure illustrates a typical xclbin::
>> +
>> +
>> +           +---------------------+
>> +           |                     |
>> +           |       HEADER        |
>> +           +---------------------+
>> +           |   SECTION  HEADER   |
>> +           |                     |
>> +           +---------------------+
>> +           |         ...         |
>> +           |                     |
>> +           +---------------------+
>> +           |   SECTION  HEADER   |
>> +           |                     |
>> +           +---------------------+
>> +           |       SECTION       |
>> +           |                     |
>> +           +---------------------+
>> +           |         ...         |
>> +           |                     |
>> +           +---------------------+
>> +           |       SECTION       |
>> +           |                     |
>> +           +---------------------+
>> +           |      SIGNATURE      |
>> +           |      (OPTIONAL)     |
>> +           +---------------------+
>> +
>> +
>> +xclbin/xsabin files can be packaged, un-packaged and inspected using an XRT
>> +utility called **xclbinutil**. xclbinutil is part of the XRT open source
>> +software stack. The source code for xclbinutil can be found at
>> +https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbinutil
>> +
>> +For example, to enumerate the contents of a xclbin/xsabin use the *--info* switch
>> +as shown below::
>> +
>> +
>> +  xclbinutil --info --input /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
>> +  xclbinutil --info --input /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsabin
>> +
>> +
>> +.. _device_tree_usage:
>> +
>> +Device Tree Usage
>> +-----------------
>> +
>> +The xsabin file stores metadata which advertise HW subsystems present in a
>> +partition. The metadata is stored in device tree format with a well defined
>> +schema. XRT management driver uses this information to bind *xrt_drivers* to
>> +the subsystem instantiations. The xrt_drivers are found in **xrt-lib.ko** kernel
>> +module.
> I'm still catching up the patchset from the very beginning, and just
> finished the Documentation part. So far, I see the DT usage concern
> which may impact the architecure a lot, so I should raise it ASAP.
>
> The concern raised by the DT maintainer:
> https://lore.kernel.org/linux-fpga/[e-mail]/
>
> First of all, directly parsing FDT in device drivers is not a normal usage of DT
> in linux. It is out of the current DT usage model. So it should be agreed by DT
> maintainers.
Thanks for reviewing XRT document and providing feedback.
Here is the reply from Sonal for Rob’s question:
https://lore.kernel.org/linux-fpga/[e-mail]/
Overall, libfdt is used by XRT driver to parse the metadata which comes 
with an Alveo board.
When XRT driver discovers an Alveo board, it will read a fdt blob from 
board firmware file resident on the host.
By parsing the fdt blob, XRT driver gets information about this Alveo 
board, such as version, uuid, IPs exposed to PCI BAR, interrupt binding etc.
So libfdt is used simply as Alveo metadata parser here. XRT drivers do 
not interact with system wide DT or present the Alveo device tree to 
host. For many systems like x86_64, system wide DT is not present but 
libfdt parsing services will still be needed.
>
> Current FPGA framework modifies kernel's live tree by DT overlay, when FPGA is
> dynamically reprogrammed and new HW devices appear. See
> Documentation/devicetree/bindings/fpga/fpga-region.txt.
>
> Then something less important:
>
>    1. The bindings should be documented in Documentation/devicetree/bindings/.
>    2. Are all the example DT usage conform to the exsiting bindings? I
>       didn't go through all device classes, but remember like the
>       interrupt-controller should have a "interrupt-controller" property, and
>       the PCI properties are also different from PCI bindings.

The fdt properties are defined for Alveo firmware files. XRT driver is 
the only consumer of this data. I am wondering if 
Documentation/devicetree/bindings is the right place for Alveo/XRT 
private format or should it be documented as part of XRT driver 
documentation? Looking for guidance here.


Thanks,

Lizhi

>
> Thanks,
> Yilun
>
>> +
>> +Logic UUID
>> +^^^^^^^^^^
>> +A partition is identified uniquely through ``logic_uuid`` property::
>> +
>> +  /dts-v1/;
>> +  / {
>> +      logic_uuid = "0123456789abcdef0123456789abcdef";
>> +      ...
>> +    }
>> +
>> +Schema Version
>> +^^^^^^^^^^^^^^
>> +Schema version is defined through the ``schema_version`` node. It contains
>> +``major`` and ``minor`` properties as below::
>> +
>> +  /dts-v1/;
>> +  / {
>> +       schema_version {
>> +           major = <0x01>;
>> +           minor = <0x00>;
>> +       };
>> +       ...
>> +    }
>> +
>> +.. _partition_uuids:
>> +
>> +Partition UUIDs
>> +^^^^^^^^^^^^^^^
>> +Each partition may have parent and child UUIDs. These UUIDs are
>> +defined by ``interfaces`` node and ``interface_uuid`` property::
>> +
>> +  /dts-v1/;
>> +  / {
>> +       interfaces {
>> +           @0 {
>> +                  interface_uuid = "0123456789abcdef0123456789abcdef";
>> +           };
>> +           @1 {
>> +                  interface_uuid = "fedcba9876543210fedcba9876543210";
>> +           };
>> +           ...
>> +        };
>> +       ...
>> +    }
>> +
>> +
>> +Subsystem Instantiations
>> +^^^^^^^^^^^^^^^^^^^^^^^^
>> +Subsystem instantiations are captured as children of ``addressable_endpoints``
>> +node::
>> +
>> +  /dts-v1/;
>> +  / {
>> +       addressable_endpoints {
>> +           abc {
>> +               ...
>> +           };
>> +           def {
>> +               ...
>> +           };
>> +           ...
>> +       }
>> +  }
>> +
>> +Subnode 'abc' and 'def' are the name of subsystem nodes
>> +
>> +Subsystem Node
>> +^^^^^^^^^^^^^^
>> +Each subsystem node and its properties define a hardware instance::
>> +
>> +
>> +  addressable_endpoints {
>> +      abc {
>> +          reg = <0x00 0x1f05000 0x00 0x1000>>
>> +          pcie_physical_function = <0x0>;
>> +          pcie_bar_mapping = <0x2>;
>> +          compatible = "abc def";
>> +          interrupts = <0x09 0x0c>;
>> +          firmware {
>> +              firmware_product_name = "abc"
>> +              firmware_branch_name = "def"
>> +              firmware_version_major = <1>
>> +              firmware_version_minor = <2>
>> +          };
>> +      }
>> +      ...
>> +  }
>> +
>> +:reg:
>> + Property defines an address range. `<0x00 0x1f05000 0x00 0x1000>` indicates
>> + *0x00 0x1f05000* as BAR offset and *0x00 0x1000* as address length.
>> +:pcie_physical_function:
>> + Property specifies which PCIe physical function the subsystem node resides.
>> + `<0x0>` implies physical function 0.
>> +:pcie_bar_mapping:
>> + Property specifies which PCIe BAR the subsystem node resides. `<0x2>` implies
>> + BAR 2. A value of 0 means the property is not defined.
>> +:compatible:
>> + Property is a list of strings. The first string in the list specifies the exact
>> + subsystem node. The following strings represent other devices that the device
>> + is compatible with.
>> +:interrupts:
>> + Property specifies start and end interrupts for this subsystem node.
>> + `<0x09 0x0c>` implies interrupts 9 to 13 are used by this subsystem.
>> +:firmware:
>> + Subnode defines the firmware required by this subsystem node.
>> +
>> +Alveo U50 Platform Example
>> +^^^^^^^^^^^^^^^^^^^^^^^^^^
>> +::
>> +
>> +  /dts-v1/;
>> +
>> +  /{
>> +        logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
>> +
>> +        schema_version {
>> +                major = <0x01>;
>> +                minor = <0x00>;
>> +        };
>> +
>> +        interfaces {
>> +
>> +                @0 {
>> +                        interface_uuid = "862c7020a250293e32036f19956669e5";
>> +                };
>> +        };
>> +
>> +        addressable_endpoints {
>> +
>> +                ep_blp_rom_00 {
>> +                        reg = <0x00 0x1f04000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
>> +                };
>> +
>> +                ep_card_flash_program_00 {
>> +                        reg = <0x00 0x1f06000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_quad_spi-1.0\0axi_quad_spi";
>> +                        interrupts = <0x03 0x03>;
>> +                };
>> +
>> +                ep_cmc_firmware_mem_00 {
>> +                        reg = <0x00 0x1e20000 0x00 0x20000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
>> +
>> +                        firmware {
>> +                                firmware_product_name = "cmc";
>> +                                firmware_branch_name = "u50";
>> +                                firmware_version_major = <0x01>;
>> +                                firmware_version_minor = <0x00>;
>> +                        };
>> +                };
>> +
>> +                ep_cmc_intc_00 {
>> +                        reg = <0x00 0x1e03000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
>> +                        interrupts = <0x04 0x04>;
>> +                };
>> +
>> +                ep_cmc_mutex_00 {
>> +                        reg = <0x00 0x1e02000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>> +                };
>> +
>> +                ep_cmc_regmap_00 {
>> +                        reg = <0x00 0x1e08000 0x00 0x2000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
>> +
>> +                        firmware {
>> +                                firmware_product_name = "sc-fw";
>> +                                firmware_branch_name = "u50";
>> +                                firmware_version_major = <0x05>;
>> +                        };
>> +                };
>> +
>> +                ep_cmc_reset_00 {
>> +                        reg = <0x00 0x1e01000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>> +                };
>> +
>> +                ep_ddr_mem_calib_00 {
>> +                        reg = <0x00 0x63000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>> +                };
>> +
>> +                ep_debug_bscan_mgmt_00 {
>> +                        reg = <0x00 0x1e90000 0x00 0x10000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-debug_bridge-1.0\0debug_bridge";
>> +                };
>> +
>> +                ep_ert_base_address_00 {
>> +                        reg = <0x00 0x21000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>> +                };
>> +
>> +                ep_ert_command_queue_mgmt_00 {
>> +                        reg = <0x00 0x40000 0x00 0x10000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
>> +                };
>> +
>> +                ep_ert_command_queue_user_00 {
>> +                        reg = <0x00 0x40000 0x00 0x10000>;
>> +                        pcie_physical_function = <0x01>;
>> +                        compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
>> +                };
>> +
>> +                ep_ert_firmware_mem_00 {
>> +                        reg = <0x00 0x30000 0x00 0x8000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
>> +
>> +                        firmware {
>> +                                firmware_product_name = "ert";
>> +                                firmware_branch_name = "v20";
>> +                                firmware_version_major = <0x01>;
>> +                        };
>> +                };
>> +
>> +                ep_ert_intc_00 {
>> +                        reg = <0x00 0x23000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
>> +                        interrupts = <0x05 0x05>;
>> +                };
>> +
>> +                ep_ert_reset_00 {
>> +                        reg = <0x00 0x22000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>> +                };
>> +
>> +                ep_ert_sched_00 {
>> +                        reg = <0x00 0x50000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x01>;
>> +                        compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
>> +                        interrupts = <0x09 0x0c>;
>> +                };
>> +
>> +                ep_fpga_configuration_00 {
>> +                        reg = <0x00 0x1e88000 0x00 0x8000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
>> +                        interrupts = <0x02 0x02>;
>> +                };
>> +
>> +                ep_icap_reset_00 {
>> +                        reg = <0x00 0x1f07000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>> +                };
>> +
>> +                ep_msix_00 {
>> +                        reg = <0x00 0x00 0x00 0x20000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
>> +                        pcie_bar_mapping = <0x02>;
>> +                };
>> +
>> +                ep_pcie_link_mon_00 {
>> +                        reg = <0x00 0x1f05000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>> +                };
>> +
>> +                ep_pr_isolate_plp_00 {
>> +                        reg = <0x00 0x1f01000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>> +                };
>> +
>> +                ep_pr_isolate_ulp_00 {
>> +                        reg = <0x00 0x1000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>> +                };
>> +
>> +                ep_uuid_rom_00 {
>> +                        reg = <0x00 0x64000 0x00 0x1000>;
>> +                        pcie_physical_function = <0x00>;
>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
>> +                };
>> +
>> +                ep_xdma_00 {
>> +                        reg = <0x00 0x00 0x00 0x10000>;
>> +                        pcie_physical_function = <0x01>;
>> +                        compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
>> +                        pcie_bar_mapping = <0x02>;
>> +                };
>> +        };
>> +
>> +  }
>> +
>> +
>> +
>> +Deployment Models
>> +=================
>> +
>> +Baremetal
>> +---------
>> +
>> +In bare-metal deployments, both MPF and UPF are visible and accessible. The
>> +xrt-mgmt driver binds to MPF. The xrt-mgmt driver operations are privileged and
>> +available to system administrator. The full stack is illustrated below::
>> +
>> +                            HOST
>> +
>> +               [XRT-MGMT]         [XRT-USER]
>> +                    |                  |
>> +                    |                  |
>> +                 +-----+            +-----+
>> +                 | MPF |            | UPF |
>> +                 |     |            |     |
>> +                 | PF0 |            | PF1 |
>> +                 +--+--+            +--+--+
>> +          ......... ^................. ^..........
>> +                    |                  |
>> +                    |   PCIe DEVICE    |
>> +                    |                  |
>> +                 +--+------------------+--+
>> +                 |         SHELL          |
>> +                 |                        |
>> +                 +------------------------+
>> +                 |         USER           |
>> +                 |                        |
>> +                 |                        |
>> +                 |                        |
>> +                 |                        |
>> +                 +------------------------+
>> +
>> +
>> +
>> +Virtualized
>> +-----------
>> +
>> +In virtualized deployments, the privileged MPF is assigned to the host but the
>> +unprivileged UPF is assigned to a guest VM via PCIe pass-through. The xrt-mgmt
>> +driver in host binds to MPF. The xrt-mgmt driver operations are privileged and
>> +only accessible to the MPF. The full stack is illustrated below::
>> +
>> +
>> +                                 ..............
>> +                  HOST           .    VM      .
>> +                                 .            .
>> +               [XRT-MGMT]        . [XRT-USER] .
>> +                    |            .     |      .
>> +                    |            .     |      .
>> +                 +-----+         .  +-----+   .
>> +                 | MPF |         .  | UPF |   .
>> +                 |     |         .  |     |   .
>> +                 | PF0 |         .  | PF1 |   .
>> +                 +--+--+         .  +--+--+   .
>> +          ......... ^................. ^..........
>> +                    |                  |
>> +                    |   PCIe DEVICE    |
>> +                    |                  |
>> +                 +--+------------------+--+
>> +                 |         SHELL          |
>> +                 |                        |
>> +                 +------------------------+
>> +                 |         USER           |
>> +                 |                        |
>> +                 |                        |
>> +                 |                        |
>> +                 |                        |
>> +                 +------------------------+
>> +
>> +
>> +
>> +
>> +
>> +Platform Security Considerations
>> +================================
>> +
>> +`Security of Alveo Platform <https://xilinx.github.io/XRT/master/html/security.html>`_
>> +discusses the deployment options and security implications in great detail.
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 056966c9aac9..beeaf0257364 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -7274,6 +7274,17 @@ F:     Documentation/fpga/
>>   F:   drivers/fpga/
>>   F:   include/linux/fpga/
>>
>> +FPGA XRT DRIVERS
>> +M:   Lizhi Hou <[e-mail]>
>> +R:   Max Zhen <[e-mail]>
>> +R:   Sonal Santan <[e-mail]>
>> +L:   [e-mail]
>> +S:   Supported
>> +W:   https://github.com/Xilinx/XRT
>> +F:   Documentation/fpga/xrt.rst
>> +F:   drivers/fpga/xrt/
>> +F:   include/uapi/linux/xrt/
>> +
>>   FPU EMULATOR
>>   M:   Bill Metzenthen <[e-mail]>
>>   S:   Maintained
>> --
>> 2.27.0

Xu Yilun Oct. 14, 2021, 2:21 a.m. UTC | #3

> > > +.. _device_tree_usage:
> > > +
> > > +Device Tree Usage
> > > +-----------------
> > > +
> > > +The xsabin file stores metadata which advertise HW subsystems present in a
> > > +partition. The metadata is stored in device tree format with a well defined
> > > +schema. XRT management driver uses this information to bind *xrt_drivers* to
> > > +the subsystem instantiations. The xrt_drivers are found in **xrt-lib.ko** kernel
> > > +module.
> > I'm still catching up the patchset from the very beginning, and just
> > finished the Documentation part. So far, I see the DT usage concern
> > which may impact the architecure a lot, so I should raise it ASAP.
> > 
> > The concern raised by the DT maintainer:
> > https://lore.kernel.org/linux-fpga/[e-mail]/
> > 
> > First of all, directly parsing FDT in device drivers is not a normal usage of DT
> > in linux. It is out of the current DT usage model. So it should be agreed by DT
> > maintainers.
> Thanks for reviewing XRT document and providing feedback.
> Here is the reply from Sonal for Rob’s question:
> https://lore.kernel.org/linux-fpga/[e-mail]/
> Overall, libfdt is used by XRT driver to parse the metadata which comes with
> an Alveo board.
> When XRT driver discovers an Alveo board, it will read a fdt blob from board
> firmware file resident on the host.
> By parsing the fdt blob, XRT driver gets information about this Alveo board,
> such as version, uuid, IPs exposed to PCI BAR, interrupt binding etc.
> So libfdt is used simply as Alveo metadata parser here. XRT drivers do not
> interact with system wide DT or present the Alveo device tree to host. For
> many systems like x86_64, system wide DT is not present but libfdt parsing
> services will still be needed.

Yes, I understand the use case.

My concern is, directly parsing an isolated FDT in device driver and
populate sub devices, skipping the unflattening, this is a new working
model of device tree usage, but for the same purpose as the existing
one.

So I really need the confirmation of DT maintainers.

> > 
> > Current FPGA framework modifies kernel's live tree by DT overlay, when FPGA is
> > dynamically reprogrammed and new HW devices appear. See
> > Documentation/devicetree/bindings/fpga/fpga-region.txt.
> > 
> > Then something less important:
> > 
> >    1. The bindings should be documented in Documentation/devicetree/bindings/.
> >    2. Are all the example DT usage conform to the exsiting bindings? I
> >       didn't go through all device classes, but remember like the
> >       interrupt-controller should have a "interrupt-controller" property, and
> >       the PCI properties are also different from PCI bindings.
> 
> The fdt properties are defined for Alveo firmware files. XRT driver is the
> only consumer of this data. I am wondering if

Personally I don't like the idea of a different binding definition set,
even if the new device tree usage is accepted. We all use device tree
for device enumeration, so why the different definitions.

Thanks,
Yilun

> Documentation/devicetree/bindings is the right place for Alveo/XRT private
> format or should it be documented as part of XRT driver documentation?
> Looking for guidance here.
> 
> 
> Thanks,
> 
> Lizhi
> 
> > 
> > Thanks,
> > Yilun
> > 
> > > +
> > > +Logic UUID
> > > +^^^^^^^^^^
> > > +A partition is identified uniquely through ``logic_uuid`` property::
> > > +
> > > +  /dts-v1/;
> > > +  / {
> > > +      logic_uuid = "0123456789abcdef0123456789abcdef";
> > > +      ...
> > > +    }
> > > +
> > > +Schema Version
> > > +^^^^^^^^^^^^^^
> > > +Schema version is defined through the ``schema_version`` node. It contains
> > > +``major`` and ``minor`` properties as below::
> > > +
> > > +  /dts-v1/;
> > > +  / {
> > > +       schema_version {
> > > +           major = <0x01>;
> > > +           minor = <0x00>;
> > > +       };
> > > +       ...
> > > +    }
> > > +
> > > +.. _partition_uuids:
> > > +
> > > +Partition UUIDs
> > > +^^^^^^^^^^^^^^^
> > > +Each partition may have parent and child UUIDs. These UUIDs are
> > > +defined by ``interfaces`` node and ``interface_uuid`` property::
> > > +
> > > +  /dts-v1/;
> > > +  / {
> > > +       interfaces {
> > > +           @0 {
> > > +                  interface_uuid = "0123456789abcdef0123456789abcdef";
> > > +           };
> > > +           @1 {
> > > +                  interface_uuid = "fedcba9876543210fedcba9876543210";
> > > +           };
> > > +           ...
> > > +        };
> > > +       ...
> > > +    }
> > > +
> > > +
> > > +Subsystem Instantiations
> > > +^^^^^^^^^^^^^^^^^^^^^^^^
> > > +Subsystem instantiations are captured as children of ``addressable_endpoints``
> > > +node::
> > > +
> > > +  /dts-v1/;
> > > +  / {
> > > +       addressable_endpoints {
> > > +           abc {
> > > +               ...
> > > +           };
> > > +           def {
> > > +               ...
> > > +           };
> > > +           ...
> > > +       }
> > > +  }
> > > +
> > > +Subnode 'abc' and 'def' are the name of subsystem nodes
> > > +
> > > +Subsystem Node
> > > +^^^^^^^^^^^^^^
> > > +Each subsystem node and its properties define a hardware instance::
> > > +
> > > +
> > > +  addressable_endpoints {
> > > +      abc {
> > > +          reg = <0x00 0x1f05000 0x00 0x1000>>
> > > +          pcie_physical_function = <0x0>;
> > > +          pcie_bar_mapping = <0x2>;
> > > +          compatible = "abc def";
> > > +          interrupts = <0x09 0x0c>;
> > > +          firmware {
> > > +              firmware_product_name = "abc"
> > > +              firmware_branch_name = "def"
> > > +              firmware_version_major = <1>
> > > +              firmware_version_minor = <2>
> > > +          };
> > > +      }
> > > +      ...
> > > +  }
> > > +
> > > +:reg:
> > > + Property defines an address range. `<0x00 0x1f05000 0x00 0x1000>` indicates
> > > + *0x00 0x1f05000* as BAR offset and *0x00 0x1000* as address length.
> > > +:pcie_physical_function:
> > > + Property specifies which PCIe physical function the subsystem node resides.
> > > + `<0x0>` implies physical function 0.
> > > +:pcie_bar_mapping:
> > > + Property specifies which PCIe BAR the subsystem node resides. `<0x2>` implies
> > > + BAR 2. A value of 0 means the property is not defined.
> > > +:compatible:
> > > + Property is a list of strings. The first string in the list specifies the exact
> > > + subsystem node. The following strings represent other devices that the device
> > > + is compatible with.
> > > +:interrupts:
> > > + Property specifies start and end interrupts for this subsystem node.
> > > + `<0x09 0x0c>` implies interrupts 9 to 13 are used by this subsystem.
> > > +:firmware:
> > > + Subnode defines the firmware required by this subsystem node.
> > > +
> > > +Alveo U50 Platform Example
> > > +^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > +::
> > > +
> > > +  /dts-v1/;
> > > +
> > > +  /{
> > > +        logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
> > > +
> > > +        schema_version {
> > > +                major = <0x01>;
> > > +                minor = <0x00>;
> > > +        };
> > > +
> > > +        interfaces {
> > > +
> > > +                @0 {
> > > +                        interface_uuid = "862c7020a250293e32036f19956669e5";
> > > +                };
> > > +        };
> > > +
> > > +        addressable_endpoints {
> > > +
> > > +                ep_blp_rom_00 {
> > > +                        reg = <0x00 0x1f04000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > > +                };
> > > +
> > > +                ep_card_flash_program_00 {
> > > +                        reg = <0x00 0x1f06000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_quad_spi-1.0\0axi_quad_spi";
> > > +                        interrupts = <0x03 0x03>;
> > > +                };
> > > +
> > > +                ep_cmc_firmware_mem_00 {
> > > +                        reg = <0x00 0x1e20000 0x00 0x20000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > > +
> > > +                        firmware {
> > > +                                firmware_product_name = "cmc";
> > > +                                firmware_branch_name = "u50";
> > > +                                firmware_version_major = <0x01>;
> > > +                                firmware_version_minor = <0x00>;
> > > +                        };
> > > +                };
> > > +
> > > +                ep_cmc_intc_00 {
> > > +                        reg = <0x00 0x1e03000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> > > +                        interrupts = <0x04 0x04>;
> > > +                };
> > > +
> > > +                ep_cmc_mutex_00 {
> > > +                        reg = <0x00 0x1e02000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > +                };
> > > +
> > > +                ep_cmc_regmap_00 {
> > > +                        reg = <0x00 0x1e08000 0x00 0x2000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > > +
> > > +                        firmware {
> > > +                                firmware_product_name = "sc-fw";
> > > +                                firmware_branch_name = "u50";
> > > +                                firmware_version_major = <0x05>;
> > > +                        };
> > > +                };
> > > +
> > > +                ep_cmc_reset_00 {
> > > +                        reg = <0x00 0x1e01000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > +                };
> > > +
> > > +                ep_ddr_mem_calib_00 {
> > > +                        reg = <0x00 0x63000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > +                };
> > > +
> > > +                ep_debug_bscan_mgmt_00 {
> > > +                        reg = <0x00 0x1e90000 0x00 0x10000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-debug_bridge-1.0\0debug_bridge";
> > > +                };
> > > +
> > > +                ep_ert_base_address_00 {
> > > +                        reg = <0x00 0x21000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > +                };
> > > +
> > > +                ep_ert_command_queue_mgmt_00 {
> > > +                        reg = <0x00 0x40000 0x00 0x10000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
> > > +                };
> > > +
> > > +                ep_ert_command_queue_user_00 {
> > > +                        reg = <0x00 0x40000 0x00 0x10000>;
> > > +                        pcie_physical_function = <0x01>;
> > > +                        compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
> > > +                };
> > > +
> > > +                ep_ert_firmware_mem_00 {
> > > +                        reg = <0x00 0x30000 0x00 0x8000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > > +
> > > +                        firmware {
> > > +                                firmware_product_name = "ert";
> > > +                                firmware_branch_name = "v20";
> > > +                                firmware_version_major = <0x01>;
> > > +                        };
> > > +                };
> > > +
> > > +                ep_ert_intc_00 {
> > > +                        reg = <0x00 0x23000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> > > +                        interrupts = <0x05 0x05>;
> > > +                };
> > > +
> > > +                ep_ert_reset_00 {
> > > +                        reg = <0x00 0x22000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > +                };
> > > +
> > > +                ep_ert_sched_00 {
> > > +                        reg = <0x00 0x50000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x01>;
> > > +                        compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
> > > +                        interrupts = <0x09 0x0c>;
> > > +                };
> > > +
> > > +                ep_fpga_configuration_00 {
> > > +                        reg = <0x00 0x1e88000 0x00 0x8000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
> > > +                        interrupts = <0x02 0x02>;
> > > +                };
> > > +
> > > +                ep_icap_reset_00 {
> > > +                        reg = <0x00 0x1f07000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > +                };
> > > +
> > > +                ep_msix_00 {
> > > +                        reg = <0x00 0x00 0x00 0x20000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
> > > +                        pcie_bar_mapping = <0x02>;
> > > +                };
> > > +
> > > +                ep_pcie_link_mon_00 {
> > > +                        reg = <0x00 0x1f05000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > +                };
> > > +
> > > +                ep_pr_isolate_plp_00 {
> > > +                        reg = <0x00 0x1f01000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > +                };
> > > +
> > > +                ep_pr_isolate_ulp_00 {
> > > +                        reg = <0x00 0x1000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > +                };
> > > +
> > > +                ep_uuid_rom_00 {
> > > +                        reg = <0x00 0x64000 0x00 0x1000>;
> > > +                        pcie_physical_function = <0x00>;
> > > +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > > +                };
> > > +
> > > +                ep_xdma_00 {
> > > +                        reg = <0x00 0x00 0x00 0x10000>;
> > > +                        pcie_physical_function = <0x01>;
> > > +                        compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
> > > +                        pcie_bar_mapping = <0x02>;
> > > +                };
> > > +        };
> > > +
> > > +  }
> > > +
> > > +
> > > +
> > > +Deployment Models
> > > +=================
> > > +
> > > +Baremetal
> > > +---------
> > > +
> > > +In bare-metal deployments, both MPF and UPF are visible and accessible. The
> > > +xrt-mgmt driver binds to MPF. The xrt-mgmt driver operations are privileged and
> > > +available to system administrator. The full stack is illustrated below::
> > > +
> > > +                            HOST
> > > +
> > > +               [XRT-MGMT]         [XRT-USER]
> > > +                    |                  |
> > > +                    |                  |
> > > +                 +-----+            +-----+
> > > +                 | MPF |            | UPF |
> > > +                 |     |            |     |
> > > +                 | PF0 |            | PF1 |
> > > +                 +--+--+            +--+--+
> > > +          ......... ^................. ^..........
> > > +                    |                  |
> > > +                    |   PCIe DEVICE    |
> > > +                    |                  |
> > > +                 +--+------------------+--+
> > > +                 |         SHELL          |
> > > +                 |                        |
> > > +                 +------------------------+
> > > +                 |         USER           |
> > > +                 |                        |
> > > +                 |                        |
> > > +                 |                        |
> > > +                 |                        |
> > > +                 +------------------------+
> > > +
> > > +
> > > +
> > > +Virtualized
> > > +-----------
> > > +
> > > +In virtualized deployments, the privileged MPF is assigned to the host but the
> > > +unprivileged UPF is assigned to a guest VM via PCIe pass-through. The xrt-mgmt
> > > +driver in host binds to MPF. The xrt-mgmt driver operations are privileged and
> > > +only accessible to the MPF. The full stack is illustrated below::
> > > +
> > > +
> > > +                                 ..............
> > > +                  HOST           .    VM      .
> > > +                                 .            .
> > > +               [XRT-MGMT]        . [XRT-USER] .
> > > +                    |            .     |      .
> > > +                    |            .     |      .
> > > +                 +-----+         .  +-----+   .
> > > +                 | MPF |         .  | UPF |   .
> > > +                 |     |         .  |     |   .
> > > +                 | PF0 |         .  | PF1 |   .
> > > +                 +--+--+         .  +--+--+   .
> > > +          ......... ^................. ^..........
> > > +                    |                  |
> > > +                    |   PCIe DEVICE    |
> > > +                    |                  |
> > > +                 +--+------------------+--+
> > > +                 |         SHELL          |
> > > +                 |                        |
> > > +                 +------------------------+
> > > +                 |         USER           |
> > > +                 |                        |
> > > +                 |                        |
> > > +                 |                        |
> > > +                 |                        |
> > > +                 +------------------------+
> > > +
> > > +
> > > +
> > > +
> > > +
> > > +Platform Security Considerations
> > > +================================
> > > +
> > > +`Security of Alveo Platform <https://xilinx.github.io/XRT/master/html/security.html>`_
> > > +discusses the deployment options and security implications in great detail.
> > > diff --git a/MAINTAINERS b/MAINTAINERS
> > > index 056966c9aac9..beeaf0257364 100644
> > > --- a/MAINTAINERS
> > > +++ b/MAINTAINERS
> > > @@ -7274,6 +7274,17 @@ F:     Documentation/fpga/
> > >   F:   drivers/fpga/
> > >   F:   include/linux/fpga/
> > > 
> > > +FPGA XRT DRIVERS
> > > +M:   Lizhi Hou <[e-mail]>
> > > +R:   Max Zhen <[e-mail]>
> > > +R:   Sonal Santan <[e-mail]>
> > > +L:   [e-mail]
> > > +S:   Supported
> > > +W:   https://github.com/Xilinx/XRT
> > > +F:   Documentation/fpga/xrt.rst
> > > +F:   drivers/fpga/xrt/
> > > +F:   include/uapi/linux/xrt/
> > > +
> > >   FPU EMULATOR
> > >   M:   Bill Metzenthen <[e-mail]>
> > >   S:   Maintained
> > > --
> > > 2.27.0

Lizhi Hou Oct. 14, 2021, 4:12 p.m. UTC | #4

Hello Rob,

Please help with the review of the proposed FDT usage by Alveo/XRT drivers.

Thanks,
Lizhi

On 10/13/21 7:21 PM, Xu Yilun wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>
>
>>>> +.. _device_tree_usage:
>>>> +
>>>> +Device Tree Usage
>>>> +-----------------
>>>> +
>>>> +The xsabin file stores metadata which advertise HW subsystems present in a
>>>> +partition. The metadata is stored in device tree format with a well defined
>>>> +schema. XRT management driver uses this information to bind *xrt_drivers* to
>>>> +the subsystem instantiations. The xrt_drivers are found in **xrt-lib.ko** kernel
>>>> +module.
>>> I'm still catching up the patchset from the very beginning, and just
>>> finished the Documentation part. So far, I see the DT usage concern
>>> which may impact the architecure a lot, so I should raise it ASAP.
>>>
>>> The concern raised by the DT maintainer:
>>> https://lore.kernel.org/linux-fpga/[e-mail]/
>>>
>>> First of all, directly parsing FDT in device drivers is not a normal usage of DT
>>> in linux. It is out of the current DT usage model. So it should be agreed by DT
>>> maintainers.
>> Thanks for reviewing XRT document and providing feedback.
>> Here is the reply from Sonal for Rob’s question:
>> https://lore.kernel.org/linux-fpga/[e-mail]/
>> Overall, libfdt is used by XRT driver to parse the metadata which comes with
>> an Alveo board.
>> When XRT driver discovers an Alveo board, it will read a fdt blob from board
>> firmware file resident on the host.
>> By parsing the fdt blob, XRT driver gets information about this Alveo board,
>> such as version, uuid, IPs exposed to PCI BAR, interrupt binding etc.
>> So libfdt is used simply as Alveo metadata parser here. XRT drivers do not
>> interact with system wide DT or present the Alveo device tree to host. For
>> many systems like x86_64, system wide DT is not present but libfdt parsing
>> services will still be needed.
> Yes, I understand the use case.
>
> My concern is, directly parsing an isolated FDT in device driver and
> populate sub devices, skipping the unflattening, this is a new working
> model of device tree usage, but for the same purpose as the existing
> one.
>
> So I really need the confirmation of DT maintainers.
>
>>> Current FPGA framework modifies kernel's live tree by DT overlay, when FPGA is
>>> dynamically reprogrammed and new HW devices appear. See
>>> Documentation/devicetree/bindings/fpga/fpga-region.txt.
>>>
>>> Then something less important:
>>>
>>>     1. The bindings should be documented in Documentation/devicetree/bindings/.
>>>     2. Are all the example DT usage conform to the exsiting bindings? I
>>>        didn't go through all device classes, but remember like the
>>>        interrupt-controller should have a "interrupt-controller" property, and
>>>        the PCI properties are also different from PCI bindings.
>> The fdt properties are defined for Alveo firmware files. XRT driver is the
>> only consumer of this data. I am wondering if
> Personally I don't like the idea of a different binding definition set,
> even if the new device tree usage is accepted. We all use device tree
> for device enumeration, so why the different definitions.
>
> Thanks,
> Yilun
>
>> Documentation/devicetree/bindings is the right place for Alveo/XRT private
>> format or should it be documented as part of XRT driver documentation?
>> Looking for guidance here.
>>
>>
>> Thanks,
>>
>> Lizhi
>>
>>> Thanks,
>>> Yilun
>>>
>>>> +
>>>> +Logic UUID
>>>> +^^^^^^^^^^
>>>> +A partition is identified uniquely through ``logic_uuid`` property::
>>>> +
>>>> +  /dts-v1/;
>>>> +  / {
>>>> +      logic_uuid = "0123456789abcdef0123456789abcdef";
>>>> +      ...
>>>> +    }
>>>> +
>>>> +Schema Version
>>>> +^^^^^^^^^^^^^^
>>>> +Schema version is defined through the ``schema_version`` node. It contains
>>>> +``major`` and ``minor`` properties as below::
>>>> +
>>>> +  /dts-v1/;
>>>> +  / {
>>>> +       schema_version {
>>>> +           major = <0x01>;
>>>> +           minor = <0x00>;
>>>> +       };
>>>> +       ...
>>>> +    }
>>>> +
>>>> +.. _partition_uuids:
>>>> +
>>>> +Partition UUIDs
>>>> +^^^^^^^^^^^^^^^
>>>> +Each partition may have parent and child UUIDs. These UUIDs are
>>>> +defined by ``interfaces`` node and ``interface_uuid`` property::
>>>> +
>>>> +  /dts-v1/;
>>>> +  / {
>>>> +       interfaces {
>>>> +           @0 {
>>>> +                  interface_uuid = "0123456789abcdef0123456789abcdef";
>>>> +           };
>>>> +           @1 {
>>>> +                  interface_uuid = "fedcba9876543210fedcba9876543210";
>>>> +           };
>>>> +           ...
>>>> +        };
>>>> +       ...
>>>> +    }
>>>> +
>>>> +
>>>> +Subsystem Instantiations
>>>> +^^^^^^^^^^^^^^^^^^^^^^^^
>>>> +Subsystem instantiations are captured as children of ``addressable_endpoints``
>>>> +node::
>>>> +
>>>> +  /dts-v1/;
>>>> +  / {
>>>> +       addressable_endpoints {
>>>> +           abc {
>>>> +               ...
>>>> +           };
>>>> +           def {
>>>> +               ...
>>>> +           };
>>>> +           ...
>>>> +       }
>>>> +  }
>>>> +
>>>> +Subnode 'abc' and 'def' are the name of subsystem nodes
>>>> +
>>>> +Subsystem Node
>>>> +^^^^^^^^^^^^^^
>>>> +Each subsystem node and its properties define a hardware instance::
>>>> +
>>>> +
>>>> +  addressable_endpoints {
>>>> +      abc {
>>>> +          reg = <0x00 0x1f05000 0x00 0x1000>>
>>>> +          pcie_physical_function = <0x0>;
>>>> +          pcie_bar_mapping = <0x2>;
>>>> +          compatible = "abc def";
>>>> +          interrupts = <0x09 0x0c>;
>>>> +          firmware {
>>>> +              firmware_product_name = "abc"
>>>> +              firmware_branch_name = "def"
>>>> +              firmware_version_major = <1>
>>>> +              firmware_version_minor = <2>
>>>> +          };
>>>> +      }
>>>> +      ...
>>>> +  }
>>>> +
>>>> +:reg:
>>>> + Property defines an address range. `<0x00 0x1f05000 0x00 0x1000>` indicates
>>>> + *0x00 0x1f05000* as BAR offset and *0x00 0x1000* as address length.
>>>> +:pcie_physical_function:
>>>> + Property specifies which PCIe physical function the subsystem node resides.
>>>> + `<0x0>` implies physical function 0.
>>>> +:pcie_bar_mapping:
>>>> + Property specifies which PCIe BAR the subsystem node resides. `<0x2>` implies
>>>> + BAR 2. A value of 0 means the property is not defined.
>>>> +:compatible:
>>>> + Property is a list of strings. The first string in the list specifies the exact
>>>> + subsystem node. The following strings represent other devices that the device
>>>> + is compatible with.
>>>> +:interrupts:
>>>> + Property specifies start and end interrupts for this subsystem node.
>>>> + `<0x09 0x0c>` implies interrupts 9 to 13 are used by this subsystem.
>>>> +:firmware:
>>>> + Subnode defines the firmware required by this subsystem node.
>>>> +
>>>> +Alveo U50 Platform Example
>>>> +^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>> +::
>>>> +
>>>> +  /dts-v1/;
>>>> +
>>>> +  /{
>>>> +        logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
>>>> +
>>>> +        schema_version {
>>>> +                major = <0x01>;
>>>> +                minor = <0x00>;
>>>> +        };
>>>> +
>>>> +        interfaces {
>>>> +
>>>> +                @0 {
>>>> +                        interface_uuid = "862c7020a250293e32036f19956669e5";
>>>> +                };
>>>> +        };
>>>> +
>>>> +        addressable_endpoints {
>>>> +
>>>> +                ep_blp_rom_00 {
>>>> +                        reg = <0x00 0x1f04000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
>>>> +                };
>>>> +
>>>> +                ep_card_flash_program_00 {
>>>> +                        reg = <0x00 0x1f06000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_quad_spi-1.0\0axi_quad_spi";
>>>> +                        interrupts = <0x03 0x03>;
>>>> +                };
>>>> +
>>>> +                ep_cmc_firmware_mem_00 {
>>>> +                        reg = <0x00 0x1e20000 0x00 0x20000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
>>>> +
>>>> +                        firmware {
>>>> +                                firmware_product_name = "cmc";
>>>> +                                firmware_branch_name = "u50";
>>>> +                                firmware_version_major = <0x01>;
>>>> +                                firmware_version_minor = <0x00>;
>>>> +                        };
>>>> +                };
>>>> +
>>>> +                ep_cmc_intc_00 {
>>>> +                        reg = <0x00 0x1e03000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
>>>> +                        interrupts = <0x04 0x04>;
>>>> +                };
>>>> +
>>>> +                ep_cmc_mutex_00 {
>>>> +                        reg = <0x00 0x1e02000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>>>> +                };
>>>> +
>>>> +                ep_cmc_regmap_00 {
>>>> +                        reg = <0x00 0x1e08000 0x00 0x2000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
>>>> +
>>>> +                        firmware {
>>>> +                                firmware_product_name = "sc-fw";
>>>> +                                firmware_branch_name = "u50";
>>>> +                                firmware_version_major = <0x05>;
>>>> +                        };
>>>> +                };
>>>> +
>>>> +                ep_cmc_reset_00 {
>>>> +                        reg = <0x00 0x1e01000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>>>> +                };
>>>> +
>>>> +                ep_ddr_mem_calib_00 {
>>>> +                        reg = <0x00 0x63000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>>>> +                };
>>>> +
>>>> +                ep_debug_bscan_mgmt_00 {
>>>> +                        reg = <0x00 0x1e90000 0x00 0x10000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-debug_bridge-1.0\0debug_bridge";
>>>> +                };
>>>> +
>>>> +                ep_ert_base_address_00 {
>>>> +                        reg = <0x00 0x21000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>>>> +                };
>>>> +
>>>> +                ep_ert_command_queue_mgmt_00 {
>>>> +                        reg = <0x00 0x40000 0x00 0x10000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
>>>> +                };
>>>> +
>>>> +                ep_ert_command_queue_user_00 {
>>>> +                        reg = <0x00 0x40000 0x00 0x10000>;
>>>> +                        pcie_physical_function = <0x01>;
>>>> +                        compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
>>>> +                };
>>>> +
>>>> +                ep_ert_firmware_mem_00 {
>>>> +                        reg = <0x00 0x30000 0x00 0x8000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
>>>> +
>>>> +                        firmware {
>>>> +                                firmware_product_name = "ert";
>>>> +                                firmware_branch_name = "v20";
>>>> +                                firmware_version_major = <0x01>;
>>>> +                        };
>>>> +                };
>>>> +
>>>> +                ep_ert_intc_00 {
>>>> +                        reg = <0x00 0x23000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
>>>> +                        interrupts = <0x05 0x05>;
>>>> +                };
>>>> +
>>>> +                ep_ert_reset_00 {
>>>> +                        reg = <0x00 0x22000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>>>> +                };
>>>> +
>>>> +                ep_ert_sched_00 {
>>>> +                        reg = <0x00 0x50000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x01>;
>>>> +                        compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
>>>> +                        interrupts = <0x09 0x0c>;
>>>> +                };
>>>> +
>>>> +                ep_fpga_configuration_00 {
>>>> +                        reg = <0x00 0x1e88000 0x00 0x8000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
>>>> +                        interrupts = <0x02 0x02>;
>>>> +                };
>>>> +
>>>> +                ep_icap_reset_00 {
>>>> +                        reg = <0x00 0x1f07000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>>>> +                };
>>>> +
>>>> +                ep_msix_00 {
>>>> +                        reg = <0x00 0x00 0x00 0x20000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
>>>> +                        pcie_bar_mapping = <0x02>;
>>>> +                };
>>>> +
>>>> +                ep_pcie_link_mon_00 {
>>>> +                        reg = <0x00 0x1f05000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>>>> +                };
>>>> +
>>>> +                ep_pr_isolate_plp_00 {
>>>> +                        reg = <0x00 0x1f01000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>>>> +                };
>>>> +
>>>> +                ep_pr_isolate_ulp_00 {
>>>> +                        reg = <0x00 0x1000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
>>>> +                };
>>>> +
>>>> +                ep_uuid_rom_00 {
>>>> +                        reg = <0x00 0x64000 0x00 0x1000>;
>>>> +                        pcie_physical_function = <0x00>;
>>>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
>>>> +                };
>>>> +
>>>> +                ep_xdma_00 {
>>>> +                        reg = <0x00 0x00 0x00 0x10000>;
>>>> +                        pcie_physical_function = <0x01>;
>>>> +                        compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
>>>> +                        pcie_bar_mapping = <0x02>;
>>>> +                };
>>>> +        };
>>>> +
>>>> +  }
>>>> +
>>>> +
>>>> +
>>>> +Deployment Models
>>>> +=================
>>>> +
>>>> +Baremetal
>>>> +---------
>>>> +
>>>> +In bare-metal deployments, both MPF and UPF are visible and accessible. The
>>>> +xrt-mgmt driver binds to MPF. The xrt-mgmt driver operations are privileged and
>>>> +available to system administrator. The full stack is illustrated below::
>>>> +
>>>> +                            HOST
>>>> +
>>>> +               [XRT-MGMT]         [XRT-USER]
>>>> +                    |                  |
>>>> +                    |                  |
>>>> +                 +-----+            +-----+
>>>> +                 | MPF |            | UPF |
>>>> +                 |     |            |     |
>>>> +                 | PF0 |            | PF1 |
>>>> +                 +--+--+            +--+--+
>>>> +          ......... ^................. ^..........
>>>> +                    |                  |
>>>> +                    |   PCIe DEVICE    |
>>>> +                    |                  |
>>>> +                 +--+------------------+--+
>>>> +                 |         SHELL          |
>>>> +                 |                        |
>>>> +                 +------------------------+
>>>> +                 |         USER           |
>>>> +                 |                        |
>>>> +                 |                        |
>>>> +                 |                        |
>>>> +                 |                        |
>>>> +                 +------------------------+
>>>> +
>>>> +
>>>> +
>>>> +Virtualized
>>>> +-----------
>>>> +
>>>> +In virtualized deployments, the privileged MPF is assigned to the host but the
>>>> +unprivileged UPF is assigned to a guest VM via PCIe pass-through. The xrt-mgmt
>>>> +driver in host binds to MPF. The xrt-mgmt driver operations are privileged and
>>>> +only accessible to the MPF. The full stack is illustrated below::
>>>> +
>>>> +
>>>> +                                 ..............
>>>> +                  HOST           .    VM      .
>>>> +                                 .            .
>>>> +               [XRT-MGMT]        . [XRT-USER] .
>>>> +                    |            .     |      .
>>>> +                    |            .     |      .
>>>> +                 +-----+         .  +-----+   .
>>>> +                 | MPF |         .  | UPF |   .
>>>> +                 |     |         .  |     |   .
>>>> +                 | PF0 |         .  | PF1 |   .
>>>> +                 +--+--+         .  +--+--+   .
>>>> +          ......... ^................. ^..........
>>>> +                    |                  |
>>>> +                    |   PCIe DEVICE    |
>>>> +                    |                  |
>>>> +                 +--+------------------+--+
>>>> +                 |         SHELL          |
>>>> +                 |                        |
>>>> +                 +------------------------+
>>>> +                 |         USER           |
>>>> +                 |                        |
>>>> +                 |                        |
>>>> +                 |                        |
>>>> +                 |                        |
>>>> +                 +------------------------+
>>>> +
>>>> +
>>>> +
>>>> +
>>>> +
>>>> +Platform Security Considerations
>>>> +================================
>>>> +
>>>> +`Security of Alveo Platform <https://xilinx.github.io/XRT/master/html/security.html>`_
>>>> +discusses the deployment options and security implications in great detail.
>>>> diff --git a/MAINTAINERS b/MAINTAINERS
>>>> index 056966c9aac9..beeaf0257364 100644
>>>> --- a/MAINTAINERS
>>>> +++ b/MAINTAINERS
>>>> @@ -7274,6 +7274,17 @@ F:     Documentation/fpga/
>>>>    F:   drivers/fpga/
>>>>    F:   include/linux/fpga/
>>>>
>>>> +FPGA XRT DRIVERS
>>>> +M:   Lizhi Hou <[e-mail]>
>>>> +R:   Max Zhen <[e-mail]>
>>>> +R:   Sonal Santan <[e-mail]>
>>>> +L:   [e-mail]
>>>> +S:   Supported
>>>> +W:   https://github.com/Xilinx/XRT
>>>> +F:   Documentation/fpga/xrt.rst
>>>> +F:   drivers/fpga/xrt/
>>>> +F:   include/uapi/linux/xrt/
>>>> +
>>>>    FPU EMULATOR
>>>>    M:   Bill Metzenthen <[e-mail]>
>>>>    S:   Maintained
>>>> --
>>>> 2.27.0

Sonal Santan Oct. 19, 2021, 4:32 a.m. UTC | #5

Hello Rob,

> -----Original Message-----
> From: Lizhi Hou <[e-mail]>
> Sent: Thursday, October 14, 2021 9:13 AM
> To: [e-mail]
> Cc: [e-mail]; [e-mail]; Max Zhen
> <[e-mail]>; Sonal Santan <[e-mail]>; Yu Liu
> <[e-mail]>; Michal Simek <[e-mail]>; Stefano Stabellini
> <[e-mail]>; [e-mail]; [e-mail];
> [e-mail]; Max Zhen <[e-mail]>; Lizhi Hou <[e-mail]>;
> Xu Yilun <[e-mail]>
> Subject: Re: [PATCH V9 XRT Alveo 01/14] Documentation: fpga: Add a
> document describing XRT Alveo drivers
> 
> Hello Rob,
> 
> Please help with the review of the proposed FDT usage by Alveo/XRT drivers.
> 
> Thanks,
> Lizhi
> 
> On 10/13/21 7:21 PM, Xu Yilun wrote:
> > CAUTION: This message has originated from an External Source. Please use
> proper judgment and caution when opening attachments, clicking links, or
> responding to this email.
> >
> >
> >>>> +.. _device_tree_usage:
> >>>> +
> >>>> +Device Tree Usage
> >>>> +-----------------
> >>>> +
> >>>> +The xsabin file stores metadata which advertise HW subsystems
> >>>> +present in a partition. The metadata is stored in device tree
> >>>> +format with a well defined schema. XRT management driver uses this
> >>>> +information to bind *xrt_drivers* to the subsystem instantiations.
> >>>> +The xrt_drivers are found in **xrt-lib.ko** kernel module.
> >>> I'm still catching up the patchset from the very beginning, and just
> >>> finished the Documentation part. So far, I see the DT usage concern
> >>> which may impact the architecure a lot, so I should raise it ASAP.
> >>>
> >>> The concern raised by the DT maintainer:
> >>> https://lore.kernel.org/linux-fpga/CAL_JsqLod6FBGFhu7WXtMrB_z7wj8-up
> >>> [e-mail]/
> >>>
> >>> First of all, directly parsing FDT in device drivers is not a normal
> >>> usage of DT in linux. It is out of the current DT usage model. So it
> >>> should be agreed by DT maintainers.

Friendly reminder that we are waiting for your feedback on the proposed FDT use model by the Alveo/XRT drivers. Please advise on how to proceed forward.

Thanks,
-Sonal
> >> Thanks for reviewing XRT document and providing feedback.
> >> Here is the reply from Sonal for Rob’s question:
> >> https://lore.kernel.org/linux-
> fpga/BY5PR02MB62604B87C66A1AD139A6F153B
> >> [e-mail]/
> >> Overall, libfdt is used by XRT driver to parse the metadata which
> >> comes with an Alveo board.
> >> When XRT driver discovers an Alveo board, it will read a fdt blob
> >> from board firmware file resident on the host.
> >> By parsing the fdt blob, XRT driver gets information about this Alveo
> >> board, such as version, uuid, IPs exposed to PCI BAR, interrupt binding etc.
> >> So libfdt is used simply as Alveo metadata parser here. XRT drivers
> >> do not interact with system wide DT or present the Alveo device tree
> >> to host. For many systems like x86_64, system wide DT is not present
> >> but libfdt parsing services will still be needed.
> > Yes, I understand the use case.
> >
> > My concern is, directly parsing an isolated FDT in device driver and
> > populate sub devices, skipping the unflattening, this is a new working
> > model of device tree usage, but for the same purpose as the existing
> > one.
> >
> > So I really need the confirmation of DT maintainers.
> >
> >>> Current FPGA framework modifies kernel's live tree by DT overlay,
> >>> when FPGA is dynamically reprogrammed and new HW devices appear. See
> >>> Documentation/devicetree/bindings/fpga/fpga-region.txt.
> >>>
> >>> Then something less important:
> >>>
> >>>     1. The bindings should be documented in
> Documentation/devicetree/bindings/.
> >>>     2. Are all the example DT usage conform to the exsiting bindings? I
> >>>        didn't go through all device classes, but remember like the
> >>>        interrupt-controller should have a "interrupt-controller" property, and
> >>>        the PCI properties are also different from PCI bindings.
> >> The fdt properties are defined for Alveo firmware files. XRT driver
> >> is the only consumer of this data. I am wondering if
> > Personally I don't like the idea of a different binding definition
> > set, even if the new device tree usage is accepted. We all use device
> > tree for device enumeration, so why the different definitions.
> >
> > Thanks,
> > Yilun
> >
> >> Documentation/devicetree/bindings is the right place for Alveo/XRT
> >> private format or should it be documented as part of XRT driver
> documentation?
> >> Looking for guidance here.
> >>
> >>
> >> Thanks,
> >>
> >> Lizhi
> >>
> >>> Thanks,
> >>> Yilun
> >>>
> >>>> +
> >>>> +Logic UUID
> >>>> +^^^^^^^^^^
> >>>> +A partition is identified uniquely through ``logic_uuid`` property::
> >>>> +
> >>>> +  /dts-v1/;
> >>>> +  / {
> >>>> +      logic_uuid = "0123456789abcdef0123456789abcdef";
> >>>> +      ...
> >>>> +    }
> >>>> +
> >>>> +Schema Version
> >>>> +^^^^^^^^^^^^^^
> >>>> +Schema version is defined through the ``schema_version`` node. It
> >>>> +contains ``major`` and ``minor`` properties as below::
> >>>> +
> >>>> +  /dts-v1/;
> >>>> +  / {
> >>>> +       schema_version {
> >>>> +           major = <0x01>;
> >>>> +           minor = <0x00>;
> >>>> +       };
> >>>> +       ...
> >>>> +    }
> >>>> +
> >>>> +.. _partition_uuids:
> >>>> +
> >>>> +Partition UUIDs
> >>>> +^^^^^^^^^^^^^^^
> >>>> +Each partition may have parent and child UUIDs. These UUIDs are
> >>>> +defined by ``interfaces`` node and ``interface_uuid`` property::
> >>>> +
> >>>> +  /dts-v1/;
> >>>> +  / {
> >>>> +       interfaces {
> >>>> +           @0 {
> >>>> +                  interface_uuid = "0123456789abcdef0123456789abcdef";
> >>>> +           };
> >>>> +           @1 {
> >>>> +                  interface_uuid = "fedcba9876543210fedcba9876543210";
> >>>> +           };
> >>>> +           ...
> >>>> +        };
> >>>> +       ...
> >>>> +    }
> >>>> +
> >>>> +
> >>>> +Subsystem Instantiations
> >>>> +^^^^^^^^^^^^^^^^^^^^^^^^
> >>>> +Subsystem instantiations are captured as children of
> >>>> +``addressable_endpoints``
> >>>> +node::
> >>>> +
> >>>> +  /dts-v1/;
> >>>> +  / {
> >>>> +       addressable_endpoints {
> >>>> +           abc {
> >>>> +               ...
> >>>> +           };
> >>>> +           def {
> >>>> +               ...
> >>>> +           };
> >>>> +           ...
> >>>> +       }
> >>>> +  }
> >>>> +
> >>>> +Subnode 'abc' and 'def' are the name of subsystem nodes
> >>>> +
> >>>> +Subsystem Node
> >>>> +^^^^^^^^^^^^^^
> >>>> +Each subsystem node and its properties define a hardware instance::
> >>>> +
> >>>> +
> >>>> +  addressable_endpoints {
> >>>> +      abc {
> >>>> +          reg = <0x00 0x1f05000 0x00 0x1000>>
> >>>> +          pcie_physical_function = <0x0>;
> >>>> +          pcie_bar_mapping = <0x2>;
> >>>> +          compatible = "abc def";
> >>>> +          interrupts = <0x09 0x0c>;
> >>>> +          firmware {
> >>>> +              firmware_product_name = "abc"
> >>>> +              firmware_branch_name = "def"
> >>>> +              firmware_version_major = <1>
> >>>> +              firmware_version_minor = <2>
> >>>> +          };
> >>>> +      }
> >>>> +      ...
> >>>> +  }
> >>>> +
> >>>> +:reg:
> >>>> + Property defines an address range. `<0x00 0x1f05000 0x00 0x1000>`
> >>>> +indicates
> >>>> + *0x00 0x1f05000* as BAR offset and *0x00 0x1000* as address length.
> >>>> +:pcie_physical_function:
> >>>> + Property specifies which PCIe physical function the subsystem node
> resides.
> >>>> + `<0x0>` implies physical function 0.
> >>>> +:pcie_bar_mapping:
> >>>> + Property specifies which PCIe BAR the subsystem node resides.
> >>>> +`<0x2>` implies  BAR 2. A value of 0 means the property is not defined.
> >>>> +:compatible:
> >>>> + Property is a list of strings. The first string in the list
> >>>> +specifies the exact  subsystem node. The following strings
> >>>> +represent other devices that the device  is compatible with.
> >>>> +:interrupts:
> >>>> + Property specifies start and end interrupts for this subsystem node.
> >>>> + `<0x09 0x0c>` implies interrupts 9 to 13 are used by this subsystem.
> >>>> +:firmware:
> >>>> + Subnode defines the firmware required by this subsystem node.
> >>>> +
> >>>> +Alveo U50 Platform Example
> >>>> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> >>>> +::
> >>>> +
> >>>> +  /dts-v1/;
> >>>> +
> >>>> +  /{
> >>>> +        logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
> >>>> +
> >>>> +        schema_version {
> >>>> +                major = <0x01>;
> >>>> +                minor = <0x00>;
> >>>> +        };
> >>>> +
> >>>> +        interfaces {
> >>>> +
> >>>> +                @0 {
> >>>> +                        interface_uuid =
> "862c7020a250293e32036f19956669e5";
> >>>> +                };
> >>>> +        };
> >>>> +
> >>>> +        addressable_endpoints {
> >>>> +
> >>>> +                ep_blp_rom_00 {
> >>>> +                        reg = <0x00 0x1f04000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-
> 1.0\0axi_bram_ctrl";
> >>>> +                };
> >>>> +
> >>>> +                ep_card_flash_program_00 {
> >>>> +                        reg = <0x00 0x1f06000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_quad_spi-
> 1.0\0axi_quad_spi";
> >>>> +                        interrupts = <0x03 0x03>;
> >>>> +                };
> >>>> +
> >>>> +                ep_cmc_firmware_mem_00 {
> >>>> +                        reg = <0x00 0x1e20000 0x00 0x20000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible =
> >>>> + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> >>>> +
> >>>> +                        firmware {
> >>>> +                                firmware_product_name = "cmc";
> >>>> +                                firmware_branch_name = "u50";
> >>>> +                                firmware_version_major = <0x01>;
> >>>> +                                firmware_version_minor = <0x00>;
> >>>> +                        };
> >>>> +                };
> >>>> +
> >>>> +                ep_cmc_intc_00 {
> >>>> +                        reg = <0x00 0x1e03000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> >>>> +                        interrupts = <0x04 0x04>;
> >>>> +                };
> >>>> +
> >>>> +                ep_cmc_mutex_00 {
> >>>> +                        reg = <0x00 0x1e02000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_cmc_regmap_00 {
> >>>> +                        reg = <0x00 0x1e08000 0x00 0x2000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible =
> >>>> + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> >>>> +
> >>>> +                        firmware {
> >>>> +                                firmware_product_name = "sc-fw";
> >>>> +                                firmware_branch_name = "u50";
> >>>> +                                firmware_version_major = <0x05>;
> >>>> +                        };
> >>>> +                };
> >>>> +
> >>>> +                ep_cmc_reset_00 {
> >>>> +                        reg = <0x00 0x1e01000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_ddr_mem_calib_00 {
> >>>> +                        reg = <0x00 0x63000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_debug_bscan_mgmt_00 {
> >>>> +                        reg = <0x00 0x1e90000 0x00 0x10000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-debug_bridge-
> 1.0\0debug_bridge";
> >>>> +                };
> >>>> +
> >>>> +                ep_ert_base_address_00 {
> >>>> +                        reg = <0x00 0x21000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_ert_command_queue_mgmt_00 {
> >>>> +                        reg = <0x00 0x40000 0x00 0x10000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-ert_command_queue-
> 1.0\0ert_command_queue";
> >>>> +                };
> >>>> +
> >>>> +                ep_ert_command_queue_user_00 {
> >>>> +                        reg = <0x00 0x40000 0x00 0x10000>;
> >>>> +                        pcie_physical_function = <0x01>;
> >>>> +                        compatible = "xilinx.com,reg_abs-ert_command_queue-
> 1.0\0ert_command_queue";
> >>>> +                };
> >>>> +
> >>>> +                ep_ert_firmware_mem_00 {
> >>>> +                        reg = <0x00 0x30000 0x00 0x8000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible =
> >>>> + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> >>>> +
> >>>> +                        firmware {
> >>>> +                                firmware_product_name = "ert";
> >>>> +                                firmware_branch_name = "v20";
> >>>> +                                firmware_version_major = <0x01>;
> >>>> +                        };
> >>>> +                };
> >>>> +
> >>>> +                ep_ert_intc_00 {
> >>>> +                        reg = <0x00 0x23000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> >>>> +                        interrupts = <0x05 0x05>;
> >>>> +                };
> >>>> +
> >>>> +                ep_ert_reset_00 {
> >>>> +                        reg = <0x00 0x22000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_ert_sched_00 {
> >>>> +                        reg = <0x00 0x50000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x01>;
> >>>> +                        compatible = "xilinx.com,reg_abs-ert_sched-
> 1.0\0ert_sched";
> >>>> +                        interrupts = <0x09 0x0c>;
> >>>> +                };
> >>>> +
> >>>> +                ep_fpga_configuration_00 {
> >>>> +                        reg = <0x00 0x1e88000 0x00 0x8000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_hwicap-
> 1.0\0axi_hwicap";
> >>>> +                        interrupts = <0x02 0x02>;
> >>>> +                };
> >>>> +
> >>>> +                ep_icap_reset_00 {
> >>>> +                        reg = <0x00 0x1f07000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_msix_00 {
> >>>> +                        reg = <0x00 0x00 0x00 0x20000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
> >>>> +                        pcie_bar_mapping = <0x02>;
> >>>> +                };
> >>>> +
> >>>> +                ep_pcie_link_mon_00 {
> >>>> +                        reg = <0x00 0x1f05000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_pr_isolate_plp_00 {
> >>>> +                        reg = <0x00 0x1f01000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_pr_isolate_ulp_00 {
> >>>> +                        reg = <0x00 0x1000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_uuid_rom_00 {
> >>>> +                        reg = <0x00 0x64000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-
> 1.0\0axi_bram_ctrl";
> >>>> +                };
> >>>> +
> >>>> +                ep_xdma_00 {
> >>>> +                        reg = <0x00 0x00 0x00 0x10000>;
> >>>> +                        pcie_physical_function = <0x01>;
> >>>> +                        compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
> >>>> +                        pcie_bar_mapping = <0x02>;
> >>>> +                };
> >>>> +        };
> >>>> +
> >>>> +  }
> >>>> +
> >>>> +
> >>>> +
> >>>> +Deployment Models
> >>>> +=================
> >>>> +
> >>>> +Baremetal
> >>>> +---------
> >>>> +
> >>>> +In bare-metal deployments, both MPF and UPF are visible and
> >>>> +accessible. The xrt-mgmt driver binds to MPF. The xrt-mgmt driver
> >>>> +operations are privileged and available to system administrator. The full
> stack is illustrated below::
> >>>> +
> >>>> +                            HOST
> >>>> +
> >>>> +               [XRT-MGMT]         [XRT-USER]
> >>>> +                    |                  |
> >>>> +                    |                  |
> >>>> +                 +-----+            +-----+
> >>>> +                 | MPF |            | UPF |
> >>>> +                 |     |            |     |
> >>>> +                 | PF0 |            | PF1 |
> >>>> +                 +--+--+            +--+--+
> >>>> +          ......... ^................. ^..........
> >>>> +                    |                  |
> >>>> +                    |   PCIe DEVICE    |
> >>>> +                    |                  |
> >>>> +                 +--+------------------+--+
> >>>> +                 |         SHELL          |
> >>>> +                 |                        |
> >>>> +                 +------------------------+
> >>>> +                 |         USER           |
> >>>> +                 |                        |
> >>>> +                 |                        |
> >>>> +                 |                        |
> >>>> +                 |                        |
> >>>> +                 +------------------------+
> >>>> +
> >>>> +
> >>>> +
> >>>> +Virtualized
> >>>> +-----------
> >>>> +
> >>>> +In virtualized deployments, the privileged MPF is assigned to the
> >>>> +host but the unprivileged UPF is assigned to a guest VM via PCIe
> >>>> +pass-through. The xrt-mgmt driver in host binds to MPF. The
> >>>> +xrt-mgmt driver operations are privileged and only accessible to the
> MPF. The full stack is illustrated below::
> >>>> +
> >>>> +
> >>>> +                                 ..............
> >>>> +                  HOST           .    VM      .
> >>>> +                                 .            .
> >>>> +               [XRT-MGMT]        . [XRT-USER] .
> >>>> +                    |            .     |      .
> >>>> +                    |            .     |      .
> >>>> +                 +-----+         .  +-----+   .
> >>>> +                 | MPF |         .  | UPF |   .
> >>>> +                 |     |         .  |     |   .
> >>>> +                 | PF0 |         .  | PF1 |   .
> >>>> +                 +--+--+         .  +--+--+   .
> >>>> +          ......... ^................. ^..........
> >>>> +                    |                  |
> >>>> +                    |   PCIe DEVICE    |
> >>>> +                    |                  |
> >>>> +                 +--+------------------+--+
> >>>> +                 |         SHELL          |
> >>>> +                 |                        |
> >>>> +                 +------------------------+
> >>>> +                 |         USER           |
> >>>> +                 |                        |
> >>>> +                 |                        |
> >>>> +                 |                        |
> >>>> +                 |                        |
> >>>> +                 +------------------------+
> >>>> +
> >>>> +
> >>>> +
> >>>> +
> >>>> +
> >>>> +Platform Security Considerations
> >>>> +================================
> >>>> +
> >>>> +`Security of Alveo Platform
> >>>> +<https://xilinx.github.io/XRT/master/html/security.html>`_
> >>>> +discusses the deployment options and security implications in great
> detail.
> >>>> diff --git a/MAINTAINERS b/MAINTAINERS index
> >>>> 056966c9aac9..beeaf0257364 100644
> >>>> --- a/MAINTAINERS
> >>>> +++ b/MAINTAINERS
> >>>> @@ -7274,6 +7274,17 @@ F:     Documentation/fpga/
> >>>>    F:   drivers/fpga/
> >>>>    F:   include/linux/fpga/
> >>>>
> >>>> +FPGA XRT DRIVERS
> >>>> +M:   Lizhi Hou <[e-mail]>
> >>>> +R:   Max Zhen <[e-mail]>
> >>>> +R:   Sonal Santan <[e-mail]>
> >>>> +L:   [e-mail]
> >>>> +S:   Supported
> >>>> +W:   https://github.com/Xilinx/XRT
> >>>> +F:   Documentation/fpga/xrt.rst
> >>>> +F:   drivers/fpga/xrt/
> >>>> +F:   include/uapi/linux/xrt/
> >>>> +
> >>>>    FPU EMULATOR
> >>>>    M:   Bill Metzenthen <[e-mail]>
> >>>>    S:   Maintained
> >>>> --
> >>>> 2.27.0

Rob Herring (Arm) Oct. 19, 2021, 1:02 p.m. UTC | #6

On Thu, Oct 14, 2021 at 11:12 AM Lizhi Hou <[e-mail]> wrote:
>
> Hello Rob,
>
> Please help with the review of the proposed FDT usage by Alveo/XRT drivers.
>
> Thanks,
> Lizhi
>
> On 10/13/21 7:21 PM, Xu Yilun wrote:
> > CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
> >
> >
> >>>> +.. _device_tree_usage:
> >>>> +
> >>>> +Device Tree Usage
> >>>> +-----------------
> >>>> +
> >>>> +The xsabin file stores metadata which advertise HW subsystems present in a
> >>>> +partition. The metadata is stored in device tree format with a well defined
> >>>> +schema. XRT management driver uses this information to bind *xrt_drivers* to
> >>>> +the subsystem instantiations. The xrt_drivers are found in **xrt-lib.ko** kernel
> >>>> +module.
> >>> I'm still catching up the patchset from the very beginning, and just
> >>> finished the Documentation part. So far, I see the DT usage concern
> >>> which may impact the architecure a lot, so I should raise it ASAP.
> >>>
> >>> The concern raised by the DT maintainer:
> >>> https://lore.kernel.org/linux-fpga/[e-mail]/
> >>>
> >>> First of all, directly parsing FDT in device drivers is not a normal usage of DT
> >>> in linux. It is out of the current DT usage model. So it should be agreed by DT
> >>> maintainers.
> >> Thanks for reviewing XRT document and providing feedback.
> >> Here is the reply from Sonal for Rob’s question:
> >> https://lore.kernel.org/linux-fpga/[e-mail]/
> >> Overall, libfdt is used by XRT driver to parse the metadata which comes with
> >> an Alveo board.
> >> When XRT driver discovers an Alveo board, it will read a fdt blob from board
> >> firmware file resident on the host.
> >> By parsing the fdt blob, XRT driver gets information about this Alveo board,
> >> such as version, uuid, IPs exposed to PCI BAR, interrupt binding etc.
> >> So libfdt is used simply as Alveo metadata parser here. XRT drivers do not
> >> interact with system wide DT or present the Alveo device tree to host. For
> >> many systems like x86_64, system wide DT is not present but libfdt parsing
> >> services will still be needed.
> > Yes, I understand the use case.
> >
> > My concern is, directly parsing an isolated FDT in device driver and
> > populate sub devices, skipping the unflattening, this is a new working
> > model of device tree usage, but for the same purpose as the existing
> > one.
> >
> > So I really need the confirmation of DT maintainers.

Perhaps you could explain why you think you need to use FDT instead of
unflattening. Without that, the answer is don't use FDT.

> >
> >>> Current FPGA framework modifies kernel's live tree by DT overlay, when FPGA is
> >>> dynamically reprogrammed and new HW devices appear. See
> >>> Documentation/devicetree/bindings/fpga/fpga-region.txt.
> >>>
> >>> Then something less important:
> >>>
> >>>     1. The bindings should be documented in Documentation/devicetree/bindings/.

Yes.

> >>>     2. Are all the example DT usage conform to the exsiting bindings? I
> >>>        didn't go through all device classes, but remember like the
> >>>        interrupt-controller should have a "interrupt-controller" property, and
> >>>        the PCI properties are also different from PCI bindings.

They don't, but should. I can't tell what you are trying to do here,
but it looks like a mess.

> >> The fdt properties are defined for Alveo firmware files. XRT driver is the
> >> only consumer of this data. I am wondering if
> > Personally I don't like the idea of a different binding definition set,
> > even if the new device tree usage is accepted. We all use device tree
> > for device enumeration, so why the different definitions.
> >
> > Thanks,
> > Yilun
> >
> >> Documentation/devicetree/bindings is the right place for Alveo/XRT private
> >> format or should it be documented as part of XRT driver documentation?
> >> Looking for guidance here.
> >>
> >>
> >> Thanks,
> >>
> >> Lizhi
> >>
> >>> Thanks,
> >>> Yilun
> >>>
> >>>> +
> >>>> +Logic UUID
> >>>> +^^^^^^^^^^
> >>>> +A partition is identified uniquely through ``logic_uuid`` property::
> >>>> +
> >>>> +  /dts-v1/;
> >>>> +  / {
> >>>> +      logic_uuid = "0123456789abcdef0123456789abcdef";
> >>>> +      ...
> >>>> +    }
> >>>> +
> >>>> +Schema Version
> >>>> +^^^^^^^^^^^^^^
> >>>> +Schema version is defined through the ``schema_version`` node. It contains
> >>>> +``major`` and ``minor`` properties as below::
> >>>> +
> >>>> +  /dts-v1/;
> >>>> +  / {
> >>>> +       schema_version {
> >>>> +           major = <0x01>;
> >>>> +           minor = <0x00>;
> >>>> +       };
> >>>> +       ...
> >>>> +    }
> >>>> +
> >>>> +.. _partition_uuids:
> >>>> +
> >>>> +Partition UUIDs
> >>>> +^^^^^^^^^^^^^^^
> >>>> +Each partition may have parent and child UUIDs. These UUIDs are
> >>>> +defined by ``interfaces`` node and ``interface_uuid`` property::
> >>>> +
> >>>> +  /dts-v1/;
> >>>> +  / {
> >>>> +       interfaces {
> >>>> +           @0 {
> >>>> +                  interface_uuid = "0123456789abcdef0123456789abcdef";
> >>>> +           };
> >>>> +           @1 {
> >>>> +                  interface_uuid = "fedcba9876543210fedcba9876543210";
> >>>> +           };
> >>>> +           ...
> >>>> +        };
> >>>> +       ...
> >>>> +    }
> >>>> +
> >>>> +
> >>>> +Subsystem Instantiations
> >>>> +^^^^^^^^^^^^^^^^^^^^^^^^
> >>>> +Subsystem instantiations are captured as children of ``addressable_endpoints``
> >>>> +node::
> >>>> +
> >>>> +  /dts-v1/;
> >>>> +  / {
> >>>> +       addressable_endpoints {
> >>>> +           abc {
> >>>> +               ...
> >>>> +           };
> >>>> +           def {
> >>>> +               ...
> >>>> +           };
> >>>> +           ...
> >>>> +       }
> >>>> +  }
> >>>> +
> >>>> +Subnode 'abc' and 'def' are the name of subsystem nodes
> >>>> +
> >>>> +Subsystem Node
> >>>> +^^^^^^^^^^^^^^
> >>>> +Each subsystem node and its properties define a hardware instance::
> >>>> +
> >>>> +
> >>>> +  addressable_endpoints {
> >>>> +      abc {
> >>>> +          reg = <0x00 0x1f05000 0x00 0x1000>>
> >>>> +          pcie_physical_function = <0x0>;
> >>>> +          pcie_bar_mapping = <0x2>;
> >>>> +          compatible = "abc def";
> >>>> +          interrupts = <0x09 0x0c>;
> >>>> +          firmware {
> >>>> +              firmware_product_name = "abc"
> >>>> +              firmware_branch_name = "def"
> >>>> +              firmware_version_major = <1>
> >>>> +              firmware_version_minor = <2>
> >>>> +          };
> >>>> +      }
> >>>> +      ...
> >>>> +  }
> >>>> +
> >>>> +:reg:
> >>>> + Property defines an address range. `<0x00 0x1f05000 0x00 0x1000>` indicates
> >>>> + *0x00 0x1f05000* as BAR offset and *0x00 0x1000* as address length.
> >>>> +:pcie_physical_function:
> >>>> + Property specifies which PCIe physical function the subsystem node resides.
> >>>> + `<0x0>` implies physical function 0.
> >>>> +:pcie_bar_mapping:
> >>>> + Property specifies which PCIe BAR the subsystem node resides. `<0x2>` implies
> >>>> + BAR 2. A value of 0 means the property is not defined.
> >>>> +:compatible:
> >>>> + Property is a list of strings. The first string in the list specifies the exact
> >>>> + subsystem node. The following strings represent other devices that the device
> >>>> + is compatible with.
> >>>> +:interrupts:
> >>>> + Property specifies start and end interrupts for this subsystem node.
> >>>> + `<0x09 0x0c>` implies interrupts 9 to 13 are used by this subsystem.
> >>>> +:firmware:
> >>>> + Subnode defines the firmware required by this subsystem node.
> >>>> +
> >>>> +Alveo U50 Platform Example
> >>>> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> >>>> +::
> >>>> +
> >>>> +  /dts-v1/;
> >>>> +
> >>>> +  /{
> >>>> +        logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
> >>>> +
> >>>> +        schema_version {
> >>>> +                major = <0x01>;
> >>>> +                minor = <0x00>;
> >>>> +        };
> >>>> +
> >>>> +        interfaces {
> >>>> +
> >>>> +                @0 {
> >>>> +                        interface_uuid = "862c7020a250293e32036f19956669e5";
> >>>> +                };
> >>>> +        };
> >>>> +
> >>>> +        addressable_endpoints {
> >>>> +
> >>>> +                ep_blp_rom_00 {
> >>>> +                        reg = <0x00 0x1f04000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> >>>> +                };
> >>>> +
> >>>> +                ep_card_flash_program_00 {
> >>>> +                        reg = <0x00 0x1f06000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_quad_spi-1.0\0axi_quad_spi";
> >>>> +                        interrupts = <0x03 0x03>;
> >>>> +                };
> >>>> +
> >>>> +                ep_cmc_firmware_mem_00 {
> >>>> +                        reg = <0x00 0x1e20000 0x00 0x20000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> >>>> +
> >>>> +                        firmware {
> >>>> +                                firmware_product_name = "cmc";
> >>>> +                                firmware_branch_name = "u50";
> >>>> +                                firmware_version_major = <0x01>;
> >>>> +                                firmware_version_minor = <0x00>;
> >>>> +                        };
> >>>> +                };
> >>>> +
> >>>> +                ep_cmc_intc_00 {
> >>>> +                        reg = <0x00 0x1e03000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> >>>> +                        interrupts = <0x04 0x04>;
> >>>> +                };
> >>>> +
> >>>> +                ep_cmc_mutex_00 {
> >>>> +                        reg = <0x00 0x1e02000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_cmc_regmap_00 {
> >>>> +                        reg = <0x00 0x1e08000 0x00 0x2000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> >>>> +
> >>>> +                        firmware {
> >>>> +                                firmware_product_name = "sc-fw";
> >>>> +                                firmware_branch_name = "u50";
> >>>> +                                firmware_version_major = <0x05>;
> >>>> +                        };
> >>>> +                };
> >>>> +
> >>>> +                ep_cmc_reset_00 {
> >>>> +                        reg = <0x00 0x1e01000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_ddr_mem_calib_00 {
> >>>> +                        reg = <0x00 0x63000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_debug_bscan_mgmt_00 {
> >>>> +                        reg = <0x00 0x1e90000 0x00 0x10000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-debug_bridge-1.0\0debug_bridge";
> >>>> +                };
> >>>> +
> >>>> +                ep_ert_base_address_00 {
> >>>> +                        reg = <0x00 0x21000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_ert_command_queue_mgmt_00 {
> >>>> +                        reg = <0x00 0x40000 0x00 0x10000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
> >>>> +                };
> >>>> +
> >>>> +                ep_ert_command_queue_user_00 {
> >>>> +                        reg = <0x00 0x40000 0x00 0x10000>;
> >>>> +                        pcie_physical_function = <0x01>;
> >>>> +                        compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
> >>>> +                };
> >>>> +
> >>>> +                ep_ert_firmware_mem_00 {
> >>>> +                        reg = <0x00 0x30000 0x00 0x8000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> >>>> +
> >>>> +                        firmware {
> >>>> +                                firmware_product_name = "ert";
> >>>> +                                firmware_branch_name = "v20";
> >>>> +                                firmware_version_major = <0x01>;
> >>>> +                        };
> >>>> +                };
> >>>> +
> >>>> +                ep_ert_intc_00 {
> >>>> +                        reg = <0x00 0x23000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> >>>> +                        interrupts = <0x05 0x05>;
> >>>> +                };
> >>>> +
> >>>> +                ep_ert_reset_00 {
> >>>> +                        reg = <0x00 0x22000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_ert_sched_00 {
> >>>> +                        reg = <0x00 0x50000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x01>;
> >>>> +                        compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
> >>>> +                        interrupts = <0x09 0x0c>;
> >>>> +                };
> >>>> +
> >>>> +                ep_fpga_configuration_00 {
> >>>> +                        reg = <0x00 0x1e88000 0x00 0x8000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
> >>>> +                        interrupts = <0x02 0x02>;
> >>>> +                };
> >>>> +
> >>>> +                ep_icap_reset_00 {
> >>>> +                        reg = <0x00 0x1f07000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_msix_00 {
> >>>> +                        reg = <0x00 0x00 0x00 0x20000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
> >>>> +                        pcie_bar_mapping = <0x02>;
> >>>> +                };
> >>>> +
> >>>> +                ep_pcie_link_mon_00 {
> >>>> +                        reg = <0x00 0x1f05000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_pr_isolate_plp_00 {
> >>>> +                        reg = <0x00 0x1f01000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_pr_isolate_ulp_00 {
> >>>> +                        reg = <0x00 0x1000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> >>>> +                };
> >>>> +
> >>>> +                ep_uuid_rom_00 {
> >>>> +                        reg = <0x00 0x64000 0x00 0x1000>;
> >>>> +                        pcie_physical_function = <0x00>;
> >>>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> >>>> +                };
> >>>> +
> >>>> +                ep_xdma_00 {
> >>>> +                        reg = <0x00 0x00 0x00 0x10000>;
> >>>> +                        pcie_physical_function = <0x01>;
> >>>> +                        compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
> >>>> +                        pcie_bar_mapping = <0x02>;
> >>>> +                };
> >>>> +        };
> >>>> +
> >>>> +  }
> >>>> +
> >>>> +
> >>>> +
> >>>> +Deployment Models
> >>>> +=================
> >>>> +
> >>>> +Baremetal
> >>>> +---------
> >>>> +
> >>>> +In bare-metal deployments, both MPF and UPF are visible and accessible. The
> >>>> +xrt-mgmt driver binds to MPF. The xrt-mgmt driver operations are privileged and
> >>>> +available to system administrator. The full stack is illustrated below::
> >>>> +
> >>>> +                            HOST
> >>>> +
> >>>> +               [XRT-MGMT]         [XRT-USER]
> >>>> +                    |                  |
> >>>> +                    |                  |
> >>>> +                 +-----+            +-----+
> >>>> +                 | MPF |            | UPF |
> >>>> +                 |     |            |     |
> >>>> +                 | PF0 |            | PF1 |
> >>>> +                 +--+--+            +--+--+
> >>>> +          ......... ^................. ^..........
> >>>> +                    |                  |
> >>>> +                    |   PCIe DEVICE    |
> >>>> +                    |                  |
> >>>> +                 +--+------------------+--+
> >>>> +                 |         SHELL          |
> >>>> +                 |                        |
> >>>> +                 +------------------------+
> >>>> +                 |         USER           |
> >>>> +                 |                        |
> >>>> +                 |                        |
> >>>> +                 |                        |
> >>>> +                 |                        |
> >>>> +                 +------------------------+
> >>>> +
> >>>> +
> >>>> +
> >>>> +Virtualized
> >>>> +-----------
> >>>> +
> >>>> +In virtualized deployments, the privileged MPF is assigned to the host but the
> >>>> +unprivileged UPF is assigned to a guest VM via PCIe pass-through. The xrt-mgmt
> >>>> +driver in host binds to MPF. The xrt-mgmt driver operations are privileged and
> >>>> +only accessible to the MPF. The full stack is illustrated below::
> >>>> +
> >>>> +
> >>>> +                                 ..............
> >>>> +                  HOST           .    VM      .
> >>>> +                                 .            .
> >>>> +               [XRT-MGMT]        . [XRT-USER] .
> >>>> +                    |            .     |      .
> >>>> +                    |            .     |      .
> >>>> +                 +-----+         .  +-----+   .
> >>>> +                 | MPF |         .  | UPF |   .
> >>>> +                 |     |         .  |     |   .
> >>>> +                 | PF0 |         .  | PF1 |   .
> >>>> +                 +--+--+         .  +--+--+   .
> >>>> +          ......... ^................. ^..........
> >>>> +                    |                  |
> >>>> +                    |   PCIe DEVICE    |
> >>>> +                    |                  |
> >>>> +                 +--+------------------+--+
> >>>> +                 |         SHELL          |
> >>>> +                 |                        |
> >>>> +                 +------------------------+
> >>>> +                 |         USER           |
> >>>> +                 |                        |
> >>>> +                 |                        |
> >>>> +                 |                        |
> >>>> +                 |                        |
> >>>> +                 +------------------------+
> >>>> +
> >>>> +
> >>>> +
> >>>> +
> >>>> +
> >>>> +Platform Security Considerations
> >>>> +================================
> >>>> +
> >>>> +`Security of Alveo Platform <https://xilinx.github.io/XRT/master/html/security.html>`_
> >>>> +discusses the deployment options and security implications in great detail.
> >>>> diff --git a/MAINTAINERS b/MAINTAINERS
> >>>> index 056966c9aac9..beeaf0257364 100644
> >>>> --- a/MAINTAINERS
> >>>> +++ b/MAINTAINERS
> >>>> @@ -7274,6 +7274,17 @@ F:     Documentation/fpga/
> >>>>    F:   drivers/fpga/
> >>>>    F:   include/linux/fpga/
> >>>>
> >>>> +FPGA XRT DRIVERS
> >>>> +M:   Lizhi Hou <[e-mail]>
> >>>> +R:   Max Zhen <[e-mail]>
> >>>> +R:   Sonal Santan <[e-mail]>
> >>>> +L:   [e-mail]
> >>>> +S:   Supported
> >>>> +W:   https://github.com/Xilinx/XRT
> >>>> +F:   Documentation/fpga/xrt.rst
> >>>> +F:   drivers/fpga/xrt/
> >>>> +F:   include/uapi/linux/xrt/
> >>>> +
> >>>>    FPU EMULATOR
> >>>>    M:   Bill Metzenthen <[e-mail]>
> >>>>    S:   Maintained
> >>>> --
> >>>> 2.27.0

Sonal Santan Oct. 21, 2021, 5:36 a.m. UTC | #7

Hello Rob,

> -----Original Message-----
> From: Rob Herring <[e-mail]>
> Sent: Tuesday, October 19, 2021 6:03 AM
> To: Lizhi Hou <[e-mail]>
> Cc: [e-mail]; [e-mail]; Max Zhen
> <[e-mail]>; Sonal Santan <[e-mail]>; Yu Liu
> <[e-mail]>; Michal Simek <[e-mail]>; Stefano Stabellini
> <[e-mail]>; [e-mail]; [e-mail]; Moritz
> Fischer <[e-mail]>; Max Zhen <[e-mail]>; Xu Yilun
> <[e-mail]>
> Subject: Re: [PATCH V9 XRT Alveo 01/14] Documentation: fpga: Add a
> document describing XRT Alveo drivers
> 
> On Thu, Oct 14, 2021 at 11:12 AM Lizhi Hou <[e-mail]> wrote:
> >
> > Hello Rob,
> >
> > Please help with the review of the proposed FDT usage by Alveo/XRT drivers.
> >
> > Thanks,
> > Lizhi
> >
> > On 10/13/21 7:21 PM, Xu Yilun wrote:
> > > CAUTION: This message has originated from an External Source. Please use
> proper judgment and caution when opening attachments, clicking links, or
> responding to this email.
> > >
> > >
> > >>>> +.. _device_tree_usage:
> > >>>> +
> > >>>> +Device Tree Usage
> > >>>> +-----------------
> > >>>> +
> > >>>> +The xsabin file stores metadata which advertise HW subsystems
> > >>>> +present in a partition. The metadata is stored in device tree
> > >>>> +format with a well defined schema. XRT management driver uses
> > >>>> +this information to bind *xrt_drivers* to the subsystem
> > >>>> +instantiations. The xrt_drivers are found in **xrt-lib.ko** kernel
> module.
> > >>> I'm still catching up the patchset from the very beginning, and
> > >>> just finished the Documentation part. So far, I see the DT usage
> > >>> concern which may impact the architecure a lot, so I should raise it ASAP.
> > >>>
> > >>> The concern raised by the DT maintainer:
> > >>> https://lore.kernel.org/linux-fpga/CAL_JsqLod6FBGFhu7WXtMrB_z7wj8-
> > >>> [e-mail]/
> > >>>
> > >>> First of all, directly parsing FDT in device drivers is not a
> > >>> normal usage of DT in linux. It is out of the current DT usage
> > >>> model. So it should be agreed by DT maintainers.
> > >> Thanks for reviewing XRT document and providing feedback.
> > >> Here is the reply from Sonal for Rob’s question:
> > >> https://lore.kernel.org/linux-
> fpga/BY5PR02MB62604B87C66A1AD139A6F15
> > >> [e-mail]/
> > >> Overall, libfdt is used by XRT driver to parse the metadata which
> > >> comes with an Alveo board.
> > >> When XRT driver discovers an Alveo board, it will read a fdt blob
> > >> from board firmware file resident on the host.
> > >> By parsing the fdt blob, XRT driver gets information about this
> > >> Alveo board, such as version, uuid, IPs exposed to PCI BAR, interrupt
> binding etc.
> > >> So libfdt is used simply as Alveo metadata parser here. XRT drivers
> > >> do not interact with system wide DT or present the Alveo device
> > >> tree to host. For many systems like x86_64, system wide DT is not
> > >> present but libfdt parsing services will still be needed.
> > > Yes, I understand the use case.
> > >
> > > My concern is, directly parsing an isolated FDT in device driver and
> > > populate sub devices, skipping the unflattening, this is a new
> > > working model of device tree usage, but for the same purpose as the
> > > existing one.
> > >
> > > So I really need the confirmation of DT maintainers.
> 
> Perhaps you could explain why you think you need to use FDT instead of
> unflattening. Without that, the answer is don't use FDT.
> 
Xilinx Alveo PCIe cards are predominantly used in x86_64 systems which do not have device tree support compiled into the kernel. XRT driver uses a matching FDT to discover IP subsystems sitting behind the PCIe BARs exposed by an Alveo PCIe card. The FDT blob (as part of an Alveo PCIe card firmware) can be freely downloaded from xilinx.com.  

If using an unflattened tree (instead of FDT) is the right solution then we would certainly look into it. Should the PCIe driver then call something like of_fdt_unflatten_tree() with a FDT blob and the kernel would then build an unflattened tree and hang it off the PCIe device node? The FDT blob for a PCIe device is only known to the driver since different PCIe platforms may store it differently: a known location in the PCIe BAR, the flash on the PCIe board or a file on the filesystem. If the kernel can provide a general on demand unflattening service similar to DTO use model, we will have a more scalable solution to describe IP subsystems exposed by a PCIe device and make their discovery data driven. Can this feature also work on x86_64 systems which does not use OF?

> > >
> > >>> Current FPGA framework modifies kernel's live tree by DT overlay,
> > >>> when FPGA is dynamically reprogrammed and new HW devices appear.
> > >>> See Documentation/devicetree/bindings/fpga/fpga-region.txt.
> > >>>
> > >>> Then something less important:
> > >>>
> > >>>     1. The bindings should be documented in
> Documentation/devicetree/bindings/.
> 
> Yes.
> 
> > >>>     2. Are all the example DT usage conform to the exsiting bindings? I
> > >>>        didn't go through all device classes, but remember like the
> > >>>        interrupt-controller should have a "interrupt-controller" property,
> and
> > >>>        the PCI properties are also different from PCI bindings.
> 
> They don't, but should. I can't tell what you are trying to do here, but it looks
> like a mess.
> 
Will appreciate any pointers on existing PCIe use case for the device tree. 

Thanks,
-Sonal

> > >> The fdt properties are defined for Alveo firmware files. XRT driver
> > >> is the only consumer of this data. I am wondering if
> > > Personally I don't like the idea of a different binding definition
> > > set, even if the new device tree usage is accepted. We all use
> > > device tree for device enumeration, so why the different definitions.
> > >
> > > Thanks,
> > > Yilun
> > >
> > >> Documentation/devicetree/bindings is the right place for Alveo/XRT
> > >> private format or should it be documented as part of XRT driver
> documentation?
> > >> Looking for guidance here.
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> Lizhi
> > >>
> > >>> Thanks,
> > >>> Yilun
> > >>>
> > >>>> +
> > >>>> +Logic UUID
> > >>>> +^^^^^^^^^^
> > >>>> +A partition is identified uniquely through ``logic_uuid`` property::
> > >>>> +
> > >>>> +  /dts-v1/;
> > >>>> +  / {
> > >>>> +      logic_uuid = "0123456789abcdef0123456789abcdef";
> > >>>> +      ...
> > >>>> +    }
> > >>>> +
> > >>>> +Schema Version
> > >>>> +^^^^^^^^^^^^^^
> > >>>> +Schema version is defined through the ``schema_version`` node.
> > >>>> +It contains ``major`` and ``minor`` properties as below::
> > >>>> +
> > >>>> +  /dts-v1/;
> > >>>> +  / {
> > >>>> +       schema_version {
> > >>>> +           major = <0x01>;
> > >>>> +           minor = <0x00>;
> > >>>> +       };
> > >>>> +       ...
> > >>>> +    }
> > >>>> +
> > >>>> +.. _partition_uuids:
> > >>>> +
> > >>>> +Partition UUIDs
> > >>>> +^^^^^^^^^^^^^^^
> > >>>> +Each partition may have parent and child UUIDs. These UUIDs are
> > >>>> +defined by ``interfaces`` node and ``interface_uuid`` property::
> > >>>> +
> > >>>> +  /dts-v1/;
> > >>>> +  / {
> > >>>> +       interfaces {
> > >>>> +           @0 {
> > >>>> +                  interface_uuid = "0123456789abcdef0123456789abcdef";
> > >>>> +           };
> > >>>> +           @1 {
> > >>>> +                  interface_uuid = "fedcba9876543210fedcba9876543210";
> > >>>> +           };
> > >>>> +           ...
> > >>>> +        };
> > >>>> +       ...
> > >>>> +    }
> > >>>> +
> > >>>> +
> > >>>> +Subsystem Instantiations
> > >>>> +^^^^^^^^^^^^^^^^^^^^^^^^
> > >>>> +Subsystem instantiations are captured as children of
> > >>>> +``addressable_endpoints``
> > >>>> +node::
> > >>>> +
> > >>>> +  /dts-v1/;
> > >>>> +  / {
> > >>>> +       addressable_endpoints {
> > >>>> +           abc {
> > >>>> +               ...
> > >>>> +           };
> > >>>> +           def {
> > >>>> +               ...
> > >>>> +           };
> > >>>> +           ...
> > >>>> +       }
> > >>>> +  }
> > >>>> +
> > >>>> +Subnode 'abc' and 'def' are the name of subsystem nodes
> > >>>> +
> > >>>> +Subsystem Node
> > >>>> +^^^^^^^^^^^^^^
> > >>>> +Each subsystem node and its properties define a hardware instance::
> > >>>> +
> > >>>> +
> > >>>> +  addressable_endpoints {
> > >>>> +      abc {
> > >>>> +          reg = <0x00 0x1f05000 0x00 0x1000>>
> > >>>> +          pcie_physical_function = <0x0>;
> > >>>> +          pcie_bar_mapping = <0x2>;
> > >>>> +          compatible = "abc def";
> > >>>> +          interrupts = <0x09 0x0c>;
> > >>>> +          firmware {
> > >>>> +              firmware_product_name = "abc"
> > >>>> +              firmware_branch_name = "def"
> > >>>> +              firmware_version_major = <1>
> > >>>> +              firmware_version_minor = <2>
> > >>>> +          };
> > >>>> +      }
> > >>>> +      ...
> > >>>> +  }
> > >>>> +
> > >>>> +:reg:
> > >>>> + Property defines an address range. `<0x00 0x1f05000 0x00
> > >>>> +0x1000>` indicates
> > >>>> + *0x00 0x1f05000* as BAR offset and *0x00 0x1000* as address
> length.
> > >>>> +:pcie_physical_function:
> > >>>> + Property specifies which PCIe physical function the subsystem node
> resides.
> > >>>> + `<0x0>` implies physical function 0.
> > >>>> +:pcie_bar_mapping:
> > >>>> + Property specifies which PCIe BAR the subsystem node resides.
> > >>>> +`<0x2>` implies  BAR 2. A value of 0 means the property is not defined.
> > >>>> +:compatible:
> > >>>> + Property is a list of strings. The first string in the list
> > >>>> +specifies the exact  subsystem node. The following strings
> > >>>> +represent other devices that the device  is compatible with.
> > >>>> +:interrupts:
> > >>>> + Property specifies start and end interrupts for this subsystem node.
> > >>>> + `<0x09 0x0c>` implies interrupts 9 to 13 are used by this subsystem.
> > >>>> +:firmware:
> > >>>> + Subnode defines the firmware required by this subsystem node.
> > >>>> +
> > >>>> +Alveo U50 Platform Example
> > >>>> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> > >>>> +::
> > >>>> +
> > >>>> +  /dts-v1/;
> > >>>> +
> > >>>> +  /{
> > >>>> +        logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
> > >>>> +
> > >>>> +        schema_version {
> > >>>> +                major = <0x01>;
> > >>>> +                minor = <0x00>;
> > >>>> +        };
> > >>>> +
> > >>>> +        interfaces {
> > >>>> +
> > >>>> +                @0 {
> > >>>> +                        interface_uuid =
> "862c7020a250293e32036f19956669e5";
> > >>>> +                };
> > >>>> +        };
> > >>>> +
> > >>>> +        addressable_endpoints {
> > >>>> +
> > >>>> +                ep_blp_rom_00 {
> > >>>> +                        reg = <0x00 0x1f04000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-
> 1.0\0axi_bram_ctrl";
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_card_flash_program_00 {
> > >>>> +                        reg = <0x00 0x1f06000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_quad_spi-
> 1.0\0axi_quad_spi";
> > >>>> +                        interrupts = <0x03 0x03>;
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_cmc_firmware_mem_00 {
> > >>>> +                        reg = <0x00 0x1e20000 0x00 0x20000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible =
> > >>>> + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > >>>> +
> > >>>> +                        firmware {
> > >>>> +                                firmware_product_name = "cmc";
> > >>>> +                                firmware_branch_name = "u50";
> > >>>> +                                firmware_version_major = <0x01>;
> > >>>> +                                firmware_version_minor = <0x00>;
> > >>>> +                        };
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_cmc_intc_00 {
> > >>>> +                        reg = <0x00 0x1e03000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_intc-
> 1.0\0axi_intc";
> > >>>> +                        interrupts = <0x04 0x04>;
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_cmc_mutex_00 {
> > >>>> +                        reg = <0x00 0x1e02000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_cmc_regmap_00 {
> > >>>> +                        reg = <0x00 0x1e08000 0x00 0x2000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible =
> > >>>> + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > >>>> +
> > >>>> +                        firmware {
> > >>>> +                                firmware_product_name = "sc-fw";
> > >>>> +                                firmware_branch_name = "u50";
> > >>>> +                                firmware_version_major = <0x05>;
> > >>>> +                        };
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_cmc_reset_00 {
> > >>>> +                        reg = <0x00 0x1e01000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_ddr_mem_calib_00 {
> > >>>> +                        reg = <0x00 0x63000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_debug_bscan_mgmt_00 {
> > >>>> +                        reg = <0x00 0x1e90000 0x00 0x10000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-debug_bridge-
> 1.0\0debug_bridge";
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_ert_base_address_00 {
> > >>>> +                        reg = <0x00 0x21000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_ert_command_queue_mgmt_00 {
> > >>>> +                        reg = <0x00 0x40000 0x00 0x10000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-ert_command_queue-
> 1.0\0ert_command_queue";
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_ert_command_queue_user_00 {
> > >>>> +                        reg = <0x00 0x40000 0x00 0x10000>;
> > >>>> +                        pcie_physical_function = <0x01>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-ert_command_queue-
> 1.0\0ert_command_queue";
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_ert_firmware_mem_00 {
> > >>>> +                        reg = <0x00 0x30000 0x00 0x8000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible =
> > >>>> + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > >>>> +
> > >>>> +                        firmware {
> > >>>> +                                firmware_product_name = "ert";
> > >>>> +                                firmware_branch_name = "v20";
> > >>>> +                                firmware_version_major = <0x01>;
> > >>>> +                        };
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_ert_intc_00 {
> > >>>> +                        reg = <0x00 0x23000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_intc-
> 1.0\0axi_intc";
> > >>>> +                        interrupts = <0x05 0x05>;
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_ert_reset_00 {
> > >>>> +                        reg = <0x00 0x22000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_ert_sched_00 {
> > >>>> +                        reg = <0x00 0x50000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x01>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-ert_sched-
> 1.0\0ert_sched";
> > >>>> +                        interrupts = <0x09 0x0c>;
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_fpga_configuration_00 {
> > >>>> +                        reg = <0x00 0x1e88000 0x00 0x8000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_hwicap-
> 1.0\0axi_hwicap";
> > >>>> +                        interrupts = <0x02 0x02>;
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_icap_reset_00 {
> > >>>> +                        reg = <0x00 0x1f07000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_msix_00 {
> > >>>> +                        reg = <0x00 0x00 0x00 0x20000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
> > >>>> +                        pcie_bar_mapping = <0x02>;
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_pcie_link_mon_00 {
> > >>>> +                        reg = <0x00 0x1f05000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_pr_isolate_plp_00 {
> > >>>> +                        reg = <0x00 0x1f01000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_pr_isolate_ulp_00 {
> > >>>> +                        reg = <0x00 0x1000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_uuid_rom_00 {
> > >>>> +                        reg = <0x00 0x64000 0x00 0x1000>;
> > >>>> +                        pcie_physical_function = <0x00>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-
> 1.0\0axi_bram_ctrl";
> > >>>> +                };
> > >>>> +
> > >>>> +                ep_xdma_00 {
> > >>>> +                        reg = <0x00 0x00 0x00 0x10000>;
> > >>>> +                        pcie_physical_function = <0x01>;
> > >>>> +                        compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
> > >>>> +                        pcie_bar_mapping = <0x02>;
> > >>>> +                };
> > >>>> +        };
> > >>>> +
> > >>>> +  }
> > >>>> +
> > >>>> +
> > >>>> +
> > >>>> +Deployment Models
> > >>>> +=================
> > >>>> +
> > >>>> +Baremetal
> > >>>> +---------
> > >>>> +
> > >>>> +In bare-metal deployments, both MPF and UPF are visible and
> > >>>> +accessible. The xrt-mgmt driver binds to MPF. The xrt-mgmt
> > >>>> +driver operations are privileged and available to system administrator.
> The full stack is illustrated below::
> > >>>> +
> > >>>> +                            HOST
> > >>>> +
> > >>>> +               [XRT-MGMT]         [XRT-USER]
> > >>>> +                    |                  |
> > >>>> +                    |                  |
> > >>>> +                 +-----+            +-----+
> > >>>> +                 | MPF |            | UPF |
> > >>>> +                 |     |            |     |
> > >>>> +                 | PF0 |            | PF1 |
> > >>>> +                 +--+--+            +--+--+
> > >>>> +          ......... ^................. ^..........
> > >>>> +                    |                  |
> > >>>> +                    |   PCIe DEVICE    |
> > >>>> +                    |                  |
> > >>>> +                 +--+------------------+--+
> > >>>> +                 |         SHELL          |
> > >>>> +                 |                        |
> > >>>> +                 +------------------------+
> > >>>> +                 |         USER           |
> > >>>> +                 |                        |
> > >>>> +                 |                        |
> > >>>> +                 |                        |
> > >>>> +                 |                        |
> > >>>> +                 +------------------------+
> > >>>> +
> > >>>> +
> > >>>> +
> > >>>> +Virtualized
> > >>>> +-----------
> > >>>> +
> > >>>> +In virtualized deployments, the privileged MPF is assigned to
> > >>>> +the host but the unprivileged UPF is assigned to a guest VM via
> > >>>> +PCIe pass-through. The xrt-mgmt driver in host binds to MPF. The
> > >>>> +xrt-mgmt driver operations are privileged and only accessible to the
> MPF. The full stack is illustrated below::
> > >>>> +
> > >>>> +
> > >>>> +                                 ..............
> > >>>> +                  HOST           .    VM      .
> > >>>> +                                 .            .
> > >>>> +               [XRT-MGMT]        . [XRT-USER] .
> > >>>> +                    |            .     |      .
> > >>>> +                    |            .     |      .
> > >>>> +                 +-----+         .  +-----+   .
> > >>>> +                 | MPF |         .  | UPF |   .
> > >>>> +                 |     |         .  |     |   .
> > >>>> +                 | PF0 |         .  | PF1 |   .
> > >>>> +                 +--+--+         .  +--+--+   .
> > >>>> +          ......... ^................. ^..........
> > >>>> +                    |                  |
> > >>>> +                    |   PCIe DEVICE    |
> > >>>> +                    |                  |
> > >>>> +                 +--+------------------+--+
> > >>>> +                 |         SHELL          |
> > >>>> +                 |                        |
> > >>>> +                 +------------------------+
> > >>>> +                 |         USER           |
> > >>>> +                 |                        |
> > >>>> +                 |                        |
> > >>>> +                 |                        |
> > >>>> +                 |                        |
> > >>>> +                 +------------------------+
> > >>>> +
> > >>>> +
> > >>>> +
> > >>>> +
> > >>>> +
> > >>>> +Platform Security Considerations
> > >>>> +================================
> > >>>> +
> > >>>> +`Security of Alveo Platform
> > >>>> +<https://xilinx.github.io/XRT/master/html/security.html>`_
> > >>>> +discusses the deployment options and security implications in great
> detail.
> > >>>> diff --git a/MAINTAINERS b/MAINTAINERS index
> > >>>> 056966c9aac9..beeaf0257364 100644
> > >>>> --- a/MAINTAINERS
> > >>>> +++ b/MAINTAINERS
> > >>>> @@ -7274,6 +7274,17 @@ F:     Documentation/fpga/
> > >>>>    F:   drivers/fpga/
> > >>>>    F:   include/linux/fpga/
> > >>>>
> > >>>> +FPGA XRT DRIVERS
> > >>>> +M:   Lizhi Hou <[e-mail]>
> > >>>> +R:   Max Zhen <[e-mail]>
> > >>>> +R:   Sonal Santan <[e-mail]>
> > >>>> +L:   [e-mail]
> > >>>> +S:   Supported
> > >>>> +W:   https://github.com/Xilinx/XRT
> > >>>> +F:   Documentation/fpga/xrt.rst
> > >>>> +F:   drivers/fpga/xrt/
> > >>>> +F:   include/uapi/linux/xrt/
> > >>>> +
> > >>>>    FPU EMULATOR
> > >>>>    M:   Bill Metzenthen <[e-mail]>
> > >>>>    S:   Maintained
> > >>>> --
> > >>>> 2.27.0

Rob Herring (Arm) Oct. 21, 2021, 5:38 p.m. UTC | #8

On Thu, Oct 21, 2021 at 12:36 AM Sonal Santan <[e-mail]> wrote:
>
> Hello Rob,
>
> > -----Original Message-----
> > From: Rob Herring <[e-mail]>
> > Sent: Tuesday, October 19, 2021 6:03 AM
> > To: Lizhi Hou <[e-mail]>
> > Cc: [e-mail]; [e-mail]; Max Zhen
> > <[e-mail]>; Sonal Santan <[e-mail]>; Yu Liu
> > <[e-mail]>; Michal Simek <[e-mail]>; Stefano Stabellini
> > <[e-mail]>; [e-mail]; [e-mail]; Moritz
> > Fischer <[e-mail]>; Max Zhen <[e-mail]>; Xu Yilun
> > <[e-mail]>
> > Subject: Re: [PATCH V9 XRT Alveo 01/14] Documentation: fpga: Add a
> > document describing XRT Alveo drivers
> >
> > On Thu, Oct 14, 2021 at 11:12 AM Lizhi Hou <[e-mail]> wrote:
> > >
> > > Hello Rob,
> > >
> > > Please help with the review of the proposed FDT usage by Alveo/XRT drivers.
> > >
> > > Thanks,
> > > Lizhi
> > >
> > > On 10/13/21 7:21 PM, Xu Yilun wrote:
> > > > CAUTION: This message has originated from an External Source. Please use
> > proper judgment and caution when opening attachments, clicking links, or
> > responding to this email.
> > > >
> > > >
> > > >>>> +.. _device_tree_usage:
> > > >>>> +
> > > >>>> +Device Tree Usage
> > > >>>> +-----------------
> > > >>>> +
> > > >>>> +The xsabin file stores metadata which advertise HW subsystems
> > > >>>> +present in a partition. The metadata is stored in device tree
> > > >>>> +format with a well defined schema. XRT management driver uses
> > > >>>> +this information to bind *xrt_drivers* to the subsystem
> > > >>>> +instantiations. The xrt_drivers are found in **xrt-lib.ko** kernel
> > module.
> > > >>> I'm still catching up the patchset from the very beginning, and
> > > >>> just finished the Documentation part. So far, I see the DT usage
> > > >>> concern which may impact the architecure a lot, so I should raise it ASAP.
> > > >>>
> > > >>> The concern raised by the DT maintainer:
> > > >>> https://lore.kernel.org/linux-fpga/CAL_JsqLod6FBGFhu7WXtMrB_z7wj8-
> > > >>> [e-mail]/
> > > >>>
> > > >>> First of all, directly parsing FDT in device drivers is not a
> > > >>> normal usage of DT in linux. It is out of the current DT usage
> > > >>> model. So it should be agreed by DT maintainers.
> > > >> Thanks for reviewing XRT document and providing feedback.
> > > >> Here is the reply from Sonal for Rob’s question:
> > > >> https://lore.kernel.org/linux-
> > fpga/BY5PR02MB62604B87C66A1AD139A6F15
> > > >> [e-mail]/
> > > >> Overall, libfdt is used by XRT driver to parse the metadata which
> > > >> comes with an Alveo board.
> > > >> When XRT driver discovers an Alveo board, it will read a fdt blob
> > > >> from board firmware file resident on the host.
> > > >> By parsing the fdt blob, XRT driver gets information about this
> > > >> Alveo board, such as version, uuid, IPs exposed to PCI BAR, interrupt
> > binding etc.
> > > >> So libfdt is used simply as Alveo metadata parser here. XRT drivers
> > > >> do not interact with system wide DT or present the Alveo device
> > > >> tree to host. For many systems like x86_64, system wide DT is not
> > > >> present but libfdt parsing services will still be needed.
> > > > Yes, I understand the use case.
> > > >
> > > > My concern is, directly parsing an isolated FDT in device driver and
> > > > populate sub devices, skipping the unflattening, this is a new
> > > > working model of device tree usage, but for the same purpose as the
> > > > existing one.
> > > >
> > > > So I really need the confirmation of DT maintainers.
> >
> > Perhaps you could explain why you think you need to use FDT instead of
> > unflattening. Without that, the answer is don't use FDT.
> >
> Xilinx Alveo PCIe cards are predominantly used in x86_64 systems which do not have device tree support compiled into the kernel. XRT driver uses a matching FDT to discover IP subsystems sitting behind the PCIe BARs exposed by an Alveo PCIe card. The FDT blob (as part of an Alveo PCIe card firmware) can be freely downloaded from xilinx.com.

If the kernel is going to consume that FDT blob, then it needs to
follow upstream practices which primarily means all the device
bindings must be documented with schema and reviewed.


> If using an unflattened tree (instead of FDT) is the right solution then we would certainly look into it. Should the PCIe driver then call something like of_fdt_unflatten_tree() with a FDT blob and the kernel would then build an unflattened tree and hang it off the PCIe device node? The FDT blob for a PCIe device is only known to the driver since different PCIe platforms may store it differently: a known location in the PCIe BAR, the flash on the PCIe board or a file on the filesystem. If the kernel can provide a general on demand unflattening service similar to DTO use model, we will have a more scalable solution to describe IP subsystems exposed by a PCIe device and make their discovery data driven. Can this feature also work on x86_64 systems which does not use OF?

There's other similar usecases like this. For example, an FTDI or
similar USB to serial chip that has GPIO, I2C, etc. and could have
downstream devices hanging off of those interfaces. And then you could
plug-in multiple of those devices to the host system. For this to
work, we'd need to create a base tree (if there isn't one) with nodes
for the USB or PCI device(s) and then an overlay for the device can be
applied to those nodes. This is also partially an issue on DT based
systems as the DT node may not exist given these are 'discoverable'
buses. It's a bit easier to solve given the PCI host bridge or USB
controller exists in the DT already.

There's really 2 separate parts here. There's how to attach a DT to a
device on a non-DT system (or DT system with a device not described in
the base DT). The second part is how to describe the PCI device and
downstream devices. This part is no different than any other device.


> > > >>> Current FPGA framework modifies kernel's live tree by DT overlay,
> > > >>> when FPGA is dynamically reprogrammed and new HW devices appear.
> > > >>> See Documentation/devicetree/bindings/fpga/fpga-region.txt.
> > > >>>
> > > >>> Then something less important:
> > > >>>
> > > >>>     1. The bindings should be documented in
> > Documentation/devicetree/bindings/.
> >
> > Yes.
> >
> > > >>>     2. Are all the example DT usage conform to the exsiting bindings? I
> > > >>>        didn't go through all device classes, but remember like the
> > > >>>        interrupt-controller should have a "interrupt-controller" property,
> > and
> > > >>>        the PCI properties are also different from PCI bindings.
> >
> > They don't, but should. I can't tell what you are trying to do here, but it looks
> > like a mess.
> >
> Will appreciate any pointers on existing PCIe use case for the device tree.

Documentation/devicetree/bindings/pci/ and there's the PCI bus schema
here[1]. There's also the OpenFirmware PCI spec[2].

Rob

[1] https://github.com/devicetree-org/dt-schema/blob/main/schemas/pci/pci-bus.yaml
[2] https://www.devicetree.org/open-firmware/bindings/pci/pci2_1.pdf

Sonal Santan Oct. 22, 2021, 4:36 a.m. UTC | #9

Thanks for your suggestions and pointers. 

We will look into creating a generic patchset for attaching a device tree for PCIe use case. Will then refactor the XRT Alveo driver to use this new infrastructure.

-Sonal

Patch

diff --git a/Documentation/fpga/index.rst b/Documentation/fpga/index.rst
index f80f95667ca2..30134357b70d 100644
--- a/Documentation/fpga/index.rst
+++ b/Documentation/fpga/index.rst
@@ -8,6 +8,7 @@  fpga
     :maxdepth: 1
 
     dfl
+    xrt
 
 .. only::  subproject and html
 
diff --git a/Documentation/fpga/xrt.rst b/Documentation/fpga/xrt.rst
new file mode 100644
index 000000000000..84eb41be9ac1
--- /dev/null
+++ b/Documentation/fpga/xrt.rst
@@ -0,0 +1,870 @@ 
+.. SPDX-License-Identifier: GPL-2.0
+
+==================================
+XRTV2 Linux Kernel Driver Overview
+==================================
+
+Authors:
+
+* Sonal Santan <[e-mail]>
+* Max Zhen <[e-mail]>
+* Lizhi Hou <[e-mail]>
+
+XRTV2 drivers are second generation `XRT <https://github.com/Xilinx/XRT>`_
+drivers which support `Alveo <https://www.xilinx.com/products/boards-and-kits/alveo.html>`_
+PCIe platforms from Xilinx.
+
+XRTV2 drivers support *subsystem* style data driven platforms where driver's
+configuration and behavior are determined by metadata provided by the platform
+(in *device tree* format). Primary management physical function (MPF) driver
+is called **xrt-mgmt**. Primary user physical function (UPF) driver is called
+**xrt-user** and is under development. xrt_driver framework and HW subsystem
+drivers are packaged into a library module called **xrt-lib**, which is shared
+by **xrt-mgmt** and **xrt-user** (under development). The xrt_driver framework
+implements a ``bus_type`` called **xrt_bus_type** which is used to discover HW
+subsystems and facilitate inter HW subsystem interaction.
+
+Driver Modules
+==============
+
+xrt-lib.ko
+----------
+
+xrt-lib is the repository of all subsystem drivers and pure software modules that
+can potentially be shared between xrt-mgmt and xrt-user. All these drivers are
+structured as **xrt_driver** and are instantiated by xrt-mgmt (or xrt-user under
+development) based on the metadata associated with the hardware. The metadata is
+in the form of a device tree as mentioned before. Each xrt_driver statically
+defines a subsystem node array by using a node name or a string in its ``.endpoints``
+property. And this array is eventually translated to IOMEM resources in the
+instantiated **xrt_device**.
+
+The xrt-lib infrastructure provides hooks to xrt_drivers for device node
+management, user file operations and ioctl callbacks. The core infrastructure also
+provides a bus functionality called **xrt_bus_type** for xrt_driver registration,
+discovery and inter xrt_driver calls. xrt-lib does not have any dependency on PCIe
+subsystem.
+
+.. note::
+   See code in ``include/xleaf.h`` and ``include/xdevice.h``
+
+
+xrt-mgmt.ko
+------------
+
+The xrt-mgmt driver is a PCIe device driver driving MPF found on Xilinx's Alveo
+PCIe device. It consists of one *root* driver, one or more *group* drivers
+and one or more *xleaf* drivers. The group and xleaf drivers are instantiations
+of the xrt_driver but are called group and xleaf to symbolize the logical operation
+performed by them.
+
+The root driver manages the life cycle of multiple group drivers, which, in turn,
+manages multiple xleaf drivers. This flexibility allows xrt-mgmt.ko and xrt-lib.ko
+to support various HW subsystems exposed by different Alveo shells. The differences
+among these Alveo shells is handled in xleaf drivers. The root and group
+drivers are part of the infrastructure which provide common services to xleaf
+drivers found on various Alveo shells. See :ref:`alveo_platform_overview`.
+
+The instantiation of specific group driver or xleaf drivers is completely data
+driven based on metadata (mostly in device tree format) found through VSEC
+capability and inside the firmware files, such as platform xsabin or user xclbin
+file.
+
+
+Driver Object Model
+===================
+
+The driver object model looks like the following::
+
+                    +-----------+
+                    |   xroot   |
+                    +-----+-----+
+                          |
+              +-----------+-----------+
+              |                       |
+              v                       v
+        +-----------+          +-----------+
+        |   group   |    ...   |   group   |
+        +-----+-----+          +------+----+
+              |                       |
+              |                       |
+        +-----+----+            +-----+----+
+        |          |            |          |
+        v          v            v          v
+    +-------+  +-------+    +-------+  +-------+
+    | xleaf |..| xleaf |    | xleaf |..| xleaf |
+    +-------+  +-------+    +-------+  +-------+
+
+As an example, for Xilinx Alveo U50 before user xclbin download, the tree
+looks like the following::
+
+                                +-----------+
+                                |  xrt-mgmt |
+                                +-----+-----+
+                                      |
+            +-------------------------+--------------------+
+            |                         |                    |
+            v                         v                    v
+       +--------+                +--------+            +--------+
+       | group0 |                | group1 |            | group2 |
+       +----+---+                +----+---+            +---+----+
+            |                         |                    |
+            |                         |                    |
+      +-----+-----+        +----+-----+---+    +-----+-----+----+--------+
+      |           |        |    |         |    |     |          |        |
+      v           v        |    v         v    |     v          v        |
+ +------------+  +------+  | +------+ +------+ |  +------+ +-----------+ |
+ | xmgmt_main |  | VSEC |  | | GPIO | | QSPI | |  |  CMC | | AXI-GATE0 | |
+ +------------+  +------+  | +------+ +------+ |  +------+ +-----------+ |
+                           | +---------+       |  +------+ +-----------+ |
+                           +>| MAILBOX |       +->| ICAP | | AXI-GATE1 |<+
+                             +---------+       |  +------+ +-----------+
+                                               |  +-------+
+                                               +->| CALIB |
+                                                  +-------+
+
+After a xclbin is downloaded, group3 will be added and the tree looks like the
+following::
+
+                                +-----------+
+                                |  xrt-mgmt |
+                                +-----+-----+
+                                      |
+            +-------------------------+--------------------+-----------------+
+            |                         |                    |                 |
+            v                         v                    v                 |
+       +--------+                +--------+            +--------+            |
+       | group0 |                | group1 |            | group2 |            |
+       +----+---+                +----+---+            +---+----+            |
+            |                         |                    |                 |
+            |                         |                    |                 |
+      +-----+-----+       +-----+-----+---+    +-----+-----+----+--------+   |
+      |           |       |     |         |    |     |          |        |   |
+      v           v       |     v         v    |     v          v        |   |
+ +------------+  +------+ | +------+ +------+  |  +------+ +-----------+ |   |
+ | xmgmt_main |  | VSEC | | | GPIO | | QSPI |  |  |  CMC | | AXI-GATE0 | |   |
+ +------------+  +------+ | +------+ +------+  |  +------+ +-----------+ |   |
+                          | +---------+        |  +------+ +-----------+ |   |
+                          +>| MAILBOX |        +->| ICAP | | AXI-GATE1 |<+   |
+                            +---------+        |  +------+ +-----------+     |
+                                               |  +-------+                  |
+                                               +->| CALIB |                  |
+                                                  +-------+                  |
+                      +---+----+                                             |
+                      | group3 |<--------------------------------------------+
+                      +--------+
+                          |
+                          |
+     +-------+--------+---+--+--------+------+-------+
+     |       |        |      |        |      |       |
+     v       |        v      |        v      |       v
+ +--------+  |   +--------+  |   +--------+  |    +-----+
+ | CLOCK0 |  |   | CLOCK1 |  |   | CLOCK2 |  |    | UCS |
+ +--------+  v   +--------+  v   +--------+  v    +-----+
+ +-------------+ +-------------+ +-------------+
+ | CLOCK-FREQ0 | | CLOCK-FREQ1 | | CLOCK-FREQ2 |
+ +-------------+ +-------------+ +-------------+
+
+
+root
+----
+
+The root driver is a PCIe device driver attached to MPF. It's part of the
+infrastructure of the MPF driver and resides in xrt-mgmt.ko. This driver
+
+* manages one or more group drivers
+* provides access to functionalities that requires pci_dev, such as PCIE config
+  space access, to other xleaf drivers through root calls
+* facilities inter xleaf driver calls for other xleaf drivers
+* facilities event callbacks for other xleaf drivers
+
+When the root driver starts, it will explicitly create an initial group instance,
+which contains xleaf drivers that will trigger the creation of other group
+instances. The root driver will wait for all group and xleaf drivers to be
+created before it returns from its probe routine and claim success of the
+initialization of the entire xrt-mgmt driver. If any xleaf fails to initialize
+the xrt-mgmt driver will still come online but with limited functionality.
+
+.. note::
+   See code in ``lib/xroot.c`` and ``mgmt/root.c``
+
+
+group
+-----
+
+The group driver represents a pseudo device whose life cycle is managed by
+root and does not have real IO mem or IRQ resources. It's part of the
+infrastructure of the MPF driver and resides in xrt-lib.ko. This driver
+
+* manages one or more xleaf drivers
+* provides access to root from xleaf drivers, so that root calls, event
+  notifications and inter xleaf calls can happen
+
+In xrt-mgmt, an initial group driver instance will be created by the root. This
+instance contains xleaf drivers that will trigger group instances to be created
+to manage groups of xleaf drivers found on different partitions of hardware,
+such as VSEC, Shell, and User.
+
+Every *fpga_region* has a group driver associated with it. The group driver is
+created when a xclbin image is loaded on the fpga_region. The existing group
+is destroyed when a new xclbin image is loaded. The fpga_region persists
+across xclbin downloads.
+
+.. note::
+   See code in ``lib/group.c``
+
+
+xleaf
+-----
+
+The xleaf driver is a xrt_driver whose life cycle is managed by
+a group driver and may or may not have real IO mem or IRQ resources. They
+manage HW subsystems they are attached to.
+
+A xleaf driver without real hardware resources manages in-memory states for
+xrt-mgmt. These states are shareable by other xleaf drivers.
+
+Xleaf drivers assigned to specific hardware resources drive a specific subsystem
+in the device. To manipulate the subsystem or carry out a task, a xleaf driver
+may ask for help from the root via root calls and/or from other leaves via
+inter xleaf calls.
+
+A xleaf can also broadcast events through infrastructure code for other leaves
+to process. It can also receive event notification from infrastructure about
+certain events, such as post-creation or pre-exit of a particular xleaf.
+
+.. note::
+   See code in ``lib/xleaf/*.c``
+
+
+xrt_bus_type
+------------
+
+xrt_bus_type defines a virtual bus which handles xrt_driver probe, remove and match
+operations. All xrt_drivers register with xrt_bus_type as part of xrt-lib driver
+``module_init`` and un-register as part of xrt-lib driver ``module_exit``.
+
+.. note::
+   See code in ``lib/lib-drv.c``
+
+FPGA Manager Interaction
+========================
+
+fpga_manager
+------------
+
+An instance of fpga_manager is created by xmgmt_main and is used for xclbin
+image download. fpga_manager requires the full xclbin image before it can
+start programming the FPGA configuration engine via Internal Configuration
+Access Port (ICAP) xrt_driver.
+
+fpga_region
+-----------
+
+For every interface exposed by the currently loaded xclbin/xsabin in the
+*parent* fpga_region a new instance of fpga_region is created like a *child*
+fpga_region. The device tree of the *parent* fpga_region defines the
+resources for a new instance of fpga_bridge which isolates the parent from
+child fpga_region. This new instance of fpga_bridge will be used when a
+xclbin image is loaded on the child fpga_region. After the xclbin image is
+downloaded to the fpga_region, an instance of a group is created for the
+fpga_region using the device tree obtained as part of the xclbin. If this
+device tree defines any child interfaces, it can trigger the creation of
+fpga_bridge and fpga_region for the next region in the chain.
+
+fpga_bridge
+-----------
+
+Like the fpga_region, an fpga_bridge is created by walking the device tree
+of the parent group. The bridge is used for isolation between a parent and
+its child.
+
+Driver Interfaces
+=================
+
+xrt-mgmt Driver Ioctls
+----------------------
+
+Ioctls exposed by the xrt-mgmt driver to user space are enumerated in the
+following table:
+
+== ===================== ============================ ==========================
+#  Functionality         ioctl request code            data format
+== ===================== ============================ ==========================
+1  FPGA image download   XMGMT_IOCICAPDOWNLOAD_AXLF    xmgmt_ioc_bitstream_axlf
+== ===================== ============================ ==========================
+
+A user xclbin can be downloaded by using the xbmgmt tool from the XRT open source
+suite. See example usage below::
+
+  xbmgmt partition --program --path /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/test/verify.xclbin --force
+
+xrt-mgmt Driver Sysfs
+----------------------
+
+The xrt-mgmt driver exposes a rich set of sysfs interfaces. Subsystem xrt
+drivers export sysfs node for every platform instance.
+
+Every partition also exports its UUIDs. See below for examples::
+
+  /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
+  /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
+
+
+hwmon
+-----
+
+The xrt-mgmt driver exposes standard hwmon interface to report voltage, current,
+temperature, power, etc. These can easily be viewed using *sensors* command line
+utility.
+
+.. _alveo_platform_overview:
+
+Alveo Platform Overview
+=======================
+
+Alveo platforms are architected as two physical FPGA partitions: *Shell* and
+*User*. The Shell provides basic infrastructure for the Alveo platform like
+PCIe connectivity, board management, Dynamic Function Exchange (DFX), sensors,
+clocking, reset, and security. DFX, partial reconfiguration, is responsible for
+loading the user compiled FPGA binary.
+
+For DFX to work properly, physical partitions require strict HW compatibility
+with each other. Every physical partition has two interface UUIDs: the *parent*
+UUID and the *child* UUID. For simple single stage platforms, Shell → User forms
+the parent child relationship.
+
+.. note::
+   Partition compatibility matching is a key design component of the Alveo platforms
+   and XRT. Partitions have child and parent relationship. A loaded partition
+   exposes child partition UUID to advertise its compatibility requirement. When
+   loading a child partition, the xrt-mgmt driver matches the parent
+   UUID of the child partition against the child UUID exported by the parent.
+   The parent and child partition UUIDs are stored in the *xclbin* (for the user)
+   and the *xsabin* (for the shell). Except for the root UUID exported by VSEC,
+   the hardware itself does not know about the UUIDs. The UUIDs are stored in
+   xsabin and xclbin. The image format has a special node called Partition UUIDs
+   which define the compatibility UUIDs. See :ref:`partition_uuids`.
+
+
+The physical partitions and their loading are illustrated below::
+
+           SHELL                               USER
+        +-----------+                  +-------------------+
+        |           |                  |                   |
+        | VSEC UUID | CHILD     PARENT |    LOGIC UUID     |
+        |           o------->|<--------o                   |
+        |           | UUID       UUID  |                   |
+        +-----+-----+                  +--------+----------+
+              |                                 |
+              .                                 .
+              |                                 |
+          +---+---+                      +------+--------+
+          |  POR  |                      | USER COMPILED |
+          | FLASH |                      |    XCLBIN     |
+          +-------+                      +---------------+
+
+
+Loading Sequence
+----------------
+
+The Shell partition is loaded from flash at system boot time. It establishes the
+PCIe link and exposes two physical functions to the BIOS. After the OS boots,
+the xrt-mgmt driver attaches to the PCIe physical function 0 exposed by the Shell
+and then looks for VSEC in the PCIe extended configuration space. Using VSEC, it
+determines the logic UUID of the Shell and uses the UUID to load matching *xsabin*
+file from Linux firmware directory. The xsabin file contains the metadata to
+discover the peripherals that are part of the Shell and the firmware for any
+embedded soft processors in the Shell. The xsabin file also contains Partition
+UUIDs as described here :ref:`partition_uuids`.
+
+The Shell exports a child interface UUID which is used for the compatibility
+check when loading the user compiled xclbin over the User partition as part of DFX.
+When a user requests loading of a specific xclbin, the xrt-mgmt driver reads
+the parent interface UUID specified in the xclbin and matches it with the child
+interface UUID exported by the Shell to determine if the xclbin is compatible with
+the Shell. If the match fails, loading of xclbin is denied.
+
+xclbin loading is requested using the ICAP_DOWNLOAD_AXLF ioctl command. When loading
+a xclbin, the xrt-mgmt driver performs the following *logical* operations:
+
+1. Copy xclbin from user to kernel memory
+2. Sanity check the xclbin contents
+3. Isolate the User partition
+4. Download the bitstream using the FPGA config engine (ICAP)
+5. De-isolate the User partition
+6. Program the clocks (ClockWiz) driving the User partition
+7. Wait for the memory controller (MIG) calibration
+8. Return the loading status back to the caller
+
+`Platform Loading Overview <https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`_
+provides more detailed information on platform loading.
+
+
+xsabin
+------
+
+Each Alveo platform comes packaged with its own xsabin. The xsabin is a trusted
+component of the platform. For format details refer to :ref:`xsabin_xclbin_container_format`
+below. xsabin contains basic information like UUIDs, platform name and metadata in the
+form of device tree. See :ref:`device_tree_usage` below for details and example.
+
+xclbin
+------
+
+xclbin is compiled by end user using
+`Vitis <https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html>`_
+tool set from Xilinx. The xclbin contains sections describing user compiled
+acceleration engines/kernels, memory subsystems, clocking information etc. It also
+contains an FPGA bitstream for the user partition, UUIDs, platform name, etc.
+
+
+.. _xsabin_xclbin_container_format:
+
+xsabin/xclbin Container Format
+------------------------------
+
+xclbin/xsabin is ELF-like binary container format. It is structured as series of
+sections. There is a file header followed by several section headers which is
+followed by sections. A section header points to an actual section. There is an
+optional signature at the end. The format is defined by the header file ``xclbin.h``.
+The following figure illustrates a typical xclbin::
+
+
+           +---------------------+
+           |                     |
+           |       HEADER        |
+           +---------------------+
+           |   SECTION  HEADER   |
+           |                     |
+           +---------------------+
+           |         ...         |
+           |                     |
+           +---------------------+
+           |   SECTION  HEADER   |
+           |                     |
+           +---------------------+
+           |       SECTION       |
+           |                     |
+           +---------------------+
+           |         ...         |
+           |                     |
+           +---------------------+
+           |       SECTION       |
+           |                     |
+           +---------------------+
+           |      SIGNATURE      |
+           |      (OPTIONAL)     |
+           +---------------------+
+
+
+xclbin/xsabin files can be packaged, un-packaged and inspected using an XRT
+utility called **xclbinutil**. xclbinutil is part of the XRT open source
+software stack. The source code for xclbinutil can be found at
+https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbinutil
+
+For example, to enumerate the contents of a xclbin/xsabin use the *--info* switch
+as shown below::
+
+
+  xclbinutil --info --input /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
+  xclbinutil --info --input /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsabin
+
+
+.. _device_tree_usage:
+
+Device Tree Usage
+-----------------
+
+The xsabin file stores metadata which advertise HW subsystems present in a
+partition. The metadata is stored in device tree format with a well defined
+schema. XRT management driver uses this information to bind *xrt_drivers* to
+the subsystem instantiations. The xrt_drivers are found in **xrt-lib.ko** kernel
+module.
+
+Logic UUID
+^^^^^^^^^^
+A partition is identified uniquely through ``logic_uuid`` property::
+
+  /dts-v1/;
+  / {
+      logic_uuid = "0123456789abcdef0123456789abcdef";
+      ...
+    }
+
+Schema Version
+^^^^^^^^^^^^^^
+Schema version is defined through the ``schema_version`` node. It contains
+``major`` and ``minor`` properties as below::
+
+  /dts-v1/;
+  / {
+       schema_version {
+           major = <0x01>;
+           minor = <0x00>;
+       };
+       ...
+    }
+
+.. _partition_uuids:
+
+Partition UUIDs
+^^^^^^^^^^^^^^^
+Each partition may have parent and child UUIDs. These UUIDs are
+defined by ``interfaces`` node and ``interface_uuid`` property::
+
+  /dts-v1/;
+  / {
+       interfaces {
+           @0 {
+                  interface_uuid = "0123456789abcdef0123456789abcdef";
+           };
+           @1 {
+                  interface_uuid = "fedcba9876543210fedcba9876543210";
+           };
+           ...
+        };
+       ...
+    }
+
+
+Subsystem Instantiations
+^^^^^^^^^^^^^^^^^^^^^^^^
+Subsystem instantiations are captured as children of ``addressable_endpoints``
+node::
+
+  /dts-v1/;
+  / {
+       addressable_endpoints {
+           abc {
+               ...
+           };
+           def {
+               ...
+           };
+           ...
+       }
+  }
+
+Subnode 'abc' and 'def' are the name of subsystem nodes
+
+Subsystem Node
+^^^^^^^^^^^^^^
+Each subsystem node and its properties define a hardware instance::
+
+
+  addressable_endpoints {
+      abc {
+          reg = <0x00 0x1f05000 0x00 0x1000>>
+          pcie_physical_function = <0x0>;
+          pcie_bar_mapping = <0x2>;
+          compatible = "abc def";
+          interrupts = <0x09 0x0c>;
+          firmware {
+              firmware_product_name = "abc"
+              firmware_branch_name = "def"
+              firmware_version_major = <1>
+              firmware_version_minor = <2>
+          };
+      }
+      ...
+  }
+
+:reg:
+ Property defines an address range. `<0x00 0x1f05000 0x00 0x1000>` indicates
+ *0x00 0x1f05000* as BAR offset and *0x00 0x1000* as address length.
+:pcie_physical_function:
+ Property specifies which PCIe physical function the subsystem node resides.
+ `<0x0>` implies physical function 0.
+:pcie_bar_mapping:
+ Property specifies which PCIe BAR the subsystem node resides. `<0x2>` implies
+ BAR 2. A value of 0 means the property is not defined.
+:compatible:
+ Property is a list of strings. The first string in the list specifies the exact
+ subsystem node. The following strings represent other devices that the device
+ is compatible with.
+:interrupts:
+ Property specifies start and end interrupts for this subsystem node.
+ `<0x09 0x0c>` implies interrupts 9 to 13 are used by this subsystem.
+:firmware:
+ Subnode defines the firmware required by this subsystem node.
+
+Alveo U50 Platform Example
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+::
+
+  /dts-v1/;
+
+  /{
+        logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
+
+        schema_version {
+                major = <0x01>;
+                minor = <0x00>;
+        };
+
+        interfaces {
+
+                @0 {
+                        interface_uuid = "862c7020a250293e32036f19956669e5";
+                };
+        };
+
+        addressable_endpoints {
+
+                ep_blp_rom_00 {
+                        reg = <0x00 0x1f04000 0x00 0x1000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+                };
+
+                ep_card_flash_program_00 {
+                        reg = <0x00 0x1f06000 0x00 0x1000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_quad_spi-1.0\0axi_quad_spi";
+                        interrupts = <0x03 0x03>;
+                };
+
+                ep_cmc_firmware_mem_00 {
+                        reg = <0x00 0x1e20000 0x00 0x20000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+                        firmware {
+                                firmware_product_name = "cmc";
+                                firmware_branch_name = "u50";
+                                firmware_version_major = <0x01>;
+                                firmware_version_minor = <0x00>;
+                        };
+                };
+
+                ep_cmc_intc_00 {
+                        reg = <0x00 0x1e03000 0x00 0x1000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
+                        interrupts = <0x04 0x04>;
+                };
+
+                ep_cmc_mutex_00 {
+                        reg = <0x00 0x1e02000 0x00 0x1000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+                };
+
+                ep_cmc_regmap_00 {
+                        reg = <0x00 0x1e08000 0x00 0x2000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+                        firmware {
+                                firmware_product_name = "sc-fw";
+                                firmware_branch_name = "u50";
+                                firmware_version_major = <0x05>;
+                        };
+                };
+
+                ep_cmc_reset_00 {
+                        reg = <0x00 0x1e01000 0x00 0x1000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+                };
+
+                ep_ddr_mem_calib_00 {
+                        reg = <0x00 0x63000 0x00 0x1000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+                };
+
+                ep_debug_bscan_mgmt_00 {
+                        reg = <0x00 0x1e90000 0x00 0x10000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-debug_bridge-1.0\0debug_bridge";
+                };
+
+                ep_ert_base_address_00 {
+                        reg = <0x00 0x21000 0x00 0x1000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+                };
+
+                ep_ert_command_queue_mgmt_00 {
+                        reg = <0x00 0x40000 0x00 0x10000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
+                };
+
+                ep_ert_command_queue_user_00 {
+                        reg = <0x00 0x40000 0x00 0x10000>;
+                        pcie_physical_function = <0x01>;
+                        compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
+                };
+
+                ep_ert_firmware_mem_00 {
+                        reg = <0x00 0x30000 0x00 0x8000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+                        firmware {
+                                firmware_product_name = "ert";
+                                firmware_branch_name = "v20";
+                                firmware_version_major = <0x01>;
+                        };
+                };
+
+                ep_ert_intc_00 {
+                        reg = <0x00 0x23000 0x00 0x1000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
+                        interrupts = <0x05 0x05>;
+                };
+
+                ep_ert_reset_00 {
+                        reg = <0x00 0x22000 0x00 0x1000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+                };
+
+                ep_ert_sched_00 {
+                        reg = <0x00 0x50000 0x00 0x1000>;
+                        pcie_physical_function = <0x01>;
+                        compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
+                        interrupts = <0x09 0x0c>;
+                };
+
+                ep_fpga_configuration_00 {
+                        reg = <0x00 0x1e88000 0x00 0x8000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
+                        interrupts = <0x02 0x02>;
+                };
+
+                ep_icap_reset_00 {
+                        reg = <0x00 0x1f07000 0x00 0x1000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+                };
+
+                ep_msix_00 {
+                        reg = <0x00 0x00 0x00 0x20000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
+                        pcie_bar_mapping = <0x02>;
+                };
+
+                ep_pcie_link_mon_00 {
+                        reg = <0x00 0x1f05000 0x00 0x1000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+                };
+
+                ep_pr_isolate_plp_00 {
+                        reg = <0x00 0x1f01000 0x00 0x1000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+                };
+
+                ep_pr_isolate_ulp_00 {
+                        reg = <0x00 0x1000 0x00 0x1000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+                };
+
+                ep_uuid_rom_00 {
+                        reg = <0x00 0x64000 0x00 0x1000>;
+                        pcie_physical_function = <0x00>;
+                        compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+                };
+
+                ep_xdma_00 {
+                        reg = <0x00 0x00 0x00 0x10000>;
+                        pcie_physical_function = <0x01>;
+                        compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
+                        pcie_bar_mapping = <0x02>;
+                };
+        };
+
+  }
+
+
+
+Deployment Models
+=================
+
+Baremetal
+---------
+
+In bare-metal deployments, both MPF and UPF are visible and accessible. The
+xrt-mgmt driver binds to MPF. The xrt-mgmt driver operations are privileged and
+available to system administrator. The full stack is illustrated below::
+
+                            HOST
+
+               [XRT-MGMT]         [XRT-USER]
+                    |                  |
+                    |                  |
+                 +-----+            +-----+
+                 | MPF |            | UPF |
+                 |     |            |     |
+                 | PF0 |            | PF1 |
+                 +--+--+            +--+--+
+          ......... ^................. ^..........
+                    |                  |
+                    |   PCIe DEVICE    |
+                    |                  |
+                 +--+------------------+--+
+                 |         SHELL          |
+                 |                        |
+                 +------------------------+
+                 |         USER           |
+                 |                        |
+                 |                        |
+                 |                        |
+                 |                        |
+                 +------------------------+
+
+
+
+Virtualized
+-----------
+
+In virtualized deployments, the privileged MPF is assigned to the host but the
+unprivileged UPF is assigned to a guest VM via PCIe pass-through. The xrt-mgmt
+driver in host binds to MPF. The xrt-mgmt driver operations are privileged and
+only accessible to the MPF. The full stack is illustrated below::
+
+
+                                 ..............
+                  HOST           .    VM      .
+                                 .            .
+               [XRT-MGMT]        . [XRT-USER] .
+                    |            .     |      .
+                    |            .     |      .
+                 +-----+         .  +-----+   .
+                 | MPF |         .  | UPF |   .
+                 |     |         .  |     |   .
+                 | PF0 |         .  | PF1 |   .
+                 +--+--+         .  +--+--+   .
+          ......... ^................. ^..........
+                    |                  |
+                    |   PCIe DEVICE    |
+                    |                  |
+                 +--+------------------+--+
+                 |         SHELL          |
+                 |                        |
+                 +------------------------+
+                 |         USER           |
+                 |                        |
+                 |                        |
+                 |                        |
+                 |                        |
+                 +------------------------+
+
+
+
+
+
+Platform Security Considerations
+================================
+
+`Security of Alveo Platform <https://xilinx.github.io/XRT/master/html/security.html>`_
+discusses the deployment options and security implications in great detail.
diff --git a/MAINTAINERS b/MAINTAINERS
index 056966c9aac9..beeaf0257364 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7274,6 +7274,17 @@  F:	Documentation/fpga/
 F:	drivers/fpga/
 F:	include/linux/fpga/
 
+FPGA XRT DRIVERS
+M:	Lizhi Hou <[e-mail]>
+R:	Max Zhen <[e-mail]>
+R:	Sonal Santan <[e-mail]>
+L:	[e-mail]
+S:	Supported
+W:	https://github.com/Xilinx/XRT
+F:	Documentation/fpga/xrt.rst
+F:	drivers/fpga/xrt/
+F:	include/uapi/linux/xrt/
+
 FPU EMULATOR
 M:	Bill Metzenthen <[e-mail]>
 S:	Maintained