mbox series

[V8,XRT,Alveo,00/14] XRT Alveo driver overview

Message ID 20210719212628.134129-1-lizhi.hou@xilinx.com (mailing list archive)
Headers show
Series XRT Alveo driver overview | expand

Message

Lizhi Hou July 19, 2021, 9:26 p.m. UTC
Hello,

This is V8 of patch series which adds management physical function driver
for Xilinx Alveo PCIe accelerator cards.
    https://www.xilinx.com/products/boards-and-kits/alveo.html

This driver is part of Xilinx Runtime (XRT) open source stack.

XILINX ALVEO PLATFORM ARCHITECTURE

Alveo PCIe FPGA based platforms have a static *shell* partition and a
partial re-configurable *user* partition. The shell partition is
automatically loaded from flash when host is booted and PCIe is enumerated
by BIOS. Shell cannot be changed till the next cold reboot. The shell
exposes two PCIe physical functions:

1. management physical function
2. user physical function

The patch series includes Documentation/xrt.rst which describes Alveo
platform, XRT driver architecture and deployment model in more detail.

Users compile their high level design in C/C++/OpenCL or RTL into FPGA
image using Vitis tools.
    https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html

The compiled image is packaged as xclbin which contains partial bitstream
for the user partition and necessary metadata. Users can dynamically swap
the image running on the user partition in order to switch between
different workloads by loading different xclbins.

XRT DRIVERS FOR XILINX ALVEO

XRT Linux kernel driver *xrt-mgmt* binds to management physical function of
Alveo platform. The modular driver framework is organized into several
platform drivers which primarily handle the following functionality:

1.  Loading firmware container also called xsabin at driver attach time
2.  Loading of user compiled xclbin with FPGA Manager integration
3.  Clock scaling of image running on user partition
4.  In-band sensors: temp, voltage, power, etc.
5.  Device reset and rescan

The platform drivers are packaged into *xrt-lib* helper module with well
defined interfaces. The module provides a pseudo-bus implementation for the
platform drivers. More details on the driver model can be found in
Documentation/xrt.rst.

User physical function driver is not included in this patch series.

LIBFDT REQUIREMENT

XRT driver infrastructure uses Device Tree as a metadata format to discover
HW subsystems in the Alveo PCIe device. The Device Tree schema used by XRT
is documented in Documentation/xrt.rst.

TESTING AND VALIDATION

xrt-mgmt driver can be tested with full XRT open source stack which
includes user space libraries, board utilities and (out of tree) first
generation user physical function driver xocl. XRT open source runtime
stack is available at https://github.com/Xilinx/XRT

Complete documentation for XRT open source stack including sections on
Alveo/XRT security and platform architecture can be found here:

https://xilinx.github.io/XRT/master/html/index.html
https://xilinx.github.io/XRT/master/html/security.html
https://xilinx.github.io/XRT/master/html/platforms_partitions.html

Changes since v7:
- Followed review comment to remove non fpga subdevices:
   clock, clkfrq, ucs, ddr_calibaration, devctl and vsec
- Collapsed include/uapi/linux/xrt/*.h into include/uapi/linux/fpga-xrt.h
- Cleaned up comments in fpga-xrt.h
- Fixed spelling errors in xrt.rst

Changes since v6:
- Resolved grammatical errors and cleaned up taxonomy in xrt.rst
  documentation.
- Fixed clang warnings.
- Updated code base to include v6 code review comments.

Changes since v5:
- Revert all changes 'mgnt/MGNT' back to 'mgmt/MGMT'
- Updated code base to include v5 code review comments.
   xrt.rst: address grammar and taxonomy
   subdev_id.h: defines XRT_SUBDEV_INVALID = 0
   xclbin.c: change shift operation to be_to_cpu
- Resolved kernel test robot errors.

Changes since v4:
- Added xrt_bus_type and xrt_device. All sub devices were changed from
  platform_bus_type/platform_device to xrt_bus_type/xrt_device.
- Renamed xrt-mgmt driver to xrt-mgnt driver.
- Replaced 'MGMT' with 'MGNT' and 'mgmt' with 'mgnt' in code and file names
- Moved pci function calls from infrastructure to xrt-mgnt driver.
- Renamed files: mgmt/main.c -> mgnt/xmgnt-main.c
                 mgmt/main-region.c -> mgnt/xmgnt-main-region.c
                 include/xmgmt-main.h -> include/xmgnt-main.h
                 mgmt/fmgr-drv.c -> mgnt/xrt-mgr.c
                 mgmt/fmgr.h -> mgnt/xrt-mgr.h
- Updated code base to include v4 code review comments.

Changes since v3:
- Leaf drivers use regmap-mmio to access hardware registers.
- Renamed driver module: xmgmt.ko -> xrt-mgmt.ko
- Renamed files: calib.[c|h] -> ddr_calibration.[c|h],
                 lib/main.[c|h] -> lib/lib-drv.[c|h],
                 mgmt/main-impl.h - > mgmt/xmgnt.h
- Updated code base to include v3 code review comments.

Changes since v2:
- Streamlined the driver framework into *xleaf*, *group* and *xroot*
- Updated documentation to show the driver model with examples
- Addressed kernel test robot errors
- Added a selftest for basic driver framework
- Documented device tree schema
- Removed need to export libfdt symbols

Changes since v1:
- Updated the driver to use fpga_region and fpga_bridge for FPGA
  programming
- Dropped platform drivers not related to PR programming to focus on XRT
  core framework
- Updated Documentation/fpga/xrt.rst with information on XRT core framework
- Addressed checkpatch issues
- Dropped xrt- prefix from some header files

For reference V7 version of patch series can be found here:
https://lore.kernel.org/lkml/20210528004959.61354-1-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-2-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-3-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-4-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-5-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-6-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-7-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-8-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-9-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-10-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-11-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-12-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-13-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-14-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-15-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-16-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-17-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-18-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-19-lizhi.hou@xilinx.com/
https://lore.kernel.org/lkml/20210528004959.61354-20-lizhi.hou@xilinx.com/

Lizhi Hou (14):
  Documentation: fpga: Add a document describing XRT Alveo drivers
  fpga: xrt: driver metadata helper functions
  fpga: xrt: xclbin file helper functions
  fpga: xrt: xrt-lib driver manager
  fpga: xrt: group driver
  fpga: xrt: char dev node helper functions
  fpga: xrt: root driver infrastructure
  fpga: xrt: driver infrastructure
  fpga: xrt: management physical function driver (root)
  fpga: xrt: main driver for management function device
  fpga: xrt: fpga-mgr and region implementation for xclbin download
  fpga: xrt: ICAP driver
  fpga: xrt: partition isolation driver
  fpga: xrt: Kconfig and Makefile updates for XRT drivers

 Documentation/fpga/index.rst                  |   1 +
 Documentation/fpga/xrt.rst                    | 870 ++++++++++++++++++
 MAINTAINERS                                   |  11 +
 drivers/Makefile                              |   1 +
 drivers/fpga/Kconfig                          |   2 +
 drivers/fpga/Makefile                         |   5 +
 drivers/fpga/xrt/Kconfig                      |   8 +
 drivers/fpga/xrt/include/events.h             |  45 +
 drivers/fpga/xrt/include/group.h              |  25 +
 drivers/fpga/xrt/include/metadata.h           | 236 +++++
 drivers/fpga/xrt/include/subdev_id.h          |  39 +
 drivers/fpga/xrt/include/xclbin-helper.h      |  48 +
 drivers/fpga/xrt/include/xdevice.h            | 131 +++
 drivers/fpga/xrt/include/xleaf.h              | 205 +++++
 drivers/fpga/xrt/include/xleaf/axigate.h      |  23 +
 drivers/fpga/xrt/include/xleaf/clkfreq.h      |  21 +
 drivers/fpga/xrt/include/xleaf/clock.h        |  29 +
 .../fpga/xrt/include/xleaf/ddr_calibration.h  |  28 +
 drivers/fpga/xrt/include/xleaf/devctl.h       |  40 +
 drivers/fpga/xrt/include/xleaf/icap.h         |  27 +
 drivers/fpga/xrt/include/xmgmt-main.h         |  34 +
 drivers/fpga/xrt/include/xroot.h              | 117 +++
 drivers/fpga/xrt/lib/Kconfig                  |  17 +
 drivers/fpga/xrt/lib/Makefile                 |  24 +
 drivers/fpga/xrt/lib/cdev.c                   | 209 +++++
 drivers/fpga/xrt/lib/group.c                  | 278 ++++++
 drivers/fpga/xrt/lib/lib-drv.c                | 322 +++++++
 drivers/fpga/xrt/lib/lib-drv.h                |  21 +
 drivers/fpga/xrt/lib/subdev.c                 | 859 +++++++++++++++++
 drivers/fpga/xrt/lib/subdev_pool.h            |  53 ++
 drivers/fpga/xrt/lib/xclbin.c                 | 381 ++++++++
 drivers/fpga/xrt/lib/xleaf/axigate.c          | 325 +++++++
 drivers/fpga/xrt/lib/xleaf/icap.c             | 328 +++++++
 drivers/fpga/xrt/lib/xroot.c                  | 536 +++++++++++
 drivers/fpga/xrt/metadata/Kconfig             |  12 +
 drivers/fpga/xrt/metadata/Makefile            |  16 +
 drivers/fpga/xrt/metadata/metadata.c          | 578 ++++++++++++
 drivers/fpga/xrt/mgmt/Kconfig                 |  15 +
 drivers/fpga/xrt/mgmt/Makefile                |  19 +
 drivers/fpga/xrt/mgmt/root.c                  | 420 +++++++++
 drivers/fpga/xrt/mgmt/xmgmt-main-region.c     | 483 ++++++++++
 drivers/fpga/xrt/mgmt/xmgmt-main.c            | 662 +++++++++++++
 drivers/fpga/xrt/mgmt/xmgmt.h                 |  33 +
 drivers/fpga/xrt/mgmt/xrt-mgr.c               | 190 ++++
 drivers/fpga/xrt/mgmt/xrt-mgr.h               |  16 +
 include/uapi/linux/fpga-xrt.h                 | 428 +++++++++
 46 files changed, 8171 insertions(+)
 create mode 100644 Documentation/fpga/xrt.rst
 create mode 100644 drivers/fpga/xrt/Kconfig
 create mode 100644 drivers/fpga/xrt/include/events.h
 create mode 100644 drivers/fpga/xrt/include/group.h
 create mode 100644 drivers/fpga/xrt/include/metadata.h
 create mode 100644 drivers/fpga/xrt/include/subdev_id.h
 create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
 create mode 100644 drivers/fpga/xrt/include/xdevice.h
 create mode 100644 drivers/fpga/xrt/include/xleaf.h
 create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
 create mode 100644 drivers/fpga/xrt/include/xleaf/clkfreq.h
 create mode 100644 drivers/fpga/xrt/include/xleaf/clock.h
 create mode 100644 drivers/fpga/xrt/include/xleaf/ddr_calibration.h
 create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
 create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
 create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
 create mode 100644 drivers/fpga/xrt/include/xroot.h
 create mode 100644 drivers/fpga/xrt/lib/Kconfig
 create mode 100644 drivers/fpga/xrt/lib/Makefile
 create mode 100644 drivers/fpga/xrt/lib/cdev.c
 create mode 100644 drivers/fpga/xrt/lib/group.c
 create mode 100644 drivers/fpga/xrt/lib/lib-drv.c
 create mode 100644 drivers/fpga/xrt/lib/lib-drv.h
 create mode 100644 drivers/fpga/xrt/lib/subdev.c
 create mode 100644 drivers/fpga/xrt/lib/subdev_pool.h
 create mode 100644 drivers/fpga/xrt/lib/xclbin.c
 create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
 create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
 create mode 100644 drivers/fpga/xrt/lib/xroot.c
 create mode 100644 drivers/fpga/xrt/metadata/Kconfig
 create mode 100644 drivers/fpga/xrt/metadata/Makefile
 create mode 100644 drivers/fpga/xrt/metadata/metadata.c
 create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
 create mode 100644 drivers/fpga/xrt/mgmt/Makefile
 create mode 100644 drivers/fpga/xrt/mgmt/root.c
 create mode 100644 drivers/fpga/xrt/mgmt/xmgmt-main-region.c
 create mode 100644 drivers/fpga/xrt/mgmt/xmgmt-main.c
 create mode 100644 drivers/fpga/xrt/mgmt/xmgmt.h
 create mode 100644 drivers/fpga/xrt/mgmt/xrt-mgr.c
 create mode 100644 drivers/fpga/xrt/mgmt/xrt-mgr.h
 create mode 100644 include/uapi/linux/fpga-xrt.h

Comments

Tom Rix July 20, 2021, 7:01 p.m. UTC | #1
On 7/19/21 2:26 PM, Lizhi Hou wrote:
> Hello,
>
> This is V8 of patch series which adds management physical function driver
> for Xilinx Alveo PCIe accelerator cards.
>      https://www.xilinx.com/products/boards-and-kits/alveo.html
>
> This driver is part of Xilinx Runtime (XRT) open source stack.
>
> XILINX ALVEO PLATFORM ARCHITECTURE
>
> Alveo PCIe FPGA based platforms have a static *shell* partition and a
> partial re-configurable *user* partition. The shell partition is
> automatically loaded from flash when host is booted and PCIe is enumerated
> by BIOS. Shell cannot be changed till the next cold reboot. The shell
> exposes two PCIe physical functions:
>
> 1. management physical function
> 2. user physical function
>
> The patch series includes Documentation/xrt.rst which describes Alveo
> platform, XRT driver architecture and deployment model in more detail.
>
> Users compile their high level design in C/C++/OpenCL or RTL into FPGA
> image using Vitis tools.
>      https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
>
> The compiled image is packaged as xclbin which contains partial bitstream
> for the user partition and necessary metadata. Users can dynamically swap
> the image running on the user partition in order to switch between
> different workloads by loading different xclbins.
>
> XRT DRIVERS FOR XILINX ALVEO
>
> XRT Linux kernel driver *xrt-mgmt* binds to management physical function of
> Alveo platform. The modular driver framework is organized into several
> platform drivers which primarily handle the following functionality:
>
> 1.  Loading firmware container also called xsabin at driver attach time
> 2.  Loading of user compiled xclbin with FPGA Manager integration
> 3.  Clock scaling of image running on user partition
> 4.  In-band sensors: temp, voltage, power, etc.
> 5.  Device reset and rescan
>
> The platform drivers are packaged into *xrt-lib* helper module with well
> defined interfaces. The module provides a pseudo-bus implementation for the
> platform drivers. More details on the driver model can be found in
> Documentation/xrt.rst.
>
> User physical function driver is not included in this patch series.
>
> LIBFDT REQUIREMENT
>
> XRT driver infrastructure uses Device Tree as a metadata format to discover
> HW subsystems in the Alveo PCIe device. The Device Tree schema used by XRT
> is documented in Documentation/xrt.rst.
>
> TESTING AND VALIDATION
>
> xrt-mgmt driver can be tested with full XRT open source stack which
> includes user space libraries, board utilities and (out of tree) first
> generation user physical function driver xocl. XRT open source runtime
> stack is available at https://github.com/Xilinx/XRT
>
> Complete documentation for XRT open source stack including sections on
> Alveo/XRT security and platform architecture can be found here:
>
> https://xilinx.github.io/XRT/master/html/index.html
> https://xilinx.github.io/XRT/master/html/security.html
> https://xilinx.github.io/XRT/master/html/platforms_partitions.html
>
> Changes since v7:
> - Followed review comment to remove non fpga subdevices:
>     clock, clkfrq, ucs, ddr_calibaration, devctl and vsec
> - Collapsed include/uapi/linux/xrt/*.h into include/uapi/linux/fpga-xrt.h
> - Cleaned up comments in fpga-xrt.h
> - Fixed spelling errors in xrt.rst

Lizhi,

Thanks, the changes look good.

Moritz,

I have no outstanding issues and I do not believe there are any from 
anyone else.

Can this set be included in fpga-next ?

Tom

>
> Changes since v6:
> - Resolved grammatical errors and cleaned up taxonomy in xrt.rst
>    documentation.
> - Fixed clang warnings.
> - Updated code base to include v6 code review comments.
>
> Changes since v5:
> - Revert all changes 'mgnt/MGNT' back to 'mgmt/MGMT'
> - Updated code base to include v5 code review comments.
>     xrt.rst: address grammar and taxonomy
>     subdev_id.h: defines XRT_SUBDEV_INVALID = 0
>     xclbin.c: change shift operation to be_to_cpu
> - Resolved kernel test robot errors.
>
> Changes since v4:
> - Added xrt_bus_type and xrt_device. All sub devices were changed from
>    platform_bus_type/platform_device to xrt_bus_type/xrt_device.
> - Renamed xrt-mgmt driver to xrt-mgnt driver.
> - Replaced 'MGMT' with 'MGNT' and 'mgmt' with 'mgnt' in code and file names
> - Moved pci function calls from infrastructure to xrt-mgnt driver.
> - Renamed files: mgmt/main.c -> mgnt/xmgnt-main.c
>                   mgmt/main-region.c -> mgnt/xmgnt-main-region.c
>                   include/xmgmt-main.h -> include/xmgnt-main.h
>                   mgmt/fmgr-drv.c -> mgnt/xrt-mgr.c
>                   mgmt/fmgr.h -> mgnt/xrt-mgr.h
> - Updated code base to include v4 code review comments.
>
> Changes since v3:
> - Leaf drivers use regmap-mmio to access hardware registers.
> - Renamed driver module: xmgmt.ko -> xrt-mgmt.ko
> - Renamed files: calib.[c|h] -> ddr_calibration.[c|h],
>                   lib/main.[c|h] -> lib/lib-drv.[c|h],
>                   mgmt/main-impl.h - > mgmt/xmgnt.h
> - Updated code base to include v3 code review comments.
>
> Changes since v2:
> - Streamlined the driver framework into *xleaf*, *group* and *xroot*
> - Updated documentation to show the driver model with examples
> - Addressed kernel test robot errors
> - Added a selftest for basic driver framework
> - Documented device tree schema
> - Removed need to export libfdt symbols
>
> Changes since v1:
> - Updated the driver to use fpga_region and fpga_bridge for FPGA
>    programming
> - Dropped platform drivers not related to PR programming to focus on XRT
>    core framework
> - Updated Documentation/fpga/xrt.rst with information on XRT core framework
> - Addressed checkpatch issues
> - Dropped xrt- prefix from some header files
>
> For reference V7 version of patch series can be found here:
> https://lore.kernel.org/lkml/20210528004959.61354-1-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-2-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-3-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-4-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-5-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-6-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-7-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-8-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-9-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-10-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-11-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-12-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-13-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-14-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-15-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-16-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-17-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-18-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-19-lizhi.hou@xilinx.com/
> https://lore.kernel.org/lkml/20210528004959.61354-20-lizhi.hou@xilinx.com/
>
> Lizhi Hou (14):
>    Documentation: fpga: Add a document describing XRT Alveo drivers
>    fpga: xrt: driver metadata helper functions
>    fpga: xrt: xclbin file helper functions
>    fpga: xrt: xrt-lib driver manager
>    fpga: xrt: group driver
>    fpga: xrt: char dev node helper functions
>    fpga: xrt: root driver infrastructure
>    fpga: xrt: driver infrastructure
>    fpga: xrt: management physical function driver (root)
>    fpga: xrt: main driver for management function device
>    fpga: xrt: fpga-mgr and region implementation for xclbin download
>    fpga: xrt: ICAP driver
>    fpga: xrt: partition isolation driver
>    fpga: xrt: Kconfig and Makefile updates for XRT drivers
>
>   Documentation/fpga/index.rst                  |   1 +
>   Documentation/fpga/xrt.rst                    | 870 ++++++++++++++++++
>   MAINTAINERS                                   |  11 +
>   drivers/Makefile                              |   1 +
>   drivers/fpga/Kconfig                          |   2 +
>   drivers/fpga/Makefile                         |   5 +
>   drivers/fpga/xrt/Kconfig                      |   8 +
>   drivers/fpga/xrt/include/events.h             |  45 +
>   drivers/fpga/xrt/include/group.h              |  25 +
>   drivers/fpga/xrt/include/metadata.h           | 236 +++++
>   drivers/fpga/xrt/include/subdev_id.h          |  39 +
>   drivers/fpga/xrt/include/xclbin-helper.h      |  48 +
>   drivers/fpga/xrt/include/xdevice.h            | 131 +++
>   drivers/fpga/xrt/include/xleaf.h              | 205 +++++
>   drivers/fpga/xrt/include/xleaf/axigate.h      |  23 +
>   drivers/fpga/xrt/include/xleaf/clkfreq.h      |  21 +
>   drivers/fpga/xrt/include/xleaf/clock.h        |  29 +
>   .../fpga/xrt/include/xleaf/ddr_calibration.h  |  28 +
>   drivers/fpga/xrt/include/xleaf/devctl.h       |  40 +
>   drivers/fpga/xrt/include/xleaf/icap.h         |  27 +
>   drivers/fpga/xrt/include/xmgmt-main.h         |  34 +
>   drivers/fpga/xrt/include/xroot.h              | 117 +++
>   drivers/fpga/xrt/lib/Kconfig                  |  17 +
>   drivers/fpga/xrt/lib/Makefile                 |  24 +
>   drivers/fpga/xrt/lib/cdev.c                   | 209 +++++
>   drivers/fpga/xrt/lib/group.c                  | 278 ++++++
>   drivers/fpga/xrt/lib/lib-drv.c                | 322 +++++++
>   drivers/fpga/xrt/lib/lib-drv.h                |  21 +
>   drivers/fpga/xrt/lib/subdev.c                 | 859 +++++++++++++++++
>   drivers/fpga/xrt/lib/subdev_pool.h            |  53 ++
>   drivers/fpga/xrt/lib/xclbin.c                 | 381 ++++++++
>   drivers/fpga/xrt/lib/xleaf/axigate.c          | 325 +++++++
>   drivers/fpga/xrt/lib/xleaf/icap.c             | 328 +++++++
>   drivers/fpga/xrt/lib/xroot.c                  | 536 +++++++++++
>   drivers/fpga/xrt/metadata/Kconfig             |  12 +
>   drivers/fpga/xrt/metadata/Makefile            |  16 +
>   drivers/fpga/xrt/metadata/metadata.c          | 578 ++++++++++++
>   drivers/fpga/xrt/mgmt/Kconfig                 |  15 +
>   drivers/fpga/xrt/mgmt/Makefile                |  19 +
>   drivers/fpga/xrt/mgmt/root.c                  | 420 +++++++++
>   drivers/fpga/xrt/mgmt/xmgmt-main-region.c     | 483 ++++++++++
>   drivers/fpga/xrt/mgmt/xmgmt-main.c            | 662 +++++++++++++
>   drivers/fpga/xrt/mgmt/xmgmt.h                 |  33 +
>   drivers/fpga/xrt/mgmt/xrt-mgr.c               | 190 ++++
>   drivers/fpga/xrt/mgmt/xrt-mgr.h               |  16 +
>   include/uapi/linux/fpga-xrt.h                 | 428 +++++++++
>   46 files changed, 8171 insertions(+)
>   create mode 100644 Documentation/fpga/xrt.rst
>   create mode 100644 drivers/fpga/xrt/Kconfig
>   create mode 100644 drivers/fpga/xrt/include/events.h
>   create mode 100644 drivers/fpga/xrt/include/group.h
>   create mode 100644 drivers/fpga/xrt/include/metadata.h
>   create mode 100644 drivers/fpga/xrt/include/subdev_id.h
>   create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
>   create mode 100644 drivers/fpga/xrt/include/xdevice.h
>   create mode 100644 drivers/fpga/xrt/include/xleaf.h
>   create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
>   create mode 100644 drivers/fpga/xrt/include/xleaf/clkfreq.h
>   create mode 100644 drivers/fpga/xrt/include/xleaf/clock.h
>   create mode 100644 drivers/fpga/xrt/include/xleaf/ddr_calibration.h
>   create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
>   create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
>   create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
>   create mode 100644 drivers/fpga/xrt/include/xroot.h
>   create mode 100644 drivers/fpga/xrt/lib/Kconfig
>   create mode 100644 drivers/fpga/xrt/lib/Makefile
>   create mode 100644 drivers/fpga/xrt/lib/cdev.c
>   create mode 100644 drivers/fpga/xrt/lib/group.c
>   create mode 100644 drivers/fpga/xrt/lib/lib-drv.c
>   create mode 100644 drivers/fpga/xrt/lib/lib-drv.h
>   create mode 100644 drivers/fpga/xrt/lib/subdev.c
>   create mode 100644 drivers/fpga/xrt/lib/subdev_pool.h
>   create mode 100644 drivers/fpga/xrt/lib/xclbin.c
>   create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
>   create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
>   create mode 100644 drivers/fpga/xrt/lib/xroot.c
>   create mode 100644 drivers/fpga/xrt/metadata/Kconfig
>   create mode 100644 drivers/fpga/xrt/metadata/Makefile
>   create mode 100644 drivers/fpga/xrt/metadata/metadata.c
>   create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
>   create mode 100644 drivers/fpga/xrt/mgmt/Makefile
>   create mode 100644 drivers/fpga/xrt/mgmt/root.c
>   create mode 100644 drivers/fpga/xrt/mgmt/xmgmt-main-region.c
>   create mode 100644 drivers/fpga/xrt/mgmt/xmgmt-main.c
>   create mode 100644 drivers/fpga/xrt/mgmt/xmgmt.h
>   create mode 100644 drivers/fpga/xrt/mgmt/xrt-mgr.c
>   create mode 100644 drivers/fpga/xrt/mgmt/xrt-mgr.h
>   create mode 100644 include/uapi/linux/fpga-xrt.h
>
Moritz Fischer July 22, 2021, 3:20 a.m. UTC | #2
Hi all,

On Tue, Jul 20, 2021 at 12:01:34PM -0700, Tom Rix wrote:
> 
> On 7/19/21 2:26 PM, Lizhi Hou wrote:
> > Hello,
> > 
> > This is V8 of patch series which adds management physical function driver
> > for Xilinx Alveo PCIe accelerator cards.
> >      https://www.xilinx.com/products/boards-and-kits/alveo.html
> > 
> > This driver is part of Xilinx Runtime (XRT) open source stack.
> > 
> > XILINX ALVEO PLATFORM ARCHITECTURE
> > 
> > Alveo PCIe FPGA based platforms have a static *shell* partition and a
> > partial re-configurable *user* partition. The shell partition is
> > automatically loaded from flash when host is booted and PCIe is enumerated
> > by BIOS. Shell cannot be changed till the next cold reboot. The shell
> > exposes two PCIe physical functions:
> > 
> > 1. management physical function
> > 2. user physical function
> > 
> > The patch series includes Documentation/xrt.rst which describes Alveo
> > platform, XRT driver architecture and deployment model in more detail.
> > 
> > Users compile their high level design in C/C++/OpenCL or RTL into FPGA
> > image using Vitis tools.
> >      https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
> > 
> > The compiled image is packaged as xclbin which contains partial bitstream
> > for the user partition and necessary metadata. Users can dynamically swap
> > the image running on the user partition in order to switch between
> > different workloads by loading different xclbins.
> > 
> > XRT DRIVERS FOR XILINX ALVEO
> > 
> > XRT Linux kernel driver *xrt-mgmt* binds to management physical function of
> > Alveo platform. The modular driver framework is organized into several
> > platform drivers which primarily handle the following functionality:
> > 
> > 1.  Loading firmware container also called xsabin at driver attach time
> > 2.  Loading of user compiled xclbin with FPGA Manager integration
> > 3.  Clock scaling of image running on user partition
> > 4.  In-band sensors: temp, voltage, power, etc.
> > 5.  Device reset and rescan
> > 
> > The platform drivers are packaged into *xrt-lib* helper module with well
> > defined interfaces. The module provides a pseudo-bus implementation for the
> > platform drivers. More details on the driver model can be found in
> > Documentation/xrt.rst.
> > 
> > User physical function driver is not included in this patch series.
> > 
> > LIBFDT REQUIREMENT
> > 
> > XRT driver infrastructure uses Device Tree as a metadata format to discover
> > HW subsystems in the Alveo PCIe device. The Device Tree schema used by XRT
> > is documented in Documentation/xrt.rst.
> > 
> > TESTING AND VALIDATION
> > 
> > xrt-mgmt driver can be tested with full XRT open source stack which
> > includes user space libraries, board utilities and (out of tree) first
> > generation user physical function driver xocl. XRT open source runtime
> > stack is available at https://github.com/Xilinx/XRT
> > 
> > Complete documentation for XRT open source stack including sections on
> > Alveo/XRT security and platform architecture can be found here:
> > 
> > https://xilinx.github.io/XRT/master/html/index.html
> > https://xilinx.github.io/XRT/master/html/security.html
> > https://xilinx.github.io/XRT/master/html/platforms_partitions.html
> > 
> > Changes since v7:
> > - Followed review comment to remove non fpga subdevices:
> >     clock, clkfrq, ucs, ddr_calibaration, devctl and vsec
> > - Collapsed include/uapi/linux/xrt/*.h into include/uapi/linux/fpga-xrt.h
> > - Cleaned up comments in fpga-xrt.h
> > - Fixed spelling errors in xrt.rst
> 
> Lizhi,
> 
> Thanks, the changes look good.
> 
> Moritz,
> 
> I have no outstanding issues and I do not believe there are any from anyone
> else.
> 
> Can this set be included in fpga-next ?

I'll get to it this weekend, apologies for the delay guys.

- Moritz