mbox series

[V3,XRT,Alveo,00/18] XRT Alveo driver overview

Message ID 20210218064019.29189-1-lizhih@xilinx.com (mailing list archive)
Headers show
Series XRT Alveo driver overview | expand

Message

Lizhi Hou Feb. 18, 2021, 6:40 a.m. UTC
Hello,

This is V3 of patch series which adds management physical function driver for Xilinx
Alveo PCIe accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
This driver is part of Xilinx Runtime (XRT) open source stack.

XILINX ALVEO PLATFORM ARCHITECTURE

Alveo PCIe FPGA based platforms have a static *shell* partition and a partial
re-configurable *user* partition. The shell partition is automatically loaded from
flash when host is booted and PCIe is enumerated by BIOS. Shell cannot be changed
till the next cold reboot. The shell exposes two PCIe physical functions:

1. management physical function
2. user physical function

The patch series includes Documentation/xrt.rst which describes Alveo platform,
XRT driver architecture and deployment model in more detail.

Users compile their high level design in C/C++/OpenCL or RTL into FPGA image using
Vitis https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
tools. The compiled image is packaged as xclbin which contains partial bitstream
for the user partition and necessary metadata. Users can dynamically swap the image
running on the user partition in order to switch between different workloads by
loading different xclbins.

XRT DRIVERS FOR XILINX ALVEO

XRT Linux kernel driver *xmgmt* binds to management physical function of Alveo
platform. The modular driver framework is organized into several platform drivers
which primarily handle the following functionality:

1.  Loading firmware container also called xsabin at driver attach time
2.  Loading of user compiled xclbin with FPGA Manager integration
3.  Clock scaling of image running on user partition
4.  In-band sensors: temp, voltage, power, etc.
5.  Device reset and rescan

The platform drivers are packaged into *xrt-lib* helper module with well
defined interfaces. The module provides a pseudo-bus implementation for the
platform drivers. More details on the driver model can be found in
Documentation/xrt.rst.

User physical function driver is not included in this patch series.

LIBFDT REQUIREMENT

XRT driver infrastructure uses Device Tree as a metadata format to discover
HW subsystems in the Alveo PCIe device. The Device Tree schema used by XRT
is documented in Documentation/xrt.rst. Unlike previous V1 and V2 version
of patch series, V3 version does not require export of libfdt symbols.

TESTING AND VALIDATION

xmgmt driver can be tested with full XRT open source stack which includes user
space libraries, board utilities and (out of tree) first generation user physical
function driver xocl. XRT open source runtime stack is available at
https://github.com/Xilinx/XRT

Complete documentation for XRT open source stack including sections on Alveo/XRT
security and platform architecture can be found here:

https://xilinx.github.io/XRT/master/html/index.html
https://xilinx.github.io/XRT/master/html/security.html
https://xilinx.github.io/XRT/master/html/platforms_partitions.html

Changes since v2:
- Streamlined the driver framework into *xleaf*, *group* and *xroot*
- Updated documentation to show the driver model with examples
- Addressed kernel test robot errors
- Added a selftest for basic driver framework
- Documented device tree schema
- Removed need to export libfdt symbols

Changes since v1:
- Updated the driver to use fpga_region and fpga_bridge for FPGA
  programming
- Dropped platform drivers not related to PR programming to focus on XRT
  core framework
- Updated Documentation/fpga/xrt.rst with information on XRT core framework
- Addressed checkpatch issues
- Dropped xrt- prefix from some header files

For reference V1 version of patch series can be found here:

https://lore.kernel.org/lkml/20201217075046.28553-1-sonals@xilinx.com/
https://lore.kernel.org/lkml/20201217075046.28553-2-sonals@xilinx.com/
https://lore.kernel.org/lkml/20201217075046.28553-3-sonals@xilinx.com/
https://lore.kernel.org/lkml/20201217075046.28553-4-sonals@xilinx.com/
https://lore.kernel.org/lkml/20201217075046.28553-5-sonals@xilinx.com/
https://lore.kernel.org/lkml/20201217075046.28553-6-sonals@xilinx.com/
https://lore.kernel.org/lkml/20201217075046.28553-7-sonals@xilinx.com/

Lizhi Hou (18):
  Documentation: fpga: Add a document describing XRT Alveo drivers
  fpga: xrt: driver metadata helper functions
  fpga: xrt: xclbin file helper functions
  fpga: xrt: xrt-lib platform driver manager
  fpga: xrt: group platform driver
  fpga: xrt: platform driver infrastructure
  fpga: xrt: management physical function driver (root)
  fpga: xrt: main platform driver for management function device
  fpga: xrt: fpga-mgr and region implementation for xclbin download
  fpga: xrt: VSEC platform driver
  fpga: xrt: UCS platform driver
  fpga: xrt: ICAP platform driver
  fpga: xrt: devctl platform driver
  fpga: xrt: clock platform driver
  fpga: xrt: clock frequence counter platform driver
  fpga: xrt: DDR calibration platform driver
  fpga: xrt: partition isolation platform driver
  fpga: xrt: Kconfig and Makefile updates for XRT drivers

 Documentation/fpga/index.rst             |   1 +
 Documentation/fpga/xrt.rst               | 842 ++++++++++++++++++++++
 MAINTAINERS                              |  11 +
 drivers/Makefile                         |   1 +
 drivers/fpga/Kconfig                     |   2 +
 drivers/fpga/Makefile                    |   4 +
 drivers/fpga/xrt/Kconfig                 |   8 +
 drivers/fpga/xrt/include/events.h        |  48 ++
 drivers/fpga/xrt/include/group.h         |  27 +
 drivers/fpga/xrt/include/metadata.h      | 229 ++++++
 drivers/fpga/xrt/include/subdev_id.h     |  43 ++
 drivers/fpga/xrt/include/xclbin-helper.h |  52 ++
 drivers/fpga/xrt/include/xleaf.h         | 276 +++++++
 drivers/fpga/xrt/include/xleaf/axigate.h |  25 +
 drivers/fpga/xrt/include/xleaf/calib.h   |  30 +
 drivers/fpga/xrt/include/xleaf/clkfreq.h |  23 +
 drivers/fpga/xrt/include/xleaf/clock.h   |  31 +
 drivers/fpga/xrt/include/xleaf/devctl.h  |  43 ++
 drivers/fpga/xrt/include/xleaf/icap.h    |  29 +
 drivers/fpga/xrt/include/xleaf/ucs.h     |  24 +
 drivers/fpga/xrt/include/xmgmt-main.h    |  37 +
 drivers/fpga/xrt/include/xroot.h         | 114 +++
 drivers/fpga/xrt/lib/Kconfig             |  16 +
 drivers/fpga/xrt/lib/Makefile            |  30 +
 drivers/fpga/xrt/lib/cdev.c              | 231 ++++++
 drivers/fpga/xrt/lib/group.c             | 265 +++++++
 drivers/fpga/xrt/lib/main.c              | 274 +++++++
 drivers/fpga/xrt/lib/main.h              |  17 +
 drivers/fpga/xrt/lib/subdev.c            | 871 +++++++++++++++++++++++
 drivers/fpga/xrt/lib/subdev_pool.h       |  53 ++
 drivers/fpga/xrt/lib/xclbin.c            | 394 ++++++++++
 drivers/fpga/xrt/lib/xleaf/axigate.c     | 298 ++++++++
 drivers/fpga/xrt/lib/xleaf/calib.c       | 226 ++++++
 drivers/fpga/xrt/lib/xleaf/clkfreq.c     | 221 ++++++
 drivers/fpga/xrt/lib/xleaf/clock.c       | 648 +++++++++++++++++
 drivers/fpga/xrt/lib/xleaf/devctl.c      | 206 ++++++
 drivers/fpga/xrt/lib/xleaf/icap.c        | 317 +++++++++
 drivers/fpga/xrt/lib/xleaf/ucs.c         | 235 ++++++
 drivers/fpga/xrt/lib/xleaf/vsec.c        | 359 ++++++++++
 drivers/fpga/xrt/lib/xroot.c             | 598 ++++++++++++++++
 drivers/fpga/xrt/metadata/Kconfig        |  12 +
 drivers/fpga/xrt/metadata/Makefile       |  16 +
 drivers/fpga/xrt/metadata/metadata.c     | 524 ++++++++++++++
 drivers/fpga/xrt/mgmt/Kconfig            |  15 +
 drivers/fpga/xrt/mgmt/Makefile           |  19 +
 drivers/fpga/xrt/mgmt/fmgr-drv.c         | 187 +++++
 drivers/fpga/xrt/mgmt/fmgr.h             |  28 +
 drivers/fpga/xrt/mgmt/main-impl.h        |  37 +
 drivers/fpga/xrt/mgmt/main-region.c      | 471 ++++++++++++
 drivers/fpga/xrt/mgmt/main.c             | 693 ++++++++++++++++++
 drivers/fpga/xrt/mgmt/root.c             | 342 +++++++++
 include/uapi/linux/xrt/xclbin.h          | 408 +++++++++++
 include/uapi/linux/xrt/xmgmt-ioctl.h     |  46 ++
 53 files changed, 9957 insertions(+)
 create mode 100644 Documentation/fpga/xrt.rst
 create mode 100644 drivers/fpga/xrt/Kconfig
 create mode 100644 drivers/fpga/xrt/include/events.h
 create mode 100644 drivers/fpga/xrt/include/group.h
 create mode 100644 drivers/fpga/xrt/include/metadata.h
 create mode 100644 drivers/fpga/xrt/include/subdev_id.h
 create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
 create mode 100644 drivers/fpga/xrt/include/xleaf.h
 create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
 create mode 100644 drivers/fpga/xrt/include/xleaf/calib.h
 create mode 100644 drivers/fpga/xrt/include/xleaf/clkfreq.h
 create mode 100644 drivers/fpga/xrt/include/xleaf/clock.h
 create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
 create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
 create mode 100644 drivers/fpga/xrt/include/xleaf/ucs.h
 create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
 create mode 100644 drivers/fpga/xrt/include/xroot.h
 create mode 100644 drivers/fpga/xrt/lib/Kconfig
 create mode 100644 drivers/fpga/xrt/lib/Makefile
 create mode 100644 drivers/fpga/xrt/lib/cdev.c
 create mode 100644 drivers/fpga/xrt/lib/group.c
 create mode 100644 drivers/fpga/xrt/lib/main.c
 create mode 100644 drivers/fpga/xrt/lib/main.h
 create mode 100644 drivers/fpga/xrt/lib/subdev.c
 create mode 100644 drivers/fpga/xrt/lib/subdev_pool.h
 create mode 100644 drivers/fpga/xrt/lib/xclbin.c
 create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
 create mode 100644 drivers/fpga/xrt/lib/xleaf/calib.c
 create mode 100644 drivers/fpga/xrt/lib/xleaf/clkfreq.c
 create mode 100644 drivers/fpga/xrt/lib/xleaf/clock.c
 create mode 100644 drivers/fpga/xrt/lib/xleaf/devctl.c
 create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
 create mode 100644 drivers/fpga/xrt/lib/xleaf/ucs.c
 create mode 100644 drivers/fpga/xrt/lib/xleaf/vsec.c
 create mode 100644 drivers/fpga/xrt/lib/xroot.c
 create mode 100644 drivers/fpga/xrt/metadata/Kconfig
 create mode 100644 drivers/fpga/xrt/metadata/Makefile
 create mode 100644 drivers/fpga/xrt/metadata/metadata.c
 create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
 create mode 100644 drivers/fpga/xrt/mgmt/Makefile
 create mode 100644 drivers/fpga/xrt/mgmt/fmgr-drv.c
 create mode 100644 drivers/fpga/xrt/mgmt/fmgr.h
 create mode 100644 drivers/fpga/xrt/mgmt/main-impl.h
 create mode 100644 drivers/fpga/xrt/mgmt/main-region.c
 create mode 100644 drivers/fpga/xrt/mgmt/main.c
 create mode 100644 drivers/fpga/xrt/mgmt/root.c
 create mode 100644 include/uapi/linux/xrt/xclbin.h
 create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h

Comments

Tom Rix Feb. 18, 2021, 1:52 p.m. UTC | #1
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> Hello,
>
> This is V3 of patch series which adds management physical function driver for Xilinx
> Alveo PCIe accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
> This driver is part of Xilinx Runtime (XRT) open source stack.
>
> XILINX ALVEO PLATFORM ARCHITECTURE

Thanks for refreshing this patchset.

It will take me a while to do the full review, so I thought I would give some early feed back.

It applies to char-misc-next, but will have conflicts with in-flight patches around the MAINTAINERS file. This is not a big deal.

The checkpatch is much better over v2, the complaints are

WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#21:
new file mode 100644

WARNING: From:/Signed-off-by: email address mismatch: 'From: Lizhi Hou <lizhi.hou@xilinx.com>' != 'Signed-off-by: Lizhi Hou <lizhih@xilinx.com>'

MAINTAINERS warning i believe you address in the last patch.

In the next revisions, please fix the signoff.

The test robot is complaining about hppa64.  While it may be an unlikely config, it would be best to fix it.

Tom

>
> Alveo PCIe FPGA based platforms have a static *shell* partition and a partial
> re-configurable *user* partition. The shell partition is automatically loaded from
> flash when host is booted and PCIe is enumerated by BIOS. Shell cannot be changed
> till the next cold reboot. The shell exposes two PCIe physical functions:
>
> 1. management physical function
> 2. user physical function
>
> The patch series includes Documentation/xrt.rst which describes Alveo platform,
> XRT driver architecture and deployment model in more detail.
>
> Users compile their high level design in C/C++/OpenCL or RTL into FPGA image using
> Vitis https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
> tools. The compiled image is packaged as xclbin which contains partial bitstream
> for the user partition and necessary metadata. Users can dynamically swap the image
> running on the user partition in order to switch between different workloads by
> loading different xclbins.
>
> XRT DRIVERS FOR XILINX ALVEO
>
> XRT Linux kernel driver *xmgmt* binds to management physical function of Alveo
> platform. The modular driver framework is organized into several platform drivers
> which primarily handle the following functionality:
>
> 1.  Loading firmware container also called xsabin at driver attach time
> 2.  Loading of user compiled xclbin with FPGA Manager integration
> 3.  Clock scaling of image running on user partition
> 4.  In-band sensors: temp, voltage, power, etc.
> 5.  Device reset and rescan
>
> The platform drivers are packaged into *xrt-lib* helper module with well
> defined interfaces. The module provides a pseudo-bus implementation for the
> platform drivers. More details on the driver model can be found in
> Documentation/xrt.rst.
>
> User physical function driver is not included in this patch series.
>
> LIBFDT REQUIREMENT
>
> XRT driver infrastructure uses Device Tree as a metadata format to discover
> HW subsystems in the Alveo PCIe device. The Device Tree schema used by XRT
> is documented in Documentation/xrt.rst. Unlike previous V1 and V2 version
> of patch series, V3 version does not require export of libfdt symbols.
>
> TESTING AND VALIDATION
>
> xmgmt driver can be tested with full XRT open source stack which includes user
> space libraries, board utilities and (out of tree) first generation user physical
> function driver xocl. XRT open source runtime stack is available at
> https://github.com/Xilinx/XRT
>
> Complete documentation for XRT open source stack including sections on Alveo/XRT
> security and platform architecture can be found here:
>
> https://xilinx.github.io/XRT/master/html/index.html
> https://xilinx.github.io/XRT/master/html/security.html
> https://xilinx.github.io/XRT/master/html/platforms_partitions.html
>
> Changes since v2:
> - Streamlined the driver framework into *xleaf*, *group* and *xroot*
> - Updated documentation to show the driver model with examples
> - Addressed kernel test robot errors
> - Added a selftest for basic driver framework
> - Documented device tree schema
> - Removed need to export libfdt symbols
>
> Changes since v1:
> - Updated the driver to use fpga_region and fpga_bridge for FPGA
>   programming
> - Dropped platform drivers not related to PR programming to focus on XRT
>   core framework
> - Updated Documentation/fpga/xrt.rst with information on XRT core framework
> - Addressed checkpatch issues
> - Dropped xrt- prefix from some header files
>
> For reference V1 version of patch series can be found here:
>
> https://lore.kernel.org/lkml/20201217075046.28553-1-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-2-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-3-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-4-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-5-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-6-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-7-sonals@xilinx.com/
>
> Lizhi Hou (18):
>   Documentation: fpga: Add a document describing XRT Alveo drivers
>   fpga: xrt: driver metadata helper functions
>   fpga: xrt: xclbin file helper functions
>   fpga: xrt: xrt-lib platform driver manager
>   fpga: xrt: group platform driver
>   fpga: xrt: platform driver infrastructure
>   fpga: xrt: management physical function driver (root)
>   fpga: xrt: main platform driver for management function device
>   fpga: xrt: fpga-mgr and region implementation for xclbin download
>   fpga: xrt: VSEC platform driver
>   fpga: xrt: UCS platform driver
>   fpga: xrt: ICAP platform driver
>   fpga: xrt: devctl platform driver
>   fpga: xrt: clock platform driver
>   fpga: xrt: clock frequence counter platform driver
>   fpga: xrt: DDR calibration platform driver
>   fpga: xrt: partition isolation platform driver
>   fpga: xrt: Kconfig and Makefile updates for XRT drivers
>
>  Documentation/fpga/index.rst             |   1 +
>  Documentation/fpga/xrt.rst               | 842 ++++++++++++++++++++++
>  MAINTAINERS                              |  11 +
>  drivers/Makefile                         |   1 +
>  drivers/fpga/Kconfig                     |   2 +
>  drivers/fpga/Makefile                    |   4 +
>  drivers/fpga/xrt/Kconfig                 |   8 +
>  drivers/fpga/xrt/include/events.h        |  48 ++
>  drivers/fpga/xrt/include/group.h         |  27 +
>  drivers/fpga/xrt/include/metadata.h      | 229 ++++++
>  drivers/fpga/xrt/include/subdev_id.h     |  43 ++
>  drivers/fpga/xrt/include/xclbin-helper.h |  52 ++
>  drivers/fpga/xrt/include/xleaf.h         | 276 +++++++
>  drivers/fpga/xrt/include/xleaf/axigate.h |  25 +
>  drivers/fpga/xrt/include/xleaf/calib.h   |  30 +
>  drivers/fpga/xrt/include/xleaf/clkfreq.h |  23 +
>  drivers/fpga/xrt/include/xleaf/clock.h   |  31 +
>  drivers/fpga/xrt/include/xleaf/devctl.h  |  43 ++
>  drivers/fpga/xrt/include/xleaf/icap.h    |  29 +
>  drivers/fpga/xrt/include/xleaf/ucs.h     |  24 +
>  drivers/fpga/xrt/include/xmgmt-main.h    |  37 +
>  drivers/fpga/xrt/include/xroot.h         | 114 +++
>  drivers/fpga/xrt/lib/Kconfig             |  16 +
>  drivers/fpga/xrt/lib/Makefile            |  30 +
>  drivers/fpga/xrt/lib/cdev.c              | 231 ++++++
>  drivers/fpga/xrt/lib/group.c             | 265 +++++++
>  drivers/fpga/xrt/lib/main.c              | 274 +++++++
>  drivers/fpga/xrt/lib/main.h              |  17 +
>  drivers/fpga/xrt/lib/subdev.c            | 871 +++++++++++++++++++++++
>  drivers/fpga/xrt/lib/subdev_pool.h       |  53 ++
>  drivers/fpga/xrt/lib/xclbin.c            | 394 ++++++++++
>  drivers/fpga/xrt/lib/xleaf/axigate.c     | 298 ++++++++
>  drivers/fpga/xrt/lib/xleaf/calib.c       | 226 ++++++
>  drivers/fpga/xrt/lib/xleaf/clkfreq.c     | 221 ++++++
>  drivers/fpga/xrt/lib/xleaf/clock.c       | 648 +++++++++++++++++
>  drivers/fpga/xrt/lib/xleaf/devctl.c      | 206 ++++++
>  drivers/fpga/xrt/lib/xleaf/icap.c        | 317 +++++++++
>  drivers/fpga/xrt/lib/xleaf/ucs.c         | 235 ++++++
>  drivers/fpga/xrt/lib/xleaf/vsec.c        | 359 ++++++++++
>  drivers/fpga/xrt/lib/xroot.c             | 598 ++++++++++++++++
>  drivers/fpga/xrt/metadata/Kconfig        |  12 +
>  drivers/fpga/xrt/metadata/Makefile       |  16 +
>  drivers/fpga/xrt/metadata/metadata.c     | 524 ++++++++++++++
>  drivers/fpga/xrt/mgmt/Kconfig            |  15 +
>  drivers/fpga/xrt/mgmt/Makefile           |  19 +
>  drivers/fpga/xrt/mgmt/fmgr-drv.c         | 187 +++++
>  drivers/fpga/xrt/mgmt/fmgr.h             |  28 +
>  drivers/fpga/xrt/mgmt/main-impl.h        |  37 +
>  drivers/fpga/xrt/mgmt/main-region.c      | 471 ++++++++++++
>  drivers/fpga/xrt/mgmt/main.c             | 693 ++++++++++++++++++
>  drivers/fpga/xrt/mgmt/root.c             | 342 +++++++++
>  include/uapi/linux/xrt/xclbin.h          | 408 +++++++++++
>  include/uapi/linux/xrt/xmgmt-ioctl.h     |  46 ++
>  53 files changed, 9957 insertions(+)
>  create mode 100644 Documentation/fpga/xrt.rst
>  create mode 100644 drivers/fpga/xrt/Kconfig
>  create mode 100644 drivers/fpga/xrt/include/events.h
>  create mode 100644 drivers/fpga/xrt/include/group.h
>  create mode 100644 drivers/fpga/xrt/include/metadata.h
>  create mode 100644 drivers/fpga/xrt/include/subdev_id.h
>  create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf/calib.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf/clkfreq.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf/clock.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf/ucs.h
>  create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
>  create mode 100644 drivers/fpga/xrt/include/xroot.h
>  create mode 100644 drivers/fpga/xrt/lib/Kconfig
>  create mode 100644 drivers/fpga/xrt/lib/Makefile
>  create mode 100644 drivers/fpga/xrt/lib/cdev.c
>  create mode 100644 drivers/fpga/xrt/lib/group.c
>  create mode 100644 drivers/fpga/xrt/lib/main.c
>  create mode 100644 drivers/fpga/xrt/lib/main.h
>  create mode 100644 drivers/fpga/xrt/lib/subdev.c
>  create mode 100644 drivers/fpga/xrt/lib/subdev_pool.h
>  create mode 100644 drivers/fpga/xrt/lib/xclbin.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/calib.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/clkfreq.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/clock.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/devctl.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/ucs.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/vsec.c
>  create mode 100644 drivers/fpga/xrt/lib/xroot.c
>  create mode 100644 drivers/fpga/xrt/metadata/Kconfig
>  create mode 100644 drivers/fpga/xrt/metadata/Makefile
>  create mode 100644 drivers/fpga/xrt/metadata/metadata.c
>  create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
>  create mode 100644 drivers/fpga/xrt/mgmt/Makefile
>  create mode 100644 drivers/fpga/xrt/mgmt/fmgr-drv.c
>  create mode 100644 drivers/fpga/xrt/mgmt/fmgr.h
>  create mode 100644 drivers/fpga/xrt/mgmt/main-impl.h
>  create mode 100644 drivers/fpga/xrt/mgmt/main-region.c
>  create mode 100644 drivers/fpga/xrt/mgmt/main.c
>  create mode 100644 drivers/fpga/xrt/mgmt/root.c
>  create mode 100644 include/uapi/linux/xrt/xclbin.h
>  create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
>
Lizhi Hou Feb. 19, 2021, 5:15 a.m. UTC | #2
On 02/18/2021 05:52 AM, Tom Rix wrote:
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> Hello,
>>
>> This is V3 of patch series which adds management physical function driver for Xilinx
>> Alveo PCIe accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
>> This driver is part of Xilinx Runtime (XRT) open source stack.
>>
>> XILINX ALVEO PLATFORM ARCHITECTURE
> Thanks for refreshing this patchset.
>
> It will take me a while to do the full review, so I thought I would give some early feed back.
>
> It applies to char-misc-next, but will have conflicts with in-flight patches around the MAINTAINERS file. This is not a big deal.
>
> The checkpatch is much better over v2, the complaints are
>
> WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
> #21:
> new file mode 100644
>
> WARNING: From:/Signed-off-by: email address mismatch: 'From: Lizhi Hou <lizhi.hou@xilinx.com>' != 'Signed-off-by: Lizhi Hou <lizhih@xilinx.com>'
>
> MAINTAINERS warning i believe you address in the last patch.
>
> In the next revisions, please fix the signoff.
>
> The test robot is complaining about hppa64.  While it may be an unlikely config, it would be best to fix it.
Thanks for reviewing. I will fix signoff, hppa64 and arm build issue 
reported by robot in next revision.
>
> Tom
>
>> Alveo PCIe FPGA based platforms have a static *shell* partition and a partial
>> re-configurable *user* partition. The shell partition is automatically loaded from
>> flash when host is booted and PCIe is enumerated by BIOS. Shell cannot be changed
>> till the next cold reboot. The shell exposes two PCIe physical functions:
>>
>> 1. management physical function
>> 2. user physical function
>>
>> The patch series includes Documentation/xrt.rst which describes Alveo platform,
>> XRT driver architecture and deployment model in more detail.
>>
>> Users compile their high level design in C/C++/OpenCL or RTL into FPGA image using
>> Vitis https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
>> tools. The compiled image is packaged as xclbin which contains partial bitstream
>> for the user partition and necessary metadata. Users can dynamically swap the image
>> running on the user partition in order to switch between different workloads by
>> loading different xclbins.
>>
>> XRT DRIVERS FOR XILINX ALVEO
>>
>> XRT Linux kernel driver *xmgmt* binds to management physical function of Alveo
>> platform. The modular driver framework is organized into several platform drivers
>> which primarily handle the following functionality:
>>
>> 1.  Loading firmware container also called xsabin at driver attach time
>> 2.  Loading of user compiled xclbin with FPGA Manager integration
>> 3.  Clock scaling of image running on user partition
>> 4.  In-band sensors: temp, voltage, power, etc.
>> 5.  Device reset and rescan
>>
>> The platform drivers are packaged into *xrt-lib* helper module with well
>> defined interfaces. The module provides a pseudo-bus implementation for the
>> platform drivers. More details on the driver model can be found in
>> Documentation/xrt.rst.
>>
>> User physical function driver is not included in this patch series.
>>
>> LIBFDT REQUIREMENT
>>
>> XRT driver infrastructure uses Device Tree as a metadata format to discover
>> HW subsystems in the Alveo PCIe device. The Device Tree schema used by XRT
>> is documented in Documentation/xrt.rst. Unlike previous V1 and V2 version
>> of patch series, V3 version does not require export of libfdt symbols.
>>
>> TESTING AND VALIDATION
>>
>> xmgmt driver can be tested with full XRT open source stack which includes user
>> space libraries, board utilities and (out of tree) first generation user physical
>> function driver xocl. XRT open source runtime stack is available at
>> https://github.com/Xilinx/XRT
>>
>> Complete documentation for XRT open source stack including sections on Alveo/XRT
>> security and platform architecture can be found here:
>>
>> https://xilinx.github.io/XRT/master/html/index.html
>> https://xilinx.github.io/XRT/master/html/security.html
>> https://xilinx.github.io/XRT/master/html/platforms_partitions.html
>>
>> Changes since v2:
>> - Streamlined the driver framework into *xleaf*, *group* and *xroot*
>> - Updated documentation to show the driver model with examples
>> - Addressed kernel test robot errors
>> - Added a selftest for basic driver framework
>> - Documented device tree schema
>> - Removed need to export libfdt symbols
>>
>> Changes since v1:
>> - Updated the driver to use fpga_region and fpga_bridge for FPGA
>>    programming
>> - Dropped platform drivers not related to PR programming to focus on XRT
>>    core framework
>> - Updated Documentation/fpga/xrt.rst with information on XRT core framework
>> - Addressed checkpatch issues
>> - Dropped xrt- prefix from some header files
>>
>> For reference V1 version of patch series can be found here:
>>
>> https://lore.kernel.org/lkml/20201217075046.28553-1-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-2-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-3-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-4-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-5-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-6-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-7-sonals@xilinx.com/
>>
>> Lizhi Hou (18):
>>    Documentation: fpga: Add a document describing XRT Alveo drivers
>>    fpga: xrt: driver metadata helper functions
>>    fpga: xrt: xclbin file helper functions
>>    fpga: xrt: xrt-lib platform driver manager
>>    fpga: xrt: group platform driver
>>    fpga: xrt: platform driver infrastructure
>>    fpga: xrt: management physical function driver (root)
>>    fpga: xrt: main platform driver for management function device
>>    fpga: xrt: fpga-mgr and region implementation for xclbin download
>>    fpga: xrt: VSEC platform driver
>>    fpga: xrt: UCS platform driver
>>    fpga: xrt: ICAP platform driver
>>    fpga: xrt: devctl platform driver
>>    fpga: xrt: clock platform driver
>>    fpga: xrt: clock frequence counter platform driver
>>    fpga: xrt: DDR calibration platform driver
>>    fpga: xrt: partition isolation platform driver
>>    fpga: xrt: Kconfig and Makefile updates for XRT drivers
>>
>>   Documentation/fpga/index.rst             |   1 +
>>   Documentation/fpga/xrt.rst               | 842 ++++++++++++++++++++++
>>   MAINTAINERS                              |  11 +
>>   drivers/Makefile                         |   1 +
>>   drivers/fpga/Kconfig                     |   2 +
>>   drivers/fpga/Makefile                    |   4 +
>>   drivers/fpga/xrt/Kconfig                 |   8 +
>>   drivers/fpga/xrt/include/events.h        |  48 ++
>>   drivers/fpga/xrt/include/group.h         |  27 +
>>   drivers/fpga/xrt/include/metadata.h      | 229 ++++++
>>   drivers/fpga/xrt/include/subdev_id.h     |  43 ++
>>   drivers/fpga/xrt/include/xclbin-helper.h |  52 ++
>>   drivers/fpga/xrt/include/xleaf.h         | 276 +++++++
>>   drivers/fpga/xrt/include/xleaf/axigate.h |  25 +
>>   drivers/fpga/xrt/include/xleaf/calib.h   |  30 +
>>   drivers/fpga/xrt/include/xleaf/clkfreq.h |  23 +
>>   drivers/fpga/xrt/include/xleaf/clock.h   |  31 +
>>   drivers/fpga/xrt/include/xleaf/devctl.h  |  43 ++
>>   drivers/fpga/xrt/include/xleaf/icap.h    |  29 +
>>   drivers/fpga/xrt/include/xleaf/ucs.h     |  24 +
>>   drivers/fpga/xrt/include/xmgmt-main.h    |  37 +
>>   drivers/fpga/xrt/include/xroot.h         | 114 +++
>>   drivers/fpga/xrt/lib/Kconfig             |  16 +
>>   drivers/fpga/xrt/lib/Makefile            |  30 +
>>   drivers/fpga/xrt/lib/cdev.c              | 231 ++++++
>>   drivers/fpga/xrt/lib/group.c             | 265 +++++++
>>   drivers/fpga/xrt/lib/main.c              | 274 +++++++
>>   drivers/fpga/xrt/lib/main.h              |  17 +
>>   drivers/fpga/xrt/lib/subdev.c            | 871 +++++++++++++++++++++++
>>   drivers/fpga/xrt/lib/subdev_pool.h       |  53 ++
>>   drivers/fpga/xrt/lib/xclbin.c            | 394 ++++++++++
>>   drivers/fpga/xrt/lib/xleaf/axigate.c     | 298 ++++++++
>>   drivers/fpga/xrt/lib/xleaf/calib.c       | 226 ++++++
>>   drivers/fpga/xrt/lib/xleaf/clkfreq.c     | 221 ++++++
>>   drivers/fpga/xrt/lib/xleaf/clock.c       | 648 +++++++++++++++++
>>   drivers/fpga/xrt/lib/xleaf/devctl.c      | 206 ++++++
>>   drivers/fpga/xrt/lib/xleaf/icap.c        | 317 +++++++++
>>   drivers/fpga/xrt/lib/xleaf/ucs.c         | 235 ++++++
>>   drivers/fpga/xrt/lib/xleaf/vsec.c        | 359 ++++++++++
>>   drivers/fpga/xrt/lib/xroot.c             | 598 ++++++++++++++++
>>   drivers/fpga/xrt/metadata/Kconfig        |  12 +
>>   drivers/fpga/xrt/metadata/Makefile       |  16 +
>>   drivers/fpga/xrt/metadata/metadata.c     | 524 ++++++++++++++
>>   drivers/fpga/xrt/mgmt/Kconfig            |  15 +
>>   drivers/fpga/xrt/mgmt/Makefile           |  19 +
>>   drivers/fpga/xrt/mgmt/fmgr-drv.c         | 187 +++++
>>   drivers/fpga/xrt/mgmt/fmgr.h             |  28 +
>>   drivers/fpga/xrt/mgmt/main-impl.h        |  37 +
>>   drivers/fpga/xrt/mgmt/main-region.c      | 471 ++++++++++++
>>   drivers/fpga/xrt/mgmt/main.c             | 693 ++++++++++++++++++
>>   drivers/fpga/xrt/mgmt/root.c             | 342 +++++++++
>>   include/uapi/linux/xrt/xclbin.h          | 408 +++++++++++
>>   include/uapi/linux/xrt/xmgmt-ioctl.h     |  46 ++
>>   53 files changed, 9957 insertions(+)
>>   create mode 100644 Documentation/fpga/xrt.rst
>>   create mode 100644 drivers/fpga/xrt/Kconfig
>>   create mode 100644 drivers/fpga/xrt/include/events.h
>>   create mode 100644 drivers/fpga/xrt/include/group.h
>>   create mode 100644 drivers/fpga/xrt/include/metadata.h
>>   create mode 100644 drivers/fpga/xrt/include/subdev_id.h
>>   create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf/calib.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf/clkfreq.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf/clock.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf/ucs.h
>>   create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
>>   create mode 100644 drivers/fpga/xrt/include/xroot.h
>>   create mode 100644 drivers/fpga/xrt/lib/Kconfig
>>   create mode 100644 drivers/fpga/xrt/lib/Makefile
>>   create mode 100644 drivers/fpga/xrt/lib/cdev.c
>>   create mode 100644 drivers/fpga/xrt/lib/group.c
>>   create mode 100644 drivers/fpga/xrt/lib/main.c
>>   create mode 100644 drivers/fpga/xrt/lib/main.h
>>   create mode 100644 drivers/fpga/xrt/lib/subdev.c
>>   create mode 100644 drivers/fpga/xrt/lib/subdev_pool.h
>>   create mode 100644 drivers/fpga/xrt/lib/xclbin.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/calib.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/clkfreq.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/clock.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/devctl.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/ucs.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/vsec.c
>>   create mode 100644 drivers/fpga/xrt/lib/xroot.c
>>   create mode 100644 drivers/fpga/xrt/metadata/Kconfig
>>   create mode 100644 drivers/fpga/xrt/metadata/Makefile
>>   create mode 100644 drivers/fpga/xrt/metadata/metadata.c
>>   create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
>>   create mode 100644 drivers/fpga/xrt/mgmt/Makefile
>>   create mode 100644 drivers/fpga/xrt/mgmt/fmgr-drv.c
>>   create mode 100644 drivers/fpga/xrt/mgmt/fmgr.h
>>   create mode 100644 drivers/fpga/xrt/mgmt/main-impl.h
>>   create mode 100644 drivers/fpga/xrt/mgmt/main-region.c
>>   create mode 100644 drivers/fpga/xrt/mgmt/main.c
>>   create mode 100644 drivers/fpga/xrt/mgmt/root.c
>>   create mode 100644 include/uapi/linux/xrt/xclbin.h
>>   create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
>>
Moritz Fischer Feb. 21, 2021, 8:43 p.m. UTC | #3
Lizhi,

On Wed, Feb 17, 2021 at 10:40:01PM -0800, Lizhi Hou wrote:
> Hello,
> 
> This is V3 of patch series which adds management physical function driver for Xilinx
> Alveo PCIe accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
> This driver is part of Xilinx Runtime (XRT) open source stack.
> 
> XILINX ALVEO PLATFORM ARCHITECTURE
> 
> Alveo PCIe FPGA based platforms have a static *shell* partition and a partial
> re-configurable *user* partition. The shell partition is automatically loaded from
> flash when host is booted and PCIe is enumerated by BIOS. Shell cannot be changed
> till the next cold reboot. The shell exposes two PCIe physical functions:
> 
> 1. management physical function
> 2. user physical function
> 
> The patch series includes Documentation/xrt.rst which describes Alveo platform,
> XRT driver architecture and deployment model in more detail.
> 
> Users compile their high level design in C/C++/OpenCL or RTL into FPGA image using
> Vitis https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
> tools. The compiled image is packaged as xclbin which contains partial bitstream
> for the user partition and necessary metadata. Users can dynamically swap the image
> running on the user partition in order to switch between different workloads by
> loading different xclbins.
> 
> XRT DRIVERS FOR XILINX ALVEO
> 
> XRT Linux kernel driver *xmgmt* binds to management physical function of Alveo
> platform. The modular driver framework is organized into several platform drivers
> which primarily handle the following functionality:
> 
> 1.  Loading firmware container also called xsabin at driver attach time
> 2.  Loading of user compiled xclbin with FPGA Manager integration
> 3.  Clock scaling of image running on user partition
> 4.  In-band sensors: temp, voltage, power, etc.
> 5.  Device reset and rescan
> 
> The platform drivers are packaged into *xrt-lib* helper module with well
> defined interfaces. The module provides a pseudo-bus implementation for the
> platform drivers. More details on the driver model can be found in
> Documentation/xrt.rst.
> 
> User physical function driver is not included in this patch series.
> 
> LIBFDT REQUIREMENT
> 
> XRT driver infrastructure uses Device Tree as a metadata format to discover
> HW subsystems in the Alveo PCIe device. The Device Tree schema used by XRT
> is documented in Documentation/xrt.rst. Unlike previous V1 and V2 version
> of patch series, V3 version does not require export of libfdt symbols.
> 
> TESTING AND VALIDATION
> 
> xmgmt driver can be tested with full XRT open source stack which includes user
> space libraries, board utilities and (out of tree) first generation user physical
> function driver xocl. XRT open source runtime stack is available at
> https://github.com/Xilinx/XRT
> 
> Complete documentation for XRT open source stack including sections on Alveo/XRT
> security and platform architecture can be found here:
> 
> https://xilinx.github.io/XRT/master/html/index.html
> https://xilinx.github.io/XRT/master/html/security.html
> https://xilinx.github.io/XRT/master/html/platforms_partitions.html
> 
> Changes since v2:
> - Streamlined the driver framework into *xleaf*, *group* and *xroot*
> - Updated documentation to show the driver model with examples
> - Addressed kernel test robot errors
> - Added a selftest for basic driver framework
> - Documented device tree schema
> - Removed need to export libfdt symbols
> 
> Changes since v1:
> - Updated the driver to use fpga_region and fpga_bridge for FPGA
>   programming
> - Dropped platform drivers not related to PR programming to focus on XRT
>   core framework
> - Updated Documentation/fpga/xrt.rst with information on XRT core framework
> - Addressed checkpatch issues
> - Dropped xrt- prefix from some header files
> 
> For reference V1 version of patch series can be found here:
> 
> https://lore.kernel.org/lkml/20201217075046.28553-1-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-2-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-3-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-4-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-5-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-6-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-7-sonals@xilinx.com/
> 
> Lizhi Hou (18):
>   Documentation: fpga: Add a document describing XRT Alveo drivers
>   fpga: xrt: driver metadata helper functions
>   fpga: xrt: xclbin file helper functions
>   fpga: xrt: xrt-lib platform driver manager
>   fpga: xrt: group platform driver
>   fpga: xrt: platform driver infrastructure
>   fpga: xrt: management physical function driver (root)
>   fpga: xrt: main platform driver for management function device
>   fpga: xrt: fpga-mgr and region implementation for xclbin download
>   fpga: xrt: VSEC platform driver
>   fpga: xrt: UCS platform driver
>   fpga: xrt: ICAP platform driver
>   fpga: xrt: devctl platform driver
>   fpga: xrt: clock platform driver
>   fpga: xrt: clock frequence counter platform driver
>   fpga: xrt: DDR calibration platform driver
>   fpga: xrt: partition isolation platform driver
>   fpga: xrt: Kconfig and Makefile updates for XRT drivers
> 
>  Documentation/fpga/index.rst             |   1 +
>  Documentation/fpga/xrt.rst               | 842 ++++++++++++++++++++++
>  MAINTAINERS                              |  11 +
>  drivers/Makefile                         |   1 +
>  drivers/fpga/Kconfig                     |   2 +
>  drivers/fpga/Makefile                    |   4 +
>  drivers/fpga/xrt/Kconfig                 |   8 +
>  drivers/fpga/xrt/include/events.h        |  48 ++
>  drivers/fpga/xrt/include/group.h         |  27 +
>  drivers/fpga/xrt/include/metadata.h      | 229 ++++++
>  drivers/fpga/xrt/include/subdev_id.h     |  43 ++
>  drivers/fpga/xrt/include/xclbin-helper.h |  52 ++
>  drivers/fpga/xrt/include/xleaf.h         | 276 +++++++
>  drivers/fpga/xrt/include/xleaf/axigate.h |  25 +
>  drivers/fpga/xrt/include/xleaf/calib.h   |  30 +
>  drivers/fpga/xrt/include/xleaf/clkfreq.h |  23 +
>  drivers/fpga/xrt/include/xleaf/clock.h   |  31 +
>  drivers/fpga/xrt/include/xleaf/devctl.h  |  43 ++
>  drivers/fpga/xrt/include/xleaf/icap.h    |  29 +
>  drivers/fpga/xrt/include/xleaf/ucs.h     |  24 +
>  drivers/fpga/xrt/include/xmgmt-main.h    |  37 +
>  drivers/fpga/xrt/include/xroot.h         | 114 +++
>  drivers/fpga/xrt/lib/Kconfig             |  16 +
>  drivers/fpga/xrt/lib/Makefile            |  30 +
>  drivers/fpga/xrt/lib/cdev.c              | 231 ++++++
>  drivers/fpga/xrt/lib/group.c             | 265 +++++++
>  drivers/fpga/xrt/lib/main.c              | 274 +++++++
>  drivers/fpga/xrt/lib/main.h              |  17 +
>  drivers/fpga/xrt/lib/subdev.c            | 871 +++++++++++++++++++++++
>  drivers/fpga/xrt/lib/subdev_pool.h       |  53 ++
>  drivers/fpga/xrt/lib/xclbin.c            | 394 ++++++++++
>  drivers/fpga/xrt/lib/xleaf/axigate.c     | 298 ++++++++
>  drivers/fpga/xrt/lib/xleaf/calib.c       | 226 ++++++
>  drivers/fpga/xrt/lib/xleaf/clkfreq.c     | 221 ++++++
>  drivers/fpga/xrt/lib/xleaf/clock.c       | 648 +++++++++++++++++
>  drivers/fpga/xrt/lib/xleaf/devctl.c      | 206 ++++++
>  drivers/fpga/xrt/lib/xleaf/icap.c        | 317 +++++++++
>  drivers/fpga/xrt/lib/xleaf/ucs.c         | 235 ++++++
>  drivers/fpga/xrt/lib/xleaf/vsec.c        | 359 ++++++++++
>  drivers/fpga/xrt/lib/xroot.c             | 598 ++++++++++++++++
>  drivers/fpga/xrt/metadata/Kconfig        |  12 +
>  drivers/fpga/xrt/metadata/Makefile       |  16 +
>  drivers/fpga/xrt/metadata/metadata.c     | 524 ++++++++++++++
>  drivers/fpga/xrt/mgmt/Kconfig            |  15 +
>  drivers/fpga/xrt/mgmt/Makefile           |  19 +
>  drivers/fpga/xrt/mgmt/fmgr-drv.c         | 187 +++++
>  drivers/fpga/xrt/mgmt/fmgr.h             |  28 +
>  drivers/fpga/xrt/mgmt/main-impl.h        |  37 +
>  drivers/fpga/xrt/mgmt/main-region.c      | 471 ++++++++++++
>  drivers/fpga/xrt/mgmt/main.c             | 693 ++++++++++++++++++
>  drivers/fpga/xrt/mgmt/root.c             | 342 +++++++++
>  include/uapi/linux/xrt/xclbin.h          | 408 +++++++++++
>  include/uapi/linux/xrt/xmgmt-ioctl.h     |  46 ++
>  53 files changed, 9957 insertions(+)
>  create mode 100644 Documentation/fpga/xrt.rst
>  create mode 100644 drivers/fpga/xrt/Kconfig
>  create mode 100644 drivers/fpga/xrt/include/events.h
>  create mode 100644 drivers/fpga/xrt/include/group.h
>  create mode 100644 drivers/fpga/xrt/include/metadata.h
>  create mode 100644 drivers/fpga/xrt/include/subdev_id.h
>  create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf/calib.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf/clkfreq.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf/clock.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
>  create mode 100644 drivers/fpga/xrt/include/xleaf/ucs.h
>  create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
>  create mode 100644 drivers/fpga/xrt/include/xroot.h
>  create mode 100644 drivers/fpga/xrt/lib/Kconfig
>  create mode 100644 drivers/fpga/xrt/lib/Makefile
>  create mode 100644 drivers/fpga/xrt/lib/cdev.c
>  create mode 100644 drivers/fpga/xrt/lib/group.c
>  create mode 100644 drivers/fpga/xrt/lib/main.c
>  create mode 100644 drivers/fpga/xrt/lib/main.h
>  create mode 100644 drivers/fpga/xrt/lib/subdev.c
>  create mode 100644 drivers/fpga/xrt/lib/subdev_pool.h
>  create mode 100644 drivers/fpga/xrt/lib/xclbin.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/calib.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/clkfreq.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/clock.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/devctl.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/ucs.c
>  create mode 100644 drivers/fpga/xrt/lib/xleaf/vsec.c
>  create mode 100644 drivers/fpga/xrt/lib/xroot.c
>  create mode 100644 drivers/fpga/xrt/metadata/Kconfig
>  create mode 100644 drivers/fpga/xrt/metadata/Makefile
>  create mode 100644 drivers/fpga/xrt/metadata/metadata.c
>  create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
>  create mode 100644 drivers/fpga/xrt/mgmt/Makefile
>  create mode 100644 drivers/fpga/xrt/mgmt/fmgr-drv.c
>  create mode 100644 drivers/fpga/xrt/mgmt/fmgr.h
>  create mode 100644 drivers/fpga/xrt/mgmt/main-impl.h
>  create mode 100644 drivers/fpga/xrt/mgmt/main-region.c
>  create mode 100644 drivers/fpga/xrt/mgmt/main.c
>  create mode 100644 drivers/fpga/xrt/mgmt/root.c
>  create mode 100644 include/uapi/linux/xrt/xclbin.h
>  create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
> 
> -- 
> 2.18.4
> 

Please fix the indents all across this patchset. Doesn't checkpatch with
--strict complain about this?

Also more generally this looks like it should be a bus. Look at DFL for
reference.

- Moritz
Lizhi Hou March 1, 2021, 6:29 p.m. UTC | #4
Hi Moritz,


On 02/21/2021 12:43 PM, Moritz Fischer wrote:
> Lizhi,
>
> On Wed, Feb 17, 2021 at 10:40:01PM -0800, Lizhi Hou wrote:
>> Hello,
>>
>> This is V3 of patch series which adds management physical function driver for Xilinx
>> Alveo PCIe accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
>> This driver is part of Xilinx Runtime (XRT) open source stack.
>>
>> XILINX ALVEO PLATFORM ARCHITECTURE
>>
>> Alveo PCIe FPGA based platforms have a static *shell* partition and a partial
>> re-configurable *user* partition. The shell partition is automatically loaded from
>> flash when host is booted and PCIe is enumerated by BIOS. Shell cannot be changed
>> till the next cold reboot. The shell exposes two PCIe physical functions:
>>
>> 1. management physical function
>> 2. user physical function
>>
>> The patch series includes Documentation/xrt.rst which describes Alveo platform,
>> XRT driver architecture and deployment model in more detail.
>>
>> Users compile their high level design in C/C++/OpenCL or RTL into FPGA image using
>> Vitis https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
>> tools. The compiled image is packaged as xclbin which contains partial bitstream
>> for the user partition and necessary metadata. Users can dynamically swap the image
>> running on the user partition in order to switch between different workloads by
>> loading different xclbins.
>>
>> XRT DRIVERS FOR XILINX ALVEO
>>
>> XRT Linux kernel driver *xmgmt* binds to management physical function of Alveo
>> platform. The modular driver framework is organized into several platform drivers
>> which primarily handle the following functionality:
>>
>> 1.  Loading firmware container also called xsabin at driver attach time
>> 2.  Loading of user compiled xclbin with FPGA Manager integration
>> 3.  Clock scaling of image running on user partition
>> 4.  In-band sensors: temp, voltage, power, etc.
>> 5.  Device reset and rescan
>>
>> The platform drivers are packaged into *xrt-lib* helper module with well
>> defined interfaces. The module provides a pseudo-bus implementation for the
>> platform drivers. More details on the driver model can be found in
>> Documentation/xrt.rst.
>>
>> User physical function driver is not included in this patch series.
>>
>> LIBFDT REQUIREMENT
>>
>> XRT driver infrastructure uses Device Tree as a metadata format to discover
>> HW subsystems in the Alveo PCIe device. The Device Tree schema used by XRT
>> is documented in Documentation/xrt.rst. Unlike previous V1 and V2 version
>> of patch series, V3 version does not require export of libfdt symbols.
>>
>> TESTING AND VALIDATION
>>
>> xmgmt driver can be tested with full XRT open source stack which includes user
>> space libraries, board utilities and (out of tree) first generation user physical
>> function driver xocl. XRT open source runtime stack is available at
>> https://github.com/Xilinx/XRT
>>
>> Complete documentation for XRT open source stack including sections on Alveo/XRT
>> security and platform architecture can be found here:
>>
>> https://xilinx.github.io/XRT/master/html/index.html
>> https://xilinx.github.io/XRT/master/html/security.html
>> https://xilinx.github.io/XRT/master/html/platforms_partitions.html
>>
>> Changes since v2:
>> - Streamlined the driver framework into *xleaf*, *group* and *xroot*
>> - Updated documentation to show the driver model with examples
>> - Addressed kernel test robot errors
>> - Added a selftest for basic driver framework
>> - Documented device tree schema
>> - Removed need to export libfdt symbols
>>
>> Changes since v1:
>> - Updated the driver to use fpga_region and fpga_bridge for FPGA
>>    programming
>> - Dropped platform drivers not related to PR programming to focus on XRT
>>    core framework
>> - Updated Documentation/fpga/xrt.rst with information on XRT core framework
>> - Addressed checkpatch issues
>> - Dropped xrt- prefix from some header files
>>
>> For reference V1 version of patch series can be found here:
>>
>> https://lore.kernel.org/lkml/20201217075046.28553-1-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-2-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-3-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-4-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-5-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-6-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-7-sonals@xilinx.com/
>>
>> Lizhi Hou (18):
>>    Documentation: fpga: Add a document describing XRT Alveo drivers
>>    fpga: xrt: driver metadata helper functions
>>    fpga: xrt: xclbin file helper functions
>>    fpga: xrt: xrt-lib platform driver manager
>>    fpga: xrt: group platform driver
>>    fpga: xrt: platform driver infrastructure
>>    fpga: xrt: management physical function driver (root)
>>    fpga: xrt: main platform driver for management function device
>>    fpga: xrt: fpga-mgr and region implementation for xclbin download
>>    fpga: xrt: VSEC platform driver
>>    fpga: xrt: UCS platform driver
>>    fpga: xrt: ICAP platform driver
>>    fpga: xrt: devctl platform driver
>>    fpga: xrt: clock platform driver
>>    fpga: xrt: clock frequence counter platform driver
>>    fpga: xrt: DDR calibration platform driver
>>    fpga: xrt: partition isolation platform driver
>>    fpga: xrt: Kconfig and Makefile updates for XRT drivers
>>
>>   Documentation/fpga/index.rst             |   1 +
>>   Documentation/fpga/xrt.rst               | 842 ++++++++++++++++++++++
>>   MAINTAINERS                              |  11 +
>>   drivers/Makefile                         |   1 +
>>   drivers/fpga/Kconfig                     |   2 +
>>   drivers/fpga/Makefile                    |   4 +
>>   drivers/fpga/xrt/Kconfig                 |   8 +
>>   drivers/fpga/xrt/include/events.h        |  48 ++
>>   drivers/fpga/xrt/include/group.h         |  27 +
>>   drivers/fpga/xrt/include/metadata.h      | 229 ++++++
>>   drivers/fpga/xrt/include/subdev_id.h     |  43 ++
>>   drivers/fpga/xrt/include/xclbin-helper.h |  52 ++
>>   drivers/fpga/xrt/include/xleaf.h         | 276 +++++++
>>   drivers/fpga/xrt/include/xleaf/axigate.h |  25 +
>>   drivers/fpga/xrt/include/xleaf/calib.h   |  30 +
>>   drivers/fpga/xrt/include/xleaf/clkfreq.h |  23 +
>>   drivers/fpga/xrt/include/xleaf/clock.h   |  31 +
>>   drivers/fpga/xrt/include/xleaf/devctl.h  |  43 ++
>>   drivers/fpga/xrt/include/xleaf/icap.h    |  29 +
>>   drivers/fpga/xrt/include/xleaf/ucs.h     |  24 +
>>   drivers/fpga/xrt/include/xmgmt-main.h    |  37 +
>>   drivers/fpga/xrt/include/xroot.h         | 114 +++
>>   drivers/fpga/xrt/lib/Kconfig             |  16 +
>>   drivers/fpga/xrt/lib/Makefile            |  30 +
>>   drivers/fpga/xrt/lib/cdev.c              | 231 ++++++
>>   drivers/fpga/xrt/lib/group.c             | 265 +++++++
>>   drivers/fpga/xrt/lib/main.c              | 274 +++++++
>>   drivers/fpga/xrt/lib/main.h              |  17 +
>>   drivers/fpga/xrt/lib/subdev.c            | 871 +++++++++++++++++++++++
>>   drivers/fpga/xrt/lib/subdev_pool.h       |  53 ++
>>   drivers/fpga/xrt/lib/xclbin.c            | 394 ++++++++++
>>   drivers/fpga/xrt/lib/xleaf/axigate.c     | 298 ++++++++
>>   drivers/fpga/xrt/lib/xleaf/calib.c       | 226 ++++++
>>   drivers/fpga/xrt/lib/xleaf/clkfreq.c     | 221 ++++++
>>   drivers/fpga/xrt/lib/xleaf/clock.c       | 648 +++++++++++++++++
>>   drivers/fpga/xrt/lib/xleaf/devctl.c      | 206 ++++++
>>   drivers/fpga/xrt/lib/xleaf/icap.c        | 317 +++++++++
>>   drivers/fpga/xrt/lib/xleaf/ucs.c         | 235 ++++++
>>   drivers/fpga/xrt/lib/xleaf/vsec.c        | 359 ++++++++++
>>   drivers/fpga/xrt/lib/xroot.c             | 598 ++++++++++++++++
>>   drivers/fpga/xrt/metadata/Kconfig        |  12 +
>>   drivers/fpga/xrt/metadata/Makefile       |  16 +
>>   drivers/fpga/xrt/metadata/metadata.c     | 524 ++++++++++++++
>>   drivers/fpga/xrt/mgmt/Kconfig            |  15 +
>>   drivers/fpga/xrt/mgmt/Makefile           |  19 +
>>   drivers/fpga/xrt/mgmt/fmgr-drv.c         | 187 +++++
>>   drivers/fpga/xrt/mgmt/fmgr.h             |  28 +
>>   drivers/fpga/xrt/mgmt/main-impl.h        |  37 +
>>   drivers/fpga/xrt/mgmt/main-region.c      | 471 ++++++++++++
>>   drivers/fpga/xrt/mgmt/main.c             | 693 ++++++++++++++++++
>>   drivers/fpga/xrt/mgmt/root.c             | 342 +++++++++
>>   include/uapi/linux/xrt/xclbin.h          | 408 +++++++++++
>>   include/uapi/linux/xrt/xmgmt-ioctl.h     |  46 ++
>>   53 files changed, 9957 insertions(+)
>>   create mode 100644 Documentation/fpga/xrt.rst
>>   create mode 100644 drivers/fpga/xrt/Kconfig
>>   create mode 100644 drivers/fpga/xrt/include/events.h
>>   create mode 100644 drivers/fpga/xrt/include/group.h
>>   create mode 100644 drivers/fpga/xrt/include/metadata.h
>>   create mode 100644 drivers/fpga/xrt/include/subdev_id.h
>>   create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf/calib.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf/clkfreq.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf/clock.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
>>   create mode 100644 drivers/fpga/xrt/include/xleaf/ucs.h
>>   create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
>>   create mode 100644 drivers/fpga/xrt/include/xroot.h
>>   create mode 100644 drivers/fpga/xrt/lib/Kconfig
>>   create mode 100644 drivers/fpga/xrt/lib/Makefile
>>   create mode 100644 drivers/fpga/xrt/lib/cdev.c
>>   create mode 100644 drivers/fpga/xrt/lib/group.c
>>   create mode 100644 drivers/fpga/xrt/lib/main.c
>>   create mode 100644 drivers/fpga/xrt/lib/main.h
>>   create mode 100644 drivers/fpga/xrt/lib/subdev.c
>>   create mode 100644 drivers/fpga/xrt/lib/subdev_pool.h
>>   create mode 100644 drivers/fpga/xrt/lib/xclbin.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/calib.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/clkfreq.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/clock.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/devctl.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/ucs.c
>>   create mode 100644 drivers/fpga/xrt/lib/xleaf/vsec.c
>>   create mode 100644 drivers/fpga/xrt/lib/xroot.c
>>   create mode 100644 drivers/fpga/xrt/metadata/Kconfig
>>   create mode 100644 drivers/fpga/xrt/metadata/Makefile
>>   create mode 100644 drivers/fpga/xrt/metadata/metadata.c
>>   create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
>>   create mode 100644 drivers/fpga/xrt/mgmt/Makefile
>>   create mode 100644 drivers/fpga/xrt/mgmt/fmgr-drv.c
>>   create mode 100644 drivers/fpga/xrt/mgmt/fmgr.h
>>   create mode 100644 drivers/fpga/xrt/mgmt/main-impl.h
>>   create mode 100644 drivers/fpga/xrt/mgmt/main-region.c
>>   create mode 100644 drivers/fpga/xrt/mgmt/main.c
>>   create mode 100644 drivers/fpga/xrt/mgmt/root.c
>>   create mode 100644 include/uapi/linux/xrt/xclbin.h
>>   create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
>>
>> --
>> 2.18.4
>>
> Please fix the indents all across this patchset. Doesn't checkpatch with
> --strict complain about this?
checkpatch --strict did not complain. And we will fix this.
>
> Also more generally this looks like it should be a bus. Look at DFL for
> reference.
Tom asked the same question in 01/18 and Sonal replied. We may discuss 
with that thread.

Thanks,
Lizhi
>
> - Moritz
Joe Perches March 3, 2021, 6:49 a.m. UTC | #5
On Sun, 2021-02-21 at 12:43 -0800, Moritz Fischer wrote:
> On Wed, Feb 17, 2021 at 10:40:01PM -0800, Lizhi Hou wrote:
> > This is V3 of patch series which adds management physical function driver for Xilinx
> > Alveo PCIe accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
> > This driver is part of Xilinx Runtime (XRT) open source stack.
[]
> Please fix the indents all across this patchset. Doesn't checkpatch with
> --strict complain about this?

I glanced at a couple bits of these patches and didn't
notice any of what I consider poor indentation style.

What indent is wrong here?
Moritz Fischer March 3, 2021, 11:15 p.m. UTC | #6
On Tue, Mar 02, 2021 at 10:49:43PM -0800, Joe Perches wrote:
> On Sun, 2021-02-21 at 12:43 -0800, Moritz Fischer wrote:
> > On Wed, Feb 17, 2021 at 10:40:01PM -0800, Lizhi Hou wrote:
> > > This is V3 of patch series which adds management physical function driver for Xilinx
> > > Alveo PCIe accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
> > > This driver is part of Xilinx Runtime (XRT) open source stack.
> []
> > Please fix the indents all across this patchset. Doesn't checkpatch with
> > --strict complain about this?
> 
> I glanced at a couple bits of these patches and didn't
> notice any of what I consider poor indentation style.
> 
> What indent is wrong here?

Maybe I dreamed it, or confused it with the CamelCase issues instead?

Sorry for the noise in that case,

- Moritz