[RFC,00/24] drm/msm: sparse / "VM_BIND" support

Message

Rob Clark Dec. 7, 2024, 4:15 p.m. UTC

From: Rob Clark <[e-mail]>

Conversion to DRM GPU VA Manager[1], and adding support for Vulkan Sparse
Memory[2] in the form of:
1. A new VM_BIND submitqueue type for executing VM MSM_SUBMIT_BO_OP_MAP/
   MAP_NULL/UNMAP commands
2. Extending the SUBMIT` ioctl to allow submitting batches of one or more
   MAP/MAP_NULL/UNMAP commands to a VM_BIND submitqueue

The UABI takes a slightly different approach from what other drivers have
done, and what would make sense if starting from a clean sheet, ie separate
VM_BIND and EXEC ioctls.  But since we have to maintain support for the
existing SUBMIT ioctl, and because the fence, syncobj, and BO pinning is
largely the same between legacy "BO-table" style SUBMIT ioctls, and new-
style VM updates submitted to a VM_BIND submitqueue, I chose to go the
route of extending the existing `SUBMIT` ioctl rather than adding a new
ioctl.

I also did not implement support for synchronous VM_BIND commands.  Since
userspace could just immediately wait for the `SUBMIT` to complete, I don't
think we need this extra complexity in the kernel.

The corresponding mesa MR: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32533

### Notes/TODOs/Open Questions:
1. The first handful of patches are from Bibek Kumar Patro's series, 
   "iommu/arm-smmu: introduction of ACTLR implementation for Qualcomm SoCs[3],
   which introduces PRR (Partially-Resident-Region) support, needed to
   implement MAP_NULL (for Vulkan Sparse Residency[4]
2. Why do VM_BIND commands need fence fd support, instead of just syncobjs?
   Mainly for the benefit of virtgpu drm native context guest<->host fence
   passing[5], where the host VMM is operating in terms of fence fd's
   (syncobs are just a convenience wrapper above a dma_fence, and don't
   exist below the guest kernel).
3. Currently shrinker support is disabled (hence this being in Draft/RFC
   state).  To properly support the shrinker, we need to pre-allocate
   various objects and pages needed for the pagetables themselves, to
   move memory allocations out of the fence signaling path.  This short-
   cut was taken to unblock userspace implementation of sparse buffer/
   image support.
4. Could/should we do all the vm/vma updates synchronously and defer _only_
   the io-pgtable updates to the VM_BIND scheduler queue?  This would
   simplify the previous point, in that we'd only have to pre-allocate
   pages for the io-pgtable updates.
5. Currently we lose support for BO dumping for devcoredump.  Ideally we'd
   plumb `MSM_SUBMIT_BO_DUMP` flag in a `MAP` commands thru to the resulting
   drm_gpuva's.  To do this, I think we need to extend drm_gpuva with a
   flags field.. the flags can be driver defined, but drm_gpuvm needs to
   know not to merge drm_gpuva's with different flags.

This series can be found in MR form, if you prefer:
https://gitlab.freedesktop.org/drm/msm/-/merge_requests/144

[1] https://www.kernel.org/doc/html/next/gpu/drm-mm.html#drm-gpuvm
[2] https://docs.vulkan.org/spec/latest/chapters/sparsemem.html
[3] https://patchwork.kernel.org/project/linux-arm-kernel/list/?series=909700
[4] https://docs.vulkan.org/spec/latest/chapters/sparsemem.html#sparsememory-partially-resident-buffers
[5] https://patchew.org/linux/[e-mail]/

Rob Clark (24):
  HACK: drm/msm: Disable shrinker
  drm/gpuvm: Don't require obj lock in destructor path
  drm/gpuvm: Remove bogus lock assert
  drm/msm: Rename msm_file_private -> msm_context
  drm/msm: Improve msm_context comments
  drm/msm: Rename msm_gem_address_space -> msm_gem_vm
  drm/msm: Remove vram carveout support
  drm/msm: Collapse vma allocation and initialization
  drm/msm: Collapse vma close and delete
  drm/msm: drm_gpuvm conversion
  drm/msm: Use drm_gpuvm types more
  drm/msm: Split submit_pin_objects()
  drm/msm: Lazily create context VM
  drm/msm: Add opt-in for VM_BIND
  drm/msm: Mark VM as unusable on faults
  drm/msm: Extend SUBMIT ioctl for VM_BIND
  drm/msm: Add VM_BIND submitqueue
  drm/msm: Add _NO_SHARE flag
  drm/msm: Split out helper to get iommu prot flags
  drm/msm: Add mmu support for non-zero offset
  drm/msm: Add PRR support
  drm/msm: Rename msm_gem_vma_purge() -> _unmap()
  drm/msm: Wire up gpuvm ops
  drm/msm: Bump UAPI version

 drivers/gpu/drm/drm_gpuvm.c                   |  10 +-
 drivers/gpu/drm/msm/Kconfig                   |   1 +
 drivers/gpu/drm/msm/adreno/a2xx_gpu.c         |  19 +-
 drivers/gpu/drm/msm/adreno/a2xx_gpummu.c      |   5 +-
 drivers/gpu/drm/msm/adreno/a3xx_gpu.c         |   4 +-
 drivers/gpu/drm/msm/adreno/a4xx_gpu.c         |   4 +-
 drivers/gpu/drm/msm/adreno/a5xx_debugfs.c     |   4 +-
 drivers/gpu/drm/msm/adreno/a5xx_gpu.c         |  24 +-
 drivers/gpu/drm/msm/adreno/a5xx_power.c       |   2 +-
 drivers/gpu/drm/msm/adreno/a5xx_preempt.c     |  10 +-
 drivers/gpu/drm/msm/adreno/a6xx_gmu.c         |  32 +-
 drivers/gpu/drm/msm/adreno/a6xx_gmu.h         |   2 +-
 drivers/gpu/drm/msm/adreno/a6xx_gpu.c         |  51 +-
 drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c   |   6 +-
 drivers/gpu/drm/msm/adreno/a6xx_preempt.c     |  10 +-
 drivers/gpu/drm/msm/adreno/adreno_gpu.c       |  78 ++-
 drivers/gpu/drm/msm/adreno/adreno_gpu.h       |  22 +-
 .../drm/msm/disp/dpu1/dpu_encoder_phys_wb.c   |  14 +-
 drivers/gpu/drm/msm/disp/dpu1/dpu_formats.c   |  18 +-
 drivers/gpu/drm/msm/disp/dpu1/dpu_formats.h   |   2 +-
 drivers/gpu/drm/msm/disp/dpu1/dpu_kms.c       |  18 +-
 drivers/gpu/drm/msm/disp/dpu1/dpu_plane.c     |  14 +-
 drivers/gpu/drm/msm/disp/dpu1/dpu_plane.h     |   4 +-
 drivers/gpu/drm/msm/disp/mdp4/mdp4_crtc.c     |   6 +-
 drivers/gpu/drm/msm/disp/mdp4/mdp4_kms.c      |  28 +-
 drivers/gpu/drm/msm/disp/mdp4/mdp4_plane.c    |  12 +-
 drivers/gpu/drm/msm/disp/mdp5/mdp5_crtc.c     |   4 +-
 drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c      |  19 +-
 drivers/gpu/drm/msm/disp/mdp5/mdp5_plane.c    |  12 +-
 drivers/gpu/drm/msm/dsi/dsi_host.c            |  14 +-
 drivers/gpu/drm/msm/msm_drv.c                 | 175 ++----
 drivers/gpu/drm/msm/msm_drv.h                 |  31 +-
 drivers/gpu/drm/msm/msm_fb.c                  |  18 +-
 drivers/gpu/drm/msm/msm_fbdev.c               |   2 +-
 drivers/gpu/drm/msm/msm_gem.c                 | 403 ++++++-------
 drivers/gpu/drm/msm/msm_gem.h                 | 193 +++++--
 drivers/gpu/drm/msm/msm_gem_prime.c           |  15 +
 drivers/gpu/drm/msm/msm_gem_submit.c          | 223 +++++--
 drivers/gpu/drm/msm/msm_gem_vma.c             | 543 +++++++++++++++---
 drivers/gpu/drm/msm/msm_gpu.c                 |  66 ++-
 drivers/gpu/drm/msm/msm_gpu.h                 | 132 +++--
 drivers/gpu/drm/msm/msm_iommu.c               |  84 ++-
 drivers/gpu/drm/msm/msm_kms.c                 |  14 +-
 drivers/gpu/drm/msm/msm_kms.h                 |   2 +-
 drivers/gpu/drm/msm/msm_mmu.h                 |   2 +-
 drivers/gpu/drm/msm/msm_ringbuffer.c          |   4 +-
 drivers/gpu/drm/msm/msm_submitqueue.c         |  86 ++-
 include/uapi/drm/msm_drm.h                    |  98 +++-
 48 files changed, 1637 insertions(+), 903 deletions(-)