[RFC,0/2,RFC] Honor PCI prefetchable attributes for a virtual machine on ARM64

Message

Shanker Donthineni April 29, 2021, 4:29 p.m. UTC

Problem statement: Virtual machine crashes when NVIDIA GPU driver access a prefetchable BAR space due to the unaligned reads/writes for pass-through devices. The same binary works fine as expected in the host kernel. Only one BAR has control & status registers (CSR) and other PCI BARs are marked as prefetchable. NVIDIA GPU driver uses the write-combine feature for mapping the prefetchable BARs to improve performance. This problem applies to all other drivers which want to enable WC.
 
Solution: Honor PCI prefetchable attributes for the guest operating systems.
 
Proposal: ARM64-KVM uses VMA struct for the needed information e.g. region physical address, size, and memory-type (struct page backed mapping or anonymous memory) for setting up a stage-2 page table. Right now memory region either can be mapped as DEVICE (strongly ordered) or NORMAL (write-back cache) depends on the flag VM_PFNMAP in VMA. VFIO-PCI will keep the prefetchable (write-combine) information in vma->vm_page_prot similar to other fields, and KVM will prepare stage-2 entries based on the memory-type attribute that was set in VMA.

Shanker Donthineni (2):
  vfio/pci: keep the prefetchable attribute of a BAR region in VMA
  KVM: arm64: Add write-combine support for stage-2 entries

 arch/arm64/include/asm/kvm_mmu.h     |  3 ++-
 arch/arm64/include/asm/kvm_pgtable.h |  2 ++
 arch/arm64/include/asm/memory.h      |  4 +++-
 arch/arm64/kvm/hyp/pgtable.c         |  9 +++++++--
 arch/arm64/kvm/mmu.c                 | 22 +++++++++++++++++++---
 arch/arm64/kvm/vgic/vgic-v2.c        |  2 +-
 drivers/vfio/pci/vfio_pci.c          |  6 +++++-
 7 files changed, 39 insertions(+), 9 deletions(-)

Comments

Christoph Hellwig May 3, 2021, 7:01 a.m. UTC | #1

On Thu, Apr 29, 2021 at 11:29:04AM -0500, Shanker Donthineni wrote:
> Problem statement: Virtual machine crashes when NVIDIA GPU driver access a prefetchable BAR space due to the unaligned reads/writes for pass-through devices. The same binary works fine as expected in the host kernel. Only one BAR has control & status registers (CSR) and other PCI BARs are marked as prefetchable. NVIDIA GPU driver uses the write-combine feature for mapping the prefetchable BARs to improve performance. This problem applies to all other drivers which want to enable WC.

Unless you mean the noveau drivers this simply does not matter.  Please
don't spam the kernel lists with issues with your broken and license
violating drivers.