diff mbox series

vfio/pci: Support 8-byte PCI loads and stores

Message ID 20240419135323.1282064-1-gbayer@linux.ibm.com (mailing list archive)
State New, archived
Headers show
Series vfio/pci: Support 8-byte PCI loads and stores | expand

Commit Message

Gerd Bayer April 19, 2024, 1:53 p.m. UTC 
From: Ben Segal <bpsegal@us.ibm.com>

Many PCI adapters can benefit or even require full 64bit read
and write access to their registers. In order to enable work on
user-space drivers for these devices add two new variations
vfio_pci_core_io{read|write}64 of the existing access methods
when the architecture supports 64-bit ioreads and iowrites.

Signed-off-by: Ben Segal <bpsegal@us.ibm.com>
Co-developed-by: Gerd Bayer <gbayer@linux.ibm.com>
Signed-off-by: Gerd Bayer <gbayer@linux.ibm.com>
---

Hi all,

we've successfully used this patch with a user-mode driver for a PCI
device that requires 64bit register read/writes on s390. A quick grep
showed that there are several other drivers for PCI devices in the kernel
that use readq/writeq and eventually could use this too.
So we decided to propose this for general inclusion.

We've added conditional compiles for non-64bit architectures that
produce graceful run-time errors. However, that path is just
compile-tested.

Thank you,
Gerd Bayer

 drivers/vfio/pci/vfio_pci_rdwr.c | 39 +++++++++++++++++++++++++++++++-
 include/linux/vfio_pci_core.h    |  3 +++
 2 files changed, 41 insertions(+), 1 deletion(-)

Comments

Jason Gunthorpe April 19, 2024, 1:58 p.m. UTC | #1 
On Fri, Apr 19, 2024 at 03:53:23PM +0200, Gerd Bayer wrote:
> From: Ben Segal <bpsegal@us.ibm.com>
> 
> Many PCI adapters can benefit or even require full 64bit read
> and write access to their registers. In order to enable work on
> user-space drivers for these devices add two new variations
> vfio_pci_core_io{read|write}64 of the existing access methods
> when the architecture supports 64-bit ioreads and iowrites.
> 
> Signed-off-by: Ben Segal <bpsegal@us.ibm.com>
> Co-developed-by: Gerd Bayer <gbayer@linux.ibm.com>
> Signed-off-by: Gerd Bayer <gbayer@linux.ibm.com>
> ---
> 
> Hi all,
> 
> we've successfully used this patch with a user-mode driver for a PCI
> device that requires 64bit register read/writes on s390.

But why? S390 already has a system call for userspace to do the 64 bit
write, and newer S390 has a userspace instruction to do it.

Why would you want to use a VFIO system call on the mmio emulation
path?

mmap the registers and access them normally?

>   * Read or write from an __iomem region (MMIO or I/O port) with an excluded
> @@ -114,7 +117,41 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
>  		else
>  			fillable = 0;
>  
> -		if (fillable >= 4 && !(off % 4)) {
> +		if (fillable >= 8 && !(off % 8)) {
> +#if defined(ioread64) || defined(iowrite64)
> +			u64 val;
> +#endif
> +
> +			if (iswrite) {
> +#ifndef iowrite64
> +				pr_err_once("vfio does not support iowrite64 on this arch");
> +				return -EIO;

can't do that you have to go back to what the old stuff did and do the
4 byte copy.

Jason
Niklas Schnelle April 19, 2024, 3:57 p.m. UTC | #2 
On Fri, 2024-04-19 at 10:58 -0300, Jason Gunthorpe wrote:
> On Fri, Apr 19, 2024 at 03:53:23PM +0200, Gerd Bayer wrote:
> > From: Ben Segal <bpsegal@us.ibm.com>
> > 
> > Many PCI adapters can benefit or even require full 64bit read
> > and write access to their registers. In order to enable work on
> > user-space drivers for these devices add two new variations
> > vfio_pci_core_io{read|write}64 of the existing access methods
> > when the architecture supports 64-bit ioreads and iowrites.
> > 
> > Signed-off-by: Ben Segal <bpsegal@us.ibm.com>
> > Co-developed-by: Gerd Bayer <gbayer@linux.ibm.com>
> > Signed-off-by: Gerd Bayer <gbayer@linux.ibm.com>
> > ---
> > 
> > Hi all,
> > 
> > we've successfully used this patch with a user-mode driver for a PCI
> > device that requires 64bit register read/writes on s390.
> 
> But why? S390 already has a system call for userspace to do the 64 bit
> write, and newer S390 has a userspace instruction to do it.
> 
> Why would you want to use a VFIO system call on the mmio emulation
> path?
> 
> mmap the registers and access them normally?

It's a very good point and digging into why this wasn't used by
Benjamin. It turns out VFIO_PCI_MMAP is disabled for S390 which it
really shouldn't be especially now that we have the user-space
instructions. Before that though Benjamin turned to this interface
which then lead him to this limitation. So yeah we'll definitely verify
that it also works via VFIO_PCI_MMAP and send a patch to enable that.

That said I still think it's odd not to have the 8 byte case working
here even if it isn't the right approach. Could still be useful for
debug/testing without having to add the MIO instructions or the our
special syscall.

Thanks,
Niklas
Jason Gunthorpe April 19, 2024, 4:11 p.m. UTC | #3 
On Fri, Apr 19, 2024 at 05:57:52PM +0200, Niklas Schnelle wrote:
> On Fri, 2024-04-19 at 10:58 -0300, Jason Gunthorpe wrote:
> > On Fri, Apr 19, 2024 at 03:53:23PM +0200, Gerd Bayer wrote:
> > > From: Ben Segal <bpsegal@us.ibm.com>
> > > 
> > > Many PCI adapters can benefit or even require full 64bit read
> > > and write access to their registers. In order to enable work on
> > > user-space drivers for these devices add two new variations
> > > vfio_pci_core_io{read|write}64 of the existing access methods
> > > when the architecture supports 64-bit ioreads and iowrites.
> > > 
> > > Signed-off-by: Ben Segal <bpsegal@us.ibm.com>
> > > Co-developed-by: Gerd Bayer <gbayer@linux.ibm.com>
> > > Signed-off-by: Gerd Bayer <gbayer@linux.ibm.com>
> > > ---
> > > 
> > > Hi all,
> > > 
> > > we've successfully used this patch with a user-mode driver for a PCI
> > > device that requires 64bit register read/writes on s390.
> > 
> > But why? S390 already has a system call for userspace to do the 64 bit
> > write, and newer S390 has a userspace instruction to do it.
> > 
> > Why would you want to use a VFIO system call on the mmio emulation
> > path?
> > 
> > mmap the registers and access them normally?
> 
> It's a very good point and digging into why this wasn't used by
> Benjamin. It turns out VFIO_PCI_MMAP is disabled for S390 which it
> really shouldn't be especially now that we have the user-space
> instructions. Before that though Benjamin turned to this interface
> which then lead him to this limitation. So yeah we'll definitely verify
> that it also works via VFIO_PCI_MMAP and send a patch to enable that.

Make sense to me!

> That said I still think it's odd not to have the 8 byte case working
> here even if it isn't the right approach. Could still be useful for
> debug/testing without having to add the MIO instructions or the our
> special syscall.

Yes, this also makes sense, but this patch needs some adjusting

Jason
Alex Williamson April 19, 2024, 4:47 p.m. UTC | #4 
On Fri, 19 Apr 2024 13:11:35 -0300
Jason Gunthorpe <jgg@ziepe.ca> wrote:

> On Fri, Apr 19, 2024 at 05:57:52PM +0200, Niklas Schnelle wrote:
> > On Fri, 2024-04-19 at 10:58 -0300, Jason Gunthorpe wrote:  
> > > On Fri, Apr 19, 2024 at 03:53:23PM +0200, Gerd Bayer wrote:  
> > > > From: Ben Segal <bpsegal@us.ibm.com>
> > > > 
> > > > Many PCI adapters can benefit or even require full 64bit read
> > > > and write access to their registers. In order to enable work on
> > > > user-space drivers for these devices add two new variations
> > > > vfio_pci_core_io{read|write}64 of the existing access methods
> > > > when the architecture supports 64-bit ioreads and iowrites.
> > > > 
> > > > Signed-off-by: Ben Segal <bpsegal@us.ibm.com>
> > > > Co-developed-by: Gerd Bayer <gbayer@linux.ibm.com>
> > > > Signed-off-by: Gerd Bayer <gbayer@linux.ibm.com>
> > > > ---
> > > > 
> > > > Hi all,
> > > > 
> > > > we've successfully used this patch with a user-mode driver for a PCI
> > > > device that requires 64bit register read/writes on s390.  
> > > 
> > > But why? S390 already has a system call for userspace to do the 64 bit
> > > write, and newer S390 has a userspace instruction to do it.
> > > 
> > > Why would you want to use a VFIO system call on the mmio emulation
> > > path?
> > > 
> > > mmap the registers and access them normally?  
> > 
> > It's a very good point and digging into why this wasn't used by
> > Benjamin. It turns out VFIO_PCI_MMAP is disabled for S390 which it
> > really shouldn't be especially now that we have the user-space
> > instructions. Before that though Benjamin turned to this interface
> > which then lead him to this limitation. So yeah we'll definitely verify
> > that it also works via VFIO_PCI_MMAP and send a patch to enable that.  
> 
> Make sense to me!
> 
> > That said I still think it's odd not to have the 8 byte case working
> > here even if it isn't the right approach. Could still be useful for
> > debug/testing without having to add the MIO instructions or the our
> > special syscall.  
> 
> Yes, this also makes sense, but this patch needs some adjusting

Yes, I think so too, falling back to 4-byte accesses of course if
8-byte is not available.  Thanks,

Alex
Gerd Bayer April 22, 2024, 11:08 a.m. UTC | #5 
On Fri, 2024-04-19 at 10:47 -0600, Alex Williamson wrote:
> On Fri, 19 Apr 2024 13:11:35 -0300
> Jason Gunthorpe <jgg@ziepe.ca> wrote:
> 
> > On Fri, Apr 19, 2024 at 05:57:52PM +0200, Niklas Schnelle wrote:
> > > On Fri, 2024-04-19 at 10:58 -0300, Jason Gunthorpe wrote:  
> > > > On Fri, Apr 19, 2024 at 03:53:23PM +0200, Gerd Bayer wrote:  
> > > > > From: Ben Segal <bpsegal@us.ibm.com>
> > > > > 
> > > > > Many PCI adapters can benefit or even require full 64bit read
> > > > > and write access to their registers. In order to enable work
> > > > > on
> > > > > user-space drivers for these devices add two new variations
> > > > > vfio_pci_core_io{read|write}64 of the existing access methods
> > > > > when the architecture supports 64-bit ioreads and iowrites.
> > > > > 
> > > > > Signed-off-by: Ben Segal <bpsegal@us.ibm.com>
> > > > > Co-developed-by: Gerd Bayer <gbayer@linux.ibm.com>
> > > > > Signed-off-by: Gerd Bayer <gbayer@linux.ibm.com>
> > > > > ---
> > > > > 
> > > > > Hi all,
> > > > > 
> > > > > we've successfully used this patch with a user-mode driver
> > > > > for a PCI device that requires 64bit register read/writes on
> > > > > s390.  
> > > > 
> > > > But why? S390 already has a system call for userspace to do the
> > > > 64 bit write, and newer S390 has a userspace instruction to do
> > > > it.
> > > > 
> > > > Why would you want to use a VFIO system call on the mmio
> > > > emulation path?
> > > > 
> > > > mmap the registers and access them normally?  
> > > 
> > > It's a very good point and digging into why this wasn't used by
> > > Benjamin. It turns out VFIO_PCI_MMAP is disabled for S390 which
> > > it really shouldn't be especially now that we have the user-space
> > > instructions. Before that though Benjamin turned to this
> > > interface which then lead him to this limitation. So yeah we'll
> > > definitely verify that it also works via VFIO_PCI_MMAP and send a
> > > patch to enable that.  
> > 
> > Make sense to me!
> > 
> > > That said I still think it's odd not to have the 8 byte case
> > > working here even if it isn't the right approach. Could still be
> > > useful for debug/testing without having to add the MIO
> > > instructions or the our special syscall.  
> > 
> > Yes, this also makes sense, but this patch needs some adjusting
> 
> Yes, I think so too, falling back to 4-byte accesses of course if
> 8-byte is not available.  Thanks,

So I'll rework this to simply fall back to 32-bit if 64-bit is not
available in a v2. And we'll investigate the VFIO_PCI_MMAP case
separately.

Thank you,
Gerd
diff mbox series

Patch

diff --git a/drivers/vfio/pci/vfio_pci_rdwr.c b/drivers/vfio/pci/vfio_pci_rdwr.c
index 03b8f7ada1ac..3f91945ea3ff 100644
--- a/drivers/vfio/pci/vfio_pci_rdwr.c
+++ b/drivers/vfio/pci/vfio_pci_rdwr.c
@@ -89,6 +89,9 @@  EXPORT_SYMBOL_GPL(vfio_pci_core_ioread##size);
 VFIO_IOREAD(8)
 VFIO_IOREAD(16)
 VFIO_IOREAD(32)
+#ifdef ioread64
+VFIO_IOREAD(64)
+#endif
 
 /*
  * Read or write from an __iomem region (MMIO or I/O port) with an excluded
@@ -114,7 +117,41 @@  ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
 		else
 			fillable = 0;
 
-		if (fillable >= 4 && !(off % 4)) {
+		if (fillable >= 8 && !(off % 8)) {
+#if defined(ioread64) || defined(iowrite64)
+			u64 val;
+#endif
+
+			if (iswrite) {
+#ifndef iowrite64
+				pr_err_once("vfio does not support iowrite64 on this arch");
+				return -EIO;
+#else
+				if (copy_from_user(&val, buf, 8))
+					return -EFAULT;
+
+				ret = vfio_pci_core_iowrite64(vdev, test_mem,
+							 val, io + off);
+				if (ret)
+					return ret;
+#endif
+			} else {
+#ifndef ioread64
+				pr_err_once("vfio does not support ioread64 on this arch");
+				return -EIO;
+#else
+				ret = vfio_pci_core_ioread64(vdev, test_mem,
+							&val, io + off);
+				if (ret)
+					return ret;
+
+				if (copy_to_user(buf, &val, 8))
+					return -EFAULT;
+#endif
+			}
+
+			filled = 8;
+		} else if (fillable >= 4 && !(off % 4)) {
 			u32 val;
 
 			if (iswrite) {
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index a2c8b8bba711..f4cf5fd2350c 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -157,5 +157,8 @@  int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev,	\
 VFIO_IOREAD_DECLATION(8)
 VFIO_IOREAD_DECLATION(16)
 VFIO_IOREAD_DECLATION(32)
+#ifdef ioread64
+VFIO_IOREAD_DECLATION(64)
+#endif
 
 #endif /* VFIO_PCI_CORE_H */