[v1,1/2] vfio:iommu: Use capabilities do report IOMMU informations

Commit Message

Pierre Morel Jan. 9, 2019, 12:41 p.m. UTC

We add a new flag, VFIO_IOMMU_INFO_CAPABILITIES, inside the
vfio_iommu_type1_info to specify the support for capabilities.

We add a new capability, with id VFIO_IOMMU_INFO_CAP_DMA
in the capability list of the VFIO_IOMMU_GET_INFO ioctl.

Signed-off-by: Pierre Morel <[e-mail]>
---
 include/uapi/linux/vfio.h | 9 +++++++++
 1 file changed, 9 insertions(+)

Comments

Alex Williamson Jan. 9, 2019, 3:37 p.m. UTC | #1

On Wed,  9 Jan 2019 13:41:53 +0100
Pierre Morel <[e-mail]> wrote:

> We add a new flag, VFIO_IOMMU_INFO_CAPABILITIES, inside the
> vfio_iommu_type1_info to specify the support for capabilities.
> 
> We add a new capability, with id VFIO_IOMMU_INFO_CAP_DMA
> in the capability list of the VFIO_IOMMU_GET_INFO ioctl.
> 
> Signed-off-by: Pierre Morel <[e-mail]>
> ---
>  include/uapi/linux/vfio.h | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
> index 8131028..54c4fcb 100644
> --- a/include/uapi/linux/vfio.h
> +++ b/include/uapi/linux/vfio.h
> @@ -669,6 +669,15 @@ struct vfio_iommu_type1_info {
>  	__u32	flags;
>  #define VFIO_IOMMU_INFO_PGSIZES (1 << 0)	/* supported page sizes info */
>  	__u64	iova_pgsizes;		/* Bitmap of supported page sizes */
> +#define VFIO_IOMMU_INFO_CAPABILITIES (1 << 1)  /* support capabilities info */
> +	__u64   cap_offset;     /* Offset within info struct of first cap */
> +};
> +
> +#define VFIO_IOMMU_INFO_CAP_DMA 1
> +struct vfio_iommu_cap_dma {
> +	struct vfio_info_cap_header header;
> +	__u64   dma_start;
> +	__u64   dma_end;
>  };
>  
>  #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)

Unfortunately for most systems, a simple start and end is not really
sufficient to describe the available IOVA space, there are often
reserved regions intermixed, so this is not really a complete
solution.  Shameer tried to solve this last year[1] but we ran into a
road block that Intel IGD devices impose a reserved range of IOVA
spaces reported to the user that conflict with existing assignment of
this device and we haven't figured out yet how to be more selective of
the enforcement of those reserved ranges.  Thanks,

Alex

[1] https://lkml.org/lkml/2018/4/18/293

Pierre Morel Jan. 9, 2019, 5:07 p.m. UTC | #2

On 09/01/2019 16:37, Alex Williamson wrote:
> On Wed,  9 Jan 2019 13:41:53 +0100
> Pierre Morel <[e-mail]> wrote:
> 
>> We add a new flag, VFIO_IOMMU_INFO_CAPABILITIES, inside the
>> vfio_iommu_type1_info to specify the support for capabilities.
>>
>> We add a new capability, with id VFIO_IOMMU_INFO_CAP_DMA
>> in the capability list of the VFIO_IOMMU_GET_INFO ioctl.
>>
>> Signed-off-by: Pierre Morel <[e-mail]>
>> ---
>>   include/uapi/linux/vfio.h | 9 +++++++++
>>   1 file changed, 9 insertions(+)
>>
>> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
>> index 8131028..54c4fcb 100644
>> --- a/include/uapi/linux/vfio.h
>> +++ b/include/uapi/linux/vfio.h
>> @@ -669,6 +669,15 @@ struct vfio_iommu_type1_info {
>>   	__u32	flags;
>>   #define VFIO_IOMMU_INFO_PGSIZES (1 << 0)	/* supported page sizes info */
>>   	__u64	iova_pgsizes;		/* Bitmap of supported page sizes */
>> +#define VFIO_IOMMU_INFO_CAPABILITIES (1 << 1)  /* support capabilities info */
>> +	__u64   cap_offset;     /* Offset within info struct of first cap */
>> +};
>> +
>> +#define VFIO_IOMMU_INFO_CAP_DMA 1
>> +struct vfio_iommu_cap_dma {
>> +	struct vfio_info_cap_header header;
>> +	__u64   dma_start;
>> +	__u64   dma_end;
>>   };
>>   
>>   #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
> 
> Unfortunately for most systems, a simple start and end is not really
> sufficient to describe the available IOVA space, there are often
> reserved regions intermixed, so this is not really a complete
> solution.  Shameer tried to solve this last year[1] but we ran into a
> road block that Intel IGD devices impose a reserved range of IOVA
> spaces reported to the user that conflict with existing assignment of
> this device and we haven't figured out yet how to be more selective of
> the enforcement of those reserved ranges.  Thanks,
> 
> Alex
> 
> [1] https://lkml.org/lkml/2018/4/18/293
> 

I understand that some architecture may be more complex and have special 
needs.
However the IOMMU geometry is a constant for all IOMMU devices and
is reported by the geometry in the iommu operations.

This makes the IOMMU geometry a special case.
It is also a special case because it is an inclusive description of 
available memory, to oppose to the exclusive description given by the 
windows.

Isn't it possible to separate the IOMMU geometry, which is really 
related to the IOMMU chip, from other windows exclusion related to the 
system memory mapping?

Retrieving the IOMMU geometry is very important for us because the 
driver inside the guest must get it and program the IOMMU based on these 
values.

Regards,
Pierre

Shameerali Kolothum Thodi Jan. 9, 2019, 5:08 p.m. UTC | #3

> -----Original Message-----
> From: Alex Williamson [mailto:[e-mail]]
> Sent: 09 January 2019 15:37
> To: Pierre Morel <[e-mail]>
> Cc: [e-mail]; [e-mail];
> [e-mail]; [e-mail]; [e-mail];
> [e-mail]; [e-mail]; Shameerali Kolothum Thodi
> <[e-mail]>
> Subject: Re: [PATCH v1 1/2] vfio:iommu: Use capabilities do report IOMMU
> informations
> 
> On Wed,  9 Jan 2019 13:41:53 +0100
> Pierre Morel <[e-mail]> wrote:
> 
> > We add a new flag, VFIO_IOMMU_INFO_CAPABILITIES, inside the
> > vfio_iommu_type1_info to specify the support for capabilities.
> >
> > We add a new capability, with id VFIO_IOMMU_INFO_CAP_DMA
> > in the capability list of the VFIO_IOMMU_GET_INFO ioctl.
> >
> > Signed-off-by: Pierre Morel <[e-mail]>
> > ---
> >  include/uapi/linux/vfio.h | 9 +++++++++
> >  1 file changed, 9 insertions(+)
> >
> > diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
> > index 8131028..54c4fcb 100644
> > --- a/include/uapi/linux/vfio.h
> > +++ b/include/uapi/linux/vfio.h
> > @@ -669,6 +669,15 @@ struct vfio_iommu_type1_info {
> >  	__u32	flags;
> >  #define VFIO_IOMMU_INFO_PGSIZES (1 << 0)	/* supported page sizes
> info */
> >  	__u64	iova_pgsizes;		/* Bitmap of supported page sizes */
> > +#define VFIO_IOMMU_INFO_CAPABILITIES (1 << 1)  /* support capabilities
> info */
> > +	__u64   cap_offset;     /* Offset within info struct of first cap */
> > +};
> > +
> > +#define VFIO_IOMMU_INFO_CAP_DMA 1
> > +struct vfio_iommu_cap_dma {
> > +	struct vfio_info_cap_header header;
> > +	__u64   dma_start;
> > +	__u64   dma_end;
> >  };
> >
> >  #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
> 
> Unfortunately for most systems, a simple start and end is not really
> sufficient to describe the available IOVA space, there are often
> reserved regions intermixed, so this is not really a complete
> solution.  Shameer tried to solve this last year[1] but we ran into a
> road block that Intel IGD devices impose a reserved range of IOVA
> spaces reported to the user that conflict with existing assignment of
> this device and we haven't figured out yet how to be more selective of
> the enforcement of those reserved ranges.  Thanks,

Right. I had further discussions to unblock this at KVM forum/off-list with
Intel folks and was promised some help.

IIRC the discussion was at, Kevin/Ashok will take another look on your
proposed approach to exclude the RMRR[1] and see whether that is good
enough or not.

Kevin/Ashok,

Please update if you had a chance to look into it.

Thanks,
Shameer

[1]. https://lkml.org/lkml/2018/6/5/897

> Alex
> 
> [1] https://lkml.org/lkml/2018/4/18/293

Alex Williamson Jan. 9, 2019, 7:43 p.m. UTC | #4

On Wed, 9 Jan 2019 18:07:19 +0100
Pierre Morel <[e-mail]> wrote:

> On 09/01/2019 16:37, Alex Williamson wrote:
> > On Wed,  9 Jan 2019 13:41:53 +0100
> > Pierre Morel <[e-mail]> wrote:
> >   
> >> We add a new flag, VFIO_IOMMU_INFO_CAPABILITIES, inside the
> >> vfio_iommu_type1_info to specify the support for capabilities.
> >>
> >> We add a new capability, with id VFIO_IOMMU_INFO_CAP_DMA
> >> in the capability list of the VFIO_IOMMU_GET_INFO ioctl.
> >>
> >> Signed-off-by: Pierre Morel <[e-mail]>
> >> ---
> >>   include/uapi/linux/vfio.h | 9 +++++++++
> >>   1 file changed, 9 insertions(+)
> >>
> >> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
> >> index 8131028..54c4fcb 100644
> >> --- a/include/uapi/linux/vfio.h
> >> +++ b/include/uapi/linux/vfio.h
> >> @@ -669,6 +669,15 @@ struct vfio_iommu_type1_info {
> >>   	__u32	flags;
> >>   #define VFIO_IOMMU_INFO_PGSIZES (1 << 0)	/* supported page sizes info */
> >>   	__u64	iova_pgsizes;		/* Bitmap of supported page sizes */
> >> +#define VFIO_IOMMU_INFO_CAPABILITIES (1 << 1)  /* support capabilities info */
> >> +	__u64   cap_offset;     /* Offset within info struct of first cap */
> >> +};
> >> +
> >> +#define VFIO_IOMMU_INFO_CAP_DMA 1
> >> +struct vfio_iommu_cap_dma {
> >> +	struct vfio_info_cap_header header;
> >> +	__u64   dma_start;
> >> +	__u64   dma_end;
> >>   };
> >>   
> >>   #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)  
> > 
> > Unfortunately for most systems, a simple start and end is not really
> > sufficient to describe the available IOVA space, there are often
> > reserved regions intermixed, so this is not really a complete
> > solution.  Shameer tried to solve this last year[1] but we ran into a
> > road block that Intel IGD devices impose a reserved range of IOVA
> > spaces reported to the user that conflict with existing assignment of
> > this device and we haven't figured out yet how to be more selective of
> > the enforcement of those reserved ranges.  Thanks,
> > 
> > Alex
> > 
> > [1] https://lkml.org/lkml/2018/4/18/293
> >   
> 
> I understand that some architecture may be more complex and have special 
> needs.
> However the IOMMU geometry is a constant for all IOMMU devices and
> is reported by the geometry in the iommu operations.
> 
> This makes the IOMMU geometry a special case.

I'm not so sure that the geometry is a constant for all IOMMU devices,
nor am I sure how if that were true and it's part of an in-kernel
interface that it automatically qualifies it as the right way to expose
it to userspace.  The fact that we have a reserved region interface to
augment a basic contiguous range suggests it's known to be insufficient
even for in-kernel use.

> It is also a special case because it is an inclusive description of 
> available memory, to oppose to the exclusive description given by the 
> windows.

Geometry doesn't really have anything to do with available memory, it's
the minimum and maximum IOVA aperture.  Shameer's proposal gave us an
IOVA list, which is based on the IOMMU geometry, from which it excludes
various reserved ranges.  So if you have a less complex architecture,
you might only have one entry in the list, which gives you the start
and end of the base geometry.  Move complex architectures might have
more entries, but the geometry can still be deduced from the absolute
highest and lowest addresses within the list.  Therefore a basic
geometry capability is automatically redundant to the interface that's
already been proposed.

> Isn't it possible to separate the IOMMU geometry, which is really 
> related to the IOMMU chip, from other windows exclusion related to the 
> system memory mapping?

Why would we ever have both given the description above?

> Retrieving the IOMMU geometry is very important for us because the 
> driver inside the guest must get it and program the IOMMU based on these 
> values.

So you have motivation to help move the IOVA list proposal forward,
or some equally inclusive proposal that isn't just a stop-gap ;)
Thanks,

Alex

Tian, Kevin Jan. 10, 2019, 12:55 a.m. UTC | #5

> From: Shameerali Kolothum Thodi
> [mailto:[e-mail]]
> 
> > -----Original Message-----
> > From: Alex Williamson [mailto:[e-mail]]
> > Sent: 09 January 2019 15:37
> > To: Pierre Morel <[e-mail]>
> > Cc: [e-mail]; [e-mail];
> > [e-mail]; [e-mail]; [e-mail];
> > [e-mail]; [e-mail]; Shameerali Kolothum Thodi
> > <[e-mail]>
> > Subject: Re: [PATCH v1 1/2] vfio:iommu: Use capabilities do report
> IOMMU
> > informations
> >
> > On Wed,  9 Jan 2019 13:41:53 +0100
> > Pierre Morel <[e-mail]> wrote:
> >
> > > We add a new flag, VFIO_IOMMU_INFO_CAPABILITIES, inside the
> > > vfio_iommu_type1_info to specify the support for capabilities.
> > >
> > > We add a new capability, with id VFIO_IOMMU_INFO_CAP_DMA
> > > in the capability list of the VFIO_IOMMU_GET_INFO ioctl.
> > >
> > > Signed-off-by: Pierre Morel <[e-mail]>
> > > ---
> > >  include/uapi/linux/vfio.h | 9 +++++++++
> > >  1 file changed, 9 insertions(+)
> > >
> > > diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
> > > index 8131028..54c4fcb 100644
> > > --- a/include/uapi/linux/vfio.h
> > > +++ b/include/uapi/linux/vfio.h
> > > @@ -669,6 +669,15 @@ struct vfio_iommu_type1_info {
> > >  	__u32	flags;
> > >  #define VFIO_IOMMU_INFO_PGSIZES (1 << 0)	/* supported page
> sizes
> > info */
> > >  	__u64	iova_pgsizes;		/* Bitmap of supported page sizes
> */
> > > +#define VFIO_IOMMU_INFO_CAPABILITIES (1 << 1)  /* support
> capabilities
> > info */
> > > +	__u64   cap_offset;     /* Offset within info struct of first cap */
> > > +};
> > > +
> > > +#define VFIO_IOMMU_INFO_CAP_DMA 1
> > > +struct vfio_iommu_cap_dma {
> > > +	struct vfio_info_cap_header header;
> > > +	__u64   dma_start;
> > > +	__u64   dma_end;
> > >  };
> > >
> > >  #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
> >
> > Unfortunately for most systems, a simple start and end is not really
> > sufficient to describe the available IOVA space, there are often
> > reserved regions intermixed, so this is not really a complete
> > solution.  Shameer tried to solve this last year[1] but we ran into a
> > road block that Intel IGD devices impose a reserved range of IOVA
> > spaces reported to the user that conflict with existing assignment of
> > this device and we haven't figured out yet how to be more selective of
> > the enforcement of those reserved ranges.  Thanks,
> 
> Right. I had further discussions to unblock this at KVM forum/off-list with
> Intel folks and was promised some help.
> 
> IIRC the discussion was at, Kevin/Ashok will take another look on your
> proposed approach to exclude the RMRR[1] and see whether that is good
> enough or not.
> 
> Kevin/Ashok,
> 
> Please update if you had a chance to look into it.

+Jacob.

My recent check seems to indicate [1] possibly fine with IGD, but I'll let
Ashok/Jacob to double confirm and ack since they are the owner of 
intel-iommu driver. 

> 
> Thanks,
> Shameer
> 
> [1]. https://lkml.org/lkml/2018/6/5/897
> 
> > Alex
> >
> > [1] https://lkml.org/lkml/2018/4/18/293

Pierre Morel Jan. 10, 2019, 12:47 p.m. UTC | #6

On 09/01/2019 20:43, Alex Williamson wrote:
> On Wed, 9 Jan 2019 18:07:19 +0100
> Pierre Morel <[e-mail]> wrote:
> 
>> On 09/01/2019 16:37, Alex Williamson wrote:
>>> On Wed,  9 Jan 2019 13:41:53 +0100
>>> Pierre Morel <[e-mail]> wrote:
>>>    
>>>> We add a new flag, VFIO_IOMMU_INFO_CAPABILITIES, inside the
>>>> vfio_iommu_type1_info to specify the support for capabilities.
>>>>
>>>> We add a new capability, with id VFIO_IOMMU_INFO_CAP_DMA
>>>> in the capability list of the VFIO_IOMMU_GET_INFO ioctl.
>>>>
>>>> Signed-off-by: Pierre Morel <[e-mail]>
>>>> ---
>>>>    include/uapi/linux/vfio.h | 9 +++++++++
>>>>    1 file changed, 9 insertions(+)
>>>>
>>>> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
>>>> index 8131028..54c4fcb 100644
>>>> --- a/include/uapi/linux/vfio.h
>>>> +++ b/include/uapi/linux/vfio.h
>>>> @@ -669,6 +669,15 @@ struct vfio_iommu_type1_info {
>>>>    	__u32	flags;
>>>>    #define VFIO_IOMMU_INFO_PGSIZES (1 << 0)	/* supported page sizes info */
>>>>    	__u64	iova_pgsizes;		/* Bitmap of supported page sizes */
>>>> +#define VFIO_IOMMU_INFO_CAPABILITIES (1 << 1)  /* support capabilities info */
>>>> +	__u64   cap_offset;     /* Offset within info struct of first cap */
>>>> +};
>>>> +
>>>> +#define VFIO_IOMMU_INFO_CAP_DMA 1
>>>> +struct vfio_iommu_cap_dma {
>>>> +	struct vfio_info_cap_header header;
>>>> +	__u64   dma_start;
>>>> +	__u64   dma_end;
>>>>    };
>>>>    
>>>>    #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
>>>
>>> Unfortunately for most systems, a simple start and end is not really
>>> sufficient to describe the available IOVA space, there are often
>>> reserved regions intermixed, so this is not really a complete
>>> solution.  Shameer tried to solve this last year[1] but we ran into a
>>> road block that Intel IGD devices impose a reserved range of IOVA
>>> spaces reported to the user that conflict with existing assignment of
>>> this device and we haven't figured out yet how to be more selective of
>>> the enforcement of those reserved ranges.  Thanks,
>>>
>>> Alex
>>>
>>> [1] https://lkml.org/lkml/2018/4/18/293
>>>    
>>
>> I understand that some architecture may be more complex and have special
>> needs.
>> However the IOMMU geometry is a constant for all IOMMU devices and
>> is reported by the geometry in the iommu operations.
>>
>> This makes the IOMMU geometry a special case.
> 
> I'm not so sure that the geometry is a constant for all IOMMU devices,

Sorry, I did not express myself correctly, what I mean is that the IOMMU 
geometry is independent of the system memory map but is a constant of 
the IOMMU device.
May be different for another IOMMU device, as we can have several 
different IOMMU device, typically one per PCI device, in the Z architecture.

> nor am I sure how if that were true and it's part of an in-kernel
> interface that it automatically qualifies it as the right way to expose
> it to userspace.  The fact that we have a reserved region interface to
> augment a basic contiguous range suggests it's known to be insufficient
> even for in-kernel use.
> 
>> It is also a special case because it is an inclusive description of
>> available memory, to oppose to the exclusive description given by the
>> windows.
> 
> Geometry doesn't really have anything to do with available memory, it's
> the minimum and maximum IOVA aperture.  Shameer's proposal gave us an
> IOVA list, which is based on the IOMMU geometry, from which it excludes
> various reserved ranges.  So if you have a less complex architecture,
> you might only have one entry in the list, which gives you the start
> and end of the base geometry.  Move complex architectures might have
> more entries, but the geometry can still be deduced from the absolute
> highest and lowest addresses within the list.  Therefore a basic
> geometry capability is automatically redundant to the interface that's
> already been proposed.
> 
>> Isn't it possible to separate the IOMMU geometry, which is really
>> related to the IOMMU chip, from other windows exclusion related to the
>> system memory mapping?
> 
> Why would we ever have both given the description above?

My idea for this is based on that restrictions are comming from two 
different address spaces:
- One address space is the device view before IOMMU (aperture)
- The other is the physical system view (reserved windows)

which, I thought, makes things difficult to merge.
However reading you and Shameer and the pointers you both sent, I 
understand that there are much more for me to learn, especially about 
other architectures (Intel / RMRR)

> 
>> Retrieving the IOMMU geometry is very important for us because the
>> driver inside the guest must get it and program the IOMMU based on these
>> values.
> 
> So you have motivation to help move the IOVA list proposal forward,
> or some equally inclusive proposal that isn't just a stop-gap ;)
> Thanks,
> 
> Alex
> 

I will be happy to help.
I will need some time to catch up however.
Thanks for your comments.

Regards,
Pierre

Patch

diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 8131028..54c4fcb 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -669,6 +669,15 @@  struct vfio_iommu_type1_info {
 	__u32	flags;
 #define VFIO_IOMMU_INFO_PGSIZES (1 << 0)	/* supported page sizes info */
 	__u64	iova_pgsizes;		/* Bitmap of supported page sizes */
+#define VFIO_IOMMU_INFO_CAPABILITIES (1 << 1)  /* support capabilities info */
+	__u64   cap_offset;     /* Offset within info struct of first cap */
+};
+
+#define VFIO_IOMMU_INFO_CAP_DMA 1
+struct vfio_iommu_cap_dma {
+	struct vfio_info_cap_header header;
+	__u64   dma_start;
+	__u64   dma_end;
 };
 
 #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)