[PATCHv2,08/10] vfio/mdev: Improve the create/remove sequence

Commit Message

Parav Pandit April 30, 2019, 10:49 p.m. UTC

This patch addresses below two issues and prepares the code to address
3rd issue listed below.

1. mdev device is placed on the mdev bus before it is created in the
vendor driver. Once a device is placed on the mdev bus without creating
its supporting underlying vendor device, mdev driver's probe() gets triggered.
However there isn't a stable mdev available to work on.

   create_store()
     mdev_create_device()
       device_register()
          ...
         vfio_mdev_probe()
        [...]
        parent->ops->create()
          vfio_ap_mdev_create()
            mdev_set_drvdata(mdev, matrix_mdev);
            /* Valid pointer set above */

Due to this way of initialization, mdev driver who want to use the mdev,
doesn't have a valid mdev to work on.

2. Current creation sequence is,
   parent->ops_create()
   groups_register()

Remove sequence is,
   parent->ops->remove()
   groups_unregister()

However, remove sequence should be exact mirror of creation sequence.
Once this is achieved, all users of the mdev will be terminated first
before removing underlying vendor device.
(Follow standard linux driver model).
At that point vendor's remove() ops shouldn't failed because device is
taken off the bus that should terminate the users.

3. When remove operation fails, mdev sysfs removal attempts to add the
file back on already removed device. Following call trace [1] is observed.

[1] call trace:
kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327 sysfs_create_file_ns+0x7f/0x90
kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted 5.1.0-rc6-vdevbus+ #6
kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS 2.0b 08/09/2016
kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
kernel: Call Trace:
kernel: remove_store+0xdc/0x100 [mdev]
kernel: kernfs_fop_write+0x113/0x1a0
kernel: vfs_write+0xad/0x1b0
kernel: ksys_write+0x5a/0xe0
kernel: do_syscall_64+0x5a/0x210
kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe

Therefore, mdev core is improved in following ways.

1. Before placing mdev devices on the bus, perform vendor drivers
creation which supports the mdev creation.
This ensures that mdev specific all necessary fields are initialized
before a given mdev can be accessed by bus driver.
This follows standard Linux kernel bus and device model similar to other
widely used PCI bus.

2. During remove flow, first remove the device from the bus. This
ensures that any bus specific devices and data is cleared.
Once device is taken of the mdev bus, perform remove() of mdev from the
vendor driver.

3. Linux core device model provides way to register and auto unregister
the device sysfs attribute groups at dev->groups.
Make use of this groups to let core create the groups and simplify code
to avoid explicit groups creation and removal.

A below stack dump of a mdev device remove process also ensures that
vfio driver guards against device removal already in use.

 cat /proc/21962/stack
[<0>] vfio_del_group_dev+0x216/0x3c0 [vfio]
[<0>] mdev_remove+0x21/0x40 [mdev]
[<0>] device_release_driver_internal+0xe8/0x1b0
[<0>] bus_remove_device+0xf9/0x170
[<0>] device_del+0x168/0x350
[<0>] mdev_device_remove_common+0x1d/0x50 [mdev]
[<0>] mdev_device_remove+0x8c/0xd0 [mdev]
[<0>] remove_store+0x71/0x90 [mdev]
[<0>] kernfs_fop_write+0x113/0x1a0
[<0>] vfs_write+0xad/0x1b0
[<0>] ksys_write+0x5a/0xe0
[<0>] do_syscall_64+0x5a/0x210
[<0>] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[<0>] 0xffffffffffffffff

This prepares the code to eliminate calling device_create_file() in
subsquent patch.

Signed-off-by: Parav Pandit <[e-mail]>
---
 drivers/vfio/mdev/mdev_core.c    | 94 +++++++++-----------------------
 drivers/vfio/mdev/mdev_private.h |  2 +-
 drivers/vfio/mdev/mdev_sysfs.c   |  2 +-
 3 files changed, 27 insertions(+), 71 deletions(-)

Comments

Cornelia Huck May 8, 2019, 5:09 p.m. UTC | #1

On Tue, 30 Apr 2019 17:49:35 -0500
Parav Pandit <[e-mail]> wrote:

> This patch addresses below two issues and prepares the code to address
> 3rd issue listed below.
> 
> 1. mdev device is placed on the mdev bus before it is created in the
> vendor driver. Once a device is placed on the mdev bus without creating
> its supporting underlying vendor device, mdev driver's probe() gets triggered.
> However there isn't a stable mdev available to work on.
> 
>    create_store()
>      mdev_create_device()
>        device_register()
>           ...
>          vfio_mdev_probe()
>         [...]
>         parent->ops->create()
>           vfio_ap_mdev_create()
>             mdev_set_drvdata(mdev, matrix_mdev);
>             /* Valid pointer set above */
> 
> Due to this way of initialization, mdev driver who want to use the mdev,
> doesn't have a valid mdev to work on.
> 
> 2. Current creation sequence is,
>    parent->ops_create()
>    groups_register()
> 
> Remove sequence is,
>    parent->ops->remove()
>    groups_unregister()
> 
> However, remove sequence should be exact mirror of creation sequence.
> Once this is achieved, all users of the mdev will be terminated first
> before removing underlying vendor device.
> (Follow standard linux driver model).
> At that point vendor's remove() ops shouldn't failed because device is
> taken off the bus that should terminate the users.
> 
> 3. When remove operation fails, mdev sysfs removal attempts to add the
> file back on already removed device. Following call trace [1] is observed.
> 
> [1] call trace:
> kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327 sysfs_create_file_ns+0x7f/0x90
> kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted 5.1.0-rc6-vdevbus+ #6
> kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS 2.0b 08/09/2016
> kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
> kernel: Call Trace:
> kernel: remove_store+0xdc/0x100 [mdev]
> kernel: kernfs_fop_write+0x113/0x1a0
> kernel: vfs_write+0xad/0x1b0
> kernel: ksys_write+0x5a/0xe0
> kernel: do_syscall_64+0x5a/0x210
> kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe
> 
> Therefore, mdev core is improved in following ways.
> 
> 1. Before placing mdev devices on the bus, perform vendor drivers
> creation which supports the mdev creation.
> This ensures that mdev specific all necessary fields are initialized
> before a given mdev can be accessed by bus driver.
> This follows standard Linux kernel bus and device model similar to other
> widely used PCI bus.
> 
> 2. During remove flow, first remove the device from the bus. This
> ensures that any bus specific devices and data is cleared.
> Once device is taken of the mdev bus, perform remove() of mdev from the
> vendor driver.
> 
> 3. Linux core device model provides way to register and auto unregister
> the device sysfs attribute groups at dev->groups.
> Make use of this groups to let core create the groups and simplify code
> to avoid explicit groups creation and removal.
> 
> A below stack dump of a mdev device remove process also ensures that
> vfio driver guards against device removal already in use.
> 
>  cat /proc/21962/stack
> [<0>] vfio_del_group_dev+0x216/0x3c0 [vfio]
> [<0>] mdev_remove+0x21/0x40 [mdev]
> [<0>] device_release_driver_internal+0xe8/0x1b0
> [<0>] bus_remove_device+0xf9/0x170
> [<0>] device_del+0x168/0x350
> [<0>] mdev_device_remove_common+0x1d/0x50 [mdev]
> [<0>] mdev_device_remove+0x8c/0xd0 [mdev]
> [<0>] remove_store+0x71/0x90 [mdev]
> [<0>] kernfs_fop_write+0x113/0x1a0
> [<0>] vfs_write+0xad/0x1b0
> [<0>] ksys_write+0x5a/0xe0
> [<0>] do_syscall_64+0x5a/0x210
> [<0>] entry_SYSCALL_64_after_hwframe+0x49/0xbe
> [<0>] 0xffffffffffffffff
> 
> This prepares the code to eliminate calling device_create_file() in
> subsquent patch.

I'm afraid I have a bit of a problem following this explanation, so let
me try to summarize what the patch does to make sure that I understand
it correctly:

- Add the sysfs groups to device->groups so that the driver core deals
  with proper registration/deregistration.
- Split the device registration/deregistration sequence so that some
  things can be done between initialization of the device and hooking
  it up to the infrastructure respectively after deregistering it from
  the infrastructure but before giving up our final reference. In
  particular, this means invoking the ->create and ->remove callback in
  those new windows. This gives the vendor driver an initialized mdev
  device to work with during creation.
- Don't allow ->remove to fail, as the device is already removed from
  the infrastructure at that point in time.

> 
> Signed-off-by: Parav Pandit <[e-mail]>
> ---
>  drivers/vfio/mdev/mdev_core.c    | 94 +++++++++-----------------------
>  drivers/vfio/mdev/mdev_private.h |  2 +-
>  drivers/vfio/mdev/mdev_sysfs.c   |  2 +-
>  3 files changed, 27 insertions(+), 71 deletions(-)

(...)

> @@ -310,41 +265,43 @@ int mdev_device_create(struct kobject *kobj,
>  
>  	mdev->parent = parent;
>  
> +	device_initialize(&mdev->dev);
>  	mdev->dev.parent  = dev;
>  	mdev->dev.bus     = &mdev_bus_type;
>  	mdev->dev.release = mdev_device_release;
>  	dev_set_name(&mdev->dev, "%pUl", uuid);
> +	mdev->dev.groups = parent->ops->mdev_attr_groups;

I like that, that makes things much easier.

> +	mdev->type_kobj = kobj;
>  
> -	ret = device_register(&mdev->dev);
> -	if (ret) {
> -		put_device(&mdev->dev);
> -		goto mdev_fail;
> -	}
> +	ret = parent->ops->create(kobj, mdev);
> +	if (ret)
> +		goto ops_create_fail;
>  
> -	ret = mdev_device_create_ops(kobj, mdev);
> +	ret = device_add(&mdev->dev);
>  	if (ret)
> -		goto create_fail;
> +		goto add_fail;
>  
>  	ret = mdev_create_sysfs_files(&mdev->dev, type);
> -	if (ret) {
> -		mdev_device_remove_ops(mdev, true);
> -		goto create_fail;
> -	}
> +	if (ret)
> +		goto sysfs_fail;
>  
> -	mdev->type_kobj = kobj;
>  	mdev->active = true;
>  	dev_dbg(&mdev->dev, "MDEV: created\n");
>  
>  	return 0;
>  
> -create_fail:
> -	device_unregister(&mdev->dev);
> +sysfs_fail:
> +	device_del(&mdev->dev);
> +add_fail:
> +	parent->ops->remove(mdev);
> +ops_create_fail:
> +	put_device(&mdev->dev);
>  mdev_fail:
>  	mdev_put_parent(parent);
>  	return ret;
>  }
>  
> -int mdev_device_remove(struct device *dev, bool force_remove)
> +int mdev_device_remove(struct device *dev)
>  {
>  	struct mdev_device *mdev, *tmp;
>  	struct mdev_parent *parent;
> @@ -373,16 +330,15 @@ int mdev_device_remove(struct device *dev, bool force_remove)
>  	mutex_unlock(&mdev_list_lock);
>  
>  	type = to_mdev_type(mdev->type_kobj);
> +	mdev_remove_sysfs_files(dev, type);
> +	device_del(&mdev->dev);
>  	parent = mdev->parent;
> +	ret = parent->ops->remove(mdev);
> +	if (ret)
> +		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);

I think carrying on with removal regardless of the return code of the
->remove callback makes sense, as it simply matches usual practice.
However, are we sure that every vendor driver works well with that? I
think it should, as removal from bus unregistration (vs. from the sysfs
file) was always something it could not veto, but have you looked at
the individual drivers?

>  
> -	ret = mdev_device_remove_ops(mdev, force_remove);
> -	if (ret) {
> -		mdev->active = true;
> -		return ret;
> -	}
> -
> -	mdev_remove_sysfs_files(dev, type);
> -	device_unregister(dev);
> +	/* Balances with device_initialize() */
> +	put_device(&mdev->dev);
>  	mdev_put_parent(parent);
>  
>  	return 0;

I think that looks sane in general, but the commit message might
benefit from tweaking.

Parav Pandit May 8, 2019, 10:06 p.m. UTC | #2

> -----Original Message-----
> From: Cornelia Huck <[e-mail]>
> Sent: Wednesday, May 8, 2019 12:10 PM
> To: Parav Pandit <[e-mail]>
> Cc: [e-mail]; [e-mail];
> [e-mail]; [e-mail]; [e-mail]
> Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
> sequence
> 
> On Tue, 30 Apr 2019 17:49:35 -0500
> Parav Pandit <[e-mail]> wrote:
> 
> > This patch addresses below two issues and prepares the code to address
> > 3rd issue listed below.
> >
> > 1. mdev device is placed on the mdev bus before it is created in the
> > vendor driver. Once a device is placed on the mdev bus without
> > creating its supporting underlying vendor device, mdev driver's probe()
> gets triggered.
> > However there isn't a stable mdev available to work on.
> >
> >    create_store()
> >      mdev_create_device()
> >        device_register()
> >           ...
> >          vfio_mdev_probe()
> >         [...]
> >         parent->ops->create()
> >           vfio_ap_mdev_create()
> >             mdev_set_drvdata(mdev, matrix_mdev);
> >             /* Valid pointer set above */
> >
> > Due to this way of initialization, mdev driver who want to use the
> > mdev, doesn't have a valid mdev to work on.
> >
> > 2. Current creation sequence is,
> >    parent->ops_create()
> >    groups_register()
> >
> > Remove sequence is,
> >    parent->ops->remove()
> >    groups_unregister()
> >
> > However, remove sequence should be exact mirror of creation sequence.
> > Once this is achieved, all users of the mdev will be terminated first
> > before removing underlying vendor device.
> > (Follow standard linux driver model).
> > At that point vendor's remove() ops shouldn't failed because device is
> > taken off the bus that should terminate the users.
> >
> > 3. When remove operation fails, mdev sysfs removal attempts to add the
> > file back on already removed device. Following call trace [1] is observed.
> >
> > [1] call trace:
> > kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327
> > sysfs_create_file_ns+0x7f/0x90
> > kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted
> > 5.1.0-rc6-vdevbus+ #6
> > kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS 2.0b
> > 08/09/2016
> > kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
> > kernel: Call Trace:
> > kernel: remove_store+0xdc/0x100 [mdev]
> > kernel: kernfs_fop_write+0x113/0x1a0
> > kernel: vfs_write+0xad/0x1b0
> > kernel: ksys_write+0x5a/0xe0
> > kernel: do_syscall_64+0x5a/0x210
> > kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe
> >
> > Therefore, mdev core is improved in following ways.
> >
> > 1. Before placing mdev devices on the bus, perform vendor drivers
> > creation which supports the mdev creation.
> > This ensures that mdev specific all necessary fields are initialized
> > before a given mdev can be accessed by bus driver.
> > This follows standard Linux kernel bus and device model similar to
> > other widely used PCI bus.
> >
> > 2. During remove flow, first remove the device from the bus. This
> > ensures that any bus specific devices and data is cleared.
> > Once device is taken of the mdev bus, perform remove() of mdev from
> > the vendor driver.
> >
> > 3. Linux core device model provides way to register and auto
> > unregister the device sysfs attribute groups at dev->groups.
> > Make use of this groups to let core create the groups and simplify
> > code to avoid explicit groups creation and removal.
> >
> > A below stack dump of a mdev device remove process also ensures that
> > vfio driver guards against device removal already in use.
> >
> >  cat /proc/21962/stack
> > [<0>] vfio_del_group_dev+0x216/0x3c0 [vfio] [<0>]
> > mdev_remove+0x21/0x40 [mdev] [<0>]
> > device_release_driver_internal+0xe8/0x1b0
> > [<0>] bus_remove_device+0xf9/0x170
> > [<0>] device_del+0x168/0x350
> > [<0>] mdev_device_remove_common+0x1d/0x50 [mdev] [<0>]
> > mdev_device_remove+0x8c/0xd0 [mdev] [<0>] remove_store+0x71/0x90
> > [mdev] [<0>] kernfs_fop_write+0x113/0x1a0 [<0>] vfs_write+0xad/0x1b0
> > [<0>] ksys_write+0x5a/0xe0 [<0>] do_syscall_64+0x5a/0x210 [<0>]
> > entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > [<0>] 0xffffffffffffffff
> >
> > This prepares the code to eliminate calling device_create_file() in
> > subsquent patch.
> 
> I'm afraid I have a bit of a problem following this explanation, so let me try
> to summarize what the patch does to make sure that I understand it
> correctly:
> 
> - Add the sysfs groups to device->groups so that the driver core deals
>   with proper registration/deregistration.
> - Split the device registration/deregistration sequence so that some
>   things can be done between initialization of the device and hooking
>   it up to the infrastructure respectively after deregistering it from
>   the infrastructure but before giving up our final reference. In
>   particular, this means invoking the ->create and ->remove callback in
>   those new windows. This gives the vendor driver an initialized mdev
>   device to work with during creation.
> - Don't allow ->remove to fail, as the device is already removed from
>   the infrastructure at that point in time.
> 
You got all the points pretty accurate.

> >
> > Signed-off-by: Parav Pandit <[e-mail]>
> > ---
> >  drivers/vfio/mdev/mdev_core.c    | 94 +++++++++-----------------------
> >  drivers/vfio/mdev/mdev_private.h |  2 +-
> >  drivers/vfio/mdev/mdev_sysfs.c   |  2 +-
> >  3 files changed, 27 insertions(+), 71 deletions(-)
> 
> (...)
> 
> > @@ -310,41 +265,43 @@ int mdev_device_create(struct kobject *kobj,
> >
> >  	mdev->parent = parent;
> >
> > +	device_initialize(&mdev->dev);
> >  	mdev->dev.parent  = dev;
> >  	mdev->dev.bus     = &mdev_bus_type;
> >  	mdev->dev.release = mdev_device_release;
> >  	dev_set_name(&mdev->dev, "%pUl", uuid);
> > +	mdev->dev.groups = parent->ops->mdev_attr_groups;
> 
> I like that, that makes things much easier.
> 
True.

> > +	mdev->type_kobj = kobj;
> >
> > -	ret = device_register(&mdev->dev);
> > -	if (ret) {
> > -		put_device(&mdev->dev);
> > -		goto mdev_fail;
> > -	}
> > +	ret = parent->ops->create(kobj, mdev);
> > +	if (ret)
> > +		goto ops_create_fail;
> >
> > -	ret = mdev_device_create_ops(kobj, mdev);
> > +	ret = device_add(&mdev->dev);
> >  	if (ret)
> > -		goto create_fail;
> > +		goto add_fail;
> >
> >  	ret = mdev_create_sysfs_files(&mdev->dev, type);
> > -	if (ret) {
> > -		mdev_device_remove_ops(mdev, true);
> > -		goto create_fail;
> > -	}
> > +	if (ret)
> > +		goto sysfs_fail;
> >
> > -	mdev->type_kobj = kobj;
> >  	mdev->active = true;
> >  	dev_dbg(&mdev->dev, "MDEV: created\n");
> >
> >  	return 0;
> >
> > -create_fail:
> > -	device_unregister(&mdev->dev);
> > +sysfs_fail:
> > +	device_del(&mdev->dev);
> > +add_fail:
> > +	parent->ops->remove(mdev);
> > +ops_create_fail:
> > +	put_device(&mdev->dev);
> >  mdev_fail:
> >  	mdev_put_parent(parent);
> >  	return ret;
> >  }
> >
> > -int mdev_device_remove(struct device *dev, bool force_remove)
> > +int mdev_device_remove(struct device *dev)
> >  {
> >  	struct mdev_device *mdev, *tmp;
> >  	struct mdev_parent *parent;
> > @@ -373,16 +330,15 @@ int mdev_device_remove(struct device *dev,
> bool force_remove)
> >  	mutex_unlock(&mdev_list_lock);
> >
> >  	type = to_mdev_type(mdev->type_kobj);
> > +	mdev_remove_sysfs_files(dev, type);
> > +	device_del(&mdev->dev);
> >  	parent = mdev->parent;
> > +	ret = parent->ops->remove(mdev);
> > +	if (ret)
> > +		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
> 
> I think carrying on with removal regardless of the return code of the
> ->remove callback makes sense, as it simply matches usual practice.
> However, are we sure that every vendor driver works well with that? I think
> it should, as removal from bus unregistration (vs. from the sysfs
> file) was always something it could not veto, but have you looked at the
> individual drivers?
> 
I looked at following drivers a little while back.
Looked again now.

drivers/gpu/drm/i915/gvt/kvmgt.c which clears the handle valid in intel_vgpu_release(), which should finish first before remove() is invoked.

s390 vfio_ccw_mdev_remove() driver drivers/s390/cio/vfio_ccw_ops.c remove() always returns 0.
s39 crypo fails the remove() once vfio_ap_mdev_release marks kvm null, which should finish before remove() is invoked.
samples/vfio-mdev/mbochs.c mbochs_remove() always returns 0.

> >
> > -	ret = mdev_device_remove_ops(mdev, force_remove);
> > -	if (ret) {
> > -		mdev->active = true;
> > -		return ret;
> > -	}
> > -
> > -	mdev_remove_sysfs_files(dev, type);
> > -	device_unregister(dev);
> > +	/* Balances with device_initialize() */
> > +	put_device(&mdev->dev);
> >  	mdev_put_parent(parent);
> >
> >  	return 0;
> 
> I think that looks sane in general, but the commit message might benefit
> from tweaking.
Part of your description is more crisp than my commit message, I can probably take snippet from it to improve?
Or any specific entries in commit message that I should address?

Cornelia Huck May 9, 2019, 9:06 a.m. UTC | #3

[vfio-ap folks: find a question regarding removal further down]

On Wed, 8 May 2019 22:06:48 +0000
Parav Pandit <[e-mail]> wrote:

> > -----Original Message-----
> > From: Cornelia Huck <[e-mail]>
> > Sent: Wednesday, May 8, 2019 12:10 PM
> > To: Parav Pandit <[e-mail]>
> > Cc: [e-mail]; [e-mail];
> > [e-mail]; [e-mail]; [e-mail]
> > Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
> > sequence
> > 
> > On Tue, 30 Apr 2019 17:49:35 -0500
> > Parav Pandit <[e-mail]> wrote:
> >   
> > > This patch addresses below two issues and prepares the code to address
> > > 3rd issue listed below.
> > >
> > > 1. mdev device is placed on the mdev bus before it is created in the
> > > vendor driver. Once a device is placed on the mdev bus without
> > > creating its supporting underlying vendor device, mdev driver's probe()  
> > gets triggered.  
> > > However there isn't a stable mdev available to work on.
> > >
> > >    create_store()
> > >      mdev_create_device()
> > >        device_register()
> > >           ...
> > >          vfio_mdev_probe()
> > >         [...]
> > >         parent->ops->create()
> > >           vfio_ap_mdev_create()
> > >             mdev_set_drvdata(mdev, matrix_mdev);
> > >             /* Valid pointer set above */
> > >
> > > Due to this way of initialization, mdev driver who want to use the

s/want/wants/

> > > mdev, doesn't have a valid mdev to work on.
> > >
> > > 2. Current creation sequence is,
> > >    parent->ops_create()
> > >    groups_register()
> > >
> > > Remove sequence is,
> > >    parent->ops->remove()
> > >    groups_unregister()
> > >
> > > However, remove sequence should be exact mirror of creation sequence.
> > > Once this is achieved, all users of the mdev will be terminated first
> > > before removing underlying vendor device.
> > > (Follow standard linux driver model).
> > > At that point vendor's remove() ops shouldn't failed because device is

s/failed/fail/

> > > taken off the bus that should terminate the users.

"because taking the device off the bus should terminate any usage" ?

> > >
> > > 3. When remove operation fails, mdev sysfs removal attempts to add the
> > > file back on already removed device. Following call trace [1] is observed.
> > >
> > > [1] call trace:
> > > kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327
> > > sysfs_create_file_ns+0x7f/0x90
> > > kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted
> > > 5.1.0-rc6-vdevbus+ #6
> > > kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS 2.0b
> > > 08/09/2016
> > > kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
> > > kernel: Call Trace:
> > > kernel: remove_store+0xdc/0x100 [mdev]
> > > kernel: kernfs_fop_write+0x113/0x1a0
> > > kernel: vfs_write+0xad/0x1b0
> > > kernel: ksys_write+0x5a/0xe0
> > > kernel: do_syscall_64+0x5a/0x210
> > > kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > >
> > > Therefore, mdev core is improved in following ways.
> > >
> > > 1. Before placing mdev devices on the bus, perform vendor drivers
> > > creation which supports the mdev creation.

"invoke the vendor driver ->create callback" ?

> > > This ensures that mdev specific all necessary fields are initialized

"that all necessary mdev specific fields are initialized" ?

> > > before a given mdev can be accessed by bus driver.
> > > This follows standard Linux kernel bus and device model similar to
> > > other widely used PCI bus.

"This follows standard practice on other Linux device model buses." ?

> > >
> > > 2. During remove flow, first remove the device from the bus. This
> > > ensures that any bus specific devices and data is cleared.
> > > Once device is taken of the mdev bus, perform remove() of mdev from

s/of/off/

> > > the vendor driver.
> > >
> > > 3. Linux core device model provides way to register and auto
> > > unregister the device sysfs attribute groups at dev->groups.

"The driver core provides a way to automatically register and
unregister sysfs attributes via dev->groups." ?

> > > Make use of this groups to let core create the groups and simplify
> > > code to avoid explicit groups creation and removal.
> > >
> > > A below stack dump of a mdev device remove process also ensures that
> > > vfio driver guards against device removal already in use.
> > >
> > >  cat /proc/21962/stack
> > > [<0>] vfio_del_group_dev+0x216/0x3c0 [vfio] [<0>]
> > > mdev_remove+0x21/0x40 [mdev] [<0>]
> > > device_release_driver_internal+0xe8/0x1b0
> > > [<0>] bus_remove_device+0xf9/0x170
> > > [<0>] device_del+0x168/0x350
> > > [<0>] mdev_device_remove_common+0x1d/0x50 [mdev] [<0>]
> > > mdev_device_remove+0x8c/0xd0 [mdev] [<0>] remove_store+0x71/0x90
> > > [mdev] [<0>] kernfs_fop_write+0x113/0x1a0 [<0>] vfs_write+0xad/0x1b0
> > > [<0>] ksys_write+0x5a/0xe0 [<0>] do_syscall_64+0x5a/0x210 [<0>]
> > > entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > > [<0>] 0xffffffffffffffff
> > >
> > > This prepares the code to eliminate calling device_create_file() in
> > > subsquent patch.  

I find this stack dump and explanation more confusing than
enlightening. Maybe just drop it?

> > 
> > I'm afraid I have a bit of a problem following this explanation, so let me try
> > to summarize what the patch does to make sure that I understand it
> > correctly:
> > 
> > - Add the sysfs groups to device->groups so that the driver core deals
> >   with proper registration/deregistration.
> > - Split the device registration/deregistration sequence so that some
> >   things can be done between initialization of the device and hooking
> >   it up to the infrastructure respectively after deregistering it from
> >   the infrastructure but before giving up our final reference. In
> >   particular, this means invoking the ->create and ->remove callback in
> >   those new windows. This gives the vendor driver an initialized mdev
> >   device to work with during creation.
> > - Don't allow ->remove to fail, as the device is already removed from
> >   the infrastructure at that point in time.
> >   
> You got all the points pretty accurate.

Ok, good.

> 
> > >
> > > Signed-off-by: Parav Pandit <[e-mail]>
> > > ---
> > >  drivers/vfio/mdev/mdev_core.c    | 94 +++++++++-----------------------
> > >  drivers/vfio/mdev/mdev_private.h |  2 +-
> > >  drivers/vfio/mdev/mdev_sysfs.c   |  2 +-
> > >  3 files changed, 27 insertions(+), 71 deletions(-)  
> > 
> > (...)

> > > @@ -373,16 +330,15 @@ int mdev_device_remove(struct device *dev,  
> > bool force_remove)  
> > >  	mutex_unlock(&mdev_list_lock);
> > >
> > >  	type = to_mdev_type(mdev->type_kobj);
> > > +	mdev_remove_sysfs_files(dev, type);
> > > +	device_del(&mdev->dev);
> > >  	parent = mdev->parent;
> > > +	ret = parent->ops->remove(mdev);
> > > +	if (ret)
> > > +		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);  
> > 
> > I think carrying on with removal regardless of the return code of the  
> > ->remove callback makes sense, as it simply matches usual practice.  
> > However, are we sure that every vendor driver works well with that? I think
> > it should, as removal from bus unregistration (vs. from the sysfs
> > file) was always something it could not veto, but have you looked at the
> > individual drivers?
> >   
> I looked at following drivers a little while back.
> Looked again now.
> 
> drivers/gpu/drm/i915/gvt/kvmgt.c which clears the handle valid in intel_vgpu_release(), which should finish first before remove() is invoked.
> 
> s390 vfio_ccw_mdev_remove() driver drivers/s390/cio/vfio_ccw_ops.c remove() always returns 0.
> s39 crypo fails the remove() once vfio_ap_mdev_release marks kvm null, which should finish before remove() is invoked.

That one is giving me a bit of a headache (the ->kvm reference is
supposed to keep us from detaching while a vm is running), so let's cc:
the vfio-ap maintainers to see whether they have any concerns.

> samples/vfio-mdev/mbochs.c mbochs_remove() always returns 0.
> 
> > >
> > > -	ret = mdev_device_remove_ops(mdev, force_remove);
> > > -	if (ret) {
> > > -		mdev->active = true;
> > > -		return ret;
> > > -	}
> > > -
> > > -	mdev_remove_sysfs_files(dev, type);
> > > -	device_unregister(dev);
> > > +	/* Balances with device_initialize() */
> > > +	put_device(&mdev->dev);
> > >  	mdev_put_parent(parent);
> > >
> > >  	return 0;  
> > 
> > I think that looks sane in general, but the commit message might benefit
> > from tweaking.  
> Part of your description is more crisp than my commit message, I can probably take snippet from it to improve?
> Or any specific entries in commit message that I should address?

I have added some comments inline (mostly some wording tweaks).

Feel free to take anything from my summary as well.

Pierre Morel May 9, 2019, 4:26 p.m. UTC | #4

On 09/05/2019 11:06, Cornelia Huck wrote:
> [vfio-ap folks: find a question regarding removal further down]
> 
> On Wed, 8 May 2019 22:06:48 +0000
> Parav Pandit <[e-mail]> wrote:
> 
>>> -----Original Message-----
>>> From: Cornelia Huck <[e-mail]>
>>> Sent: Wednesday, May 8, 2019 12:10 PM
>>> To: Parav Pandit <[e-mail]>
>>> Cc: [e-mail]; [e-mail];
>>> [e-mail]; [e-mail]; [e-mail]
>>> Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
>>> sequence
>>>
>>> On Tue, 30 Apr 2019 17:49:35 -0500
>>> Parav Pandit <[e-mail]> wrote:
>>>    

...snip...

>>>> @@ -373,16 +330,15 @@ int mdev_device_remove(struct device *dev,
>>> bool force_remove)
>>>>   	mutex_unlock(&mdev_list_lock);
>>>>
>>>>   	type = to_mdev_type(mdev->type_kobj);
>>>> +	mdev_remove_sysfs_files(dev, type);
>>>> +	device_del(&mdev->dev);
>>>>   	parent = mdev->parent;
>>>> +	ret = parent->ops->remove(mdev);
>>>> +	if (ret)
>>>> +		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
>>>
>>> I think carrying on with removal regardless of the return code of the
>>> ->remove callback makes sense, as it simply matches usual practice.
>>> However, are we sure that every vendor driver works well with that? I think
>>> it should, as removal from bus unregistration (vs. from the sysfs
>>> file) was always something it could not veto, but have you looked at the
>>> individual drivers?
>>>    
>> I looked at following drivers a little while back.
>> Looked again now.
>>
>> drivers/gpu/drm/i915/gvt/kvmgt.c which clears the handle valid in intel_vgpu_release(), which should finish first before remove() is invoked.
>>
>> s390 vfio_ccw_mdev_remove() driver drivers/s390/cio/vfio_ccw_ops.c remove() always returns 0.
>> s39 crypo fails the remove() once vfio_ap_mdev_release marks kvm null, which should finish before remove() is invoked.
> 
> That one is giving me a bit of a headache (the ->kvm reference is
> supposed to keep us from detaching while a vm is running), so let's cc:
> the vfio-ap maintainers to see whether they have any concerns.
> 

We are aware of this race and we did correct this in the IRQ patches for 
which it would have become a real issue.
We now increment/decrement the KVM reference counter inside open and 
release.
Should be right after this.

Thanks for the cc,
Pierre

Parav Pandit May 9, 2019, 7:19 p.m. UTC | #5

> -----Original Message-----
> From: Cornelia Huck <[e-mail]>
> Sent: Thursday, May 9, 2019 4:06 AM
> To: Parav Pandit <[e-mail]>
> Cc: [e-mail]; [e-mail];
> [e-mail]; [e-mail]; [e-mail];
> Tony Krowiak <[e-mail]>; Pierre Morel
> <[e-mail]>; Halil Pasic <[e-mail]>
> Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
> sequence
> 
> [vfio-ap folks: find a question regarding removal further down]
> 
> On Wed, 8 May 2019 22:06:48 +0000
> Parav Pandit <[e-mail]> wrote:
> 
> > > -----Original Message-----
> > > From: Cornelia Huck <[e-mail]>
> > > Sent: Wednesday, May 8, 2019 12:10 PM
> > > To: Parav Pandit <[e-mail]>
> > > Cc: [e-mail]; [e-mail];
> > > [e-mail]; [e-mail]; [e-mail]
> > > Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
> > > sequence
> > >
> > > On Tue, 30 Apr 2019 17:49:35 -0500
> > > Parav Pandit <[e-mail]> wrote:
> > >
> > > > This patch addresses below two issues and prepares the code to
> > > > address 3rd issue listed below.
> > > >
> > > > 1. mdev device is placed on the mdev bus before it is created in
> > > > the vendor driver. Once a device is placed on the mdev bus without
> > > > creating its supporting underlying vendor device, mdev driver's
> > > > probe()
> > > gets triggered.
> > > > However there isn't a stable mdev available to work on.
> > > >
> > > >    create_store()
> > > >      mdev_create_device()
> > > >        device_register()
> > > >           ...
> > > >          vfio_mdev_probe()
> > > >         [...]
> > > >         parent->ops->create()
> > > >           vfio_ap_mdev_create()
> > > >             mdev_set_drvdata(mdev, matrix_mdev);
> > > >             /* Valid pointer set above */
> > > >
> > > > Due to this way of initialization, mdev driver who want to use the
> 
> s/want/wants/
> 
> > > > mdev, doesn't have a valid mdev to work on.
> > > >
> > > > 2. Current creation sequence is,
> > > >    parent->ops_create()
> > > >    groups_register()
> > > >
> > > > Remove sequence is,
> > > >    parent->ops->remove()
> > > >    groups_unregister()
> > > >
> > > > However, remove sequence should be exact mirror of creation
> sequence.
> > > > Once this is achieved, all users of the mdev will be terminated
> > > > first before removing underlying vendor device.
> > > > (Follow standard linux driver model).
> > > > At that point vendor's remove() ops shouldn't failed because
> > > > device is
> 
> s/failed/fail/
> 
> > > > taken off the bus that should terminate the users.
> 
> "because taking the device off the bus should terminate any usage" ?
> 
> > > >
> > > > 3. When remove operation fails, mdev sysfs removal attempts to add
> > > > the file back on already removed device. Following call trace [1] is
> observed.
> > > >
> > > > [1] call trace:
> > > > kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327
> > > > sysfs_create_file_ns+0x7f/0x90
> > > > kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted
> > > > 5.1.0-rc6-vdevbus+ #6
> > > > kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS
> > > > 2.0b
> > > > 08/09/2016
> > > > kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
> > > > kernel: Call Trace:
> > > > kernel: remove_store+0xdc/0x100 [mdev]
> > > > kernel: kernfs_fop_write+0x113/0x1a0
> > > > kernel: vfs_write+0xad/0x1b0
> > > > kernel: ksys_write+0x5a/0xe0
> > > > kernel: do_syscall_64+0x5a/0x210
> > > > kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > > >
> > > > Therefore, mdev core is improved in following ways.
> > > >
> > > > 1. Before placing mdev devices on the bus, perform vendor drivers
> > > > creation which supports the mdev creation.
> 
> "invoke the vendor driver ->create callback" ?
> 
> > > > This ensures that mdev specific all necessary fields are
> > > > initialized
> 
> "that all necessary mdev specific fields are initialized" ?
> 
> > > > before a given mdev can be accessed by bus driver.
> > > > This follows standard Linux kernel bus and device model similar to
> > > > other widely used PCI bus.
> 
> "This follows standard practice on other Linux device model buses." ?
> 
> > > >
> > > > 2. During remove flow, first remove the device from the bus. This
> > > > ensures that any bus specific devices and data is cleared.
> > > > Once device is taken of the mdev bus, perform remove() of mdev
> > > > from
> 
> s/of/off/
> 
> > > > the vendor driver.
> > > >
> > > > 3. Linux core device model provides way to register and auto
> > > > unregister the device sysfs attribute groups at dev->groups.
> 
> "The driver core provides a way to automatically register and unregister sysfs
> attributes via dev->groups." ?
> 
> > > > Make use of this groups to let core create the groups and simplify
> > > > code to avoid explicit groups creation and removal.
> > > >
> > > > A below stack dump of a mdev device remove process also ensures
> > > > that vfio driver guards against device removal already in use.
> > > >
> > > >  cat /proc/21962/stack
> > > > [<0>] vfio_del_group_dev+0x216/0x3c0 [vfio] [<0>]
> > > > mdev_remove+0x21/0x40 [mdev] [<0>]
> > > > device_release_driver_internal+0xe8/0x1b0
> > > > [<0>] bus_remove_device+0xf9/0x170 [<0>] device_del+0x168/0x350
> > > > [<0>] mdev_device_remove_common+0x1d/0x50 [mdev] [<0>]
> > > > mdev_device_remove+0x8c/0xd0 [mdev] [<0>]
> remove_store+0x71/0x90
> > > > [mdev] [<0>] kernfs_fop_write+0x113/0x1a0 [<0>]
> > > > vfs_write+0xad/0x1b0 [<0>] ksys_write+0x5a/0xe0 [<0>]
> > > > do_syscall_64+0x5a/0x210 [<0>]
> > > > entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > > > [<0>] 0xffffffffffffffff
> > > >
> > > > This prepares the code to eliminate calling device_create_file()
> > > > in subsquent patch.
> 
> I find this stack dump and explanation more confusing than enlightening.
> Maybe just drop it?
> 
> > >
> > > I'm afraid I have a bit of a problem following this explanation, so
> > > let me try to summarize what the patch does to make sure that I
> > > understand it
> > > correctly:
> > >
> > > - Add the sysfs groups to device->groups so that the driver core deals
> > >   with proper registration/deregistration.
> > > - Split the device registration/deregistration sequence so that some
> > >   things can be done between initialization of the device and hooking
> > >   it up to the infrastructure respectively after deregistering it from
> > >   the infrastructure but before giving up our final reference. In
> > >   particular, this means invoking the ->create and ->remove callback in
> > >   those new windows. This gives the vendor driver an initialized mdev
> > >   device to work with during creation.
> > > - Don't allow ->remove to fail, as the device is already removed from
> > >   the infrastructure at that point in time.
> > >
> > You got all the points pretty accurate.
> 
> Ok, good.
> 
> >
> > > >
> > > > Signed-off-by: Parav Pandit <[e-mail]>
> > > > ---
> > > >  drivers/vfio/mdev/mdev_core.c    | 94 +++++++++-----------------------
> > > >  drivers/vfio/mdev/mdev_private.h |  2 +-
> > > >  drivers/vfio/mdev/mdev_sysfs.c   |  2 +-
> > > >  3 files changed, 27 insertions(+), 71 deletions(-)
> > >
> > > (...)
> 
> > > > @@ -373,16 +330,15 @@ int mdev_device_remove(struct device *dev,
> > > bool force_remove)
> > > >  	mutex_unlock(&mdev_list_lock);
> > > >
> > > >  	type = to_mdev_type(mdev->type_kobj);
> > > > +	mdev_remove_sysfs_files(dev, type);
> > > > +	device_del(&mdev->dev);
> > > >  	parent = mdev->parent;
> > > > +	ret = parent->ops->remove(mdev);
> > > > +	if (ret)
> > > > +		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
> > >
> > > I think carrying on with removal regardless of the return code of
> > > the
> > > ->remove callback makes sense, as it simply matches usual practice.
> > > However, are we sure that every vendor driver works well with that?
> > > I think it should, as removal from bus unregistration (vs. from the
> > > sysfs
> > > file) was always something it could not veto, but have you looked at
> > > the individual drivers?
> > >
> > I looked at following drivers a little while back.
> > Looked again now.
> >
> > drivers/gpu/drm/i915/gvt/kvmgt.c which clears the handle valid in
> intel_vgpu_release(), which should finish first before remove() is invoked.
> >
> > s390 vfio_ccw_mdev_remove() driver drivers/s390/cio/vfio_ccw_ops.c
> remove() always returns 0.
> > s39 crypo fails the remove() once vfio_ap_mdev_release marks kvm null,
> which should finish before remove() is invoked.
> 
> That one is giving me a bit of a headache (the ->kvm reference is supposed
> to keep us from detaching while a vm is running), so let's cc:
> the vfio-ap maintainers to see whether they have any concerns.
> 
I probably wrote wrongly.
vfio_ap_mdev_remove() fails if the VM is already running (i.e. vfio_ap_mdev_release() is not yet called).

And if VM is running it guarded by the vfio_mdev driver which is the one who binds to the device mdev device.
That is why I shown the above stack trace in the commit log, indicating that vfio driver is guarding it.

> > samples/vfio-mdev/mbochs.c mbochs_remove() always returns 0.
> >
> > > >
> > > > -	ret = mdev_device_remove_ops(mdev, force_remove);
> > > > -	if (ret) {
> > > > -		mdev->active = true;
> > > > -		return ret;
> > > > -	}
> > > > -
> > > > -	mdev_remove_sysfs_files(dev, type);
> > > > -	device_unregister(dev);
> > > > +	/* Balances with device_initialize() */
> > > > +	put_device(&mdev->dev);
> > > >  	mdev_put_parent(parent);
> > > >
> > > >  	return 0;
> > >
> > > I think that looks sane in general, but the commit message might
> > > benefit from tweaking.
> > Part of your description is more crisp than my commit message, I can
> probably take snippet from it to improve?
> > Or any specific entries in commit message that I should address?
> 
> I have added some comments inline (mostly some wording tweaks).
> 
> Feel free to take anything from my summary as well.

Halil Pasic May 9, 2019, 10:12 p.m. UTC | #6

On Thu, 9 May 2019 18:26:59 +0200
Pierre Morel <[e-mail]> wrote:

> On 09/05/2019 11:06, Cornelia Huck wrote:
> > [vfio-ap folks: find a question regarding removal further down]
> > 
> > On Wed, 8 May 2019 22:06:48 +0000
> > Parav Pandit <[e-mail]> wrote:
> > 
> >>> -----Original Message-----
> >>> From: Cornelia Huck <[e-mail]>
> >>> Sent: Wednesday, May 8, 2019 12:10 PM
> >>> To: Parav Pandit <[e-mail]>
> >>> Cc: [e-mail]; [e-mail];
> >>> [e-mail]; [e-mail]; [e-mail]
> >>> Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
> >>> sequence
> >>>
> >>> On Tue, 30 Apr 2019 17:49:35 -0500
> >>> Parav Pandit <[e-mail]> wrote:
> >>>    
> 
> ...snip...
> 
> >>>> @@ -373,16 +330,15 @@ int mdev_device_remove(struct device *dev,
> >>> bool force_remove)
> >>>>   	mutex_unlock(&mdev_list_lock);
> >>>>
> >>>>   	type = to_mdev_type(mdev->type_kobj);
> >>>> +	mdev_remove_sysfs_files(dev, type);
> >>>> +	device_del(&mdev->dev);
> >>>>   	parent = mdev->parent;
> >>>> +	ret = parent->ops->remove(mdev);
> >>>> +	if (ret)
> >>>> +		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
> >>>
> >>> I think carrying on with removal regardless of the return code of the
> >>> ->remove callback makes sense, as it simply matches usual practice.
> >>> However, are we sure that every vendor driver works well with that? I think
> >>> it should, as removal from bus unregistration (vs. from the sysfs
> >>> file) was always something it could not veto, but have you looked at the
> >>> individual drivers?
> >>>    
> >> I looked at following drivers a little while back.
> >> Looked again now.
> >>
> >> drivers/gpu/drm/i915/gvt/kvmgt.c which clears the handle valid in intel_vgpu_release(), which should finish first before remove() is invoked.
> >>
> >> s390 vfio_ccw_mdev_remove() driver drivers/s390/cio/vfio_ccw_ops.c remove() always returns 0.
> >> s39 crypo fails the remove() once vfio_ap_mdev_release marks kvm null, which should finish before remove() is invoked.
> > 
> > That one is giving me a bit of a headache (the ->kvm reference is
> > supposed to keep us from detaching while a vm is running), so let's cc:
> > the vfio-ap maintainers to see whether they have any concerns.
> > 
> 
> We are aware of this race and we did correct this in the IRQ patches for 
> which it would have become a real issue.
> We now increment/decrement the KVM reference counter inside open and 
> release.
> Should be right after this.
> 

Tony, what is your take on this? I don't have the bandwidth to think
this through properly, but my intuition tells me: this might be more
complicated than what Pierre's response suggests.

Regards,
Halil

Pierre Morel May 10, 2019, 7:08 a.m. UTC | #7

On 09/05/2019 21:19, Parav Pandit wrote:
> 
> 
>> -----Original Message-----
>> From: Cornelia Huck <[e-mail]>
>> Sent: Thursday, May 9, 2019 4:06 AM
>> To: Parav Pandit <[e-mail]>
>> Cc: [e-mail]; [e-mail];
>> [e-mail]; [e-mail]; [e-mail];
>> Tony Krowiak <[e-mail]>; Pierre Morel
>> <[e-mail]>; Halil Pasic <[e-mail]>
>> Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
>> sequence
>>
>> [vfio-ap folks: find a question regarding removal further down]
>>
>> On Wed, 8 May 2019 22:06:48 +0000
>> Parav Pandit <[e-mail]> wrote:
>>
>>>> -----Original Message-----
>>>> From: Cornelia Huck <[e-mail]>
>>>> Sent: Wednesday, May 8, 2019 12:10 PM
>>>> To: Parav Pandit <[e-mail]>
>>>> Cc: [e-mail]; [e-mail];
>>>> [e-mail]; [e-mail]; [e-mail]
>>>> Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
>>>> sequence
>>>>
>>>> On Tue, 30 Apr 2019 17:49:35 -0500
>>>> Parav Pandit <[e-mail]> wrote:
>>>>
>>>>> This patch addresses below two issues and prepares the code to
>>>>> address 3rd issue listed below.
>>>>>
>>>>> 1. mdev device is placed on the mdev bus before it is created in
>>>>> the vendor driver. Once a device is placed on the mdev bus without
>>>>> creating its supporting underlying vendor device, mdev driver's
>>>>> probe()
>>>> gets triggered.
>>>>> However there isn't a stable mdev available to work on.
>>>>>
>>>>>     create_store()
>>>>>       mdev_create_device()
>>>>>         device_register()
>>>>>            ...
>>>>>           vfio_mdev_probe()
>>>>>          [...]
>>>>>          parent->ops->create()
>>>>>            vfio_ap_mdev_create()
>>>>>              mdev_set_drvdata(mdev, matrix_mdev);
>>>>>              /* Valid pointer set above */
>>>>>
>>>>> Due to this way of initialization, mdev driver who want to use the
>>
>> s/want/wants/
>>
>>>>> mdev, doesn't have a valid mdev to work on.
>>>>>
>>>>> 2. Current creation sequence is,
>>>>>     parent->ops_create()
>>>>>     groups_register()
>>>>>
>>>>> Remove sequence is,
>>>>>     parent->ops->remove()
>>>>>     groups_unregister()
>>>>>
>>>>> However, remove sequence should be exact mirror of creation
>> sequence.
>>>>> Once this is achieved, all users of the mdev will be terminated
>>>>> first before removing underlying vendor device.
>>>>> (Follow standard linux driver model).
>>>>> At that point vendor's remove() ops shouldn't failed because
>>>>> device is
>>
>> s/failed/fail/
>>
>>>>> taken off the bus that should terminate the users.
>>
>> "because taking the device off the bus should terminate any usage" ?
>>
>>>>>
>>>>> 3. When remove operation fails, mdev sysfs removal attempts to add
>>>>> the file back on already removed device. Following call trace [1] is
>> observed.
>>>>>
>>>>> [1] call trace:
>>>>> kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327
>>>>> sysfs_create_file_ns+0x7f/0x90
>>>>> kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted
>>>>> 5.1.0-rc6-vdevbus+ #6
>>>>> kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS
>>>>> 2.0b
>>>>> 08/09/2016
>>>>> kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
>>>>> kernel: Call Trace:
>>>>> kernel: remove_store+0xdc/0x100 [mdev]
>>>>> kernel: kernfs_fop_write+0x113/0x1a0
>>>>> kernel: vfs_write+0xad/0x1b0
>>>>> kernel: ksys_write+0x5a/0xe0
>>>>> kernel: do_syscall_64+0x5a/0x210
>>>>> kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe
>>>>>
>>>>> Therefore, mdev core is improved in following ways.
>>>>>
>>>>> 1. Before placing mdev devices on the bus, perform vendor drivers
>>>>> creation which supports the mdev creation.
>>
>> "invoke the vendor driver ->create callback" ?
>>
>>>>> This ensures that mdev specific all necessary fields are
>>>>> initialized
>>
>> "that all necessary mdev specific fields are initialized" ?
>>
>>>>> before a given mdev can be accessed by bus driver.
>>>>> This follows standard Linux kernel bus and device model similar to
>>>>> other widely used PCI bus.
>>
>> "This follows standard practice on other Linux device model buses." ?
>>
>>>>>
>>>>> 2. During remove flow, first remove the device from the bus. This
>>>>> ensures that any bus specific devices and data is cleared.
>>>>> Once device is taken of the mdev bus, perform remove() of mdev
>>>>> from
>>
>> s/of/off/
>>
>>>>> the vendor driver.
>>>>>
>>>>> 3. Linux core device model provides way to register and auto
>>>>> unregister the device sysfs attribute groups at dev->groups.
>>
>> "The driver core provides a way to automatically register and unregister sysfs
>> attributes via dev->groups." ?
>>
>>>>> Make use of this groups to let core create the groups and simplify
>>>>> code to avoid explicit groups creation and removal.
>>>>>
>>>>> A below stack dump of a mdev device remove process also ensures
>>>>> that vfio driver guards against device removal already in use.
>>>>>
>>>>>   cat /proc/21962/stack
>>>>> [<0>] vfio_del_group_dev+0x216/0x3c0 [vfio] [<0>]
>>>>> mdev_remove+0x21/0x40 [mdev] [<0>]
>>>>> device_release_driver_internal+0xe8/0x1b0
>>>>> [<0>] bus_remove_device+0xf9/0x170 [<0>] device_del+0x168/0x350
>>>>> [<0>] mdev_device_remove_common+0x1d/0x50 [mdev] [<0>]
>>>>> mdev_device_remove+0x8c/0xd0 [mdev] [<0>]
>> remove_store+0x71/0x90
>>>>> [mdev] [<0>] kernfs_fop_write+0x113/0x1a0 [<0>]
>>>>> vfs_write+0xad/0x1b0 [<0>] ksys_write+0x5a/0xe0 [<0>]
>>>>> do_syscall_64+0x5a/0x210 [<0>]
>>>>> entry_SYSCALL_64_after_hwframe+0x49/0xbe
>>>>> [<0>] 0xffffffffffffffff
>>>>>
>>>>> This prepares the code to eliminate calling device_create_file()
>>>>> in subsquent patch.
>>
>> I find this stack dump and explanation more confusing than enlightening.
>> Maybe just drop it?
>>
>>>>
>>>> I'm afraid I have a bit of a problem following this explanation, so
>>>> let me try to summarize what the patch does to make sure that I
>>>> understand it
>>>> correctly:
>>>>
>>>> - Add the sysfs groups to device->groups so that the driver core deals
>>>>    with proper registration/deregistration.
>>>> - Split the device registration/deregistration sequence so that some
>>>>    things can be done between initialization of the device and hooking
>>>>    it up to the infrastructure respectively after deregistering it from
>>>>    the infrastructure but before giving up our final reference. In
>>>>    particular, this means invoking the ->create and ->remove callback in
>>>>    those new windows. This gives the vendor driver an initialized mdev
>>>>    device to work with during creation.
>>>> - Don't allow ->remove to fail, as the device is already removed from
>>>>    the infrastructure at that point in time.
>>>>
>>> You got all the points pretty accurate.
>>
>> Ok, good.
>>
>>>
>>>>>
>>>>> Signed-off-by: Parav Pandit <[e-mail]>
>>>>> ---
>>>>>   drivers/vfio/mdev/mdev_core.c    | 94 +++++++++-----------------------
>>>>>   drivers/vfio/mdev/mdev_private.h |  2 +-
>>>>>   drivers/vfio/mdev/mdev_sysfs.c   |  2 +-
>>>>>   3 files changed, 27 insertions(+), 71 deletions(-)
>>>>
>>>> (...)
>>
>>>>> @@ -373,16 +330,15 @@ int mdev_device_remove(struct device *dev,
>>>> bool force_remove)
>>>>>   	mutex_unlock(&mdev_list_lock);
>>>>>
>>>>>   	type = to_mdev_type(mdev->type_kobj);
>>>>> +	mdev_remove_sysfs_files(dev, type);
>>>>> +	device_del(&mdev->dev);
>>>>>   	parent = mdev->parent;
>>>>> +	ret = parent->ops->remove(mdev);
>>>>> +	if (ret)
>>>>> +		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
>>>>
>>>> I think carrying on with removal regardless of the return code of
>>>> the
>>>> ->remove callback makes sense, as it simply matches usual practice.
>>>> However, are we sure that every vendor driver works well with that?
>>>> I think it should, as removal from bus unregistration (vs. from the
>>>> sysfs
>>>> file) was always something it could not veto, but have you looked at
>>>> the individual drivers?
>>>>
>>> I looked at following drivers a little while back.
>>> Looked again now.
>>>
>>> drivers/gpu/drm/i915/gvt/kvmgt.c which clears the handle valid in
>> intel_vgpu_release(), which should finish first before remove() is invoked.
>>>
>>> s390 vfio_ccw_mdev_remove() driver drivers/s390/cio/vfio_ccw_ops.c
>> remove() always returns 0.
>>> s39 crypo fails the remove() once vfio_ap_mdev_release marks kvm null,
>> which should finish before remove() is invoked.
>>
>> That one is giving me a bit of a headache (the ->kvm reference is supposed
>> to keep us from detaching while a vm is running), so let's cc:
>> the vfio-ap maintainers to see whether they have any concerns.
>>
> I probably wrote wrongly.
> vfio_ap_mdev_remove() fails if the VM is already running (i.e. vfio_ap_mdev_release() is not yet called).
> 
> And if VM is running it guarded by the vfio_mdev driver which is the one who binds to the device mdev device.
> That is why I shown the above stack trace in the commit log, indicating that vfio driver is guarding it.
> 

I looked again, and see the race you are speaking of, it is not the one 
I thought first, which does not really mater at this stage of the driver 
but indeed between release and remove we do not take the lock in the 
right way.

We will correct this.
Thanks,

Pierre


>>> samples/vfio-mdev/mbochs.c mbochs_remove() always returns 0.
>>>
>>>>>
>>>>> -	ret = mdev_device_remove_ops(mdev, force_remove);
>>>>> -	if (ret) {
>>>>> -		mdev->active = true;
>>>>> -		return ret;
>>>>> -	}
>>>>> -
>>>>> -	mdev_remove_sysfs_files(dev, type);
>>>>> -	device_unregister(dev);
>>>>> +	/* Balances with device_initialize() */
>>>>> +	put_device(&mdev->dev);
>>>>>   	mdev_put_parent(parent);
>>>>>
>>>>>   	return 0;
>>>>
>>>> I think that looks sane in general, but the commit message might
>>>> benefit from tweaking.
>>> Part of your description is more crisp than my commit message, I can
>> probably take snippet from it to improve?
>>> Or any specific entries in commit message that I should address?
>>
>> I have added some comments inline (mostly some wording tweaks).
>>
>> Feel free to take anything from my summary as well.
>

Parav Pandit May 14, 2019, 8:34 p.m. UTC | #8

Hi Alex, Cornelia,


> -----Original Message-----
> From: [e-mail] <linux-kernel-
> [e-mail]> On Behalf Of Parav Pandit
> Sent: Thursday, May 9, 2019 2:20 PM
> To: Cornelia Huck <[e-mail]>
> Cc: [e-mail]; [e-mail];
> [e-mail]; [e-mail]; [e-mail];
> Tony Krowiak <[e-mail]>; Pierre Morel
> <[e-mail]>; Halil Pasic <[e-mail]>
> Subject: RE: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
> sequence
> 
> 
> 
> > -----Original Message-----
> > From: Cornelia Huck <[e-mail]>
> > Sent: Thursday, May 9, 2019 4:06 AM
> > To: Parav Pandit <[e-mail]>
> > Cc: [e-mail]; [e-mail];
> > [e-mail]; [e-mail]; [e-mail];
> > Tony Krowiak <[e-mail]>; Pierre Morel
> > <[e-mail]>; Halil Pasic <[e-mail]>
> > Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
> > sequence
> >
> > [vfio-ap folks: find a question regarding removal further down]
> >
> > On Wed, 8 May 2019 22:06:48 +0000
> > Parav Pandit <[e-mail]> wrote:
> >
> > > > -----Original Message-----
> > > > From: Cornelia Huck <[e-mail]>
> > > > Sent: Wednesday, May 8, 2019 12:10 PM
> > > > To: Parav Pandit <[e-mail]>
> > > > Cc: [e-mail]; [e-mail];
> > > > [e-mail]; [e-mail];
> [e-mail]
> > > > Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
> > > > sequence
> > > >
> > > > On Tue, 30 Apr 2019 17:49:35 -0500 Parav Pandit
> > > > <[e-mail]> wrote:
> > > >
> > > > > This patch addresses below two issues and prepares the code to
> > > > > address 3rd issue listed below.
> > > > >
> > > > > 1. mdev device is placed on the mdev bus before it is created in
> > > > > the vendor driver. Once a device is placed on the mdev bus
> > > > > without creating its supporting underlying vendor device, mdev
> > > > > driver's
> > > > > probe()
> > > > gets triggered.
> > > > > However there isn't a stable mdev available to work on.
> > > > >
> > > > >    create_store()
> > > > >      mdev_create_device()
> > > > >        device_register()
> > > > >           ...
> > > > >          vfio_mdev_probe()
> > > > >         [...]
> > > > >         parent->ops->create()
> > > > >           vfio_ap_mdev_create()
> > > > >             mdev_set_drvdata(mdev, matrix_mdev);
> > > > >             /* Valid pointer set above */
> > > > >
> > > > > Due to this way of initialization, mdev driver who want to use
> > > > > the
> >
> > s/want/wants/
> >
> > > > > mdev, doesn't have a valid mdev to work on.
> > > > >
> > > > > 2. Current creation sequence is,
> > > > >    parent->ops_create()
> > > > >    groups_register()
> > > > >
> > > > > Remove sequence is,
> > > > >    parent->ops->remove()
> > > > >    groups_unregister()
> > > > >
> > > > > However, remove sequence should be exact mirror of creation
> > sequence.
> > > > > Once this is achieved, all users of the mdev will be terminated
> > > > > first before removing underlying vendor device.
> > > > > (Follow standard linux driver model).
> > > > > At that point vendor's remove() ops shouldn't failed because
> > > > > device is
> >
> > s/failed/fail/
> >
> > > > > taken off the bus that should terminate the users.
> >
> > "because taking the device off the bus should terminate any usage" ?
> >
> > > > >
> > > > > 3. When remove operation fails, mdev sysfs removal attempts to
> > > > > add the file back on already removed device. Following call
> > > > > trace [1] is
> > observed.
> > > > >
> > > > > [1] call trace:
> > > > > kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327
> > > > > sysfs_create_file_ns+0x7f/0x90
> > > > > kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted
> > > > > 5.1.0-rc6-vdevbus+ #6
> > > > > kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS
> > > > > 2.0b
> > > > > 08/09/2016
> > > > > kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
> > > > > kernel: Call Trace:
> > > > > kernel: remove_store+0xdc/0x100 [mdev]
> > > > > kernel: kernfs_fop_write+0x113/0x1a0
> > > > > kernel: vfs_write+0xad/0x1b0
> > > > > kernel: ksys_write+0x5a/0xe0
> > > > > kernel: do_syscall_64+0x5a/0x210
> > > > > kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > > > >
> > > > > Therefore, mdev core is improved in following ways.
> > > > >
> > > > > 1. Before placing mdev devices on the bus, perform vendor
> > > > > drivers creation which supports the mdev creation.
> >
> > "invoke the vendor driver ->create callback" ?
> >
> > > > > This ensures that mdev specific all necessary fields are
> > > > > initialized
> >
> > "that all necessary mdev specific fields are initialized" ?
> >
> > > > > before a given mdev can be accessed by bus driver.
> > > > > This follows standard Linux kernel bus and device model similar
> > > > > to other widely used PCI bus.
> >
> > "This follows standard practice on other Linux device model buses." ?
> >
> > > > >
> > > > > 2. During remove flow, first remove the device from the bus.
> > > > > This ensures that any bus specific devices and data is cleared.
> > > > > Once device is taken of the mdev bus, perform remove() of mdev
> > > > > from
> >
> > s/of/off/
> >
> > > > > the vendor driver.
> > > > >
> > > > > 3. Linux core device model provides way to register and auto
> > > > > unregister the device sysfs attribute groups at dev->groups.
> >
> > "The driver core provides a way to automatically register and
> > unregister sysfs attributes via dev->groups." ?
> >
> > > > > Make use of this groups to let core create the groups and
> > > > > simplify code to avoid explicit groups creation and removal.
> > > > >
> > > > > A below stack dump of a mdev device remove process also ensures
> > > > > that vfio driver guards against device removal already in use.
> > > > >
> > > > >  cat /proc/21962/stack
> > > > > [<0>] vfio_del_group_dev+0x216/0x3c0 [vfio] [<0>]
> > > > > mdev_remove+0x21/0x40 [mdev] [<0>]
> > > > > device_release_driver_internal+0xe8/0x1b0
> > > > > [<0>] bus_remove_device+0xf9/0x170 [<0>] device_del+0x168/0x350
> > > > > [<0>] mdev_device_remove_common+0x1d/0x50 [mdev] [<0>]
> > > > > mdev_device_remove+0x8c/0xd0 [mdev] [<0>]
> > remove_store+0x71/0x90
> > > > > [mdev] [<0>] kernfs_fop_write+0x113/0x1a0 [<0>]
> > > > > vfs_write+0xad/0x1b0 [<0>] ksys_write+0x5a/0xe0 [<0>]
> > > > > do_syscall_64+0x5a/0x210 [<0>]
> > > > > entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > > > > [<0>] 0xffffffffffffffff
> > > > >
> > > > > This prepares the code to eliminate calling device_create_file()
> > > > > in subsquent patch.
> >
> > I find this stack dump and explanation more confusing than enlightening.
> > Maybe just drop it?
> >
> > > >
> > > > I'm afraid I have a bit of a problem following this explanation,
> > > > so let me try to summarize what the patch does to make sure that I
> > > > understand it
> > > > correctly:
> > > >
> > > > - Add the sysfs groups to device->groups so that the driver core deals
> > > >   with proper registration/deregistration.
> > > > - Split the device registration/deregistration sequence so that some
> > > >   things can be done between initialization of the device and hooking
> > > >   it up to the infrastructure respectively after deregistering it from
> > > >   the infrastructure but before giving up our final reference. In
> > > >   particular, this means invoking the ->create and ->remove callback in
> > > >   those new windows. This gives the vendor driver an initialized mdev
> > > >   device to work with during creation.
> > > > - Don't allow ->remove to fail, as the device is already removed from
> > > >   the infrastructure at that point in time.
> > > >
> > > You got all the points pretty accurate.
> >
> > Ok, good.
> >
> > >
> > > > >
> > > > > Signed-off-by: Parav Pandit <[e-mail]>
> > > > > ---
> > > > >  drivers/vfio/mdev/mdev_core.c    | 94 +++++++++-----------------------
> > > > >  drivers/vfio/mdev/mdev_private.h |  2 +-
> > > > >  drivers/vfio/mdev/mdev_sysfs.c   |  2 +-
> > > > >  3 files changed, 27 insertions(+), 71 deletions(-)
> > > >
> > > > (...)
> >
> > > > > @@ -373,16 +330,15 @@ int mdev_device_remove(struct device
> *dev,
> > > > bool force_remove)
> > > > >  	mutex_unlock(&mdev_list_lock);
> > > > >
> > > > >  	type = to_mdev_type(mdev->type_kobj);
> > > > > +	mdev_remove_sysfs_files(dev, type);
> > > > > +	device_del(&mdev->dev);
> > > > >  	parent = mdev->parent;
> > > > > +	ret = parent->ops->remove(mdev);
> > > > > +	if (ret)
> > > > > +		dev_err(&mdev->dev, "Remove failed: err=%d\n",
> ret);
> > > >
> > > > I think carrying on with removal regardless of the return code of
> > > > the
> > > > ->remove callback makes sense, as it simply matches usual practice.
> > > > However, are we sure that every vendor driver works well with that?
> > > > I think it should, as removal from bus unregistration (vs. from
> > > > the sysfs
> > > > file) was always something it could not veto, but have you looked
> > > > at the individual drivers?
> > > >
> > > I looked at following drivers a little while back.
> > > Looked again now.
> > >
> > > drivers/gpu/drm/i915/gvt/kvmgt.c which clears the handle valid in
> > intel_vgpu_release(), which should finish first before remove() is invoked.
> > >
> > > s390 vfio_ccw_mdev_remove() driver drivers/s390/cio/vfio_ccw_ops.c
> > remove() always returns 0.
> > > s39 crypo fails the remove() once vfio_ap_mdev_release marks kvm
> > > null,
> > which should finish before remove() is invoked.
> >
> > That one is giving me a bit of a headache (the ->kvm reference is
> > supposed to keep us from detaching while a vm is running), so let's cc:
> > the vfio-ap maintainers to see whether they have any concerns.
> >
> I probably wrote wrongly.
> vfio_ap_mdev_remove() fails if the VM is already running (i.e.
> vfio_ap_mdev_release() is not yet called).
> 
> And if VM is running it guarded by the vfio_mdev driver which is the one
> who binds to the device mdev device.
> That is why I shown the above stack trace in the commit log, indicating that
> vfio driver is guarding it.
> 
> > > samples/vfio-mdev/mbochs.c mbochs_remove() always returns 0.
> > >
> > > > >
> > > > > -	ret = mdev_device_remove_ops(mdev, force_remove);
> > > > > -	if (ret) {
> > > > > -		mdev->active = true;
> > > > > -		return ret;
> > > > > -	}
> > > > > -
> > > > > -	mdev_remove_sysfs_files(dev, type);
> > > > > -	device_unregister(dev);
> > > > > +	/* Balances with device_initialize() */
> > > > > +	put_device(&mdev->dev);
> > > > >  	mdev_put_parent(parent);
> > > > >
> > > > >  	return 0;
> > > >
> > > > I think that looks sane in general, but the commit message might
> > > > benefit from tweaking.
> > > Part of your description is more crisp than my commit message, I can
> > probably take snippet from it to improve?
> > > Or any specific entries in commit message that I should address?
> >
> > I have added some comments inline (mostly some wording tweaks).
> >
> > Feel free to take anything from my summary as well.

I want to send v3 addressing commit log comment and take updated description from Cornelia, if this 3 patches looks reasonable enough.
What do you think?

Alex Williamson May 14, 2019, 10:20 p.m. UTC | #9

On Tue, 14 May 2019 20:34:12 +0000
Parav Pandit <[e-mail]> wrote:

> Hi Alex, Cornelia,
> 
> 
> > -----Original Message-----
> > From: [e-mail] <linux-kernel-  
> > [e-mail]> On Behalf Of Parav Pandit  
> > Sent: Thursday, May 9, 2019 2:20 PM
> > To: Cornelia Huck <[e-mail]>
> > Cc: [e-mail]; [e-mail];
> > [e-mail]; [e-mail]; [e-mail];
> > Tony Krowiak <[e-mail]>; Pierre Morel
> > <[e-mail]>; Halil Pasic <[e-mail]>
> > Subject: RE: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
> > sequence
> > 
> > 
> >   
> > > -----Original Message-----
> > > From: Cornelia Huck <[e-mail]>
> > > Sent: Thursday, May 9, 2019 4:06 AM
> > > To: Parav Pandit <[e-mail]>
> > > Cc: [e-mail]; [e-mail];
> > > [e-mail]; [e-mail]; [e-mail];
> > > Tony Krowiak <[e-mail]>; Pierre Morel
> > > <[e-mail]>; Halil Pasic <[e-mail]>
> > > Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
> > > sequence
> > >
> > > [vfio-ap folks: find a question regarding removal further down]
> > >
> > > On Wed, 8 May 2019 22:06:48 +0000
> > > Parav Pandit <[e-mail]> wrote:
> > >  
> > > > > -----Original Message-----
> > > > > From: Cornelia Huck <[e-mail]>
> > > > > Sent: Wednesday, May 8, 2019 12:10 PM
> > > > > To: Parav Pandit <[e-mail]>
> > > > > Cc: [e-mail]; [e-mail];
> > > > > [e-mail]; [e-mail];  
> > [e-mail]  
> > > > > Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
> > > > > sequence
> > > > >
> > > > > On Tue, 30 Apr 2019 17:49:35 -0500 Parav Pandit
> > > > > <[e-mail]> wrote:
> > > > >  
> > > > > > This patch addresses below two issues and prepares the code to
> > > > > > address 3rd issue listed below.
> > > > > >
> > > > > > 1. mdev device is placed on the mdev bus before it is created in
> > > > > > the vendor driver. Once a device is placed on the mdev bus
> > > > > > without creating its supporting underlying vendor device, mdev
> > > > > > driver's
> > > > > > probe()  
> > > > > gets triggered.  
> > > > > > However there isn't a stable mdev available to work on.
> > > > > >
> > > > > >    create_store()
> > > > > >      mdev_create_device()
> > > > > >        device_register()
> > > > > >           ...
> > > > > >          vfio_mdev_probe()
> > > > > >         [...]
> > > > > >         parent->ops->create()
> > > > > >           vfio_ap_mdev_create()
> > > > > >             mdev_set_drvdata(mdev, matrix_mdev);
> > > > > >             /* Valid pointer set above */
> > > > > >
> > > > > > Due to this way of initialization, mdev driver who want to use
> > > > > > the  
> > >
> > > s/want/wants/
> > >  
> > > > > > mdev, doesn't have a valid mdev to work on.
> > > > > >
> > > > > > 2. Current creation sequence is,
> > > > > >    parent->ops_create()
> > > > > >    groups_register()
> > > > > >
> > > > > > Remove sequence is,
> > > > > >    parent->ops->remove()
> > > > > >    groups_unregister()
> > > > > >
> > > > > > However, remove sequence should be exact mirror of creation  
> > > sequence.  
> > > > > > Once this is achieved, all users of the mdev will be terminated
> > > > > > first before removing underlying vendor device.
> > > > > > (Follow standard linux driver model).
> > > > > > At that point vendor's remove() ops shouldn't failed because
> > > > > > device is  
> > >
> > > s/failed/fail/
> > >  
> > > > > > taken off the bus that should terminate the users.  
> > >
> > > "because taking the device off the bus should terminate any usage" ?
> > >  
> > > > > >
> > > > > > 3. When remove operation fails, mdev sysfs removal attempts to
> > > > > > add the file back on already removed device. Following call
> > > > > > trace [1] is  
> > > observed.  
> > > > > >
> > > > > > [1] call trace:
> > > > > > kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327
> > > > > > sysfs_create_file_ns+0x7f/0x90
> > > > > > kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted
> > > > > > 5.1.0-rc6-vdevbus+ #6
> > > > > > kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS
> > > > > > 2.0b
> > > > > > 08/09/2016
> > > > > > kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
> > > > > > kernel: Call Trace:
> > > > > > kernel: remove_store+0xdc/0x100 [mdev]
> > > > > > kernel: kernfs_fop_write+0x113/0x1a0
> > > > > > kernel: vfs_write+0xad/0x1b0
> > > > > > kernel: ksys_write+0x5a/0xe0
> > > > > > kernel: do_syscall_64+0x5a/0x210
> > > > > > kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > > > > >
> > > > > > Therefore, mdev core is improved in following ways.
> > > > > >
> > > > > > 1. Before placing mdev devices on the bus, perform vendor
> > > > > > drivers creation which supports the mdev creation.  
> > >
> > > "invoke the vendor driver ->create callback" ?
> > >  
> > > > > > This ensures that mdev specific all necessary fields are
> > > > > > initialized  
> > >
> > > "that all necessary mdev specific fields are initialized" ?
> > >  
> > > > > > before a given mdev can be accessed by bus driver.
> > > > > > This follows standard Linux kernel bus and device model similar
> > > > > > to other widely used PCI bus.  
> > >
> > > "This follows standard practice on other Linux device model buses." ?
> > >  
> > > > > >
> > > > > > 2. During remove flow, first remove the device from the bus.
> > > > > > This ensures that any bus specific devices and data is cleared.
> > > > > > Once device is taken of the mdev bus, perform remove() of mdev
> > > > > > from  
> > >
> > > s/of/off/
> > >  
> > > > > > the vendor driver.
> > > > > >
> > > > > > 3. Linux core device model provides way to register and auto
> > > > > > unregister the device sysfs attribute groups at dev->groups.  
> > >
> > > "The driver core provides a way to automatically register and
> > > unregister sysfs attributes via dev->groups." ?
> > >  
> > > > > > Make use of this groups to let core create the groups and
> > > > > > simplify code to avoid explicit groups creation and removal.
> > > > > >
> > > > > > A below stack dump of a mdev device remove process also ensures
> > > > > > that vfio driver guards against device removal already in use.
> > > > > >
> > > > > >  cat /proc/21962/stack
> > > > > > [<0>] vfio_del_group_dev+0x216/0x3c0 [vfio] [<0>]
> > > > > > mdev_remove+0x21/0x40 [mdev] [<0>]
> > > > > > device_release_driver_internal+0xe8/0x1b0
> > > > > > [<0>] bus_remove_device+0xf9/0x170 [<0>] device_del+0x168/0x350
> > > > > > [<0>] mdev_device_remove_common+0x1d/0x50 [mdev] [<0>]
> > > > > > mdev_device_remove+0x8c/0xd0 [mdev] [<0>]  
> > > remove_store+0x71/0x90  
> > > > > > [mdev] [<0>] kernfs_fop_write+0x113/0x1a0 [<0>]
> > > > > > vfs_write+0xad/0x1b0 [<0>] ksys_write+0x5a/0xe0 [<0>]
> > > > > > do_syscall_64+0x5a/0x210 [<0>]
> > > > > > entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > > > > > [<0>] 0xffffffffffffffff
> > > > > >
> > > > > > This prepares the code to eliminate calling device_create_file()
> > > > > > in subsquent patch.  
> > >
> > > I find this stack dump and explanation more confusing than enlightening.
> > > Maybe just drop it?
> > >  
> > > > >
> > > > > I'm afraid I have a bit of a problem following this explanation,
> > > > > so let me try to summarize what the patch does to make sure that I
> > > > > understand it
> > > > > correctly:
> > > > >
> > > > > - Add the sysfs groups to device->groups so that the driver core deals
> > > > >   with proper registration/deregistration.
> > > > > - Split the device registration/deregistration sequence so that some
> > > > >   things can be done between initialization of the device and hooking
> > > > >   it up to the infrastructure respectively after deregistering it from
> > > > >   the infrastructure but before giving up our final reference. In
> > > > >   particular, this means invoking the ->create and ->remove callback in
> > > > >   those new windows. This gives the vendor driver an initialized mdev
> > > > >   device to work with during creation.
> > > > > - Don't allow ->remove to fail, as the device is already removed from
> > > > >   the infrastructure at that point in time.
> > > > >  
> > > > You got all the points pretty accurate.  
> > >
> > > Ok, good.
> > >  
> > > >  
> > > > > >
> > > > > > Signed-off-by: Parav Pandit <[e-mail]>
> > > > > > ---
> > > > > >  drivers/vfio/mdev/mdev_core.c    | 94 +++++++++-----------------------
> > > > > >  drivers/vfio/mdev/mdev_private.h |  2 +-
> > > > > >  drivers/vfio/mdev/mdev_sysfs.c   |  2 +-
> > > > > >  3 files changed, 27 insertions(+), 71 deletions(-)  
> > > > >
> > > > > (...)  
> > >  
> > > > > > @@ -373,16 +330,15 @@ int mdev_device_remove(struct device  
> > *dev,  
> > > > > bool force_remove)  
> > > > > >  	mutex_unlock(&mdev_list_lock);
> > > > > >
> > > > > >  	type = to_mdev_type(mdev->type_kobj);
> > > > > > +	mdev_remove_sysfs_files(dev, type);
> > > > > > +	device_del(&mdev->dev);
> > > > > >  	parent = mdev->parent;
> > > > > > +	ret = parent->ops->remove(mdev);
> > > > > > +	if (ret)
> > > > > > +		dev_err(&mdev->dev, "Remove failed: err=%d\n",  
> > ret);  
> > > > >
> > > > > I think carrying on with removal regardless of the return code of
> > > > > the  
> > > > > ->remove callback makes sense, as it simply matches usual practice.  
> > > > > However, are we sure that every vendor driver works well with that?
> > > > > I think it should, as removal from bus unregistration (vs. from
> > > > > the sysfs
> > > > > file) was always something it could not veto, but have you looked
> > > > > at the individual drivers?
> > > > >  
> > > > I looked at following drivers a little while back.
> > > > Looked again now.
> > > >
> > > > drivers/gpu/drm/i915/gvt/kvmgt.c which clears the handle valid in  
> > > intel_vgpu_release(), which should finish first before remove() is invoked.  
> > > >
> > > > s390 vfio_ccw_mdev_remove() driver drivers/s390/cio/vfio_ccw_ops.c  
> > > remove() always returns 0.  
> > > > s39 crypo fails the remove() once vfio_ap_mdev_release marks kvm
> > > > null,  
> > > which should finish before remove() is invoked.
> > >
> > > That one is giving me a bit of a headache (the ->kvm reference is
> > > supposed to keep us from detaching while a vm is running), so let's cc:
> > > the vfio-ap maintainers to see whether they have any concerns.
> > >  
> > I probably wrote wrongly.
> > vfio_ap_mdev_remove() fails if the VM is already running (i.e.
> > vfio_ap_mdev_release() is not yet called).
> > 
> > And if VM is running it guarded by the vfio_mdev driver which is the one
> > who binds to the device mdev device.
> > That is why I shown the above stack trace in the commit log, indicating that
> > vfio driver is guarding it.
> >   
> > > > samples/vfio-mdev/mbochs.c mbochs_remove() always returns 0.
> > > >  
> > > > > >
> > > > > > -	ret = mdev_device_remove_ops(mdev, force_remove);
> > > > > > -	if (ret) {
> > > > > > -		mdev->active = true;
> > > > > > -		return ret;
> > > > > > -	}
> > > > > > -
> > > > > > -	mdev_remove_sysfs_files(dev, type);
> > > > > > -	device_unregister(dev);
> > > > > > +	/* Balances with device_initialize() */
> > > > > > +	put_device(&mdev->dev);
> > > > > >  	mdev_put_parent(parent);
> > > > > >
> > > > > >  	return 0;  
> > > > >
> > > > > I think that looks sane in general, but the commit message might
> > > > > benefit from tweaking.  
> > > > Part of your description is more crisp than my commit message, I can  
> > > probably take snippet from it to improve?  
> > > > Or any specific entries in commit message that I should address?  
> > >
> > > I have added some comments inline (mostly some wording tweaks).
> > >
> > > Feel free to take anything from my summary as well.  
> 
> I want to send v3 addressing commit log comment and take updated description from Cornelia, if this 3 patches looks reasonable enough.
> What do you think?

The kref removal in the last patch still makes me uncomfortable, but I
can't really find a reason to keep it or see any problems with the way
you're using refcount either.  It's probably good to see a new version
of the series regardless.  Thanks,

Alex

Parav Pandit May 15, 2019, 8:42 p.m. UTC | #10

> -----Original Message-----
> From: Alex Williamson <[e-mail]>
> Sent: Tuesday, May 14, 2019 5:20 PM
> To: Parav Pandit <[e-mail]>
> Cc: Cornelia Huck <[e-mail]>; [e-mail]; linux-
> [e-mail]; [e-mail]; [e-mail]; Tony
> Krowiak <[e-mail]>; Pierre Morel <[e-mail]>;
> Halil Pasic <[e-mail]>
> Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
> sequence
> 
> On Tue, 14 May 2019 20:34:12 +0000
> Parav Pandit <[e-mail]> wrote:
> 
> > Hi Alex, Cornelia,
> >
> >
> > > -----Original Message-----
> > > From: [e-mail] <linux-kernel-
> > > [e-mail]> On Behalf Of Parav Pandit
> > > Sent: Thursday, May 9, 2019 2:20 PM
> > > To: Cornelia Huck <[e-mail]>
> > > Cc: [e-mail]; [e-mail];
> > > [e-mail]; [e-mail]; [e-mail];
> > > Tony Krowiak <[e-mail]>; Pierre Morel
> > > <[e-mail]>; Halil Pasic <[e-mail]>
> > > Subject: RE: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
> > > sequence
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Cornelia Huck <[e-mail]>
> > > > Sent: Thursday, May 9, 2019 4:06 AM
> > > > To: Parav Pandit <[e-mail]>
> > > > Cc: [e-mail]; [e-mail];
> > > > [e-mail]; [e-mail];
> [e-mail];
> > > > Tony Krowiak <[e-mail]>; Pierre Morel
> > > > <[e-mail]>; Halil Pasic <[e-mail]>
> > > > Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the create/remove
> > > > sequence
> > > >
> > > > [vfio-ap folks: find a question regarding removal further down]
> > > >
> > > > On Wed, 8 May 2019 22:06:48 +0000
> > > > Parav Pandit <[e-mail]> wrote:
> > > >
> > > > > > -----Original Message-----
> > > > > > From: Cornelia Huck <[e-mail]>
> > > > > > Sent: Wednesday, May 8, 2019 12:10 PM
> > > > > > To: Parav Pandit <[e-mail]>
> > > > > > Cc: [e-mail]; [e-mail];
> > > > > > [e-mail]; [e-mail];
> > > [e-mail]
> > > > > > Subject: Re: [PATCHv2 08/10] vfio/mdev: Improve the
> > > > > > create/remove sequence
> > > > > >
> > > > > > On Tue, 30 Apr 2019 17:49:35 -0500 Parav Pandit
> > > > > > <[e-mail]> wrote:
> > > > > >
> > > > > > > This patch addresses below two issues and prepares the code
> > > > > > > to address 3rd issue listed below.
> > > > > > >
> > > > > > > 1. mdev device is placed on the mdev bus before it is
> > > > > > > created in the vendor driver. Once a device is placed on the
> > > > > > > mdev bus without creating its supporting underlying vendor
> > > > > > > device, mdev driver's
> > > > > > > probe()
> > > > > > gets triggered.
> > > > > > > However there isn't a stable mdev available to work on.
> > > > > > >
> > > > > > >    create_store()
> > > > > > >      mdev_create_device()
> > > > > > >        device_register()
> > > > > > >           ...
> > > > > > >          vfio_mdev_probe()
> > > > > > >         [...]
> > > > > > >         parent->ops->create()
> > > > > > >           vfio_ap_mdev_create()
> > > > > > >             mdev_set_drvdata(mdev, matrix_mdev);
> > > > > > >             /* Valid pointer set above */
> > > > > > >
> > > > > > > Due to this way of initialization, mdev driver who want to
> > > > > > > use the
> > > >
> > > > s/want/wants/
> > > >
> > > > > > > mdev, doesn't have a valid mdev to work on.
> > > > > > >
> > > > > > > 2. Current creation sequence is,
> > > > > > >    parent->ops_create()
> > > > > > >    groups_register()
> > > > > > >
> > > > > > > Remove sequence is,
> > > > > > >    parent->ops->remove()
> > > > > > >    groups_unregister()
> > > > > > >
> > > > > > > However, remove sequence should be exact mirror of creation
> > > > sequence.
> > > > > > > Once this is achieved, all users of the mdev will be
> > > > > > > terminated first before removing underlying vendor device.
> > > > > > > (Follow standard linux driver model).
> > > > > > > At that point vendor's remove() ops shouldn't failed because
> > > > > > > device is
> > > >
> > > > s/failed/fail/
> > > >
> > > > > > > taken off the bus that should terminate the users.
> > > >
> > > > "because taking the device off the bus should terminate any usage" ?
> > > >
> > > > > > >
> > > > > > > 3. When remove operation fails, mdev sysfs removal attempts
> > > > > > > to add the file back on already removed device. Following
> > > > > > > call trace [1] is
> > > > observed.
> > > > > > >
> > > > > > > [1] call trace:
> > > > > > > kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327
> > > > > > > sysfs_create_file_ns+0x7f/0x90
> > > > > > > kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not
> > > > > > > tainted 5.1.0-rc6-vdevbus+ #6
> > > > > > > kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+,
> > > > > > > BIOS 2.0b
> > > > > > > 08/09/2016
> > > > > > > kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90
> > > > > > > kernel: Call Trace:
> > > > > > > kernel: remove_store+0xdc/0x100 [mdev]
> > > > > > > kernel: kernfs_fop_write+0x113/0x1a0
> > > > > > > kernel: vfs_write+0xad/0x1b0
> > > > > > > kernel: ksys_write+0x5a/0xe0
> > > > > > > kernel: do_syscall_64+0x5a/0x210
> > > > > > > kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > > > > > >
> > > > > > > Therefore, mdev core is improved in following ways.
> > > > > > >
> > > > > > > 1. Before placing mdev devices on the bus, perform vendor
> > > > > > > drivers creation which supports the mdev creation.
> > > >
> > > > "invoke the vendor driver ->create callback" ?
> > > >
> > > > > > > This ensures that mdev specific all necessary fields are
> > > > > > > initialized
> > > >
> > > > "that all necessary mdev specific fields are initialized" ?
> > > >
> > > > > > > before a given mdev can be accessed by bus driver.
> > > > > > > This follows standard Linux kernel bus and device model
> > > > > > > similar to other widely used PCI bus.
> > > >
> > > > "This follows standard practice on other Linux device model buses." ?
> > > >
> > > > > > >
> > > > > > > 2. During remove flow, first remove the device from the bus.
> > > > > > > This ensures that any bus specific devices and data is cleared.
> > > > > > > Once device is taken of the mdev bus, perform remove() of
> > > > > > > mdev from
> > > >
> > > > s/of/off/
> > > >
> > > > > > > the vendor driver.
> > > > > > >
> > > > > > > 3. Linux core device model provides way to register and auto
> > > > > > > unregister the device sysfs attribute groups at dev->groups.
> > > >
> > > > "The driver core provides a way to automatically register and
> > > > unregister sysfs attributes via dev->groups." ?
> > > >
> > > > > > > Make use of this groups to let core create the groups and
> > > > > > > simplify code to avoid explicit groups creation and removal.
> > > > > > >
> > > > > > > A below stack dump of a mdev device remove process also
> > > > > > > ensures that vfio driver guards against device removal already in
> use.
> > > > > > >
> > > > > > >  cat /proc/21962/stack
> > > > > > > [<0>] vfio_del_group_dev+0x216/0x3c0 [vfio] [<0>]
> > > > > > > mdev_remove+0x21/0x40 [mdev] [<0>]
> > > > > > > device_release_driver_internal+0xe8/0x1b0
> > > > > > > [<0>] bus_remove_device+0xf9/0x170 [<0>]
> > > > > > > device_del+0x168/0x350 [<0>]
> > > > > > > mdev_device_remove_common+0x1d/0x50 [mdev] [<0>]
> > > > > > > mdev_device_remove+0x8c/0xd0 [mdev] [<0>]
> > > > remove_store+0x71/0x90
> > > > > > > [mdev] [<0>] kernfs_fop_write+0x113/0x1a0 [<0>]
> > > > > > > vfs_write+0xad/0x1b0 [<0>] ksys_write+0x5a/0xe0 [<0>]
> > > > > > > do_syscall_64+0x5a/0x210 [<0>]
> > > > > > > entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > > > > > > [<0>] 0xffffffffffffffff
> > > > > > >
> > > > > > > This prepares the code to eliminate calling
> > > > > > > device_create_file() in subsquent patch.
> > > >
> > > > I find this stack dump and explanation more confusing than
> enlightening.
> > > > Maybe just drop it?
> > > >
> > > > > >
> > > > > > I'm afraid I have a bit of a problem following this
> > > > > > explanation, so let me try to summarize what the patch does to
> > > > > > make sure that I understand it
> > > > > > correctly:
> > > > > >
> > > > > > - Add the sysfs groups to device->groups so that the driver core
> deals
> > > > > >   with proper registration/deregistration.
> > > > > > - Split the device registration/deregistration sequence so that some
> > > > > >   things can be done between initialization of the device and
> hooking
> > > > > >   it up to the infrastructure respectively after deregistering it from
> > > > > >   the infrastructure but before giving up our final reference. In
> > > > > >   particular, this means invoking the ->create and ->remove callback
> in
> > > > > >   those new windows. This gives the vendor driver an initialized
> mdev
> > > > > >   device to work with during creation.
> > > > > > - Don't allow ->remove to fail, as the device is already removed
> from
> > > > > >   the infrastructure at that point in time.
> > > > > >
> > > > > You got all the points pretty accurate.
> > > >
> > > > Ok, good.
> > > >
> > > > >
> > > > > > >
> > > > > > > Signed-off-by: Parav Pandit <[e-mail]>
> > > > > > > ---
> > > > > > >  drivers/vfio/mdev/mdev_core.c    | 94 +++++++++-----------------------
> > > > > > >  drivers/vfio/mdev/mdev_private.h |  2 +-
> > > > > > >  drivers/vfio/mdev/mdev_sysfs.c   |  2 +-
> > > > > > >  3 files changed, 27 insertions(+), 71 deletions(-)
> > > > > >
> > > > > > (...)
> > > >
> > > > > > > @@ -373,16 +330,15 @@ int mdev_device_remove(struct device
> > > *dev,
> > > > > > bool force_remove)
> > > > > > >  	mutex_unlock(&mdev_list_lock);
> > > > > > >
> > > > > > >  	type = to_mdev_type(mdev->type_kobj);
> > > > > > > +	mdev_remove_sysfs_files(dev, type);
> > > > > > > +	device_del(&mdev->dev);
> > > > > > >  	parent = mdev->parent;
> > > > > > > +	ret = parent->ops->remove(mdev);
> > > > > > > +	if (ret)
> > > > > > > +		dev_err(&mdev->dev, "Remove failed: err=%d\n",
> > > ret);
> > > > > >
> > > > > > I think carrying on with removal regardless of the return code
> > > > > > of the
> > > > > > ->remove callback makes sense, as it simply matches usual practice.
> > > > > > However, are we sure that every vendor driver works well with
> that?
> > > > > > I think it should, as removal from bus unregistration (vs.
> > > > > > from the sysfs
> > > > > > file) was always something it could not veto, but have you
> > > > > > looked at the individual drivers?
> > > > > >
> > > > > I looked at following drivers a little while back.
> > > > > Looked again now.
> > > > >
> > > > > drivers/gpu/drm/i915/gvt/kvmgt.c which clears the handle valid
> > > > > in
> > > > intel_vgpu_release(), which should finish first before remove() is
> invoked.
> > > > >
> > > > > s390 vfio_ccw_mdev_remove() driver
> > > > > drivers/s390/cio/vfio_ccw_ops.c
> > > > remove() always returns 0.
> > > > > s39 crypo fails the remove() once vfio_ap_mdev_release marks kvm
> > > > > null,
> > > > which should finish before remove() is invoked.
> > > >
> > > > That one is giving me a bit of a headache (the ->kvm reference is
> > > > supposed to keep us from detaching while a vm is running), so let's cc:
> > > > the vfio-ap maintainers to see whether they have any concerns.
> > > >
> > > I probably wrote wrongly.
> > > vfio_ap_mdev_remove() fails if the VM is already running (i.e.
> > > vfio_ap_mdev_release() is not yet called).
> > >
> > > And if VM is running it guarded by the vfio_mdev driver which is the
> > > one who binds to the device mdev device.
> > > That is why I shown the above stack trace in the commit log,
> > > indicating that vfio driver is guarding it.
> > >
> > > > > samples/vfio-mdev/mbochs.c mbochs_remove() always returns 0.
> > > > >
> > > > > > >
> > > > > > > -	ret = mdev_device_remove_ops(mdev, force_remove);
> > > > > > > -	if (ret) {
> > > > > > > -		mdev->active = true;
> > > > > > > -		return ret;
> > > > > > > -	}
> > > > > > > -
> > > > > > > -	mdev_remove_sysfs_files(dev, type);
> > > > > > > -	device_unregister(dev);
> > > > > > > +	/* Balances with device_initialize() */
> > > > > > > +	put_device(&mdev->dev);
> > > > > > >  	mdev_put_parent(parent);
> > > > > > >
> > > > > > >  	return 0;
> > > > > >
> > > > > > I think that looks sane in general, but the commit message
> > > > > > might benefit from tweaking.
> > > > > Part of your description is more crisp than my commit message, I
> > > > > can
> > > > probably take snippet from it to improve?
> > > > > Or any specific entries in commit message that I should address?
> > > >
> > > > I have added some comments inline (mostly some wording tweaks).
> > > >
> > > > Feel free to take anything from my summary as well.
> >
> > I want to send v3 addressing commit log comment and take updated
> description from Cornelia, if this 3 patches looks reasonable enough.
> > What do you think?
> 
> The kref removal in the last patch still makes me uncomfortable, but I can't
> really find a reason to keep it or see any problems with the way you're using
> refcount either.  It's probably good to see a new version of the series
> regardless.  Thanks,
> 
Ok. thanks. Right. I will send these 3 patches with updated cover letter and commit log enhancements from Cornelia.

> Alex

Patch

diff --git a/drivers/vfio/mdev/mdev_core.c b/drivers/vfio/mdev/mdev_core.c
index 1040a4a2dcbc..2b98da2ee361 100644
--- a/drivers/vfio/mdev/mdev_core.c
+++ b/drivers/vfio/mdev/mdev_core.c
@@ -102,55 +102,10 @@  static void mdev_put_parent(struct mdev_parent *parent)
 		kref_put(&parent->ref, mdev_release_parent);
 }
 
-static int mdev_device_create_ops(struct kobject *kobj,
-				  struct mdev_device *mdev)
-{
-	struct mdev_parent *parent = mdev->parent;
-	int ret;
-
-	ret = parent->ops->create(kobj, mdev);
-	if (ret)
-		return ret;
-
-	ret = sysfs_create_groups(&mdev->dev.kobj,
-				  parent->ops->mdev_attr_groups);
-	if (ret)
-		parent->ops->remove(mdev);
-
-	return ret;
-}
-
-/*
- * mdev_device_remove_ops gets called from sysfs's 'remove' and when parent
- * device is being unregistered from mdev device framework.
- * - 'force_remove' is set to 'false' when called from sysfs's 'remove' which
- *   indicates that if the mdev device is active, used by VMM or userspace
- *   application, vendor driver could return error then don't remove the device.
- * - 'force_remove' is set to 'true' when called from mdev_unregister_device()
- *   which indicate that parent device is being removed from mdev device
- *   framework so remove mdev device forcefully.
- */
-static int mdev_device_remove_ops(struct mdev_device *mdev, bool force_remove)
-{
-	struct mdev_parent *parent = mdev->parent;
-	int ret;
-
-	/*
-	 * Vendor driver can return error if VMM or userspace application is
-	 * using this mdev device.
-	 */
-	ret = parent->ops->remove(mdev);
-	if (ret && !force_remove)
-		return ret;
-
-	sysfs_remove_groups(&mdev->dev.kobj, parent->ops->mdev_attr_groups);
-	return 0;
-}
-
 static int mdev_device_remove_cb(struct device *dev, void *data)
 {
 	if (dev_is_mdev(dev))
-		mdev_device_remove(dev, true);
+		mdev_device_remove(dev);
 
 	return 0;
 }
@@ -310,41 +265,43 @@  int mdev_device_create(struct kobject *kobj,
 
 	mdev->parent = parent;
 
+	device_initialize(&mdev->dev);
 	mdev->dev.parent  = dev;
 	mdev->dev.bus     = &mdev_bus_type;
 	mdev->dev.release = mdev_device_release;
 	dev_set_name(&mdev->dev, "%pUl", uuid);
+	mdev->dev.groups = parent->ops->mdev_attr_groups;
+	mdev->type_kobj = kobj;
 
-	ret = device_register(&mdev->dev);
-	if (ret) {
-		put_device(&mdev->dev);
-		goto mdev_fail;
-	}
+	ret = parent->ops->create(kobj, mdev);
+	if (ret)
+		goto ops_create_fail;
 
-	ret = mdev_device_create_ops(kobj, mdev);
+	ret = device_add(&mdev->dev);
 	if (ret)
-		goto create_fail;
+		goto add_fail;
 
 	ret = mdev_create_sysfs_files(&mdev->dev, type);
-	if (ret) {
-		mdev_device_remove_ops(mdev, true);
-		goto create_fail;
-	}
+	if (ret)
+		goto sysfs_fail;
 
-	mdev->type_kobj = kobj;
 	mdev->active = true;
 	dev_dbg(&mdev->dev, "MDEV: created\n");
 
 	return 0;
 
-create_fail:
-	device_unregister(&mdev->dev);
+sysfs_fail:
+	device_del(&mdev->dev);
+add_fail:
+	parent->ops->remove(mdev);
+ops_create_fail:
+	put_device(&mdev->dev);
 mdev_fail:
 	mdev_put_parent(parent);
 	return ret;
 }
 
-int mdev_device_remove(struct device *dev, bool force_remove)
+int mdev_device_remove(struct device *dev)
 {
 	struct mdev_device *mdev, *tmp;
 	struct mdev_parent *parent;
@@ -373,16 +330,15 @@  int mdev_device_remove(struct device *dev, bool force_remove)
 	mutex_unlock(&mdev_list_lock);
 
 	type = to_mdev_type(mdev->type_kobj);
+	mdev_remove_sysfs_files(dev, type);
+	device_del(&mdev->dev);
 	parent = mdev->parent;
+	ret = parent->ops->remove(mdev);
+	if (ret)
+		dev_err(&mdev->dev, "Remove failed: err=%d\n", ret);
 
-	ret = mdev_device_remove_ops(mdev, force_remove);
-	if (ret) {
-		mdev->active = true;
-		return ret;
-	}
-
-	mdev_remove_sysfs_files(dev, type);
-	device_unregister(dev);
+	/* Balances with device_initialize() */
+	put_device(&mdev->dev);
 	mdev_put_parent(parent);
 
 	return 0;
diff --git a/drivers/vfio/mdev/mdev_private.h b/drivers/vfio/mdev/mdev_private.h
index ddcf9c72bd8a..067dc5d8c5de 100644
--- a/drivers/vfio/mdev/mdev_private.h
+++ b/drivers/vfio/mdev/mdev_private.h
@@ -59,6 +59,6 @@  void mdev_remove_sysfs_files(struct device *dev, struct mdev_type *type);
 
 int  mdev_device_create(struct kobject *kobj,
 			struct device *dev, const guid_t *uuid);
-int  mdev_device_remove(struct device *dev, bool force_remove);
+int  mdev_device_remove(struct device *dev);
 
 #endif /* MDEV_PRIVATE_H */
diff --git a/drivers/vfio/mdev/mdev_sysfs.c b/drivers/vfio/mdev/mdev_sysfs.c
index cbf94b8165ea..9f774b91d275 100644
--- a/drivers/vfio/mdev/mdev_sysfs.c
+++ b/drivers/vfio/mdev/mdev_sysfs.c
@@ -236,7 +236,7 @@  static ssize_t remove_store(struct device *dev, struct device_attribute *attr,
 	if (val && device_remove_file_self(dev, attr)) {
 		int ret;
 
-		ret = mdev_device_remove(dev, false);
+		ret = mdev_device_remove(dev);
 		if (ret) {
 			device_create_file(dev, attr);
 			return ret;