| Message ID | 20210210194031.7422-2-bostroesser@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | uio and tcmu: Fix memory leak in tcmu by adding new uio feature | expand |
On Wed, Feb 10, 2021 at 08:40:30PM +0100, Bodo Stroesser wrote: > If uio_unregister_device() is called while userspace daemon > still holds the uio device open or mmap'ed, uio will not call > uio_info->release() on later close / munmap. > > At least one user of uio (tcmu) should not free resources (pages > allocated by tcmu which are mmap'ed to userspace) while uio > device still is open, because that could cause userspace daemon > to be killed by SIGSEGV or SIGBUS. Therefore tcmu frees the > pages only after it called uio_unregister_device _and_ the device > was closed. > So, uio not calling uio_info->release causes trouble. > tcmu currently leaks memory in that case. > > Just waiting for userspace daemon to exit before calling > uio_unregister_device I think is not the right solution, because > daemon would not become aware of kernel code wanting to destroy > the uio device. > After uio_unregister_device was called, reading or writing the > uio device returns -EIO, which normally results in daemon exit. > > This patch adds new callback pointer 'late_release' to struct > uio_info. If uio user sets this callback, it will be called by > uio if userspace closes / munmaps the device after > uio_unregister_device was executed. > > That way we can use uio_unregister_device() to notify userspace > that we are going to destroy the device, but still get a call > to late_release when uio device is finally closed. > > Signed-off-by: Bodo Stroesser <bostroesser@gmail.com> > --- > Documentation/driver-api/uio-howto.rst | 10 ++++++++++ > drivers/uio/uio.c | 4 ++++ > include/linux/uio_driver.h | 4 ++++ > 3 files changed, 18 insertions(+) > > diff --git a/Documentation/driver-api/uio-howto.rst b/Documentation/driver-api/uio-howto.rst > index 907ffa3b38f5..a2d57a7d623a 100644 > --- a/Documentation/driver-api/uio-howto.rst > +++ b/Documentation/driver-api/uio-howto.rst > @@ -265,6 +265,16 @@ the members are required, others are optional. > function. The parameter ``irq_on`` will be 0 to disable interrupts > and 1 to enable them. > > +- ``int (*late_release)(struct uio_info *info, struct inode *inode)``: > + Optional. If you define your own :c:func:`open()`, you will > + in certain cases also want a custom :c:func:`late_release()` > + function. If uio device is unregistered - by calling > + :c:func:`uio_unregister_device()` - while it is open or mmap'ed by > + userspace, the custom :c:func:`release()` function will not be > + called when userspace later closes the device. An optionally > + specified :c:func:`late_release()` function will be called in that > + situation. > + > Usually, your device will have one or more memory regions that can be > mapped to user space. For each region, you have to set up a > ``struct uio_mem`` in the ``mem[]`` array. Here's a description of the > diff --git a/drivers/uio/uio.c b/drivers/uio/uio.c > index ea96e319c8a0..0b2636f8d373 100644 > --- a/drivers/uio/uio.c > +++ b/drivers/uio/uio.c > @@ -532,6 +532,8 @@ static int uio_release(struct inode *inode, struct file *filep) > mutex_lock(&idev->info_lock); > if (idev->info && idev->info->release) > ret = idev->info->release(idev->info, inode); > + else if (idev->late_info && idev->late_info->late_release) > + ret = idev->late_info->late_release(idev->late_info, inode); > mutex_unlock(&idev->info_lock); Why can't release() be called here? Why doesn't your driver define a release() if it cares about this information? Why do we need 2 different callbacks that fire at exactly the same time? This feels really wrong. greg k-h
On 10.02.21 20:47, Greg Kroah-Hartman wrote: > On Wed, Feb 10, 2021 at 08:40:30PM +0100, Bodo Stroesser wrote: >> If uio_unregister_device() is called while userspace daemon >> still holds the uio device open or mmap'ed, uio will not call >> uio_info->release() on later close / munmap. >> >> At least one user of uio (tcmu) should not free resources (pages >> allocated by tcmu which are mmap'ed to userspace) while uio >> device still is open, because that could cause userspace daemon >> to be killed by SIGSEGV or SIGBUS. Therefore tcmu frees the >> pages only after it called uio_unregister_device _and_ the device >> was closed. >> So, uio not calling uio_info->release causes trouble. >> tcmu currently leaks memory in that case. >> >> Just waiting for userspace daemon to exit before calling >> uio_unregister_device I think is not the right solution, because >> daemon would not become aware of kernel code wanting to destroy >> the uio device. >> After uio_unregister_device was called, reading or writing the >> uio device returns -EIO, which normally results in daemon exit. >> >> This patch adds new callback pointer 'late_release' to struct >> uio_info. If uio user sets this callback, it will be called by >> uio if userspace closes / munmaps the device after >> uio_unregister_device was executed. >> >> That way we can use uio_unregister_device() to notify userspace >> that we are going to destroy the device, but still get a call >> to late_release when uio device is finally closed. >> >> Signed-off-by: Bodo Stroesser <bostroesser@gmail.com> >> --- >> Documentation/driver-api/uio-howto.rst | 10 ++++++++++ >> drivers/uio/uio.c | 4 ++++ >> include/linux/uio_driver.h | 4 ++++ >> 3 files changed, 18 insertions(+) >> >> diff --git a/Documentation/driver-api/uio-howto.rst b/Documentation/driver-api/uio-howto.rst >> index 907ffa3b38f5..a2d57a7d623a 100644 >> --- a/Documentation/driver-api/uio-howto.rst >> +++ b/Documentation/driver-api/uio-howto.rst >> @@ -265,6 +265,16 @@ the members are required, others are optional. >> function. The parameter ``irq_on`` will be 0 to disable interrupts >> and 1 to enable them. >> >> +- ``int (*late_release)(struct uio_info *info, struct inode *inode)``: >> + Optional. If you define your own :c:func:`open()`, you will >> + in certain cases also want a custom :c:func:`late_release()` >> + function. If uio device is unregistered - by calling >> + :c:func:`uio_unregister_device()` - while it is open or mmap'ed by >> + userspace, the custom :c:func:`release()` function will not be >> + called when userspace later closes the device. An optionally >> + specified :c:func:`late_release()` function will be called in that >> + situation. >> + >> Usually, your device will have one or more memory regions that can be >> mapped to user space. For each region, you have to set up a >> ``struct uio_mem`` in the ``mem[]`` array. Here's a description of the >> diff --git a/drivers/uio/uio.c b/drivers/uio/uio.c >> index ea96e319c8a0..0b2636f8d373 100644 >> --- a/drivers/uio/uio.c >> +++ b/drivers/uio/uio.c >> @@ -532,6 +532,8 @@ static int uio_release(struct inode *inode, struct file *filep) >> mutex_lock(&idev->info_lock); >> if (idev->info && idev->info->release) >> ret = idev->info->release(idev->info, inode); >> + else if (idev->late_info && idev->late_info->late_release) >> + ret = idev->late_info->late_release(idev->late_info, inode); >> mutex_unlock(&idev->info_lock); > > Why can't release() be called here? Why doesn't your driver define a > release() if it cares about this information? Why do we need 2 > different callbacks that fire at exactly the same time? > > This feels really wrong. > > greg k-h > tcmu has a release callback. But uio can't call it after uio_unregister_device was executed, because in uio_unregister_device uio sets the uio_device::info to NULL. So, uio would never call both callbacks for the same release action, but would call release before uio_unregister_device is executed, and late_release after that. Of course it would be good for tcmu if uio would call uio_info:release even after uio_unregister_device, but changing this AFAICS could cause trouble in other drivers using uio. Bodo after uio_unregister_device.
On Wed, Feb 10, 2021 at 08:57:11PM +0100, Bodo Stroesser wrote: > On 10.02.21 20:47, Greg Kroah-Hartman wrote: > > On Wed, Feb 10, 2021 at 08:40:30PM +0100, Bodo Stroesser wrote: > > > If uio_unregister_device() is called while userspace daemon > > > still holds the uio device open or mmap'ed, uio will not call > > > uio_info->release() on later close / munmap. > > > > > > At least one user of uio (tcmu) should not free resources (pages > > > allocated by tcmu which are mmap'ed to userspace) while uio > > > device still is open, because that could cause userspace daemon > > > to be killed by SIGSEGV or SIGBUS. Therefore tcmu frees the > > > pages only after it called uio_unregister_device _and_ the device > > > was closed. > > > So, uio not calling uio_info->release causes trouble. > > > tcmu currently leaks memory in that case. > > > > > > Just waiting for userspace daemon to exit before calling > > > uio_unregister_device I think is not the right solution, because > > > daemon would not become aware of kernel code wanting to destroy > > > the uio device. > > > After uio_unregister_device was called, reading or writing the > > > uio device returns -EIO, which normally results in daemon exit. > > > > > > This patch adds new callback pointer 'late_release' to struct > > > uio_info. If uio user sets this callback, it will be called by > > > uio if userspace closes / munmaps the device after > > > uio_unregister_device was executed. > > > > > > That way we can use uio_unregister_device() to notify userspace > > > that we are going to destroy the device, but still get a call > > > to late_release when uio device is finally closed. > > > > > > Signed-off-by: Bodo Stroesser <bostroesser@gmail.com> > > > --- > > > Documentation/driver-api/uio-howto.rst | 10 ++++++++++ > > > drivers/uio/uio.c | 4 ++++ > > > include/linux/uio_driver.h | 4 ++++ > > > 3 files changed, 18 insertions(+) > > > > > > diff --git a/Documentation/driver-api/uio-howto.rst b/Documentation/driver-api/uio-howto.rst > > > index 907ffa3b38f5..a2d57a7d623a 100644 > > > --- a/Documentation/driver-api/uio-howto.rst > > > +++ b/Documentation/driver-api/uio-howto.rst > > > @@ -265,6 +265,16 @@ the members are required, others are optional. > > > function. The parameter ``irq_on`` will be 0 to disable interrupts > > > and 1 to enable them. > > > +- ``int (*late_release)(struct uio_info *info, struct inode *inode)``: > > > + Optional. If you define your own :c:func:`open()`, you will > > > + in certain cases also want a custom :c:func:`late_release()` > > > + function. If uio device is unregistered - by calling > > > + :c:func:`uio_unregister_device()` - while it is open or mmap'ed by > > > + userspace, the custom :c:func:`release()` function will not be > > > + called when userspace later closes the device. An optionally > > > + specified :c:func:`late_release()` function will be called in that > > > + situation. > > > + > > > Usually, your device will have one or more memory regions that can be > > > mapped to user space. For each region, you have to set up a > > > ``struct uio_mem`` in the ``mem[]`` array. Here's a description of the > > > diff --git a/drivers/uio/uio.c b/drivers/uio/uio.c > > > index ea96e319c8a0..0b2636f8d373 100644 > > > --- a/drivers/uio/uio.c > > > +++ b/drivers/uio/uio.c > > > @@ -532,6 +532,8 @@ static int uio_release(struct inode *inode, struct file *filep) > > > mutex_lock(&idev->info_lock); > > > if (idev->info && idev->info->release) > > > ret = idev->info->release(idev->info, inode); > > > + else if (idev->late_info && idev->late_info->late_release) > > > + ret = idev->late_info->late_release(idev->late_info, inode); > > > mutex_unlock(&idev->info_lock); > > > > Why can't release() be called here? Why doesn't your driver define a > > release() if it cares about this information? Why do we need 2 > > different callbacks that fire at exactly the same time? > > > > This feels really wrong. > > > > greg k-h > > > > tcmu has a release callback. But uio can't call it after > uio_unregister_device was executed, because in uio_unregister_device > uio sets the uio_device::info to NULL. As it should because the driver could then be gone. It should NEVER call back into it again. > So, uio would never call both callbacks for the same release action, > but would call release before uio_unregister_device is executed, and > late_release after that. That's not ok. > Of course it would be good for tcmu if uio would call uio_info:release even > after uio_unregister_device, but changing this AFAICS could cause > trouble in other drivers using uio. You are confusing two different lifetime rules here it seems. One is the char device and one is the struct device. They work independently as different users affect them. So if one is removed from the system, do not try to keep a callback to it, otherwise you will crash. And why is scsi using the uio driver in the first place? That feels really odd to me. Why not just make a "real" driver if you want to somehow tie these two lifetimes together? thanks, greg k-h
On 11.02.21 07:51, Greg Kroah-Hartman wrote: > On Wed, Feb 10, 2021 at 08:57:11PM +0100, Bodo Stroesser wrote: >> On 10.02.21 20:47, Greg Kroah-Hartman wrote: >>> On Wed, Feb 10, 2021 at 08:40:30PM +0100, Bodo Stroesser wrote: >>>> If uio_unregister_device() is called while userspace daemon >>>> still holds the uio device open or mmap'ed, uio will not call >>>> uio_info->release() on later close / munmap. >>>> >>>> At least one user of uio (tcmu) should not free resources (pages >>>> allocated by tcmu which are mmap'ed to userspace) while uio >>>> device still is open, because that could cause userspace daemon >>>> to be killed by SIGSEGV or SIGBUS. Therefore tcmu frees the >>>> pages only after it called uio_unregister_device _and_ the device >>>> was closed. >>>> So, uio not calling uio_info->release causes trouble. >>>> tcmu currently leaks memory in that case. >>>> >>>> Just waiting for userspace daemon to exit before calling >>>> uio_unregister_device I think is not the right solution, because >>>> daemon would not become aware of kernel code wanting to destroy >>>> the uio device. >>>> After uio_unregister_device was called, reading or writing the >>>> uio device returns -EIO, which normally results in daemon exit. >>>> >>>> This patch adds new callback pointer 'late_release' to struct >>>> uio_info. If uio user sets this callback, it will be called by >>>> uio if userspace closes / munmaps the device after >>>> uio_unregister_device was executed. >>>> >>>> That way we can use uio_unregister_device() to notify userspace >>>> that we are going to destroy the device, but still get a call >>>> to late_release when uio device is finally closed. >>>> >>>> Signed-off-by: Bodo Stroesser <bostroesser@gmail.com> >>>> --- >>>> Documentation/driver-api/uio-howto.rst | 10 ++++++++++ >>>> drivers/uio/uio.c | 4 ++++ >>>> include/linux/uio_driver.h | 4 ++++ >>>> 3 files changed, 18 insertions(+) >>>> >>>> diff --git a/Documentation/driver-api/uio-howto.rst b/Documentation/driver-api/uio-howto.rst >>>> index 907ffa3b38f5..a2d57a7d623a 100644 >>>> --- a/Documentation/driver-api/uio-howto.rst >>>> +++ b/Documentation/driver-api/uio-howto.rst >>>> @@ -265,6 +265,16 @@ the members are required, others are optional. >>>> function. The parameter ``irq_on`` will be 0 to disable interrupts >>>> and 1 to enable them. >>>> +- ``int (*late_release)(struct uio_info *info, struct inode *inode)``: >>>> + Optional. If you define your own :c:func:`open()`, you will >>>> + in certain cases also want a custom :c:func:`late_release()` >>>> + function. If uio device is unregistered - by calling >>>> + :c:func:`uio_unregister_device()` - while it is open or mmap'ed by >>>> + userspace, the custom :c:func:`release()` function will not be >>>> + called when userspace later closes the device. An optionally >>>> + specified :c:func:`late_release()` function will be called in that >>>> + situation. >>>> + >>>> Usually, your device will have one or more memory regions that can be >>>> mapped to user space. For each region, you have to set up a >>>> ``struct uio_mem`` in the ``mem[]`` array. Here's a description of the >>>> diff --git a/drivers/uio/uio.c b/drivers/uio/uio.c >>>> index ea96e319c8a0..0b2636f8d373 100644 >>>> --- a/drivers/uio/uio.c >>>> +++ b/drivers/uio/uio.c >>>> @@ -532,6 +532,8 @@ static int uio_release(struct inode *inode, struct file *filep) >>>> mutex_lock(&idev->info_lock); >>>> if (idev->info && idev->info->release) >>>> ret = idev->info->release(idev->info, inode); >>>> + else if (idev->late_info && idev->late_info->late_release) >>>> + ret = idev->late_info->late_release(idev->late_info, inode); >>>> mutex_unlock(&idev->info_lock); >>> >>> Why can't release() be called here? Why doesn't your driver define a >>> release() if it cares about this information? Why do we need 2 >>> different callbacks that fire at exactly the same time? >>> >>> This feels really wrong. >>> >>> greg k-h >>> >> >> tcmu has a release callback. But uio can't call it after >> uio_unregister_device was executed, because in uio_unregister_device >> uio sets the uio_device::info to NULL. > > As it should because the driver could then be gone. It should NEVER > call back into it again. OTOH, uio does try_module_get(idev->owner) in uio_open before calling the driver's open callback and module_put(idev_owner) in uio_release after calling driver's release callback. So driver's release callback is guaranteed to exist until last release is done. Apart from that, tcmu also has an uio_info::mmap callback. In that callback it installs its own vm_operations_struct::fault handler. This handler also can happen to be called as long as userspace holds the uio device mmap'ed. I think, this is not a problem due to the above mentioned mechanism. tcmu just has to ensure, that the tcmu device, which contains the uio_info - is kept until the final release call happens. Unfortunately this call will not happen if uio device is open during uio_unregister_device. That's why tcmu sometimes leaks memory. > >> So, uio would never call both callbacks for the same release action, >> but would call release before uio_unregister_device is executed, and >> late_release after that. > > That's not ok. > >> Of course it would be good for tcmu if uio would call uio_info:release even >> after uio_unregister_device, but changing this AFAICS could cause >> trouble in other drivers using uio. > > You are confusing two different lifetime rules here it seems. One is > the char device and one is the struct device. They work independently > as different users affect them. I'm not sure I get your point. > > So if one is removed from the system, do not try to keep a callback to > it, otherwise you will crash. That's why I tried to change uio in a compatible way, so other drivers based on it are not afflicted by the change. I saw, that some drivers based on uio free their resources directly after calling uio_unregister_device. Executing their release callback later would definitely cause trouble. > > And why is scsi using the uio driver in the first place? That feels > really odd to me. Why not just make a "real" driver if you want to > somehow tie these two lifetimes together? Why tcmu driver is based on uio I don't know. I inherited the driver as it is. Maybe it would have been better to not base it on uio, I don't know. But changing this now would cause an API change for all existing userspace apps, e.g. tcmu-runner. I think I should avoid that and therefore have to find an acceptable solution for the tcmu/uio combination. > > thanks, > > greg k-h >
diff --git a/Documentation/driver-api/uio-howto.rst b/Documentation/driver-api/uio-howto.rst index 907ffa3b38f5..a2d57a7d623a 100644 --- a/Documentation/driver-api/uio-howto.rst +++ b/Documentation/driver-api/uio-howto.rst @@ -265,6 +265,16 @@ the members are required, others are optional. function. The parameter ``irq_on`` will be 0 to disable interrupts and 1 to enable them. +- ``int (*late_release)(struct uio_info *info, struct inode *inode)``: + Optional. If you define your own :c:func:`open()`, you will + in certain cases also want a custom :c:func:`late_release()` + function. If uio device is unregistered - by calling + :c:func:`uio_unregister_device()` - while it is open or mmap'ed by + userspace, the custom :c:func:`release()` function will not be + called when userspace later closes the device. An optionally + specified :c:func:`late_release()` function will be called in that + situation. + Usually, your device will have one or more memory regions that can be mapped to user space. For each region, you have to set up a ``struct uio_mem`` in the ``mem[]`` array. Here's a description of the diff --git a/drivers/uio/uio.c b/drivers/uio/uio.c index ea96e319c8a0..0b2636f8d373 100644 --- a/drivers/uio/uio.c +++ b/drivers/uio/uio.c @@ -532,6 +532,8 @@ static int uio_release(struct inode *inode, struct file *filep) mutex_lock(&idev->info_lock); if (idev->info && idev->info->release) ret = idev->info->release(idev->info, inode); + else if (idev->late_info && idev->late_info->late_release) + ret = idev->late_info->late_release(idev->late_info, inode); mutex_unlock(&idev->info_lock); module_put(idev->owner); @@ -1057,6 +1059,8 @@ void uio_unregister_device(struct uio_info *info) free_irq(info->irq, idev); idev->info = NULL; + if (info->late_release) + idev->late_info = info; mutex_unlock(&idev->info_lock); wake_up_interruptible(&idev->wait); diff --git a/include/linux/uio_driver.h b/include/linux/uio_driver.h index 47c5962b876b..5712e149f9ac 100644 --- a/include/linux/uio_driver.h +++ b/include/linux/uio_driver.h @@ -74,6 +74,7 @@ struct uio_device { struct fasync_struct *async_queue; wait_queue_head_t wait; struct uio_info *info; + struct uio_info *late_info; struct mutex info_lock; struct kobject *map_dir; struct kobject *portio_dir; @@ -94,6 +95,8 @@ struct uio_device { * @open: open operation for this uio device * @release: release operation for this uio device * @irqcontrol: disable/enable irqs when 0/1 is written to /dev/uioX + * @late_release: optional late release operation for this uio device, + * called if device is released after uio_unregister_device() */ struct uio_info { struct uio_device *uio_dev; @@ -109,6 +112,7 @@ struct uio_info { int (*open)(struct uio_info *info, struct inode *inode); int (*release)(struct uio_info *info, struct inode *inode); int (*irqcontrol)(struct uio_info *info, s32 irq_on); + int (*late_release)(struct uio_info *info, struct inode *inode); }; extern int __must_check
If uio_unregister_device() is called while userspace daemon still holds the uio device open or mmap'ed, uio will not call uio_info->release() on later close / munmap. At least one user of uio (tcmu) should not free resources (pages allocated by tcmu which are mmap'ed to userspace) while uio device still is open, because that could cause userspace daemon to be killed by SIGSEGV or SIGBUS. Therefore tcmu frees the pages only after it called uio_unregister_device _and_ the device was closed. So, uio not calling uio_info->release causes trouble. tcmu currently leaks memory in that case. Just waiting for userspace daemon to exit before calling uio_unregister_device I think is not the right solution, because daemon would not become aware of kernel code wanting to destroy the uio device. After uio_unregister_device was called, reading or writing the uio device returns -EIO, which normally results in daemon exit. This patch adds new callback pointer 'late_release' to struct uio_info. If uio user sets this callback, it will be called by uio if userspace closes / munmaps the device after uio_unregister_device was executed. That way we can use uio_unregister_device() to notify userspace that we are going to destroy the device, but still get a call to late_release when uio device is finally closed. Signed-off-by: Bodo Stroesser <bostroesser@gmail.com> --- Documentation/driver-api/uio-howto.rst | 10 ++++++++++ drivers/uio/uio.c | 4 ++++ include/linux/uio_driver.h | 4 ++++ 3 files changed, 18 insertions(+)