[0/5] padata: fix liftime issues after ->serial() has completed

Message

Nicolai Stange Oct. 19, 2022, 8:37 a.m. UTC

Hi all,

this series is supposed to fix some lifetime issues all related to the fact that
once the last ->serial() has been invoked, the padata user (i.e. pcrypt) is well
with its right to tear down the associated padata_shell or parallel_data
instance respectively.

Only the first one, addressed by patch [2/5], has actually been observed, namely
on a (downstream) RT kernel under a very specific workload involving LTP's
pcrypt_aead01. On non-RT, I've been unable to reproduce.

The remainder of this series, 3-5/5, fixes two more, somewhat related, but
purely theoretical issues I spotted when scratching my head about possible
reasons for the original Oops.

Thanks!

Nicolai

Nicolai Stange (5):
  padata: introduce internal padata_get/put_pd() helpers
  padata: make padata_free_shell() to respect pd's ->refcnt
  padata: grab parallel_data refcnt for reorder
  padata: split out dequeue operation from padata_find_next()
  padata: avoid potential UAFs to the padata_shell from padata_reorder()

 kernel/padata.c | 129 +++++++++++++++++++++++++++++++++++-------------
 1 file changed, 96 insertions(+), 33 deletions(-)

Comments

Daniel Jordan Oct. 21, 2022, 9:35 p.m. UTC | #1

Hi Nicolai,

On Wed, Oct 19, 2022 at 10:37:03AM +0200, Nicolai Stange wrote:
> Hi all,
> 
> this series is supposed to fix some lifetime issues all related to the fact that
> once the last ->serial() has been invoked, the padata user (i.e. pcrypt) is well
> with its right to tear down the associated padata_shell or parallel_data
> instance respectively.
> 
> Only the first one, addressed by patch [2/5], has actually been observed, namely
> on a (downstream) RT kernel under a very specific workload involving LTP's
> pcrypt_aead01. On non-RT, I've been unable to reproduce.

I haven't been able to hit the issue in 2/5 on RT on a v6.0 kernel in an
x86 vm.  Were there any other things running on the system besides
pcrypt_aead01?  More details about your environment and your kernel
config would be helpful.

The first two patches seem ok but I want to think about the second more
next week.  I'll look over the rest of the series then too.

Nicolai Stange Oct. 24, 2022, 8:47 a.m. UTC | #2

Hi Daniel,

Daniel Jordan <[e-mail]> writes:

> On Wed, Oct 19, 2022 at 10:37:03AM +0200, Nicolai Stange wrote:
>> this series is supposed to fix some lifetime issues all related to the fact that
>> once the last ->serial() has been invoked, the padata user (i.e. pcrypt) is well
>> with its right to tear down the associated padata_shell or parallel_data
>> instance respectively.
>> 
>> Only the first one, addressed by patch [2/5], has actually been observed, namely
>> on a (downstream) RT kernel under a very specific workload involving LTP's
>> pcrypt_aead01. On non-RT, I've been unable to reproduce.
>
> I haven't been able to hit the issue in 2/5 on RT on a v6.0 kernel in an
> x86 vm.  Were there any other things running on the system besides
> pcrypt_aead01?  More details about your environment and your kernel
> config would be helpful.

Right, the issue is indeed hard to reproduce, unfortunately. It has
originally been reported internally by our QA Maintenance team, which --
for unknown reason -- suddenly started to hit the issue once every while
in their testing environment. I did manage to reproduce it once or twice
myself, but it took me several days running pcrypt_aead01 in a loop each
time. AFAIR, I allocated a single cpu to the VM only and increased the
priority of pcrypt_aead01 a bit, with the intent to make preemption of
the ->serial() worker by DELALG more likely. But really, I cannot tell
if that did in fact contribute to the likelihood of triggering the race
or whether I've just been lucky.

Also, as mentioned in the cover letter, the RT kernel this has been
observed on is a downstream one, based on 5.3.18 (source tree at [1],
config at [2]), but with quite some additional patches on top. A
backport of this patch series here had been subject to testing in the
same environment the issue originally showed up in on a fairly regular
basis and no new crashes have been observed since.

Let me know if I could provide you with any more details.

Thanks,

Nicolai

[1] https://github.com/SUSE/kernel/tree/SLE15-SP3-RT
[2] https://github.com/SUSE/kernel-source/blob/SLE15-SP3-RT/config/x86_64/rt