| Message ID | 20220803050230.30152-1-yepeilin.cs@gmail.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | audit, io_uring, io-wq: Fix memory leak in io_sq_thread() and io_wqe_worker() | expand |
On Wed, Aug 3, 2022 at 1:03 AM Peilin Ye <yepeilin.cs@gmail.com> wrote: > > From: Peilin Ye <peilin.ye@bytedance.com> > > Currently @audit_context is allocated twice for io_uring workers: > > 1. copy_process() calls audit_alloc(); > 2. io_sq_thread() or io_wqe_worker() calls audit_alloc_kernel() (which > is effectively audit_alloc()) and overwrites @audit_context, > causing: > > BUG: memory leak > unreferenced object 0xffff888144547400 (size 1024): > <...> > hex dump (first 32 bytes): > 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 ................ > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > backtrace: > [<ffffffff8135cfc3>] audit_alloc+0x133/0x210 > [<ffffffff81239e63>] copy_process+0xcd3/0x2340 > [<ffffffff8123b5f3>] create_io_thread+0x63/0x90 > [<ffffffff81686604>] create_io_worker+0xb4/0x230 > [<ffffffff81686f68>] io_wqe_enqueue+0x248/0x3b0 > [<ffffffff8167663a>] io_queue_iowq+0xba/0x200 > [<ffffffff816768b3>] io_queue_async+0x113/0x180 > [<ffffffff816840df>] io_req_task_submit+0x18f/0x1a0 > [<ffffffff816841cd>] io_apoll_task_func+0xdd/0x120 > [<ffffffff8167d49f>] tctx_task_work+0x11f/0x570 > [<ffffffff81272c4e>] task_work_run+0x7e/0xc0 > [<ffffffff8125a688>] get_signal+0xc18/0xf10 > [<ffffffff8111645b>] arch_do_signal_or_restart+0x2b/0x730 > [<ffffffff812ea44e>] exit_to_user_mode_prepare+0x5e/0x180 > [<ffffffff844ae1b2>] syscall_exit_to_user_mode+0x12/0x20 > [<ffffffff844a7e80>] do_syscall_64+0x40/0x80 > > Then, > > 3. io_sq_thread() or io_wqe_worker() frees @audit_context using > audit_free(); > 4. do_exit() eventually calls audit_free() again, which is okay > because audit_free() does a NULL check. > > Free the old @audit_context first in audit_alloc_kernel(), and delete > the redundant calls to audit_free() for less confusion. > > Fixes: 5bd2182d58e9 ("audit,io_uring,io-wq: add some basic audit support to io_uring") > Cc: stable@vger.kernel.org > Signed-off-by: Peilin Ye <peilin.ye@bytedance.com> > --- > Hi all, > > A better way to fix this memleak would probably be checking > @args->io_thread in copy_process()? Something like: > > if (args->io_thread) > retval = audit_alloc_kernel(); > else > retval = audit_alloc(); > > But I didn't want to add another if to copy_process() for this bugfix. > Please suggest, thanks! Thanks for the report and patch! I'll take a closer look at this today and get back to you. > fs/io-wq.c | 1 - > fs/io_uring.c | 2 -- > kernel/auditsc.c | 1 + > 3 files changed, 1 insertion(+), 3 deletions(-)
On Wed, Aug 3, 2022 at 9:16 AM Paul Moore <paul@paul-moore.com> wrote: > On Wed, Aug 3, 2022 at 1:03 AM Peilin Ye <yepeilin.cs@gmail.com> wrote: > > > > Currently @audit_context is allocated twice for io_uring workers: > > > > 1. copy_process() calls audit_alloc(); > > 2. io_sq_thread() or io_wqe_worker() calls audit_alloc_kernel() (which > > is effectively audit_alloc()) and overwrites @audit_context, > > causing: > > > > BUG: memory leak > > unreferenced object 0xffff888144547400 (size 1024): > > <...> > > hex dump (first 32 bytes): > > 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 ................ > > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > backtrace: > > [<ffffffff8135cfc3>] audit_alloc+0x133/0x210 > > [<ffffffff81239e63>] copy_process+0xcd3/0x2340 > > [<ffffffff8123b5f3>] create_io_thread+0x63/0x90 > > [<ffffffff81686604>] create_io_worker+0xb4/0x230 > > [<ffffffff81686f68>] io_wqe_enqueue+0x248/0x3b0 > > [<ffffffff8167663a>] io_queue_iowq+0xba/0x200 > > [<ffffffff816768b3>] io_queue_async+0x113/0x180 > > [<ffffffff816840df>] io_req_task_submit+0x18f/0x1a0 > > [<ffffffff816841cd>] io_apoll_task_func+0xdd/0x120 > > [<ffffffff8167d49f>] tctx_task_work+0x11f/0x570 > > [<ffffffff81272c4e>] task_work_run+0x7e/0xc0 > > [<ffffffff8125a688>] get_signal+0xc18/0xf10 > > [<ffffffff8111645b>] arch_do_signal_or_restart+0x2b/0x730 > > [<ffffffff812ea44e>] exit_to_user_mode_prepare+0x5e/0x180 > > [<ffffffff844ae1b2>] syscall_exit_to_user_mode+0x12/0x20 > > [<ffffffff844a7e80>] do_syscall_64+0x40/0x80 > > > > Then, > > > > 3. io_sq_thread() or io_wqe_worker() frees @audit_context using > > audit_free(); > > 4. do_exit() eventually calls audit_free() again, which is okay > > because audit_free() does a NULL check. > > > > Free the old @audit_context first in audit_alloc_kernel(), and delete > > the redundant calls to audit_free() for less confusion. > > > > Fixes: 5bd2182d58e9 ("audit,io_uring,io-wq: add some basic audit support to io_uring") > > Cc: stable@vger.kernel.org > > Signed-off-by: Peilin Ye <peilin.ye@bytedance.com> > > --- > > Hi all, > > > > A better way to fix this memleak would probably be checking > > @args->io_thread in copy_process()? Something like: > > > > if (args->io_thread) > > retval = audit_alloc_kernel(); > > else > > retval = audit_alloc(); > > > > But I didn't want to add another if to copy_process() for this bugfix. > > Please suggest, thanks! > > Thanks for the report and patch! I'll take a closer look at this > today and get back to you. I think the best solution to this is simply to remove the calls to audit_alloc_kernel() in the io_uring and io-wq code, as well as the audit_alloc_kernel() function itself. As long as create_io_thread() ends up calling copy_process to create the new kernel thread the audit_context should be allocated correctly. Peilin Ye, are you able to draft a patch to do that and give it a test? For those that may be wondering how this happened (I definitely was!), it looks like when I first started working on the LSM/audit support for io_uring it was before the v5.12-rc1 release when create_io_thread() was introduced. Prior to create_io_thread() it appears that io_uring/io-wq wasn't calling into copy_process() and thus was not getting an audit_context allocated in the kernel thread's task_struct; the solution for those original development drafts was to add a call to a new audit_alloc_kernel() which would handle the audit_context allocation. Unfortunately, I didn't notice the move to create_io_thread() during development and the redundant audit_alloc_kernel() calls remained :/
On 8/3/22 1:28 PM, Paul Moore wrote: > On Wed, Aug 3, 2022 at 9:16 AM Paul Moore <paul@paul-moore.com> wrote: >> On Wed, Aug 3, 2022 at 1:03 AM Peilin Ye <yepeilin.cs@gmail.com> wrote: >>> >>> Currently @audit_context is allocated twice for io_uring workers: >>> >>> 1. copy_process() calls audit_alloc(); >>> 2. io_sq_thread() or io_wqe_worker() calls audit_alloc_kernel() (which >>> is effectively audit_alloc()) and overwrites @audit_context, >>> causing: >>> >>> BUG: memory leak >>> unreferenced object 0xffff888144547400 (size 1024): >>> <...> >>> hex dump (first 32 bytes): >>> 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 ................ >>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >>> backtrace: >>> [<ffffffff8135cfc3>] audit_alloc+0x133/0x210 >>> [<ffffffff81239e63>] copy_process+0xcd3/0x2340 >>> [<ffffffff8123b5f3>] create_io_thread+0x63/0x90 >>> [<ffffffff81686604>] create_io_worker+0xb4/0x230 >>> [<ffffffff81686f68>] io_wqe_enqueue+0x248/0x3b0 >>> [<ffffffff8167663a>] io_queue_iowq+0xba/0x200 >>> [<ffffffff816768b3>] io_queue_async+0x113/0x180 >>> [<ffffffff816840df>] io_req_task_submit+0x18f/0x1a0 >>> [<ffffffff816841cd>] io_apoll_task_func+0xdd/0x120 >>> [<ffffffff8167d49f>] tctx_task_work+0x11f/0x570 >>> [<ffffffff81272c4e>] task_work_run+0x7e/0xc0 >>> [<ffffffff8125a688>] get_signal+0xc18/0xf10 >>> [<ffffffff8111645b>] arch_do_signal_or_restart+0x2b/0x730 >>> [<ffffffff812ea44e>] exit_to_user_mode_prepare+0x5e/0x180 >>> [<ffffffff844ae1b2>] syscall_exit_to_user_mode+0x12/0x20 >>> [<ffffffff844a7e80>] do_syscall_64+0x40/0x80 >>> >>> Then, >>> >>> 3. io_sq_thread() or io_wqe_worker() frees @audit_context using >>> audit_free(); >>> 4. do_exit() eventually calls audit_free() again, which is okay >>> because audit_free() does a NULL check. >>> >>> Free the old @audit_context first in audit_alloc_kernel(), and delete >>> the redundant calls to audit_free() for less confusion. >>> >>> Fixes: 5bd2182d58e9 ("audit,io_uring,io-wq: add some basic audit support to io_uring") >>> Cc: stable@vger.kernel.org >>> Signed-off-by: Peilin Ye <peilin.ye@bytedance.com> >>> --- >>> Hi all, >>> >>> A better way to fix this memleak would probably be checking >>> @args->io_thread in copy_process()? Something like: >>> >>> if (args->io_thread) >>> retval = audit_alloc_kernel(); >>> else >>> retval = audit_alloc(); >>> >>> But I didn't want to add another if to copy_process() for this bugfix. >>> Please suggest, thanks! >> >> Thanks for the report and patch! I'll take a closer look at this >> today and get back to you. > > I think the best solution to this is simply to remove the calls to > audit_alloc_kernel() in the io_uring and io-wq code, as well as the > audit_alloc_kernel() function itself. As long as create_io_thread() > ends up calling copy_process to create the new kernel thread the > audit_context should be allocated correctly. Peilin Ye, are you able > to draft a patch to do that and give it a test? > > For those that may be wondering how this happened (I definitely was!), > it looks like when I first started working on the LSM/audit support > for io_uring it was before the v5.12-rc1 release when > create_io_thread() was introduced. Prior to create_io_thread() it > appears that io_uring/io-wq wasn't calling into copy_process() and > thus was not getting an audit_context allocated in the kernel thread's > task_struct; the solution for those original development drafts was to > add a call to a new audit_alloc_kernel() which would handle the > audit_context allocation. Unfortunately, I didn't notice the move to > create_io_thread() during development and the redundant > audit_alloc_kernel() calls remained :/ I agree with your analysis and suggested solution. Post the native io-wq workers create_io_thread() -> copy_process() is always used for io-wq (and sqpoll, for that matter).
Hi, On Wed, Aug 03, 2022 at 03:28:19PM -0400, Paul Moore wrote: > On Wed, Aug 3, 2022 at 9:16 AM Paul Moore <paul@paul-moore.com> wrote: > > On Wed, Aug 3, 2022 at 1:03 AM Peilin Ye <yepeilin.cs@gmail.com> wrote: > > > Hi all, > > > > > > A better way to fix this memleak would probably be checking > > > @args->io_thread in copy_process()? Something like: > > > > > > if (args->io_thread) > > > retval = audit_alloc_kernel(); > > > else > > > retval = audit_alloc(); > > > > > > But I didn't want to add another if to copy_process() for this bugfix. > > > Please suggest, thanks! > > > > Thanks for the report and patch! I'll take a closer look at this > > today and get back to you. > > I think the best solution to this is simply to remove the calls to > audit_alloc_kernel() in the io_uring and io-wq code, as well as the > audit_alloc_kernel() function itself. As long as create_io_thread() > ends up calling copy_process to create the new kernel thread the > audit_context should be allocated correctly. Peilin Ye, are you able > to draft a patch to do that and give it a test? Sure, I will write a v2 today. Thanks for the suggestion! > For those that may be wondering how this happened (I definitely was!), > it looks like when I first started working on the LSM/audit support > for io_uring it was before the v5.12-rc1 release when > create_io_thread() was introduced. Prior to create_io_thread() it > appears that io_uring/io-wq wasn't calling into copy_process() and > thus was not getting an audit_context allocated in the kernel thread's > task_struct; the solution for those original development drafts was to > add a call to a new audit_alloc_kernel() which would handle the > audit_context allocation. Unfortunately, I didn't notice the move to > create_io_thread() during development and the redundant > audit_alloc_kernel() calls remained :/ Thanks, Peilin Ye
diff --git a/fs/io-wq.c b/fs/io-wq.c index 824623bcf1a5..0f4804a5e873 100644 --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -670,7 +670,6 @@ static int io_wqe_worker(void *data) if (test_bit(IO_WQ_BIT_EXIT, &wq->state)) io_worker_handle_work(worker); - audit_free(current); io_worker_exit(worker); return 0; } diff --git a/fs/io_uring.c b/fs/io_uring.c index e8e769be9ed0..0f27914f37f2 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -9283,8 +9283,6 @@ static int io_sq_thread(void *data) io_run_task_work(); mutex_unlock(&sqd->lock); - audit_free(current); - complete(&sqd->exited); do_exit(0); } diff --git a/kernel/auditsc.c b/kernel/auditsc.c index 3a8c9d744800..7948090fd12f 100644 --- a/kernel/auditsc.c +++ b/kernel/auditsc.c @@ -1095,6 +1095,7 @@ int audit_alloc_kernel(struct task_struct *tsk) * 2. The {set,clear}_task_syscall_work() ops likely have zero effect * on these internal kernel tasks, but they probably don't hurt either. */ + __audit_free(current); return audit_alloc(tsk); }