| Message ID | 20180731005615.GA2911@visor (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [RESEND] exec: don't force_sigsegv processes with a pending fatal signal | expand |
Hi Ivan, 2018-07-31 1:56 GMT+01:00 Ivan Delalande <colona@arista.com>: > We were seeing unexplained segfaults in coreutils processes and other > basic utilities that we tracked down to binfmt_elf failing to load > segments for ld.so. Digging further, the actual problem seems to occur > when a process gets sigkilled while it is still being loaded by the > kernel. In our case when _do_page_fault goes for a retry it will return > early as it first checks for fatal_signal_pending(), so load_elf_interp > also returns with error and as a result search_binary_handler will > force_sigsegv() which is pretty confusing as nothing actually failed > here. > > Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()") > Reference: https://lkml.org/lkml/2013/2/14/5 > Signed-off-by: Ivan Delalande <colona@arista.com> +Cc: Oleg Nesterov <oleg@redhat.com> +Cc: Andy Lutomirski <luto@kernel.org> > --- > fs/exec.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/fs/exec.c b/fs/exec.c > index bdd0eacefdf5..6e8007edbb2d 100644 > --- a/fs/exec.c > +++ b/fs/exec.c > @@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm) > if (retval < 0 && !bprm->mm) { > /* we got to flush_old_exec() and failed after it */ > read_unlock(&binfmt_lock); > - force_sigsegv(SIGSEGV, current); > + if (!fatal_signal_pending(current)) > + force_sigsegv(SIGSEGV, current); I would suggest to add something like: : if (print_fatal_signals) : pr_info("load_binary() failed: %d\n", retval); It was interesting to catch that it actually segfaults during loading, probably will save someone a couple of minutes too ;-) > return retval; > } > if (retval != -ENOEXEC || !bprm->file) { Thanks, Dmitry
2018-08-02 20:53 GMT+01:00 Dmitry Safonov <0x7f454c46@gmail.com>: > Hi Ivan, > > 2018-07-31 1:56 GMT+01:00 Ivan Delalande <colona@arista.com>: >> We were seeing unexplained segfaults in coreutils processes and other >> basic utilities that we tracked down to binfmt_elf failing to load >> segments for ld.so. Digging further, the actual problem seems to occur >> when a process gets sigkilled while it is still being loaded by the >> kernel. In our case when _do_page_fault goes for a retry it will return >> early as it first checks for fatal_signal_pending(), so load_elf_interp >> also returns with error and as a result search_binary_handler will >> force_sigsegv() which is pretty confusing as nothing actually failed >> here. >> >> Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()") >> Reference: https://lkml.org/lkml/2013/2/14/5 >> Signed-off-by: Ivan Delalande <colona@arista.com> > > +Cc: Oleg Nesterov <oleg@redhat.com> > +Cc: Andy Lutomirski <luto@kernel.org> Also worth to add to commit message an example of previously user-visible message in dmesg: [ 1545.889604] potentially unexpected fatal signal 11. [ 1545.889614] CPU: 2 PID: 7462 Comm: grep Tainted: P O 3.18.28 #1 [ 1545.889617] Hardware name: Celestica D4040/D4040, BIOS 5.6.5 08/18/2016 [ 1545.889621] task: ffff880011282280 ti: ffff880100938000 task.ti: ffff880100938000 [ 1545.889624] RIP: 0023:[<00000000f760eb70>] [<00000000f760eb70>] 0xf760eb70 [ 1545.889641] RSP: 002b:00000000fffa3454 EFLAGS: 00000296 [ 1545.889644] RAX: fffffffffffffff2 RBX: 00000000f7a5c3e8 RCX: 00000000f7a5c718 [ 1545.889647] RDX: 00000000f7a5b230 RSI: 00000000f7a5c718 RDI: 00000000f757c000 [ 1545.889650] RBP: 00000000f7a5c3e8 R08: 0000000000000000 R09: 0000000000000000 [ 1545.889653] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 1545.889656] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 1545.889659] FS: 0000000000000000(0000) GS:ffff88017fb00000(0000) knlGS:0000000000000000 [ 1545.889662] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b [ 1545.889665] CR2: 00000000f77d7838 CR3: 000000010bb90000 CR4: 00000000001007e0 (which now will be suppressed if there was a fatal signal) > >> --- >> fs/exec.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/fs/exec.c b/fs/exec.c >> index bdd0eacefdf5..6e8007edbb2d 100644 >> --- a/fs/exec.c >> +++ b/fs/exec.c >> @@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm) >> if (retval < 0 && !bprm->mm) { >> /* we got to flush_old_exec() and failed after it */ >> read_unlock(&binfmt_lock); >> - force_sigsegv(SIGSEGV, current); >> + if (!fatal_signal_pending(current)) >> + force_sigsegv(SIGSEGV, current); > > I would suggest to add something like: > : if (print_fatal_signals) > : pr_info("load_binary() failed: %d\n", retval); > > It was interesting to catch that it actually segfaults during loading, > probably will save someone a couple of minutes too ;-) Not sure if it's easy to trigger, but it might require a ratelimit too.. > >> return retval; >> } >> if (retval != -ENOEXEC || !bprm->file) {
On 08/02, Dmitry Safonov wrote: > > Hi Ivan, > > 2018-07-31 1:56 GMT+01:00 Ivan Delalande <colona@arista.com>: > > We were seeing unexplained segfaults in coreutils processes and other > > basic utilities that we tracked down to binfmt_elf failing to load > > segments for ld.so. Digging further, the actual problem seems to occur > > when a process gets sigkilled while it is still being loaded by the > > kernel. In our case when _do_page_fault goes for a retry it will return > > early as it first checks for fatal_signal_pending(), so load_elf_interp > > also returns with error and as a result search_binary_handler will > > force_sigsegv() which is pretty confusing as nothing actually failed > > here. > > > > Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()") > > Reference: https://lkml.org/lkml/2013/2/14/5 > > Signed-off-by: Ivan Delalande <colona@arista.com> > > +Cc: Oleg Nesterov <oleg@redhat.com> > +Cc: Andy Lutomirski <luto@kernel.org> Thanks... and sorry, I fail to understand the problem and what/how this patch tries to fix. Hmm. After I read the next email from Dmitry it seems to me that the whole purpose of this patch is to avoid print_fatal_signal()? If yes, the changelog should clearly explain this. > > --- a/fs/exec.c > > +++ b/fs/exec.c > > @@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm) > > if (retval < 0 && !bprm->mm) { > > /* we got to flush_old_exec() and failed after it */ > > read_unlock(&binfmt_lock); > > - force_sigsegv(SIGSEGV, current); > > + if (!fatal_signal_pending(current)) > > + force_sigsegv(SIGSEGV, current); I won't argue, but may be force_sigsegv() should check fatal_signal_pending() itself. setup_rt_frame() can too fail if fatal_signal_pending() by the same reason. Oleg.
Hi, On Fri, Aug 03, 2018 at 03:39:24PM +0200, Oleg Nesterov wrote: > On 08/02, Dmitry Safonov wrote: > > 2018-07-31 1:56 GMT+01:00 Ivan Delalande <colona@arista.com>: > > > We were seeing unexplained segfaults in coreutils processes and other > > > basic utilities that we tracked down to binfmt_elf failing to load > > > segments for ld.so. Digging further, the actual problem seems to occur > > > when a process gets sigkilled while it is still being loaded by the > > > kernel. In our case when _do_page_fault goes for a retry it will return > > > early as it first checks for fatal_signal_pending(), so load_elf_interp > > > also returns with error and as a result search_binary_handler will > > > force_sigsegv() which is pretty confusing as nothing actually failed > > > here. > > > > > > Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()") > > > Reference: https://lkml.org/lkml/2013/2/14/5 > > > Signed-off-by: Ivan Delalande <colona@arista.com> > > > > +Cc: Oleg Nesterov <oleg@redhat.com> > > +Cc: Andy Lutomirski <luto@kernel.org> > > Thanks... > > and sorry, I fail to understand the problem and what/how this patch tries to fix. > > Hmm. After I read the next email from Dmitry it seems to me that the whole purpose > of this patch is to avoid print_fatal_signal()? If yes, the changelog should clearly > explain this. Sorry about that, yes this is purely to avoid printing the segfault messages for these processes when they were in fact killed. I'll definitely send a v2 to clarify that, and probably add the helpful message Dimitry suggested as well. > > > --- a/fs/exec.c > > > +++ b/fs/exec.c > > > @@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm) > > > if (retval < 0 && !bprm->mm) { > > > /* we got to flush_old_exec() and failed after it */ > > > read_unlock(&binfmt_lock); > > > - force_sigsegv(SIGSEGV, current); > > > + if (!fatal_signal_pending(current)) > > > + force_sigsegv(SIGSEGV, current); > > I won't argue, but may be force_sigsegv() should check fatal_signal_pending() > itself. setup_rt_frame() can too fail if fatal_signal_pending() by the same > reason. I'm not sure, I think it would feel out of place in force_sigsegv() as other callers might not expect this check in different contexts. I could add a similar call to fatal_signal_pending() in signal_setup_done() though, if you think we can hit the same problem from setup_rt_frame(). Thanks,
diff --git a/fs/exec.c b/fs/exec.c index bdd0eacefdf5..6e8007edbb2d 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm) if (retval < 0 && !bprm->mm) { /* we got to flush_old_exec() and failed after it */ read_unlock(&binfmt_lock); - force_sigsegv(SIGSEGV, current); + if (!fatal_signal_pending(current)) + force_sigsegv(SIGSEGV, current); return retval; } if (retval != -ENOEXEC || !bprm->file) {
We were seeing unexplained segfaults in coreutils processes and other basic utilities that we tracked down to binfmt_elf failing to load segments for ld.so. Digging further, the actual problem seems to occur when a process gets sigkilled while it is still being loaded by the kernel. In our case when _do_page_fault goes for a retry it will return early as it first checks for fatal_signal_pending(), so load_elf_interp also returns with error and as a result search_binary_handler will force_sigsegv() which is pretty confusing as nothing actually failed here. Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()") Reference: https://lkml.org/lkml/2013/2/14/5 Signed-off-by: Ivan Delalande <colona@arista.com> --- fs/exec.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)