From patchwork Fri Oct 16 02:40:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 11840493 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C427514B4 for ; Fri, 16 Oct 2020 02:40:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4B6D320897 for ; Fri, 16 Oct 2020 02:40:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="HYu9FnLy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4B6D320897 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CEB03940008; Thu, 15 Oct 2020 22:40:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C7627900002; Thu, 15 Oct 2020 22:40:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC743940008; Thu, 15 Oct 2020 22:40:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0212.hostedemail.com [216.40.44.212]) by kanga.kvack.org (Postfix) with ESMTP id 79BE7900002 for ; Thu, 15 Oct 2020 22:40:36 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 1A48A181AEF10 for ; Fri, 16 Oct 2020 02:40:36 +0000 (UTC) X-FDA: 77376235272.25.mind83_130238927219 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id EC2B01804E3A0 for ; Fri, 16 Oct 2020 02:40:35 +0000 (UTC) X-Spam-Summary: 1,0,0,872f6bb3eabebb5a,d41d8cd98f00b204,jannh@google.com,,RULES_HIT:2:41:69:355:379:541:800:960:966:973:982:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1606:1730:1747:1777:1792:2196:2198:2199:2200:2393:2553:2559:2562:2691:2693:2731:2894:2898:2901:3138:3139:3140:3141:3142:3152:3865:3866:3867:3868:3870:3871:3872:3874:4119:4250:4321:4385:4605:5007:6117:6120:6261:6653:7875:7901:7903:9592:9969:10004:11026:11232:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12895:12986:13161:13180:13229:13255:13894:14096:14877:21080:21222:21324:21444:21451:21627:21939:21972:21987:30003:30045:30054:30070:30090,0,RBL:209.85.128.67:@google.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100;04y87hxxd37w5j69zsh4dn7ffc6bxyc4tzyr3u3mxmg5jc3imkd7ac1qcq3z9e1.cb11iithbnj9zsyh8mnm1eaem4mymm7bncid9dbss1hp8iqrgm913arehpn5yu7.6-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0, DNSBL:ne X-HE-Tag: mind83_130238927219 X-Filterd-Recvd-Size: 8964 Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Fri, 16 Oct 2020 02:40:35 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id k18so1190745wmj.5 for ; Thu, 15 Oct 2020 19:40:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pvEughYKhq6CPlCwy3Gb88MacOrMK6GsoP9z00bpKRo=; b=HYu9FnLyQSFSJmzxotOAKKwXADM27PjlJfnilMOVzzmweirs2scSTtjZxmW19WFqbs qKhVuz2d4dNSGIxsoo5WUcepJaEhXiT/mvkHYzy+ITRq0rrraQ9Tb3Dafm8iwb7v0lwt hbwAfsD7vq3O8xlwMJ4oUxxCNqRvcwURiW46MBI557AdjxFD0AMoL6LSrzRfZfbgnPoe N1urYaQDLUyElj/nIYY6u3bDOYoCOG12lB+tNkzQBw3Z9dsapQKSf2tgcWW/x4UacSwx 1JrOInTm+F21s2HkGMw/6OK9Hc6RbGwkLUxjMU2X7mR+AeFOYGcoxHfohgRoK8ybyCmK NLZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pvEughYKhq6CPlCwy3Gb88MacOrMK6GsoP9z00bpKRo=; b=Zme5uyfQ57sGGyBk3X6AdC74dcDhhv90c/QT2xyu4UwDhwTUTIHEDoY1uNuEaJzAYu u8qrk2qw1XsmokUTSIhkHwLB5WzwYJcUne/6eYeNzUhyGO5snUXK/ZbNZf9IfcuOt2Mi 0w8Uv3PFS5/wy0w16h9fn49kijL/scJ/d8iVU1+AxXGAwrpztJC8KMkCByCtDXyEveLy trNy6nQzT6apui7FQNFPkkJpO8n2/fYzvcY1mY10buA6QIUArZMEYoRgsOGEWokLcTFn OHW0GsDZp+o95Pw5yk0hW5AGDMzNj0CT0Vk+8XGVeaafF+goNYfCWAFoLuIZgIJKVhEG TKeA== X-Gm-Message-State: AOAM532MFYB/E1wI+YvWtC7I4cgbhDqWgkfcnKWq6eErVgKtR4rHV6jA klRPHS1+IfbNmqYQcMoetjByiQ== X-Google-Smtp-Source: ABdhPJwVT5RN/jLxGSRyml+j1Ke4WERT80MS8k60n76yy/6+aaqzCEh+GU7QKYNDx5gbZUww9+LjmQ== X-Received: by 2002:a1c:f70b:: with SMTP id v11mr1371136wmh.21.1602816034173; Thu, 15 Oct 2020 19:40:34 -0700 (PDT) Received: from localhost ([2a02:168:96c5:1:55ed:514f:6ad7:5bcc]) by smtp.gmail.com with ESMTPSA id x18sm1543168wrg.4.2020.10.15.19.40.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Oct 2020 19:40:33 -0700 (PDT) From: Jann Horn To: Andrew Morton , linux-mm@kvack.org, Eric Biederman , Oleg Nesterov Cc: linux-kernel@vger.kernel.org, Will Deacon , Kees Cook , Ingo Molnar Subject: [RFC PATCH 1/6] ptrace: Keep mm around after exit_mm() for __ptrace_may_access() Date: Fri, 16 Oct 2020 04:40:14 +0200 Message-Id: <20201016024019.1882062-2-jannh@google.com> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201016024019.1882062-1-jannh@google.com> References: <20201016024019.1882062-1-jannh@google.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __ptrace_may_access() checks can happen on target tasks that are in the middle of do_exit(), past exit_mm(). At that point, the ->mm pointer has been NULLed out, and the mm_struct has been mmput(). Unfortunately, the mm_struct contains the dumpability and the user_ns in which the task last went through execve(), and we need those for __ptrace_may_access(). Currently, that problem is handled by failing open: If the ->mm is gone, we assume that the task was dumpable. In some edge cases, this could potentially expose access to things like /proc/$pid/fd/$fd of originally non-dumpable processes. (exit_files() comes after exit_mm(), so the file descriptor table is still there when we've gone through exit_mm().) One way to fix this would be to move mm->user_ns and the dumpability state over into the task_struct. However, that gets quite ugly if we want to preserve existing semantics because e.g. PR_SET_DUMPABLE and commit_creds() would then have to scan through all tasks sharing the mm_struct and keep them in sync manually - that'd be a bit error-prone and overcomplicated. (Moving these things into the signal_struct is not an option because that is kept across executions, and pre-execve co-threads will share the signal_struct that is also used by the task that has gone through execve().) I believe that this patch may be the least bad option to fix this - keep the mm_struct (but not process memory) around with an mmgrab() reference from exit_mm() until the task goes away completely. Note that this moves free_task() down in order to make mmdrop_async() available without a forward declaration. Cc: stable@vger.kernel.org Fixes: bfedb589252c ("mm: Add a user_ns owner to mm_struct and fix ptrace permission checks") Signed-off-by: Jann Horn --- include/linux/sched.h | 8 +++++++ kernel/exit.c | 2 ++ kernel/fork.c | 54 ++++++++++++++++++++++--------------------- kernel/ptrace.c | 10 ++++++++ 4 files changed, 48 insertions(+), 26 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index afe01e232935..55bec6ff5626 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -747,6 +747,14 @@ struct task_struct { struct mm_struct *mm; struct mm_struct *active_mm; + /* + * When we exit and ->mm (the reference pinning ->mm's address space) + * goes away, we stash a reference to the mm_struct itself (counted via + * exit_mm->mm_count) in this member. + * This allows us to continue using the mm_struct for security checks + * and such even after the task has started exiting. + */ + struct mm_struct *exit_mm; /* Per-thread vma caching: */ struct vmacache vmacache; diff --git a/kernel/exit.c b/kernel/exit.c index 733e80f334e7..97253ef33486 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -476,6 +476,8 @@ static void exit_mm(void) /* more a memory barrier than a real lock */ task_lock(current); current->mm = NULL; + mmgrab(mm); /* for current->exit_mm */ + current->exit_mm = mm; mmap_read_unlock(mm); enter_lazy_tlb(mm, current); task_unlock(current); diff --git a/kernel/fork.c b/kernel/fork.c index da8d360fb032..4942428a217c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -438,32 +438,6 @@ void put_task_stack(struct task_struct *tsk) } #endif -void free_task(struct task_struct *tsk) -{ - scs_release(tsk); - -#ifndef CONFIG_THREAD_INFO_IN_TASK - /* - * The task is finally done with both the stack and thread_info, - * so free both. - */ - release_task_stack(tsk); -#else - /* - * If the task had a separate stack allocation, it should be gone - * by now. - */ - WARN_ON_ONCE(refcount_read(&tsk->stack_refcount) != 0); -#endif - rt_mutex_debug_task_free(tsk); - ftrace_graph_exit_task(tsk); - arch_release_task_struct(tsk); - if (tsk->flags & PF_KTHREAD) - free_kthread_struct(tsk); - free_task_struct(tsk); -} -EXPORT_SYMBOL(free_task); - #ifdef CONFIG_MMU static __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) @@ -722,6 +696,34 @@ static inline void put_signal_struct(struct signal_struct *sig) free_signal_struct(sig); } +void free_task(struct task_struct *tsk) +{ + scs_release(tsk); + +#ifndef CONFIG_THREAD_INFO_IN_TASK + /* + * The task is finally done with both the stack and thread_info, + * so free both. + */ + release_task_stack(tsk); +#else + /* + * If the task had a separate stack allocation, it should be gone + * by now. + */ + WARN_ON_ONCE(refcount_read(&tsk->stack_refcount) != 0); +#endif + rt_mutex_debug_task_free(tsk); + ftrace_graph_exit_task(tsk); + arch_release_task_struct(tsk); + if (tsk->flags & PF_KTHREAD) + free_kthread_struct(tsk); + if (tsk->exit_mm) + mmdrop_async(tsk->exit_mm); + free_task_struct(tsk); +} +EXPORT_SYMBOL(free_task); + void __put_task_struct(struct task_struct *tsk) { WARN_ON(!tsk->exit_state); diff --git a/kernel/ptrace.c b/kernel/ptrace.c index 43d6179508d6..0aedc6cf5bdc 100644 --- a/kernel/ptrace.c +++ b/kernel/ptrace.c @@ -342,7 +342,17 @@ static int __ptrace_may_access(struct task_struct *task, unsigned int mode) * Pairs with a write barrier in commit_creds(). */ smp_rmb(); + /* + * Look up the target task's mm_struct. If it fails because the task is + * exiting and has gone through exit_mm(), we can instead use ->exit_mm + * as long as we only use members that are preserved by an mmgrab() + * reference. + * The only case in which both ->mm and ->exit_mm can be NULL should be + * kernel threads. + */ mm = task->mm; + if (!mm) + mm = task->exit_mm; if (mm && ((get_dumpable(mm) != SUID_DUMP_USER) && !ptrace_has_cap(cred, mm->user_ns, mode))) From patchwork Fri Oct 16 02:40:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 11840495 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E614A16BC for ; Fri, 16 Oct 2020 02:40:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 957F920878 for ; Fri, 16 Oct 2020 02:40:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="syzN3L/W" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 957F920878 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 23BA6940009; Thu, 15 Oct 2020 22:40:38 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1EED3900002; Thu, 15 Oct 2020 22:40:38 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 107E8940009; Thu, 15 Oct 2020 22:40:38 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CCE09900002 for ; Thu, 15 Oct 2020 22:40:37 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 69059362E for ; Fri, 16 Oct 2020 02:40:37 +0000 (UTC) X-FDA: 77376235314.10.leaf77_5d08fdb27219 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id 4BA7516A0B9 for ; Fri, 16 Oct 2020 02:40:37 +0000 (UTC) X-Spam-Summary: 1,0,0,93bb8a1d20ee9c6e,d41d8cd98f00b204,jannh@google.com,,RULES_HIT:41:69:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:3138:3139:3140:3141:3142:3152:3353:3865:3866:3867:3868:3870:3871:3874:4321:4385:5007:6120:6261:6653:7875:9592:9969:10004:11658:11914:12043:12048:12296:12297:12517:12519:12555:12895:12986:13138:13231:13618:13894:14096:14181:14721:21080:21444:21451:21627:30054,0,RBL:209.85.128.68:@google.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100;04yrs4wqkguyu4mizpqytgy76s9stypaeqpnhdh1bx96wittjnw9fhesinukm5i.qsnga6b5uwnzyx9b9i7rny1iskts7ywy7s35x4bn4mbtz4zi9g1xfidqxzwb1at.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: leaf77_5d08fdb27219 X-Filterd-Recvd-Size: 5140 Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Fri, 16 Oct 2020 02:40:36 +0000 (UTC) Received: by mail-wm1-f68.google.com with SMTP id p15so939254wmi.4 for ; Thu, 15 Oct 2020 19:40:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eMrDzdvIowItPCZxwmw+48iBRpQ/wjXi6fymrEYNArc=; b=syzN3L/Wp3YBmgLZItyTMxNEiB7mSUwRKg+lARKzCumVW9bP0wC/rA7URPv/3XF2yt NrGKLpQo13w4rdDGIqCvUm3OISGfJNWUuhWojJn6f1GySxsY9rQqnNl+nyluowWKlF/w /SfnEGkqDMZDKLHzI5YHh1fFjWw5PbMtQKqRUbHCZnkS//gMV2wIk7A2G37nuTKvHADs B/zEQmJuo6j1bA3+59r6JaC2HhzekjfyfiImcMg48D1936b52+jSyU3YhSQywN11p6mU R867i5PzCvR+idcLzFJMQt7NwJbNlQ98E3Z39zytdEiGac/EIY/UPKeRButIQE62fycf qhaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eMrDzdvIowItPCZxwmw+48iBRpQ/wjXi6fymrEYNArc=; b=VUNGd9q3dSx6X0QB/myxca2r25AWJBcgxVB6hvl+fjgi4H9wCMuYytJs7GmYDUd0jq FWhGRLgfWDmJyqwLiow2xMSnncGvaUVEwtc9rc4Ei1XMO2EyPQLWw4KRVRrvzxfsPZlq bJycxByv8OhCdeHZKB9eJn2+EqQzpn0F7gMr6kxCBuevmAgheBvxGS2BVtEURS9cqjMW OYPlMwJbfYxra4EigNqpJ/DONvmDMt2Mdj0IRF116O/7LWjb0w8MUjX9GrsppF4qoDAz e2GabLL3AGDjp0gc88u/2tbCXV2FhPiDxLvAf/azvwjEqbj0ABUuh5cvh6yPorKcx8Nv Wj3Q== X-Gm-Message-State: AOAM531SGT8vV601WtjC052h+nQC77ba8F+3EUL8wEz5RKVmmVzqjzoL mpcGAWVOXEn1s3f9pr0IYR0u0g== X-Google-Smtp-Source: ABdhPJy9aBJRo775A0oUO4Y+eGG3JV+yIwLDNGANvDx1gCLPck3vDjMcMFiN3f0sqr3F+s6pCGS/Hw== X-Received: by 2002:a1c:e919:: with SMTP id q25mr1385922wmc.142.1602816035657; Thu, 15 Oct 2020 19:40:35 -0700 (PDT) Received: from localhost ([2a02:168:96c5:1:55ed:514f:6ad7:5bcc]) by smtp.gmail.com with ESMTPSA id k5sm972649wmb.19.2020.10.15.19.40.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Oct 2020 19:40:34 -0700 (PDT) From: Jann Horn To: Andrew Morton , linux-mm@kvack.org, Eric Biederman , Oleg Nesterov Cc: linux-kernel@vger.kernel.org, Will Deacon , Kees Cook , Ingo Molnar Subject: [RFC PATCH 2/6] refcount: Move refcount_t definition into linux/types.h Date: Fri, 16 Oct 2020 04:40:15 +0200 Message-Id: <20201016024019.1882062-3-jannh@google.com> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201016024019.1882062-1-jannh@google.com> References: <20201016024019.1882062-1-jannh@google.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: I want to use refcount_t in mm_struct, but if refcount_t is defined in linux/refcount.h, that header would have to be included in linux/mm_types.h; that would be wasteful. Let's move refcount_t over into linux/types.h so that includes can be written less wastefully. Signed-off-by: Jann Horn --- include/linux/refcount.h | 13 +------------ include/linux/types.h | 12 ++++++++++++ 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/include/linux/refcount.h b/include/linux/refcount.h index 0e3ee25eb156..fd8cf65e4e2f 100644 --- a/include/linux/refcount.h +++ b/include/linux/refcount.h @@ -96,22 +96,11 @@ #include #include #include +#include /* refcount_t is defined here */ #include struct mutex; -/** - * struct refcount_t - variant of atomic_t specialized for reference counts - * @refs: atomic_t counter field - * - * The counter saturates at REFCOUNT_SATURATED and will not move once - * there. This avoids wrapping the counter and causing 'spurious' - * use-after-free bugs. - */ -typedef struct refcount_struct { - atomic_t refs; -} refcount_t; - #define REFCOUNT_INIT(n) { .refs = ATOMIC_INIT(n), } #define REFCOUNT_MAX INT_MAX #define REFCOUNT_SATURATED (INT_MIN / 2) diff --git a/include/linux/types.h b/include/linux/types.h index a147977602b5..34e4e779e767 100644 --- a/include/linux/types.h +++ b/include/linux/types.h @@ -175,6 +175,18 @@ typedef struct { } atomic64_t; #endif +/** + * struct refcount_t - variant of atomic_t specialized for reference counts + * @refs: atomic_t counter field + * + * The counter saturates at REFCOUNT_SATURATED and will not move once + * there. This avoids wrapping the counter and causing 'spurious' + * use-after-free bugs. + */ +typedef struct refcount_struct { + atomic_t refs; +} refcount_t; + struct list_head { struct list_head *next, *prev; }; From patchwork Fri Oct 16 02:40:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 11840497 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DF99214B4 for ; Fri, 16 Oct 2020 02:40:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 718F420897 for ; Fri, 16 Oct 2020 02:40:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="Mxy/DHXY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 718F420897 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C9E1094000A; Thu, 15 Oct 2020 22:40:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C7753900002; Thu, 15 Oct 2020 22:40:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC6F394000A; Thu, 15 Oct 2020 22:40:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0041.hostedemail.com [216.40.44.41]) by kanga.kvack.org (Postfix) with ESMTP id 7ABB4900002 for ; Thu, 15 Oct 2020 22:40:39 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 13D1C8249980 for ; Fri, 16 Oct 2020 02:40:39 +0000 (UTC) X-FDA: 77376235398.17.range71_111865b27219 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id E4C7B180D0180 for ; Fri, 16 Oct 2020 02:40:38 +0000 (UTC) X-Spam-Summary: 1,0,0,dedb22f1deae9c5e,d41d8cd98f00b204,jannh@google.com,,RULES_HIT:2:41:355:379:541:800:960:966:973:982:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1605:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:3138:3139:3140:3141:3142:3152:3165:3622:3865:3866:3867:3868:3870:3871:3872:3874:4049:4120:4250:4321:4385:4605:5007:6119:6120:6261:6653:7903:9969:10004:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12895:12986:13149:13161:13229:13230:13894:14096:21080:21433:21444:21451:21627:21740:30034:30045:30054,0,RBL:209.85.128.65:@google.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100;04y89umndkj81gxz5ep3r5xawnf1socdk3gycnodw6cd4ayaatjjn614n5qs9mh.o5wwhdooh6dxxme5ritkkp8yskgr9oqegm9hpnioihtt5463nkkz8karbxb8pyt.o-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: range71_111865b27219 X-Filterd-Recvd-Size: 9732 Received: from mail-wm1-f65.google.com (mail-wm1-f65.google.com [209.85.128.65]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Fri, 16 Oct 2020 02:40:38 +0000 (UTC) Received: by mail-wm1-f65.google.com with SMTP id e2so1232146wme.1 for ; Thu, 15 Oct 2020 19:40:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0v8cEFkv006H/orJHvg5H7puUbhcFvgGtNNhP7STOH0=; b=Mxy/DHXYmetsU0zQ6Xxh1pIcvrU0cHOnso1CihgTxthKlvIcT8ftKJ55SlcGGZfbDa jBawsSf7AUXCfRCxIluzdno+mafFszhLYoV7FahRcde1d5DCpJU7Y7hMoFrMJpVnJZNC ZcCiTdqLP1n1i0UI9OxPjGUBqrgy2hvq8zaNVy76gNgARBB/oJLfnNp30XaydSb3FXMx 1cH6okiO95nwslQzplGwqKVzo8RcCWQ8PVyEkMIHW7LQI+cl88YKlbNsOva4qZ614ZS2 hLonSIBr70WsYSgQbvuvy78FLDkrgoLgZJGlDYoVHTZzNIWv2AN49p3ZLxai7LqcJXav y9Cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0v8cEFkv006H/orJHvg5H7puUbhcFvgGtNNhP7STOH0=; b=OFsCtVSWQYSEUjwt8fe3C+Fh5pC9YUDbcpDSZfZK4c9dtu6lO3fcP/s+5W0+qdcueY +tLPRFhy9sPYb7/v3fEvV0au0ObNN8k06ZLPyNc51phlr+nbxoY+PcsA2WfEu99VrVj/ WpAVp/nXyKV4e/2nwJsX0+QkxdnOsJD13FDYN0537QLm+VEJ4ig9G2OrSGTS/eJ9i/L0 L/EjhJ3omLPbiJbd2t1Vc7Dg8wMwxunaSsjgb4ZB6BgiEIid55Jh5qjzPDBoX0ZY3uaI ELR30VcukNmb2YLVwsTUrn2h2+MU3yNav9FOvDo1Rial85Q5gf/EVqIBHmb6+3/rvT5L UDXw== X-Gm-Message-State: AOAM533VOLie7aUxaLiqahUWGOZF7A5ELLEfOZt2i+mllZpBoDhz18Bb Wi4Dm6y4sYprS5uzQA1v6P/5xOUSTgce3A== X-Google-Smtp-Source: ABdhPJxodChBJ6f8oDeAnkcjiQsUaWNS+KK9LGGhQxaK4kbSw1XKCZe42gB2XUNidA9gSk8JvJk4ZQ== X-Received: by 2002:a7b:c453:: with SMTP id l19mr1403751wmi.50.1602816037248; Thu, 15 Oct 2020 19:40:37 -0700 (PDT) Received: from localhost ([2a02:168:96c5:1:55ed:514f:6ad7:5bcc]) by smtp.gmail.com with ESMTPSA id a17sm1550065wra.29.2020.10.15.19.40.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Oct 2020 19:40:36 -0700 (PDT) From: Jann Horn To: Andrew Morton , linux-mm@kvack.org, Eric Biederman , Oleg Nesterov Cc: linux-kernel@vger.kernel.org, Will Deacon , Kees Cook , Ingo Molnar Subject: [RFC PATCH 3/6] mm: Add refcount for preserving mm_struct without pgd Date: Fri, 16 Oct 2020 04:40:16 +0200 Message-Id: <20201016024019.1882062-4-jannh@google.com> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201016024019.1882062-1-jannh@google.com> References: <20201016024019.1882062-1-jannh@google.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, mm_struct has two refcounts: - mm_users: preserves everything - the mm_struct, the page tables, the memory mappings, and so on - mm_count: preserves the mm_struct and pgd However, there are three types of users of mm_struct: 1. users that want page tables, memory mappings and so on 2. users that want to preserve the pgd (for lazy TLB) 3. users that just want to keep the mm_struct itself around (e.g. for mmget_not_zero() or __ptrace_may_access()) Dropping mm_count references can be messy because dropping mm_count to zero deletes the pgd, which takes the pgd_lock on x86, meaning it doesn't work from RCU callbacks (which run in IRQ context). In those cases, mmdrop_async() must be used to punt the invocation of __mmdrop() to workqueue context. That's fine when mmdrop_async() is a rare case, but the preceding patch "ptrace: Keep mm around after exit_mm() for __ptrace_may_access()" makes it the common case; we should probably avoid punting freeing to workqueue context all the time if we can avoid it? To resolve this, add a third refcount that just protects the mm_struct and the user_ns it points to, and which can be dropped with synchronous freeing from (almost) any context. Signed-off-by: Jann Horn --- arch/x86/kernel/tboot.c | 2 ++ drivers/firmware/efi/efi.c | 2 ++ include/linux/mm_types.h | 13 +++++++++++-- include/linux/sched/mm.h | 13 +++++++++++++ kernel/fork.c | 14 ++++++++++---- mm/init-mm.c | 2 ++ 6 files changed, 40 insertions(+), 6 deletions(-) diff --git a/arch/x86/kernel/tboot.c b/arch/x86/kernel/tboot.c index 992fb1415c0f..b92ea1bb3bb9 100644 --- a/arch/x86/kernel/tboot.c +++ b/arch/x86/kernel/tboot.c @@ -19,6 +19,7 @@ #include #include #include +#include #include #include @@ -93,6 +94,7 @@ static struct mm_struct tboot_mm = { .pgd = swapper_pg_dir, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), + .mm_bare_refs = REFCOUNT_INIT(1), MMAP_LOCK_INITIALIZER(init_mm) .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c index 3aa07c3b5136..3b73a0717c6e 100644 --- a/drivers/firmware/efi/efi.c +++ b/drivers/firmware/efi/efi.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include @@ -54,6 +55,7 @@ struct mm_struct efi_mm = { .mm_rb = RB_ROOT, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), + .mm_bare_refs = REFCOUNT_INIT(1), MMAP_LOCK_INITIALIZER(efi_mm) .page_table_lock = __SPIN_LOCK_UNLOCKED(efi_mm.page_table_lock), .mmlist = LIST_HEAD_INIT(efi_mm.mmlist), diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index ed028af3cb19..764d251966c7 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -429,13 +429,22 @@ struct mm_struct { /** * @mm_count: The number of references to &struct mm_struct - * (@mm_users count as 1). + * including its pgd (@mm_users count as 1). * * Use mmgrab()/mmdrop() to modify. When this drops to 0, the - * &struct mm_struct is freed. + * pgd is freed. */ atomic_t mm_count; + /** + * @mm_bare_refs: The number of references to &struct mm_struct + * that preserve no page table state whatsoever (@mm_count + * counts as 1). + * + * When this drops to 0, the &struct mm_struct is freed. + */ + refcount_t mm_bare_refs; + /** * @has_pinned: Whether this mm has pinned any pages. This can * be either replaced in the future by @pinned_vm when it diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index f889e332912f..e5b236e15757 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -109,6 +109,19 @@ extern void mmput(struct mm_struct *); void mmput_async(struct mm_struct *); #endif +static inline void mm_ref(struct mm_struct *mm) +{ + refcount_inc(&mm->mm_bare_refs); +} + +void __mm_unref(struct mm_struct *mm); + +static inline void mm_unref(struct mm_struct *mm) +{ + if (refcount_dec_and_test(&mm->mm_bare_refs)) + __mm_unref(mm); +} + /* Grab a reference to a task's mm, if it is not already going away */ extern struct mm_struct *get_task_mm(struct task_struct *task); /* diff --git a/kernel/fork.c b/kernel/fork.c index 4942428a217c..fcdd1ace79e4 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -642,10 +642,16 @@ static void check_mm(struct mm_struct *mm) #define allocate_mm() (kmem_cache_alloc(mm_cachep, GFP_KERNEL)) #define free_mm(mm) (kmem_cache_free(mm_cachep, (mm))) +void __mm_unref(struct mm_struct *mm) +{ + put_user_ns(mm->user_ns); + free_mm(mm); +} + /* - * Called when the last reference to the mm + * Called when the last PGD-preserving reference to the mm * is dropped: either by a lazy thread or by - * mmput. Free the page directory and the mm. + * mmput. Free the page directory. */ void __mmdrop(struct mm_struct *mm) { @@ -656,8 +662,7 @@ void __mmdrop(struct mm_struct *mm) destroy_context(mm); mmu_notifier_subscriptions_destroy(mm); check_mm(mm); - put_user_ns(mm->user_ns); - free_mm(mm); + mm_unref(mm); } EXPORT_SYMBOL_GPL(__mmdrop); @@ -1007,6 +1012,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mm->vmacache_seqnum = 0; atomic_set(&mm->mm_users, 1); atomic_set(&mm->mm_count, 1); + refcount_set(&mm->mm_bare_refs, 1); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); mm->core_state = NULL; diff --git a/mm/init-mm.c b/mm/init-mm.c index 3a613c85f9ed..3c3cd35236fd 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -31,6 +32,7 @@ struct mm_struct init_mm = { .pgd = swapper_pg_dir, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), + .mm_bare_refs = REFCOUNT_INIT(1), MMAP_LOCK_INITIALIZER(init_mm) .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), From patchwork Fri Oct 16 02:40:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 11840499 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C43E314B4 for ; Fri, 16 Oct 2020 02:40:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7916C20897 for ; Fri, 16 Oct 2020 02:40:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="sdfx+7bU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7916C20897 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7BC7B94000B; Thu, 15 Oct 2020 22:40:41 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 744CE900002; Thu, 15 Oct 2020 22:40:41 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60D2C94000B; Thu, 15 Oct 2020 22:40:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0227.hostedemail.com [216.40.44.227]) by kanga.kvack.org (Postfix) with ESMTP id 2EBF4900002 for ; Thu, 15 Oct 2020 22:40:41 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id CB195180AD811 for ; Fri, 16 Oct 2020 02:40:40 +0000 (UTC) X-FDA: 77376235440.07.form17_2f0754a27219 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id B05CF1803F9A8 for ; Fri, 16 Oct 2020 02:40:40 +0000 (UTC) X-Spam-Summary: 1,0,0,36d2baceac04a3b9,d41d8cd98f00b204,jannh@google.com,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2194:2196:2199:2200:2393:2559:2562:2693:2903:3138:3139:3140:3141:3142:3152:3352:3865:3867:3868:3871:3872:3874:4250:4321:4385:5007:6119:6120:6261:6653:7903:8660:9969:10004:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12895:12986:13069:13148:13161:13229:13230:13311:13357:13894:14096:14181:14384:14721:21080:21433:21444:21627:21939:30054,0,RBL:209.85.128.66:@google.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100;04yghotbjsju1ronz7wmkn4wzw3mmocou11nhwurz5psuipoopz7k9q673ucu6o.uu7o4914mzne3xhpgjqk17bt1qoizqybxwyedqa1ojczskj4bjjczrprxu9bbxg.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: form17_2f0754a27219 X-Filterd-Recvd-Size: 4649 Received: from mail-wm1-f66.google.com (mail-wm1-f66.google.com [209.85.128.66]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Fri, 16 Oct 2020 02:40:40 +0000 (UTC) Received: by mail-wm1-f66.google.com with SMTP id a72so934036wme.5 for ; Thu, 15 Oct 2020 19:40:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9hiZDiQ/TR8ehnjVeOHrF6PP0oyaC/YML4vUQjMFIVA=; b=sdfx+7bUGp3cRgeTtdQ816RouRn8j6SP7HsK5d0gZuur4EGmaW0O6FljoLA0m5OoNq M7DmZ2odccv8yyAuoF+h0TtyDy1f1mlKs/jIHC9peBhE5Q1KFK5pZamwLxXowCisoDj9 Q1sG5IuqOrG1ldID+7rHXZ8TshzViT65ThWtRwTdNmLiQt/FwHD0WbfUpjqPsM+eSUrT yy9wAPTM14hjTOyqbnGbln1lWVJyoyE+kZ5/StaNoA38OzLLCp3E6f5M5x3V7zD91BHb hFXZoQU3reKsdzc1zsedRDOR/VZo16BlDf1Z2jR9wrZxDuGcj+cqpPYbQNLsL1xn3DhR PCmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9hiZDiQ/TR8ehnjVeOHrF6PP0oyaC/YML4vUQjMFIVA=; b=Ipcm0XXuSM9gE5Oe4LVLoA6p6PicB5OMiDHd+0hK0pmKb1Z2NmpDEJwS2xQz8r/BVB jptIOoU7jcQGKnjeqcDl5zdgdZBqLan/078SHPDvQmn+zSyCqmlLEuig8Ngtc/Vy6r0c b7wIh1sUyQwvCqnsMi6kHmlTzJcanNK5gOYKcQ3/m2FtK3mAeRS/pc9LuqmAVAzTG3uO Jg4A6hZ8f9RbMjbocpF7/ANvomckUqOCcRewFbmvWAlD/x2zOH3T5QbyY6nzhRSOzvpO 8Y+3lqItjHO2fAnorbXoJ3ip2eIQRttAhH3Jr2w0vR/4UiclSzemX7HOyeSOh0xPlcwi qivA== X-Gm-Message-State: AOAM533iRZ5MvlZc8uXZSvPF4jB3hY1/Ffa3w3HE50Y7wSnx4llNmPce MHMzkbOa+U0vrSAikgNEmSJIJPwUW0L9Dw== X-Google-Smtp-Source: ABdhPJwUZByfFbdIxgdaG6l/XtUPqQBgTpgUOUyknpODLVNPNySmarleCGv1ABqEejnzZl7lKAm85A== X-Received: by 2002:a1c:3503:: with SMTP id c3mr1358021wma.43.1602816039091; Thu, 15 Oct 2020 19:40:39 -0700 (PDT) Received: from localhost ([2a02:168:96c5:1:55ed:514f:6ad7:5bcc]) by smtp.gmail.com with ESMTPSA id m14sm1479195wro.43.2020.10.15.19.40.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Oct 2020 19:40:37 -0700 (PDT) From: Jann Horn To: Andrew Morton , linux-mm@kvack.org, Eric Biederman , Oleg Nesterov Cc: linux-kernel@vger.kernel.org, Will Deacon , Kees Cook , Ingo Molnar Subject: [RFC PATCH 4/6] mm, oom: Use mm_ref()/mm_unref() and avoid mmdrop_async() Date: Fri, 16 Oct 2020 04:40:17 +0200 Message-Id: <20201016024019.1882062-5-jannh@google.com> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201016024019.1882062-1-jannh@google.com> References: <20201016024019.1882062-1-jannh@google.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The OOM killer uses MMF_OOM_SKIP to avoid running on an mm that has started __mmput(); it only uses the mmgrab() reference to ensure that the mm_struct itself stays alive. This means that we don't need a full mmgrab() reference, which will keep the pgd (and potentially also some pmd pages) alive and can't be cleaned up from RCU callback context; we can use an mm_ref() reference instead. Signed-off-by: Jann Horn --- kernel/fork.c | 6 +----- mm/oom_kill.c | 2 +- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index fcdd1ace79e4..59c119b03351 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -686,12 +686,8 @@ static inline void free_signal_struct(struct signal_struct *sig) { taskstats_tgid_free(sig); sched_autogroup_exit(sig); - /* - * __mmdrop is not safe to call from softirq context on x86 due to - * pgd_dtor so postpone it to the async context - */ if (sig->oom_mm) - mmdrop_async(sig->oom_mm); + mm_unref(sig->oom_mm); kmem_cache_free(signal_cachep, sig); } diff --git a/mm/oom_kill.c b/mm/oom_kill.c index e90f25d6385d..12967f54fbcf 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -704,7 +704,7 @@ static void mark_oom_victim(struct task_struct *tsk) /* oom_mm is bound to the signal struct life time. */ if (!cmpxchg(&tsk->signal->oom_mm, NULL, mm)) { - mmgrab(tsk->signal->oom_mm); + mm_ref(tsk->signal->oom_mm); set_bit(MMF_OOM_VICTIM, &mm->flags); } From patchwork Fri Oct 16 02:40:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 11840501 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A8F0C16BC for ; Fri, 16 Oct 2020 02:40:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5AAB3208E4 for ; Fri, 16 Oct 2020 02:40:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="MddfLAWG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5AAB3208E4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9680694000C; Thu, 15 Oct 2020 22:40:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8F627900002; Thu, 15 Oct 2020 22:40:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7BE1E94000C; Thu, 15 Oct 2020 22:40:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0171.hostedemail.com [216.40.44.171]) by kanga.kvack.org (Postfix) with ESMTP id 37895900002 for ; Thu, 15 Oct 2020 22:40:43 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C6DE63623 for ; Fri, 16 Oct 2020 02:40:42 +0000 (UTC) X-FDA: 77376235524.28.house30_4e1717a27219 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id AA4646C0B for ; Fri, 16 Oct 2020 02:40:42 +0000 (UTC) X-Spam-Summary: 1,0,0,713e4ae39f2622c3,d41d8cd98f00b204,jannh@google.com,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3152:3352:3865:3866:3867:3868:3871:3872:3874:4250:4321:4385:5007:6119:6120:6261:6653:7903:9969:10004:11026:11658:11914:12043:12048:12296:12297:12517:12519:12555:12895:13069:13311:13357:13894:14096:14181:14384:14721:14877:21080:21324:21444:21627:21939:30054,0,RBL:209.85.221.68:@google.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100;04yr3h3ufaiu3kqrxtek5r793jmmnyc4qygrnss35wufphkbdcbwyxf6xzi4u8w.oidoxsdzq1938ab8yqsxhy96sttya7qnhmzm7mtwd6a9q6et6qqmiqpdoos9j8q.n-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: house30_4e1717a27219 X-Filterd-Recvd-Size: 4396 Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Fri, 16 Oct 2020 02:40:42 +0000 (UTC) Received: by mail-wr1-f68.google.com with SMTP id i1so964720wro.1 for ; Thu, 15 Oct 2020 19:40:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IcagjNj/dK1Y8RczpuQCkX5yBuEwqqFKdmDjpHwB/uo=; b=MddfLAWGyYBRfNuN1ok8b6DU3DiyooyjcqJoJfOhV+LJKWza09r9Go8ofj1LC/QisZ RX1U7FiWILjd8tJ4OPgtOUEDISb8w3/bkgvVAJMxqUZUHmQ+HhuSYJFp/Ralp9ECoDZq SMtrG+132hhR9UTmdkq2zSCiGcjHv1krB4OeKFj1QSfxQBK595gy4ZNGwhxkNu3BZxOK bn81x1r9B6DOkbjCVmkPpnJ/P0Hiju8Dg99yreeh5ABC2m7yGbRjs7seprwLuOdcEt5s My5XZ+XMcAOZGWsPD45667KU2tZIhqcxBWzE2Ng//a54J+Tl1nGWSutjgrXIQcYbJuar Ctig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IcagjNj/dK1Y8RczpuQCkX5yBuEwqqFKdmDjpHwB/uo=; b=qmOD+nnR31F0twWuENt4b8uxZa7aknSYV4J0jazHzdwQ4xsIrsq+Dwk9lGTsowGaDn Kjfnyg8wKE6vvFg7rhYIAQ5pQGVCZCYR9u/H+ryBdKYxmRKwSeo5hXBqDOBEt65Mg3gH bmjUBqfL+mP6EqYkKEp5fTOldJCCI8v3zFtlX0G3qZLrb4l4cOmpbmdnkzHR/jRx3a33 XfV692yxY9KaQ6yCMynV2GdL7DLb1aul8YEAOvRN5f+62OLKP4TyybxBMjZEzCP9QOJx PUTmgtUuCEHRlJ2jCtyRjkae/pPJNCk+kWS52E4PIB2s9KGF73fIsbRBz2mEy55hvlq+ pScw== X-Gm-Message-State: AOAM530wefJ+VF1Q7Qj5eLmtjmn/QW5eEMqS3i1zqDfSqA/TXM+dwcQT DjzT/fXXuAKz4iRXp79Bq/GeIA== X-Google-Smtp-Source: ABdhPJximA8TmzkmMulZ0xLdHIxKgoI2ci4m2SYAs/YeDnDT1bJXj91PiKhT9FEUOj80FiKoKk0j4Q== X-Received: by 2002:adf:f74e:: with SMTP id z14mr1179844wrp.312.1602816040727; Thu, 15 Oct 2020 19:40:40 -0700 (PDT) Received: from localhost ([2a02:168:96c5:1:55ed:514f:6ad7:5bcc]) by smtp.gmail.com with ESMTPSA id z5sm1511651wrw.37.2020.10.15.19.40.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Oct 2020 19:40:40 -0700 (PDT) From: Jann Horn To: Andrew Morton , linux-mm@kvack.org, Eric Biederman , Oleg Nesterov Cc: linux-kernel@vger.kernel.org, Will Deacon , Kees Cook , Ingo Molnar Subject: [RFC PATCH 5/6] ptrace: Use mm_ref() for ->exit_mm Date: Fri, 16 Oct 2020 04:40:18 +0200 Message-Id: <20201016024019.1882062-6-jannh@google.com> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201016024019.1882062-1-jannh@google.com> References: <20201016024019.1882062-1-jannh@google.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We only use ->exit_mm to look up dumpability and the ->user_mm; we don't need to keep the PGD alive for this. mmgrab() is also inconvenient here, because it means that we need to use mmdrop_async() when dropping the reference to the mm from an RCU callback. Use mm_ref() instead of mmgrab() to make things neater. Signed-off-by: Jann Horn --- kernel/exit.c | 2 +- kernel/fork.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/exit.c b/kernel/exit.c index 97253ef33486..03ba6d13ef1e 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -476,7 +476,7 @@ static void exit_mm(void) /* more a memory barrier than a real lock */ task_lock(current); current->mm = NULL; - mmgrab(mm); /* for current->exit_mm */ + mm_ref(mm); /* for current->exit_mm */ current->exit_mm = mm; mmap_read_unlock(mm); enter_lazy_tlb(mm, current); diff --git a/kernel/fork.c b/kernel/fork.c index 59c119b03351..4383bf055b40 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -720,7 +720,7 @@ void free_task(struct task_struct *tsk) if (tsk->flags & PF_KTHREAD) free_kthread_struct(tsk); if (tsk->exit_mm) - mmdrop_async(tsk->exit_mm); + mm_unref(tsk->exit_mm); free_task_struct(tsk); } EXPORT_SYMBOL(free_task); From patchwork Fri Oct 16 02:40:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 11840503 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 795AC14B4 for ; Fri, 16 Oct 2020 02:40:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 356B22087D for ; Fri, 16 Oct 2020 02:40:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="UzPVodG7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 356B22087D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C264994000D; Thu, 15 Oct 2020 22:40:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BD5FD900002; Thu, 15 Oct 2020 22:40:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A288B94000D; Thu, 15 Oct 2020 22:40:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0141.hostedemail.com [216.40.44.141]) by kanga.kvack.org (Postfix) with ESMTP id 750CB900002 for ; Thu, 15 Oct 2020 22:40:44 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 0FE3A1EE6 for ; Fri, 16 Oct 2020 02:40:44 +0000 (UTC) X-FDA: 77376235608.06.grain72_030470127219 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id E4B0B10247F4E for ; Fri, 16 Oct 2020 02:40:43 +0000 (UTC) X-Spam-Summary: 1,0,0,979ee92b861f1a21,d41d8cd98f00b204,jannh@google.com,,RULES_HIT:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3152:3352:3865:3866:3870:3871:3872:3874:4321:4385:4605:5007:6120:6261:6653:7903:9592:9969:10004:11026:11232:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12895:12986:13069:13161:13180:13229:13311:13357:13894:14096:14181:14384:14721:21080:21444:21627:21740:30054,0,RBL:209.85.128.65:@google.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100;04y83g4fepnr7hx94mdr8wcixt4qpop3rfdcx6ba3844f5eqza5jh5n8fwfp6cd.bpbxa91xqchsrfbqwaq1k9w4tsf8n9iwqaourcabkpq6unadjg7wqbwo8p6ext8.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: grain72_030470127219 X-Filterd-Recvd-Size: 4716 Received: from mail-wm1-f65.google.com (mail-wm1-f65.google.com [209.85.128.65]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Fri, 16 Oct 2020 02:40:43 +0000 (UTC) Received: by mail-wm1-f65.google.com with SMTP id f21so946041wml.3 for ; Thu, 15 Oct 2020 19:40:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MM4vknTStYRLkGZw4tEHhTyBDfVUaCowXRmIC2TIp+c=; b=UzPVodG7vA2ic1HH1rFJ9ZFcjIcafLiynej1hsfsW7Kk8wfH5gNDzFpgzEjh3lN58l k0dAp9bXU+cG3pzw3tHSGbuYEk2L7mtIYHUFm+JeveAYlOMp75+XCbhV8IOZtonrBENQ N3EuUEXsABr3wvBaELeeMCVuFKi/re3WS+eoQMSP1pOGXHnkEyafFQENqCfVkz8ZBh12 LEr/np5tTIRUXaDrf3a6Zu5AOPiFCvfr+ogI139Xbc2dsROxhWd1zDR87GC3eWDxiU15 Mgm+yz3mfYc2ggE+SKeJcFy7gWvizCM5fduQMpvrjIhdvXokKcUaTd0xrhi19exO0wq1 CZQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MM4vknTStYRLkGZw4tEHhTyBDfVUaCowXRmIC2TIp+c=; b=UTNXdPAUEPBDFJUm0M+A4cRO2oW9WV1YS96iHzt+ggU6p5ONS59f2smdrRIlOCgHrY tbITnFQSb8Ew30VqzOipU6LimFSrGUzZiBJQFpRsgQhzYMf1QjmcW5qKD900es5848BD HQVBkMhD84POpULfRFGu3Avf8olI3UAoWXgbIvWh/DJ9F0n2+bTNBk56tpKS6dGbnUUj G7CB7hBVRCoXvvs9l49v9K4sleedO1ieXfMYabOjlG1uvp2zRuuHuMoDa14iRPB/CdnF CxcQhPCTaMfozX/x9sApaUU7CgN1ZaQg8q3+4DrqJQBNZ41R5LlOzVAvHGvWNl0yusMn to5w== X-Gm-Message-State: AOAM532azoKWyqE3vvHXAos5bqAvbCUmUvobNYowKEOrH53M/eUNKqw7 S+5NWJzEKovuin/tnhmSdfIsAbUn/epB/A== X-Google-Smtp-Source: ABdhPJzhEFMFksHr3z4YKmc2OKnKI/GGyeg8SX7OdlCeHKjZMM8NAVXhzJhEFVJQFXw7prbPniY0DA== X-Received: by 2002:a7b:c451:: with SMTP id l17mr1421587wmi.127.1602816042197; Thu, 15 Oct 2020 19:40:42 -0700 (PDT) Received: from localhost ([2a02:168:96c5:1:55ed:514f:6ad7:5bcc]) by smtp.gmail.com with ESMTPSA id p9sm990595wma.12.2020.10.15.19.40.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Oct 2020 19:40:41 -0700 (PDT) From: Jann Horn To: Andrew Morton , linux-mm@kvack.org, Eric Biederman , Oleg Nesterov Cc: linux-kernel@vger.kernel.org, Will Deacon , Kees Cook , Ingo Molnar Subject: [RFC PATCH 6/6] mm: remove now-unused mmdrop_async() Date: Fri, 16 Oct 2020 04:40:19 +0200 Message-Id: <20201016024019.1882062-7-jannh@google.com> X-Mailer: git-send-email 2.29.0.rc1.297.gfa9743e501-goog In-Reply-To: <20201016024019.1882062-1-jannh@google.com> References: <20201016024019.1882062-1-jannh@google.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The preceding patches have removed all users of mmdrop_async(); get rid of it. Note that on MMU, we still need async_put_work because mmput_async() uses it, which in turn is used by binder's shrinker callback. We could claw back those 4 words per mm if we made mmput_async() depend on CONFIG_ANDROID_BINDER_IPC. Signed-off-by: Jann Horn --- include/linux/mm_types.h | 2 ++ kernel/fork.c | 16 ---------------- 2 files changed, 2 insertions(+), 16 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 764d251966c7..8fde2068bde1 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -560,7 +560,9 @@ struct mm_struct { #ifdef CONFIG_HUGETLB_PAGE atomic_long_t hugetlb_usage; #endif +#ifdef CONFIG_MMU struct work_struct async_put_work; +#endif } __randomize_layout; /* diff --git a/kernel/fork.c b/kernel/fork.c index 4383bf055b40..c5f2ec544933 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -666,22 +666,6 @@ void __mmdrop(struct mm_struct *mm) } EXPORT_SYMBOL_GPL(__mmdrop); -static void mmdrop_async_fn(struct work_struct *work) -{ - struct mm_struct *mm; - - mm = container_of(work, struct mm_struct, async_put_work); - __mmdrop(mm); -} - -static void mmdrop_async(struct mm_struct *mm) -{ - if (unlikely(atomic_dec_and_test(&mm->mm_count))) { - INIT_WORK(&mm->async_put_work, mmdrop_async_fn); - schedule_work(&mm->async_put_work); - } -} - static inline void free_signal_struct(struct signal_struct *sig) { taskstats_tgid_free(sig);