From patchwork Wed Apr 7 01:44:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C343C43460 for ; Wed, 7 Apr 2021 01:46:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AB80C61165 for ; Wed, 7 Apr 2021 01:46:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AB80C61165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CD7F18E000C; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 14ACB8E000A; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16EE38E000F; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0021.hostedemail.com [216.40.44.21]) by kanga.kvack.org (Postfix) with ESMTP id B61DF8D0009 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6F06081D1 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.06.962C992 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf22.hostedemail.com (Postfix) with ESMTP id 36958C0007CA for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=4ziyv/pIXZ+c0Q5G5PWzYQBRlES+IZwjStJKbTMM+os=; b=1zfU2U/hZRjQOfQbf+ngGzUxUD0CAmdlCfE75iKwCJji3T80V31VCIjGrlAz4Vg2/qnUR cLMt72B2etEV5ErCA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=4ziyv/pIXZ+c0Q5G5PWzYQBRlES+IZwjStJKbTMM+os=; b=Zk4enQyEEfaPQoXqLFfW8qu8XPvJ29ctA8eQmIOu5f7cy23VXpXwFyszzj7f/B4A5c+I/ T7XEBxP5iCS4irHJX9swHIlVK9DWYUe+gnQn6KljNjZOzg5yZ16CPVI3wqpxLm4Ugn/w4VE vw38o6Ovajp+16OwAsnx9pl7eoq3wSkeokZSFYFNjcOStaJd005IjYgCf5yzpOcVgY6+kPn qzJKLnN2RLPyB7Q2I2eroqhgkEI0TZpRrB+Y74rRmtcb/E6ldd2NZIY6004/l9BS8lp5Spb vUVer8Bak0nHBymTqRtUxDOEqcnxKHoYW0EZOExcseVtnKJW9tXHTh6KJywQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 10E05160369; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 0272C19F31E; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 23/37] mm: rcu safe vma->vm_file freeing Date: Tue, 6 Apr 2021 18:44:48 -0700 Message-Id: <20210407014502.24091-24-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 36958C0007CA X-Stat-Signature: xqhqq4p9dur8su9gqx99exd65hosdfgo X-Rspamd-Server: rspam02 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf22; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759906-461697 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Defer freeing of vma->vm_file when freeing vmas. This is to allow speculative page faults in the mapped file case. Signed-off-by: Michel Lespinasse Reported-by: kernel test robot --- fs/exec.c | 1 + kernel/fork.c | 17 +++++++++++++++-- mm/mmap.c | 11 +++++++---- mm/nommu.c | 6 ++---- 4 files changed, 25 insertions(+), 10 deletions(-) diff --git a/fs/exec.c b/fs/exec.c index 18594f11c31f..c9da73eb0f53 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -286,6 +286,7 @@ static int __bprm_mm_init(struct linux_binprm *bprm) mmap_write_unlock(mm); err_free: bprm->vma = NULL; + VM_BUG_ON(vma->vm_file); vm_area_free(vma); return err; } diff --git a/kernel/fork.c b/kernel/fork.c index b6078e546114..2f20a5c5fed8 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -369,19 +369,31 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } +static inline void ____vm_area_free(struct vm_area_struct *vma) +{ + if (vma->vm_file) + fput(vma->vm_file); + kmem_cache_free(vm_area_cachep, vma); +} + #ifdef CONFIG_SPECULATIVE_PAGE_FAULT static void __vm_area_free(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); - kmem_cache_free(vm_area_cachep, vma); + ____vm_area_free(vma); } +#endif + void vm_area_free(struct vm_area_struct *vma) { +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT call_rcu(&vma->vm_rcu, __vm_area_free); +#else + ____vm_area_free(vma); +#endif } -#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ static void account_kernel_stack(struct task_struct *tsk, int account) { @@ -621,6 +633,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, fail_nomem_anon_vma_fork: mpol_put(vma_policy(tmp)); fail_nomem_policy: + tmp->vm_file = NULL; /* prevents fput within vm_area_free() */ vm_area_free(tmp); fail_nomem: retval = -ENOMEM; diff --git a/mm/mmap.c b/mm/mmap.c index 3f287599a7a3..cc2323e243bb 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -178,9 +178,8 @@ static struct vm_area_struct *remove_vma(struct vm_area_struct *vma) might_sleep(); if (vma->vm_ops && vma->vm_ops->close) vma->vm_ops->close(vma); - if (vma->vm_file) - fput(vma->vm_file); mpol_put(vma_policy(vma)); + /* fput(vma->vm_file) happens in vm_area_free after an RCU delay. */ vm_area_free(vma); return next; } @@ -949,7 +948,8 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, if (remove_next) { if (file) { uprobe_munmap(next, next->vm_start, next->vm_end); - fput(file); + /* fput(file) happens whthin vm_area_free(next) */ + VM_BUG_ON(file != next->vm_file); } if (next->anon_vma) anon_vma_merge(vma, next); @@ -1828,7 +1828,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, * fput the vma->vm_file here or we would add an extra fput for file * and cause general protection fault ultimately. */ - fput(vma->vm_file); + /* fput happens within vm_area_free */ vm_area_free(vma); vma = merge; /* Update vm_flags to pick up the change. */ @@ -1907,6 +1907,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, if (vm_flags & VM_DENYWRITE) allow_write_access(file); free_vma: + VM_BUG_ON(vma->vm_file); vm_area_free(vma); unacct_error: if (charged) @@ -2779,6 +2780,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma, out_free_mpol: mpol_put(vma_policy(new)); out_free_vma: + new->vm_file = NULL; /* prevents fput within vm_area_free() */ vm_area_free(new); return err; } @@ -3343,6 +3345,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, out_free_mempol: mpol_put(vma_policy(new_vma)); out_free_vma: + new_vma->vm_file = NULL; /* Prevent fput within vm_area_free */ vm_area_free(new_vma); out: return NULL; diff --git a/mm/nommu.c b/mm/nommu.c index 5c9ab799c0e6..06a0dc0b913b 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -664,9 +664,8 @@ static void delete_vma(struct mm_struct *mm, struct vm_area_struct *vma) { if (vma->vm_ops && vma->vm_ops->close) vma->vm_ops->close(vma); - if (vma->vm_file) - fput(vma->vm_file); put_nommu_region(vma->vm_region); + /* fput(vma->vm_file) happens within vm_area_free() */ vm_area_free(vma); } @@ -1267,8 +1266,7 @@ unsigned long do_mmap(struct file *file, if (region->vm_file) fput(region->vm_file); kmem_cache_free(vm_region_jar, region); - if (vma->vm_file) - fput(vma->vm_file); + /* fput(vma->vm_file) happens within vm_area_free() */ vm_area_free(vma); return ret;