From patchwork Fri Apr 26 14:45:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13644922 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CF77C04FFE for ; Fri, 26 Apr 2024 14:45:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BBC596B009C; Fri, 26 Apr 2024 10:45:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B6CAE6B009D; Fri, 26 Apr 2024 10:45:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A36876B009E; Fri, 26 Apr 2024 10:45:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7FF5C6B009C for ; Fri, 26 Apr 2024 10:45:24 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 32824120970 for ; Fri, 26 Apr 2024 14:45:24 +0000 (UTC) X-FDA: 82051956168.13.6D0A480 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf09.hostedemail.com (Postfix) with ESMTP id B535B14001C for ; Fri, 26 Apr 2024 14:45:22 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=RyyF7zdw; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714142722; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WfdYXkl3VmeSNFl8CeYkQkRSot5hSq2wAYXQXVO2q28=; b=O6lj6WBflZGSPkYaYJrPWXJbC4kkHfBri75RsXshQsSWKNYUMKLhRSJMSO64JtEJ/uss3Q J6AM8M9OHrXGmMrEkV01kun7u+fJ7hZCyD0fEt22zss+NUZ5YBxVDr4Tk8/kGWZJpbGO1m BfwwYWrPB4mj+qXahKP3gUm6OhYOCGg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714142722; a=rsa-sha256; cv=none; b=wtOfK2FQKS9LghzHLuc0ILrjvGtYOCaGeQ9yU7/wl5Gcj1iqZPt7oaQ2NDKINXgH0dz+3k mWasVvFfIGnBVvRwMZE6ZhytFQCdVG1OxqXH2LFdzm88eZSbORFyC8RBKfV785k8eFToG9 J6CvU66hsvtPLxnUIZR3F35IpdKbbsU= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=RyyF7zdw; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=WfdYXkl3VmeSNFl8CeYkQkRSot5hSq2wAYXQXVO2q28=; b=RyyF7zdwWk9nDa97+eyubbdcGh /w28kPOhTqJkoKMGtwC13r+m64VxEWsAi8TiLuVDtxQS2gSkWj39hwiAkuY0x1wxCTh45vs8B9FTn sD30juuwToyG9roIfdS9dV8KS47Mv8h4eldpZC0MIv0XWa+zSjDgFtCb0N0CH3m5CjLA6W/+Ermsv vCRso3N6rfSCVH2Va/m/EBte6Z/aKqUBvmbYVHj936Hjj3wYAuJc77vTg2oam8yNVg16gv4t17js7 4OfTtjSbHss3Gw2WztfJrsDnkdMN5ObcpwwKZPNiMEVbnV6Pcu+aigMm//iKd9fRHCaGg13oL7Du/ /4x72pfw==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1s0Mp2-00000005Pl0-0bLw; Fri, 26 Apr 2024 14:45:08 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Suren Baghdasaryan , Peter Xu Subject: [PATCH 1/4] mm: Assert the mmap_lock is held in __anon_vma_prepare() Date: Fri, 26 Apr 2024 15:45:00 +0100 Message-ID: <20240426144506.1290619-2-willy@infradead.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240426144506.1290619-1-willy@infradead.org> References: <20240426144506.1290619-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: B535B14001C X-Stat-Signature: in4dudizmhq56w1qqx7nhst47ke6cxci X-Rspam-User: X-HE-Tag: 1714142722-632503 X-HE-Meta: U2FsdGVkX1/4RuKLQXPuiSvUII3+swJDxT8kDH1CxJJau+ltmJ7ZUCRrgtENYFvtaA2uK4vOGLUjqNdzYq77sbzd2qmQEKzUlfPKu7Zj8zvKg1Gm3kQ7r7VtZ489cdqqF4C7lnAgC2WH9hjZ/MtrZT/BYC2S0LM5C0VKKI7jjdXUQApL8ZzvHjGaYR5Fro6McepPSrwsa7tHd6CbMnJRyoDHRFcHS+Xh9ZdlaoBEuVEjcJsZ4ehTaJcmOeINymXrxJSt6FdqmMmIu9uUxtBAUko3CZHYtAZ1VgKYgH0Nf6PxJyV0B41nTHKZLj1k2G2/8Iugq1PacuM7Sa0TJOR8F/qliVnWSrqfpv5je2VCnsLscxN9OsHw42GeIfMuO298Zkj9ANo+iYCyzIVA00sIXnGJMJl/c95B4cps7BKa+wtRoT5T5xk8FP0wsLrzyc9KLSRrUErhb19dPDSSnHuZmlWGA7POUaU545SRkr5r2rEg9Tq38JpOWnSlXGITkxiZAdev1YShPuWweHyQevF2mHl4KJn4dXcFXJrxW69GLe1Xb2qu9avaQzwm9zehTmnTNwfup3Vr1ZEkw0go+lLvVySsQJcnMAsKbOO7fzcyIFNphtJ/7pthgxMsXoJghwWfjk08MFgxe5M1udWo5fYxJF3sDEIkdlPuDY5R+NC4+6WtmxR6z8fxVAfD4+3s+JGU5kQ3N3xoKl3cO6/x0Rl9v5mT2jA+hNPYodNIeFaZ8cgB+to1oHFOzPKAE1c4iko59FZzDVuev278b4cXK8eGtQd62DdnfQ38OwmJBnbRzosZ3Za2J0j87zlR4A5yP/h7X5iev++AJ0XxAeUlI3gj0OWkOs+OpUgTVeHyO9+KkIT64ufKKS1GZh6GCp56SyqhFrX735Q5V6HS20kJI5FyyZyQ00NeGh9R X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Convert the comment into an assertion. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan Reviewed-by: David Hildenbrand --- mm/rmap.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 2608c40dffad..619d4d65d99b 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -182,8 +182,6 @@ static void anon_vma_chain_link(struct vm_area_struct *vma, * for the new allocation. At the same time, we do not want * to do any locking for the common case of already having * an anon_vma. - * - * This must be called with the mmap_lock held for reading. */ int __anon_vma_prepare(struct vm_area_struct *vma) { @@ -191,6 +189,7 @@ int __anon_vma_prepare(struct vm_area_struct *vma) struct anon_vma *anon_vma, *allocated; struct anon_vma_chain *avc; + mmap_assert_locked(mm); might_sleep(); avc = anon_vma_chain_alloc(GFP_KERNEL); From patchwork Fri Apr 26 14:45:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13644919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E34F8C04FFE for ; Fri, 26 Apr 2024 14:45:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 10F056B009A; Fri, 26 Apr 2024 10:45:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F8316B0099; Fri, 26 Apr 2024 10:45:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F155F6B009A; Fri, 26 Apr 2024 10:45:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D08066B0098 for ; Fri, 26 Apr 2024 10:45:14 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 81E64C1366 for ; Fri, 26 Apr 2024 14:45:14 +0000 (UTC) X-FDA: 82051955748.06.9C4B624 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id F22A112000F for ; Fri, 26 Apr 2024 14:45:12 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=D4g+WnOb; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714142713; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5xao1faSLSLg0StYuxJJ8KUL1V23wgo3qyoN0YuKNr8=; b=voYXu8CSl1b8Jr6CYRfq/3x/POhOBUS1iK365gNw4+YTAbVUUQhrq05FpsrtEJYSfpsvW+ iTm9P3se9UOnXRikdTLht8M8fe0jXPH7U7YSg6RImyTBeIFUSrlnSQSu0xwEfAO0cBPkyO bO7LMh9TfymQqZ11d9GKKvUqgAqZshk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714142713; a=rsa-sha256; cv=none; b=PUhN++aVtl73m9BpO1h98Kf+mvp6o3g8ZN3OVz2RMt64S16AmXLWHr2eLvh10MbbscrfSf ujNX/L1yRsn+IJ7OEozIsEBiCCgx+hC8Vl7WMjvN/kltIxR5R/vAYuOXehE2QO2dpPT45U FxKcvFmD4oO86tkFuApPdj2tbmk6rOI= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=D4g+WnOb; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=5xao1faSLSLg0StYuxJJ8KUL1V23wgo3qyoN0YuKNr8=; b=D4g+WnOba5ewplYbiUpqL0OM87 0MRzHx47bxn/V4Qa5spqbIR364xnBh9JfepXuF8jyGySd8A8pZ+H6r2qYfEeuerd4sOtcKzqlYOVu AabUF3EG+/v1J2JfpwcqT9db+NpVdaNLwS3vJPSgOxAU2osU2MUTnnycIY8rA8HukYnKgSdcZ4L37 E6oQejNp/b8yzkzlsHL1n5q6KiFl4wi2pfJekzlduFmafXG4LilZJC4IxIwOrx2q81lvyUa3baHKO wezsagK/b5j3avamBA5CmIZHXR79ukfVOcG463vBnwOioDbPKg2lZQ5S0VMZZ4Q4knXXJKIUN0q5a E1/1Wllw==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1s0Mp2-00000005Pl2-103M; Fri, 26 Apr 2024 14:45:08 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Suren Baghdasaryan , Peter Xu Subject: [PATCH 2/4] mm: Delay the check for a NULL anon_vma Date: Fri, 26 Apr 2024 15:45:01 +0100 Message-ID: <20240426144506.1290619-3-willy@infradead.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240426144506.1290619-1-willy@infradead.org> References: <20240426144506.1290619-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: F22A112000F X-Rspam-User: X-Stat-Signature: y5gpryzegddc6w79oo1s9ptgbiehaipm X-HE-Tag: 1714142712-986600 X-HE-Meta: U2FsdGVkX19zF6hsDwHO7kLzd2vachj0/WrFvdHDSEhsFtE1H8N9EOUQYkfbC/9O5cgMIymieMs58qW/7t5s3qlTF2wozyqMSuSmsXvdqfxnwmsGHmsLor/Hb81z4K4xA02Bcs++Wf9aSBoXYqIHZloz7slKRMurDdFAAdaR5lA3/sGvKKUbZ8mEHi0vm87TEvMFLIeVfkIUl790rvhuMYCvsSabJwF2pRuaT/Jrdinrnlesbjb3h9fpboldR6FkwGiWZc6tE9fu3qlmvumLwfCsqAMuLrbOymXxN1qpi/2J8JULvf7dRlxCxp9G+66G5ZyJ+BX1BFBt2AxSxP/yOSiIQH4FEGbtWWYUV0pdbKvI9JrYdUlWP1z9ig2KBwe+QTGRqeasvP0QS62sfEAtOcxGfHX9pQU5Xj/WYEiP6ZRGLcK+KQM/epqVjOUTYyWpkZd4O3bDuPRyNzJyjHaJlGjw/z/tlyOb0i2CtzkjcHegWzTOpw1w4HwEnAtc4MX+G6TtBMUnTzCtkyHuXKdYTtqaGCYkvfp+Mdyo2jjrqSx9Z8jOqztFrHX4qoFx118Knq1zNeoE+ynpZiCca7JHOc5p7dUWH5f7uoh6Wae7FkARq/zMk+hnWRjTjipi20cow5hy9vL7XOhJiUNPNEOWIU2oEa3kqqem9LanJT8Zj8dvsqJ09JJ+uhkvTTUPlkbrkvI2IRguQSv513KN3vImwa3+zK/9frQ//tWgMJOQjLLxbGgpZK8Z1NAHHbr6bZpi+k4IwuxQu/YDUGxscDa0ODARhhkbob8kIgBsIuARQuZUfGeVJuIOwYOawiJccy2oHMWHQraXOjMrBUrUQ4OIlMPQaIogdRypZeGw+6FRcrUS1l5hdflsZztiObQCMCimzlFQy2KPD7O13vmBAEE3gr+ImJ9oR3wYbVe61LmbuXQg9aVhRstTICODhf44oEa8y2u6x5rzTJ9QEBXVZtW 6oe+WPp+ XHHhtSWx374tihHcTJjxriwACjhtfmmSQSdLlgJP0CC3e/sXJyi4/sgCmU69hSQFkTN0hxq3JorgdqolamGFYfzZc5U5bylWdgSQcGoKwrd3sXOWJLsySJ/njzm39cUJBYU0bJOLTGYly9nzyJu5kt+33nLm8mIFopfVBdXi24fvuOuTk5iIZ+DCqRGJmAnUam4LF+96OZ5Jb0LUHGCrYSPchEnORlmzTtKMk6OM392Md6FK/NkzM2GjrmyOolTcRLZYlqOtW0PJTY7cq8FTWZk7n6sDgdk/5eT0OebJWHLonvetZe04jhEy9G6U6rpaRCBZyYlI3u8zsrqk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Instead of checking the anon_vma early in the fault path where all page faults pay the cost, delay it until we know we're going to need the anon_vma to be filled in. This will have a slight negative effect on the first fault in an anonymous VMA, but it shortens every other page fault. It also makes the code slightly cleaner as the anon and file backed fault handling look more similar. The Intel kernel test bot reports a 3x improvement in vm-scalability throughput with the small-allocs-mt test. This is clearly an extreme situation that won't be replicated in any real-world workload, but it's a nice win. https://lore.kernel.org/all/202404261055.c5e24608-oliver.sang@intel.com/ Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/huge_memory.c | 6 ++++-- mm/memory.c | 29 ++++++++++++++++++----------- 2 files changed, 22 insertions(+), 13 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 19000fc2c43c..8261b5669397 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1057,11 +1057,13 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) gfp_t gfp; struct folio *folio; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; + vm_fault_t ret; if (!thp_vma_suitable_order(vma, haddr, PMD_ORDER)) return VM_FAULT_FALLBACK; - if (unlikely(anon_vma_prepare(vma))) - return VM_FAULT_OOM; + ret = vmf_anon_prepare(vmf); + if (ret) + return ret; khugepaged_enter_vma(vma, vma->vm_flags); if (!(vmf->flags & FAULT_FLAG_WRITE) && diff --git a/mm/memory.c b/mm/memory.c index 6647685fd3c4..7dc112d3a7e4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3214,6 +3214,21 @@ static inline vm_fault_t vmf_can_call_fault(const struct vm_fault *vmf) return VM_FAULT_RETRY; } +/** + * vmf_anon_prepare - Prepare to handle an anonymous fault. + * @vmf: The vm_fault descriptor passed from the fault handler. + * + * When preparing to insert an anonymous page into a VMA from a + * fault handler, call this function rather than anon_vma_prepare(). + * If this vma does not already have an associated anon_vma and we are + * only protected by the per-VMA lock, the caller must retry with the + * mmap_lock held. __anon_vma_prepare() will look at adjacent VMAs to + * determine if this VMA can share its anon_vma, and that's not safe to + * do with only the per-VMA lock held for this VMA. + * + * Return: 0 if fault handling can proceed. Any other value should be + * returned to the caller. + */ vm_fault_t vmf_anon_prepare(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; @@ -4434,8 +4449,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) } /* Allocate our own private page. */ - if (unlikely(anon_vma_prepare(vma))) - goto oom; + ret = vmf_anon_prepare(vmf); + if (ret) + return ret; /* Returns NULL on OOM or ERR_PTR(-EAGAIN) if we must retry the fault */ folio = alloc_anon_folio(vmf); if (IS_ERR(folio)) @@ -5823,15 +5839,6 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_start_read(vma)) goto inval; - /* - * find_mergeable_anon_vma uses adjacent vmas which are not locked. - * This check must happen after vma_start_read(); otherwise, a - * concurrent mremap() with MREMAP_DONTUNMAP could dissociate the VMA - * from its anon_vma. - */ - if (unlikely(vma_is_anonymous(vma) && !vma->anon_vma)) - goto inval_end_read; - /* Check since vm_start/vm_end might change before we lock the VMA */ if (unlikely(address < vma->vm_start || address >= vma->vm_end)) goto inval_end_read; From patchwork Fri Apr 26 14:45:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13644920 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04AE8C4345F for ; Fri, 26 Apr 2024 14:45:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 295C36B0099; Fri, 26 Apr 2024 10:45:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 246076B009B; Fri, 26 Apr 2024 10:45:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E72D6B009C; Fri, 26 Apr 2024 10:45:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D62376B0099 for ; Fri, 26 Apr 2024 10:45:17 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 936AA1C07E4 for ; Fri, 26 Apr 2024 14:45:17 +0000 (UTC) X-FDA: 82051955874.15.1A9E7BC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id EF1434000B for ; Fri, 26 Apr 2024 14:45:15 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=H3Kru3bJ; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714142716; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ahgxolPjR35rGtX1/SJWQbu72Aplypg3iRuwgc7l0ko=; b=oUR8+w8q7fHcyyo1AytOpPRaidmDrwqTXOIiiJWWbUrcFiJ78xsJN+HtEeFkHACGalpfnr s65h3BXkOJMurDKRSmGJqgU2Lnxap0kU9IN3FwP3UY0ol/MZRtbDervrX4wp0M9SGkYTnX ISAz1xYlX80MrLACvYgqAHdxmPBnDEg= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=H3Kru3bJ; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714142716; a=rsa-sha256; cv=none; b=t4MUs27IXFBlZU5p61i9dKdI7+Ud7WxEyL45j8XYGEbyaUHC0pghSaUq9egvCbzUVM4xqX I+gXgx2siqqD2wVa5C54zOjlHclSAzwvXu8ftvT1oUXL5yej83hQlF36g28ULcO3btFm9p IIQYR9HoE4C7bSiwWZkVeUw0VvwccrI= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ahgxolPjR35rGtX1/SJWQbu72Aplypg3iRuwgc7l0ko=; b=H3Kru3bJevR/UbQEi+rQZVHYBs 4jZpcvre5O2AGbp8OOvPMiDJkyyXKYEwehrbAEXn46zMrJ7KBuWpFNQeP5GvM2ym4Znpe2rFHtJGY msYEmfK9UvuayFULCI8kk6G7LUB6YPrB2Zx+N03fvRhfkOQQbI+vtr45609Tt2Kwn6FJlK1MeXJxS WMcWkh72E4cJosWiQFX1pYa+1vdoenBQAajpeUfrFZWmbzAJ9JWV2hofQM8juXYox4jLlGGaod9A5 tg72zpBJGzqFbzLuE2zjQPW+GvalGrwtimdgo9buMc+VIRx3vHv5q3cPTlFfAlqho5X7/s/abOeuJ CWaJ58PA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1s0Mp2-00000005Pl4-1Nds; Fri, 26 Apr 2024 14:45:08 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Suren Baghdasaryan , Peter Xu Subject: [PATCH 3/4] mm: Fix some minor per-VMA lock issues in userfaultfd Date: Fri, 26 Apr 2024 15:45:02 +0100 Message-ID: <20240426144506.1290619-4-willy@infradead.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240426144506.1290619-1-willy@infradead.org> References: <20240426144506.1290619-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: EF1434000B X-Stat-Signature: osc8j85jdistk14zmpctwsfoof9c6bc9 X-Rspam-User: X-Rspamd-Server: rspam07 X-HE-Tag: 1714142715-67426 X-HE-Meta: U2FsdGVkX19B9OjbNfUL+JXD3oygEBezV+G10ex74Kuw8Uk5dDmHrFBtHwsIDWYxmoWJcmAAaRqY2av0jeWcFobA1eah9iChm/q2LAgZ3vAxlSkpGQA2tFzolSXvUBzU23SMdTFEGUUoUrJ8ke3Qtyy1VNfjiyCCX6DbXjtVnUt80pzJ9ZR0/a3aEc1RlE6gKkojQydU+Ni17jQVCBWC4d+mvcgbCwnT/hD6gxWI/KzuK57psu9SJ0sPHQHD9e9/xK8Gp1jKNqNaYoVmVUrARVxgrGSPJiPA2RnHLQfkBY9xe7VyZXNFqxWQF6BjHHkw2KwZ6QDXxSztZezB6gG/3tUJztp3FcDXoyB+AqB4JQ0HPfcfVSiqg49ELk6RG+eyIrwW59YM9f8rf3Gf8tJrtbZtOFgsUyjUKGend2XShZyg36wLmn7OS69ywGHDHjTYyMQHurdeks2gtwoZXtigS1rWsX6L6fMeHFL1ew8t+dfkJTgY3SZ3w/YuDwpMvITZpHJ2MIk3NoTqlWc/6/UatIwsxlYlKbMbmAlHfM4Os7xcs9QEAY7rXwYHHCG97a/xmTCTDFuM5OnJkQvqTtxEzUK3zCtl6vHHE4EndyjxPgUrN48fS/l1u0YSk+whXqUdnP8/nmHPC8cqlDNC/e/adZv6IoArqkUmdSjRX/Yu801Xj2zuAE4uS722UCVDAnTdwojRVYOgdXfxrEcjNQ5ReLtYe3MG3kvl4NaFm9T3tDNZUhPnH2uZDzYo+mCkoJRxu3++mUlzjKFpCM+jYHGTV5i4gp0N6ek2veeD+skJzEU1v3uyXg3PEP0KJLFg+6q1py4HDLxJggkZeVONZgvAHELuGtHAunHjh1sc8yaPsgnm0FEgGPBjYkJi3wUoApr4Fx1KntQmewUaNyKfSB74QOPpn0+WGWED/xzb3DbQNfcgnPpmfazBxdkIGgiG8EsRrBeqoiRNtjHxa+zRU0Z eDyQz+e1 uTlMxmSFWEbpjDt72eUzgCW7nMl7LiZe9oVwpSsG9xbwSUFCbyzvn5aGVgCGfh5gq1GfmrUpb2j66un1L/fNvTom+5WzbAnDCp+BYH6OcUEdvA9jULuw90pHgF0FgIfagYy/965YWHSypkR/n4s9q9IOkioiNseJFSYbfL5EnfRA5r1mpQf6MRl+UMQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Rename lock_vma() to uffd_lock_vma() because it really is uffd specific. Remove comment referencing unlock_vma() which doesn't exist. Fix the comment about lock_vma_under_rcu() which I just made incorrect. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/userfaultfd.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index e6486923263c..defa5109cc62 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -56,17 +56,16 @@ struct vm_area_struct *find_vma_and_prepare_anon(struct mm_struct *mm, #ifdef CONFIG_PER_VMA_LOCK /* - * lock_vma() - Lookup and lock vma corresponding to @address. + * uffd_lock_vma() - Lookup and lock vma corresponding to @address. * @mm: mm to search vma in. * @address: address that the vma should contain. * - * Should be called without holding mmap_lock. vma should be unlocked after use - * with unlock_vma(). + * Should be called without holding mmap_lock. * * Return: A locked vma containing @address, -ENOENT if no vma is found, or * -ENOMEM if anon_vma couldn't be allocated. */ -static struct vm_area_struct *lock_vma(struct mm_struct *mm, +static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm, unsigned long address) { struct vm_area_struct *vma; @@ -74,9 +73,8 @@ static struct vm_area_struct *lock_vma(struct mm_struct *mm, vma = lock_vma_under_rcu(mm, address); if (vma) { /* - * lock_vma_under_rcu() only checks anon_vma for private - * anonymous mappings. But we need to ensure it is assigned in - * private file-backed vmas as well. + * We know we're going to need to use anon_vma, so check + * that early. */ if (!(vma->vm_flags & VM_SHARED) && unlikely(!vma->anon_vma)) vma_end_read(vma); @@ -107,7 +105,7 @@ static struct vm_area_struct *uffd_mfill_lock(struct mm_struct *dst_mm, { struct vm_area_struct *dst_vma; - dst_vma = lock_vma(dst_mm, dst_start); + dst_vma = uffd_lock_vma(dst_mm, dst_start); if (IS_ERR(dst_vma) || validate_dst_vma(dst_vma, dst_start + len)) return dst_vma; @@ -1436,7 +1434,7 @@ static int uffd_move_lock(struct mm_struct *mm, struct vm_area_struct *vma; int err; - vma = lock_vma(mm, dst_start); + vma = uffd_lock_vma(mm, dst_start); if (IS_ERR(vma)) return PTR_ERR(vma); @@ -1451,7 +1449,7 @@ static int uffd_move_lock(struct mm_struct *mm, } /* - * Using lock_vma() to get src_vma can lead to following deadlock: + * Using uffd_lock_vma() to get src_vma can lead to following deadlock: * * Thread1 Thread2 * ------- ------- @@ -1473,7 +1471,7 @@ static int uffd_move_lock(struct mm_struct *mm, err = find_vmas_mm_locked(mm, dst_start, src_start, dst_vmap, src_vmap); if (!err) { /* - * See comment in lock_vma() as to why not using + * See comment in uffd_lock_vma() as to why not using * vma_start_read() here. */ down_read(&(*dst_vmap)->vm_lock->lock); From patchwork Fri Apr 26 14:45:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13644921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB402C4345F for ; Fri, 26 Apr 2024 14:45:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7EDEC6B009B; Fri, 26 Apr 2024 10:45:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 776746B009C; Fri, 26 Apr 2024 10:45:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63DEB6B009D; Fri, 26 Apr 2024 10:45:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 459FD6B009B for ; Fri, 26 Apr 2024 10:45:22 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 51D28A0B8A for ; Fri, 26 Apr 2024 14:45:21 +0000 (UTC) X-FDA: 82051956042.07.B56782D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id AED3F40013 for ; Fri, 26 Apr 2024 14:45:19 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="avEdWTM/"; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714142719; a=rsa-sha256; cv=none; b=temu3ip+neaczJmdN+qSd9C4+LsL0EPp6GegUGGWE12LB+UCx3wo7JnpTtPlN1GUm74+Lq 6Sas9XhjSEJ2Gq1lLHVfMxfoYZHtWBdDfTUe01zxKdwoagyRkHt4rWtfqG1LQUHFhvVyx2 TM2WAayokC9JvJMH9nHMDm0frC56oKA= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="avEdWTM/"; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714142719; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xiasNW2bjz4XwywyYv4zK4GYnJbH5N9m8U5+b7/FG0g=; b=4vb6QfTgAXtKIdwnwgD7okvoXYcqQhuqg/k5aRjdN3kfPdwgSJEi7AbezrtQFEB5nLcJFj TpZ3rTua5+n0REvvJT5WhG+WlyN3ibsHqFunDTT0X/AV2qMOUtFMcv3l6Ljeeel0WSlMV4 5IEdYmkKhayMSx1twWKsmeP6DeoSHFU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=xiasNW2bjz4XwywyYv4zK4GYnJbH5N9m8U5+b7/FG0g=; b=avEdWTM/yAlNRrylBNlMhcItT8 yT3lf5sbN2RA11wRK57vjoI6+cQ4lOshf22Lj4uL7AZ/b1ZfApFJfPwIoV+o5X89LxrtQKmovdkWp d6GdOd+oLFrtddsrmgjkb3tHhclq/QbDFb2K9UAk9I0CHB+msjtBqcV+NNzBx3RflOlaD9xy8Dxhk es3hCflPEq9bjg0W2B3ZvZt2nuIbUQ94Vbc6N/36fbLOt32SrhktKVGwgIfqJcg4UWo/Y/MU/xiW+ LQeO3PqBRGk0YU47AeSqIQtKcSKBURhocodIXVgGTbeLOQplgj5BFtuB0HLDEw1g+kXORauvDvxQ1 4dMG+cJw==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1s0Mp2-00000005Pl6-1hWT; Fri, 26 Apr 2024 14:45:08 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Suren Baghdasaryan , Peter Xu , Jann Horn Subject: [PATCH 4/4] mm: Optimise vmf_anon_prepare() for VMAs without an anon_vma Date: Fri, 26 Apr 2024 15:45:03 +0100 Message-ID: <20240426144506.1290619-5-willy@infradead.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240426144506.1290619-1-willy@infradead.org> References: <20240426144506.1290619-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: AED3F40013 X-Stat-Signature: ffroes6j1twr58fz6oz1ge817q4tir3h X-Rspam-User: X-HE-Tag: 1714142719-944121 X-HE-Meta: U2FsdGVkX18hygKFWAxEkA/Ojy5QDrS0zDIAxdRJ3RKlec+CDaSzWZZCuux7gKsQ/JLy/phrNxRyvUvjC5C53cTVrRkaB+l/AngxoePfFEwjTgr58NMXLq5fKZ5uGZff/XyBrqFD+2/hF+4JezHDnLTKr2kebl69hHpySx5sHw3UdvuLhsSdBWMuM3khH1s5bia8L+WZmTilKjl9vHnw+rdskKuLOwPCPbnOkoiDzXAViJfQZQDHTHncHSWuBvARpaMZploZRHsHGiNHh5y6xh6yUhpaSJvLnZ3kf9m1Luqw8m1JrKVb0lhaFLlN1GAUcGz+pv9H9uJMoCpwwgWvKUtqetGZrY6zVzOvkF/bzk1nohW78zbxZxmoHhJj7VDLgthos3YHN+wwwtbzO2QPPoL/HVgKm2KLjwh/3gND5ZWziHIVSESRB3nHyADBohILVZULuL+EmU9/WglQKm5yo0cPbyKsoSI0ZLPwp74F+7KLfCEPugFj8mIq17+F3IUfdnfo6kSaVNLTfmoDmCRDMteFW4d7pJG9Jfzd0PRv3dI5L2hnVX24Ggxsuy8Ccw6Qt513r5AscS9S5bB0Ak+rmJT+0469jbsa0BmWmp+7TlM/nLoBWyTmdDehAH9PGqBt7vJ/rg8p6zWaJ32rsTTYYa9BmslmoWMTiD941dfzjGETVL713yrOHFJtfi87g0UOEI0J66Uz/rjKlWSauEa4IrnA/hHboaM1ZnGmXclpE6Lm7Ou6UkhVpb2qE1wLK/Cmwtr9HhiZUaUOuG/1/0vZrxqSYb+ItSiXbeEiCxRMuh+ZXzWMIk/CY+e7YU/v32mbaZjnNSPZWIxSv4R/u2TauBvy52sSOw9Txqxwv8DEppecOA0b83GT1LqdgyddZ3WQwpZKTV8FqciCJCxf9cImEpp2qR38QISlCcyfuhSXgTb+aAGvz+MdVjrkrqZmCw7qtsnDPlhbSEe+g+RJ1qf zuHP3V9+ cZ7QRdh5+X0eIKKCnfrhswPNXLwZ078wh1Zla2P6WLWuaGsbg/CiDdWAv4mGUc85LFlaWej+rOZ2P3qMC2N40Oksmxu/j6O7jEdvwRsUR3zhJXxKQQuY6gI9kI9yeOMl3fhjXFPUYzqTj9FA9TCuzxd+4RZrJg5aNAwkk9MV1iYwxHm6DQ7QLkq4qiwHjbGkfYYw+x9EiC0F8t83wpeyZ7tOd4qhyMv3hVfcXLMYSnWRCWV/5KQQMbCaldw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If the mmap_lock can be taken for read, we can call __anon_vma_prepare() while holding it, saving ourselves a trip back through the fault handler. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jann Horn Reviewed-by: Suren Baghdasaryan Reviewed-by: David Hildenbrand --- mm/memory.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 7dc112d3a7e4..b5453b86ec4b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3232,16 +3232,21 @@ static inline vm_fault_t vmf_can_call_fault(const struct vm_fault *vmf) vm_fault_t vmf_anon_prepare(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; + vm_fault_t ret = 0; if (likely(vma->anon_vma)) return 0; if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vma); - return VM_FAULT_RETRY; + if (!mmap_read_trylock(vma->vm_mm)) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } } if (__anon_vma_prepare(vma)) - return VM_FAULT_OOM; - return 0; + ret = VM_FAULT_OOM; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) + mmap_read_unlock(vma->vm_mm); + return ret; } /*