From patchwork Fri May 24 04:10:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13672700 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D49EC25B74 for ; Fri, 24 May 2024 04:10:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A2CB96B0092; Fri, 24 May 2024 00:10:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B5716B0093; Fri, 24 May 2024 00:10:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 87DA96B0096; Fri, 24 May 2024 00:10:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 675E56B0092 for ; Fri, 24 May 2024 00:10:44 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DD82B8082F for ; Fri, 24 May 2024 04:10:43 +0000 (UTC) X-FDA: 82151963166.28.599E6FE Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf11.hostedemail.com (Postfix) with ESMTP id 392B74000C for ; Fri, 24 May 2024 04:10:42 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=qFp08kfJ; spf=pass (imf11.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716523842; a=rsa-sha256; cv=none; b=EIsbcnmwcPyU0GERwuyrdnYBeye/aGQm6C0J0mGalMaIA2AcWTCGHOI9nqTb8KgyixWK6x FIWB+BV9RCGn+ihcoVDmppt64ZXb47OLPuWFDFC/0EF8+SqVjYJRFWN5l1ZhIqlHYzf3kS DlzZ5IcepLqeInhuyVj+guAekZ+xA34= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=qFp08kfJ; spf=pass (imf11.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716523842; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=f8/b++6yMlYROYD5+SRnRTc6GVeU1KHe+gCGLSZfF2c=; b=mM9IOlgqxUrkxBopnb0uVQEzLUfTpK+OHCuGqSNcEmQzHUmB5M5kSgtI/qPnimDAuhJq4J 9/pVVdFjBM5Ffd/RfSgc27a7LWJC+QRMo5YDvUfPeONvgCEB5fJh0bhaEv0iaUXLJg2Myp zYaSmvk/VEL3AMDw0xznx7uaQQmT580= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 5CA7B63034; Fri, 24 May 2024 04:10:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 06448C2BBFC; Fri, 24 May 2024 04:10:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1716523841; bh=cJ+pScGp7VQ+eleJC++23Mq/1Sbkvghbyy7uqJ0sROc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qFp08kfJT/u230KfRh8KqGt5sSUogcNzDpvTxTL3DehKhSM2HsZXTiaJaOUvv6DMn W8/fhxdpRcolBrDIFowYh93P2QEDsWjAskP8Un7vrZyo7WCoS8QHczfNzVst468c0X +SVv9r/Z8Q3Q5zo4tovdIn9zvdrdsBWI3Fih+kSlmUj1B0nqbi3i0DwkJc9+qO4TCF 1bI8tRsRzwCUEmvyyB7FlOknLWOgexqHRqK52owJf1l/rVDXIRrI+PsKW3BCwnupVN eAbpq9A4Df1NN3hJaRAS5Xudvrxw3W5IZeJXr503yLUuMGhypQPSfhkXFXjRnTvJIA +K7dt48W1qQUg== From: Andrii Nakryiko To: linux-fsdevel@vger.kernel.org, brauner@kernel.org, viro@zeniv.linux.org.uk, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, bpf@vger.kernel.org, gregkh@linuxfoundation.org, linux-mm@kvack.org, liam.howlett@oracle.com, surenb@google.com, rppt@kernel.org, Andrii Nakryiko Subject: [PATCH v2 1/9] mm: add find_vma()-like API but RCU protected and taking VMA lock Date: Thu, 23 May 2024 21:10:23 -0700 Message-ID: <20240524041032.1048094-2-andrii@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240524041032.1048094-1-andrii@kernel.org> References: <20240524041032.1048094-1-andrii@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 392B74000C X-Stat-Signature: yqt6deox88g4a3g8841rfwmhm1cm791c X-HE-Tag: 1716523842-371904 X-HE-Meta: U2FsdGVkX18MfWTbVTEKrIu3aHmndAAJ7C52UIWhJ5OHJsga2XZbn3NIVw9UFCfEqR6HiyMZ6rtewFtaVG3VM+/JGKXsb53XTLm8XIT68VTi/LyTILmfUe+OUlC+mRpDgjgKnT+JcpEKGpThXTeZbmG1qAESE9R/3S7Hj7IboEB4nd9kw2bRH66v05vHF8vnqTYFI1ypKzjiM4YknRHB298yxVhZbrgPX0uW0wQiFTrSi8hyU+Uc18aUHsSZ+OkAuK2F0/5e3qodIdqezr+JOGyBzfxfs7rd2But3Lx2hZfeWUzUnmb9fgaQ8s6fwmjhopRkcBXO/FnRVsuhezBzzEdrUjSjRp0VKT+RJXdI91qwPJSXorOkfgoaCzN2gbL6Cted5+jt1ApTcLIgxsAOU3KvKPsUpjIeTlaSlx0WzYcb9pDI2LoE3zM88rwLWHmv1T0Z/MdJIuWSD7lSzI0h86wo2OMKwY6hdqpUncKR+mjMtxCJ5l4vVas33f/FH7lrjg5Zt5f6Bnv+ZuJm66PZJzur5/oeDE5EHWVjrVOszQQ4HPzR7Of/Cqygl4pC82/r7LXJRz88VPTB3pMC2RjOnGeRcTzM7i1G5uHukDEtFlCH2wqnSZzIc70O5RFNvYaTIGH5OquxjBGL7NetnvfvsecK5QNhjvKTRcTLvaADaCq58zBoy9V3ckoB2jKYlmEEEhPZeYfX/VXu4fBjeYLxxMQtXJb/WtrtOJw6xyd+/Qwhbd8pHcKlEmGvrKQl1boeysF9r3ufNjXmqIeKu5Rotu9fd/tLp0uDZ+FrsTwiw8RAv8vF0Rq+0Fvl5Sr4sDMbZE81UKjPEfMMixxtg1l2iCZ5dNqMEjURs9EGNLerIPKxccfF31yrDtRTDfzz54fgkwreNV0M1L31R6jPQAI+ExBmzdjBO2VZ2qbOGVJ4EzA/rWu6fAPoksn2a5YIUfYxzzj5R4x4VUCVNbmfOVi lB5prF09 NT6mprYIqt2YcvZCBsniUAd+5NqY+RjoWTZLqPzh+5L1ZfZQjA49CpBR5v4lto97tDTG2zCgoCLAZDZXEAj6ByLRpNeZK/np5NVAk+BI34HNCvvLPPFqDYGslU8Cc29rMAowStsJr6v8YVvm3eU4fXdu+Oip+QGFt9jnq30yy8Hd9CpE5yDDOIL1AJ7EDQSRRs91F7yzbkNqH9P2TNiQ3/r+p6+3MJ+JWaSPPlYdX/CknPTai+OIEzX42n7FPYUjbcPqh8VG91csWh4QwjiBxviM6yTjNMJ1vP7r/UNHArcFp4bnrnEysifxpYmAKL4usnaHYXj291yXKaOc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Existing lock_vma_under_rcu() API assumes exact VMA match, so it's not a 100% equivalent of find_vma(). There are use cases that do want find_vma() semantics of finding an exact VMA or the next one. Also, it's important for such an API to let user distinguish between not being able to get per-VMA lock and not having any VMAs at or after provided address. As such, this patch adds a new find_vma()-like API, find_and_lock_vma_rcu(), which finds exact or next VMA, attempts to take per-VMA lock, and if that fails, returns ERR_PTR(-EBUSY). It still returns NULL if there is no VMA at or after address. In successfuly case it will return valid and non-isolated VMA with VMA lock taken. This API will be used in subsequent patch in this patch set to implement a new user-facing API for querying process VMAs. Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Liam Howlett Signed-off-by: Andrii Nakryiko --- include/linux/mm.h | 8 ++++++ mm/memory.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 9849dfda44d4..a6846401da77 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -776,6 +776,8 @@ static inline void assert_fault_locked(struct vm_fault *vmf) mmap_assert_locked(vmf->vma->vm_mm); } +struct vm_area_struct *find_and_lock_vma_rcu(struct mm_struct *mm, + unsigned long address); struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address); @@ -790,6 +792,12 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) {} +struct vm_area_struct *find_and_lock_vma_rcu(struct mm_struct *mm, + unsigned long address) +{ + return -EOPNOTSUPP; +} + static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address) { diff --git a/mm/memory.c b/mm/memory.c index b5453b86ec4b..9d0413e98d8b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5824,6 +5824,68 @@ struct vm_area_struct *lock_mm_and_find_vma(struct mm_struct *mm, #endif #ifdef CONFIG_PER_VMA_LOCK +/* + * find_and_lock_vma_rcu() - Find and lock the VMA for a given address, or the + * next VMA. Search is done under RCU protection, without taking or assuming + * mmap_lock. Returned VMA is guaranteed to be stable and not isolated. + + * @mm: The mm_struct to check + * @addr: The address + * + * Returns: The VMA associated with addr, or the next VMA. + * May return %NULL in the case of no VMA at addr or above. + * If the VMA is being modified and can't be locked, -EBUSY is returned. + */ +struct vm_area_struct *find_and_lock_vma_rcu(struct mm_struct *mm, + unsigned long address) +{ + MA_STATE(mas, &mm->mm_mt, address, address); + struct vm_area_struct *vma; + int err; + + rcu_read_lock(); +retry: + vma = mas_find(&mas, ULONG_MAX); + if (!vma) { + err = 0; /* no VMA, return NULL */ + goto inval; + } + + if (!vma_start_read(vma)) { + err = -EBUSY; + goto inval; + } + + /* + * Check since vm_start/vm_end might change before we lock the VMA. + * Note, unlike lock_vma_under_rcu() we are searching for VMA covering + * address or the next one, so we only make sure VMA wasn't updated to + * end before the address. + */ + if (unlikely(vma->vm_end <= address)) { + err = -EBUSY; + goto inval_end_read; + } + + /* Check if the VMA got isolated after we found it */ + if (vma->detached) { + vma_end_read(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + /* The area was replaced with another one */ + goto retry; + } + + rcu_read_unlock(); + return vma; + +inval_end_read: + vma_end_read(vma); +inval: + rcu_read_unlock(); + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return ERR_PTR(err); +} + /* * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be * stable and not isolated. If the VMA is not found or is being modified the