From patchwork Wed Jun 5 00:24:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13686015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61696C25B78 for ; Wed, 5 Jun 2024 00:25:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D2C3F6B008C; Tue, 4 Jun 2024 20:25:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CDD166B0092; Tue, 4 Jun 2024 20:25:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B56636B0093; Tue, 4 Jun 2024 20:25:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 968656B0092 for ; Tue, 4 Jun 2024 20:25:17 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 00185A08AB for ; Wed, 5 Jun 2024 00:25:16 +0000 (UTC) X-FDA: 82194940632.26.8CDCD34 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf20.hostedemail.com (Postfix) with ESMTP id 532AC1C000B for ; Wed, 5 Jun 2024 00:25:15 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=NOGRQgVN; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf20.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717547115; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7jDdTLkK44GLfBM/z1/kgCxTgHlamAC1xEk2N7hacqM=; b=E7U7pz4AKL9ipQ2N4uk/5XoH+zYJ7iTNtx04oa7PzSJGlCmv1+gB+EYk9rDebXy35/Is94 Yo8FXJCBQzFCa/ABZwJWjCjakI/kWb6opW8I6kh9TCppvS25TVuIDnCXzgellOUqGtS68C WvgaVJ43s+/j8NL3GvhjyALyEgVsDvI= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=NOGRQgVN; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf20.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717547115; a=rsa-sha256; cv=none; b=2tIp3VK549Fz2nOrcJwu1nyTzzDfoKRPzF6eudX4K93iCfSCjwVy2iUnxm7ZLdf1lUwJIy gWgVCsil2nfEsFkXB0N+EWJD+kj6Uzs/W4wRlh/PyPmQC8YdsGQUSwkfR635Tx+l9QbKZy rYSqPp34JcgrQ7KALAc1X+gUFRq4Ytg= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 57207615F7; Wed, 5 Jun 2024 00:25:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D5E94C2BBFC; Wed, 5 Jun 2024 00:25:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717547114; bh=MdL1CnIYtJxVuaoJQ4v36gHWpY8r7Nqra5mr+RAHejU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NOGRQgVNavyC3fwxEaZyJrjxDcYUvSoz/8YM6QedZh1sBl8wowm7FizTiavEjNHn/ EvjD1syryMww2h/Cd4Bk1oUVRd0afZpPTXHLoi/FeUHaRnDFDDMMPRviTgxBJSCq12 tt9tGFK9cPMKulFga2lx7O7/gt+WZaNagmDdfWm6cGfLykpdKe8Wt4dLAvnv8fo30u qTNSE/DJb4zv2vaKfCZ/NqrLUbBBvVZqt3vYDDF4cw4Wz8o6nIehEZOjQ2NDnzHcqM gl4tEPY3uixZj4n7PhMAbTTCssXdChHgeYz/B+Quhh2vTHPQC0HknreP7LHgGlYcRy WSMJWgtfEVf9w== From: Andrii Nakryiko To: linux-fsdevel@vger.kernel.org, brauner@kernel.org, viro@zeniv.linux.org.uk, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, bpf@vger.kernel.org, gregkh@linuxfoundation.org, linux-mm@kvack.org, liam.howlett@oracle.com, surenb@google.com, rppt@kernel.org, Andrii Nakryiko Subject: [PATCH v3 1/9] mm: add find_vma()-like API but RCU protected and taking VMA lock Date: Tue, 4 Jun 2024 17:24:46 -0700 Message-ID: <20240605002459.4091285-2-andrii@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240605002459.4091285-1-andrii@kernel.org> References: <20240605002459.4091285-1-andrii@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 532AC1C000B X-Stat-Signature: zyb1mu97q1x4tfzkebij34rqof3jaukz X-HE-Tag: 1717547115-314306 X-HE-Meta: U2FsdGVkX18pWJN576KUFiDaLj0CGdG53voPrsRG/08DtwQATJsqTQxHyPBV8QOnSQzjsCgjZ4OUJJOdveTCyHGQAovRIDOdv+pqjc3x43dLxRC3knvaemkHh/9qz5BZAjFKxJw4jCDj4f+czt2vLsc+k/rMpFcxwPTHK2VWz7rQ79jXjqSJA8Qz32c9JF4GPhWVuHuUS/+JlONghNwahT6YAgjSA3D+qb5XPJIJrQXiSxxJC9ICnMRXZB+XsJpMI6GnmA4c3ijEWXFZVRYTl87KVHeLg9cVJ0s8p1sDUquMlAu8S1N7HXHIQG05ax+dOTKSsk+HmyLeXLqHdEb9+9CNrXmTD6T2WLomP4NYgpYp1YMHN6uN2mT6TKMqWK3x/gyGrkFSCb5/LdS3yxsaQjJBMhL0p3b8yFGxGVxhlfDXaA6dvv13bnkacm/5kGsfKUsEH2DdkH3Gdklf6XyG0IyF3rA+lZUmio7lflCJ6+KpaoBXY4SAs0+0wqaZ9g613EQHEq+S+he1TR8YZe5RJxXbnjR25Wy+V7301pyd9l/p98Eu2+KhzqTO2GLCUl2ej1WLSaa+kqhruV7poNd4PCvPCp1daWM0/Xn23CgVxt+EeHnmMW98c+N14yS+9udttODxemMfO/nGAuOlQRS36kXna/RQFBi/Z42o0b69r6BUOhnr3qiiJ5NSxz5UMMTlOId9W/KQyql+uwdFeTnqbGI2TNyhBmPVfhHl/GhZ4ql2/H2XywtaPwW5k1FvBk0RO/iLzVaE6hrrPiyyyXKdC2Y2K4EC9mBtpHvdxNxLGSLUMXE9pdEIERkwdKWQ5M58gOTCJ2PkzZ82R3TigCLnnlIrviqwJaKfZ0xcOfRHz/B1LskV5jN0dhR14f1tuXRSKea3kQfdHpHzjLJJSPNZNyQ49zOlVU+qhHhIrEBIw7jr9Q6u/oEaPkpLUpmpP4SmjmEe6IXKbnwrAHAGTFk 0QbEIaBW hEq2Ff+j8YIv0eJ5hQdZ9BJ4fgq0U+qDGqd1d413/yWu5FgJKvVGphDZVX9x1JiGvbewT3GDTYpuNZP8nO7LuW3Am9J09lbe3vsKftU2vTk4hJoxJ6D/hzjVUnIej6sqt2mfFNXxTZMdW1p3lzoMO5RIVAtyHmZ+VYICsgvTHRWJ3tSnZI0gy+0I+rPdSg+b3wbp/vwpekJQ6QTZFXUl3qt3jf8shrrVC3RJaDPXWXRR7TjNaTXo7U7pa7SsmOm/PZKQ5IB7iETwkQNn2obGb7bNvisTaOQbF7o9pGjsvzsUv8huNI7+2t2UwF01Juu5UoXkcAsWbnc7GmSs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Existing lock_vma_under_rcu() API assumes exact VMA match, so it's not a 100% equivalent of find_vma(). There are use cases that do want find_vma() semantics of finding an exact VMA or the next one. Also, it's important for such an API to let user distinguish between not being able to get per-VMA lock and not having any VMAs at or after provided address. As such, this patch adds a new find_vma()-like API, find_and_lock_vma_rcu(), which finds exact or next VMA, attempts to take per-VMA lock, and if that fails, returns ERR_PTR(-EBUSY). It still returns NULL if there is no VMA at or after address. In successfuly case it will return valid and non-isolated VMA with VMA lock taken. This API will be used in subsequent patch in this patch set to implement a new user-facing API for querying process VMAs. Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Liam Howlett Signed-off-by: Andrii Nakryiko --- include/linux/mm.h | 8 ++++++ mm/memory.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index c41c82bcbec2..3ab52b7e124c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -776,6 +776,8 @@ static inline void assert_fault_locked(struct vm_fault *vmf) mmap_assert_locked(vmf->vma->vm_mm); } +struct vm_area_struct *find_and_lock_vma_rcu(struct mm_struct *mm, + unsigned long address); struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address); @@ -790,6 +792,12 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) {} +struct vm_area_struct *find_and_lock_vma_rcu(struct mm_struct *mm, + unsigned long address) +{ + return -EOPNOTSUPP; +} + static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address) { diff --git a/mm/memory.c b/mm/memory.c index eef4e482c0c2..c9517742bd6d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5913,6 +5913,68 @@ struct vm_area_struct *lock_mm_and_find_vma(struct mm_struct *mm, #endif #ifdef CONFIG_PER_VMA_LOCK +/* + * find_and_lock_vma_rcu() - Find and lock the VMA for a given address, or the + * next VMA. Search is done under RCU protection, without taking or assuming + * mmap_lock. Returned VMA is guaranteed to be stable and not isolated. + + * @mm: The mm_struct to check + * @addr: The address + * + * Returns: The VMA associated with addr, or the next VMA. + * May return %NULL in the case of no VMA at addr or above. + * If the VMA is being modified and can't be locked, -EBUSY is returned. + */ +struct vm_area_struct *find_and_lock_vma_rcu(struct mm_struct *mm, + unsigned long address) +{ + MA_STATE(mas, &mm->mm_mt, address, address); + struct vm_area_struct *vma; + int err; + + rcu_read_lock(); +retry: + vma = mas_find(&mas, ULONG_MAX); + if (!vma) { + err = 0; /* no VMA, return NULL */ + goto inval; + } + + if (!vma_start_read(vma)) { + err = -EBUSY; + goto inval; + } + + /* + * Check since vm_start/vm_end might change before we lock the VMA. + * Note, unlike lock_vma_under_rcu() we are searching for VMA covering + * address or the next one, so we only make sure VMA wasn't updated to + * end before the address. + */ + if (unlikely(vma->vm_end <= address)) { + err = -EBUSY; + goto inval_end_read; + } + + /* Check if the VMA got isolated after we found it */ + if (vma->detached) { + vma_end_read(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + /* The area was replaced with another one */ + goto retry; + } + + rcu_read_unlock(); + return vma; + +inval_end_read: + vma_end_read(vma); +inval: + rcu_read_unlock(); + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return ERR_PTR(err); +} + /* * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be * stable and not isolated. If the VMA is not found or is being modified the