From patchwork Fri May 24 04:10:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13672703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50AB3C25B74 for ; Fri, 24 May 2024 04:10:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D8D5D6B0083; Fri, 24 May 2024 00:10:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D38496B009C; Fri, 24 May 2024 00:10:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B63596B009D; Fri, 24 May 2024 00:10:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9628A6B0083 for ; Fri, 24 May 2024 00:10:57 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 53F1780838 for ; Fri, 24 May 2024 04:10:57 +0000 (UTC) X-FDA: 82151963754.11.8034974 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf20.hostedemail.com (Postfix) with ESMTP id 163081C0015 for ; Fri, 24 May 2024 04:10:54 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=oGF0nsJ4; spf=pass (imf20.hostedemail.com: domain of andrii@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716523855; a=rsa-sha256; cv=none; b=m9+XiH2g+WYRwuQyVuV4R+cH1eSHCOYZ8U9tCqVhh90tDapE3ml93YL1uh9pSz96UGsGRh wsq8x33knWS34pJ7ff5DGKNGqp9tKVZZ0i+cedcXJL7Vzx3BhKbTHOA9dSO8FAMFi33Q4D Eo38pamgwsbNPLFjpsgOmlh7vWsVFd4= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=oGF0nsJ4; spf=pass (imf20.hostedemail.com: domain of andrii@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716523855; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=78VARJriAuzO1L2dfIt4EY+wmkVZ35j9wTTIvpJ3oCg=; b=EmVl1XJRoG/wN8yS3bfOFXdMg8Otsd0GXhyMvJ9EWHfbzIDYKvCN3RsFnzT3baXcZSJrzL 1ZarwjddXFZdedaq3qlpyD0eUFCuq8w97SkmGocnB6NtYl3usdjo0Z6wgruu3SG0ZTzTN3 Aln67WI3ojx3jrWDaDZn7kFEd8NpI2Q= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 187D1CE18C7; Fri, 24 May 2024 04:10:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F30C7C2BD11; Fri, 24 May 2024 04:10:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1716523851; bh=zTCFNBXKntn3uKNCfQDhX1RMYgRb96MCd/HOF52B5+s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oGF0nsJ4lWUgidH6hqxg85XT6O9QU7bGEJNrb3GUoUgXu3YNA/gte/0Anz3cym+fi RLnmgOxqPJE9j1T9j9iHMwPsG8FCKPMvhH4phSmQFfIL5nzXbogzdgDcVqz9vgnJP6 /BxeoQL47G/TpCJLKoQqdpfxx97EPTB4ZRk/4Od/AHkjZlSecdlSR6s336gRHM/XMT echfa9RMAX8053WZ2GLw2zDVLtprp8Fa3/UsIwDd3+P0CPT9OwxEkmGn4O3UGLjIlC //fdBmcelA5uhED0h0hVtCoQIPvYIUDySrU1awCDh7hfPfntgo1ndJkaNdCjhCB0bS 6JJ8I/ed7K09w== From: Andrii Nakryiko To: linux-fsdevel@vger.kernel.org, brauner@kernel.org, viro@zeniv.linux.org.uk, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, bpf@vger.kernel.org, gregkh@linuxfoundation.org, linux-mm@kvack.org, liam.howlett@oracle.com, surenb@google.com, rppt@kernel.org, Andrii Nakryiko Subject: [PATCH v2 4/9] fs/procfs: use per-VMA RCU-protected locking in PROCMAP_QUERY API Date: Thu, 23 May 2024 21:10:26 -0700 Message-ID: <20240524041032.1048094-5-andrii@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240524041032.1048094-1-andrii@kernel.org> References: <20240524041032.1048094-1-andrii@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 163081C0015 X-Rspam-User: X-Rspamd-Server: rspam12 X-Stat-Signature: usjbwkipptry1ufu698tzk7oomnc9pm7 X-HE-Tag: 1716523854-347462 X-HE-Meta: U2FsdGVkX1+QfyIihuNzrdz5kskpo6m0E6MMj6JNh0JJyj0/DdNA0kp2xI14LfjcWX8zwXngS+NQQFB5cZwQJCTu/Bo8Hg9BZZYZiv8PkNY4Xw2L4bEGP3im0T9acnpCMHPWYIFEzkAK9WbwmbIdkAp85C+QVQQaeZYVcrEqwAShU3lJRB1kn8KHqZNtp//AyOpakVGhiiLrXqOmLms7EDwluvGSg1QVRC5TCUz+ihAfhNJPyBNzEnf7FOsmJy81rdnTdD9d/GPpvMiWHf09wC9zX5tmkniLxE9UWuIV68xdSq7Qqa6DT6FVLnfe/85+24e8YNlxb+LpNWSJnLzSPEMpywSZ+zIJAGS6MQhEZ3WzAMP2JOH2kgD7Wg8qxneZfxZZugMhVQiY7xIrgJ+5IzIvFdUxrJ0IdGF7Q7M/fIjWAK2K4B30sdRvIRn/6+SSwcnliRWROv7HL/CfPQZnkDuf+br9DrkTWpKDe/poOk2ExLGV24emUzkJwJv+HzMkxLiMmweE3W+IDybkl89f6KoTh56nDOIv6GzDU91xpF40K58ZBrNTgGQs10/XrzDHrnD2sep0ZZVn7V2xulJf9mp8oSA7CGBU6PUhcvX5axladImPEYptDI8ysHFLVKFX4U1dA0ym9uAMb6l4TqsHl+3HnenV7+JQOMWySKFh55KnTTtoZ5jKI0rec0yNuxDPA2ryk2eHvsJOtBs3hdlFsS9ilnH+bUONz5VRDNL5BiPvdzu4QwUJEO7b/A91R7R0FESWdCBvssiy8Sq9vkUwdJVnSQ6K8GNbRCpb042H/0DcQRgnCuqg8Tints1ChE6HPxDUZXArRxe/HAjVaFXOFN7WL2S3Joz8NulV+fRDa0b1Rtxrv8Ju9zzwjgx3MIe+T/WwPSPVhS0tCbrxxj/PInLdC4H9m3G+3laZpA2ixnH5CxLCx6E4peg15UbLuFg+Zx+Z+rjY8VHs4oRKUFt Ns3s6FMM Ya9fhNTiWBIeaKGNHTgJdjt471nZxzxRPJolKvpkIGRJJYUzHjTYVVj2sjLYSrJPZXN5b7/GmZoJWrvoLQZXc8VDTm/UHkAi2c8r6T2yPCs5CRvKWjRTqicCB+UqWbF5EJDojFANEFMOmKr0/QABmD6IEmJS/gnGIyCyaCfFh6nt0O/BrAReFn6eC1UUr6y5O0hvF8/T8OiSyWhHKDs+EN1jMQ/nZs3FEXOpG4LLuB6HgDkObkGp1iQTt7g6I/7uyXHyfOyo02dcOwOn6t1PNpn+2UsoCZIYIgV/D X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Attempt to use RCU-protected per-VAM lock when looking up requested VMA as much as possible, only falling back to mmap_lock if per-VMA lock failed. This is done so that querying of VMAs doesn't interfere with other critical tasks, like page fault handling. This has been suggested by mm folks, and we make use of a newly added internal API that works like find_vma(), but tries to use per-VMA lock. Signed-off-by: Andrii Nakryiko --- fs/proc/task_mmu.c | 42 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 34 insertions(+), 8 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 8ad547efd38d..2b14d06d1def 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -389,12 +389,30 @@ static int pid_maps_open(struct inode *inode, struct file *file) ) static struct vm_area_struct *query_matching_vma(struct mm_struct *mm, - unsigned long addr, u32 flags) + unsigned long addr, u32 flags, + bool *mm_locked) { struct vm_area_struct *vma; + bool mmap_locked; + + *mm_locked = mmap_locked = false; next_vma: - vma = find_vma(mm, addr); + if (!mmap_locked) { + /* if we haven't yet acquired mmap_lock, try to use less disruptive per-VMA */ + vma = find_and_lock_vma_rcu(mm, addr); + if (IS_ERR(vma)) { + /* failed to take per-VMA lock, fallback to mmap_lock */ + if (mmap_read_lock_killable(mm)) + return ERR_PTR(-EINTR); + + *mm_locked = mmap_locked = true; + vma = find_vma(mm, addr); + } + } else { + /* if we have mmap_lock, get through the search as fast as possible */ + vma = find_vma(mm, addr); + } /* no VMA found */ if (!vma) @@ -428,18 +446,25 @@ static struct vm_area_struct *query_matching_vma(struct mm_struct *mm, skip_vma: /* * If the user needs closest matching VMA, keep iterating. + * But before we proceed we might need to unlock current VMA. */ addr = vma->vm_end; + if (!mmap_locked) + vma_end_read(vma); if (flags & PROCMAP_QUERY_COVERING_OR_NEXT_VMA) goto next_vma; no_vma: - mmap_read_unlock(mm); + if (mmap_locked) + mmap_read_unlock(mm); return ERR_PTR(-ENOENT); } -static void unlock_vma(struct vm_area_struct *vma) +static void unlock_vma(struct vm_area_struct *vma, bool mm_locked) { - mmap_read_unlock(vma->vm_mm); + if (mm_locked) + mmap_read_unlock(vma->vm_mm); + else + vma_end_read(vma); } static int do_procmap_query(struct proc_maps_private *priv, void __user *uarg) @@ -447,6 +472,7 @@ static int do_procmap_query(struct proc_maps_private *priv, void __user *uarg) struct procmap_query karg; struct vm_area_struct *vma; struct mm_struct *mm; + bool mm_locked; const char *name = NULL; char *name_buf = NULL; __u64 usize; @@ -475,7 +501,7 @@ static int do_procmap_query(struct proc_maps_private *priv, void __user *uarg) if (!mm || !mmget_not_zero(mm)) return -ESRCH; - vma = query_matching_vma(mm, karg.query_addr, karg.query_flags); + vma = query_matching_vma(mm, karg.query_addr, karg.query_flags, &mm_locked); if (IS_ERR(vma)) { mmput(mm); return PTR_ERR(vma); @@ -542,7 +568,7 @@ static int do_procmap_query(struct proc_maps_private *priv, void __user *uarg) } /* unlock vma/mm_struct and put mm_struct before copying data to user */ - unlock_vma(vma); + unlock_vma(vma, mm_locked); mmput(mm); if (karg.vma_name_size && copy_to_user((void __user *)karg.vma_name_addr, @@ -558,7 +584,7 @@ static int do_procmap_query(struct proc_maps_private *priv, void __user *uarg) return 0; out: - unlock_vma(vma); + unlock_vma(vma, mm_locked); mmput(mm); kfree(name_buf); return err;