From patchwork Fri Apr 18 17:49:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 14057562 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1387AC369CA for ; Fri, 18 Apr 2025 17:50:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 20B646B02B1; Fri, 18 Apr 2025 13:50:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B9716B02B3; Fri, 18 Apr 2025 13:50:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F2DE56B02B4; Fri, 18 Apr 2025 13:50:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D32FC6B02B1 for ; Fri, 18 Apr 2025 13:50:22 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3BC0E160AB8 for ; Fri, 18 Apr 2025 17:50:24 +0000 (UTC) X-FDA: 83347903968.25.2B9F23D Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf14.hostedemail.com (Postfix) with ESMTP id D8804100008 for ; Fri, 18 Apr 2025 17:50:21 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=0qWUbmrs; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 33JACaAYKCE89B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=33JACaAYKCE89B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744998621; a=rsa-sha256; cv=none; b=lEBRwBVwFlBsT2OJ+FB2DcRvagD6frI/Hpf4wkAIPCjK9rdoAegfo8EWjzsoxQ1Bga67lt EHnGiQUVkgsBczpQxhoYx93Ryku8kiOxPW3WUebWTwN2pzGgSCgJNStpzyfkhTOKxTXl8f pTDwNuzyDD6nJoKld0rcvSvYYXLGWiU= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=0qWUbmrs; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 33JACaAYKCE89B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=33JACaAYKCE89B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744998621; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GDNsq3YX7eW5e3apKRRmkWWvDDDuWE0xbjvQjwBgCsE=; b=qkKQcVJTe51sp5hbWYx8GBBcWQ+Hivuj2YIC5E3VJnOESuyuBdRlGbV1C4KlgvxfTE5crL OcAdUg+r3BvdMqphE0jXbz1TKydXNpcBvMI0aX9LoI4Qo8wnXMHmp5QrFheGVgc63Rv9aH QXbLCdkIAZpQQG7sgJU9q6U8PxGLV+E= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-22c35bafdbdso30245925ad.1 for ; Fri, 18 Apr 2025 10:50:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1744998621; x=1745603421; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GDNsq3YX7eW5e3apKRRmkWWvDDDuWE0xbjvQjwBgCsE=; b=0qWUbmrszho6lf/6bsKu4TFOmR4Ou6Zc0iXRlDKGhJcJhuigOFH3cIzub2bFO6rD+e xx6Y3JdJ4XqeFeGClmPaZXjbqXeGPEo3nyWj0Be4YBCRA+1naQ8RdI7vYI+mbAwKRxxm 8fhLU6RmFSuFyihLd5o3qbaO939RCUOeMGMyMVDMKBfixGnY2Q/8iLFj6W7CcsJVtECz iBHjyjplqnKgVZdKLhChEodQVORe7Oj1bYatyE0hGZBbUZMMOyPqzQ+cpMGIaaeITYgU 3q58WfF5BC4RO3j8nkgaY8LoCUIa2tCJ2wvcEIOQmd8mB+wBUPPtdTYBcH2JHwn5iEU/ 0caw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744998621; x=1745603421; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GDNsq3YX7eW5e3apKRRmkWWvDDDuWE0xbjvQjwBgCsE=; b=LmGLEpOjyEug94UJE5avVW6TD05fTB7r2xfakQJpN+pWzlnpFAmgL0kmZEmHZkkS/y ScqURSRnHKB+hfyx7OJoLsVYH3FwuK1NflIERDX07ZOj0IyTT1y1VxnHSt9bnyawoPG1 m3VyPH5GkT+O+RS1w72GxteC0yodtQ7Kc4fDqGcKddWRzi1mOldUye3ybiKQunmnR7mo y8U8j4DItZhahndVSh3VKVKl5RhE15qZediiRgjiv913EpKjGq1wi3xGfOcGna0He5Nk FQ57+hWRB8Ust85U5R4tbnsx8JMOY7CvYCRTW0ZpAk7/JpKwdH/1OtqyvU6sA4Lk9/WG Cmvg== X-Forwarded-Encrypted: i=1; AJvYcCULa+Rz9miqu2XTqf6IwPxqJgIw5K+jbrHebOxZm+V1+xQeC8rCY0Su1KmQmW76nzY8BVx9HMQpKw==@kvack.org X-Gm-Message-State: AOJu0YwFDOOhZ5q2s6dMKPL+Rhp9eNEhys8PNQJS+fmSZMGgrOclkAKA W2Ngh3o2WyYhfXjyZvL5gbTQzrrleBjVXmoRMKZDKgvrTGRuKyFs4tGgCWwuOxepcGcf1aHk28j SgQ== X-Google-Smtp-Source: AGHT+IH4SzFcgriEZeEyINeEOYHsKsBduZvh1a8UF65TGxe4ng4DHNhTgv3meBOCFXGKpW1iswhVW2S93bM= X-Received: from plrd9.prod.google.com ([2002:a17:902:aa89:b0:223:225b:3d83]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:19cc:b0:224:24d5:f20a with SMTP id d9443c01a7336-22c53620caamr57870345ad.48.1744998620802; Fri, 18 Apr 2025 10:50:20 -0700 (PDT) Date: Fri, 18 Apr 2025 10:49:59 -0700 In-Reply-To: <20250418174959.1431962-1-surenb@google.com> Mime-Version: 1.0 References: <20250418174959.1431962-1-surenb@google.com> X-Mailer: git-send-email 2.49.0.805.g082f7c87e0-goog Message-ID: <20250418174959.1431962-9-surenb@google.com> Subject: [PATCH v3 8/8] mm/maps: execute PROCMAP_QUERY ioctl under RCU From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, david@redhat.com, vbabka@suse.cz, peterx@redhat.com, jannh@google.com, hannes@cmpxchg.org, mhocko@kernel.org, paulmck@kernel.org, shuah@kernel.org, adobriyan@gmail.com, brauner@kernel.org, josef@toxicpanda.com, yebin10@huawei.com, linux@weissschuh.net, willy@infradead.org, osalvador@suse.de, andrii@kernel.org, ryan.roberts@arm.com, christophe.leroy@csgroup.eu, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, surenb@google.com X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: D8804100008 X-Stat-Signature: j4yryccr6bfwwn6b4wqqabfiynah4bnk X-Rspam-User: X-HE-Tag: 1744998621-25102 X-HE-Meta: U2FsdGVkX1+s1SNeQqKuSK3hhhvpdBEzNa4XGv4qWFZ4zF30xmOlUOTfzZB2YE9E7c4s3Sxbv+CfnFSJh4VXlgfeDhH4pK5XD7/NpLzPr6OCPQOSYMpLmv8DouXfbT/8bsp7D94RLGFX5j86fwVrm+wy8O/kxcIihdyvNqJFxI1jJrio2dMc1fu21g8NNfg5txnoF5Bm1E7GmAzrpkfZNoxF+SXDK0O/JP6C7naNS9gpmB3e/IBbW1NF1p8YMSQoTj/9jhpWmmBPBLgEq5gtjYynArfu0WvhNqS5ALB2q+6JnsGe9MkjGJY23DdniqoEctCX65JsbnEahPBkPKwGsI7NHnbbPjB2rBfRlMEyY/tFUGAi/xY0Um0RHrjweCSJFd3bxjQDfPk2qg9xrX7dbTcDuNUbd2VGQxigHuD+OLz134C4dhY+9so2pp5ha6Q5vKV6VQR3HmKKoxDHQYoitSfROu8FyyX+W8+r45n3vWQxvyaKynyXh6OdRlcO9S7Ey3eMbaCF5vFoTckR99nRhdf8RMkBELcH7zgerEe5SvaY5FDSJlCXfrqkVL/MIhDRrQhyOE8qdke5fOxVQ56tVJsj/4vvCKIkEBIrNe2LWt7RwQVIP9cpI6xvDWZu264QiJ97+IYnT3n76cvNRyh1VHhORe/dpkNtq9OOoF288QbWpdF1zqKRygmmdrsEjPDGBlF6T6vhOwTIJEhEzXq1ZK6QPeSee/smpWXY5Vn8YZjvZsU+eao5xsNrJ3yTBnYd6OQt1kpeN+f00sQjj2oUDMCnekOg+2arR+MYOtV97OtTUiILYa8fR2ChqzKWYrP6I90SnafjDaTE/Lvtv81fh/XuO52YDXuBk4APBjieHy9cVQZI/eSNYATb7J8025jJLJdNHGm2sGsgSrCgcHxgNBROm/5lqcUON4Xbt1xwDkxVp9P+kDkO20Hx3KSl1Z/Vo1fq8twcvfIXcY/LFXy oN1wT4FD LdJHCqXMPaU5oBk8zjtEdaOuMQpcK8XW9BsYhCQTfjOnyL4PI9rS/zkKRuAWhF6SNY/QAbN/AxQlTwDiXH2PA4WXcuNpZUg5Qytnr/WpNM/cQrUCZTZI5W0/4251Y7r3OFMryaWjSBjEZIc8oQHn6axEzx8QrXB+BazL/vGEFgMPiwNMe7uhF5tF5QS/EQXwp4WFT6nLMDpoRhGtO2COPLdG1DccPvj3YmXmLhgU6w7N/JvUpQ3tqVcADwWH9WYqcMd/pQc/E7nYHh3zp6ZsMMKrtsGFNrjqH8CljQQHiOBuVLtNosuisJAe0pwOmrWYfAfSo3LsBggzMu5AG010sJr0epvz+nFname2Y0vt5BvtZ0iTcCgQ1MPUKhlGnOrjAR5vKsy1iOyDfsdu5VFCNkWfysrHd/geVzKetHRsO0+1t3M1yIeEULKdC3QFdQWVePzso+cuRHZ86MCgU117YoQdTpvPG1Wbod77wsguPfnExUbsxRpM4Z9qyJBUCLYNQrLcGopCZqNVDd1HAzDguJgdGLXaXaP2bR7ieGYP1kJKlbzN5pPIGuYab+tSV2PpvuGUvfX3EyGkJdBY+4tbs68uLbQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Utilize speculative vma lookup to find and snapshot a vma without taking mmap_lock during PROCMAP_QUERY ioctl execution. Concurrent address space modifications are detected and the lookup is retried. While we take the mmap_lock for reading during such contention, we do that momentarily only to record new mm_wr_seq counter. This change is designed to reduce mmap_lock contention and prevent PROCMAP_QUERY ioctl calls (often a low priority task, such as monitoring/data collection services) from blocking address space updates. Signed-off-by: Suren Baghdasaryan --- fs/proc/task_mmu.c | 63 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 55 insertions(+), 8 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index f9d50a61167c..28b975ddff26 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -525,9 +525,53 @@ static int pid_maps_open(struct inode *inode, struct file *file) PROCMAP_QUERY_VMA_FLAGS \ ) -static int query_vma_setup(struct mm_struct *mm) +#ifdef CONFIG_PER_VMA_LOCK + +static int query_vma_setup(struct proc_maps_private *priv) { - return mmap_read_lock_killable(mm); + if (!mmap_lock_speculate_try_begin(priv->mm, &priv->mm_wr_seq)) { + int ret = mmap_read_lock_killable(priv->mm); + + if (ret) + return ret; + + /* mmap_lock_speculate_try_begin() succeeds when holding mmap_read_lock */ + mmap_lock_speculate_try_begin(priv->mm, &priv->mm_wr_seq); + mmap_read_unlock(priv->mm); + } + + memset(&priv->vma_copy, 0, sizeof(priv->vma_copy)); + rcu_read_lock(); + + return 0; +} + +static void query_vma_teardown(struct mm_struct *mm, struct vm_area_struct *vma) +{ + rcu_read_unlock(); +} + +static struct vm_area_struct *query_vma_find_by_addr(struct proc_maps_private *priv, + unsigned long addr) +{ + struct vm_area_struct *vma; + + vma_iter_init(&priv->iter, priv->mm, addr); + vma = vma_next(&priv->iter); + if (!vma) + return NULL; + + vma = get_stable_vma(vma, priv, addr); + + /* The only possible error is EINTR, just pretend we found nothing */ + return IS_ERR(vma) ? NULL : vma; +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static int query_vma_setup(struct proc_maps_private *priv) +{ + return mmap_read_lock_killable(priv->mm); } static void query_vma_teardown(struct mm_struct *mm, struct vm_area_struct *vma) @@ -535,18 +579,21 @@ static void query_vma_teardown(struct mm_struct *mm, struct vm_area_struct *vma) mmap_read_unlock(mm); } -static struct vm_area_struct *query_vma_find_by_addr(struct mm_struct *mm, unsigned long addr) +static struct vm_area_struct *query_vma_find_by_addr(struct proc_maps_private *priv, + unsigned long addr) { - return find_vma(mm, addr); + return find_vma(priv->mm, addr); } -static struct vm_area_struct *query_matching_vma(struct mm_struct *mm, +#endif /* CONFIG_PER_VMA_LOCK */ + +static struct vm_area_struct *query_matching_vma(struct proc_maps_private *priv, unsigned long addr, u32 flags) { struct vm_area_struct *vma; next_vma: - vma = query_vma_find_by_addr(mm, addr); + vma = query_vma_find_by_addr(priv, addr); if (!vma) goto no_vma; @@ -622,13 +669,13 @@ static int do_procmap_query(struct proc_maps_private *priv, void __user *uarg) if (!mm || !mmget_not_zero(mm)) return -ESRCH; - err = query_vma_setup(mm); + err = query_vma_setup(priv); if (err) { mmput(mm); return err; } - vma = query_matching_vma(mm, karg.query_addr, karg.query_flags); + vma = query_matching_vma(priv, karg.query_addr, karg.query_flags); if (IS_ERR(vma)) { err = PTR_ERR(vma); vma = NULL;