From patchwork Mon Dec 16 19:24:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A2F2E77180 for ; Mon, 16 Dec 2024 19:24:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 449BA6B00AF; Mon, 16 Dec 2024 14:24:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F9016B00B1; Mon, 16 Dec 2024 14:24:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E6696B00B2; Mon, 16 Dec 2024 14:24:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 120266B00AF for ; Mon, 16 Dec 2024 14:24:33 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id AF3471608EE for ; Mon, 16 Dec 2024 19:24:32 +0000 (UTC) X-FDA: 82901797860.30.FF80CA2 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf30.hostedemail.com (Postfix) with ESMTP id 3F16E80009 for ; Mon, 16 Dec 2024 19:23:34 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JRHdqEUA; spf=pass (imf30.hostedemail.com: domain of 3bX5gZwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3bX5gZwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377048; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tPcYammI4P+bTAfQ+4gtRvMOMTfi6gzN2lD/68NUW0U=; b=BJYC5gUtYz2jEKTRGdu03qNGSUkkJf3zOUat4+M6/tQU8bsPiQwoH/d1ho/J75q5ubgzNh mPiXrpmoACMs3xjYrIrKzieOGMwmMiXEJryqaJ02dkY1jXZoulJkEuPTzSzrS9elr4UD1Y hIqEluTaiI8xFNzc4PoPd/dQAy2xvQk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JRHdqEUA; spf=pass (imf30.hostedemail.com: domain of 3bX5gZwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3bX5gZwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377048; a=rsa-sha256; cv=none; b=71fLUg/YlF20MESwKV8lsPGxpb9JeWince49eRgtSnpk4AsUSDSCwGuebz9SYEYwidvF8/ Toem8SnoGbXHYZqyZNH2DUJqlUpK5BSgJwxvJa47VxzCMLkdsYGF3F/KhViVCc77ARfEF3 lOA6Rxe+pyweoUph+ixT2r5pMxH0jSI= Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-725c882576aso3306792b3a.3 for ; Mon, 16 Dec 2024 11:24:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377069; x=1734981869; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tPcYammI4P+bTAfQ+4gtRvMOMTfi6gzN2lD/68NUW0U=; b=JRHdqEUAKyiImQJ/sgDagfw1r9Zmw64U/QFhfe6dAZPy0JQqqPrc0jR3fBzO4QOeLL 3Qx8o9i2L3/+vWcgpUZLIBjfHDNxhmM8cuBCpxSJCSAa0d4kQusXmKg8eM1sBwhNYSbr AT0qYm2xp6GJ9mzufnFroqwLMndOQRYQIDvady1SQry+fZ/vdueqF0ImIs7StdqE9kc9 eofH8czmx9L6X+vi2x5DLfhPblcu1fiiKjhqHZEfe0VuH/TtX76YKxVqKo+L5l41hgm2 bRkHM+E/2xAlQBm8yYfHnwnY90dXuO/4IlU7JoMG9BFL5E7to/ih9G7v2V2vhXpBoawd mPXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377069; x=1734981869; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tPcYammI4P+bTAfQ+4gtRvMOMTfi6gzN2lD/68NUW0U=; b=d3+28XPt1AWFU6UURN/ptcro/xxYTyJ+kuKUvLovdIqrjjxOK3+TsnBWS+PopgNkHa rqXZGKK4mYVEtkDfWyk6rrYlwXRt3Kvk26xnaD4ySYR5IZi95AngfBmNlnk/hq/EUbU5 ZrLkdbUck9fpViQBmZDCzC93j6W3lkOjcD3WpdZ5xe1DcUuQQ114dm7rZd+X6e2HwkYl 5VaG8nwLITIYf7lxgqTKbAQeJ8oStmNAAlca93cYM1N57Osms8pj/sFzR0T3yucIFiZS 7B4hX5I8ZQS0WLKZ5w11LPWNtHW3HetTdIy3sdH0RDguyZePhh7eN8mo14B3LUgjUaLs PauA== X-Forwarded-Encrypted: i=1; AJvYcCVk23KvoKlCFixnuUiOefMJY25Mmer/e1JeHx1rExkLZxadIcFhFU2M2F61sMAfieMM+SR/QJA0vA==@kvack.org X-Gm-Message-State: AOJu0YxUkDRnVtnK6fKvGMc66IiXSAZTRSA0v3/hs2RZNmWrAH+Zq3eN xtWQKFXuvY05cqJnoeINRfDkfLTZaNdBBUdADiOsFotH7Pyo4TN1L5kDjH6g7jqff8319CfG1L0 HNw== X-Google-Smtp-Source: AGHT+IHuIz3KcGOEBSL7UPXAppbQsIUkVSUTnN31px7An8YO7VAcBSifJ3+A4XtsuM0O2Sd96qkE9oQILpU= X-Received: from pgbcz14.prod.google.com ([2002:a05:6a02:230e:b0:7fd:4e21:2f5a]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:a120:b0:1e0:d32f:24e2 with SMTP id adf61e73a8af0-1e462d97415mr1129422637.38.1734377069542; Mon, 16 Dec 2024 11:24:29 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:06 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-4-surenb@google.com> Subject: [PATCH v6 03/16] mm: mark vma as detached until it's added into vma tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 3F16E80009 X-Rspamd-Server: rspam12 X-Stat-Signature: c98qb9ip7xhnmq7dm1c9h3xioynobqt5 X-Rspam-User: X-HE-Tag: 1734377014-239697 X-HE-Meta: U2FsdGVkX1822SFB/y09aTPd07rG/wKQkb3DevCTXlxGk0qboU/Qpu4hMi3euDrCWq7+zB+1dLl/7CZ8F5hiRKI+me18lXrB3x6fHzpeSrFX2LtDxkXFyKF6+j3tNqDp3t+tSbWlV5UrjdQnKWzY5IRKbGjN4EG40VBKUKcPoOaNvGnwHPkbYywHWayt25pWrerSdaesYhCGIMLYZrRjrkl+0q4DSmvC7iToU/245QfTGTtY8tUPgC96lZxcNDl/xyxReLCEHoCabe6fjoiw8/qmgn6PvXNRjTMzZh5YGDHLiJYEnXZ/l17ALairzVstP3xXwl43yNB4r0O64mKaf1uNBLXo8Bn8S4kAxgC/UVY2/XUnQ5ncT/4MBD7Y+HlM7rmNSkMHNEsKG1/kVWx4hgacMbBwqN08MqtdDhTBTb6S6q0YOJFr5cHZN14k9fICjN+UsgydMZ5b8OKik8yKGqvie8VXRhwm4tqHYBv8kkgkWF1x/2t2j2nwHQZigOmh/20rg1WvSb3UPFEAxd7P3ssbSUhwz/CHFCGtJQOV9LKOqSSzrkNqUPTDGV2JzP4573pl2/VJu/mrAdBsPJBzQ57TSyG+TGcPJqncs7vY8ZlcvCWw8vU2t/DUSToPruzB55BxFY6RhoAeIYOySPW6Ec+25/2P5oxtLwyLnpkch9UL5szRLaFad5dtGBQn1+Z+c5xFy6xg7wOQ29F5CkJ/c8uxjJaxRT4FnlA5b7E7m70NAtRgn3LScYI72EDEyknu30Qe4uUtxAHDzbbBvw7PlaedIuYUnYM2klyalTO+5hJea86eS4OE8RUqM6iLC0hhETfDKXlYztXWsXd2TVw1KCV+zAWRNGDVXribAk+XlXC6KHW9RjEnWgIULXc3AO5xvSAWAoknL9OhV4gMZBkNkP3+rGiPcC17Mq7Qd+7TGwfsfUdbNWz6gGZAxCh0Y2QasTfCqUkonPde5ee6eDl ovA4MYw1 4UXioxJ6ht93M4o+rQiFXdz0igpxravvixCM/EXBvQ5DmgkRC4wYCDTuMFQnuVlT/z2sfqGGB4y4OlRBJt8/DzAu4KBe/70BwcUDUiYuJ2Zng6kSWoPK/bHLCMF+lbrlVW/vqxKpI6wE4AwKlfkxub9nmqNtwbVaujWmhrZEZg47qbYNs1TjrkRotEB1gUtGKKVI7tT5sahc6pbWg54JT/TL2F1FljscOiJh5OBSuzmiTOFCS6uXIICg+wBZy+FxqWdICSk9w5OPcKbwD61QdH8CkdCqfLEW0ioWE0iydCaXbUYdiBhdRbRw4e89ROSgSokxiIgq5bQDMFLqw2y9A8QZg3VhflE9cl4UZmOHf999ZHpwGnia4E4FSthfkOkDwa/BbgEhSEc66KAADE52v55lBZI371rCw+tLuZ4szxiPI9HgbLcHPFukqBYL8ThoUsQWBzX13aoaor1JmC5tN/EDRmsD0OChc696sy/PN71W5oHOgxcMXv3bFJJntB17P/eMbFXmU/sJgXHri/QGG3804V+WHzhPgI5SrmxZVWBZxW15bqnYo+8Heqvm+X6BRHSGz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree. Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter. Signed-off-by: Suren Baghdasaryan Reviewed-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes Reviewed-by: Vlastimil Babka --- include/linux/mm.h | 27 ++++++++++++++++++++------- kernel/fork.c | 4 ++++ mm/memory.c | 2 +- mm/vma.c | 6 +++--- mm/vma.h | 2 ++ tools/testing/vma/vma_internal.h | 17 ++++++++++++----- 6 files changed, 42 insertions(+), 16 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index e1768a9395c9..689f5a1e2181 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -821,12 +821,21 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) vma_assert_write_locked(vma); } -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; +} + +static inline bool is_vma_detached(struct vm_area_struct *vma) +{ + return vma->detached; } static inline void release_fault_lock(struct vm_fault *vmf) @@ -857,8 +866,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) { mmap_assert_write_locked(vma->vm_mm); } -static inline void vma_mark_detached(struct vm_area_struct *vma, - bool detached) {} +static inline void vma_mark_attached(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma) {} static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address) @@ -891,7 +900,10 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; +#endif vma_numab_state_init(vma); vma_lock_init(vma); } @@ -1086,6 +1098,7 @@ static inline int vma_iter_bulk_store(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } diff --git a/kernel/fork.c b/kernel/fork.c index eb3e35d65e95..57dc5b935f79 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -465,6 +465,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) data_race(memcpy(new, orig, sizeof(*new))); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; +#endif vma_numab_state_init(new); dup_anon_vma_name(orig, new); diff --git a/mm/memory.c b/mm/memory.c index 2d97a17dd3ba..cc7159aef918 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6350,7 +6350,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, goto inval; /* Check if the VMA got isolated after we found it */ - if (vma->detached) { + if (is_vma_detached(vma)) { vma_end_read(vma); count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ diff --git a/mm/vma.c b/mm/vma.c index 6fa240e5b0c5..fbd7254517d6 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -327,7 +327,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, if (vp->remove) { again: - vma_mark_detached(vp->remove, true); + vma_mark_detached(vp->remove); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -1222,7 +1222,7 @@ static void reattach_vmas(struct ma_state *mas_detach) mas_set(mas_detach, 0); mas_for_each(mas_detach, vma, ULONG_MAX) - vma_mark_detached(vma, false); + vma_mark_attached(vma); __mt_destroy(mas_detach->tree); } @@ -1297,7 +1297,7 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, if (error) goto munmap_gather_failed; - vma_mark_detached(next, true); + vma_mark_detached(next); nrpages = vma_pages(next); vms->nr_pages += nrpages; diff --git a/mm/vma.h b/mm/vma.h index 61ed044b6145..24636a2b0acf 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -157,6 +157,7 @@ static inline int vma_iter_store_gfp(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } @@ -389,6 +390,7 @@ static inline void vma_iter_store(struct vma_iterator *vmi, __mas_set_range(&vmi->mas, vma->vm_start, vma->vm_end - 1); mas_store_prealloc(&vmi->mas, vma); + vma_mark_attached(vma); } static inline unsigned long vma_iter_addr(struct vma_iterator *vmi) diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 568c18d24d53..0cdc5f8c3d60 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -465,13 +465,17 @@ static inline void vma_lock_init(struct vm_area_struct *vma) vma->vm_lock_seq = UINT_MAX; } +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + static inline void vma_assert_write_locked(struct vm_area_struct *); -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; } extern const struct vm_operations_struct vma_dummy_vm_ops; @@ -484,7 +488,8 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; vma_lock_init(vma); } @@ -510,6 +515,8 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) memcpy(new, orig, sizeof(*new)); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; return new; }