From patchwork Sat Jan 11 04:26:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13935781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 125AAE7719A for ; Sat, 11 Jan 2025 04:26:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 394096B00A8; Fri, 10 Jan 2025 23:26:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 342E36B00A9; Fri, 10 Jan 2025 23:26:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1959C6B00AA; Fri, 10 Jan 2025 23:26:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id ED54C6B00A8 for ; Fri, 10 Jan 2025 23:26:42 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A40CC1210AC for ; Sat, 11 Jan 2025 04:26:42 +0000 (UTC) X-FDA: 82993885044.24.89134C1 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf14.hostedemail.com (Postfix) with ESMTP id CEF07100002 for ; Sat, 11 Jan 2025 04:26:40 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=WT37ABmU; spf=pass (imf14.hostedemail.com: domain of 3__KBZwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3__KBZwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736569600; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sqweZn90u3nkindBXw7O0ID50FUbfbOsDRIP3ctel0c=; b=5Qw2zOPMsTfB3L0KD8T9EkK3pu0gyEjUvKqR7KEVZVyUHcs6JCCwmafDZRVdNKdjZTIccm +xmkFj7J+Tqi427CAxMbDgvPio3JyKXvhfNgZ6wUYzHB8yGBS1KdgvIydkXzGfCf1T+8if AjV/rfGyp9x20upfBq1iu6Tcu3awKcs= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=WT37ABmU; spf=pass (imf14.hostedemail.com: domain of 3__KBZwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3__KBZwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736569600; a=rsa-sha256; cv=none; b=vQHL9m4FW99azlqXL7fPqo+ULGI0bEN7JEEHQOfP7b+7yZH+wiTf5h+23s1y9ojdQoiuto 1bWwxgZD0VsU4WeB6oAixlJ2x1xYwIXxa2HhmixstdqyFcPZGuOGm7OIp/sb21o9SAmoBH wUK172Hca58z9kjVMOXstNs9OtD7RzU= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-216387ddda8so52050045ad.3 for ; Fri, 10 Jan 2025 20:26:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736569600; x=1737174400; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sqweZn90u3nkindBXw7O0ID50FUbfbOsDRIP3ctel0c=; b=WT37ABmUID1zJ6zDdDmFtfAGIE/Ey7pY7hj6j1/zjZEStyZnilrgF8CD0nNCpZ4c7y AxV4jPJDYlRhuhAXdBQGeGPmaA8PkPuMhZxUgUjAtp3qMU2WjPNGpAeBdN6Io4mSYaEX HO9gZbS8jJixq1L8ouhSTRxOwQCh89OmzL02pYYUI2sxGvIrtg0XgNeKBi5kvR6ysUy6 HKrPpMUHSql1ddNlwjaL3xBPxYdWxGIwBYmNkNyf+7M/f4scseYwnHse5xjIwtVGTs0C Fi7yh0OvVzM9wU5Yr+kyfCKPFyHmo81CuESkaJMBOTcQq/Jo00s9Jh7rAliqm+dCkC3J dmdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736569600; x=1737174400; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sqweZn90u3nkindBXw7O0ID50FUbfbOsDRIP3ctel0c=; b=KE4G360AbWLjkCXuxN8i9s0cvq6PFOGB/sk+j5q9d7PYFMK/LaP/JBQV2UoxIbUngg NMUZamtBgq7AZyuaOb1lBAvlX+KPa2y2Ivi5RCHITBqwwQQ20KwbIMsD7ec9+p7fi3p+ YWiNuewE4pQcVzQovvNifk1hLiBHNrZmihFPNqr+qf/6PSnhuy1Ze+zn2i/U2lS4391h 3cc6qX3BJJAkCOgne/jj0lsD9AML+EyHJLeLYXHu/I6g5nkt1tRhj7inCQE3Vc3BMPk8 OEHqQstvc9TZSCI25yIGCRC4YZic9TCDN+/4t8/HOfNWQHFt0juj1tgqyuHGXZRdpdOE TQVA== X-Forwarded-Encrypted: i=1; AJvYcCUy51abZhCKmvo1vgabtJ5kPPxBZuafFtJxxmvvJSeN6qFamL6RFlFp+1sILJK8omuifyZ23kUNgg==@kvack.org X-Gm-Message-State: AOJu0Yxq28TyS7DNOowq8bSHN3IkTI4mgvPRoTsPxQmSj8vtbuqABaXL UYiGvXmIQ/ty1bP1cie/YyHwgoq/9VDR4nvDvZFXrxS81LfcCSZ0cHf8isu/qMq0CuPFyOljOxN ZKg== X-Google-Smtp-Source: AGHT+IEDi9vLxcvUpqCWBPMXXF0Y8lrMfIatGu39C3DhS1ZkGtjS5dUciQm8YtZJQiW79lEU0cSQ1XI24iE= X-Received: from pgg14.prod.google.com ([2002:a05:6a02:4d8e:b0:7ff:d6:4f07]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:c996:b0:1e3:e680:8c91 with SMTP id adf61e73a8af0-1e88d2d5ea9mr24704039637.31.1736569599645; Fri, 10 Jan 2025 20:26:39 -0800 (PST) Date: Fri, 10 Jan 2025 20:26:02 -0800 In-Reply-To: <20250111042604.3230628-1-surenb@google.com> Mime-Version: 1.0 References: <20250111042604.3230628-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20250111042604.3230628-16-surenb@google.com> Subject: [PATCH v9 15/17] mm: prepare lock_vma_under_rcu() for vma reuse possibility From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: CEF07100002 X-Stat-Signature: ancnfb3bwyqyj85gnukmzieapgwzwenp X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1736569600-701351 X-HE-Meta: U2FsdGVkX19WcPrPOtniMuI0iTR9v1YrVFkaikYjF7lOf3d7flt1+xCh98XyafqN4vrDBCz2d4101ZnWJc8q+htDS57ePg5O3ItEVdIV0FpUkAw7uSLe8FqUkJgd7hD1lH1SulWs5/FWuy1d4IzqwdeSTyLm2MeNNsYUE3Re6W52h0Mgi5Vnhb+Bdbdr8BEtY8sWVuf43lPMGcB1w1LPDL4o/tLGbfEpZXE7+oChTMu1e1rTbYFn4R8/uq1PrelHhYI/kDz2CiylgfbdUsc4hHeQ3PiT1aG+FYd90uPaQ5Cb5JmDEM+grECt0PQufc3/WhJyjcyRv0yWrQn3D37v8PSqYT2rJIN9/OOqMS0ZLvCmhHJ62h6oUnFAyrJa1AxDtjhpxBKUrpBk3U9xL96TN1kjbuW8UoQjF8+/HdBqBl4+327Fp1rQZiTY/qk7rRsaSTpPnjX8hiTSaDRLQVd2mxBo7wgnRyF2K+9AXdmzaREo0g8hosYFJd4r3G2iDMHb5oR8Evnxkd9nFeRymZfQlVkLf4UMDHwZdmMv4eglHhXSWaOTmgLYB4GRH+ICsI05T1cpBmjLXohj9M2jHHECSXDl+4crOkgnk/mcKgq4Tk/oVFYnpHpnNqX4D4M7nZN9s9cyrptH/z2yRo/zh085897468j934J52IxXIzUGX9ZZf460xSojKshwISTNQwutvZqNlef/yqh9YjEmXXGicw7PMyVNzRhRbktxZyLm5rPeORgg44dolZkViHp0dHaNLh1rihdAUxyUk0txa81ZgNABBAE66kkCtCoXPrKVYJBnkefp5JvEVvAIoPSLzMM1ccznacUA2PZXTmJLCyUo8X0CjYVta3FUO1fqLuyuqUEjl4bw34QJyyy6/WVMzlSI+ECkz67sfsr42c5MNW6Rdy6GYX07salcx3RJ2aVzC+/nWuN5BhsbM89RkGJFkNjZS2kuMdGIavrSQAPdV1H 2WllbebE tasvaK3JjSfw19RMwujPIYoYmhyQoavcGf0d58taxpareep0ia+8Ynp/zRNDvEZvV/qJEtAkYbLEfCwtioaQ7stHwvcqgIyqRQnx3r2p9fDMdU/0MLK5V/b4zfYnfKQlZx5WPfFR4ztkxTlsmg+reziznXl/aDj7/FltctziEjKenXhle/Eyd8cmoVp+jh+3F/MdTKrgJjG1jjkl1IPh0NR5yQZYW8pE5Am92pTrGHIO8BIFAkrzblBubsVytVq28YAiiAlKyloBvbtGTjup5jc9nEA5pdbZXXymjTdBeTU3l6MR050tw+tdHkLbvZs2X5EB6b0XDG/0CR7i9ru9gUcSGei52BHmBcfiFxdyRIcZmzijVdtAr+B1eDB7eIPvDN3/jBLJZJDMVG+muJ7X3cd6KdyJBgTE/emwD8JnVznpojzuUU0TbTslMBu0LuMpk0zkNLDxY7zUUp1IndKHScuR/yTwhkFtCdwJMobbfLRd3XIiN4dPrracv03aBEOFTgm6sAuMSKPLtvWUYBOk+SakL6LkcfXZ7lh1PN7Si+qo2KfclPc+chHBMEd92RzcqSZc/s5mkIXsnMqgGwcr0kpnKdhCdcRF19+Qjt/04EJNqV6TiTYKeTWPxScGQDkuhbmH43CkM/WQh8MRqR6sUB6p9g13nKL1rItw+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Once we make vma cache SLAB_TYPESAFE_BY_RCU, it will be possible for a vma to be reused and attached to another mm after lock_vma_under_rcu() locks the vma. lock_vma_under_rcu() should ensure that vma_start_read() is using the original mm and after locking the vma it should ensure that vma->vm_mm has not changed from under us. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- include/linux/mm.h | 10 ++++++---- mm/memory.c | 7 ++++--- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index c8da64b114d1..cb29eb7360c5 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -739,8 +739,10 @@ static inline void vma_refcount_put(struct vm_area_struct *vma) * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to * using mmap_lock. The function should never yield false unlocked result. + * False locked result is possible if mm_lock_seq overflows or if vma gets + * reused and attached to a different mm before we lock it. */ -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { int oldcnt; @@ -751,7 +753,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * we don't rely on for anything - the mm_lock_seq read against which we * need ordering is below. */ - if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) + if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) return false; /* @@ -774,7 +776,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ - if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { + if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) { vma_refcount_put(vma); return false; } @@ -906,7 +908,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt) {} -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { return false; } static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index dc16b67beefa..67cfcebb0f94 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6426,7 +6426,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma) goto inval; - if (!vma_start_read(vma)) + if (!vma_start_read(mm, vma)) goto inval; /* @@ -6436,8 +6436,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, * fields are accessible for RCU readers. */ - /* Check since vm_start/vm_end might change before we lock the VMA */ - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + /* Check if the vma we locked is the right one. */ + if (unlikely(vma->vm_mm != mm || + address < vma->vm_start || address >= vma->vm_end)) goto inval_end_read; rcu_read_unlock();