From patchwork Mon Oct 28 01:08:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13852773 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7569ED13590 for ; Mon, 28 Oct 2024 01:09:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 07B066B0089; Sun, 27 Oct 2024 21:09:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 004676B008C; Sun, 27 Oct 2024 21:09:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D97746B0092; Sun, 27 Oct 2024 21:09:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B0F906B0089 for ; Sun, 27 Oct 2024 21:09:12 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DAAD4A1D5A for ; Mon, 28 Oct 2024 01:08:31 +0000 (UTC) X-FDA: 82721226672.30.8450AF7 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf17.hostedemail.com (Postfix) with ESMTP id F3AC640003 for ; Mon, 28 Oct 2024 01:08:52 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=XcEsDuxZ; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf17.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730077696; a=rsa-sha256; cv=none; b=G3HmU0GEswxDpAXQInIXaBBDsWYqshvjTWRAupkxw9wFuhwLuD2tIdIL5QHGUQBHsXHCnx /lDdbb3ZlwDnKKBFXBE+PDSTg2sWRs8u1LkNjHIbm8FmvkJ8gFfs4DwBpr3wKg47MaV8E7 UvXi4HeAW7a2zp0fGf/J4fpUBpKbiHI= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=XcEsDuxZ; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf17.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730077696; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aJ9yRWXYZ/yelMyUDnfmysXw9x3hWWCe4ChOBPCba1Q=; b=WcgDxTfOUI9Ro7JhM4bhRYZCO/e+Fw6Txstuyw0F3YRAFokLIo232jojcFQ4v/T8lRTfci mG4s1DxzwUsAOS2jZfC/bJq6CU+Zw30FGRzcTuIgCUhBIfWoyRRluHQ62F+p7rvQvqAvcw SqG+EPbR/6J4NjiOxNoR3Yn6Qio4Ewk= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id AC2A65C486B; Mon, 28 Oct 2024 01:08:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 65703C4CEE5; Mon, 28 Oct 2024 01:09:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730077749; bh=Ribok69L0V+IQIfX1LKAa+j7slCTXJ/a5HygjFWCqNY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XcEsDuxZ44cpUaR4KUbORo/ERRrafxKAF05h7Dd+n1ohShoR33iYzss2xVzkX9bZy NUA2c2/P93p+1NY9BMPbz4ZhkGNAColwIYcLIkLlh2xb7UNFCr/2OVBIwND0TfPClO beYu28enquoY3AqAvW6Rn0Hh1VpYSsaOhhkqDSzC3pNPmq+ARjE5S+TFvAdIRba7vC B251sDrDieSzrwoxJpURaYek9EDwPpQG1pvuqNwhl+W23/1EpA/uFDnIxtKnlI9F/A FmYTcm7H7O71F77iLvdRJxipSDNTwtQR1UO+1v/V0OA1iqT0qzRg8WsMBTP86OkiSe E2A9BGrGM6ZnA== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, peterz@infradead.org Cc: oleg@redhat.com, rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, shakeel.butt@linux.dev, hannes@cmpxchg.org, Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, david@redhat.com, arnd@arndb.de, richard.weiyang@gmail.com, zhangpeng.00@bytedance.com, linmiaohe@huawei.com, viro@zeniv.linux.org.uk, hca@linux.ibm.com, Andrii Nakryiko Subject: [PATCH v4 tip/perf/core 1/4] mm: Convert mm_lock_seq to a proper seqcount Date: Sun, 27 Oct 2024 18:08:15 -0700 Message-ID: <20241028010818.2487581-2-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241028010818.2487581-1-andrii@kernel.org> References: <20241028010818.2487581-1-andrii@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: F3AC640003 X-Rspamd-Server: rspam01 X-Stat-Signature: aa5t8o7grsz8jfagod7g4w75b5kgzg7k X-HE-Tag: 1730077732-82833 X-HE-Meta: U2FsdGVkX1/vErlgRfYWsP/s7TXrE12mf4uEvwTllA/BE4Oh4CupPfAjxAKe1UAmHtEqc7nNKy8viq3RNl2kXT7q52sBol7b3WwnpO+oE6bBsGRMVY0lFHuWSP9HvgqR49Rl4CLFib0DVysI5bbmoLTy8ulwuqxAATjh+3xVcdiviThpeELbk2HPnD3Gf4obbyOY697UO/uZzGlCh/JochL5uwtsbMAA9O/57BR9pjy9F0SyXmxfteIKUsFxy0IAOaODBWqyNUPOgGQ6zczZp9KBezokIUaPH6iy8By5lw9+sKzUuqxMvvOhJYg676fwOpvOYyuO7strquJRodRYdwDYOjcJGu8t12ZpT047T5wW8cu9HNBnlkwXb+xltjxMQv1ZfbHcLYbA6CxSBijeD1JMGzpX+mAPdx49m5TgTS8Dym+IV9CJ2FkDen5DJhNqgph+y3eDxvWZ4oBZP7JSAQcMlWIFQlO7JUJ+PFzoHqslcManjCRVS9pIr90IaLMwiuV60tTYDlR5h+Ok0iTwPRa+rBuCPYI7APY9DY0XvaYa8KxhwXzc6jqnCCMJaqDD3+iut09MkjcdvcR1ClzxwxM3U75OiV31Eql2aNBKoBWB5/RKc2NSukbgX1GMnh0KXN/VLZoYM41F5GwdSFtMckqecsYIjEAD8ooZxjmcszOyTBPX8Z/dBR9kyNZ+smyH5a42cG17VRiYTveyQkFESTjyQdHq0ViiWFeUTKTNZ88TC7YbXccGor66fpbsur/FUYOkgMuBTYbLNk9uV2y0Usry9SAGPnFQI5zSIpirvoLT/NlBYQN9rbLt89oBtqFvUZl1Q3ngKrcB0mYkm3ptw2IfbH9ll7IHi28zvAv7RiurPoUTkYpLkjNxhhaQIAqK7Qdv2bgUG6/fWiGBtkrjfaKnr/wHDayQvGU7MbtloMstR+8ip/yGk+Rc/3U52mB1wQJdacLDiJzRoVLkQVl zqoqv9E7 o6MK6ZqMloEC+w20tY6l+zJOPv+a8xb56/XfyflbdZK5nGM3KT+18Fj0LmAuhOH0yK2zedmJp+28l9dbkdr0UX1Pcjcfhg0T1YtttoQbARZqBay727TDw1chlWl0ud1He3kcyGxfcTVKBrAXui5/akXrlZTswy/8NdjxeiXE3S5Y8ZJLLUNY8Ln4Uxk9KxMI0Sowxla9qHHyu6/d28XZ65KUzAX0l/u+o9mKJPNSPLQH610II+g9I5lNCc6dqSLATlq9XYWRqTSX/X8/VCuLKYEPyFiD//GqjROEw2xOZoAf5n0mKknOmhFFqiziVkvu1ZrbNcpi4zlLKGzkHLJ8xLYhKYk2lNXgafLvMoAVHWrd/wB5fnkkfb0HvJZKl9EGo3sJz8cZSJZeBr5qEkYJTCVPZT49IS+DbR+vgy+qxeD9guTaoMn22FyrN/Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Suren Baghdasaryan Convert mm_lock_seq to be seqcount_t and change all mmap_write_lock variants to increment it, in-line with the usual seqcount usage pattern. This lets us check whether the mmap_lock is write-locked by checking mm_lock_seq.sequence counter (odd=locked, even=unlocked). This will be used when implementing mmap_lock speculation functions. As a result vm_lock_seq is also change to be unsigned to match the type of mm_lock_seq.sequence. Suggested-by: Peter Zijlstra Signed-off-by: Suren Baghdasaryan Signed-off-by: Andrii Nakryiko Acked-by: Vlastimil Babka --- include/linux/mm.h | 12 +++---- include/linux/mm_types.h | 7 ++-- include/linux/mmap_lock.h | 58 +++++++++++++++++++++----------- kernel/fork.c | 5 +-- mm/init-mm.c | 2 +- tools/testing/vma/vma.c | 4 +-- tools/testing/vma/vma_internal.h | 4 +-- 7 files changed, 56 insertions(+), 36 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ecf63d2b0582..94b537088142 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -698,7 +698,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * we don't rely on for anything - the mm_lock_seq read against which we * need ordering is below. */ - if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq)) + if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) return false; if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) @@ -715,7 +715,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ - if (unlikely(vma->vm_lock_seq == smp_load_acquire(&vma->vm_mm->mm_lock_seq))) { + if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { up_read(&vma->vm_lock->lock); return false; } @@ -730,7 +730,7 @@ static inline void vma_end_read(struct vm_area_struct *vma) } /* WARNING! Can only be used if mmap_lock is expected to be write-locked */ -static bool __is_vma_write_locked(struct vm_area_struct *vma, int *mm_lock_seq) +static bool __is_vma_write_locked(struct vm_area_struct *vma, unsigned int *mm_lock_seq) { mmap_assert_write_locked(vma->vm_mm); @@ -738,7 +738,7 @@ static bool __is_vma_write_locked(struct vm_area_struct *vma, int *mm_lock_seq) * current task is holding mmap_write_lock, both vma->vm_lock_seq and * mm->mm_lock_seq can't be concurrently modified. */ - *mm_lock_seq = vma->vm_mm->mm_lock_seq; + *mm_lock_seq = vma->vm_mm->mm_lock_seq.sequence; return (vma->vm_lock_seq == *mm_lock_seq); } @@ -749,7 +749,7 @@ static bool __is_vma_write_locked(struct vm_area_struct *vma, int *mm_lock_seq) */ static inline void vma_start_write(struct vm_area_struct *vma) { - int mm_lock_seq; + unsigned int mm_lock_seq; if (__is_vma_write_locked(vma, &mm_lock_seq)) return; @@ -767,7 +767,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) static inline void vma_assert_write_locked(struct vm_area_struct *vma) { - int mm_lock_seq; + unsigned int mm_lock_seq; VM_BUG_ON_VMA(!__is_vma_write_locked(vma, &mm_lock_seq), vma); } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6e3bdf8e38bc..76e0cdc0462b 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -715,7 +715,7 @@ struct vm_area_struct { * counter reuse can only lead to occasional unnecessary use of the * slowpath. */ - int vm_lock_seq; + unsigned int vm_lock_seq; /* Unstable RCU readers are allowed to read this. */ struct vma_lock *vm_lock; #endif @@ -887,6 +887,9 @@ struct mm_struct { * Roughly speaking, incrementing the sequence number is * equivalent to releasing locks on VMAs; reading the sequence * number can be part of taking a read lock on a VMA. + * Incremented every time mmap_lock is write-locked/unlocked. + * Initialized to 0, therefore odd values indicate mmap_lock + * is write-locked and even values that it's released. * * Can be modified under write mmap_lock using RELEASE * semantics. @@ -895,7 +898,7 @@ struct mm_struct { * Can be read with ACQUIRE semantics if not holding write * mmap_lock. */ - int mm_lock_seq; + seqcount_t mm_lock_seq; #endif diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index de9dc20b01ba..6b3272686860 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -71,39 +71,38 @@ static inline void mmap_assert_write_locked(const struct mm_struct *mm) } #ifdef CONFIG_PER_VMA_LOCK -/* - * Drop all currently-held per-VMA locks. - * This is called from the mmap_lock implementation directly before releasing - * a write-locked mmap_lock (or downgrading it to read-locked). - * This should normally NOT be called manually from other places. - * If you want to call this manually anyway, keep in mind that this will release - * *all* VMA write locks, including ones from further up the stack. - */ -static inline void vma_end_write_all(struct mm_struct *mm) +static inline void mm_lock_seqcount_init(struct mm_struct *mm) { - mmap_assert_write_locked(mm); - /* - * Nobody can concurrently modify mm->mm_lock_seq due to exclusive - * mmap_lock being held. - * We need RELEASE semantics here to ensure that preceding stores into - * the VMA take effect before we unlock it with this store. - * Pairs with ACQUIRE semantics in vma_start_read(). - */ - smp_store_release(&mm->mm_lock_seq, mm->mm_lock_seq + 1); + seqcount_init(&mm->mm_lock_seq); +} + +static inline void mm_lock_seqcount_begin(struct mm_struct *mm) +{ + do_raw_write_seqcount_begin(&mm->mm_lock_seq); +} + +static inline void mm_lock_seqcount_end(struct mm_struct *mm) +{ + do_raw_write_seqcount_end(&mm->mm_lock_seq); } + #else -static inline void vma_end_write_all(struct mm_struct *mm) {} +static inline void mm_lock_seqcount_init(struct mm_struct *mm) {} +static inline void mm_lock_seqcount_begin(struct mm_struct *mm) {} +static inline void mm_lock_seqcount_end(struct mm_struct *mm) {} #endif static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); + mm_lock_seqcount_init(mm); } static inline void mmap_write_lock(struct mm_struct *mm) { __mmap_lock_trace_start_locking(mm, true); down_write(&mm->mmap_lock); + mm_lock_seqcount_begin(mm); __mmap_lock_trace_acquire_returned(mm, true, true); } @@ -111,6 +110,7 @@ static inline void mmap_write_lock_nested(struct mm_struct *mm, int subclass) { __mmap_lock_trace_start_locking(mm, true); down_write_nested(&mm->mmap_lock, subclass); + mm_lock_seqcount_begin(mm); __mmap_lock_trace_acquire_returned(mm, true, true); } @@ -120,10 +120,30 @@ static inline int mmap_write_lock_killable(struct mm_struct *mm) __mmap_lock_trace_start_locking(mm, true); ret = down_write_killable(&mm->mmap_lock); + if (!ret) + mm_lock_seqcount_begin(mm); __mmap_lock_trace_acquire_returned(mm, true, ret == 0); return ret; } +/* + * Drop all currently-held per-VMA locks. + * This is called from the mmap_lock implementation directly before releasing + * a write-locked mmap_lock (or downgrading it to read-locked). + * This should normally NOT be called manually from other places. + * If you want to call this manually anyway, keep in mind that this will release + * *all* VMA write locks, including ones from further up the stack. + */ +static inline void vma_end_write_all(struct mm_struct *mm) +{ + mmap_assert_write_locked(mm); + /* + * Nobody can concurrently modify mm->mm_lock_seq due to exclusive + * mmap_lock being held. + */ + mm_lock_seqcount_end(mm); +} + static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true); diff --git a/kernel/fork.c b/kernel/fork.c index 89ceb4a68af2..55c4088543dc 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -448,7 +448,7 @@ static bool vma_lock_alloc(struct vm_area_struct *vma) return false; init_rwsem(&vma->vm_lock->lock); - vma->vm_lock_seq = -1; + vma->vm_lock_seq = UINT_MAX; return true; } @@ -1261,9 +1261,6 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); -#ifdef CONFIG_PER_VMA_LOCK - mm->mm_lock_seq = 0; -#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index 24c809379274..6af3ad675930 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -40,7 +40,7 @@ struct mm_struct init_mm = { .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), #ifdef CONFIG_PER_VMA_LOCK - .mm_lock_seq = 0, + .mm_lock_seq = SEQCNT_ZERO(init_mm.mm_lock_seq), #endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, diff --git a/tools/testing/vma/vma.c b/tools/testing/vma/vma.c index c53f220eb6cc..bcdf831dfe3e 100644 --- a/tools/testing/vma/vma.c +++ b/tools/testing/vma/vma.c @@ -87,7 +87,7 @@ static struct vm_area_struct *alloc_and_link_vma(struct mm_struct *mm, * begun. Linking to the tree will have caused this to be incremented, * which means we will get a false positive otherwise. */ - vma->vm_lock_seq = -1; + vma->vm_lock_seq = UINT_MAX; return vma; } @@ -212,7 +212,7 @@ static bool vma_write_started(struct vm_area_struct *vma) int seq = vma->vm_lock_seq; /* We reset after each check. */ - vma->vm_lock_seq = -1; + vma->vm_lock_seq = UINT_MAX; /* The vma_start_write() stub simply increments this value. */ return seq > -1; diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index c5b9da034511..4007ec580f85 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -231,7 +231,7 @@ struct vm_area_struct { * counter reuse can only lead to occasional unnecessary use of the * slowpath. */ - int vm_lock_seq; + unsigned int vm_lock_seq; struct vma_lock *vm_lock; #endif @@ -406,7 +406,7 @@ static inline bool vma_lock_alloc(struct vm_area_struct *vma) return false; init_rwsem(&vma->vm_lock->lock); - vma->vm_lock_seq = -1; + vma->vm_lock_seq = UINT_MAX; return true; }