From patchwork Mon Oct 28 01:08:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13852774 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF147D13591 for ; Mon, 28 Oct 2024 01:09:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E6C86B0092; Sun, 27 Oct 2024 21:09:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4961A6B0095; Sun, 27 Oct 2024 21:09:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C31E6B0093; Sun, 27 Oct 2024 21:09:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 07ED36B008C for ; Sun, 27 Oct 2024 21:09:16 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9955BC1C32 for ; Mon, 28 Oct 2024 01:08:51 +0000 (UTC) X-FDA: 82721226462.19.B3209E3 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf29.hostedemail.com (Postfix) with ESMTP id 43745120011 for ; Mon, 28 Oct 2024 01:08:43 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=R3D7Ov04; spf=pass (imf29.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730077674; a=rsa-sha256; cv=none; b=0mVjVlqSJJWtnHE8SKQ2677DAS5GtfdHcXDGk/iyKXmO2YNCw6bvg+DzdF/qeHVJOGkGje qxOasB7a514rqlsHbhKkB00lErhXFmnYSxED4MRGb8vS7/SXLVrqZ03tAZHVJ0qH4hpxi4 BRxh/zEL7M3LwmWro+qRJLF2H8C+QXo= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=R3D7Ov04; spf=pass (imf29.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730077674; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=f+rhBCaUIOowBr7L6zyysIJf2njGDW7nlOA5qUUDwWE=; b=SZG74M+YiPQoW4Eo+1hbtjl7VxoLuFNQzQrPG0QhMUmOrmP+0dVS2aoJ+ijooArDBqWJnC dY/wz4afi0v+3OfIbcz36Pyjm55oxjEPTp7aCV+Qwest1z3GTjwVvVlhqFbWUITMiU73y6 iOea6iXrxycVjJb3y1QwTftHc+CHvFA= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id EF9635C4850; Mon, 28 Oct 2024 01:08:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9E76EC4CEE5; Mon, 28 Oct 2024 01:09:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730077752; bh=hDtDfS8JqT/fLQMiv3m8WVup/RU6yhW70oJ6haQb60E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=R3D7Ov04DpaTyYJl+4SfPWoXLMI5EldjfTRT6VKHmSxekQRYCT+7e+vroidwAIM/y c9UyoWB0yeHAQH9wTW7L78nHqBKgCt8ub4xAaiUUNjh1wTF2ra5CO9zN6Y1+ZM9WXu XyH8q6zDh3Q2i8O3Z0GTJGneeTd+hwvD+2koWe7Tt+V5W6IjIOahBvgr/w8QyRKLZX zh3XZeFvlXwBaoY/ZbL/16DKFuEy1Pqn1S42YcpEXwRg9wU7Db45SReuz3aoSXoDcX pYqPLyrc4kC79COC4fnmbguofXh/ZDGrmKdLWt0rsIyDhYk4VXtGueWWod/9Y+SFIn 07MKzAQ58jXyA== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, peterz@infradead.org Cc: oleg@redhat.com, rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, mjguzik@gmail.com, brauner@kernel.org, jannh@google.com, mhocko@kernel.org, vbabka@suse.cz, shakeel.butt@linux.dev, hannes@cmpxchg.org, Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, david@redhat.com, arnd@arndb.de, richard.weiyang@gmail.com, zhangpeng.00@bytedance.com, linmiaohe@huawei.com, viro@zeniv.linux.org.uk, hca@linux.ibm.com, Andrii Nakryiko Subject: [PATCH v4 tip/perf/core 2/4] mm: Introduce mmap_lock_speculation_{begin|end} Date: Sun, 27 Oct 2024 18:08:16 -0700 Message-ID: <20241028010818.2487581-3-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241028010818.2487581-1-andrii@kernel.org> References: <20241028010818.2487581-1-andrii@kernel.org> MIME-Version: 1.0 X-Stat-Signature: 7t3ww8rou6fmbtuaxk3hgfba3kztmwy7 X-Rspamd-Queue-Id: 43745120011 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1730077723-126534 X-HE-Meta: U2FsdGVkX18jmmJ5UPRYh6hwx+O/KqM6ny2u5oiM95MbzC0Nc+ER/03TK9nCZOfmec7LOQGRcy7riR7CiIcqVQTaMPg8zvu2wIPpJR7jXVgVk/y/5fkGyGeXtGAQUR9j8w0Nr1UhjuOsvVE/rIHIaqSuAuNiBz4Uw5YSJr/LW0DOhM3077l4BtSIy0cMD3TEU0AI32TxmB9BhtMbQmcnGSal1qdy+QEQGwvUE2UNoWXZdBovrk7wCp+L0+0v+IyHAp8QXaXBzc1uOQRkiy7YWyGRygchxuy1uzHKkQecnUgje0tiLyLJHEU5DpzU1nQTzcw/kbYoetR/qkcbW3uQwDLuRxvPUDMy+iFCqo1g+s56NuTJ6kgGgIcoNFMcABA0ZDsezEqE0DkjrsW5hQt4i0a7KDohqFbralhVvxZk7+mqv9Hf1dM4wKMRsLUbhaQexNvRKQncw2qtO8NVdn76ZiDneMG3nRSfAgAY3iNfM2/tBRId6d18PCqEm6T+yEl9j8YdHUaIKm79MA4UmoMSgtYvzgRJ3LF5Qd1tg2Lp+xj4h1oo3MF95EODSBmOtx0gGYntWbdWfkqm+yYJwBjmrYrHPVr+u6l86O/y+d7YfGeM072SWQflnXF0xsOI5tNl5E7Wee97W4P+DVBnIXfc5bsIUi9r6desn48xmveJ/wUzNxfNg6SDE0Xs7seIJ1PfKw5LCAiw2yiSiOASkTSEzachzIbVFoPYnvwwchl6knUmSjYggLMsFFKt01dvLECOywGtSqOn5heSIYbbx2rjhpcCkAOknSy5Bqu9p/GRgD7eI+oLnFCjqTyR5WD4E5oaW7TMmcq2KxoHibQMhm1kwVbddBJPQycfwc0IYGvRzYn4JCls3MTmQ18jJlmaVR18DmLp+9TSVXwRApxFcvLfxm4QABnadXyRQ5Twbaehuqyc5HQ73/Ddn/29dQxVme5+y3XUAekSw9IBYfhELfw wcKEkft4 UR7AWHDDiS36TuwoKjLf9VmIhYO2GMocOaycwUVgtYkIU6AnF3IlsS5Nu1pPhpjVbTd9AeQzBtiuKe7RAYe/gbID3eRcDoqJyMc4ecyQRRJ7id9aZnsGa9lMgopK+a8nqG0VpyK17Us0gSmXZ9MnYZZ3p26cj07i/h6DkBMza7t+X633u77Hx7FgQCMIazfUsKHInJ0UtbqQWAhMqEuu/LaW1J/2O6p1aUIS/2puOAVjmTIWBroAWDUiMXsKIMkBpgfl56KvVGreBDrt21EdSr5GxQPLbV6UYj+YeRmppJyHrER1WmpAemwv2ubAF4yMhgPRFepvwj5wjFAZeae+/au0mSxA1VaPozxazr+lKWjsC1JE2E2ky4gcT/7rCjRV5Vbax3+RYT6MtSn7L4ifDtNF61Skk203bC68fzQE31ZFwT2sn+mSMdcTu5A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Suren Baghdasaryan Add helper functions to speculatively perform operations without read-locking mmap_lock, expecting that mmap_lock will not be write-locked and mm is not modified from under us. Suggested-by: Peter Zijlstra Signed-off-by: Suren Baghdasaryan Signed-off-by: Andrii Nakryiko Acked-by: Vlastimil Babka --- include/linux/mmap_lock.h | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 6b3272686860..58dde2e35f7e 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -71,6 +71,7 @@ static inline void mmap_assert_write_locked(const struct mm_struct *mm) } #ifdef CONFIG_PER_VMA_LOCK + static inline void mm_lock_seqcount_init(struct mm_struct *mm) { seqcount_init(&mm->mm_lock_seq); @@ -86,11 +87,35 @@ static inline void mm_lock_seqcount_end(struct mm_struct *mm) do_raw_write_seqcount_end(&mm->mm_lock_seq); } -#else +static inline bool mmap_lock_speculation_begin(struct mm_struct *mm, unsigned int *seq) +{ + *seq = raw_read_seqcount(&mm->mm_lock_seq); + /* Allow speculation if mmap_lock is not write-locked */ + return (*seq & 1) == 0; +} + +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, unsigned int seq) +{ + return !do_read_seqcount_retry(&mm->mm_lock_seq, seq); +} + +#else /* CONFIG_PER_VMA_LOCK */ + static inline void mm_lock_seqcount_init(struct mm_struct *mm) {} static inline void mm_lock_seqcount_begin(struct mm_struct *mm) {} static inline void mm_lock_seqcount_end(struct mm_struct *mm) {} -#endif + +static inline bool mmap_lock_speculation_begin(struct mm_struct *mm, unsigned int *seq) +{ + return false; +} + +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, unsigned int seq) +{ + return false; +} + +#endif /* CONFIG_PER_VMA_LOCK */ static inline void mmap_init_lock(struct mm_struct *mm) {