From patchwork Mon Jan 22 07:13:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13524878 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E1FBC47DDC for ; Mon, 22 Jan 2024 07:13:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E93E16B0071; Mon, 22 Jan 2024 02:13:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E44248D0006; Mon, 22 Jan 2024 02:13:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CBD398D0001; Mon, 22 Jan 2024 02:13:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B72F06B0071 for ; Mon, 22 Jan 2024 02:13:34 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 94D44C08AA for ; Mon, 22 Jan 2024 07:13:33 +0000 (UTC) X-FDA: 81706081506.04.7F05923 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf11.hostedemail.com (Postfix) with ESMTP id DAEF24000F for ; Mon, 22 Jan 2024 07:13:31 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VQoQ2Xpn; spf=pass (imf11.hostedemail.com: domain of 3mxWuZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3mxWuZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705907611; a=rsa-sha256; cv=none; b=T5m8wJlbBJhtGxJHQW/xREHov2pyI3XtZ8SJ2Wxmbp4jPEVIGFqH6dp36LnwcUNDeH0hVE EEh7yJjJAs8GvzAQZVp1TZr1gTurr1ao0HLziZ6PyFXNDzN/NObdzpaPdC89g6mpwNjjvy IZQpX3qPDvj2LuvnQP/mVicQl31ZKes= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VQoQ2Xpn; spf=pass (imf11.hostedemail.com: domain of 3mxWuZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3mxWuZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705907611; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OLp6U6EsMSw583KQsA7U2jc3UsvPAv4xgtxefIxudO0=; b=jnVksyGD1H6yO/ncPSGZUKrlV8BoT5X2Q/W0XWs7pZv49R0NPoyqWXa0OUVy8djE/HNzVT 7praoifKvLLq0bRNVPa46cJIh26zsGRPvejGPakeAk6h27gaLlg59g5XHPqZkC+Hlk9R56 WP6iCbh5j4i3/4yOFeWu8AlM4mqnd64= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5ffee6fcdc1so7648777b3.2 for ; Sun, 21 Jan 2024 23:13:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1705907611; x=1706512411; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OLp6U6EsMSw583KQsA7U2jc3UsvPAv4xgtxefIxudO0=; b=VQoQ2XpnZRSpuqclyVNqsYqWDf6z30eA18iyVMcecDXVn1yGhVo/f86QRLImEiAhzr 1JhKDe02dO9Ou5PFD9dXlewi3d6SCGhG+V/1kUbayWYAeIZncAYX3YvsLRdB0XL7sHSf 9LTqSqAPDj9qkDtbFizTymzxN81K15vgAzV+Aa/4STFS6QsZ/hE2s+z01JpgIaw/0yKZ UnB+53KV1Fn9RSHbafqnMP7k4Myvn6fHz8EnxFVHwwRGAzDGTmFMyat5hbz2ttxOPwNI WDJQNRF+D+pNBs/8KFLOqNS765d+SGpjsKf7V0Y+i6VX4AwM7lGf0UG7oYhCW7I23e15 aWRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705907611; x=1706512411; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OLp6U6EsMSw583KQsA7U2jc3UsvPAv4xgtxefIxudO0=; b=Wf2rluJdq/INw63y5TFgYiAQ+zsc+Q6j3uDwH1+XaEaaPT9lF3H13OhixSxwXfqoZA X1XFY1cI1kSj+GdorCpmX2EKbEkKv9sUbIM1GPR8lJvsvjwU9vxVUNq7LxmirMr7pQrL QIbS+ZUTt70KRiEWn7QrFx5OD8Uy1TjH5qemybrDrdLlLJAGHSXzD71bsOOzOf81kU9c qbw/nTqONXfq7T3hy6Ukhc+asvfoEHRtmbYtmv3sqp9n2Nwx6q5XebrzZfkTW30YrQTv 6rIox6AmbP6BHMHCXDxh5PmCs4Ivbs1EaUQhdGxVwJd5pIYyw+wc4v7UtJKUK2EVDv3H +wxQ== X-Gm-Message-State: AOJu0YyQnN68Xl8VyW76kQFAVmhi0Hh9jvNQfhARVQ7tp6T/P6mMj+fS xcGNUxCrOWEJlAQfGJzyY6JBuW8oZPZvgPf9EE9BKsJvpdpGHIFIrMQZmdkLWAcccbUVbPiSa86 hRg== X-Google-Smtp-Source: AGHT+IFDFx+rU5MMV2joyEArNdxSbn9+Uqv0sdgUNYpKz6Fm9lpqOqNw6v9H//7PI5LcIGO1EwFhJ6qHlkY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:4979:1d79:d572:5708]) (user=surenb job=sendgmr) by 2002:a81:a096:0:b0:5e6:27ee:67fb with SMTP id x144-20020a81a096000000b005e627ee67fbmr1310854ywg.4.1705907611085; Sun, 21 Jan 2024 23:13:31 -0800 (PST) Date: Sun, 21 Jan 2024 23:13:23 -0800 In-Reply-To: <20240122071324.2099712-1-surenb@google.com> Mime-Version: 1.0 References: <20240122071324.2099712-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240122071324.2099712-2-surenb@google.com> Subject: [PATCH 2/3] mm: add mm_struct sequence number to detect write locks From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, dchinner@redhat.com, casey@schaufler-ca.com, ben.wolsieffer@hefring.com, paulmck@kernel.org, david@redhat.com, avagin@google.com, usama.anjum@collabora.com, peterx@redhat.com, hughd@google.com, ryan.roberts@arm.com, wangkefeng.wang@huawei.com, Liam.Howlett@Oracle.com, yuzhao@google.com, axelrasmussen@google.com, lstoakes@gmail.com, talumbau@google.com, willy@infradead.org, vbabka@suse.cz, mgorman@techsingularity.net, jhubbard@nvidia.com, vishal.moola@gmail.com, mathieu.desnoyers@efficios.com, dhowells@redhat.com, jgg@ziepe.ca, sidhartha.kumar@oracle.com, andriy.shevchenko@linux.intel.com, yangxingui@huawei.com, keescook@chromium.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: DAEF24000F X-Stat-Signature: 9j8d93ik3fpjqfifpttoqnqqz3fnkzka X-Rspam-User: X-HE-Tag: 1705907611-158232 X-HE-Meta: U2FsdGVkX1+baU6+ZGPhc/sLlxJJiAolEN5m9X4DAGzxIPTp7+shdJuODa8UvFeKtNwwVUrMA8P0QfjWOvaGi4p/AaDzZ3OKDT60ISBhs2iaoGUg4slc8icteFT1zocGZtKNWNs2e3k7fO7hjhNNF8qiVrKF/2EqzoNnWyNlYF0venzCSwz5u2Y0xFxlfM6rn+Bjfu1z10Bj96e+9opHHovPyjlxzd9v3e8o2xbHRaZp6fy8h/M0Bp4zTUS+uVkVaZhwdgT4QAjhWYKR5mej7S7mfL/5H4EUgvj1EUE7oJHnqiJrY5hDQNPuiF6ySZc2BOvMPrvyntpLo6IvtDFnC5IzSNuSA0umL+oG33ulTSkYrLmdsW7kXVFUgHkK3l2aurn9h9upu8mCDpwHQpjlqAM4CZ3JL4hd+zUXxhz1s1dveX8+KuovBP0ryPgneUuGFkFJTzZl1jiQIm2zeXqbCqS/xHKOtQZ8uXdM8ymvOOV8vWg0mvYIAMPrMkyrDoT/v9JAoxJ2AWN2NmojJC7520dfkZxzLSGpJftXL5irvgzIJcLkB8d0AefMk1zKsyBdXpGtatuwwcuBdsnx0GRYB3i4uapUaP3Ey7AMeRZiVG8fQjk4MUgci5lrHiPEkyRuc2+kUdR31GGJkt4+smHu/XUiughMeRCic3/iD/A0vedHSNdVHy8zIYVQZ1/2MoE4zMkn5kCOuNUWpy/r2mCK/bFcwjPLtleHHjqTqAQ88PW70EFJb9yCaP3q6XkoKOjxczG5jQPW1vA2Q5TTEONdBT7+ZED56SCxBpUINCKZpFYgp7HOdnBhLYpeIycOpXmGmirlFDaSSTwizi1XDftCzr6azl4odqY8QJmLpBC9uaz31MWy+I2EDzpkOThkUryO1JW/Pt30MYfMijYMU5nXJFkIb7IkDCWHzO692RnzxGxGA9gqWEoTlZvZT8wDaKv0k1/09Z1bQpKhgz3hhOJ qsIBvdPM bbeK1wcQKeaqJGaUzE0g9xiOJQ/CZlfFjZYS/dhHI5M0m2UB4SorsPYKs7Wm1/f8IuNHezDzlUS5SkJrwUru17lfmdtx4/aSLxMwBljF+mCkDqHley6BC4qz/SQzWzntJDXtlr8NmuulQULdnJIy4c2agu5VOsD/qf8X033clyRSkPfzzmdB2uLXac9uKXvfzKquhDNJ57iMlHmOGTJ1afALUlTJPFArDtpWKxxSlFV9TCqsKDoq14t7DAnSXOdqImupIwLpdmKQu3NU12qjxNLv7YF1IblzqaXlvxoCqIGn9NO8aXUkjp9uP5DacofxBPaVbIji0UHLFBLMW4H1Qsm4cNOZR9G7VDBdu6SwqZDH7Na3KJmfgJqqbev+uMeWisb4MAT0qvbT5lc8aJKNWMr4NXx9PtU6lMUNDw8hslvKhDOFQNE9G1QRhdmox2OIgc2iEFO2BQqVvf6GscrmJn6ioxBRD27s0ZP/CNGp2VjjSc7AB1/o1UgFiBXwXLDFc9qE3dPaCXksYWrmCoBannsJ/teYXN46LncIBNoME8YZKPAW9Ef7dgueOobL/lB69v26uv1qWHoXd9yN5IPuhInIC6JTu5utq4r7i2xzt3iGbUeC9SnIq12lgbYjdbKLMCuKOTSmzQUpJi4SPAjfLGyFID/A0ZoR8ofioAelbvaxFoj/iLqsPd2QnGHGW9BdZu/BwspWgmNvy3zrDXjCYSw8cYoke/2jquR/h X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Provide a way for lockless mm_struct users to detect whether mm might have been changed since some specific point in time. The API provided allows the user to record a counter when it starts using the mm and later use that counter to check if anyone write-locked mmap_lock since the counter was recorded. Recording the counter value should be done while holding mmap_lock at least for reading to prevent the counter from concurrent changes. Every time mmap_lock is write-locked mm_struct updates its mm_wr_seq counter so that checks against counters recorded before that would fail, indicating a possibility of mm being modified. Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 2 ++ include/linux/mmap_lock.h | 22 ++++++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index bbe1223cd992..e749f7f09314 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -846,6 +846,8 @@ struct mm_struct { */ int mm_lock_seq; #endif + /* Counter incremented each time mm gets write-locked */ + unsigned long mm_wr_seq; unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 8d38dcb6d044..0197079cb6fe 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -106,6 +106,8 @@ static inline void mmap_write_lock(struct mm_struct *mm) { __mmap_lock_trace_start_locking(mm, true); down_write(&mm->mmap_lock); + /* Pairs with ACQUIRE semantics in mmap_write_seq_read */ + smp_store_release(&mm->mm_wr_seq, mm->mm_wr_seq + 1); __mmap_lock_trace_acquire_returned(mm, true, true); } @@ -113,6 +115,8 @@ static inline void mmap_write_lock_nested(struct mm_struct *mm, int subclass) { __mmap_lock_trace_start_locking(mm, true); down_write_nested(&mm->mmap_lock, subclass); + /* Pairs with ACQUIRE semantics in mmap_write_seq_read */ + smp_store_release(&mm->mm_wr_seq, mm->mm_wr_seq + 1); __mmap_lock_trace_acquire_returned(mm, true, true); } @@ -122,6 +126,10 @@ static inline int mmap_write_lock_killable(struct mm_struct *mm) __mmap_lock_trace_start_locking(mm, true); ret = down_write_killable(&mm->mmap_lock); + if (!ret) { + /* Pairs with ACQUIRE semantics in mmap_write_seq_read */ + smp_store_release(&mm->mm_wr_seq, mm->mm_wr_seq + 1); + } __mmap_lock_trace_acquire_returned(mm, true, ret == 0); return ret; } @@ -140,6 +148,20 @@ static inline void mmap_write_downgrade(struct mm_struct *mm) downgrade_write(&mm->mmap_lock); } +static inline unsigned long mmap_write_seq_read(struct mm_struct *mm) +{ + /* Pairs with RELEASE semantics in mmap_write_lock */ + return smp_load_acquire(&mm->mm_wr_seq); +} + +static inline void mmap_write_seq_record(struct mm_struct *mm, + unsigned long *mm_wr_seq) +{ + mmap_assert_locked(mm); + /* Nobody can concurrently modify since we hold the mmap_lock */ + *mm_wr_seq = mm->mm_wr_seq; +} + static inline void mmap_read_lock(struct mm_struct *mm) { __mmap_lock_trace_start_locking(mm, false);