From patchwork Sun Nov 17 08:09:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13877796 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02220D68BF2 for ; Sun, 17 Nov 2024 08:09:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 66D559C0018; Sun, 17 Nov 2024 03:09:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 61C596B00C3; Sun, 17 Nov 2024 03:09:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 471939C0018; Sun, 17 Nov 2024 03:09:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 272626B00C1 for ; Sun, 17 Nov 2024 03:09:42 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9ADB21A0165 for ; Sun, 17 Nov 2024 08:09:41 +0000 (UTC) X-FDA: 82794861828.15.51E27EA Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf18.hostedemail.com (Postfix) with ESMTP id A73501C0003 for ; Sun, 17 Nov 2024 08:09:18 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=L5LGEs2o; spf=pass (imf18.hostedemail.com: domain of 3wqQ5ZwYKCDMhjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3wqQ5ZwYKCDMhjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731830921; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CZkyrFrgAwaR5MxHWX7P6WpGLYaVbagX6YvpJwNn5k0=; b=aLDXQ5QzXrcQAXLFqGM2QAGZkEMu2sYxZyq9kcskzcYJ7K+n2keJwGgGDU77YvQI+qlqhn YPBv8Y1nfEqmtcYXLyTI44Z8wizOwOxQW3JLk3QJoNChjQkpM9ZSF2G/DgYMgus4oklXWn O7IdvuwZ9cwu9P7cYTAY4FsQ1L6RRWw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731830921; a=rsa-sha256; cv=none; b=x9oM61Zz3rG6db4bchJnP5eeWjP3fJIQc7+ikDMBR3cQkLk8Uylk+wdQcPFb4mYsuT5iFZ va4vm79+UajqjkxRCCsJv8Uy1Or1bc1HBJm0/q1ERp6jj96jV4lgmRwnWauYwqD5fuhCCn vJMigHEdJKeqlGIbb2VVpUE+HE3y358= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=L5LGEs2o; spf=pass (imf18.hostedemail.com: domain of 3wqQ5ZwYKCDMhjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3wqQ5ZwYKCDMhjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e3886f4cee2so770679276.0 for ; Sun, 17 Nov 2024 00:09:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731830979; x=1732435779; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CZkyrFrgAwaR5MxHWX7P6WpGLYaVbagX6YvpJwNn5k0=; b=L5LGEs2oIHuhsUbE9c5/S4F0fY5zICJvhO9gUVR5tKC4UOzP6ge0yHsgU8b5zSHPyu W990Fsg9SiAjPIs8Exa3LYVBKyyTTvubxLNSgvNCUuP0yLDRU983zKnKN06xFxZ5ZQix YiDpa7UkdqmqW950yPU//5HM5xkqT6FcqrnK3I2gLwCALLfHF0cZEARgCcZxxRFqUllS sCYN6x2EGQC1cYJ0NEMeJRkOPFY6qF1tMIW5l0ZE/VHy+NGjQvigaEqxogcucAsOkQhL AxhxlJCm56eGg6VeJJv26XuGwrN6Qq9jWOOQ1/wfg4CE8q0kXyIpFhDfRyuDnhIzbS/2 l5kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731830979; x=1732435779; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CZkyrFrgAwaR5MxHWX7P6WpGLYaVbagX6YvpJwNn5k0=; b=OTW57LF1FHLRRB+dFN9lW8gwowYIUs2aK45oSLijIXexmzOpxkOpCC8ZmYdUPoBkna ZvT76REeMl5COAEgmmVvGFCgZSiot8MPw4hBUTgEeIZP84FyCYl9PAKVlIdMcixYrPY2 yOhm2HA/8XnnyisJheuO7tIR0BJb7CvELQi+UHxv70MsLfTmUcPl4ej4TCiNCt+pyE9W LI9gb3IBHuy90RQvtK+w6tMMgCv7U47sYbksGL+oR9MDSwZzjWvtbhvT3s64GGnSu8iR JT78PObC/LL0J5ePhzHQS6HObNy5rQ4xUt12dj5f9LVXk+spCNHDR0ihe7jjgoHX7ent 8ZJA== X-Forwarded-Encrypted: i=1; AJvYcCW8TlCCezFVtG/eFkqaU5ncIBbpaysSQWoA1kMgARdA9F63nCMNTo/0zRTG9P1k2OAyj0RWK2zKKQ==@kvack.org X-Gm-Message-State: AOJu0Yz+X98rRfKzh+B3FEXjAxAY6fyp3dH4/1EunO3rrZPHxSoTWkPK U5lM1DFRGMYRiYwUkAMqzJIOI8mn25Er7CWkRTIH1WeQYHtNJAYtEjxcxBKwen5Jcx9ByXdzrmh /cA== X-Google-Smtp-Source: AGHT+IH7Nd8pXV7CscbKL6Oe02lMMMIInGW3q+Det1fKSUjvn9CktAIAKXjQIOt60hBNRgC9PV8ouBWRm4Y= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:bafc:6633:f766:6415]) (user=surenb job=sendgmr) by 2002:a25:9d85:0:b0:e2e:317a:d599 with SMTP id 3f1490d57ef6-e3824ac0065mr340301276.2.1731830978836; Sun, 17 Nov 2024 00:09:38 -0800 (PST) Date: Sun, 17 Nov 2024 00:09:27 -0800 In-Reply-To: <20241117080931.600731-1-surenb@google.com> Mime-Version: 1.0 References: <20241117080931.600731-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241117080931.600731-2-surenb@google.com> Subject: [PATCH v3 1/5] mm: introduce vma_start_read_locked{_nested} helpers From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: tnmr4q68r9du8wgcu1fnmh6zu7tbrsgh X-Rspam-User: X-Rspamd-Queue-Id: A73501C0003 X-Rspamd-Server: rspam02 X-HE-Tag: 1731830958-564597 X-HE-Meta: U2FsdGVkX1/uWt0dz5L5ApUwurm/xhjv0tSd8BTXJsjuI2iKal1SRjyOTFy6BCuelQjsaq78XuHP65iKDN0R0B7f3V5YCQX4sXgbCrV4Z9U141ang0j+5GtmK7ELsZ5hlxvqc/F5Gc6jhClFkj1bA9sWR2gXDxeDyp9kMX/1r0Wog0VAQHx04Q+6Rv/7qMFPv6TUB+ATXHwNJN3MR+cqrYRm4db8h5XLlOwebM5GcQ5bgACHTOqtJI0W5FqR6ryIiOnu5nI3n3mG0s9z0g2rtAN8lbQkQQRJuABe8FlYpZOb+ijI1vWWln2Yzw/vUGDJicU+B5S4lnKPCnX7fcIowk42690Tnvv69vNQBGBxX2ge8dR2Li9iexRtaebE1Il6N644ZpN3KFmIuqf02RpFJYI53maFhjCk9belFUYdnuduC86qzbGizuiBjJQDfMf4+gkAu1OMvUn6s3q1jVOcZ2qbESj8WpvZXp3jeJRMoIbVzbR5D3uRUBh882SJ1xcAQb+CbhBeX0IAdLxQlhwBtcdj80g4Fo/wG64P7aOodi+ubfxyWRmO/2YccnXMND2f2ZjHdWPaFIWQCBSHqTtbTWmfGLijV18Y560SRY5zycncS9HmxZxrFZJ1Nky+TlU+rAhj7txgmrYvWIvXmDz3kEztABykGR9VsRbJFNxSovJaG337by/UZcjvneN+Gaf1ounCOs+lh+AhKo31rxgl3FoTAnpDyaaaS8G/Ow/8fiOW74GGUDhaSnJ/KfYbn9W/Olxk8zV5w3mQEeSWryJIc3KlANHRdk/AJuUAodJX0v/Cm1eTDNRKNaKdx0HopatJUYgyeqCkVWAlTNs0kHV7b1WNMtBHFiTX2D4YuU6ka0CnKfoIt9ul4bPgqr6y9J+mNlGjDtA+eYEreVRGr/W+oOiGRy/0FI2+rLbNRRmC8d7GdYuDxoMVj72bkuKO3Dr7tW+NFbPNO5cu5bSH65J f0gCIL69 3zUlqmu+A+6ESgHAOQ5tsO6fCGcz1DhBmVVGk0v77NR0eo1mkiYDVxxRAxdpwrRQCJTYXcKtFXwwdLiARSCgdTU5Bb7smqu3vUFgtNIuGU0YbI1L9/Nd5kJDySvaNja7cDKrREc/H/ttUbkJLfgG0hWO+jrdujNiv6/Qd0GYoc37kzS5LCXdlSU1782nGLI+DZMXb1HSmM5SzektmyfGTY/8F9zg9JcF1Tmw8JYZfXZNw2Gk4zIVNkizVt4kgUohpGFQK/3L9YvQN+KprB1pw01vbmxak+gAHgsVWy/x75y9o3HMJthtr7tYecYaTCS0nLYKP0bJgKAPQi4Q1/zVC1MCaPhq6APMzQGpn3w1SZpblAOhnF2CNg/MRu4PE1X7Cg9MeBDCp1tXweuPlmYmZX9OvnfXMSUpH2NqZjtEjUpwv3n6neWNrWWVip2vrpCm6ohDdE7XqNGPdA60uQtl9Lm+TjsbxtoJujcCQ53lj/vk6Zb+E1ED1c3ZBbFBrkubkI8G4DlCqtgtITSXNTU3/Pld45lvGifld4TWV1OkzT+q2Bn8vDpDOFy461qOyk9NuKV4OGtrdqBm6+oGOPe+k4VHJqA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce helper functions which can be used to read-lock a VMA when holding mmap_lock for read. Replace direct accesses to vma->vm_lock with these new helpers. Signed-off-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes Reviewed-by: Davidlohr Bueso --- include/linux/mm.h | 24 ++++++++++++++++++++++++ mm/userfaultfd.c | 22 +++++----------------- 2 files changed, 29 insertions(+), 17 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fecd47239fa9..1ba2e480ae63 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -722,6 +722,30 @@ static inline bool vma_start_read(struct vm_area_struct *vma) return true; } +/* + * Use only while holding mmap read lock which guarantees that locking will not + * fail (nobody can concurrently write-lock the vma). vma_start_read() should + * not be used in such cases because it might fail due to mm_lock_seq overflow. + * This functionality is used to obtain vma read lock and drop the mmap read lock. + */ +static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) +{ + mmap_assert_locked(vma->vm_mm); + down_read_nested(&vma->vm_lock->lock, subclass); +} + +/* + * Use only while holding mmap read lock which guarantees that locking will not + * fail (nobody can concurrently write-lock the vma). vma_start_read() should + * not be used in such cases because it might fail due to mm_lock_seq overflow. + * This functionality is used to obtain vma read lock and drop the mmap read lock. + */ +static inline void vma_start_read_locked(struct vm_area_struct *vma) +{ + mmap_assert_locked(vma->vm_mm); + down_read(&vma->vm_lock->lock); +} + static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 60a0be33766f..87db4b32b82a 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -84,16 +84,8 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm, mmap_read_lock(mm); vma = find_vma_and_prepare_anon(mm, address); - if (!IS_ERR(vma)) { - /* - * We cannot use vma_start_read() as it may fail due to - * false locked (see comment in vma_start_read()). We - * can avoid that by directly locking vm_lock under - * mmap_lock, which guarantees that nobody can lock the - * vma for write (vma_start_write()) under us. - */ - down_read(&vma->vm_lock->lock); - } + if (!IS_ERR(vma)) + vma_start_read_locked(vma); mmap_read_unlock(mm); return vma; @@ -1476,14 +1468,10 @@ static int uffd_move_lock(struct mm_struct *mm, mmap_read_lock(mm); err = find_vmas_mm_locked(mm, dst_start, src_start, dst_vmap, src_vmap); if (!err) { - /* - * See comment in uffd_lock_vma() as to why not using - * vma_start_read() here. - */ - down_read(&(*dst_vmap)->vm_lock->lock); + vma_start_read_locked(*dst_vmap); if (*dst_vmap != *src_vmap) - down_read_nested(&(*src_vmap)->vm_lock->lock, - SINGLE_DEPTH_NESTING); + vma_start_read_locked_nested(*src_vmap, + SINGLE_DEPTH_NESTING); } mmap_read_unlock(mm); return err; From patchwork Sun Nov 17 08:09:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13877797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50E6BD68BDD for ; Sun, 17 Nov 2024 08:09:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CFE619C001A; Sun, 17 Nov 2024 03:09:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CADE46B00C3; Sun, 17 Nov 2024 03:09:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B28CB9C001A; Sun, 17 Nov 2024 03:09:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8D27A6B00C1 for ; Sun, 17 Nov 2024 03:09:45 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1D1F21A016E for ; Sun, 17 Nov 2024 08:09:45 +0000 (UTC) X-FDA: 82794862668.14.5E7DFCF Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf07.hostedemail.com (Postfix) with ESMTP id 4368740010 for ; Sun, 17 Nov 2024 08:08:36 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MXhbcmJw; spf=pass (imf07.hostedemail.com: domain of 3xqQ5ZwYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3xqQ5ZwYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731830784; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sPYYZoVCqtknZIayzm8p1ItIUlzdQ0oYPahBTNB5G3w=; b=oUppGrR+vN7B4WGl3n9ANrxnCJGO5TWOEpqIKh0hoJBrCMNmK3wW9NiyQDriw5lCZrqy+C pM++Q/CAX0Z+m3PCdRGjOHgKHSRwzFpZi1iHrTqZGbn9woR0/AFNZi0NEEQHC3GfQDQcRJ ieZgkGSPOPJgHtFixCpdlUnHZdMbkcE= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MXhbcmJw; spf=pass (imf07.hostedemail.com: domain of 3xqQ5ZwYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3xqQ5ZwYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731830784; a=rsa-sha256; cv=none; b=raAj3cowjrV/gAIZ/Z/Pn/J2LaThQnSUuKwIVTTzfiqF46pdCQzh/iVtZx7O46rU3KvD5h BExqwC5c5rx3gTKfik4GBdW1KAgFRw409xUxRVl8GUWXZAF8z9z58I2qo89+TaAmzWemBz m0+c4sbck8XzL+grcyQUyG/lIsVdnB0= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6ee6c2b65b9so23988097b3.0 for ; Sun, 17 Nov 2024 00:09:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731830982; x=1732435782; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sPYYZoVCqtknZIayzm8p1ItIUlzdQ0oYPahBTNB5G3w=; b=MXhbcmJwYnW6QGjbTWseF+5MU/R9lbgXHPt73NfFOhr3jALs+KaubOmJ0V4pnaMxym utSgRpeC+TSE5ZoPLaZsjOt3og8fmZjFgAjC/TK49XpPLwABDMyTL8GQKxipIvwuKs87 7j9dDaIErd5doSBDWn6xU8XmwT+FxEvNqlnZjNIUTLt4N4v6cvstd+x4NdhfGlFh//EQ o1w0AxkAZ1RuSfnnLWC//90NFSRus3vGUqqXEcbPX1Pv2T/rdscpZ16DrHivLaZMMpJg /S0MKWDkECVzNyG0uWdqEANU/bIH/EZXTwrAW+xgeOjxRddTKw2ww/wqEiNr0iAkomt+ BxyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731830982; x=1732435782; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sPYYZoVCqtknZIayzm8p1ItIUlzdQ0oYPahBTNB5G3w=; b=Pi/02jRMYaYgOv6p722b1c2L6KoLYqvCqKkMg2PKJ7hn6qJfiqXuK2X3lsOP/cNtVq bw8HOybg8d2YnkME8bNhTRTM4rFz6m1vPCHZDEqqshrilRi+TdHro/LQ2tny2g6WJUst mPHxkZmSO2y4e2UgtFP6EFACMdndWwjwSyFtfieePiuqYZg3dsTB8IF8W43hEozJL9lP 2rFn92u4LoBwIxl6Xvb3J3WH34SMY2hclCZuqbHfB3Rv0p8WFl21bnyaDFR8ZxZHdBOw 7oHuba/jg9rlid02CaPoXECle8v3p9upH40HbJgnevsEvaz5livO3TczS0Q5gNLO3gze IwZA== X-Forwarded-Encrypted: i=1; AJvYcCUH77u03rW86nBxNOz0ZHb/0kgUlqX3zcP1tNLVDqjNVpi8/tclr/D4Bsn7NM103mZwoj+PyO1qKA==@kvack.org X-Gm-Message-State: AOJu0YwVNALHhXVZXWGKZs1w1Rk95rBbjXBn2X7Sm+uHLk+IQmko0Xzy /dLTigSrZ0zGhP9IJX7DftK5YF+aW0y1QqbRDqw0TVfF9gIHhzgm2kYo0JtVjJQ/KU9g8Fpe6cS XQA== X-Google-Smtp-Source: AGHT+IFmq2c1p1+TPAUk2jQ4A7Cpj3uaypUiZEZvBLe44BDcqceNyYvrL452knc8vwv/9E7sGLOMSXBRweU= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:bafc:6633:f766:6415]) (user=surenb job=sendgmr) by 2002:a0d:e505:0:b0:6e3:8562:ffa with SMTP id 00721157ae682-6ee55c729b9mr919347b3.5.1731830982118; Sun, 17 Nov 2024 00:09:42 -0800 (PST) Date: Sun, 17 Nov 2024 00:09:28 -0800 In-Reply-To: <20241117080931.600731-1-surenb@google.com> Mime-Version: 1.0 References: <20241117080931.600731-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241117080931.600731-3-surenb@google.com> Subject: [PATCH v3 2/5] mm: move per-vma lock into vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4368740010 X-Stat-Signature: 7tzxxcj16tbuir38smbqwthfhn46xz1z X-Rspam-User: X-HE-Tag: 1731830916-108516 X-HE-Meta: U2FsdGVkX1/ZExlz06QoJPmxxp6ATeVHmHHi9OHrYqj4J1JxUWNa7v4bI2s8+ucGrqqnqMfdPGBNR1uJzL3OQEm/w3LmKJBDziDO4GTLMAaI9tyXM1xDPa2mzfmhZp2RcNX9GhPu4CNvu8rcytWKw0QUGt3pyylbezE+N6Zn34YpoAmhcQ6NGIR6b8rAVnSRTFJ5UHdspQh0zWpDWDJ40/y4/pHAityEaX3B1t6RAjxCu/4o/7JxkjseGB7J1SI0f4F5r5THHhcRX6AKVDDzo7V+6Eb4XHEFkRzr5KyVXUypsAjPZ9bfpUA/sE3txXOTXn0SleNOR1dFV/A90UKJiEVfR3eI1sBznJaIsc6z3aJyx5Aw2eD2l7bxVv2DrX1yygkoEcAlt6zEcvwTnBUFLDDgdKShq7JS07Fi1vedw/iKTN671ViWX+UtQOoE5xJjj9rG7gaiYFWTAD/t63w/d7ODM7XQ+tG3OnZCIHR6FyUB6/buaKhly6RnB9aQS10XnmQqWco5TP6kvT0Sw1asFc7yVsfnb1sXmd/cfg1Qs7piVzcr4zbQhS+fFNRgl4l+gVS+1nZ71yCjSzlDtgtv9ZZ7XbswZu9cJc1855hSrBfqkL53Svkzw76H9tc3mpkeEMeNm1DOc6k0xSbRyh9FW/UuiRgsbmg9zPwK3jOb0FwuUvVzR6GZoCXFQmJ1RoG8KWM95NcLIdsJmwhWYzHBEaSdFnFfRLnu43aYOd+77SGiHQ522u8oySN2z/qcTnAKgxvS9VKcVQPrZb6SThBTv+HzEo56v9dKVQfaiKsfFc7km5nyPHuaNBbyerEZG7vlV3KhS+thfZbQTtp13K7rb5cDzq3cExC3LyvfX61y9GS748pbcWzzcSEouORghE3Jz2ewzAex2SKb9zL7E06H7oglDguRg+XFktSbA7yodssjoNBoS10mWtLkio5UKaRKfsxlOM14I9jn+CMzr4J nfCRI5Ez MckxYT+5EDV+j/m86qrwaeoMTfOlfkWw54lJ7iWixQkR5AKlNe/2xRL2NK7crbYdt36dewQAZbVnQK4AUSeZyKiziijGh+1wqarUsobWQ4el7JdTLGn213sJ+fAwWvn/rPeEexbYmodeOFjAScIX5BJiYND5d0ebpI35IZYBJSkAmG+Yw0mBJi68eFjLEc2PAgDSKWMQyQGRRr6IwFjHjHIzOewDdyvDymwHukCfV1ZRaQTfudFvNint8xZTSZYyVhdCqoiEkH36J31DkZ5RXdpRVWnkbpx2t/12aM1dF55T5CV0BPkiRv0CVU4Iuh35FSNHoDVaaNv3/VCsYUORlvmeWpjqUo7mAyabfMGsZgBPTkCuqAIBplOPI6t4GDeaY6tGZq9QD1yPu7TWP8Y19dYkp9zi4WWpFWWTjIZ8wc08PMvuW//XTUHt/2GcOHwJl3JyeSyLNQG8v/iKyMdIiuNq9PWNn7Je/tDMBnPvrkbY6KAYbUvfnJ1mgbivOyTfVk1ZTyWcMvLvikLaYPxhOM3aZ7W1WisqDQk0eBquAfYpiLGaS2uopHDsi+hkIBz+muj/Quz7rRL3ueV36RpVuYAJg/zWtntHFycKC/Ilz9JCGKlmMGTvZZCVo1jf5MzGLOqio X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Back when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3]. Splitting single logical structure into multiple ones leads to more complicated management, extra pointer dereferences and overall less maintainable code. When that split-away part is a lock, it complicates things even further. With no performance benefits, there are no reasons for this split. Merging the vm_lock back into vm_area_struct also allows vm_area_struct to use SLAB_TYPESAFE_BY_RCU later in this patchset. Move vm_lock back into vm_area_struct, aligning it at the cacheline boundary and changing the cache to be cacheline-aligned as well. With kernel compiled using defconfig, this causes VMA memory consumption to grow from 160 (vm_area_struct) + 40 (vm_lock) bytes to 256 bytes: slabinfo before: ... : ... vma_lock ... 40 102 1 : ... vm_area_struct ... 160 51 2 : ... slabinfo after moving vm_lock: ... : ... vm_area_struct ... 256 32 2 : ... Aggregate VMA memory consumption per 1000 VMAs grows from 50 to 64 pages, which is 5.5MB per 100000 VMAs. Note that the size of this structure is dependent on the kernel configuration and typically the original size is higher than 160 bytes. Therefore these calculations are close to the worst case scenario. A more realistic vm_area_struct usage before this change is: ... : ... vma_lock ... 40 102 1 : ... vm_area_struct ... 176 46 2 : ... Aggregate VMA memory consumption per 1000 VMAs grows from 54 to 64 pages, which is 3.9MB per 100000 VMAs. This memory consumption growth can be addressed later by optimizing the vm_lock. [1] https://lore.kernel.org/all/20230227173632.3292573-34-surenb@google.com/ [2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/ [3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/ Signed-off-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes --- include/linux/mm.h | 28 ++++++++++-------- include/linux/mm_types.h | 6 ++-- kernel/fork.c | 49 ++++---------------------------- tools/testing/vma/vma_internal.h | 33 +++++---------------- 4 files changed, 32 insertions(+), 84 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1ba2e480ae63..737c003b0a1e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -684,6 +684,12 @@ static inline void vma_numab_state_free(struct vm_area_struct *vma) {} #endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_PER_VMA_LOCK +static inline void vma_lock_init(struct vm_area_struct *vma) +{ + init_rwsem(&vma->vm_lock.lock); + vma->vm_lock_seq = UINT_MAX; +} + /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -701,7 +707,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) return false; - if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) + if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0)) return false; /* @@ -716,7 +722,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * This pairs with RELEASE semantics in vma_end_write_all(). */ if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { - up_read(&vma->vm_lock->lock); + up_read(&vma->vm_lock.lock); return false; } return true; @@ -731,7 +737,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) { mmap_assert_locked(vma->vm_mm); - down_read_nested(&vma->vm_lock->lock, subclass); + down_read_nested(&vma->vm_lock.lock, subclass); } /* @@ -743,13 +749,13 @@ static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int static inline void vma_start_read_locked(struct vm_area_struct *vma) { mmap_assert_locked(vma->vm_mm); - down_read(&vma->vm_lock->lock); + down_read(&vma->vm_lock.lock); } static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ - up_read(&vma->vm_lock->lock); + up_read(&vma->vm_lock.lock); rcu_read_unlock(); } @@ -778,7 +784,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) if (__is_vma_write_locked(vma, &mm_lock_seq)) return; - down_write(&vma->vm_lock->lock); + down_write(&vma->vm_lock.lock); /* * We should use WRITE_ONCE() here because we can have concurrent reads * from the early lockless pessimistic check in vma_start_read(). @@ -786,7 +792,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. */ WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); - up_write(&vma->vm_lock->lock); + up_write(&vma->vm_lock.lock); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -798,7 +804,7 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) static inline void vma_assert_locked(struct vm_area_struct *vma) { - if (!rwsem_is_locked(&vma->vm_lock->lock)) + if (!rwsem_is_locked(&vma->vm_lock.lock)) vma_assert_write_locked(vma); } @@ -831,6 +837,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ +static inline void vma_lock_init(struct vm_area_struct *vma) {} static inline bool vma_start_read(struct vm_area_struct *vma) { return false; } static inline void vma_end_read(struct vm_area_struct *vma) {} @@ -865,10 +872,6 @@ static inline void assert_fault_locked(struct vm_fault *vmf) extern const struct vm_operations_struct vma_dummy_vm_ops; -/* - * WARNING: vma_init does not initialize vma->vm_lock. - * Use vm_area_alloc()/vm_area_free() if vma needs locking. - */ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { memset(vma, 0, sizeof(*vma)); @@ -877,6 +880,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) INIT_LIST_HEAD(&vma->anon_vma_chain); vma_mark_detached(vma, false); vma_numab_state_init(vma); + vma_lock_init(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 80fef38d9d64..5c4bfdcfac72 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -716,8 +716,6 @@ struct vm_area_struct { * slowpath. */ unsigned int vm_lock_seq; - /* Unstable RCU readers are allowed to read this. */ - struct vma_lock *vm_lock; #endif /* @@ -770,6 +768,10 @@ struct vm_area_struct { struct vma_numab_state *numab_state; /* NUMA Balancing state */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_PER_VMA_LOCK + /* Unstable RCU readers are allowed to read this. */ + struct vma_lock vm_lock ____cacheline_aligned_in_smp; +#endif } __randomize_layout; #ifdef CONFIG_NUMA diff --git a/kernel/fork.c b/kernel/fork.c index 0061cf2450ef..7823797e31d2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -436,35 +436,6 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; -#ifdef CONFIG_PER_VMA_LOCK - -/* SLAB cache for vm_area_struct.lock */ -static struct kmem_cache *vma_lock_cachep; - -static bool vma_lock_alloc(struct vm_area_struct *vma) -{ - vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); - if (!vma->vm_lock) - return false; - - init_rwsem(&vma->vm_lock->lock); - vma->vm_lock_seq = UINT_MAX; - - return true; -} - -static inline void vma_lock_free(struct vm_area_struct *vma) -{ - kmem_cache_free(vma_lock_cachep, vma->vm_lock); -} - -#else /* CONFIG_PER_VMA_LOCK */ - -static inline bool vma_lock_alloc(struct vm_area_struct *vma) { return true; } -static inline void vma_lock_free(struct vm_area_struct *vma) {} - -#endif /* CONFIG_PER_VMA_LOCK */ - struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) { struct vm_area_struct *vma; @@ -474,10 +445,6 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) return NULL; vma_init(vma, mm); - if (!vma_lock_alloc(vma)) { - kmem_cache_free(vm_area_cachep, vma); - return NULL; - } return vma; } @@ -496,10 +463,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) * will be reinitialized. */ data_race(memcpy(new, orig, sizeof(*new))); - if (!vma_lock_alloc(new)) { - kmem_cache_free(vm_area_cachep, new); - return NULL; - } + vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); vma_numab_state_init(new); dup_anon_vma_name(orig, new); @@ -511,7 +475,6 @@ void __vm_area_free(struct vm_area_struct *vma) { vma_numab_state_free(vma); free_anon_vma_name(vma); - vma_lock_free(vma); kmem_cache_free(vm_area_cachep, vma); } @@ -522,7 +485,7 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) vm_rcu); /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock), vma); + VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma); __vm_area_free(vma); } #endif @@ -3168,11 +3131,9 @@ void __init proc_caches_init(void) sizeof(struct fs_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL); - - vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT); -#ifdef CONFIG_PER_VMA_LOCK - vma_lock_cachep = KMEM_CACHE(vma_lock, SLAB_PANIC|SLAB_ACCOUNT); -#endif + vm_area_cachep = KMEM_CACHE(vm_area_struct, + SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC| + SLAB_ACCOUNT); mmap_init(); nsproxy_cache_init(); } diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 1d9fc97b8e80..11c2c38ca4e8 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -230,10 +230,10 @@ struct vm_area_struct { /* * Can only be written (using WRITE_ONCE()) while holding both: * - mmap_lock (in write mode) - * - vm_lock->lock (in write mode) + * - vm_lock.lock (in write mode) * Can be read reliably while holding one of: * - mmap_lock (in read or write mode) - * - vm_lock->lock (in read or write mode) + * - vm_lock.lock (in read or write mode) * Can be read unreliably (using READ_ONCE()) for pessimistic bailout * while holding nothing (except RCU to keep the VMA struct allocated). * @@ -242,7 +242,7 @@ struct vm_area_struct { * slowpath. */ unsigned int vm_lock_seq; - struct vma_lock *vm_lock; + struct vma_lock vm_lock; #endif /* @@ -408,17 +408,10 @@ static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi) return mas_find(&vmi->mas, ULONG_MAX); } -static inline bool vma_lock_alloc(struct vm_area_struct *vma) +static inline void vma_lock_init(struct vm_area_struct *vma) { - vma->vm_lock = calloc(1, sizeof(struct vma_lock)); - - if (!vma->vm_lock) - return false; - - init_rwsem(&vma->vm_lock->lock); + init_rwsem(&vma->vm_lock.lock); vma->vm_lock_seq = UINT_MAX; - - return true; } static inline void vma_assert_write_locked(struct vm_area_struct *); @@ -439,6 +432,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); vma_mark_detached(vma, false); + vma_lock_init(vma); } static inline struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) @@ -449,10 +443,6 @@ static inline struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) return NULL; vma_init(vma, mm); - if (!vma_lock_alloc(vma)) { - free(vma); - return NULL; - } return vma; } @@ -465,10 +455,7 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return NULL; memcpy(new, orig, sizeof(*new)); - if (!vma_lock_alloc(new)) { - free(new); - return NULL; - } + vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); return new; @@ -638,14 +625,8 @@ static inline void mpol_put(struct mempolicy *) { } -static inline void vma_lock_free(struct vm_area_struct *vma) -{ - free(vma->vm_lock); -} - static inline void __vm_area_free(struct vm_area_struct *vma) { - vma_lock_free(vma); free(vma); } From patchwork Sun Nov 17 08:09:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13877798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4BFED68BDD for ; Sun, 17 Nov 2024 08:09:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 731D89C001B; Sun, 17 Nov 2024 03:09:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E1616B00C1; Sun, 17 Nov 2024 03:09:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50CBB9C001B; Sun, 17 Nov 2024 03:09:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 283036B0085 for ; Sun, 17 Nov 2024 03:09:49 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id DADFA1A0137 for ; Sun, 17 Nov 2024 08:09:48 +0000 (UTC) X-FDA: 82794861912.12.A6DA173 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf11.hostedemail.com (Postfix) with ESMTP id 11EAA4000B for ; Sun, 17 Nov 2024 08:08:47 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=KaL8imfX; spf=pass (imf11.hostedemail.com: domain of 3yqQ5ZwYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3yqQ5ZwYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731830896; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=b7bh6sSGESOsY9pdAptx272UcEa/Ygad7puDmCEq6Kg=; b=uHKZ0oO+joiun1jLywYvFhZFw3MuBV1HWYvwm7oNnAO4orcX/RWSL1j5BaM7NXpe6UEyjT EEgeEKqcmLj6IDCxGC50eRuzJNBKzRMad09KQvL4zoL97OTB8KGVkvOEuIxhbv8FVKYxbC olNAh9mkVGOKgI7l5QgRIoJj4rUZgTs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731830896; a=rsa-sha256; cv=none; b=dyOAMOXS3hXiNTVnOdHEYPZR449jb+REFyuf4qbOukbkB1x1kA0SJuC3bSgcw1qN6CnGrS e0vh6yPoQln4QMU2iFYEU97cGtsi4LVUw5Hr8RJBc5t/gr3PzxTeGxoUBiowipfFDlqAXd 1tXjcZiVI628i0Fqc9dnMWdmWgZ8Bc4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=KaL8imfX; spf=pass (imf11.hostedemail.com: domain of 3yqQ5ZwYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3yqQ5ZwYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e0b8fa94718so1351987276.0 for ; Sun, 17 Nov 2024 00:09:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731830986; x=1732435786; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=b7bh6sSGESOsY9pdAptx272UcEa/Ygad7puDmCEq6Kg=; b=KaL8imfXi7yPCHMsUhZ7h7OmJ5CR38NTQwTLRgvpUd+i9Pl59zT4zd5zW+4sdFC1qR 23d8vtOSe+MnbmMtnmXkojjf2xsctIScrrwCUonAq8tlvr//p00fH8efaiI4fMoN5jFm dSbh/jjzHiXwwWMeDvQl+NRmaDAovBBU72pEb4u3mnPFG2KHj+czEVNHK7RTGrbi/ziA EnttJHhbv8VCmprYgetBRRw971gZLcI7kuu0XrdkzVV6S5tDIm25OBl7IhKbTGJuZPpK EcwSdFYxNI71ECxZe9QMtNBTnBwSTekOXNfINVKsaGA1uZhS2w9OPhJO2WDfqaHnPz8q 2/nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731830986; x=1732435786; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=b7bh6sSGESOsY9pdAptx272UcEa/Ygad7puDmCEq6Kg=; b=VVdtp8xUjOkYDSI+S5ZqJaPBNbzJpc6mITCvy/qsAZni1Xfm5yikPY1iWwKvnbBqnA N0otQp+sL6cKFaKzDUeaGNPy8Pq8kJLZi1FPeqr8FCxO8BC87E+6OihpzVxcqE+DNZP1 WcAgXl7rdakiFurCHIW/eWsI9LiKlSPGO5g7RKGhOUhP2/aL0d5DPyS0Kmx/FgftjLk1 8TVlUxChVuh6kB17OQTjczL95ZtEQWFt3Kxy/JU9Nl5AhXA7lE2OdQI89Ggxx7gL6I3o cyNMURoF9UmyeIa4oxiD+KzbBJeT0b1cSC3V2vDKurw709+mQBj61KEPtsnHTwdSQXtR +xuw== X-Forwarded-Encrypted: i=1; AJvYcCUGcyitc73DLBSw76Y9cfjP1nOOpH6LTVwLspt+TueRFHkIbJBrx51BmoP5Z02vDUQDsKjmtUlHWQ==@kvack.org X-Gm-Message-State: AOJu0YwIWjq/LPYJ0n2t/UhZ5KQRSZJx7NNz6oOdMIpdbY4qb2e0KXC7 NMb4oIXpSCfuhYN5gFcgvtvzTcVwgs/XwZXYHVGizAryMAiI+M89m9cgfr16AZqWQ0oKiBm5tZb Z2A== X-Google-Smtp-Source: AGHT+IGF2YWbcHToc+T5cvIS4jQysZu9n+W2Aar36xDvDbuGFhbNn0TYu5GxTnL5QmwYgoOaAYtUUxzh9nI= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:bafc:6633:f766:6415]) (user=surenb job=sendgmr) by 2002:a25:ab48:0:b0:e28:e6a1:fc53 with SMTP id 3f1490d57ef6-e382639f42amr153411276.5.1731830986067; Sun, 17 Nov 2024 00:09:46 -0800 (PST) Date: Sun, 17 Nov 2024 00:09:29 -0800 In-Reply-To: <20241117080931.600731-1-surenb@google.com> Mime-Version: 1.0 References: <20241117080931.600731-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241117080931.600731-4-surenb@google.com> Subject: [PATCH v3 3/5] mm: mark vma as detached until it's added into vma tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam10 X-Stat-Signature: ynp8omhn8qnw5ca45n141k68o4wetfy6 X-Rspamd-Queue-Id: 11EAA4000B X-Rspam-User: X-HE-Tag: 1731830927-309204 X-HE-Meta: U2FsdGVkX192mXjytsqPMaY+9Nj+3V1gZgMLKxlrZRL8fjdvDcOzTbzyQRzBcJ3PNZY4NTrS7UNpZ43HIVy0+AhPrjXLOK7Xs1mhcKQI3uRAsEVO+geSu4LzSprBI43Zl8tckBSX3dbWr19rbuAN6PsllZEMWvD5q/EsQdEnfteMjzLKIzA6sI8l5fO7Y83o96/86IKf3sUZ5TC0aZK9tPMTxOiU42iynS6mKUT3Th9sAu+kcScq87euYKIPl1y6xp8qLMaKBUReIXntPktS7qt0Cb93RXK4NTpSY/zmVw+M2YSrs3pCN5Mk4MCBsDEb/tUrtbWd3XEoAlERU0iRzs5LsgV5ORFVsI8mIhnwedEtxuWMOK/5HwBVvXEgCcSuRBVWuOFH1AMIeGEWdP4mIdvfWihz9Q6SaOI8y9nhSGCaigjaFprIVZvniY8r7mF8oSurKwACzvC8sN5ck65tordL/aywJMLr02KjiOROwNSO4QY2S6sncIU//Q8QM8bOPd/JMigvtZk5AENPyw4iLjmUJgypd+en2OZp02beg8iSoFI6oWbHqilGZhV+qf4z6vT5XaTuvHZo6UgBVPBgkgq/vyjyvmpYol5tgMKdusOKP9zycOXRbPCafW2HlNF4BqVvSSe9ZDwepHeFl+J3tBOA7WpVnRcXUc/+0zjnctRPiGJTEa+9RkQ3brbYC5+ST8LUHSxobKYmvoBdCUETaSL7jNXyLDVOOM7lzXX3cyiCkbQE1hkI9u0ChAyXndvlDJ8kFToMsEVuAOZ63au5TXjg4NivNJ1yX4XzhZfsvJ6tu1BdU3MixG3fS0pMPEjR1H+cfrsm/xk5/+eQNiw0JJR/4ZsSOL+c2oKuNcmcTYmLr7wiewQoQe7KdmX2Chh6raeO4ej/VXm5VfRdnU5k8BN2yNLSvbrYUs4aChp1s55Lr3axS8z4t2KnnaI7T1LXjQl86RYXWJi73BpArT5 +6QASQeE VyDvjR99l6MT7FXXVXXtQwZ25L3a5nmW06txdb2DNYCFCogmUdZf7PZZt1ASK6U+zk1CE8B55D4fSVNsxJAw3Eb7jsbUmmYeN2FaXBxFeNBRTwhTVUuywenY0ARP+X50Flx3uU9LiKiZgxde/5jNzyYKxmkKA6ERW4LzzbKZTR41Te9rVZcwsZanPtO6ZB8qC6rvWnyhLBmmNi0UlIniipeciWH8+78MG/+3GyrlDen15WRNXZLF4bcFnOX7Vr9/8C0phfsulnZrqb8iDU1XpQzMyJpQ28RLPwycR7Sqk6cI4LH99CMTX2SIPeZLhb+DtgBUj3NOfwYN4tefEup8quzatcHD2zcJEo8kNQn+6YEkfKVzinbaS259Yy+ovyipTLTWXEWC3xZ92bqznAcz6bbWKL1TFEWPkNUC/o74aRVgWuKhF3nm5DXWtndtMaSKvrQIjOKY0SXu20vpSCL3g2N37tydc49sHqsQGHOcKhp4Auh+r4CuU/hPCu858azKEL/neQAkt9oLrNp8IWQvOmXQv/bRNF0SV2IrqgARxjJuvDfw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree. Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 27 ++++++++++++++++++++------- kernel/fork.c | 4 ++++ mm/memory.c | 2 +- mm/vma.c | 6 +++--- mm/vma.h | 2 ++ tools/testing/vma/vma_internal.h | 17 ++++++++++++----- 6 files changed, 42 insertions(+), 16 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 737c003b0a1e..dd1b6190df28 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -808,12 +808,21 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) vma_assert_write_locked(vma); } -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; +} + +static inline bool is_vma_detached(struct vm_area_struct *vma) +{ + return vma->detached; } static inline void release_fault_lock(struct vm_fault *vmf) @@ -844,8 +853,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) { mmap_assert_write_locked(vma->vm_mm); } -static inline void vma_mark_detached(struct vm_area_struct *vma, - bool detached) {} +static inline void vma_mark_attached(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma) {} static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address) @@ -878,7 +887,10 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; +#endif vma_numab_state_init(vma); vma_lock_init(vma); } @@ -1073,6 +1085,7 @@ static inline int vma_iter_bulk_store(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } diff --git a/kernel/fork.c b/kernel/fork.c index 7823797e31d2..f0cec673583c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -465,6 +465,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) data_race(memcpy(new, orig, sizeof(*new))); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; +#endif vma_numab_state_init(new); dup_anon_vma_name(orig, new); diff --git a/mm/memory.c b/mm/memory.c index 209885a4134f..d0197a0c0996 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6279,7 +6279,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, goto inval; /* Check if the VMA got isolated after we found it */ - if (vma->detached) { + if (is_vma_detached(vma)) { vma_end_read(vma); count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ diff --git a/mm/vma.c b/mm/vma.c index 8a454a7bbc80..73104d434567 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -295,7 +295,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, if (vp->remove) { again: - vma_mark_detached(vp->remove, true); + vma_mark_detached(vp->remove); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -1220,7 +1220,7 @@ static void reattach_vmas(struct ma_state *mas_detach) mas_set(mas_detach, 0); mas_for_each(mas_detach, vma, ULONG_MAX) - vma_mark_detached(vma, false); + vma_mark_attached(vma); __mt_destroy(mas_detach->tree); } @@ -1295,7 +1295,7 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, if (error) goto munmap_gather_failed; - vma_mark_detached(next, true); + vma_mark_detached(next); nrpages = vma_pages(next); vms->nr_pages += nrpages; diff --git a/mm/vma.h b/mm/vma.h index 388d34748674..2e680f357ace 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -162,6 +162,7 @@ static inline int vma_iter_store_gfp(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } @@ -385,6 +386,7 @@ static inline void vma_iter_store(struct vma_iterator *vmi, __mas_set_range(&vmi->mas, vma->vm_start, vma->vm_end - 1); mas_store_prealloc(&vmi->mas, vma); + vma_mark_attached(vma); } static inline unsigned long vma_iter_addr(struct vma_iterator *vmi) diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 11c2c38ca4e8..2fed366d20ef 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -414,13 +414,17 @@ static inline void vma_lock_init(struct vm_area_struct *vma) vma->vm_lock_seq = UINT_MAX; } +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + static inline void vma_assert_write_locked(struct vm_area_struct *); -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; } extern const struct vm_operations_struct vma_dummy_vm_ops; @@ -431,7 +435,8 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; vma_lock_init(vma); } @@ -457,6 +462,8 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) memcpy(new, orig, sizeof(*new)); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; return new; } From patchwork Sun Nov 17 08:09:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13877799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 576E4D68BDD for ; Sun, 17 Nov 2024 08:09:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B78F99C001C; Sun, 17 Nov 2024 03:09:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AFF986B00C4; Sun, 17 Nov 2024 03:09:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92BFA9C001C; Sun, 17 Nov 2024 03:09:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6C5816B00C3 for ; Sun, 17 Nov 2024 03:09:54 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E9CBF4018B for ; Sun, 17 Nov 2024 08:09:53 +0000 (UTC) X-FDA: 82794862122.12.1C0F0F2 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf10.hostedemail.com (Postfix) with ESMTP id 235DDC0006 for ; Sun, 17 Nov 2024 08:09:30 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LiVq5fsD; spf=pass (imf10.hostedemail.com: domain of 3zqQ5ZwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3zqQ5ZwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731830901; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ydc3xnWflSOvT3mKyRecGyeDU6jHNwbSKwasb1j3Pls=; b=xcChxZY9ZEot3dN4Q23zj3pxDcLZ3w+CxCoxftSrSfz1lU+8bypxpRdWapoP43XfPhlMSa r35pHyJ7IUjtM5UA44UJjo0R8RYLOeuWifIoI41gAVJN0WPYOniLe+qXmuqLE+L8v0UGem lhgP+OcwOv5pTnZfyfHiO0guPsbDIZk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731830901; a=rsa-sha256; cv=none; b=SI9AOrbVf5hwVRMteN+Ni2jRKQFgMeBOibDyTkJfATvUIkrq+70mAVzlNBlNg8unq6didH 678O9EKZmMitms4+5QQEKumZmC3CPIxykt0HzD7hrb9S+t2lc2F7htrYptJSbcWFXD2tkK l66mEirPCi2lsLMo2nSXs2kUhy9HOnE= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LiVq5fsD; spf=pass (imf10.hostedemail.com: domain of 3zqQ5ZwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3zqQ5ZwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e02fff66a83so2666007276.0 for ; Sun, 17 Nov 2024 00:09:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731830991; x=1732435791; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ydc3xnWflSOvT3mKyRecGyeDU6jHNwbSKwasb1j3Pls=; b=LiVq5fsD4ohrthLdvb0x1BEF5qQefZ+3b2urJ8B2MT90Vsp9DaScEEWURH7hlp6on0 lPPqMWKQjFOxFxWQNqiGdYDZ6RGystjr1jkPHB03hbxjcVGMFLSGCZwStRA0aqP/1pem P68tkAbMneXHyuepf8uLGDhCFAnooCAsXHoRWJvjAOPjL8SF2ijFf/JXtAPkfPZnaMtO Oi2qDWpIrFED5CCWVKgfbzaIwh3wo5sKFX50HByQ51h8VDr6EoQqVaX6oXq6uOuPZR7L ZU7Pn5ZTWZJWckLKTRyFboQbrxFFntWuQl6ozXArj1HFg0skE1S8US571PHfip6c3w/s 8T0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731830991; x=1732435791; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ydc3xnWflSOvT3mKyRecGyeDU6jHNwbSKwasb1j3Pls=; b=JKGpSAVtWAZ7VCSyk7USmjw47LTOtZiWhbPt0/DJJPdVfB5YW3H3bJbuFP44wP1DLP jnwybGLYK0RKXs85rx1NyIKx+rHQl3Yf8E7CxaYAgMY5rImgcveYQyekJtAA+FnvpgML PQXCtQzM+/risfvp8vXKFb3nAa0fIHUC52DYokUf98HdjIgN9YIHh3VaeDbhUUN1PDc2 +yrMhUM+uP1sSEIs5i8+dGvMvbXPxdIy3rjkEzRHR0SfqBZZjg3m8um0JOBPwALLzPT9 JON5aa9Z6qEtG8ZOq3mCH7Ej/8VEIsSjVmxspNurxnF9Rh/wxQ8cQmsMQL5ju7RAJpoo bb0A== X-Forwarded-Encrypted: i=1; AJvYcCV94Vv/1eUSTwR22xWJVfHqs1q1Vo0gjzBzCE+DeE94dC2rPnH5x6k8YCfnBvYE6ni6p0oFhNGWcA==@kvack.org X-Gm-Message-State: AOJu0YzzRuHb/lP2reM33+z3VtvwbHa0hKM4IW8XrF45xMM0RyAuqqHJ YYt8ioIOZAr6WT6z5A0hsC1h45Avg8syQxIZXwIUiSAz0CQn3ngGVTkD2cRyw96dRLh86Vdkw3N Zuw== X-Google-Smtp-Source: AGHT+IFBFQn/cqW/Nbk+DrZLqER7IMLE2PxGHl++p5as9o7fv/NkZldv72m8b+Y3E+tx/FPb9DpLbddXQ/Q= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:bafc:6633:f766:6415]) (user=surenb job=sendgmr) by 2002:a25:e097:0:b0:e28:eae7:f838 with SMTP id 3f1490d57ef6-e3825bc27f4mr8116276.0.1731830990609; Sun, 17 Nov 2024 00:09:50 -0800 (PST) Date: Sun, 17 Nov 2024 00:09:30 -0800 In-Reply-To: <20241117080931.600731-1-surenb@google.com> Mime-Version: 1.0 References: <20241117080931.600731-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241117080931.600731-5-surenb@google.com> Subject: [PATCH v3 4/5] mm: make vma cache SLAB_TYPESAFE_BY_RCU From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam10 X-Stat-Signature: ijdkj53ika8ry3afqjd8z97e8xgwc8sy X-Rspamd-Queue-Id: 235DDC0006 X-Rspam-User: X-HE-Tag: 1731830970-255536 X-HE-Meta: U2FsdGVkX1/WKegCnS+OM+1vN6l8OQR9sXx2MUiZ9lAN/mdJWuceC6AtdorfGlECA4f4E/9XfJpwnZUY9sebV68XgICt8PTqllgZ5O5MM611oSj1WDs1wiRylvSRf2FOjbzHN0IfIhXQ2038X2a3A6j9jXR29Bb3ZW2qsxa3uFsK2mMrUb6A/4X+3eaZv8xb1bL5YVcOINsjD3gz2iD9LR8dWsvkRBFlQVGbTeUUp2O4T+OF8oWLP+xwhLD4GBDy8N8x+2p6DhQrJKUZJ0guCSjYwd7tEj3XtiqGkbxbRRwLC2muSuOstc7dOmBBWa5gkjKraQ9TiwwSfkorSa9Ey2SnpLPeiUdP/h9tW37q9JV8wQITp/FHGiQyukTO/Pbc8DS4BKlsrUSJjpJlYrnlESN51MjxmAcGivW01yb/XpS2gimHhm/UgsGE+1DPun5dAq/lvmKBtr688Y526DgA19DflccT2AUn2hBjL9dpPXyQq/Wy1jIbPp2Z4eWk7+gIDEvgLZMd8RCpCP8q0tpbdWgs0GFIcyYFwy6T7uX2yQSmtHQjm4RCsDz5UXXxNC1x1cwM8t8a8NsTV20PsUdETJP8ihWz6pwqm0OBOjmCJ7yPF/2cMQCUTCMJLJUWF/gFdnToGpelHXJn6TZmT9mDTR4d7sywKpBfcBhkPuVDGMRGBJkh3CaCxpcn7BluLxrGBxx2cPbDgV5OLWkgYKXxuDjszfNr9KipKcz109pp3fw03irH4SbBCWxAHOraqZ5UuVQ2Sd7RWaEYnJOsXL3A4YgD/ykC2DeTsyDdQKHSyFtAF0bhjL8N9KGToBH/1xZetavgpGM3B4TpcRMBRXEwzZRKeSiLIUk0zOXdCKsDBFwWSQm7AMGy09w07b4UnDZWrf+jFuoRD1J7fNED7uPSsM+4MSL4oBVM/wOv7nipB1nAQDZXOqdpQb3El4EvtUKfGqg9gKzljx0TDGEXMKP SQz4cODT 1TKQatM6tFDi4msY6hruTwTUGpm2PDvf0P+3jlK0RYEp3EmWp++fKBpbt/PVSG1uKzRwlE63el7E7OOb1+KF0co2gTo0EfS3SblVMpeJThxRRtrgkMo/KruY08qCwAbiS8m52UKZAa+1/3neOi+5YfzljH4BbV0HSO64mjPDgr2vfbldXl/b0919IGoXgp+KplreIwCY3pLL3lS1abUHgCWQGchMQq08zIy8vNAQojmIr5pt8htQH0ywBo2b9kH2KOwkcbzjf6p9KSrCK0zQA1yIuaMWxQwYV6PL8QOnTfnQwCuwl/KXtFMBYWCrt5LPPtREr+0u3F6zOsA2cyyHDD7AgJaIc9j79JEW9/Z9vQ7ny1yTB3L8XZ3aovAAvvjNUpRsFlqWk46qlUN2q07SUYRV4Kow+6eWN9gGhXjR20w8KMNPGoKHSImHLRn7blmmX0F0I11WW0hI7P7eGTQhe6gRUpYV/qvbXfLjfrSMnIEfZbrUg/I+JnB5r2kbVqjANuwW+cPIrSQBAD834Spg82w4zo2omKQS1Tj4EdoojGtxsQWPOA2RU3RUa7yZ6/Ba66JxR8zEGHRZOhThD30ipm7kG5g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu(). lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it. vma reuse introduces several new possibilities: 1. vma can be reused after it was found but before it is locked; 2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read(); 3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while being read-locked; For case #1 current checks will help detecting cases when: - vma was reused but not yet added into the tree (detached check) - vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. For case #3, we write-lock the vma in vma_mark_attached(), ensuring that vma does not get re-attached while read-locked by a user of the vma before it was recycled. This write-locking should not cause performance issues because contention during vma_mark_attached() can happen only in the rare vma reuse case. Even when this happens, it's the slowpath (write-lock) which will be waiting, not the page fault path. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls. Adding a freeptr_t into vm_area_struct (unioned with vm_start/vm_end) could be used to avoids bloating the structure, however currently custom free pointers are not supported in combination with a ctor (see the comment for kmem_cache_args.freeptr_offset). Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 48 ++++++++++++++++++++++++----- include/linux/mm_types.h | 13 +++----- kernel/fork.c | 53 +++++++++++++++++++------------- mm/memory.c | 7 +++-- mm/vma.c | 2 +- tools/testing/vma/vma_internal.h | 7 +++-- 6 files changed, 86 insertions(+), 44 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index dd1b6190df28..d8e10e1e34ad 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -257,7 +257,7 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); /* Use only if VMA has no other users */ -void __vm_area_free(struct vm_area_struct *vma); +void vm_area_free_unreachable(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; @@ -690,12 +690,32 @@ static inline void vma_lock_init(struct vm_area_struct *vma) vma->vm_lock_seq = UINT_MAX; } +#define VMA_BEFORE_LOCK offsetof(struct vm_area_struct, vm_lock) +#define VMA_LOCK_END(vma) \ + (((void *)(vma)) + offsetofend(struct vm_area_struct, vm_lock)) +#define VMA_AFTER_LOCK \ + (sizeof(struct vm_area_struct) - offsetofend(struct vm_area_struct, vm_lock)) + +static inline void vma_clear(struct vm_area_struct *vma) +{ + /* Preserve vma->vm_lock */ + memset(vma, 0, VMA_BEFORE_LOCK); + memset(VMA_LOCK_END(vma), 0, VMA_AFTER_LOCK); +} + +static inline void vma_copy(struct vm_area_struct *new, struct vm_area_struct *orig) +{ + /* Preserve vma->vm_lock */ + data_race(memcpy(new, orig, VMA_BEFORE_LOCK)); + data_race(memcpy(VMA_LOCK_END(new), VMA_LOCK_END(orig), VMA_AFTER_LOCK)); +} + /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to * using mmap_lock. The function should never yield false unlocked result. */ -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { /* * Check before locking. A race might cause false locked result. @@ -704,7 +724,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * we don't rely on for anything - the mm_lock_seq read against which we * need ordering is below. */ - if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) + if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) return false; if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0)) @@ -721,7 +741,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ - if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { + if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) { up_read(&vma->vm_lock.lock); return false; } @@ -810,7 +830,18 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) static inline void vma_mark_attached(struct vm_area_struct *vma) { + /* vma shoudn't be already attached */ + VM_BUG_ON_VMA(!vma->detached, vma); + + /* + * Lock here can be contended only if the vma got reused after + * lock_vma_under_rcu() found it but before it had a chance to + * read-lock it. Write-locking the vma guarantees that the vma + * won't be attached until all its old users are out. + */ + down_write(&vma->vm_lock.lock); vma->detached = false; + up_write(&vma->vm_lock.lock); } static inline void vma_mark_detached(struct vm_area_struct *vma) @@ -847,7 +878,11 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_lock_init(struct vm_area_struct *vma) {} -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline void vma_clear(struct vm_area_struct *vma) + { memset(vma, 0, sizeof(*vma)); } +static inline void vma_copy(struct vm_area_struct *new, struct vm_area_struct *orig) + { data_race(memcpy(new, orig, sizeof(*new))); } +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { return false; } static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} @@ -883,7 +918,7 @@ extern const struct vm_operations_struct vma_dummy_vm_ops; static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { - memset(vma, 0, sizeof(*vma)); + vma_clear(vma); vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); @@ -892,7 +927,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->detached = true; #endif vma_numab_state_init(vma); - vma_lock_init(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5c4bfdcfac72..8f6b0c935c2b 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -667,15 +667,10 @@ struct vma_numab_state { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - union { - struct { - /* VMA covers [vm_start; vm_end) addresses within mm */ - unsigned long vm_start; - unsigned long vm_end; - }; -#ifdef CONFIG_PER_VMA_LOCK - struct rcu_head vm_rcu; /* Used for deferred freeing. */ -#endif + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start; + unsigned long vm_end; }; /* diff --git a/kernel/fork.c b/kernel/fork.c index f0cec673583c..76c68b041f8a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -436,6 +436,11 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; +static void vm_area_ctor(void *data) +{ + vma_lock_init(data); +} + struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) { struct vm_area_struct *vma; @@ -462,8 +467,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) * orig->shared.rb may be modified concurrently, but the clone * will be reinitialized. */ - data_race(memcpy(new, orig, sizeof(*new))); - vma_lock_init(new); + vma_copy(new, orig); INIT_LIST_HEAD(&new->anon_vma_chain); #ifdef CONFIG_PER_VMA_LOCK /* vma is not locked, can't use vma_mark_detached() */ @@ -475,32 +479,37 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -void __vm_area_free(struct vm_area_struct *vma) +static void __vm_area_free(struct vm_area_struct *vma, bool unreachable) { +#ifdef CONFIG_PER_VMA_LOCK + /* + * With SLAB_TYPESAFE_BY_RCU, vma can be reused and we need + * vma->detached to be set before vma is returned into the cache. + * This way reused object won't be used by readers until it's + * initialized and reattached. + * If vma is unreachable, there can be no other users and we + * can set vma->detached directly with no risk of a race. + * If vma is reachable, then it should have been already detached + * under vma write-lock or it was never attached. + */ + if (unreachable) + vma->detached = true; + else + VM_BUG_ON_VMA(!is_vma_detached(vma), vma); +#endif vma_numab_state_free(vma); free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); } -#ifdef CONFIG_PER_VMA_LOCK -static void vm_area_free_rcu_cb(struct rcu_head *head) +void vm_area_free(struct vm_area_struct *vma) { - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); - - /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma); - __vm_area_free(vma); + __vm_area_free(vma, false); } -#endif -void vm_area_free(struct vm_area_struct *vma) +void vm_area_free_unreachable(struct vm_area_struct *vma) { -#ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); -#else - __vm_area_free(vma); -#endif + __vm_area_free(vma, true); } static void account_kernel_stack(struct task_struct *tsk, int account) @@ -3135,9 +3144,11 @@ void __init proc_caches_init(void) sizeof(struct fs_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL); - vm_area_cachep = KMEM_CACHE(vm_area_struct, - SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC| - SLAB_ACCOUNT); + vm_area_cachep = kmem_cache_create("vm_area_struct", + sizeof(struct vm_area_struct), 0, + SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU| + SLAB_ACCOUNT, vm_area_ctor); + mmap_init(); nsproxy_cache_init(); } diff --git a/mm/memory.c b/mm/memory.c index d0197a0c0996..c8a3e820ed66 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6275,7 +6275,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma) goto inval; - if (!vma_start_read(vma)) + if (!vma_start_read(mm, vma)) goto inval; /* Check if the VMA got isolated after we found it */ @@ -6292,8 +6292,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, * fields are accessible for RCU readers. */ - /* Check since vm_start/vm_end might change before we lock the VMA */ - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + /* Check since vm_mm/vm_start/vm_end might change before we lock the VMA */ + if (unlikely(vma->vm_mm != mm || + address < vma->vm_start || address >= vma->vm_end)) goto inval_end_read; rcu_read_unlock(); diff --git a/mm/vma.c b/mm/vma.c index 73104d434567..050b83df3df2 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -382,7 +382,7 @@ void remove_vma(struct vm_area_struct *vma, bool unreachable) fput(vma->vm_file); mpol_put(vma_policy(vma)); if (unreachable) - __vm_area_free(vma); + vm_area_free_unreachable(vma); else vm_area_free(vma); } diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 2fed366d20ef..fd668d6cafc0 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -632,14 +632,15 @@ static inline void mpol_put(struct mempolicy *) { } -static inline void __vm_area_free(struct vm_area_struct *vma) +static inline void vm_area_free(struct vm_area_struct *vma) { free(vma); } -static inline void vm_area_free(struct vm_area_struct *vma) +static inline void vm_area_free_unreachable(struct vm_area_struct *vma) { - __vm_area_free(vma); + vma->detached = true; + vm_area_free(vma); } static inline void lru_add_drain(void) From patchwork Sun Nov 17 08:09:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13877800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF9A9D68BDD for ; Sun, 17 Nov 2024 08:09:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 472A66B00C3; Sun, 17 Nov 2024 03:09:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 422CF6B00C4; Sun, 17 Nov 2024 03:09:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C5626B00C5; Sun, 17 Nov 2024 03:09:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 085286B00C3 for ; Sun, 17 Nov 2024 03:09:58 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B1278140193 for ; Sun, 17 Nov 2024 08:09:57 +0000 (UTC) X-FDA: 82794862290.12.AB5766E Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf10.hostedemail.com (Postfix) with ESMTP id D7549C000F for ; Sun, 17 Nov 2024 08:09:34 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="zJ/mJpqQ"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 30qQ5ZwYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=30qQ5ZwYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731830930; a=rsa-sha256; cv=none; b=y8I4dnkx98keVSu56I1qb2BY14sRUZvTIyspjQel3qKWXpEJGFrHsoWVrnUXITKFpM/AKy FtKuzr43vUe/nMuuqlmsuc7FXUr5jzSsekk8esLn56Zg3VMt6UwEu2GDSbGsK6wuJTWvtH EGUZIlFjf+8tEbZo5riCPkH54NQK+hM= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="zJ/mJpqQ"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 30qQ5ZwYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=30qQ5ZwYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731830930; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TVe9xIxnbmkl7SmN6iZyRrlBkQz17Arq0DLY4ssSJic=; b=zl4EwbYWjmb9bOVWlwW6No5gLVWg2UpIpABslli/f89PNh5b9wEJHuwjjZxrq8EMuo0nps 5NVSa1ZF6DHDiRgryvDTZC5oPZHvhL/XHvuhXJGLd3GuYMzdcxGJRZYesEQ7RjQNFZCnTG xxraLDXBpsIwuMJZdoni3vHSFUP1+h4= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e3877744e45so2332838276.1 for ; Sun, 17 Nov 2024 00:09:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731830995; x=1732435795; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TVe9xIxnbmkl7SmN6iZyRrlBkQz17Arq0DLY4ssSJic=; b=zJ/mJpqQTL6PP4Pux5OIcOShfLrf/GAzByQkE/B+BqpFVIJDVoFbo8pt2zjmITBBZE UFx0SItp/Z+5U5FKqiSxU9FUXgjPoXTQmJG1i3D4taQ2baekeKfycm48z+vjKAHJEg/9 WJ/nouUzkVUB2WkvmcNbt2NT2yhnxfjojDZeBBS7p2Uw8KmioKZMz5I2dCUyfdcPZCtS nFczZop27OXPWSvC61CiqgJyHFPfvwUii0+zwD68D/Gtu4BpNN/6+fOvZ1ow5PF5ghbD l2QX1sL4S2G3/n/uwUS2kS21ahblapPcMtRjuMej+GSE96pINtsJK3bLSXN68DWQD83L jOHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731830995; x=1732435795; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TVe9xIxnbmkl7SmN6iZyRrlBkQz17Arq0DLY4ssSJic=; b=Y7rA6ylFHorPBg1s+QTrW8ZrIrx7FSwNU83JPjJhDXNMTnrgSr8nWhButFuWatJmzo TyPNy9VP0TGmXsN4GqYdWRrW/3XwoWeJpcT2H/295FqdRYv6oHsM7nnCZa5sKjZ3T4J3 Qr7ULy4mxirNT67/KtK3bu/0OjYkAcZuYOluaHA9OuPelfcOA7LbEbmcz+3TxY4JQitR UmeAyB3CnN4oIQ5364ZOb2Xw4sX4AHgqYxNCcpJgOH2vP3YaDzQX8mmg//15OQpOCz1B dGv73h4A5E0ojCkGEM588SPd/z/4r7I3nG34CEOR/M5o9yPlw0oVx11Py2FAtyW+MWWU VSIg== X-Forwarded-Encrypted: i=1; AJvYcCUGmw00YeqlhxuA/w8bSqIsh2t8wc91Ux2vt2KxCCsYvWqf+u3RELdKdEcUV4URTquCvqzifxnJGQ==@kvack.org X-Gm-Message-State: AOJu0Yz3+R1MFpPjdxcUzJXNx1q9EbLPNBDhkMdkvWPQvFGDI/CsTBZ1 pxfXpshYM6wZojy7HfdtGQLHssDQ919muBjyi7uQKSapLSl/fZdjK62eR7Y1DeKKTxoV8e3efbv kiw== X-Google-Smtp-Source: AGHT+IHOo0SNvb429sH9y3xLmKrGp2y4SlbBBZl72lF8+egqZzHX2n9q6YU+duTw+pCfixwfbSLpKdu+Ko4= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:bafc:6633:f766:6415]) (user=surenb job=sendgmr) by 2002:a25:aa8b:0:b0:e38:816c:df18 with SMTP id 3f1490d57ef6-e38816cec5dmr3614276.4.1731830994994; Sun, 17 Nov 2024 00:09:54 -0800 (PST) Date: Sun, 17 Nov 2024 00:09:31 -0800 In-Reply-To: <20241117080931.600731-1-surenb@google.com> Mime-Version: 1.0 References: <20241117080931.600731-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241117080931.600731-6-surenb@google.com> Subject: [PATCH v3 5/5] docs/mm: document latest changes to vm_lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Queue-Id: D7549C000F X-Rspamd-Server: rspam01 X-Stat-Signature: bf7f9ei7r4gz5agp1ifguxza4tceigyu X-HE-Tag: 1731830974-810920 X-HE-Meta: U2FsdGVkX19leqg5vfU9OjdfuorRiJ+KYGiZhG7eQvVKOkI2I68kZ3T24BUnEPKimF947Rwn6z5Y7CNQW9zH1f8LPDBA40S9WG31GErU/4SR+MUSnfpncPPNNXH3vLCKFjr0BDoOUf6E1w/IaYMshpuIjOu6oiAicl6leVC93a3HahJiOndDD3q3FL1vJrYbLUX8wWEW0HRXlnebf1tQ/Gzr9E6pydbQ8vTIOZ6nJmsy00or9gXpdPIPUf7JWqqF8eEHrXw803+0nttITPgZTZrgrV+ic02DOTEiDS8E7rNIszBM9gqoRmRZvEj8BkRF2lHGU33jKppuAyh8DJRxwqpPvoyEqiqLw/iUa1C/xEJFuLKaaVULQWxFjqWMWtqNQMN2THkBX5psRoCrJ03+HwfAjiynB9e68BmfFyorfCMQEWPTQKalD4ZxU6tLBZoalbn3SCAjGSR0CBEr/jgr953lw5yD6q+0rXoHYEat8UYBTvg5cpII+j+kyVhik+Irhp8pcEpuFnGBtzPm9F6AKxwSrg6/9w54A4DtkxS3oSBwib7GiDLYsTY07xFGBrIE4LxwQxTv1EyrdEg4rG3orNY9SxLKdPz4I6SsYhzhICJaoUHJrpgjdUzcpdNQAzqLevGIcRHHRP/XRVqVwSfFstJa1k7vnIpDzEDL8y/Cevmh6YDPtD0C2spWoB97USIHYfXPLPUjliWjGRwLZzbfRN6Z624Gywtbq4tT7RbRcA83CF5LeF8oCyStLQ0WYDNOfzwz8YOnV2HrIcpSUhsBeyP95a0EhhAFGtMYyRo04qaeV4nnr4LphLsdyLG9r/2CZJi4nOZ242F6p5ZUkLmqwXywsq3b2fInw07xowjC8+iJ+CqW2xLpXE7dhVvxsMzH+x+viTSjzlGKj4/zEW0vD6D4hNqP7Vb+Wg04Wc/iD6RRcLaowOO62NdF2mv7IsuXDgTn3sviQD8F0hIpJeB 3uqLC5jt uAIyF4I8MvFJFurs8vV8o3oBjCdCbfbsiFQZokCqSg2cnUbHYUzeHCsxfoWonSMtu3ydtqUdr8qmdzNOLIne15QxLCS1bqXuMEAv06xlyiRNf+Dtk8t2ih1tYIXbjfTq0Ehg8wps3Mi3jcABTfDuo3UsvyWmWxrKKh7KExiZP5fzYxGzyphWfG9cGqrI39jYCKptLEjC3bGEtpf0Er9eivC+2t2afh40zPGUNt56d1GzyEGopKbpi4hx5YoQNn5a/tMHOgD6PcFCLwZ8NZ9cNCv6toT7CVbNu2iQlV6W1tEyn1uzl7eDlsY3vj8zQlWv2RHBdpVc9o9icBt9XcbHsFETQLSiX0Lve+M2kt7eg4jw+Y3rZHIf0+NjbwGT6jLHYlQe1dPpcBf/JDzll8EMSh8SaAPNBwTzvQH1ghr5RyXP6CNcyLkLRT6FJTZ2WyveVAnysWO8sbBZV8cFS/3lf9lT1NEIH+zkxfOA82JP6+HD3dXsjqDCAywOojGdPlJoUYnqegowhvCurF4OeHBwppsQhE0D3N29taSOYjBLl7JmpShk/RFz5FxzRpmcg3N8W+klmMHAZ7JY0fGzOzDNfAMDgYzkIErNbDulYS+7huBoBoU8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Change the documentation to reflect that vm_lock is integrated into vma. Document newly introduced vma_start_read_locked{_nested} functions. Signed-off-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes --- Documentation/mm/process_addrs.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/Documentation/mm/process_addrs.rst b/Documentation/mm/process_addrs.rst index 1bf7ad010fc0..a18450b6496d 100644 --- a/Documentation/mm/process_addrs.rst +++ b/Documentation/mm/process_addrs.rst @@ -686,7 +686,11 @@ calls :c:func:`!rcu_read_lock` to ensure that the VMA is looked up in an RCU critical section, then attempts to VMA lock it via :c:func:`!vma_start_read`, before releasing the RCU lock via :c:func:`!rcu_read_unlock`. -VMA read locks hold the read lock on the :c:member:`!vma->vm_lock` semaphore for +In cases when the user already holds mmap read lock, :c:func:`!vma_start_read_locked` +and :c:func:`!vma_start_read_locked_nested` can be used. These functions always +succeed in acquiring VMA read lock. + +VMA read locks hold the read lock on the :c:member:`!vma.vm_lock` semaphore for their duration and the caller of :c:func:`!lock_vma_under_rcu` must release it via :c:func:`!vma_end_read`. @@ -750,7 +754,7 @@ keep VMAs locked across entirely separate write operations. It also maintains correct lock ordering. Each time a VMA read lock is acquired, we acquire a read lock on the -:c:member:`!vma->vm_lock` read/write semaphore and hold it, while checking that +:c:member:`!vma.vm_lock` read/write semaphore and hold it, while checking that the sequence count of the VMA does not match that of the mm. If it does, the read lock fails. If it does not, we hold the lock, excluding @@ -760,7 +764,7 @@ Importantly, maple tree operations performed in :c:func:`!lock_vma_under_rcu` are also RCU safe, so the whole read lock operation is guaranteed to function correctly. -On the write side, we acquire a write lock on the :c:member:`!vma->vm_lock` +On the write side, we acquire a write lock on the :c:member:`!vma.vm_lock` read/write semaphore, before setting the VMA's sequence number under this lock, also simultaneously holding the mmap write lock.