From patchwork Fri Jan 27 19:41:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119264 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D74DC54EAA for ; Fri, 27 Jan 2023 19:42:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12CD86B009D; Fri, 27 Jan 2023 14:42:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B5376B009E; Fri, 27 Jan 2023 14:42:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E28146B00A0; Fri, 27 Jan 2023 14:42:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CDFD16B009E for ; Fri, 27 Jan 2023 14:42:14 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9F5E0AB7D5 for ; Fri, 27 Jan 2023 19:42:14 +0000 (UTC) X-FDA: 80401600188.06.FF659EC Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf29.hostedemail.com (Postfix) with ESMTP id E4E5A12002C for ; Fri, 27 Jan 2023 19:42:12 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kp92vFUO; spf=pass (imf29.hostedemail.com: domain of 3ESnUYwYKCNgMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3ESnUYwYKCNgMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848532; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5PvsAEkfbXSvdKN+lTJ9M9+vqfLQVBlihpxSKUw/t/s=; b=UBbEa3tIwjL9ZOQe0pgcqyKqlKpi3TpjCV7vjHK5EEDHYL7+zjcXQKAnrOAf4zWdISuU14 JUV9rCl4YfQeXTnxz8doBDN4cjSWlTneukoUvmGP6b4Kj6aoSQXqLM3rDv5zUtDjqwQFn0 YW4Ninz7pl/jmL+DV6QeHNMxoh6yDng= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kp92vFUO; spf=pass (imf29.hostedemail.com: domain of 3ESnUYwYKCNgMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3ESnUYwYKCNgMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848532; a=rsa-sha256; cv=none; b=OvQTFCR+gjERuBSHzH5MkTaw0TnmKZm5f08lCr29a5WzUkqpmIfmYw1Fea0033KFtH9LJ6 1XZFMsB1/nAu3HhZI8sBvjkdZQ7Ui79e0QuhDyW5QzQimNCxSb4F5DQhYWizgcPwsVLc1e 5S760Av2lvSAfl9WFG8ZtJWd3SFjHh4= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-4bdeb1bbeafso65149737b3.4 for ; Fri, 27 Jan 2023 11:42:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5PvsAEkfbXSvdKN+lTJ9M9+vqfLQVBlihpxSKUw/t/s=; b=kp92vFUO2N4rCYUenom2PgB/dU7qqaylHAcH6HVVMYT5QMpL/oqQ/DdOm4bVNlYLNw 9rNSkse+dfgOJKwj09yKdpzV2qc7upA8mjHmEpxt3RKsaa3pDRks1QGh3Nb2hG4bp/jU 3eSArvNsdqtxNb6zpVi6gl2vf0iZr8y+PtaslVXMeV9Pu9xS/EmrXvCiajKJa9iCZe4S LpK/JO6vbmew4JxbSzbZQyffZL4sOt9sBlPWbT9WmPY7+pp+Ff0i+9BT7l4WdUaIrrgA btwhF2XkX8IUP7VpoyJ9OAYm06ddrZftAFhV0gBzofVRmYYEJ8dT0O9gMwk7XMh0b2Ct qw3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5PvsAEkfbXSvdKN+lTJ9M9+vqfLQVBlihpxSKUw/t/s=; b=0GPNSI6XBVIKO1Mymuzldkrdrp/yVyDhB1MLdgcnUEoO4wGBsdI8baV5l3wEP0DZrr lB0sFUGDK6Ot0Z+sG7/uq5dqJsnoEnEwDXvjNaTY0D/Vk2hsKltnPMH8c0yH/pnp8Edb gSM828eaZpA7GjlEqxvfGEorRA5SwKaqq1JT8JMRvQTZiPndqjypbCukIyFBViJFpDns 2kPG47yKSFC7TWcY65Qi1T/6UdMH2MTQmqSmspkVaugeTlwghtX63tjJhk928/DCz2gQ IcpZRM0/BGxIekqhGfswHaMovcx18qKjKkAQMZtK3zl/8eq1e+fe/VDtxS+1ET88aBAf U8Gg== X-Gm-Message-State: AO0yUKXk9shJvk2LPrwbYgf/s8HgaGXmwIQBmw2ZZTK/l169d7pvY/PC IpV63/9IHnh97vj8U/cYmyPLNU2G3ew= X-Google-Smtp-Source: AK7set8zsvT5wkP/7moSYA7dIHSCpHQad2RzXlV5c0Wh0zEZrq5yaOQZsQ7iykW9DnqHdXZz8PLHMfkZLc8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:6d2:0:b0:50e:e91d:f47 with SMTP id 201-20020a8106d2000000b0050ee91d0f47mr27189ywg.61.1674848529047; Fri, 27 Jan 2023 11:42:09 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:00 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-24-surenb@google.com> Subject: [PATCH v2 23/33] mm: introduce lock_vma_under_rcu to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: gw9khnpqb6sq7gt7gn3pixh11g4adut7 X-Rspamd-Queue-Id: E4E5A12002C X-HE-Tag: 1674848532-188511 X-HE-Meta: U2FsdGVkX1+CDrHj03saT6xeeD3Kb66E/+Q27wuigL6bv/8H5xZuccRrLwaDrUx5lXrQPHAuhs76nFQb/ik7SQerDAwbAuar6Yi3mMoqK1UqiN0aQ9JClpNhNZScWNh7QQLF5HiPgSxo63zZcS4C9EX5Db5LEYCDphp3o8ZoEBkwI7v3MNWHQPoEz2OLCyrJWeFkKSOL0RIYTgNQPxZFwABBsV7ba50UA/Hnxm7oWa9Q3Tw0rAvDg9s6AC4fepsBK0O1EeLqOzlnB401UrwTjBWxrKJ1ElRO8d4TinfT/Ck+oZxJSsa22801uUYmVVT71U3pWNfFvgA2dfQZCRi7yjt9nc9l5Gzjn4rtaQzhqEnGvkRBIhrTMKWlUNtUjT0w1y/FN9J/E74/bhuzC9oQhuivDmSUe4pGXb+PcC6BogNWqNDu2jvOsolWPCryjk28A6E6eDpPMKngfv2wQhHB0EeTOOddEmFc/+J+Ol769ncYBwxiSUqoWnK59GloavOQpj0yJq7eCpXtOR+QVIdG+RCyPimNGgktiocXsuV3uBC3sCJJV6X3oq5XfK9gqhWpF3/L2IGWGX1ukylr+GvqTgD5KsCxwj5rtw3Kmlz+VI6/n9ed1BvdtyIFvlzhxsGY1CES6NJBuCERMNjb+oJ+f+xC4QV674XtF85fvKtx5KOR9qZJDPNo3nGm0pAAnOkeVRQrVPrGjGBZKXcXV5qmGoGSCWDVrPW+bVWQt7Z2bkWhS10bxQLuRo+Y+Te0D5rVC+h/jYLiSrjdA1LKjCzCMjmuJZaAV87joRViwB/gIzD+wuDtvEFaAgARnpZ54q/a6zQ9JHr7qINp6vzyLrjxqAzFEyvHnpyDD2WVOkdb3tku/4L+uJsj21BSbM1oBD6D2ZN+It9QiwGXKYyKIRWJr2lohQ5xwjY27FFTjsRljoZksNjiG26Ua5bGZkQjYe18n66uWI20vdQ8vn6SlXv Dy9C3ptQ vlKQAOoqYMmiM2cwMAh6a3OMK3OG8XK+Kz1RDsDCfhasNi59b3PxBX9TDVSi3DAk53S7JaLiQ75eYjVyF0jpLcwclZtmmFEBjDiU5Bu+02NTeZ4yFwVSKahxABwusfmetSshDBjTCDIoRiVocdZ3bY4kzgp9MU3B+qfjCgVgJMDQlxux71lIj/vzTFDbNn5lDQVN+7zXuv5bpXosaVErnRCzR4DW00kpdd3BIMcwGQcCH5ve/A/N9Z0eQoKBEhWlr7UtbLx0Oxo0BOGp5ztCMKmLe4wo3Vl2X2gy2MB2D+PGPugA+kun1OaAo+Vjepfa+Ou3MdtqwnFQto3EcllZyj6JAGy9vlQDUiwUA8Vs9rnz/hkFR0bkXeKgW5xWAyWVuyRkrYFfAnEGZ/TtQp0fFLHNol+BNBg1FTmLh/V8QjdSDyIdlDsEEirXGJSE89a+hlNTXDuJyElBV0gAMAgQMAjZesPpDbOuv29TkoufxYYQaeNju9bLG1qc+CXJxRpVA6kmAL98kJD0z+RYUitY4OphoYLZP/3V32hxJh0bOXdPF5xzkUsKTQJzB4uJzPi/Jmx7xs858noHr3MSIZkVs31eHojlDpyPhctSBWp5aT7URNsaXISoicxoFEv/jsC7APY0ugFtgpD6Q1InP1G8c4KzHc2IPPMCUSlZ1gs/HI5ljMzo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce lock_vma_under_rcu function to lookup and lock a VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. For now only anonymous VMAs can be searched this way. In other cases the function returns NULL. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 66dca140695e..fa2b9d6e665e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -687,6 +687,9 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index d48c76e9fa57..5568fcb0a46b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5222,6 +5222,57 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +/* + * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be + * stable and not isolated. If the VMA is not found or is being modified the + * function returns NULL. + */ +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + MA_STATE(mas, &mm->mm_mt, address, address); + struct vm_area_struct *vma, *validate; + + rcu_read_lock(); + vma = mas_walk(&mas); +retry: + if (!vma) + goto inval; + + /* Only anonymous vmas are supported for now */ + if (!vma_is_anonymous(vma)) + goto inval; + + if (!vma_start_read(vma)) + goto inval; + + /* Check since vm_start/vm_end might change before we lock the VMA */ + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { + vma_end_read(vma); + goto inval; + } + + /* Check if the VMA got isolated after we found it */ + mas.index = address; + validate = mas_walk(&mas); + if (validate != vma) { + vma_end_read(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + /* The area was replaced with another one. */ + vma = validate; + goto retry; + } + + rcu_read_unlock(); + return vma; +inval: + rcu_read_unlock(); + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table.