From patchwork Thu Feb 16 05:17:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C6F9C636D7 for ; Thu, 16 Feb 2023 05:18:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D7A956B0073; Thu, 16 Feb 2023 00:18:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CDB1C6B0074; Thu, 16 Feb 2023 00:18:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B555B6B00A1; Thu, 16 Feb 2023 00:18:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9D4F16B0073 for ; Thu, 16 Feb 2023 00:18:55 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8310EC087E for ; Thu, 16 Feb 2023 05:18:55 +0000 (UTC) X-FDA: 80472000630.15.02D60BA Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf08.hostedemail.com (Postfix) with ESMTP id C89F916000D for ; Thu, 16 Feb 2023 05:18:53 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=e8FcGNDH; spf=pass (imf08.hostedemail.com: domain of 3vLztYwYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3vLztYwYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524733; a=rsa-sha256; cv=none; b=m/qBxgvYFlopof28LAKubdxa2U2Z/ymVldGY8D1sfTwWjDZ2VtYYYAdwOFw+zqkqQoIHOC gcEUR2YScPCit8urHWp09ce99TgZt+fC2eO1SmgE+1jVnPUeHa3XhHcF1CAZUnqg7PmG6w e0L6bYszuj0Y0ZGhlVv/lxQbgTc6pV8= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=e8FcGNDH; spf=pass (imf08.hostedemail.com: domain of 3vLztYwYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3vLztYwYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524733; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=u26HcyQZUssNB/wvgmnc8FNLt5Bjt6C27EcCpLSHQ2I=; b=vFKYEZOxT5AeKFSe3MCqHum1TJ52GhTSkRUGsY8soMeWrg8N+w0UHcFyeIU1/5KWsN+6Mm i+DVLG/QG00ewYVQn/pvgh3f3BhaKYyv5KASerrM/P5ZitG26JMgh5pBEPlxk0esYEEWWm GDul6OBvNY4CDOxTffkw06slqYxbZhU= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5337e2cbc58so8652887b3.10 for ; Wed, 15 Feb 2023 21:18:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=u26HcyQZUssNB/wvgmnc8FNLt5Bjt6C27EcCpLSHQ2I=; b=e8FcGNDH5VkVzBTYRblbtAQ0+ZP2ugsPFuxmDPyoQXwVKDnKFUyAPJn/SJMrsaQ1Ia jo+Cd4OdGYV2adIa7Ps1WdQQvf3M0XkBbQCS2vH4hvx3mR4XIU6AeYNJ4fgFw+1soX5Z Pa1xM0MUFfbg6JS2S95T1SCUKSw+S8anZTC0et5KKeuEZZM6kRmGtxME3D+du+nzcvgG qzTaGZDic4CCJCQNQmsvjPxTw8HSPKOmRJrMlp7St/TdGbLlhdx8Vn9wPFTpNdXttF8Q VA64i9yGakw2MEWw1P/yGjLggzJzYv4qMjifZYLSh4iksjXsYnlNyRAbe89au2hayYpB yYBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=u26HcyQZUssNB/wvgmnc8FNLt5Bjt6C27EcCpLSHQ2I=; b=h9Ygnwyau8o6ta4ky4b+yzWH0W+igRkXvkXM/Vw0oF7OLLYnvAjJL3zz/H1Y1FevFG ZlTZDQu2lB7gFG0GrTYshR/08G/7I8L+EoCM8H4qw6NVlHQpMMpyV6Hr/ew6qfIHaCM4 wGBTqOHt4K+B7aEnsYo9CuW2eOeu+1B2eaLhBerWvFurdL978S3kdw/9mmC8JwNOGndt LiydAMm4yYvJRslfjxfAyc22YEL2/qPbNgKWi6aw/KgP3XkxQNDOgAZCOvGNLGwXNkiv CMBcFMETjrd+8mgzoZKqRI+ipCeHneQlgN/hRkcH+yYdzgCh8egXo16cC3jHCa8n8HsL 6GCQ== X-Gm-Message-State: AO0yUKWjNUUeYOtDDJ6c+FjUMC8TnWe35T3/U6T1m319eEQ5e18Wd9fr CIUzVznBDMQ3mCortD400Wic3Z6cytQ= X-Google-Smtp-Source: AK7set8LRdaZWnTNJXEEL+5OavvLJg+SJ7ELgpuDSzdI0PbEuUZjWdo/eOURR51UCivz1X36gI0P4ir+qmc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:114f:b0:855:fdcb:4467 with SMTP id p15-20020a056902114f00b00855fdcb4467mr31ybu.0.1676524732481; Wed, 15 Feb 2023 21:18:52 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:39 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-25-surenb@google.com> Subject: [PATCH v3 24/35] mm: introduce vma detached flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Queue-Id: C89F916000D X-Rspamd-Server: rspam01 X-Stat-Signature: rxmkjmwcw1mh8fcdohuanf1o4q8uoy6q X-HE-Tag: 1676524733-472738 X-HE-Meta: U2FsdGVkX184Ksct+tMyiKTjk9gUL+nN6sX8c7YIE9uvSDiioQ8+dPFC0ztIq0yXlFgSUYlZlEJfaIvbW3Q7D5L4IsNWX2XU4Qe/jxbphGv8WBn/q96c8nDcuxRJ2Ji7Bm3AeCYTDfMu6ogG5DAaV1sh+6MYXngHDdLSi0HpbRoK4aN6R6tLUpH45m7XuDR33asSkirvuN6ZMY1hUCDjUbDx7c+qthL8UfgvOMcBHvzNNtOg0CiHBUHRJRYCgl8FBtUbFZFsOZ3OmgznDRQ+kmnYaAg+/TBsETcCXLE5TZQiqvyTUeLTvBvwNqU0BfyRXIY8PuO/OhiX+ZVekG7cHQfVTGXudr1SOPvzJQACo+9r8XJRJt05lI4tTW8vv2jZysY3550RSNNrlXXwZ4CqPDqMUKRrh+ka+JmBDdT/hiYC6mqR/JxoYR7jfWEO/vvWoTkH01hapIjJ++HM0fu+dxuL5W7B95tgZr1ydnIcBG2TVp5t2nekk27MCiV0bXnZ+1G9u8NgDPZW/ViWPH4IT8oI3GRd9KvbckHWaBGxMouxxQUzZ0s14dcI/7T/VRua8PHDk9f+HKWuzBvJcpqge4ugBh0uPiGA6YeHBvpkUE0oFtXJ1P6X8Qbm9XwhJR9F0dYAav2ZhUAOAuCP4F/GpLX0BnVhVV2J5DMZs3g0Oj6LsZ8GldNkvh39JOZlUJRq4BlUUvvik9eFRFXQHn/trE+kXaOBgm65ENqEsKKOPBxplf9cTZUItqrHMlQ6kkK5OKbxAIrpFYmbTw5ZyoUqwYW0rhwdPI6nQvMeZfJZL3lYgWPkpj5TsgzJAIgKK+mAnifmgNGzlUlqhNFgm9I+RwXl8xZ8ttjFPHPuIQPt5OQ/WvbzdlzI3CI88sK9gbTzPbkWxY15VeqRiN/ARDt1HSexy0peBcTgwmBrvZ1tzVT643tdfNu5Bci/q8qVUaFfYXT1lqZQnjO39QtTls2 oVPzH7sS Q83ogPx5pSeLV+hkB/l/ArzgfMhsfvphLG4NNVOSHUhqwggpx+BjTd/c9T0Upg5FPKVD5jI+mr+b2NDRlgnPTmLoeu/eHAnK5IiEBqkjx+uMWeTrxNl1VtAPvABsUX0R1Y1OYpVy/+l1En3tbAeBpGkK5HsAr1IsHsFKYGWEpyWXTuyqy3jdBe9lj4Rol3TYN2m79KjxTaqWAlsdkFfZVezCBnn3AeqQiqovQcwBrTdYQNosFFVBt9qXWP8oHNgvRqly0ikqmmzbeLs05i7G6FEMRqBWlSuyKoML5lF3UHPAxfeBiXd4NzO5uayhis+Ln/C+c4cOlo7R32JewtnheWKqwljWuhLKvu1NfV+ftHVMy80417tYI2UMFrUPBzjcfpXaIEZFnJ/PMtAZGM7hbz/lqa3QOEZOx0uEyDBwig0VCGy6TWezYt+v5+/94eqOp9zHihcjrXyoZF2LtSwrAd/sdcyxkpyV+cyYybbSfV2XLKqLEFzvVxIJ0ai41jJ1YIUqZY01ChqJ9ysghawePO/m/rtPSb5ZKtA9mDlKOqyDmI3DgSOCNdK1wpt10PPxS5wqncDnSFhYWWfkGYFz+BjGa+Xr3BJfUOn39EIoyB15TydCe/KQrscsWUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Per-vma locking mechanism will search for VMA under RCU protection and then after locking it, has to ensure it was not removed from the VMA tree after we found it. To make this check efficient, introduce a vma->detached flag to mark VMAs which were removed from the VMA tree. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 11 +++++++++++ include/linux/mm_types.h | 3 +++ mm/mmap.c | 2 ++ 3 files changed, 16 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index f4f702224ec5..3f98344f829c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -693,6 +693,14 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +{ + /* When detaching vma should be write-locked */ + if (detached) + vma_assert_write_locked(vma); + vma->detached = detached; +} + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} @@ -701,6 +709,8 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma, + bool detached) {} #endif /* CONFIG_PER_VMA_LOCK */ @@ -712,6 +722,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_mark_detached(vma, false); vma_init_lock(vma); } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index e268723eaf44..939f4f5a1115 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -511,6 +511,9 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; struct rw_semaphore lock; + + /* Flag to indicate areas detached from the mm->mm_mt tree */ + bool detached; #endif /* diff --git a/mm/mmap.c b/mm/mmap.c index 801608726be8..adf40177e68f 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -593,6 +593,7 @@ static inline void vma_complete(struct vma_prepare *vp, if (vp->remove) { again: + vma_mark_detached(vp->remove, true); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -2267,6 +2268,7 @@ static inline int munmap_sidetree(struct vm_area_struct *vma, if (mas_store_gfp(mas_detach, vma, GFP_KERNEL)) return -ENOMEM; + vma_mark_detached(vma, true); if (vma->vm_flags & VM_LOCKED) vma->vm_mm->locked_vm -= vma_pages(vma);