From patchwork Mon Jul 31 17:12:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13335315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CFDCC001DE for ; Mon, 31 Jul 2023 17:12:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28A71280083; Mon, 31 Jul 2023 13:12:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 23A9428007A; Mon, 31 Jul 2023 13:12:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0DB78280083; Mon, 31 Jul 2023 13:12:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id F042328007A for ; Mon, 31 Jul 2023 13:12:43 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4A05C120BC3 for ; Mon, 31 Jul 2023 17:12:43 +0000 (UTC) X-FDA: 81072551406.11.3DD2A8E Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 5151DC0014 for ; Mon, 31 Jul 2023 17:12:40 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Gui0GRYV; spf=pass (imf22.hostedemail.com: domain of 3h-vHZAYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3h-vHZAYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690823560; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OgBnAtPoNKX+KJs8/FKp+BmtZpXa1bCkw8iPPxm9v3Y=; b=Lt0JQ0l0p7xNhGgvaZ4ytDhHJlwJDEx+1mTdmPN9Heo6ZlrbO455NQqyhpAO6spXF7XnZV eQFyNOYkzh9p8waQiJqKdR6BsolCFPpSHa80nBI6XmulEQ72tKdtAJgUmjyA1TPCn+Oh1I hLCcB/Ebk2lFIz+fskVYne+/WZuRnxI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690823560; a=rsa-sha256; cv=none; b=WBHWOnwvUxOQG9h/uAlkZWMODo7BBktXJfLv5n0btoeeRiLQf3qo+Qr1N+JFFcgRQI+evB T7kXvy2WPokiQxKoh7zbWn6WxVNdIgbsegUbsUvWAiAIuOPIBPkn0q0XqFGid0oNtNO+Wl BXgoVty90qU6T0PY1cYAxfm4nVueiKc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Gui0GRYV; spf=pass (imf22.hostedemail.com: domain of 3h-vHZAYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3h-vHZAYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-d1ef7a6abacso5230974276.2 for ; Mon, 31 Jul 2023 10:12:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690823559; x=1691428359; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OgBnAtPoNKX+KJs8/FKp+BmtZpXa1bCkw8iPPxm9v3Y=; b=Gui0GRYVBMuJeIPTHfyhHWRf3ZvvQ9Lj3V7ipXxNJcDko805MiAWbxKx8mDUarhmuB q/olN0XhD4l8TaoIpr8gDksobmmqELsiDOw3r+0sx6ApTaHKI74YK3XC55P8mWDFy5hP 3P9FBMAEa8e8vsor78FFzDcyQVY4zchDIvDwMMlMLiDHo0y8hhkXv10ZVpjqVjdyAN4b 24xpkaWLkyKWOoC6+J3sDFsPEf08qqJ9mcoY4tSZQR7CAVRPQDvaUPVk6cI7UdE14AbJ cOpcpUcuFLJ9bBngvUSa13BpF0/aqtlVAHc6CDPFkqJFkVTb8czwRnhttfFT6igKBYZ4 1B8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690823559; x=1691428359; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OgBnAtPoNKX+KJs8/FKp+BmtZpXa1bCkw8iPPxm9v3Y=; b=gvk6mIyvKah7Kinof+s4u651DVXvTq4MQDlgiT+gWEs9gXOt/mS24FowfuxhJuFVzb avEUCg6e399eoE0W2CwBo1gtElCLHsaNeryo5gMcy/FUh4exvsIR4OwAcLVPi0qzCGZd cxJVkObHdI3dshYL5Xl//dMArLn8zPKkgDewnTyviJ3vnqhShAOI5bGAmXuMKgm2SGRE W8Wr/ofUjm1AMub0OhppYzZcRn40S4RLucTLY07RHfpRSq0AYyxEVVS2Y3cqHy84DXo1 BGCPVM7VEZYGjUzHoi/cHJJB2Dky9EUYkkHKWLj483l21+9MxNrALO13oF1WAH2DraIq +/7Q== X-Gm-Message-State: ABy/qLYWlG7OcmuQCKwyL9ZkBZ4QvHZow4r6rwVbjheiEGGdH4FkxRm+ Njo0cJpVhPr0q6t0K7N7FMuDUhlpYZI= X-Google-Smtp-Source: APBJJlFnvcCtbJZ73O/WCx8iV93eYXfURurzOhWScTRkBXD7hwc1ky41TRPGayUXZ/7QplAeardfbFpzlyQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:d4d3:7e41:cb80:21ea]) (user=surenb job=sendgmr) by 2002:a25:2055:0:b0:d27:bbcf:44ba with SMTP id g82-20020a252055000000b00d27bbcf44bamr66443ybg.1.1690823559416; Mon, 31 Jul 2023 10:12:39 -0700 (PDT) Date: Mon, 31 Jul 2023 10:12:27 -0700 In-Reply-To: <20230731171233.1098105-1-surenb@google.com> Mime-Version: 1.0 References: <20230731171233.1098105-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230731171233.1098105-2-surenb@google.com> Subject: [PATCH 1/6] mm: enable page walking API to lock vmas during the walk From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, Suren Baghdasaryan , Linus Torvalds X-Stat-Signature: 9jezg9m9ut8714zripbw4j1gg6z5zzb8 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 5151DC0014 X-Rspam-User: X-HE-Tag: 1690823560-669812 X-HE-Meta: U2FsdGVkX1+dGz5X1Tb9p42eHCaCgtMBznzhb8wqlSsyia9nQDS1OLon+j8+Oe1hZdL+MqlaTRdh/XmOmRlbWSNV3VxCn7YaTIE/TH2ipc+FPFp5oEiiGIb51cUoq/TXX7x16bRSa2hHV2OYutsivKM4uYLxd+WWFhnKBo0CjYadf98SUvLXNEa8ARVaXxhabq/j0DizkPFtooN+MCH8q3hptzp3ytRtBi3khS7qA1y6U9jHJUifVKX+QHX9DF/HjWQaaM2DA+ungOV7/F1/E3mIPtzjhCMRY0kBQBy4TQCPEaYyav4z9cma87tlpDj8axfcoBsiKiVjFTslU8UjyaUOVFnmHLuARexTNzyU6j4EbouN8urfSIfJupUPUlnWZa+S9gWywCMokoZCKU46r0vz8UB1kMwBdw8CkzD/bzAq5K4O18reEvfE/UAgxMnrRAsFGU3vuwS9uBWjxjiYYg50fiXIFlCLPlNcbedVFEU3kMMWyhsWiYMDhkE8LyayhMUBp/dELA97NgAs3wyQkzAxqdYL04pWq9QozGd19A2Rw84Yhq1zsd7ouXow/3ayyQMx9Txvh+6PyLvCmImE9vyiit7hBfOzp1t70jr0/afeaqMP2HWiI9/ewkwNko2qZXQpB5ziXIreqbkSePFrmnUX16hO37BZHSxzlBG+VtRwN2BRrOgg2FJ/ypPzyxEu5KfpUu+byVWEZeidQ6RBhbjsp0GLasAvNN5kIaovu3geTAO2OJ0iWN2dVaxYqqtT+AkIWHzcfvhsFN5enoIvdKkok+BJ2ZX0rQ7EEdLEF/pTY/SBBYc9Mj4TPksdmFD9EDGjC+CqneWtLm3aMyqJY1IqzMl4rZomKAeLhch0wHmCYK6xrIiIrxHIX3Kylizz2UcL5gVb+HZ6H8DrpO+Du+KwnU2ctnO6Kp85PF4rCGht/hlJrb74J9qwsgERvLSQ4RbgRp9bZFQEfdoCLYO JkZBgzzT eaEu91xQNzIygAfQPMMsXSo3ErX/bYIoxDEYou4/JNZf33YKbrsKt7RN9F9Ov1dPQBuSAc303iq9RcoGbDnQeRDtxzYhGmJDlX3dvRwWfOhi4fSFReh0IspUYlxzkU/LRxftedzUdHFvVg7MR1w8LPVxzqvgj0sfBu2pQlFal1xhkMF5HD4yYWAdYDeHeTcgx12VKLz4eHB7Kb9stp9NnN/Vu6V+C+ULSfG2wAbz4V8pqOonWt9XjjgtIzW517hVIw70cYzWtrD98n4hMLC7ZLJ7khGcONfLTqinFLWN//pX0E/LjvlYA/tAZJaLRmnqo5pI46UcDyhgT8n41KMSosMGVw/WgsWrt3NiztSdoyT7+5DWSB+5ayciQLbjGMQu+pERlUlOElDSM4zS7JoS3in4Dze9QNNAqRQS+eWqIQWMFqWIy0RmyBbRA5WsHbiZ/6ZmrwGpE4EKEv+q/hHjzp+nve5bp3X+Up7NizxLG+Byrl6wptl+N9meIqDAc5gxAuwVI++sGIc+2pt/QLBkKVNXOMOeCNCaBlsAH2apqdgfjD+Bx+tooKQwu+lkp0qy2GTLVXzpxtu+QqfrLUNd9+f4Iha/QrLcv43ssV+IyUeZyqQ8Xp6nEdydPHV5z6oG7N+meNk4sk3/anHt+IbFZgbKCVAOgR+3FGWyagVvTHDJymwbWNhWXxQ88PBoad9nJ+85VuH4/K5Ux+FCcMKDlMcM5Xy6d2kxWjAdqrF3wLs2SAZNYc8UdpD73r1FzqkLkXBiTH5TfbGKl+tr51z+q/sEEDERgMf2kjSbt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: walk_page_range() and friends often operate under write-locked mmap_lock. With introduction of vma locks, the vmas have to be locked as well during such walks to prevent concurrent page faults in these areas. Add an additional parameter to walk_page_range() functions to indicate the walks which should lock the vmas before operating on them. Cc: stable@vger.kernel.org # 6.4.x Suggested-by: Linus Torvalds Suggested-by: Jann Horn Signed-off-by: Suren Baghdasaryan --- arch/powerpc/mm/book3s64/subpage_prot.c | 2 +- arch/riscv/mm/pageattr.c | 4 ++-- arch/s390/mm/gmap.c | 10 +++++----- fs/proc/task_mmu.c | 10 +++++----- include/linux/pagewalk.h | 6 +++--- mm/damon/vaddr.c | 4 ++-- mm/hmm.c | 2 +- mm/ksm.c | 16 ++++++++-------- mm/madvise.c | 8 ++++---- mm/memcontrol.c | 4 ++-- mm/memory-failure.c | 2 +- mm/mempolicy.c | 12 ++++-------- mm/migrate_device.c | 2 +- mm/mincore.c | 2 +- mm/mlock.c | 2 +- mm/mprotect.c | 2 +- mm/pagewalk.c | 13 ++++++++++--- mm/vmscan.c | 3 ++- 18 files changed, 54 insertions(+), 50 deletions(-) diff --git a/arch/powerpc/mm/book3s64/subpage_prot.c b/arch/powerpc/mm/book3s64/subpage_prot.c index 0dc85556dec5..177e5c646d9c 100644 --- a/arch/powerpc/mm/book3s64/subpage_prot.c +++ b/arch/powerpc/mm/book3s64/subpage_prot.c @@ -159,7 +159,7 @@ static void subpage_mark_vma_nohuge(struct mm_struct *mm, unsigned long addr, */ for_each_vma_range(vmi, vma, addr + len) { vm_flags_set(vma, VM_NOHUGEPAGE); - walk_page_vma(vma, &subpage_walk_ops, NULL); + walk_page_vma(vma, &subpage_walk_ops, true, NULL); } } #else diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c index ea3d61de065b..95207994cbf0 100644 --- a/arch/riscv/mm/pageattr.c +++ b/arch/riscv/mm/pageattr.c @@ -167,7 +167,7 @@ int set_direct_map_invalid_noflush(struct page *page) }; mmap_read_lock(&init_mm); - ret = walk_page_range(&init_mm, start, end, &pageattr_ops, &masks); + ret = walk_page_range(&init_mm, start, end, &pageattr_ops, false, &masks); mmap_read_unlock(&init_mm); return ret; @@ -184,7 +184,7 @@ int set_direct_map_default_noflush(struct page *page) }; mmap_read_lock(&init_mm); - ret = walk_page_range(&init_mm, start, end, &pageattr_ops, &masks); + ret = walk_page_range(&init_mm, start, end, &pageattr_ops, false, &masks); mmap_read_unlock(&init_mm); return ret; diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 9c8af31be970..16a58c860c74 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -2523,7 +2523,7 @@ static inline void thp_split_mm(struct mm_struct *mm) for_each_vma(vmi, vma) { vm_flags_mod(vma, VM_NOHUGEPAGE, VM_HUGEPAGE); - walk_page_vma(vma, &thp_split_walk_ops, NULL); + walk_page_vma(vma, &thp_split_walk_ops, true, NULL); } mm->def_flags |= VM_NOHUGEPAGE; } @@ -2584,7 +2584,7 @@ int s390_enable_sie(void) mm->context.has_pgste = 1; /* split thp mappings and disable thp for future mappings */ thp_split_mm(mm); - walk_page_range(mm, 0, TASK_SIZE, &zap_zero_walk_ops, NULL); + walk_page_range(mm, 0, TASK_SIZE, &zap_zero_walk_ops, true, NULL); mmap_write_unlock(mm); return 0; } @@ -2672,7 +2672,7 @@ int s390_enable_skey(void) mm->context.uses_skeys = 0; goto out_up; } - walk_page_range(mm, 0, TASK_SIZE, &enable_skey_walk_ops, NULL); + walk_page_range(mm, 0, TASK_SIZE, &enable_skey_walk_ops, true, NULL); out_up: mmap_write_unlock(mm); @@ -2697,7 +2697,7 @@ static const struct mm_walk_ops reset_cmma_walk_ops = { void s390_reset_cmma(struct mm_struct *mm) { mmap_write_lock(mm); - walk_page_range(mm, 0, TASK_SIZE, &reset_cmma_walk_ops, NULL); + walk_page_range(mm, 0, TASK_SIZE, &reset_cmma_walk_ops, true, NULL); mmap_write_unlock(mm); } EXPORT_SYMBOL_GPL(s390_reset_cmma); @@ -2771,7 +2771,7 @@ int __s390_uv_destroy_range(struct mm_struct *mm, unsigned long start, while (r > 0) { state.count = 0; mmap_read_lock(mm); - r = walk_page_range(mm, state.next, end, &gather_pages_ops, &state); + r = walk_page_range(mm, state.next, end, &gather_pages_ops, false, &state); mmap_read_unlock(mm); cond_resched(); s390_uv_destroy_pfns(state.count, state.pfns); diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 507cd4e59d07..f0d0f2959f91 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -804,9 +804,9 @@ static void smap_gather_stats(struct vm_area_struct *vma, /* mmap_lock is held in m_start */ if (!start) - walk_page_vma(vma, ops, mss); + walk_page_vma(vma, ops, false, mss); else - walk_page_range(vma->vm_mm, start, vma->vm_end, ops, mss); + walk_page_range(vma->vm_mm, start, vma->vm_end, ops, false, mss); } #define SEQ_PUT_DEC(str, val) \ @@ -1307,7 +1307,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, 0, mm, 0, -1UL); mmu_notifier_invalidate_range_start(&range); } - walk_page_range(mm, 0, -1, &clear_refs_walk_ops, &cp); + walk_page_range(mm, 0, -1, &clear_refs_walk_ops, true, &cp); if (type == CLEAR_REFS_SOFT_DIRTY) { mmu_notifier_invalidate_range_end(&range); flush_tlb_mm(mm); @@ -1720,7 +1720,7 @@ static ssize_t pagemap_read(struct file *file, char __user *buf, ret = mmap_read_lock_killable(mm); if (ret) goto out_free; - ret = walk_page_range(mm, start_vaddr, end, &pagemap_ops, &pm); + ret = walk_page_range(mm, start_vaddr, end, &pagemap_ops, false, &pm); mmap_read_unlock(mm); start_vaddr = end; @@ -1981,7 +1981,7 @@ static int show_numa_map(struct seq_file *m, void *v) seq_puts(m, " huge"); /* mmap_lock is held by m_start */ - walk_page_vma(vma, &show_numa_ops, md); + walk_page_vma(vma, &show_numa_ops, false, md); if (!md->pages) goto out; diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h index 27a6df448ee5..69656ec44049 100644 --- a/include/linux/pagewalk.h +++ b/include/linux/pagewalk.h @@ -105,16 +105,16 @@ struct mm_walk { int walk_page_range(struct mm_struct *mm, unsigned long start, unsigned long end, const struct mm_walk_ops *ops, - void *private); + bool lock_vma, void *private); int walk_page_range_novma(struct mm_struct *mm, unsigned long start, unsigned long end, const struct mm_walk_ops *ops, pgd_t *pgd, void *private); int walk_page_range_vma(struct vm_area_struct *vma, unsigned long start, unsigned long end, const struct mm_walk_ops *ops, - void *private); + bool lock_vma, void *private); int walk_page_vma(struct vm_area_struct *vma, const struct mm_walk_ops *ops, - void *private); + bool lock_vma, void *private); int walk_page_mapping(struct address_space *mapping, pgoff_t first_index, pgoff_t nr, const struct mm_walk_ops *ops, void *private); diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 2fcc9731528a..54f50b1aefe4 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -391,7 +391,7 @@ static const struct mm_walk_ops damon_mkold_ops = { static void damon_va_mkold(struct mm_struct *mm, unsigned long addr) { mmap_read_lock(mm); - walk_page_range(mm, addr, addr + 1, &damon_mkold_ops, NULL); + walk_page_range(mm, addr, addr + 1, &damon_mkold_ops, false, NULL); mmap_read_unlock(mm); } @@ -536,7 +536,7 @@ static bool damon_va_young(struct mm_struct *mm, unsigned long addr, }; mmap_read_lock(mm); - walk_page_range(mm, addr, addr + 1, &damon_young_ops, &arg); + walk_page_range(mm, addr, addr + 1, &damon_young_ops, false, &arg); mmap_read_unlock(mm); return arg.young; } diff --git a/mm/hmm.c b/mm/hmm.c index 855e25e59d8f..f94f5e268e40 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -600,7 +600,7 @@ int hmm_range_fault(struct hmm_range *range) range->notifier_seq)) return -EBUSY; ret = walk_page_range(mm, hmm_vma_walk.last, range->end, - &hmm_walk_ops, &hmm_vma_walk); + &hmm_walk_ops, false, &hmm_vma_walk); /* * When -EBUSY is returned the loop restarts with * hmm_vma_walk.last set to an address that has not been stored diff --git a/mm/ksm.c b/mm/ksm.c index ba266359da55..494a1f3fcb97 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -470,7 +470,7 @@ static const struct mm_walk_ops break_ksm_ops = { * of the process that owns 'vma'. We also do not want to enforce * protection keys here anyway. */ -static int break_ksm(struct vm_area_struct *vma, unsigned long addr) +static int break_ksm(struct vm_area_struct *vma, unsigned long addr, bool lock_vma) { vm_fault_t ret = 0; @@ -479,7 +479,7 @@ static int break_ksm(struct vm_area_struct *vma, unsigned long addr) cond_resched(); ksm_page = walk_page_range_vma(vma, addr, addr + 1, - &break_ksm_ops, NULL); + &break_ksm_ops, lock_vma, NULL); if (WARN_ON_ONCE(ksm_page < 0)) return ksm_page; if (!ksm_page) @@ -565,7 +565,7 @@ static void break_cow(struct ksm_rmap_item *rmap_item) mmap_read_lock(mm); vma = find_mergeable_vma(mm, addr); if (vma) - break_ksm(vma, addr); + break_ksm(vma, addr, false); mmap_read_unlock(mm); } @@ -871,7 +871,7 @@ static void remove_trailing_rmap_items(struct ksm_rmap_item **rmap_list) * in cmp_and_merge_page on one of the rmap_items we would be removing. */ static int unmerge_ksm_pages(struct vm_area_struct *vma, - unsigned long start, unsigned long end) + unsigned long start, unsigned long end, bool lock_vma) { unsigned long addr; int err = 0; @@ -882,7 +882,7 @@ static int unmerge_ksm_pages(struct vm_area_struct *vma, if (signal_pending(current)) err = -ERESTARTSYS; else - err = break_ksm(vma, addr); + err = break_ksm(vma, addr, lock_vma); } return err; } @@ -1029,7 +1029,7 @@ static int unmerge_and_remove_all_rmap_items(void) if (!(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma) continue; err = unmerge_ksm_pages(vma, - vma->vm_start, vma->vm_end); + vma->vm_start, vma->vm_end, false); if (err) goto error; } @@ -2530,7 +2530,7 @@ static int __ksm_del_vma(struct vm_area_struct *vma) return 0; if (vma->anon_vma) { - err = unmerge_ksm_pages(vma, vma->vm_start, vma->vm_end); + err = unmerge_ksm_pages(vma, vma->vm_start, vma->vm_end, true); if (err) return err; } @@ -2668,7 +2668,7 @@ int ksm_madvise(struct vm_area_struct *vma, unsigned long start, return 0; /* just ignore the advice */ if (vma->anon_vma) { - err = unmerge_ksm_pages(vma, start, end); + err = unmerge_ksm_pages(vma, start, end, true); if (err) return err; } diff --git a/mm/madvise.c b/mm/madvise.c index 886f06066622..0e484111a1d2 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -287,7 +287,7 @@ static long madvise_willneed(struct vm_area_struct *vma, *prev = vma; #ifdef CONFIG_SWAP if (!file) { - walk_page_range(vma->vm_mm, start, end, &swapin_walk_ops, vma); + walk_page_range(vma->vm_mm, start, end, &swapin_walk_ops, false, vma); lru_add_drain(); /* Push any new pages onto the LRU now */ return 0; } @@ -546,7 +546,7 @@ static void madvise_cold_page_range(struct mmu_gather *tlb, }; tlb_start_vma(tlb, vma); - walk_page_range(vma->vm_mm, addr, end, &cold_walk_ops, &walk_private); + walk_page_range(vma->vm_mm, addr, end, &cold_walk_ops, false, &walk_private); tlb_end_vma(tlb, vma); } @@ -584,7 +584,7 @@ static void madvise_pageout_page_range(struct mmu_gather *tlb, }; tlb_start_vma(tlb, vma); - walk_page_range(vma->vm_mm, addr, end, &cold_walk_ops, &walk_private); + walk_page_range(vma->vm_mm, addr, end, &cold_walk_ops, false, &walk_private); tlb_end_vma(tlb, vma); } @@ -786,7 +786,7 @@ static int madvise_free_single_vma(struct vm_area_struct *vma, mmu_notifier_invalidate_range_start(&range); tlb_start_vma(&tlb, vma); walk_page_range(vma->vm_mm, range.start, range.end, - &madvise_free_walk_ops, &tlb); + &madvise_free_walk_ops, false, &tlb); tlb_end_vma(&tlb, vma); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e8ca4bdcb03c..76aaadbd4bf9 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6031,7 +6031,7 @@ static unsigned long mem_cgroup_count_precharge(struct mm_struct *mm) unsigned long precharge; mmap_read_lock(mm); - walk_page_range(mm, 0, ULONG_MAX, &precharge_walk_ops, NULL); + walk_page_range(mm, 0, ULONG_MAX, &precharge_walk_ops, false, NULL); mmap_read_unlock(mm); precharge = mc.precharge; @@ -6332,7 +6332,7 @@ static void mem_cgroup_move_charge(void) * When we have consumed all precharges and failed in doing * additional charge, the page walk just aborts. */ - walk_page_range(mc.mm, 0, ULONG_MAX, &charge_walk_ops, NULL); + walk_page_range(mc.mm, 0, ULONG_MAX, &charge_walk_ops, false, NULL); mmap_read_unlock(mc.mm); atomic_dec(&mc.from->moving_account); } diff --git a/mm/memory-failure.c b/mm/memory-failure.c index ece5d481b5ff..763297df9240 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -860,7 +860,7 @@ static int kill_accessing_process(struct task_struct *p, unsigned long pfn, mmap_read_lock(p->mm); ret = walk_page_range(p->mm, 0, TASK_SIZE, &hwp_walk_ops, - (void *)&priv); + false, (void *)&priv); if (ret == 1 && priv.tk.addr) kill_proc(&priv.tk, pfn, flags); else diff --git a/mm/mempolicy.c b/mm/mempolicy.c index c53f8beeb507..70ba53c70700 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -738,7 +738,7 @@ static const struct mm_walk_ops queue_pages_walk_ops = { static int queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, nodemask_t *nodes, unsigned long flags, - struct list_head *pagelist) + struct list_head *pagelist, bool lock_vma) { int err; struct queue_pages qp = { @@ -750,7 +750,7 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, .first = NULL, }; - err = walk_page_range(mm, start, end, &queue_pages_walk_ops, &qp); + err = walk_page_range(mm, start, end, &queue_pages_walk_ops, lock_vma, &qp); if (!qp.first) /* whole range in hole */ @@ -1078,7 +1078,7 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, vma = find_vma(mm, 0); VM_BUG_ON(!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))); queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask, - flags | MPOL_MF_DISCONTIG_OK, &pagelist); + flags | MPOL_MF_DISCONTIG_OK, &pagelist, false); if (!list_empty(&pagelist)) { err = migrate_pages(&pagelist, alloc_migration_target, NULL, @@ -1321,12 +1321,8 @@ static long do_mbind(unsigned long start, unsigned long len, * Lock the VMAs before scanning for pages to migrate, to ensure we don't * miss a concurrently inserted page. */ - vma_iter_init(&vmi, mm, start); - for_each_vma_range(vmi, vma, end) - vma_start_write(vma); - ret = queue_pages_range(mm, start, end, nmask, - flags | MPOL_MF_INVERT, &pagelist); + flags | MPOL_MF_INVERT, &pagelist, true); if (ret < 0) { err = ret; diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 8365158460ed..1bc9937bf1fb 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -304,7 +304,7 @@ static void migrate_vma_collect(struct migrate_vma *migrate) mmu_notifier_invalidate_range_start(&range); walk_page_range(migrate->vma->vm_mm, migrate->start, migrate->end, - &migrate_vma_walk_ops, migrate); + &migrate_vma_walk_ops, false, migrate); mmu_notifier_invalidate_range_end(&range); migrate->end = migrate->start + (migrate->npages << PAGE_SHIFT); diff --git a/mm/mincore.c b/mm/mincore.c index b7f7a516b26c..a06288c6c126 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -198,7 +198,7 @@ static long do_mincore(unsigned long addr, unsigned long pages, unsigned char *v memset(vec, 1, pages); return pages; } - err = walk_page_range(vma->vm_mm, addr, end, &mincore_walk_ops, vec); + err = walk_page_range(vma->vm_mm, addr, end, &mincore_walk_ops, false, vec); if (err < 0) return err; return (end - addr) >> PAGE_SHIFT; diff --git a/mm/mlock.c b/mm/mlock.c index 0a0c996c5c21..3634de0b28e3 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -389,7 +389,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, vm_flags_reset_once(vma, newflags); lru_add_drain(); - walk_page_range(vma->vm_mm, start, end, &mlock_walk_ops, NULL); + walk_page_range(vma->vm_mm, start, end, &mlock_walk_ops, true, NULL); lru_add_drain(); if (newflags & VM_IO) { diff --git a/mm/mprotect.c b/mm/mprotect.c index 6f658d483704..f781f709c39d 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -599,7 +599,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, pgprot_t new_pgprot = vm_get_page_prot(newflags); error = walk_page_range(current->mm, start, end, - &prot_none_walk_ops, &new_pgprot); + &prot_none_walk_ops, true, &new_pgprot); if (error) return error; } diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 2022333805d3..7503885fae75 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -406,6 +406,7 @@ static int __walk_page_range(unsigned long start, unsigned long end, * @start: start address of the virtual address range * @end: end address of the virtual address range * @ops: operation to call during the walk + * @lock_vma write-lock the vma before operating on it * @private: private data for callbacks' usage * * Recursively walk the page table tree of the process represented by @mm @@ -442,7 +443,7 @@ static int __walk_page_range(unsigned long start, unsigned long end, */ int walk_page_range(struct mm_struct *mm, unsigned long start, unsigned long end, const struct mm_walk_ops *ops, - void *private) + bool lock_vma, void *private) { int err = 0; unsigned long next; @@ -474,6 +475,8 @@ int walk_page_range(struct mm_struct *mm, unsigned long start, if (ops->pte_hole) err = ops->pte_hole(start, next, -1, &walk); } else { /* inside vma */ + if (lock_vma) + vma_start_write(vma); walk.vma = vma; next = min(end, vma->vm_end); vma = find_vma(mm, vma->vm_end); @@ -535,7 +538,7 @@ int walk_page_range_novma(struct mm_struct *mm, unsigned long start, int walk_page_range_vma(struct vm_area_struct *vma, unsigned long start, unsigned long end, const struct mm_walk_ops *ops, - void *private) + bool lock_vma, void *private) { struct mm_walk walk = { .ops = ops, @@ -550,11 +553,13 @@ int walk_page_range_vma(struct vm_area_struct *vma, unsigned long start, return -EINVAL; mmap_assert_locked(walk.mm); + if (lock_vma) + vma_start_write(vma); return __walk_page_range(start, end, &walk); } int walk_page_vma(struct vm_area_struct *vma, const struct mm_walk_ops *ops, - void *private) + bool lock_vma, void *private) { struct mm_walk walk = { .ops = ops, @@ -567,6 +572,8 @@ int walk_page_vma(struct vm_area_struct *vma, const struct mm_walk_ops *ops, return -EINVAL; mmap_assert_locked(walk.mm); + if (lock_vma) + vma_start_write(vma); return __walk_page_range(vma->vm_start, vma->vm_end, &walk); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 1080209a568b..d85f86871fd9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4306,7 +4306,8 @@ static void walk_mm(struct lruvec *lruvec, struct mm_struct *mm, struct lru_gen_ /* the caller might be holding the lock for write */ if (mmap_read_trylock(mm)) { - err = walk_page_range(mm, walk->next_addr, ULONG_MAX, &mm_walk_ops, walk); + err = walk_page_range(mm, walk->next_addr, ULONG_MAX, + &mm_walk_ops, false, walk); mmap_read_unlock(mm); } From patchwork Mon Jul 31 17:12:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13335316 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B776EC001DC for ; Mon, 31 Jul 2023 17:12:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0684D280084; Mon, 31 Jul 2023 13:12:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 017F528007A; Mon, 31 Jul 2023 13:12:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D5DD6280084; Mon, 31 Jul 2023 13:12:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C575128007A for ; Mon, 31 Jul 2023 13:12:44 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8B76F16040A for ; Mon, 31 Jul 2023 17:12:44 +0000 (UTC) X-FDA: 81072551448.27.88F72A8 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf02.hostedemail.com (Postfix) with ESMTP id C0D6A80005 for ; Mon, 31 Jul 2023 17:12:42 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=GXelEXrb; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3ievHZAYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ievHZAYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690823562; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9nr+kbatQ6YMh4Ju6KIXOvG29nOfDMrirC5hgMlL3cU=; b=O1Ao7W23ihKCwx+/OBg1xQMpvWB9/szTevdFTsO3c177fLhwSSKaukd8jJGpdo0DcnvieX Ob1B0YcioXZVRZSYQ7EKg9Etr7JlMsp9+IGn9AJud8bYUU8pHT+eY0kbUjWITRnJjo3Z0r gfcaod0UrQBRm6SQIBGLLkuZs8x4j+0= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=GXelEXrb; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3ievHZAYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ievHZAYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690823562; a=rsa-sha256; cv=none; b=HGKuElVnbQ1OjPbCYIDPWt6ICUFyP0wXeQ029HSx3NIECao0DKvBh/T2ZROTCw3m6HP/bn wqEWb2cxwE1iR0Wd3jCBSa8xjyzr14h6xJYdG1zcsLpqDVPyo/YIX81LlKL9GengH2AFd2 EIR10j/3waC2QWdGoEV40C6rKVyhRiQ= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-57320c10635so57013837b3.3 for ; Mon, 31 Jul 2023 10:12:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690823562; x=1691428362; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9nr+kbatQ6YMh4Ju6KIXOvG29nOfDMrirC5hgMlL3cU=; b=GXelEXrbYIY2IAX+kiYu0x1tkCdi2SyGtSF/9w53Zm3gPRECLbctG/dBvEhZfym31B ZpNGuvTs3RdmkliU7juePlylZr5pnrjsO4FCwqrfsr6YZw+CfZKmgNsroN9WWEkA9+ck VJdhxNNh/Cu46QjjSL9nAJzM84Yb17nR6+dWlfcQS/FVxSuAY184c48pG3oyTsWv//zV pwBrsYEyhVF3UcXFqaT0GQRjqCuBIJhqhqjimJS9XP+GadbKmGH7LM8IjHhW/7STICdG EJDRSQ3ltPFB9ycjQw7aPjUMjYM8r7OK4dOG6LXF0kGtaXGnCriJR99uz1fZ6+M5MjcR M39g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690823562; x=1691428362; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9nr+kbatQ6YMh4Ju6KIXOvG29nOfDMrirC5hgMlL3cU=; b=AJxomvfuPoeG76BmnjlVfBY7faJqDdcA5AU5NKND7YRq/lYSQrucYvPVg/fMDyEdmy DpKBrf4wSxLG+TpXaNeVJNgagN8nLZKzKjLGqLZ79plcDrqM269ardN4TwsudAoV8jw3 jKyKhYbWGpeFnms/DGUDk02rmGkFi8znfaq2q//LYO7HyqRmAt77JhVQ7A2lQz1itxC2 iaIg3lKbCiRpUVHncGdf6mcoj/MWjkbpn/mn5oVEBRFE1JG1iSbwFAFP3nMil5sSDtE3 zHWibh/1h1Z3WyEdcBAb0CqamSVA4AmnRZzuiBKF7eUBEFLl/QLLzkZWS3RxwMHXD4mn 69Ag== X-Gm-Message-State: ABy/qLapsnx6zxzYX1UNdJwyldRMpq2JEyQzJmsmIgMJWUxScFbTFClT +OrSvcFcWBqpCdcR//3WB8kMKm8DvI4= X-Google-Smtp-Source: APBJJlHglqY7H7DXFApYFeh47wS6CJVxYTVOVhaQK3HM9XVkVTrRgX4b1iDjG9+19cN8CdS+pNJRAvwhhLw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:d4d3:7e41:cb80:21ea]) (user=surenb job=sendgmr) by 2002:a25:d808:0:b0:d04:faa6:e62b with SMTP id p8-20020a25d808000000b00d04faa6e62bmr57379ybg.6.1690823561762; Mon, 31 Jul 2023 10:12:41 -0700 (PDT) Date: Mon, 31 Jul 2023 10:12:28 -0700 In-Reply-To: <20230731171233.1098105-1-surenb@google.com> Mime-Version: 1.0 References: <20230731171233.1098105-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230731171233.1098105-3-surenb@google.com> Subject: [PATCH 2/6] mm: for !CONFIG_PER_VMA_LOCK equate write lock assertion for vma and mmap From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: C0D6A80005 X-Stat-Signature: 6gx5gnrrbm8we9a1x1u4mmsc6tnkk1j4 X-HE-Tag: 1690823562-368881 X-HE-Meta: U2FsdGVkX1+j/VxQMs9CquWMx//WbqPoTJQZmYuq2m7RhT8QUqjrUpCMQSUPW4FWUrrHfJ+zMwnfD2pcwIg1zCGmNQ0GmJiD3Xm8STsjARv5awGIjvVT+nrKfi7Lhkbf6id+e/A65aiS7MpsFSGPlwGV/fdSb8oCxIUJJDkDx3Ax0FAfV5wmaC4lV40TPZ9FDD8i/TXXU8/Xgh/mgjETimOLMXprbdXRYm4CxCh6f0Se41aWzX/cmgKhv26gJGzEXazuZoNQMRoUchUTxrRdkS1oAVxeqeoNdjcfZtnHbFsGowHCGtGtlexfK+MHsEjc3Cj7I+s1bhoF7heCR/1LHGdNMXr9JSDlHWNeqqs/txg6UX9Gr/8o7cAdItGgD4yptACYSwB22nP3xOQo0WdC2jpWee4bsvk0aOmTQ95g5Dw/o/8LrqickHHd5Dd+JNcgv+psqpSXu+HByV4tJtCoIqr33AWHSIaVy6BeuaPHCI74eKArvVrAR8pKF50MHXkGpPYY3PSLfIUvvR0C9IU/xoaZMQPCsopqUFeJGJpntRTvLijTzZqNdVmPAw0Y5VMoSDXUkcaeki6xEjvD3hgOCteCtZzcd5AZFFBOWohJiA8NcSLIqA9n1CTEFo0Pnrw/hkY3gI3ZLeJpyMQHBae6jGBfYGs24vGOK6oBSreYPuB8mbfm2KQfGVlOZqF5uWhzzIBFFDDQS7xe5s/8LV+xYSf91yEtwrlxiPOaaSouC7eKhkYGDiPblgEZPbk9z0v90DP4NDbdQKHXTv3vL/uC7b/snDdQOEoyEMYVO7Yz9haneJQToYc2yn52sbyEoQ6BtMHMDEWxH4D0j52JY6bsjF54ooC+taVV6h4RXhVEX33DSJAoSwh8Y0hMZSCtzerkIwKg/fcLVtssbJYGTdrmk0HogjxeIketGxXOb+JhJaAnKhtBp4P7CxJQaL3lD8tCMY/Jpvtz5Tr2YXYYqRw V8bLgw/b OZp4YRBw/oSrT1397myoC56wR+mk9I2obP3WzXXHNorrVDwU1JUdEoQ0u5/kE6d3mqKF6ht/Z9tJuML2NYnxZq0FmIT0/u0o300NtGJtNVseGuJoxijj6ILIbNo+NUx63Y8clipyFf+YwYWlgXrAKRo6RNj6f5M02/mtymhF24pwhwgCaOibVEYQ1nrWfEntC5NYAJo7LC4sieo2cWt5sxnm9L1pDUTEVvgMw5bbQL+JPnuX9ubCpO0UScjZlXUSLxDIfLzXKzYQNEk0GRaJlDDBvBa8tkCuxBe0X9+EcWabdgMzbaIxWdIM2+D7dSWPNBZjhKawDWe9uDWfyL8+QYBhnVl3VQ6TiQoE53Tj7DaoyeYEIMBILs4zqtnJvuuf0YjB/Ell+FhEmEvHXClLuHV/tyV38r1E6bVeL/ut7gQSJfJxfsV8ZZGuoA+C9GdAhOO2jijKkJn6gnPSIAIHehdbrr4RajDE9Siyqsjac+CFwUl6yy54gq6sthKJnKLPZ5ughHUlLPwF9T4kfp2xVp/OOsEk2bZXUBYkhjpRBpAUcKaHe74jUpU0OC06W01HBQm9Mgj0YmkbTHmwNgBGVfPg18xFoUDBBAIv9z+3A9wRwglDiRgIC9epAzdRHWEv1L6nJPSiqsr12s0x1HO80pqNBSTEMuVl0oq3bNDncUUtuHFycgp4KUXhQOq4UAvv7gQhJxKmQsi1l5PKhEiNQNs9qgcs6uCzQ5tJq X-Bogosity: Ham, tests=bogofilter, spamicity=0.023001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When CONFIG_PER_VMA_LOCK=n, vma_assert_write_locked() should be equivalent to mmap_assert_write_locked(). Suggested-by: Jann Horn Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 406ab9ea818f..262b5f44101d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -750,7 +750,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline bool vma_try_start_write(struct vm_area_struct *vma) { return true; } -static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma) + { mmap_assert_write_locked(vma->vm_mm); } static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) {} From patchwork Mon Jul 31 17:12:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13335317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B1F2C04A94 for ; Mon, 31 Jul 2023 17:12:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0536280085; Mon, 31 Jul 2023 13:12:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9679D28007A; Mon, 31 Jul 2023 13:12:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B939280085; Mon, 31 Jul 2023 13:12:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6954428007A for ; Mon, 31 Jul 2023 13:12:47 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1B9BB1A020C for ; Mon, 31 Jul 2023 17:12:47 +0000 (UTC) X-FDA: 81072551574.29.B67CCC0 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf21.hostedemail.com (Postfix) with ESMTP id 4E8631C000B for ; Mon, 31 Jul 2023 17:12:45 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=f9Xkv4qD; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3jOvHZAYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3jOvHZAYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690823565; a=rsa-sha256; cv=none; b=0rYoCR59q1Hgm1oRxTWxRPuhEoYogaD1MfMduCW3CkDKnd5nQJFlEbxWDzzidjfEqb2Hpq RMk7xZhPI4n6v7zycIycldlJy1HnLKNkdP1+rLMaD0qf1430Xl8rSi5K+quik6Z45Zvqzo eXrh1HP8cNlSX90MiXUnCFLwdZO5pLs= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=f9Xkv4qD; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3jOvHZAYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3jOvHZAYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690823565; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=k54nr7y8sT8Wz3UIefWJCe/J3ZJoLd0MPdD7SX1yxV4=; b=5IlJfbvLwIcs7ccS/vnc1OwFaGVeEbMmQvzglY2ePtw74nGVdy/ed9jo+jdgGOV/xNiXO5 1CTH1kRGDZzyEGD1cTW3ExJxiixDF6lLQagDzJzxGTx7tUlJnvVUkNQRZ+auS9ysBajg24 ZV7d0WoxEZBUQ1PIJCI9NUMnFfGUTSw= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-c8f360a07a2so3571367276.2 for ; Mon, 31 Jul 2023 10:12:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690823564; x=1691428364; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=k54nr7y8sT8Wz3UIefWJCe/J3ZJoLd0MPdD7SX1yxV4=; b=f9Xkv4qDndAI0YdOZ2cqpuzigjJkWQ867WFypP7OSwCe0TjMpUnPnkD5Y6ryh7F15Y M9Q14b7Q7rdYUncpsFgbIcnXrps3cXu/n5nLbGedhZ45/p7BeS8PbqD3FkLK8kNSVcWe sJK2CotFttBgUzcy2VXRjvFzjM5g8hEPvN+5TMKbT+Soi/X9AdWfs7huzDn319kCX/N6 GMp6dPMzi2Z4psHXbwS83zAFVmyMj0uFSK8waScyRdzBafY4I+1PCGfjiTaar6EOvw8g 4FjFEveEJTGYSjlmO9UfUzptyPCWrdDCvasot5Lp1U7AI3L7Llir4ep5LDCt1MtOBHu3 6mFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690823564; x=1691428364; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=k54nr7y8sT8Wz3UIefWJCe/J3ZJoLd0MPdD7SX1yxV4=; b=OAjaO1yHe4O4G8Tcs7xEWn6ZRMxEelejWX9aB4XAPI0y12OejjkN1+Oo1F+5qkJsOr cjkRb0n6BZB6TWgxskTmOGZRcGNKOg+NK29D0z48dAA7N2vFnIReHbbiAqwXh4agtQPz AQJQuzvnT0xb8QzFhZUTb8E+PHcGrSCj0mLbruVuB3zOcHX+PpuPWvFRpIfanzLfg5MY 4Ncy4NrkexQXfFlaMFVpi6Nal5mRLdkLl80NhfIh8rnjTfLzRND/Tkvlqr/hA4Ycs2Fy Mr4Ph7hNla9lKuFAT8BPK6VmDvP3jljc0GnF3KzL04gzfEuC0r31kyPUYU8GrV+w+LFc EGWA== X-Gm-Message-State: ABy/qLb6/rVo8EPtKxo5cNybSg2itwmcOhpvTB3PaCHJfIRe5COZX8qq AOq9u7grv4McsPL1ZIRODNw+GE8OIMM= X-Google-Smtp-Source: APBJJlFUd/WIRrMRABoKwcapNJpdOFT19YiYMf0HEPYfoNPTpuOYC63k+ANMx6Qm9ThWVSfUuBUUFqFSUIk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:d4d3:7e41:cb80:21ea]) (user=surenb job=sendgmr) by 2002:a05:6902:11ca:b0:d09:b19:fe2c with SMTP id n10-20020a05690211ca00b00d090b19fe2cmr63888ybu.12.1690823564414; Mon, 31 Jul 2023 10:12:44 -0700 (PDT) Date: Mon, 31 Jul 2023 10:12:29 -0700 In-Reply-To: <20230731171233.1098105-1-surenb@google.com> Mime-Version: 1.0 References: <20230731171233.1098105-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230731171233.1098105-4-surenb@google.com> Subject: [PATCH 3/6] mm: replace mmap with vma write lock assertions when operating on a vma From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 4E8631C000B X-Stat-Signature: fns3aha1o553jqoh7prxfmsigwewg8es X-HE-Tag: 1690823565-56096 X-HE-Meta: U2FsdGVkX1+2bhnNIa52AN29YsQsvyo+XXM+Wg7qHBlFaUellx0yESmsJrImVOrRNkce7Ej5QvgHicTYq1n8g+tJMyJJAKTPxK6XYcJv2POahEhMVQ8wQk7KIWGulIQHwiTDiLk7JnvZ/zFBk6Z2QpPfRu8Om9Y/Nw0v6YMSI5yxzrF8EZx59JlUijMML8CuCyQ9Reh9vHAzbx7gUeNrHt0XIuPbvqhh3GO6lQEKHNo3qWCt2O3b3+FXDsrX7CXxaMVjIGC91lEEc5bNfCge4+RARDyGFK8H4cKksPO52sIimzrGG0nkLsT1EGzBiryF3v5DC6ZFGVQUlFqcIunjEUt/qgeblULsuE6qCwGyGAu3Jo8KWpWqVxedwVe9PtaoQzHhH9uITaadB4SdtId7mg/W3agvzo1wY3HRLT1s6FP2WQc53irxxG7O3cEpIIn41U4s0LWcjfpNWTR5lNyT0gsRsifmeW3UZ+sEplEXSpCadDxwqJnhItt2Yubylj/Dy4Ty7BuRMGLeN9Q872j3Twy9VNPtAoY+QQR5pOgo/MiKNn/3nZgG/EMyIjQMpaqOHcLzD2OtqEBUHu88uhzAVlGBvsvtkcUOy8gVO5/0PbaaIBZlQIrKHFFpMpELQJeAYohk4cg21AmvGGNwmmaTU9HCRCfZLwYdlXgmoVFum+MVy6YJplTkYRIaeb6k/TZPFOGi+K8wLeH5tbBlQOViiazlPs3ZrFP3BklH9qNGZ+IRF9LySgo5Nr5DVhcFUSISslwhSPidaQgMZBUia+wcRKcZL7CrTetzG+g8A3LMtl7q8sVZicA6rOfN6VNgkqAZ5OrAaLqvbPkFtUKqPQE1sn2+nOelvCUgU72AQ9olvBQ1xqd7wV/qMvysuAkWdtZ8PdwkmuweNIZ7jshBQ6HkfUibnXbdB7cdvKNbKJpwNoKst1bsA8jedQ9Ed0GyxLxGiycr2/ox+KG3PTL1vm3 l1wJ66+/ wYuNHiDF6S0I9FifxyLZYJ8VKA3DsUfMqMZkyp9En/jrSACn2RMAGDbYeLRHpLRuYAr70+sITZZqsBBPz5pGJ2Ze0JtPxHJ6i6c+95ACJ/UAsDmwUSpmREclTDQn4h1U8MclB3GBu/GpHjs7LN91JR041s5ZHOv1S1eThWDqOzsPlRtePs2zF5iqnm//Qnt5KKGUcBBpY9Sx79hzv7b+k/UTCd/qD+OHo52Efeb1DP1OM7TU4GJdZaSzZxuzy9a8nH5lKxC1pWaxrLoLeVuoY/ycC7nLjo6Voem9M6OT2EVD2hjiq3pFit8lYxd5hs+Xx/Qyi97HNWIq+2eKsU9a7OrpNcsgxR28DrgLTK38z218+VwDJxqOci4r3H5sVC0Ej7MC8b3jVJn+2Xt+rj6ki9785ZhxwS1u3e0y8lpfLf0RIGHIfSmoYBemjI+wp1VFaJgkzD/aexGqfZHsqP3iPXBrjfN6mNcmozPHdNVu/Qwo7hDtuYM8fFA2r6VD9EDSP9I8462tS6Vp52nBJjX1EbdjjPeok4ZRhqrg1H1fr74f7hoCcMU8M66XB8H+CTrkt8J1KraDmgKy88ypT+/klUfPLE+l9DQwNAZz+JOVFr74CIXc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Vma write lock assertion always includes mmap write lock assertion and additional vma lock checks when per-VMA locks are enabled. Replace weaker mmap_assert_write_locked() assertions with stronger vma_assert_write_locked() ones when we are operating on a vma which is expected to be locked. Suggested-by: Jann Horn Signed-off-by: Suren Baghdasaryan --- mm/hugetlb.c | 2 +- mm/khugepaged.c | 5 +++-- mm/memory.c | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 64a3239b6407..1d871a1167d8 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5028,7 +5028,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, src_vma->vm_start, src_vma->vm_end); mmu_notifier_invalidate_range_start(&range); - mmap_assert_write_locked(src); + vma_assert_write_locked(src_vma); raw_write_seqcount_begin(&src->write_protect_seq); } else { /* diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 78c8d5d8b628..1e43a56fba31 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1495,7 +1495,7 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, }; VM_BUG_ON(!PageTransHuge(hpage)); - mmap_assert_write_locked(vma->vm_mm); + vma_assert_write_locked(vma); if (do_set_pmd(&vmf, hpage)) return SCAN_FAIL; @@ -1525,7 +1525,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v pmd_t pmd; struct mmu_notifier_range range; - mmap_assert_write_locked(mm); + vma_assert_write_locked(vma); if (vma->vm_file) lockdep_assert_held_write(&vma->vm_file->f_mapping->i_mmap_rwsem); /* @@ -1570,6 +1570,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, int count = 0, result = SCAN_FAIL; int i; + /* Ensure vma can't change, it will be locked below after checks */ mmap_assert_write_locked(mm); /* Fast check before locking page if already PMD-mapped */ diff --git a/mm/memory.c b/mm/memory.c index 603b2f419948..652d99b9858a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1312,7 +1312,7 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) * Use the raw variant of the seqcount_t write API to avoid * lockdep complaining about preemptibility. */ - mmap_assert_write_locked(src_mm); + vma_assert_write_locked(src_vma); raw_write_seqcount_begin(&src_mm->write_protect_seq); } From patchwork Mon Jul 31 17:12:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13335318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EB58C001DC for ; Mon, 31 Jul 2023 17:12:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 09181280086; Mon, 31 Jul 2023 13:12:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F0C9D28007A; Mon, 31 Jul 2023 13:12:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D1249280086; Mon, 31 Jul 2023 13:12:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id AD93D28007A for ; Mon, 31 Jul 2023 13:12:49 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 79B3240138 for ; Mon, 31 Jul 2023 17:12:49 +0000 (UTC) X-FDA: 81072551658.11.BCCE06A Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf30.hostedemail.com (Postfix) with ESMTP id 8B62480010 for ; Mon, 31 Jul 2023 17:12:47 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=TqEf6Hrd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 3juvHZAYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3juvHZAYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690823567; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pWiy0eqxBgZlb+dN+Yox0ISeOPzNiRGRRyq+SZhtGFE=; b=3BCrronjs+d3t7EOU3rzaZ/wrTeJrNNmLGZ3u7YYlHyHHTicnQdt5YTSy8L5WVY38T5waE 9kDE3VrtkTPJ8ddvbslk+y94teyWhYW/5gupSNWtMQqEq6OQfdkX31eYurrdEjXBhnenRf w1S/xeBSzFWRJQo0z363yXU1Z8mUVlw= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=TqEf6Hrd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 3juvHZAYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3juvHZAYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690823567; a=rsa-sha256; cv=none; b=mmjm5SQGfCbN+RzuYIPUgO7s7jsrQ8g516HD5sVf+5hO6A9RSTP/kwwXhXz56lXEp3k9ro OlVyhqoyDY5iX+0ohXwOtHbh4aqUCli9G5l9GQtycfmQPo9r8PretneMvusf1Sp2DZPxGD SCjXGQxDafbD5XTkfetDqHa7V80IPA4= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-585f254c41aso31858357b3.1 for ; Mon, 31 Jul 2023 10:12:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690823566; x=1691428366; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pWiy0eqxBgZlb+dN+Yox0ISeOPzNiRGRRyq+SZhtGFE=; b=TqEf6HrdE+I2RnvUQAJhHrd/y5HGFhwmubpEUXYwAE4bGwVm81VSt8g5W0j9fPPAuQ sX4JCbn4xB1mj7KL3Fly3i1ij68a6VGGbaiodngH2NBWtY3W2Fduv5VAGXk0Y+AsbLAU 2uykQrOSFF3WK6HbnGND60wrUIQwprYirtJ7g4akiT7JRGvKgviVxDtLgAB37WUQewaT bIDeNPs0lImrqBS8/+XnjxQ4XT/lIUE0hzBuPIL335K48KzLJyX4EWLQ3B19lgcjlmTZ ixFnC3T+Kv/cFjI8V1Y3EgfT6F9hQnaglRNBsbIeI6o7F9OSz2bO7I7d5X/ejesuCCqD 50Gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690823566; x=1691428366; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pWiy0eqxBgZlb+dN+Yox0ISeOPzNiRGRRyq+SZhtGFE=; b=i4v9sarRHjD889+n3nTsnXoYvtF/wEDRsjkTiyh0KABFoNIgZcmD4eX/Srr/4nuB1v Cg0sU0uHq0/tjvfMTWH4FG9ZwByUBLBLsIWHjZv9MoTSkJq21Mg2cVOmnAJr692aWhBm /a4Ch9PocgG/X/wpwWzGbYdpHFAOgrESN6D8IPJvbOXZljS4/lnXQiDr6JhdZPvKixvT O9dUD8ykqn2l7qVhnQiRj1F52sNK+x4wWtH7s0vt49FdkxQGps09ogoBqW91pjUheXyb aMpa7hJgycyf98o8z6FTpkcUDxqHhcPid6j7I2u8m/Bx6XFmJXnja+dR4IB719Z/bfjA iLiA== X-Gm-Message-State: ABy/qLby+6R9fIuK44xEpa+nMyXnSbM4Yx2rjjhV3B7Vb7Dzhp1vzuFy BXeYSB70FYppJblkdbawAD+wTzAo0Zo= X-Google-Smtp-Source: APBJJlHwwvxymWxZDUL8yzdLNqvheNGhA91W/OiK0dav8XIzZNSGI7RuCw/tew0kh4tMP8lXsVeDK4HgzX0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:d4d3:7e41:cb80:21ea]) (user=surenb job=sendgmr) by 2002:a81:b625:0:b0:586:50cf:e13f with SMTP id u37-20020a81b625000000b0058650cfe13fmr9301ywh.1.1690823566545; Mon, 31 Jul 2023 10:12:46 -0700 (PDT) Date: Mon, 31 Jul 2023 10:12:30 -0700 In-Reply-To: <20230731171233.1098105-1-surenb@google.com> Mime-Version: 1.0 References: <20230731171233.1098105-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230731171233.1098105-5-surenb@google.com> Subject: [PATCH 4/6] mm: lock vma explicitly before doing vm_flags_reset and vm_flags_reset_once From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, Suren Baghdasaryan , Linus Torvalds X-Rspamd-Queue-Id: 8B62480010 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 9tmfo4tgxeo7h16jnhtqrxfznw1ymg6g X-HE-Tag: 1690823567-657211 X-HE-Meta: U2FsdGVkX19hb9jYR+CNzVihopR+x6ZqY1x1hzS5/RVrfzzJg5pBt+p+c+WMMPhoftih4T5AILrWvfNhJ38PEA+HOq0aNQxKX8dyWSaQ48q60FgD2Xjo194gkTVwsu9r8g+OTXUYEDNJvwiixNgL7ZFwFQcWHPJlUXegx+FJ9iy7mLqrNPWSLFzxkYoWmjdhf4mRAiur86tXE68oTxyHhoTlUZ9vH36E3o6oHjS3NxH4qDN/b3scATXuBIdcZgRCwyP//vgEFP2qnLh2U0y8EW3RQlaVsBq3h1Juu2lJ86/pYPcPJajRLlybzyIi5f9rfYQqATAIZlKsqBz/SPbKzNgvC0RemVVcJSLTuFkNM5l4WyjEe3l833mXNDA0QG8oT3G9/V64XtDSOZ+qmPgoQinUyUuVuiHjcttqP1zRV/tk5MInFMYGHv8sb3DYLQ1HyV//ahPSczmc0ZaaTROkQgiUfR8OtC1yAIfq9kp7YtWpaM1JDg4C9xIUGUZrCGTBf/g8gTy+7HC3hEGAoSvJ6I5Uoaj9yhGtmCvrbpnJ59IFYL6W7vX25HeWnhf6Nc5oahzB7CYnPek/OM/Hav34TRm4XPOSitIgecjgL2hthRXIpnqmBuD6Lvd8vD2cMPgdtqt7Trnc/KW73KmHUkevp+tHg4Jf88kTesE+vNVnaLO5EXEp+Z73LtfA7h27MEWzJijcTsL5kJACCvXsYEuHkpUmucYWG2uIc794Pw3ICkxr8i8b2sAXSRUK1BAmZLypN7/diDqftOhS775DfvKZiVdB9vDy7kFKfVFUctOeBcUBxLoJYbgHQR3Wvo+2yj2ZfdAoLEeAhVTQygdmGmNdbuNwvfRliIlw1OGrF0+9k7+z03ru1Y1UWh0l20gPoE/uPuUFCMqhFiDyjs4kw32wADTjDQWmINwZxq4sFc+ai+K2Qqeqig0iM/R34oiUwaAO2JPkIXpo9MIkMbPYhDZ otSY/ePL m8dqlZ8cdM2sag+scEswjnkNP3duOsHdwP4K7lx0IL9vOFVMYDBb3i2Iyhc2Emotw81FMnk4HPbPs/JqNWdbxYJoGWmLi/kQUfVweaYWsSk1Bzn6zQXcGa5uL010igNL/ice16dxEL4qqEcK63ohI4R1KgzYnHrAF53rBohFO3qCkxY8rl1SVNROlhCfKdFaX4YXP1+TNdSVAfgtCbDI9eFVW+san6perfps1IKnhdL14fWhLxyM4i/h27KacCqPgtfMI1IsMYmtiPVE/PQ92c+et1C8nx1xxmvCCdJUEMlwDaZxfogn6/Z9DJEG1PU6oepPh9XcjROG3APC/jj87ihQ/zVIAKq/p/s0JWfk5RMCHswvdfd+yVqoe8VjoSKwxEvo4e8dzVXNC4jbqr0X65cyLTFwJwde06+0Vhe+pR9A1OJQFHFCHdCXbianEbfYl53AJ6J4/JrkPssc/INBt1DfTdR9uRdyso0AZN11tn6lCLhCUFx/OmYUTC7pFRabx4mdpCtu9Kq0Bna5Zizk8bpmT1+zBEvhs5sIS2ERTgX5UM+IN1MN911qVFZ2WhpgFHpU25Q7P84ZF+80f0gZ9MX/fwitsOy53P25R7WsdK6yUNVmjUbq5MI5kLW9Qm5VPKa1Wazd/n2p3Zw0twBgtBT8cglVaKa23cLNUAPETLwhuqFzuMNYcwAALHEuU1cspLjOANtf4MEBQhV38sK94lZ4xuw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Implicit vma locking inside vm_flags_reset() and vm_flags_reset_once() is not obvious and makes it hard to understand where vma locking is happening. Also in some cases (like in dup_userfaultfd()) vma should be locked earlier than vma_flags modification. To make locking more visible, change these functions to assert that the vma write lock is taken and explicitly lock the vma beforehand. Fix userfaultfd functions which should lock the vma earlier. Suggested-by: Linus Torvalds Signed-off-by: Suren Baghdasaryan --- arch/powerpc/kvm/book3s_hv_uvmem.c | 1 + drivers/infiniband/hw/hfi1/file_ops.c | 1 + fs/userfaultfd.c | 6 ++++++ include/linux/mm.h | 10 +++++++--- mm/madvise.c | 5 ++--- mm/mlock.c | 3 ++- mm/mprotect.c | 1 + 7 files changed, 20 insertions(+), 7 deletions(-) diff --git a/arch/powerpc/kvm/book3s_hv_uvmem.c b/arch/powerpc/kvm/book3s_hv_uvmem.c index 709ebd578394..e2d6f9327f77 100644 --- a/arch/powerpc/kvm/book3s_hv_uvmem.c +++ b/arch/powerpc/kvm/book3s_hv_uvmem.c @@ -410,6 +410,7 @@ static int kvmppc_memslot_page_merge(struct kvm *kvm, ret = H_STATE; break; } + vma_start_write(vma); /* Copy vm_flags to avoid partial modifications in ksm_madvise */ vm_flags = vma->vm_flags; ret = ksm_madvise(vma, vma->vm_start, vma->vm_end, diff --git a/drivers/infiniband/hw/hfi1/file_ops.c b/drivers/infiniband/hw/hfi1/file_ops.c index a5ab22cedd41..5920bfc1e1c5 100644 --- a/drivers/infiniband/hw/hfi1/file_ops.c +++ b/drivers/infiniband/hw/hfi1/file_ops.c @@ -344,6 +344,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) goto done; } + vma_start_write(vma); /* * vm_pgoff is used as a buffer selector cookie. Always mmap from * the beginning. diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 7cecd49e078b..6cde95533dcd 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -667,6 +667,7 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx, mmap_write_lock(mm); for_each_vma(vmi, vma) { if (vma->vm_userfaultfd_ctx.ctx == release_new_ctx) { + vma_start_write(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; userfaultfd_set_vm_flags(vma, vma->vm_flags & ~__VM_UFFD_FLAGS); @@ -702,6 +703,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs) octx = vma->vm_userfaultfd_ctx.ctx; if (!octx || !(octx->features & UFFD_FEATURE_EVENT_FORK)) { + vma_start_write(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; userfaultfd_set_vm_flags(vma, vma->vm_flags & ~__VM_UFFD_FLAGS); return 0; @@ -783,6 +785,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma, atomic_inc(&ctx->mmap_changing); } else { /* Drop uffd context if remap feature not enabled */ + vma_start_write(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; userfaultfd_set_vm_flags(vma, vma->vm_flags & ~__VM_UFFD_FLAGS); } @@ -940,6 +943,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) prev = vma; } + vma_start_write(vma); userfaultfd_set_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; } @@ -1502,6 +1506,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_start_write(vma); userfaultfd_set_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx.ctx = ctx; @@ -1685,6 +1690,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_start_write(vma); userfaultfd_set_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; diff --git a/include/linux/mm.h b/include/linux/mm.h index 262b5f44101d..2c720c9bb1ae 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -780,18 +780,22 @@ static inline void vm_flags_init(struct vm_area_struct *vma, ACCESS_PRIVATE(vma, __vm_flags) = flags; } -/* Use when VMA is part of the VMA tree and modifications need coordination */ +/* + * Use when VMA is part of the VMA tree and modifications need coordination + * Note: vm_flags_reset and vm_flags_reset_once do not lock the vma and + * it should be locked explicitly beforehand. + */ static inline void vm_flags_reset(struct vm_area_struct *vma, vm_flags_t flags) { - vma_start_write(vma); + vma_assert_write_locked(vma); vm_flags_init(vma, flags); } static inline void vm_flags_reset_once(struct vm_area_struct *vma, vm_flags_t flags) { - vma_start_write(vma); + vma_assert_write_locked(vma); WRITE_ONCE(ACCESS_PRIVATE(vma, __vm_flags), flags); } diff --git a/mm/madvise.c b/mm/madvise.c index 0e484111a1d2..54628f4ca217 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -173,9 +173,8 @@ static int madvise_update_vma(struct vm_area_struct *vma, } success: - /* - * vm_flags is protected by the mmap_lock held in write mode. - */ + /* vm_flags is protected by the mmap_lock held in write mode. */ + vma_start_write(vma); vm_flags_reset(vma, new_flags); if (!vma->vm_file || vma_is_anon_shmem(vma)) { error = replace_anon_vma_name(vma, anon_name); diff --git a/mm/mlock.c b/mm/mlock.c index 3634de0b28e3..f0f5125188ba 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -386,6 +386,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, */ if (newflags & VM_LOCKED) newflags |= VM_IO; + vma_start_write(vma); vm_flags_reset_once(vma, newflags); lru_add_drain(); @@ -460,9 +461,9 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, * It's okay if try_to_unmap_one unmaps a page just after we * set VM_LOCKED, populate_vma_page_range will bring it back. */ - if ((newflags & VM_LOCKED) && (oldflags & VM_LOCKED)) { /* No work to do, and mlocking twice would be wrong */ + vma_start_write(vma); vm_flags_reset(vma, newflags); } else { mlock_vma_pages_range(vma, start, end, newflags); diff --git a/mm/mprotect.c b/mm/mprotect.c index f781f709c39d..0eab019914db 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -656,6 +656,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, * vm_flags and vm_page_prot are protected by the mmap_lock * held in write mode. */ + vma_start_write(vma); vm_flags_reset(vma, newflags); if (vma_wants_manual_pte_write_upgrade(vma)) mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE; From patchwork Mon Jul 31 17:12:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13335319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98AA9C001E0 for ; Mon, 31 Jul 2023 17:12:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 374B2280087; Mon, 31 Jul 2023 13:12:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2FEE728007A; Mon, 31 Jul 2023 13:12:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F33AF280087; Mon, 31 Jul 2023 13:12:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id CBB1328007A for ; Mon, 31 Jul 2023 13:12:51 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 7C82E4035E for ; Mon, 31 Jul 2023 17:12:51 +0000 (UTC) X-FDA: 81072551742.12.749F07B Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf30.hostedemail.com (Postfix) with ESMTP id A79EF80016 for ; Mon, 31 Jul 2023 17:12:49 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=LgDlXg7e; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 3kOvHZAYKCLMlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3kOvHZAYKCLMlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690823569; a=rsa-sha256; cv=none; b=vYnp83LYIRN6LkWRNKff3FJscQwZsHiiBBiFrwHNVXHtrjMRe8s8VqUBjpWYX7YZ8pNFxm HbGbw9is5H5lEj76QsT1k5H6G5zUgm0i3gYjtrUSPao7EU66u109p/3EydP1wgNRwpCj4o +vDAhbfrsyfJmgPldfgsLCIs5J3yNI0= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=LgDlXg7e; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 3kOvHZAYKCLMlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3kOvHZAYKCLMlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690823569; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fl37fA/fktQ3urfS+YwWUE+qcRJpc6Gt7zobpPnOyFQ=; b=1QiTFnebHfgbX4jPUri5+vellgPhDKGBXqrgmBewJ6r9GlGyOfx2XwP6GQVNjMVBPqH/8x x7MgDu/UGMN9LZkmgoMzAUUWsCML6P484hFxsUlgR0pObuthwk10sAgaP4EryHM34VTDH4 rmDY5+ruLQGqWRrjfgxMXpuKyvmJI2w= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-c647150c254so8699964276.1 for ; Mon, 31 Jul 2023 10:12:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690823569; x=1691428369; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=fl37fA/fktQ3urfS+YwWUE+qcRJpc6Gt7zobpPnOyFQ=; b=LgDlXg7e+opMqwWMES1LLqVmtZkaKcma+JDWWp+e4CicAgmkjcvE9XhQBziMq2eTa3 PXWdlSTMfbarj3wXfNgyENr1RvdnAJxrQvtE9y8G7VjJ080QaAAJjzFwuwEaNmQ0DHyN a1+nQsa+bGNOM4jP3sCAuVCP01EJzsRkj0hXximLEb2Mfn0aWk9hMuhEtMt7RzQ2IfWq dTRWuwzCf9427UAvrXN1bePAw/m+zuKvf0/dCjndjuyKyBwMXDIzsawqi0xrdLmdKZLv FQU527Nzu03LfekDUrASNgZQizxrizQv19ZDdXWFCDoALbzcJ8zvPoznDlPtXbvB3a/3 h6XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690823569; x=1691428369; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fl37fA/fktQ3urfS+YwWUE+qcRJpc6Gt7zobpPnOyFQ=; b=LS6sOxbDRKkW/0FDE8SCT1SvKibEgibz/M+TVYdN/QRURgmx3ePB+CrX5cf0y8Ak5H 55GHLbTctcMXRjvjQLx8Eb+ZkIUH9Tt2G6+6F+GO+KS8bAkxF1eiHp/L0My+IZVrxR6g HRWK7uu/4+9g+HVuL5eN7mtxLQOwxPXCF5yQB+CVOosNQiUORoSUFCYlcIhlEspG/q0J qkUHu4ZBuMO0rj+e69XiUJ877Gk4NI8p0ig1BoChUqLgZAEOlC7QEXoS+0zlxtNcQyLP rq7yWBjPkn5s1Trm+RNccbhvkgO42bXg0Ik2FoNuH6YeW4ubZagtC7pkrkL9eVmKgBwI nh+A== X-Gm-Message-State: ABy/qLa7MFN0wiTjsYybVgQj2KlHy3pbEAdEMxkJeVX1mMwj953f/uLl eMNPI/j3ergczAQOHXOaLnLT7TIH7L0= X-Google-Smtp-Source: APBJJlHhlXYyfYo/Ka1CDxxExi2V2guG40bRgvHWyuWMJtZm8ytlVRtn32TdlCPCKVgUKlv/bdcqKCA4SCM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:d4d3:7e41:cb80:21ea]) (user=surenb job=sendgmr) by 2002:a25:e658:0:b0:d1c:57aa:d267 with SMTP id d85-20020a25e658000000b00d1c57aad267mr113311ybh.5.1690823568750; Mon, 31 Jul 2023 10:12:48 -0700 (PDT) Date: Mon, 31 Jul 2023 10:12:31 -0700 In-Reply-To: <20230731171233.1098105-1-surenb@google.com> Mime-Version: 1.0 References: <20230731171233.1098105-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230731171233.1098105-6-surenb@google.com> Subject: [PATCH 5/6] mm: always lock new vma before inserting into vma tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A79EF80016 X-Stat-Signature: yg5xmmr14znnugttp5fd7sei3eu4btsc X-HE-Tag: 1690823569-451437 X-HE-Meta: U2FsdGVkX19LfVok5zdvJb3XOq/IT9VVXOtklYAg22h/Z4gIwxFZVMH/WNEGjVZVjtnauKGlpsY1p8LnCs6IrdO5WGKQBEqDK4Ng+e9SbKw/HSOtGcwKIUssAJRlJAYanuehN+hhjwXXSGShdru1//fASHFiXagNxHjX8grpOnXRTx1AYa9DOYA/vXMpr3IC7aXUkY4UcNgHq2NosemTcfT9Z+eiRnkr9xktVRSPK2rPFvDEx5nvSHiUd2SPIpj+LjLL3mErNqIKeyHAsbmBjuauwGVMzxJdNGimkC0AIMOe2VUp2QDGswYyxPmGZJwUiZbAfY8U+jVf9qU67us19CEuvfXZXctxL0DaIfAKhkg3heX7DfNGAfCEMiJO4GJbrEHXOiWk/hEcZu7eWzAVd8CkiQKRiUV0xMAHhcPUyitZ6KdC63KUo/yqSvsaVd9vc1KCPeSKC5FRMJjbO/9/pmYuWCK96dlxiqHkCYfmU/tOMQTseltnqaRG8WVaUu8EVX+3thbCSbhwi/czfTf/2gZqFxMxfzg2oDic0d4zaoe9J7C75TNmVibumYrZ/GbJtocXpn8Orj1nx01mkVThPSzvmRlsDjZ6fIExFrrzaGvSKEMe1MrMCfScQTXOufdMVoBLfLl02DoEOMLzrbn1FfYVaRPXm/RSsaKkbxbzvAX/iHFMI1Kz2OPyGyDADwGhHujN3wvDdIPmUTAUwiLnCn+g3vaUSF/rBCb9lzYQn0WlniKu2WZnKhYcCpPkw7cNrvvakKRI5IqX7vhuH8lI+DQWLhh764pLWOf9zB++zFz0dU3RoXiKQ2c7mYubGuy8w3xVbPOlH5T/kekOXFBwBFrExoRYnw2hhYTG8cTi/dDHRT4yRWbuTzgNGafzTf8MiZRhuTAv03oJil10FZSm2inbtN4lPzOv+3N0qN3A8TZY6YuXtl3aSoTr1ryY7KFMIFBfPSs1DlneB2Wp0ur 3a5FPgWQ wm4d5WD/TwLIUd1XsOTOwPHPqVETKnHurd2Rl2QDEB/39CQeX8SVJ/nRYOg1PjhGyjDQu2XvOL3z0lsxKB4ZIqzvbGqxqSiWHO1dKR8I+vQhqzs+F+iTvDNrZxH6+Qsyf+jCRYqdmlTZBwYlao6g7il9mDTqvrLUmLdHbd69O3V09CQgtzWQUXHFqOBErDGZx852O/ojrawGg3aPu2+BNLA52auI8pIE5CDaD9rt5U/hRi5js8y70SRIPMgLjFAi9ay1EX7gUVxHmFL4zl2isMeVdIwq0hkN/Um7jFgokpIXvcdfNelBbSF/ljhq4sTncqJ9mUxducdgayG1kJDVBKJI9A9ZlZyu13R4b1WpMlViPP4p6uPr5wciiEV+8r5JrLpT4ZbqzLf3S4vuP894OBTf5UVVqIHkWslDQCGtaGkbND0BAWINR4Hu90SqKKy0sQmT9Tj1XWQeYibsBpEp61f+mjGgD1kFs991tXDsfxCAcHTntrUBT0gxM6jthIVKY7EyT4B5yq1FwoTqiF75nchiTTcM2JNeOFJxudE+pYHdYImxNrJ8dL4H+aWF7ZlXSBJA24CZ6su5z2aBzHvgzWYN7QxV7MNrB/vLudb7GXZ2T7ok= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: While it's not strictly necessary to lock a newly created vma before adding it into the vma tree (as long as no further changes are performed to it), it seems like a good policy to lock it and prevent accidental changes after it becomes visible to the page faults. Lock the vma before adding it into the vma tree. Suggested-by: Jann Horn Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 3937479d0e07..850a39dee075 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -412,6 +412,8 @@ static int vma_link(struct mm_struct *mm, struct vm_area_struct *vma) if (vma_iter_prealloc(&vmi)) return -ENOMEM; + vma_start_write(vma); + if (vma->vm_file) { mapping = vma->vm_file->f_mapping; i_mmap_lock_write(mapping); @@ -477,7 +479,8 @@ static inline void vma_prepare(struct vma_prepare *vp) vma_start_write(vp->vma); if (vp->adj_next) vma_start_write(vp->adj_next); - /* vp->insert is always a newly created VMA, no need for locking */ + if (vp->insert) + vma_start_write(vp->insert); if (vp->remove) vma_start_write(vp->remove); if (vp->remove2) @@ -3098,6 +3101,7 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, vma->vm_pgoff = addr >> PAGE_SHIFT; vm_flags_init(vma, flags); vma->vm_page_prot = vm_get_page_prot(flags); + vma_start_write(vma); if (vma_iter_store_gfp(vmi, vma, GFP_KERNEL)) goto mas_store_fail; @@ -3345,7 +3349,6 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); - vma_start_write(new_vma); if (vma_link(mm, new_vma)) goto out_vma_link; *need_rmap_locks = false; From patchwork Mon Jul 31 17:12:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13335320 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCC88C001DC for ; Mon, 31 Jul 2023 17:12:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 91B47280089; Mon, 31 Jul 2023 13:12:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 82EA228007A; Mon, 31 Jul 2023 13:12:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A8EF280089; Mon, 31 Jul 2023 13:12:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 598E528007A for ; Mon, 31 Jul 2023 13:12:54 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 03F41C0358 for ; Mon, 31 Jul 2023 17:12:53 +0000 (UTC) X-FDA: 81072551868.20.B1F7ACE Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf28.hostedemail.com (Postfix) with ESMTP id 2BC03C0011 for ; Mon, 31 Jul 2023 17:12:51 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=sHGO4YkH; spf=pass (imf28.hostedemail.com: domain of 3k-vHZAYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3k-vHZAYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690823572; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nlUlu5Mhd5cq+/dnIWytp1wDOwye9pYqWME5wT1VG2E=; b=qDccPhmLA/CSOI890Jp27fIpiQMFgJ4cBmgk0IoBvixsRsNS8J1u7/+lEtMwDfuy2oJXdt gmnK5ZJwaFbxKIG5fvpEmd1srfDisAsG5Z0FGhGMq7q74LSDi4ayb9KjRcRs2rOIrxKHKf 5H174k2vkndDTEA1KETqnEtTBJ7F9Tk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690823572; a=rsa-sha256; cv=none; b=4ZbKMc+klLIthmvT1IZd8n1AwFfI6zJbkch0OZEvvKppff0pnSTcwVnjTReLwfQYrICEsW 99pjRca6Bhrn3byjy/Xox3MOHeMEsC30C9NxhWkcWVppKTojoQ+8LwVDzIt5ehbEC5JzpP z7lQ560x1wIHa10wepmvca06kJH0Umo= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=sHGO4YkH; spf=pass (imf28.hostedemail.com: domain of 3k-vHZAYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3k-vHZAYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d1ef7a6abacso5231169276.2 for ; Mon, 31 Jul 2023 10:12:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690823571; x=1691428371; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nlUlu5Mhd5cq+/dnIWytp1wDOwye9pYqWME5wT1VG2E=; b=sHGO4YkHIV7FKB9rSFdsUdceu+sd5n7SB4rQ2unW3pD+BVqkwNJbz+SITG2pysaMgl gKtSzBeA+OBMePojGafJ+11doCnzqSxZKPCriFEkRtJWp27lsz7elqgKl6T8eOE/f8yK w7zw6fW868PqBeRY5qLJOdGEIjLR+7606TSSgtlwLhhrV02LXp2unR5jYo2ZCHxqOD6/ aTLQZWV2NWG8NNR6Zbwdm5er8wOMF1YxeMP3rfPV2nuC2GAZIApB/MNhrk0KohkTdxbo aZdUz3A18SoWVWors7K7+klRf0vOn2JYaGIHOGPalUyPxopNDj0TFe/WRuIA3D+NB8/C cl3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690823571; x=1691428371; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nlUlu5Mhd5cq+/dnIWytp1wDOwye9pYqWME5wT1VG2E=; b=iiypUywBvN2KT1TU1dQE/Px5P8wvXLp2XTPiWi2wzXxOvf06yxbvvqzRX4JGu8JwaU wPLa67kYq2/i/t99p6AIt9eJjqMHYO/UHAcAIoU872KCIpG3k3wlevGbklvwakiT0gkL kR7tqb+8UmFvyh/doW3X6NaVD/QYuIwy54Eg5PBvNxUsK1w1rjl6Pul1RdZySspoVny+ ch+q9bJamhdj5Anb+MWUDhuWR8MEA2RT0QxhsaJdy9f5/ixZE147UlIM9mkOE29An1i7 KvGhpHoBI94+tSGAg1IFL1xgIijMcJcO4MAWMVC1SHaH2daM78cOBOD8cG4wOx9vz2ss x9Nw== X-Gm-Message-State: ABy/qLYZtX66Xojuu4VrKSE+ukjkP7OxfF4J6UoB//VQgJcVEhN6NmP3 axDQpJGhST3ILwd00jS/SIClutDiqgs= X-Google-Smtp-Source: APBJJlE2nxuBMZTc6tsHHVP/zDZfM1zO/qPtnEy+bfYdI+vtC/H0IUdghcKWYcF/n32JcSPB9GeJlvFqcvY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:d4d3:7e41:cb80:21ea]) (user=surenb job=sendgmr) by 2002:a05:6902:1616:b0:d12:d6e4:a08e with SMTP id bw22-20020a056902161600b00d12d6e4a08emr70507ybb.5.1690823571188; Mon, 31 Jul 2023 10:12:51 -0700 (PDT) Date: Mon, 31 Jul 2023 10:12:32 -0700 In-Reply-To: <20230731171233.1098105-1-surenb@google.com> Mime-Version: 1.0 References: <20230731171233.1098105-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230731171233.1098105-7-surenb@google.com> Subject: [PATCH 6/6] mm: move vma locking out of vma_prepare From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, Suren Baghdasaryan , Linus Torvalds X-Stat-Signature: of75sqhcm355eg7951fju1ah85ygwpfb X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 2BC03C0011 X-Rspam-User: X-HE-Tag: 1690823571-221327 X-HE-Meta: U2FsdGVkX1+jOgl9ciAyJWKio4PZS2nbwwLuF3BKW6Gs/smpfqs09gwJF6XucyRaaiRPrOipSmkOkC0BAoaW40EjqNcSXYq+Q2XmD9pUuwfCV3ij7r3XVz0XTgZkZJsvEv5PcUYtS+m1sUAbrg81Qp10wbgrumvBCNVeRmEYzBx1PPA1tJGqbuVQh2jIqNCf9RzEhFs6F2Y3Tem1FnQTb9w3klNnYEFvy/YaX2u67A44pdajAN3h5BCdLVILjdAVla7VImDz6ycB8lygXi6LnS2CpnVRxHGxEdYAn1RRTVW925xxDGCC0Kukz6HteEeN0weks+glv/CoI17VgtCNxYPa/NAYLCQgj+K8EFxnH2G4EJMRmKtHAgcgV+bgbNOpPoxUgKjmVeaZzk45pcP3epuegjJzj3zMyTERF3qBKPb7OAdUJlNXHlPKkrIh8AAk33vXICDJIwlgKWebSyxhYLnyP4xPM8R4JqlIlOi5yLATXmoREf+EJjvlbPHTmp7xzrjGhpCWbN8LW2YwfqvktCDtQYNiA+HjUjpUp2nhi7qiGRMZh2KPm//XWo+56tb9ZqJbjLZO6C6U9dau9cShzuAFDoFwz9m77kIxXNf4Hs8ARo9opOMlgwR0OH9a2ribhvpaFG8B8r3exoxtcJ9vBeujrggROfP9tmJetF8oJI4F9uECsORB2b/+iIyiGWXw37jmZ3ojzYmAUvuKxCvrjAGn/8p/8s95jrfd32rFcbCCB2r9L7OuRpT5sGcW0640ATUFKGMTKxJ3t2YY+iOaP0AoyeujsJc2zD1OH4qP35O3y+HCcZB1MEDOrx3icNBvBlC7Pwol1k2CPWc31h/BApZO08I8qJdQ/zF2bfMosRKWjdLhPWJ+VUWmH/D/5FD+/1fHkXVNf7+XClPgWSP7LkHaGWwyp+HZ4yyTjuFNTF4doV2nWpJtPhUkjbdbyr7SUlfwPtqtdVbhi54lXAc u3kA/0vg u8SwFwW5eyvWdrsweWg4xNDu/TP+n+FRQuNL9TyZzKKsJNpGxZdAg1JO2oHLvVMhkvchNqh0FPsk3CSWt7YWViXDGpNN9OMocFc19OFtQmJ5CHf6+H3/pkHLRlBdSy7Bezhwum9VWl3xgqvcTo9IOj597VqQMOxzOrLb5fMBu+TxlN4NNlP57p5WyHV49HKWubqOi5HG1mq6jCHzy8vgvT2FOKfaIUOLjPGpUwnSQp2lIq9IkIlGm8t2lIXCI0hWAnlV8t4XytqrclWqsVYrK1gGLgyjZrMv6MaHuuA8IeAmseMhGaOZiZOdH48idwF+OmDDipGwMXX3adxtImSsve2/6oChQ4KWxN+K78DuGQ3nepeyX0gdVAXSwhSvxtJQLDfEQaqjM5UFsKO7piLQLTT2DVzBLEeMtCQcGntpS9eqIJmSQIEAq2nIyXlG5t8qecW4AdWofdQkRqp1w143T1TyzwY6ozdHkX+/cUhQSyfHaix7bMuead8bO2qg4eZ3kmMGNpJ1vCVTuR0o9zS9B2dhwOhMy0TYL2i3bcvYTRnZAhj9CnkBU9MarTRTAinj+ntQDmnjBXgixO8TpASX6W84AUzQYdYRPXWymPb60ky0zPGN/dESiMzLsu48LXxd+U1F9IbPxUmvkUhNOID9U574ebuYbqdQ7LC8zvjJobWfGRwioXczdHXO2ARgN25adxpJ6VNnBUJ9lkLpMbBYRU6DCBQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma_prepare() is currently the central place where vmas are being locked before vma_complete() applies changes to them. While this is convenient, it also obscures vma locking and makes it hard to follow the locking rules. Move vma locking out of vma_prepare() and take vma locks explicitly at the locations where vmas are being modified. Suggested-by: Linus Torvalds Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett --- mm/mmap.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 850a39dee075..e59d83cb1d7a 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -476,16 +476,6 @@ static inline void init_vma_prep(struct vma_prepare *vp, */ static inline void vma_prepare(struct vma_prepare *vp) { - vma_start_write(vp->vma); - if (vp->adj_next) - vma_start_write(vp->adj_next); - if (vp->insert) - vma_start_write(vp->insert); - if (vp->remove) - vma_start_write(vp->remove); - if (vp->remove2) - vma_start_write(vp->remove2); - if (vp->file) { uprobe_munmap(vp->vma, vp->vma->vm_start, vp->vma->vm_end); @@ -650,6 +640,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, bool remove_next = false; struct vma_prepare vp; + vma_start_write(vma); if (next && (vma != next) && (end == next->vm_end)) { int ret; @@ -657,6 +648,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, ret = dup_anon_vma(vma, next); if (ret) return ret; + vma_start_write(next); } init_multi_vma_prep(&vp, vma, NULL, remove_next ? next : NULL, NULL); @@ -708,6 +700,8 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) return -ENOMEM; + vma_start_write(vma); + init_vma_prep(&vp, vma); vma_prepare(&vp); vma_adjust_trans_huge(vma, start, end, 0); @@ -946,10 +940,12 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, /* Can we merge both the predecessor and the successor? */ if (merge_prev && merge_next && is_mergeable_anon_vma(prev->anon_vma, next->anon_vma, NULL)) { + vma_start_write(next); remove = next; /* case 1 */ vma_end = next->vm_end; err = dup_anon_vma(prev, next); if (curr) { /* case 6 */ + vma_start_write(curr); remove = curr; remove2 = next; if (!next->anon_vma) @@ -958,6 +954,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, } else if (merge_prev) { /* case 2 */ if (curr) { err = dup_anon_vma(prev, curr); + vma_start_write(curr); if (end == curr->vm_end) { /* case 7 */ remove = curr; } else { /* case 5 */ @@ -969,6 +966,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, res = next; if (prev && addr < prev->vm_end) { /* case 4 */ vma_end = addr; + vma_start_write(next); adjust = next; adj_start = -(prev->vm_end - addr); err = dup_anon_vma(next, prev); @@ -983,6 +981,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, vma_pgoff = next->vm_pgoff - pglen; if (curr) { /* case 8 */ vma_pgoff = curr->vm_pgoff; + vma_start_write(curr); remove = curr; err = dup_anon_vma(next, curr); } @@ -996,6 +995,8 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, if (vma_iter_prealloc(vmi)) return NULL; + vma_start_write(vma); + init_multi_vma_prep(&vp, vma, adjust, remove, remove2); VM_WARN_ON(vp.anon_vma && adjust && adjust->anon_vma && vp.anon_vma != adjust->anon_vma); @@ -2373,6 +2374,9 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, if (new->vm_ops && new->vm_ops->open) new->vm_ops->open(new); + vma_start_write(vma); + vma_start_write(new); + init_vma_prep(&vp, vma); vp.insert = new; vma_prepare(&vp); @@ -3078,6 +3082,8 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto unacct_fail; + vma_start_write(vma); + init_vma_prep(&vp, vma); vma_prepare(&vp); vma_adjust_trans_huge(vma, vma->vm_start, addr + len, 0);