From patchwork Fri Jan 27 19:40:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119256 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30726C61DA7 for ; Fri, 27 Jan 2023 19:41:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B8BBA6B0092; Fri, 27 Jan 2023 14:41:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B16726B0093; Fri, 27 Jan 2023 14:41:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B5866B0095; Fri, 27 Jan 2023 14:41:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 863366B0092 for ; Fri, 27 Jan 2023 14:41:57 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5D16780754 for ; Fri, 27 Jan 2023 19:41:57 +0000 (UTC) X-FDA: 80401599474.16.F2CC071 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf19.hostedemail.com (Postfix) with ESMTP id 99C921A0019 for ; Fri, 27 Jan 2023 19:41:55 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=lE9qRR84; spf=pass (imf19.hostedemail.com: domain of 3_ijUYwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3_ijUYwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848515; a=rsa-sha256; cv=none; b=VRO1fW0feJlVLDn9X1BvxjxVJ5dcWbzybcmwG+fC0xQqZyEl3uDfKmZk/NxeSFlFClssh5 R2yY5Fd/ifsp1L7heo31P9ryqqE9Cg8nsCznBPrnzkcvkUpDvZwismEMVx2PudEw0WeS0W NqeiReBU0Fpo2mjBW4CZxYq5NHtSkC0= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=lE9qRR84; spf=pass (imf19.hostedemail.com: domain of 3_ijUYwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3_ijUYwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848515; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xM1gV9tZESxmDxiysmAmLZq38ZaKZXSlDI1GeX9HKic=; b=BkKKyRvWSiVYzAknmwH1ejeEaB6UwUtGyo04ZzDJ5IhTJXJKyYIy4zJhE1C79waIgyw9Sq zMAZRDzE7O7wkL3LRcRyHOp7A2TfHF1GWpvneoGDJWyKLjOo2JPTtnYov6rANoQDTUipFO i6vMm5qWsODArS9GYSGdM1E4p2isM30= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-4c11ae6ab25so65246377b3.8 for ; Fri, 27 Jan 2023 11:41:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xM1gV9tZESxmDxiysmAmLZq38ZaKZXSlDI1GeX9HKic=; b=lE9qRR84U5+rYrRmzpPAw7Ud6uFSOEezLerrdXzskk/7N7IWEzIS+P8h33l5/GHLwm pzf0b8p0OfVWGzGLb/dMUWWv/4+TG3PN/VW/HgmJDf4GTYQxz4qhZvzGmSe6iwYXQbEv pmRWin5z3kTr/dQHGmOmQouL9X7gNxEvHMfX/Mw4+j3xs8bwN3ke4zv6/7Jpk/cXGw8a hnXfsAcVwtJnLNS//9cNJnJKu2XJduYEuopAyOxXqZTZsrcp23rvXM55Jk6t7ntayedf QFH2QchxeUajMjFYvRPLkdcrU5vfcJYPLkQ5Wb/mKP36Ds4BBv5V6MGo7zAGbp1SePYq xw1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xM1gV9tZESxmDxiysmAmLZq38ZaKZXSlDI1GeX9HKic=; b=2HeZ8bHr+JYFuyc3IetC9nshd1Qat691URqw0fD61oxd15vULQYs7/3PXXtS/2H92Z uXBzV81OP/pOEiNnjnXIGnFrRpUO69YsoMtSIKS+Q0qxgZJuY0cUo5CBMN59WQu7gU+O Toqr5Zm6sbL3w5OTwLff77oJjyvK8JqqDGVqxpwsMvb7Go0AvP9FPcZxEPO/ciBkay0c daRo8xsNpwOB+oj70P6I0debN+pyHZrr7UUqHYo9nx14iZ5mPoqQMr3PMl6ElDx75VaI 2Aw07fRe4eTDVO4GRXI2rXmy+ErwVLNSZkeFt+6HG8fL9Cfee8SqnnHb2rygoYQvCUtT Nc0Q== X-Gm-Message-State: AFqh2kpfGCkIKwVlmfeAnITyxnsic0Fimt51weDvTWhMWkknc7IWlV+w aAoO+KoO539BhwJI2GlwZ+/QiIE/fe8= X-Google-Smtp-Source: AMrXdXsibCiIco7YlC7GUbshkOFJQAk8PpTkup371cElxWyegDeUsxNSE/XyOxE1JUpmHxGWPZOyG9ioimM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:b94d:0:b0:801:e503:dd0c with SMTP id s13-20020a25b94d000000b00801e503dd0cmr2559229ybm.384.1674848510225; Fri, 27 Jan 2023 11:41:50 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:52 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-16-surenb@google.com> Subject: [PATCH v2 15/33] mm/mmap: write-lock VMAs before merging, splitting or expanding them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Queue-Id: 99C921A0019 X-Rspamd-Server: rspam01 X-Stat-Signature: 446kwd7h3yn9g19nfk3jeoybse8ngx9f X-HE-Tag: 1674848515-113936 X-HE-Meta: U2FsdGVkX18W5mzTBRd2xiN8UMce5m9Os8AQ6Au9UD4tHD67fYsllKnF9GGoz8rxCHMfzMLDpIecDHnmCOt6JVfiBJIMYM+azo4+SbhAlpYbMyI3TVLqoT/DU9TOoeaCwLs8zGE4Rok299VVp0sKAa3OEb5vgIMuJZJbd2uMBEbs6GXvpIduF/mASNXHYz1MY2MPi9FofPXefNvWpKuIVDUY+LZupubDpmD4i2cQ24WtpFRjkNM1T2sWcmhxEIV4g5FQqWb6fdx/RRP6oKgnZC00d39oedj7cHWVDYleVAuGXzB34m4kENTIJy0cB5C0CPSc6PjaxUsabLI1xRmhlEZh/u4CP5KDgzjGyZMrDc3oT7NYNkJWAY6O3wUSJTsoNYd3f47P+sqWcgH1cfJbxIjejAuC7ZM4g/gaKfoV8a720QhazZn1H7YC6A3mFwcPNI+MD3+E1MsOqpOgBGVpNZpUN4OvRiApdTS+TWotSIFA+xaKmP3P8DZEgwPxaQqrLv0MaaeLgbi28aewRfiLlWLbkf2PMq3W2GHsQ0sP0skTbTXZRiKdTJPj8DDMI8QBHS4DQU2s08tNkGsvPM12JhH7U9Nu7ZQ1BNAoLAI1ZX3yDWOccOBwGISvx6RV6j4LxD9AWalj0b3WFJ/cyaiB0cMB+l0MX2hr+sFYoGY1eMiu1ZbvgM8I4XMNY23D36NYOl/6g5itl5JH3ndYRMQ227TtuLoovVDbYwNSO1ohc44xDQMXChNNPOUG5U8Sa0zSwNnjuIrvrKxs8VKT/O5t1dxLZi7YflfLItSBRxg+RrR9s9cJvnVH3zWCWCpkW6bi3wBSZWwFKa//uwvwZHazl2FGtZNAjnZPzNTo/Un6zvzPOMchT41/SBjrB+HTNwuP1/ZhQ0ZK0T80hYv9dJkN4WGmnKDo9Dy9Zpn65yJCaRaTnmfeWs0xLYcGS2t8bENvt9Ge+iGiZlrXNWS2y/g j/7biOt/ T0re0a5UDvpKwzmqRkSMdUI25PpWPUVoXPcyM9XFW/9prmoZF29PZrWwrH0GS0SDbZR1etBD1703n+k459XAvBQv69R9DVux2QD8+AYfEpBClirgkP36iJUNypB15qXLGlSaO1y9wI/f7DrdLuW3HjDqyc1wWIYMTTC//xhIRaIWaaP6CruC0rDqRt9tUlBoMb2XUe8oRWZT0nBRwNZEBiuk2xQ2vk0I11aCYC0c3LE2C9fg+i4KohVONwKH8tNDCMcKZ+ZoJHl9Sd4oNHrnN8DeEXbPzo8UYVBwb0KA8qKaH/V0n2gKsMsI09T4eN9zXbwZQN3j5cgItg9YBGI9ChYxG7r3poicBQXNLRjF0Ameoi43uq17lYcPADxIqy2BVoUul5sqPg6iQVGn8+rmc6KVHZlL/QtMKC6myeEx1+hVesWY79DbMppucSpWL7U6iTuQmHWkRJadxMoYfKdBpBRyboYqv43yM4D8f0Y4DI3GiZ0d35rHXPwbdvDzajOnKwFf0gIWTO9u39AAeOeToaaZFxu5rJ+iPszYixZOfo+OzZs4Rm6XuOJXJtQf8BAeNYID/x4gp1Ym1B6Zud6WvuYOlGEctgY9solZshFSY1XhmpZRnON/C0vg2oQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Decisions about whether VMAs can be merged, split or expanded must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Write-lock all VMAs which might be affected by a merge or split operation before making decision how such operations should be performed. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 7467d691e357..6fff76334177 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -269,8 +269,11 @@ SYSCALL_DEFINE1(brk, unsigned long, brk) */ vma_iter_init(&vmi, mm, oldbrk); next = vma_find(&vmi, newbrk + PAGE_SIZE + stack_guard_gap); - if (next && newbrk + PAGE_SIZE > vm_start_gap(next)) - goto out; + if (next) { + vma_start_write(next); + if (newbrk + PAGE_SIZE > vm_start_gap(next)) + goto out; + } brkvma = vma_prev_limit(&vmi, mm->start_brk); /* Ok, looks good - let it rip. */ @@ -912,10 +915,17 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + if (prev) + vma_start_write(prev); next = find_vma(mm, prev ? prev->vm_end : 0); + if (next) + vma_start_write(next); mid = next; - if (next && next->vm_end == end) /* cases 6, 7, 8 */ + if (next && next->vm_end == end) { /* cases 6, 7, 8 */ next = find_vma(mm, next->vm_end); + if (next) + vma_start_write(next); + } /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); @@ -2163,6 +2173,7 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, WARN_ON(vma->vm_start >= addr); WARN_ON(vma->vm_end <= addr); + vma_start_write(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split(vma, addr); if (err) @@ -2518,6 +2529,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, /* Attempt to expand an old mapping */ /* Check next */ + if (next) + vma_start_write(next); if (next && next->vm_start == end && !vma_policy(next) && can_vma_merge_before(next, vm_flags, NULL, file, pgoff+pglen, NULL_VM_UFFD_CTX, NULL)) { @@ -2527,6 +2540,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, } /* Check prev */ + if (prev) + vma_start_write(prev); if (prev && prev->vm_end == addr && !vma_policy(prev) && (vma ? can_vma_merge_after(prev, vm_flags, vma->anon_vma, file, pgoff, vma->vm_userfaultfd_ctx, NULL) : @@ -2900,6 +2915,8 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (security_vm_enough_memory_mm(mm, len >> PAGE_SHIFT)) return -ENOMEM; + if (vma) + vma_start_write(vma); /* * Expand the existing vma if possible; Note that singular lists do not * occur after forking, so the expand will only happen on new VMAs.