From patchwork Thu Feb 16 05:17:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 081A0C636CC for ; Thu, 16 Feb 2023 05:18:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 981F66B009A; Thu, 16 Feb 2023 00:18:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 90BAE6B009B; Thu, 16 Feb 2023 00:18:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D2CC6B009C; Thu, 16 Feb 2023 00:18:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 692D26B009A for ; Thu, 16 Feb 2023 00:18:46 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 42252AAD24 for ; Thu, 16 Feb 2023 05:18:46 +0000 (UTC) X-FDA: 80472000252.20.9BD298C Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf08.hostedemail.com (Postfix) with ESMTP id 894B6160013 for ; Thu, 16 Feb 2023 05:18:44 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ngrSSnFG; spf=pass (imf08.hostedemail.com: domain of 3s7ztYwYKCAo241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3s7ztYwYKCAo241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524724; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=65rn53nYPCehqawxI5VHHUH5OXR7IFklQh11X90VOGk=; b=SRcerqiK8AV3phFT0pOmQ2yDprMEztQLMbu6gtoO6sY6DNk/5OZsp6c0xWhtciJvCwLHfB 2Y0TXDe4euyejXqD/YyMmBmryXcuFP8VDSdWmdI/NTDvKpI2agYMGLqoni4UhQw9kndhK2 dJ4E5BaZ1AQahL0zxXHpUhmrarB9AwY= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ngrSSnFG; spf=pass (imf08.hostedemail.com: domain of 3s7ztYwYKCAo241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3s7ztYwYKCAo241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524724; a=rsa-sha256; cv=none; b=Xq5atkh5C4z2qiOAU1aRrY6PtdviS/ce57KqxGOv5bb7X7vTrxW5AtEEQomAJmWSn76jfQ qfdRuQKdiL2HAZOf6W2rziceqUVzEvkxW5KJO05kmsvSftK4ES1DALUJ+BlOpzeI9lcWrM HS9fZhtRzDp0AIZi9eeDWRWphYeDOCk= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-527501b56ffso8392637b3.15 for ; Wed, 15 Feb 2023 21:18:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=65rn53nYPCehqawxI5VHHUH5OXR7IFklQh11X90VOGk=; b=ngrSSnFGdDPlYz5zuJYWd2ZcJdDDxwAT/RtnIL5xzVfAFVgHCKIj7A9xwEDcTyX2aB 3vsZSlkeyPVpcwi1aa9l93eu/icfxKL0DIaqQVOS2lNklMJwu2jpUpq532q1GD3mJcXZ OYyJWX25F7RdsRsm0bO+ULQDs6kU+hJx2yHcqT8VTisE8SVpLjiLy4deYJxyu/llv5yd uNDnBwf4OPlsNNTMONtuv4dnuvjdk8hbLw2jK3U2tc3OA2ZTvVJ3/lGJPyy4Jcy7H5Vt +ZqHtuuBA7mxmkHLm02ywaCux2tqtjmQCbr0/F6E0nSC73lfB2u+G7wX3tsYtQZiJlY+ ecqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=65rn53nYPCehqawxI5VHHUH5OXR7IFklQh11X90VOGk=; b=MV/fe/8utUhpbDHwR/leOlzwOs2WiR8KT18618jCv5OjiixuvOVyKgB8rJbMkJc9xI Dj4CmZewutPejLTB8Ca4VpDkfmwKLc1kgPjdM0xUeTBxGyaBLUM/wPVV2i1rr8Vd8gX1 8VpTAzBncwBZUPx/IgQ0flDdwUHrB8/tzL2UFUkvR+EPnxbAAPpb8e+VIFPxN7axq8wG 6yHi2Z3aHgB2mX6BYqsqT+F+WT8Et6mi0zqsVi5MWwZml9q7iQhS3KUtPpvZTdeJFnLA yr+cdCN3TZUkMN2ocuGEvyMQKFdP+edYfcCU0l4SiO2dnyCqnPXRHMriZKkVGSHqqjUB clSA== X-Gm-Message-State: AO0yUKW1tGbB5DmQVKR5jL2q07hL8vCoZoByxnZmJkM/GSAhlchLsk34 PfaG8K+8yLQ+0kzikeCqb8k1i+Vk/5o= X-Google-Smtp-Source: AK7set9eBfEal0C1jR4VjCmFFWO3/hvdavzvid6pMzzd4bvHowMVw43Kn34On8e4ZBDtcDdaQHGRoUI6tN4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:b704:0:b0:52b:fcf0:4fd1 with SMTP id v4-20020a81b704000000b0052bfcf04fd1mr5ywh.1.1676524723168; Wed, 15 Feb 2023 21:18:43 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:35 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-21-surenb@google.com> Subject: [PATCH v3 20/35] mm: conditionally write-lock VMA in free_pgtables From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 894B6160013 X-Rspam-User: X-Stat-Signature: g3z6qfyfsedhxq3kzhgbx8jjqdan41ww X-HE-Tag: 1676524724-791097 X-HE-Meta: U2FsdGVkX19qmn7GTjSpQGu/FlmWFg06vpSnIX3qotr9XTBSwFmxQbSEaZsnZlFsiIzI/9NLTGgznBlYVj+Afp0FGGjhVHjeTwH1lcbHPlbkjc+If0ZznukhuYfo7X/Dw3VdY1EUoTWDnAx5aP8T50/MeWZpBu5vmcuvhg2z7NC7LHLHVx68v71gIUF+A5QfJ180XN9dyEmhrVPyul7N9lUtQNNa++bs2xbOL2Vc0SaNWC/TVpHn0BN2Faz9sxH5492lOSkj/fDzprWjaFGwg2wmeTKo0PyjC8Qts0AAFxtQi7DKINEgY2r3K9B1Ql1n+sgjB/hnMnE450zb2YhX5Qq3CY2EK3A+Hw/UPebp/gYWBqEIgtdI/+kb04iNXFNPA7Dhb/Ouk1gXWfe9Ljmv4jiKDP9h5f0suKIoBD0OJy35hZkHFGPjjlgImjqDiZMHPl+Vd555UKdObDiSp4fmmpFp5yQ+GmTpIUYxHvOY/HnG/izhAo368F3VTgOhNuWBpjdF3Cc/bz/t4DL+AFJM5vsZWTO7gcCev6TsOSTql8DSDR/p8ToMyVAhHmtudKXpo98kOoV+bF8uNfxpxXpzVt+G5f6AlRhOMnvc4kPWeCSIpoWQg7zpChqLaJ8h9j65/qMSdj2OzTtWsR8AM+LhxhdEHDZrHi5jFLf+jxcNl6ZOJbBHyFZ3txai2ogdtKkciV7dhpeVXkIO7HDXgPmTlvgkQ+gS9UHdG9oCdLiJtpn0ZvjRpuPNqa8+wcqxXkaEpq391ld1DZjwAXL5hjfc6TeuIEPJeUSiDYkIuOoQqaFRO1Pu6NWmlAM5iJfPyl9EA/SlHrauyKtWCcZ5ygc2p14elb5iGCkzzhXgAUvPpHEr8EkXG8m98Gv5TCBpfpdjWuubq5H6P7acRtdeC9UYXfXgK8BoWyX5bfaaexRh4YL/C/BAk5dgkFidy5gZS6ulVKrf7vReOpZzaEBnmEx 2lKNMoo+ 1ICUoPpBe3b+vs75eF7FFPAvAa7h0GvkCPXHFNqelCQXMtrRT+WNbGwtZIuJkSI2VuJlSW3sza3kLz9KjqLrl3A1hDST/YRHH+HnYEesPD8/+agluf0LolU1xsmc+jiEjx2MCyDIGnRjP2INWbPrsxMsHcW6boqifExOQPC14n6RZxULuy+PmNbnYo3kqvvcTrkmfjLIct7KFZga0XaqKlF/m8TejsClW89buAmVslTpzGkA02hF3xwKEfi16gZri13+m9ARtfwYsfJgqRq1TlNV1gV8mr8Iarl91zVLZgk2BwQr0z00NMtCOxuSPbi93Y3HjR7mq17QB4+2MxjBYQqWy7KdjdoCKNWpWoAzhaM+2/yh6eF5YudeBmAtOjbarYZ+kPt76fOWCHcZdI66355b9tXczxPbloFeFAFNImImBvDbPTXrPcyoxAsq62zWzrjR2Akf93e2t9ZsMCFuXR4zYAwcAaGQKXCvuboVKY1DoCqkpvDQYQ2xtCk9yIqqVLoimC2sl0jAf7GmmPBbrQOfZcRF4OUJQJNofkiwHO4g7nZDpfKJZV7c7btRwZWK95h5n5gbMVQsYQwktDWk8yhZfV/42QeNlUcZJ6zFmjlekUimY0by62+WaK41lOF78dfogWK5ylfhg0PURkdkNx/Xk63mx8XK9k49aoX5R58veFh0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Normally free_pgtables needs to lock affected VMAs except for the case when VMAs were isolated under VMA write-lock. munmap() does just that, isolating while holding appropriate locks and then downgrading mmap_lock and dropping per-VMA locks before freeing page tables. Add a parameter to free_pgtables for such scenario. Signed-off-by: Suren Baghdasaryan --- mm/internal.h | 2 +- mm/memory.c | 6 +++++- mm/mmap.c | 5 +++-- 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index fc01fd092ea5..400c302fbf47 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -105,7 +105,7 @@ void folio_activate(struct folio *folio); void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *start_vma, unsigned long floor, - unsigned long ceiling); + unsigned long ceiling, bool mm_wr_locked); void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); struct zap_details; diff --git a/mm/memory.c b/mm/memory.c index f456f3b5049c..8177c59ffd2d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -348,7 +348,7 @@ void free_pgd_range(struct mmu_gather *tlb, void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *vma, unsigned long floor, - unsigned long ceiling) + unsigned long ceiling, bool mm_wr_locked) { MA_STATE(mas, mt, vma->vm_end, vma->vm_end); @@ -366,6 +366,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + if (mm_wr_locked) + vma_start_write(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); @@ -380,6 +382,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, && !is_vm_hugetlb_page(next)) { vma = next; next = mas_find(&mas, ceiling - 1); + if (mm_wr_locked) + vma_start_write(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); } diff --git a/mm/mmap.c b/mm/mmap.c index aaa359a147b2..118b2246bba9 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2157,7 +2157,8 @@ static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, update_hiwater_rss(mm); unmap_vmas(&tlb, mt, vma, start, end, mm_wr_locked); free_pgtables(&tlb, mt, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, - next ? next->vm_start : USER_PGTABLES_CEILING); + next ? next->vm_start : USER_PGTABLES_CEILING, + mm_wr_locked); tlb_finish_mmu(&tlb); } @@ -3069,7 +3070,7 @@ void exit_mmap(struct mm_struct *mm) mmap_write_lock(mm); mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, - USER_PGTABLES_CEILING); + USER_PGTABLES_CEILING, true); tlb_finish_mmu(&tlb); /*