From patchwork Fri Jan 27 19:41:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DBB99C54EAA for ; Fri, 27 Jan 2023 19:50:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 740266B0095; Fri, 27 Jan 2023 14:50:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C8DD6B00A3; Fri, 27 Jan 2023 14:50:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56A176B00A6; Fri, 27 Jan 2023 14:50:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 49D676B0095 for ; Fri, 27 Jan 2023 14:50:08 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 279B91C688D for ; Fri, 27 Jan 2023 19:50:08 +0000 (UTC) X-FDA: 80401620096.29.8724904 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf08.hostedemail.com (Postfix) with ESMTP id E32B4160031 for ; Fri, 27 Jan 2023 19:50:03 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=R8vm5sGe; spf=pass (imf08.hostedemail.com: domain of 3JSnUYwYKCOwgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3JSnUYwYKCOwgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674849003; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=aM9TOObWyxI9lZa2RiM06JqdLQX6NmXnIDz0gFwyKgPPXxq1oeBNgYzaSk21m1ZVN6oExr dzRdZHtzVW2sSfn3GzLruC0KK1wmIY6ublN8UITPwthkXfde5c9LFnwyCgtDIKakwjY0Du 8UVutTs05QI6zZdek+C29yWAUA2kBgs= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=R8vm5sGe; spf=pass (imf08.hostedemail.com: domain of 3JSnUYwYKCOwgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3JSnUYwYKCOwgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674849003; a=rsa-sha256; cv=none; b=JpbjCg4CLoGEnE8JssLi8uL260qb1ibBZtsZmru14Bp8le0eVURCXmHPbx2z1iWb5edyil 6FvydpfuF2VFEMN/Z59f+itaQZYb1URN9tdhv9uleM3ZnpmD4YKsdGoV6IX2rV7lnPBgjt d3YAFc5pPNljivuv8+pzh5sfxpKe0V8= Received: by mail-yb1-f201.google.com with SMTP id r8-20020a252b08000000b007b989d5e105so6395108ybr.11 for ; Fri, 27 Jan 2023 11:50:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=R8vm5sGePmLU1Pe4mKnKd7BNiKDqaMQRSE2Uxaf9jHVe/UX+8wZaZnli/NAPrvHh5D Gp2YVdh1UhOUz23uxWg1VsM9bUD5cup3/ZbXShp0DX2DQS4OYSwXrDDdrwmKhIAFH0Hw 74ENSW8vtZuV6jUaLsmkcaEy+yDT+z+95V7siTwW55EWIBuns5aniQb/TRczPRcRq3g4 VhtuB5yMK58BxiemGZ2CqsONqT2FuthDA4119d3zQhl7jpXryXulujWcH7o4rSRWH8Tl w610sqrsMHevwvr509uliYrTdDC7Uzh2VKUBma4LjNdxi7kycc6HPZnQZbUV0crFM3wd Zbgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=3VZAxCiVSpDAU72O6mEUewZmTvhi/kqOr11MFQEgpJm5s1x7ekijyD05e4tHzy/qJJ brprENsxxJ+LDtzZkozfOMvyV7v2s89yDsTo4epcin/QX9yskur/uq7/g+Yfh3D9jMA9 6rVGVc3L7bBKPag2+EtWykVefd/NyRjCOURwrj3aAky8/N+Gd6/oCTggprWJ7MYe7g1z N6BdoIqankUCFv9zJuKbbRXZyRfW/5QvRUfn0wthZzSgPcuPjw/a0NJEThcAUwa3H3MZ khdxddnsqcXvSfF7qG7FkYON9mC0uOoDEv3oH1/CLULcn1GInfHDek4P4JOfWy54LbXc 2zfw== X-Gm-Message-State: AO0yUKWPCUKyNHHwEcztdI6W7/kuTHaM7n3kA3Y5rC6NdMHsc4rF0cRP +MPeImQpaxFJeBYqwbZcvVy9pCIl6a8= X-Google-Smtp-Source: AK7set9ZwBhHaiq8PgHnHZpi30bVFoqemTYpH4CqkWTR3ntjpG4ZYW0UY93Gsvx11KtEAuMPsZpv9FPCPgs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:db07:0:b0:80b:8dd0:7b35 with SMTP id g7-20020a25db07000000b0080b8dd07b35mr1614607ybf.322.1674848549978; Fri, 27 Jan 2023 11:42:29 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:09 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-33-surenb@google.com> Subject: [PATCH v2 32/33] mm/mmap: free vm_area_struct without call_rcu in exit_mmap From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: oy43wcwkhna518zgwc51o59g3yoik341 X-Rspam-User: X-Rspamd-Queue-Id: E32B4160031 X-Rspamd-Server: rspam06 X-HE-Tag: 1674849003-212394 X-HE-Meta: U2FsdGVkX18h9Jr8jNYoerN0aNCCD45g+9A3AO00y+C5dcPOalqOep8iP+PbPhUHcjnV4CBhK+v0H20TiAUmHp0nwVsr7MOV/lnZgs6To6Phz8jA632SlbKhlZdDWZt3OOoR+jEiv3Iflm52oqsr5MttH1r3vdVnI1Y6L2G+qhwbvek7bd60od5pJXk0OJITCYz7OGsuxFNNkzl+Tc+9UEpXDeHIYpBs3yYBFXq/FE9FWbYPPduvsoDjHlTGOTILkrNmxudpgW8xVM5uRzlot/b6E42ByBwmQQPRZtcq0xQDxHCN4K3xh4i+MRfAhg8cw2K0hAFIzPbjANjLMTeBqKGP56LgKfI46krXsDwVpuLlIq5yP7H2N54QDLqMoGtvZcyGSci4sCJDqoos4z4/PbLgiuSgNrI6XLU87Io/gsxa4D3A33jjzYwa2qPP30QkxDkh/1b9vNc5U1nkJCcVfMPvcszttEsbEAntzRQvJL7ZEO79ZT3CKBlpevmLKiOBrcixJOgOQ4hDC4R1YADfIdwDT16b4dYZ6pzABtDlLAbDm0RCmmf1SyNSejB3vlqTLEnDWmnLiCiUYOVZyx6KbQUuhbYdVaCYIvG4SUrCsLebJwxaXduPgwC/mX5vOMAqdUIxeZ5xrINNKl/R1xvNhH2OSBQxmvQByY3GiO2lTw5aMx/9dTVkFtJAHdu7nJOmkaC3vfZAb9gtIjk8TtCtEPoX2hebNm/iB2NVyjgJSjjumG06edMa7KACBBrzYGd2H5d2pChPZn3qZac8Lq0s9tJBB4oCFJdmnnCEH6Ja4hBihQy60S38Yp8OX4P3MzOKEFnKD6wSh9r4nRv3Z0IbF9Gi2uaMdwshZEKkXQBo2am59vXVI58YdJLlXquKzNSi7lFrpOYs7dSlLJtiB7vTTywmcYqJK+A5Swm8j+g1JVCNT+cVDEkAAW+QPwIc1jqhW/WTRhIBILyfFb8i3fX lJrLPaps KgR2hbjC2yR0zgJ29LZ1Nn3c0Yx317FBX3Kxm/IHAofG2IsVCAxolSuH9FMSQlTix5xX6wBqRwo/2N6RI8gZyguNKbTb3AnMj3/IT1BQTAAZV/hFMsqL8i3k6iNl9u0wbUJY8dHJFkZWoTzdOVzxKPVkXe+t9fYOew5ghoybK1NGjG0pP3LS4nMVwuqFBBqxX/2WWhI04DyxYEMv1qfjzG7AjGgMNl8UTS+8949oTQmWWBjR66VhwfQOI/cukB5NNiVjsc/4SQinmqyMupHgA7ioZn7LRim9BdIjp9A4SYtG1Ib2bfVV0nQkwZFyyJu474Lk7H01IPqwj3vnIi3XRVHzSsxJXG6bJJ1+IiOVmQCkQ4JGbhGNyOLgZ6owhoeVt84B9SBIsvZQh3rr3r8j0tGwuXC9Nc00Jo9YrViBR1qMeVTfGrBIP0DHUucGknD8jlhqHvWsDU4D5lpPfo4I5y8+Vn/bE+Vfov8S/0M1m1lqeqhs1qPb+Lib15OYhPPJ2rkOtS1K30b44V77iQ3KV6DOyxuFyoxPmSDqTmbffg8ZZWRE+2B1OflYWs/QzXRMtP17qHrhMjF6f834zVx51tSB378xfPB2wXnaIrixV2+uWgkSrUS+zMpmhm0wX+8dEOHRe0v4eETxaEnWXDjWcX0PNBt0PS+qh3gnaB0xv8seExfw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. Because exit_mmap() is called only after the last mm user drops its refcount, the page fault handlers can't be racing with it. Any other possible user like oom-reaper or process_mrelease are already synchronized using mmap_lock. Therefore exit_mmap() can free VMAs directly, without the use of call_rcu(). Expose __vm_area_free() and use it from exit_mmap() to avoid possible call_rcu() floods and performance regressions caused by it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 ++ kernel/fork.c | 2 +- mm/mmap.c | 11 +++++++---- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a6de58bb40c7..1c4ddcd6fd84 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -250,6 +250,8 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +/* Use only if VMA has no other users */ +void __vm_area_free(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/kernel/fork.c b/kernel/fork.c index a08cc0e2bfde..d0999de82f94 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -480,7 +480,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -static void __vm_area_free(struct vm_area_struct *vma) +void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); diff --git a/mm/mmap.c b/mm/mmap.c index 3d0cfbc92745..1028fe131bb7 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -133,7 +133,7 @@ void unlink_file_vma(struct vm_area_struct *vma) /* * Close a vm structure and free it. */ -static void remove_vma(struct vm_area_struct *vma) +static void remove_vma(struct vm_area_struct *vma, bool unreachable) { might_sleep(); if (vma->vm_ops && vma->vm_ops->close) @@ -141,7 +141,10 @@ static void remove_vma(struct vm_area_struct *vma) if (vma->vm_file) fput(vma->vm_file); mpol_put(vma_policy(vma)); - vm_area_free(vma); + if (unreachable) + __vm_area_free(vma); + else + vm_area_free(vma); } static inline struct vm_area_struct *vma_prev_limit(struct vma_iterator *vmi, @@ -2134,7 +2137,7 @@ static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) if (vma->vm_flags & VM_ACCOUNT) nr_accounted += nrpages; vm_stat_account(mm, vma->vm_flags, -nrpages); - remove_vma(vma); + remove_vma(vma, false); } vm_unacct_memory(nr_accounted); validate_mm(mm); @@ -3083,7 +3086,7 @@ void exit_mmap(struct mm_struct *mm) do { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); - remove_vma(vma); + remove_vma(vma, true); count++; cond_resched(); } while ((vma = mas_find(&mas, ULONG_MAX)) != NULL);