From patchwork Thu Feb 16 05:17:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 448C0C636D7 for ; Thu, 16 Feb 2023 05:19:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB5CB6B0095; Thu, 16 Feb 2023 00:19:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C3EFA6B00B0; Thu, 16 Feb 2023 00:19:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6A656B00B1; Thu, 16 Feb 2023 00:19:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8E3E36B0095 for ; Thu, 16 Feb 2023 00:19:19 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 76BE31C629D for ; Thu, 16 Feb 2023 05:19:19 +0000 (UTC) X-FDA: 80472001638.23.7CF5A6F Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf19.hostedemail.com (Postfix) with ESMTP id BDF3E1A000A for ; Thu, 16 Feb 2023 05:19:17 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=JAL2Aq1j; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 31LztYwYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=31LztYwYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524757; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gCR40Nt3TWmXm3CiMW8diiOUlGd9n6tZbm3lsLizpE8=; b=3UataMOu4NKWjyVFxLUppp10ejDmXS6cgwBHVgBAEiODFlwXhsUCjfc6XYGvaJWnjor/l3 HlkXhZKRKHauUZ9kOtjmTXx2wRXXEGCiNZnBqAdEgKbRUaBCgYpjt8I5wKh/D6oWqJlbz+ u9Jx97g1IWSOYui8t9DDu/FVUXG96rw= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=JAL2Aq1j; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 31LztYwYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=31LztYwYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524757; a=rsa-sha256; cv=none; b=uDtPqKb+4K3g3/5VF5HOtYbS8JQWakvhFkFRsr9YkDeS2c/qEr/7S+rjcCn/n4e4A54qfv JF5MLe1gzXph7lUahDg3KFeZfRJZfukbChPSO9MD7Awq6nAuZeE9c+SB3NXI4mChc9NVqx NSwGYly91v+2kmBHiNyoKz12FSdp+oE= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-52ed582a847so8822177b3.1 for ; Wed, 15 Feb 2023 21:19:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gCR40Nt3TWmXm3CiMW8diiOUlGd9n6tZbm3lsLizpE8=; b=JAL2Aq1jm/4r1JqeZYEWBrue/W3MPTrJmtqCzmWLkH/WIKCq/WNMS7CZcuyalyxOey IosycXB+MxcwT4QBmmHaciSLx6z1YK/qs/xuZHAZ6O8LZPapUolphd/QYky6xf7HXG4m TIajJoSTXZu0Yk9TmYuKl91OepBHI9VMsFb9wtZhCPHchjEtdJSeWxPPYnIrHgMp6AHH zReb74rJ9HRrecOXakA/GD9/YkepqNwyoeE0Te0FwV1uNkov7IGEqryUYtJWv/bTKuMJ k46shKzXYlPWW+imaaqXvyzMwcruK2EkfBqoUiODxJ9SCK0t0Bhmm1Z4Ww2xAI1rN1gq fZJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gCR40Nt3TWmXm3CiMW8diiOUlGd9n6tZbm3lsLizpE8=; b=7HtrQudQ78Jozzub41vJSmLKdUay7ICQtR40NMXBmFTqoPEF7aESrwjtuGX3qU0Dhq D2lzlhGBWHEeq+Jg5gefhi7lUbYol+YFu+uYHkj0/eAyP8yaVq+1f0O7+VIwZojlSbSc 6bon6Jvg1MJCd8y0B9nfj4wGo+kvjs7X/keBIxPwUkiHy6wmPEYF2ldmrtGyhpgJ1HYJ s3dECOBICLCQ4BzPqkOSiainPtsdJNXtDntLu6D8kVE/SofWjeVsaJwsij2yBsoTL5vg WjftYth0H8b9YoqtFQ2b+aZkCKB2XEA9OAPIiaXrGqmWrdzdJh3gh+x9N0fQ4jfJtAsC iUpA== X-Gm-Message-State: AO0yUKV4J6DV7LWcNisGf4n+ph7XO4W3bNhyIQw0MjH1kdYMIKiNvkDQ Rk0E9JlpJbBd7X/L1jLF1iuHGk8NTD4= X-Google-Smtp-Source: AK7set+zfCXRRrabneP7jbhvgAuIkh3ByJMNAQijkS9aLyX/yWFovh9nbJLYkoOZH5rfO4SiFvvO8qqtNyU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a5b:609:0:b0:80b:adf8:4f9b with SMTP id d9-20020a5b0609000000b0080badf84f9bmr575565ybq.33.1676524756940; Wed, 15 Feb 2023 21:19:16 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:49 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-35-surenb@google.com> Subject: [PATCH v3 34/35] mm/mmap: free vm_area_struct without call_rcu in exit_mmap From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: BDF3E1A000A X-Stat-Signature: qq5atu47fabhcjmhohkr6tfzhagw37wq X-HE-Tag: 1676524757-759187 X-HE-Meta: U2FsdGVkX1/EzujQ9Z4h96KSdtYj0buxetQPH9ElFQ9fvjL/5NEWX4YEDCEcY2HEZlw7G/GHQa+RzJVTx6cT7AHe3DATo4oaOaEq3CTwC6Lt52ccLY7tocw+zwLL5b6axXYUuMgjFqWVB/xlq/hpInpYpqA8til/PiLRVSvIfGpX8/KAB3M82Z28HBIrGEWonmlPBGIkp/8HpEZaozoDsfgnK3PJ0Wr4rpEvxqST1La9pbUfN9aFBwrjaSjS2zaZh5XrPcJwOnzeHyZaImyIv0bF6VSlT/EH+/cxSgFP0RVdJ2+ZiqYb4dWxbLRS3/HWq/9zl77ixT2hE7mU3EJhxvNrY39R7RZ6Lhs/Odr+sj7RB6P6Ww1db8oNRM5EnN48b89GeXUOkt0nt59vyeSsx6XMYeDNlnhnukV5DmGpSot3y5COO0jnNmv4t99JJbVBcCgs6CHMqAHMNg2otlayzWz3SZ/gNDGyEMeid0HlTygYoq8WyYJjGCe+L2xqD6E4NVZzIvhfZY8MIAVLC3QF+i7zhwl7qmWFAmldAhreQuDY1qmRWok2xUAOLcffgLGaG1M2YFgmaCiABV1CIGxjvHfRb3tFZGRjsu5dSdAGTJ/62cA99EE1lrIvkyUzLszCRMGknQWyB37T0zM/woAXGf1UQIovIT6PRzjNLVTnTjCl6NkdqfoufpO+QiareFnPVwaFKtV9KM1zkmgIPyFmd2QqUfrl4lgVx6hcVqsh51FUxXbCwu1UFB94SArVB3GM7Y6reFcVwmSwbJFVa62WLKMlNPtQ3Qg3GPOcRVAX8hUb3sHjmD5IxVYQxNCQg6oF2OkdeBinkmsT5PRCKxyKrwbE4WzrxHUikDxn2ZZOLaIfuuYKZ0Y/s3sS5p6nB8u6bVfWvJn9iaCX/tWZo58/4ubPNy5WpjZ+lNVWpoR3P0a2PAHlyi9HfdroAUJRIAUarErCln0ydRarSvqqyH2 0uomQKhE 6i8LbpYEtnS/hdGw3ohzU0+xfW4IVHkm3e/bhPHYL/WGhEU5nH4Lmjbhe0gjOzd7i8rWpSbL3fDCG2sTDaAYYzUyZ4NZKEKc06StMX5+b8pmQEZb65OVJpFL60xMHe2ZsObN1fNGgR7OZZXZdxkPVm6pDlXKcVQJQvU0HNIOntK17OFO5OfVawZw3vjjOebHVoah5qfPoWpW6YfLhvtMXCBo3dtSoiL9Zu/M6TlmIofcJJnLDIH4qjHW3FYQGjySC4C05g/fRcjGqk4UWDyWHiWOQdbVUK/o93cxAKwgMOCokpm7iBfjLZm7l1PHVGLA8w/Udz2U0wBJyL4yN3g5N3X09h99ixNdulROisdAIy/a4/qEjLNlGO0ETidER5IJ4NguAUScoUdNcPug5izC2o4QH+xxgY3RW55EIwsHNtLPuc9uYkYBUKx05ryryJMB1d99iPagPItmF7lJGwd3fv+ANCKyYvS2ITXHd9Y+MsYpgbezHfaic9b6XxHet/FtLnwKoea9NFCGNYUI+w7IYRpuauRqi+rTIp0ZXi/wWVLi8UdssL2lUqWCzHvqmV3EVRyPKryxUm3vwzjBWyHPofHH3EMgjZm0cI//pTWAbdix2GAJ/NgHCAL3eHDgbKDvK7aawis25l2/jVs3P+49TeH3Y0uVQw9JoP3rnM7DlYpHjMjE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. Because exit_mmap() is called only after the last mm user drops its refcount, the page fault handlers can't be racing with it. Any other possible user like oom-reaper or process_mrelease are already synchronized using mmap_lock. Therefore exit_mmap() can free VMAs directly, without the use of call_rcu(). Expose __vm_area_free() and use it from exit_mmap() to avoid possible call_rcu() floods and performance regressions caused by it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 ++ kernel/fork.c | 2 +- mm/mmap.c | 11 +++++++---- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3c9167529417..cedef02dfd2b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -256,6 +256,8 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +/* Use only if VMA has no other users */ +void __vm_area_free(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/kernel/fork.c b/kernel/fork.c index a08cc0e2bfde..d0999de82f94 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -480,7 +480,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -static void __vm_area_free(struct vm_area_struct *vma) +void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); diff --git a/mm/mmap.c b/mm/mmap.c index adf40177e68f..d847be615720 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -133,7 +133,7 @@ void unlink_file_vma(struct vm_area_struct *vma) /* * Close a vm structure and free it. */ -static void remove_vma(struct vm_area_struct *vma) +static void remove_vma(struct vm_area_struct *vma, bool unreachable) { might_sleep(); if (vma->vm_ops && vma->vm_ops->close) @@ -141,7 +141,10 @@ static void remove_vma(struct vm_area_struct *vma) if (vma->vm_file) fput(vma->vm_file); mpol_put(vma_policy(vma)); - vm_area_free(vma); + if (unreachable) + __vm_area_free(vma); + else + vm_area_free(vma); } static inline struct vm_area_struct *vma_prev_limit(struct vma_iterator *vmi, @@ -2135,7 +2138,7 @@ static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) if (vma->vm_flags & VM_ACCOUNT) nr_accounted += nrpages; vm_stat_account(mm, vma->vm_flags, -nrpages); - remove_vma(vma); + remove_vma(vma, false); } vm_unacct_memory(nr_accounted); validate_mm(mm); @@ -3085,7 +3088,7 @@ void exit_mmap(struct mm_struct *mm) do { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); - remove_vma(vma); + remove_vma(vma, true); count++; cond_resched(); } while ((vma = mas_find(&mas, ULONG_MAX)) != NULL);