From patchwork Fri Jan 27 19:40:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119246 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A4A6C61DA4 for ; Fri, 27 Jan 2023 19:41:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71C9E6B007E; Fri, 27 Jan 2023 14:41:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C1306B0083; Fri, 27 Jan 2023 14:41:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 39B116B0081; Fri, 27 Jan 2023 14:41:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 23F156B007B for ; Fri, 27 Jan 2023 14:41:36 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D4BDB8070F for ; Fri, 27 Jan 2023 19:41:35 +0000 (UTC) X-FDA: 80401598550.24.64C12E6 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 2E38C180029 for ; Fri, 27 Jan 2023 19:41:31 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=n77zu0oR; spf=pass (imf24.hostedemail.com: domain of 36yjUYwYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=36yjUYwYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674848492; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mDt1g22Fa9gYu2Kb7/F2UNWz7p2znEfhohgEMazkZAE=; b=oAnUozwryKPJZYsOHPjaJAC3y+r/EEsur/orpYvjbdi/ql0nGwc4ADB64AKId6QFjNxkVL BdBydALDJcruuUPKsVjflqowpqYyVSNkV3jBYG43HvlQxFn1ts9rThpMoGjGzYnDHHZjz5 twVgeNsZ7XR7qPmmOwCQEYMsNHOEEl4= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=n77zu0oR; spf=pass (imf24.hostedemail.com: domain of 36yjUYwYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=36yjUYwYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674848492; a=rsa-sha256; cv=none; b=Mm++BnXOKL10HFIYsuTnlz0QFGudYmNLVa4JwB+9T+toP6b5Kb+xIk7F1taq04hNPt19Nb 5JpEr74QnNot47Geiu5gdDVUqIMKRavzUYJE2w59i4oK1rSBK9Nxt71QDA3qXUPhfQvgjE i226nbPuPkNJqzsCobRuy0SDF4Vtqs4= Received: by mail-yb1-f201.google.com with SMTP id z17-20020a256651000000b007907852ca4dso6388189ybm.16 for ; Fri, 27 Jan 2023 11:41:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mDt1g22Fa9gYu2Kb7/F2UNWz7p2znEfhohgEMazkZAE=; b=n77zu0oRsJL7OvjwFNjS7T1FAOtu4hdvxpK7kq3XthnUipBv5szcKq0gW6Eh2qWZ+N VqA54npBtuI1r6BsRB8yCAVHYO39SuVCqNfhqCm/Ya4OMopwGqTAxzjzqXFTaLEmWefn G7d9A2dlsE9NEs9J39+IVmn9jXa2H7iuxd7eUv1r4SdiVN/BsV9E4rX7Fx/7SJnqIKhp 5pljM+LwjnmUAESpDExSAXXjwcvSQjwdpBA+epV9jtEBYonrLt6ilkcmvKbIgLrFIqhL dWwHvu2PRt1rquBCjgEA2IpAdLCPilrM/O5s952uzrtLUUUIX7VUyMYpZs8xK3jyFgDu 2ztw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mDt1g22Fa9gYu2Kb7/F2UNWz7p2znEfhohgEMazkZAE=; b=gQamu5T5hAgvtJEbwXuY/BJXk5VU7Y5CJyzZYWDHI2EoWqUU8ic/5e7bWQOTSPXa89 fkRL68745zaYFR/Bw/fEf4BO6LAur6MqUeYitDam4nq28zV8iR2/eVy+c6Iz17vNyRJG wDlzF10x+0tCgKaWogv6Zc3fXahE6ALpM+kqnUpcA1bu4vlhnKz1TKekNThlIRvbMkUO 3H69aFXThIbZiPl0rZLw6pOWyOh0s7vVsKt1RCyou5sgJXZspBOLdaDskqLb4R5LB4Zy 4fBERAwt8NmNHVv33VkJ00tygjhgzMItrTDiDgjtFgWAlR7Bh8VBsLGhPvMxmjXhiylA Swcw== X-Gm-Message-State: AO0yUKU4H6K6PdpfcrzdWQVULXUuqGWoo45chr9YH3pWSroFfU3lAjyj OMhmItvQTA053Eme8f1e/md62qSSBKY= X-Google-Smtp-Source: AK7set9vKZUOyTcPurWe/bFPJyhfVHuV/nLiUj+ueaBvjKNu+svIA1blUVC8f6GWjxXLSfMUdNgMQcXmD6M= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:354:0:b0:80b:543d:2106 with SMTP id 81-20020a250354000000b0080b543d2106mr1234365ybd.10.1674848491227; Fri, 27 Jan 2023 11:41:31 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:44 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-8-surenb@google.com> Subject: [PATCH v2 07/33] mm: Enable maple tree RCU mode by default. From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 2E38C180029 X-Rspam-User: X-Stat-Signature: rejdj743ft4zjx1qbghjjhsfnitxmzy5 X-HE-Tag: 1674848491-612304 X-HE-Meta: U2FsdGVkX18OWEg5cCWDUyKZkaGGhscUDl+bcZngrpt8WCTz9Jh5ULShlYuHsHReaCy1gya3r7HetSowcQzeX3U0kMzRn5EVTVw1aEFv2pKNleGhxd5cEKsKwCm3hcJn9v9o6K9xN215t+dTXN/Gy7n3owWYDIvFRiud5UKTfXFFzOq+42MkQqDkQuKbVCTy6PkrbNpZCM/MxPzWzDg4e7qqzFNr9nYpRS0UWFAHdvx4lks7P3kz+RS5nCFMik5PT2Fxv76aMNrAxHmJxYTKSyUPIxu52xWS1Vrr59j5jDEmsCtnk6VSvkq48Gcjj2pAS9EmCQN4udDswnal89D0jlO9KhZh9kTCCyI/XrEnWVpzc9xoak9MC8RrqyU/GxfAhSBvaCABTQtwK1cFoUvtKeH6A2piuo5MqQuPJstq4ykWYB4C1+MossLXRPpkKu6Xjj2RjDEE4TiMECBGOgs4tiW+BFdT+HIB44YuLauyOOXym2wlVNUCdLFRQ3xGIhmU1fhIijyi3DfTB6FzczO6cFKFt6WBwHIU372RquoEAQ1PGL29aty6huKDMbDy7P2WtjeCxTIGeoWQBYy3W2A2i2/3n6aj7IfAIvjMPKHzOjDk4A0S81S+cydcwsN95xCOx3voNgo05wZtgzX5Jad/nwfZotLHlmg7udfTvoj1Kv/Ph/oGj3YPUY8xfeXPz4C6mVVmm4smPPPvTbU2Mn5+QpUcCHZcDPQgR5IWAeNZ7kWRZ5uPA/ATvPkKgqocIVBeFZuf1l6C6LSRlAOKUmQuf+vamZtB2RnMoyKXLxwfRbaVKQI8EgXRC+UK+TudI5t4FH9P7dIzFj80VpXTmeiMeH7DFfgbsi/M973dogBrzbV+YzZqSEo2WXkDjHH8DBZqkQzLnSHK7IfiSDRNFkp5r/76dzVG+kzMxF8xl4mxa8qTVYtdyWDcNdwcWrIIpQ6zDSRBWlDjpzi5aaUM8Yu JxTjzgyc RWTUJW3zzclt1V+3H7F6+565wETw97zUP/PfPO1/6H24zhznHiqPcUZ1/0ph8lUDd9QhJs1RSsrKYKIZTLJcXqwTkT08jUZhJlPb7FtgYF3B+pYJMxMWdhls3mcElV+zpDOU5Zl/i3P6IoTnbIjQv92EBz48Lae7oqQwkwg3HP0KdF1xJV79i+ODkX9C10MRgkM0pR1cF0gOizDhZyk9o2L62AcLzHky4Q6Nxw5zHigFLRqH1BJfGkhAra0/06FGEK8E+sXQNM49osAwrCHQuImEEuBP8GkEf5qqhN8D1Cgc+EMy9+Ku/9IZtaf0Zw60QvN0OQHvQECFE8rBmIrylve3qLuh8unDSIbgD6ZhMcpTTvQkOOn0u7X6GI3tGwd8m1Z4gfH8rYjgo5qQf0wA28mZreFQLxel1FFoPM63sExLzie/Gn3R0jCBQ4frc2h0cQoQghLQ98+T5Tfa7KHuUnGaiTAZQNA8BzUnRQfFxEtkug037LFWL4ExgBaKCx+siluWo6zioSlBgggQoPtXbdXtBRgaQzu5nL/TDQNYM74FsOTTOFoBp2T+Ag+AevTQbF9Zl/5OTq5noh6er9HgxzhrK7e9QpwyGd/YSeCHoW4cF6cM0fxoEkUPFp0yFhyXFrgxzTTowiGzWBPT6Nh/UYg1AEg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Use the maple tree in RCU mode for VMA tracking. This is necessary for the use of per-VMA locking. RCU mode is enabled by default but disabled when exiting an mm and for the new tree during a fork. Also enable RCU for the tree used in munmap operations to ensure the nodes remain valid for readers. Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 3 ++- kernel/fork.c | 3 +++ mm/mmap.c | 4 +++- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index da983aedb741..8410c3052148 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -768,7 +768,8 @@ struct mm_struct { unsigned long cpu_bitmap[]; }; -#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN) +#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN | \ + MT_FLAGS_USE_RCU) extern struct mm_struct init_mm; /* Pointer magic because the dynamic array size confuses some compilers. */ diff --git a/kernel/fork.c b/kernel/fork.c index 5e3029ea8e1e..5f23d5e03362 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -617,6 +617,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, if (retval) goto out; + mt_clear_in_rcu(vmi.mas.tree); for_each_vma(old_vmi, mpnt) { struct file *file; @@ -700,6 +701,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, retval = arch_dup_mmap(oldmm, mm); loop_out: vma_iter_free(&vmi); + if (!retval) + mt_set_in_rcu(vmi.mas.tree); out: mmap_write_unlock(mm); flush_tlb_mm(oldmm); diff --git a/mm/mmap.c b/mm/mmap.c index 9d0dadb55e9a..9efe13d36df7 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2277,7 +2277,8 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, int count = 0; int error = -ENOMEM; MA_STATE(mas_detach, &mt_detach, 0, 0); - mt_init_flags(&mt_detach, MT_FLAGS_LOCK_EXTERN); + mt_init_flags(&mt_detach, vmi->mas.tree->ma_flags & + (MT_FLAGS_LOCK_MASK | MT_FLAGS_USE_RCU)); mt_set_external_lock(&mt_detach, &mm->mmap_lock); /* @@ -3042,6 +3043,7 @@ void exit_mmap(struct mm_struct *mm) */ set_bit(MMF_OOM_SKIP, &mm->flags); mmap_write_lock(mm); + mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); tlb_finish_mmu(&tlb);