From patchwork Thu Feb 16 05:17:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A83AC61DA4 for ; Thu, 16 Feb 2023 05:18:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A942E6B0073; Thu, 16 Feb 2023 00:17:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A43386B0074; Thu, 16 Feb 2023 00:17:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90C226B007B; Thu, 16 Feb 2023 00:17:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7EE3D6B0074 for ; Thu, 16 Feb 2023 00:17:59 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4B05CA61D3 for ; Thu, 16 Feb 2023 05:17:59 +0000 (UTC) X-FDA: 80471998278.20.C8B5FB1 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf04.hostedemail.com (Postfix) with ESMTP id 9041940010 for ; Thu, 16 Feb 2023 05:17:57 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=DtrBb0OE; spf=pass (imf04.hostedemail.com: domain of 3hLztYwYKCNkNPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3hLztYwYKCNkNPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524677; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0nC1+Z5xJ7W1lTs/T0LqT1g+dKiQp9Lc6m1pbhrJE6M=; b=LpY43sL2FFzNwtYu6DKECnL/wK+EkjK8HAoLxgAJq/9NBZRcAtD9rigOZlCNpGXZocJZFW qI8LCF6oP6rmLsLtHq4xfvkPfIvgkBQSIJdnkZqNGwMBP6SGsvw9g4rOVApSFNWvgOJJnw P5Iok2qAEfkV5UyRbqC3WSU0KJKcVm4= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=DtrBb0OE; spf=pass (imf04.hostedemail.com: domain of 3hLztYwYKCNkNPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3hLztYwYKCNkNPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524677; a=rsa-sha256; cv=none; b=44PSeNx2wUN4q7mcqfCCaP/IDEEcGwjIdl4WWF9JZqZsp9VvJnLO77UMerNh80OcI3zr02 XxLiRzM/cz7WvC8xscWWqNOs0+J9XYdAK+mUKawthvvJGIMrwd8CSfZm5pxYjQAcyyyzJZ dDjHbtloS17kbazwk7u9AQdqpUgKETY= Received: by mail-yb1-f202.google.com with SMTP id a17-20020a256611000000b00889c54916f2so850946ybc.14 for ; Wed, 15 Feb 2023 21:17:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0nC1+Z5xJ7W1lTs/T0LqT1g+dKiQp9Lc6m1pbhrJE6M=; b=DtrBb0OEqSibZGKM+tGrekgia+czdJCmNvwxB9nTfvwGTIHXKlm6TS9Q1D5NfIR4hg c3apVnOjZnfZCRE1YFzTHLthbM0S/IVW1BccnLNsJf21OgviYfhBVHNRB8wtAupWJwTS 1TEYfIbKZCSoMpt+Js0TzB9iPZbPOfnvL//puIai0IRqYdE7nHNdWFvMQyJ5Dt5csXIh 43aWlLJbXNtNaZid0iRHPyqJRyFRPoBCCZ5nKLzs/n4kbMJKA8/P4QRR3m4y665oyagy kgaLTr7SQL3GQMCbK2ePGMttOB2X0C/PFJbSA9edFZkog1VQvtGlD+HG94g+afV+vkdw M8NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0nC1+Z5xJ7W1lTs/T0LqT1g+dKiQp9Lc6m1pbhrJE6M=; b=uho0ebqtJIJlAo9erJFDTeKffoFukQguDp46vd/l7yk2eKq3mWoT0m3eZL9XCu46iu 5ZJcqVyWPwiAmt3AinoKSFHwg0f6E/RTWZmnwzqIdlrGt9tK+ZxdesAsQHeBNmbFtBMf nJekS8MH00dPByOdB9mQVCdhOpPj06BlrjnceirYxmKIFoSwsMACi6i5RjKBlG0UJcU8 c0wGOPEiIdJOiMlyvZpaaWCeLNJFxYyHu1sW/aIYEoz2GTwgAt/cOKEEIfakbeBYr4NK duTlENtlW7g2Pv4GWCOrALL0rw6LgSfum+Dl1DtO68XRX8wp1goHvzclzu+qWhnEH8qQ zLUA== X-Gm-Message-State: AO0yUKUEQsQohg6FCLNOg7vuEC7xIqrVnq3bZYI8Y+5FTMmBqTioEwXV TY6syIx+gjjy2aLubFITl+br5dGwgwA= X-Google-Smtp-Source: AK7set+Y8Yn0V0huCQq66Ej7hgCVLxUYKYuorVk92zLqUUDZhSEdy+eB6kBj2xP313LsIxhb2Sq52k6rqsw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a0d:d88c:0:b0:533:86d8:a551 with SMTP id a134-20020a0dd88c000000b0053386d8a551mr60897ywe.11.1676524676787; Wed, 15 Feb 2023 21:17:56 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:16 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-2-surenb@google.com> Subject: [PATCH v3 01/35] maple_tree: Be more cautious about dead nodes From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Liam Howlett , Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: tbd6eh6mnp5sqth9fpcehd9ze53fhqma X-Rspamd-Queue-Id: 9041940010 X-HE-Tag: 1676524677-806775 X-HE-Meta: U2FsdGVkX1/y1GIJBHd/keDgmUpy+ug8yCLapCXOQOm0YVXycR0qbM0HZpy8NAvHQjDjxGp3svPn804NKTGLj+WIbZooCcD3oYZwJ3bFo+rrA7D1kl5FSahqHCbvKTp9NSIXsf7PPDBls3twoLh8U5eGyq/Xy6K+gmKv1bvlTkEWNo3ytfGFasBC6cuFa7d6s301q4JegR0veBRUwenubM5XDPSNd+cip7Rg7z1llo6FgDLmK0oCU6QDYmzj78qDk3vH8t/SmIYC0nF87HA+kX5pIeQjPFFNhEvVpYOLcXM5Gk5ixSJGNzjEEX0q1UCW4Hkw4RARliXLF3CWEYZDuq1SSMW64Or533RTIBRpgTA48ko+7LxNh2B6hLE+mM4xgW4d6gF82mL4sM3zCqKfR8uHAk6oyqJKAInyoV2Ful8NU53qPYU90Teb0JzATRP2UHJwQDWU5pD5NeV3OkCzxMl0eLq/dyuf1l1w3QvY5pcRK5S1Z+ZQBfsb1E9EBN5DtlgOgav3KKHdi1LGgsvioKz+0d/rtU93TeZQeJ4nbR3yIBSSQMbddPB7N6Rb1KrzfoT19Zvk23nbkKvB19q7q2qS7DZmrJ7B6SZ1J2Pg7aXeE6r5txYB4GcoIKezNeeMq5TpShMkVB5fdqm8GWJCJ3UXTrXhU51bhIlyPtLE0y+PhM8seu/TakYm4dgmvRhIzsQCvJ79cHyjsePdHAjtd3RgWUPcd1fg6Jp8+DpTU108OW7hCt1HxASo+b1eYS/o2+0O5J8JDtSmWLme6ZDs8JLxmccTpRN3Ej6Uf20q/NOKBOjz/HF2vlt6GstuCjDw5cbVV4e2ewfuuNgn7AKWUwg3H7f1KfEJK96ca0FJEMu0eP9iMq4QyuhXjRvtMi8nlcC1q8zT50bb0GvGmsue9rQqOwEs/++wZkSqhuM/IwnDULgYkr6CWMBUHJMEVks7Ab5ABO0TED3dakE+VCv kxFdo3+h +kOaF4oZ5Q+R1Qc1WY3oh6tIrDhkDh0CcRvo8AS6jP/YAn59FTlXlvFeynzSY2QtchoWzfhLXZMs55n2E9/PXjgEybEmshjfheSOxIrEYaPL4tT7HOlUlIpgcW2c9of/JF//GNym+Zu1IT8Wj6VgM+FAVslfJMfAiR6HMSjDULqYYkVYojtZYh3NagNY0qjfS68OTfnxyQxK7Nv++owQVJmMg2PXf6Svb0R5NLISL2xBAmSeCnTDTXUl7Xx9rfxkVE7erK85QtiodYqjamulKUXGTLBSWWGyqbvW4Lu2HE5P4jnJO4MYQuAOZrDxp4j5x8gfOVAzk8EBj4GL4IKpoUDUW4htj3INPviQZuscGZDbtOKLf1LJkPLf2Iz4tEyUgqNsgUqHKEdzAth59DK3lX+yLNjmjVlKD4q9nXlZmqz3S9rOrxhSMhdy4OY5YT5kFxvJKwJsigscFoGNAcNzThsI7suJQ9cdu+sxLvCJJ/q6ZiW+g0bRC+j7pl6zroj2M2yrobh9PmbDwFa7fwVBlRVPTVWNd5aW80lA94/Dx4HzABUgk6SzT3WEQHkuXFRW5j+74n4KyEwV1N8QElC2ZQOipNWe8boYXGDwk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett ma_pivots() and ma_data_end() may be called with a dead node. Ensure to that the node isn't dead before using the returned values. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 52 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 646297cae5d1..cc356b8369ad 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -544,6 +544,7 @@ static inline bool ma_dead_node(const struct maple_node *node) return (parent == node); } + /* * mte_dead_node() - check if the @enode is dead. * @enode: The encoded maple node @@ -625,6 +626,8 @@ static inline unsigned int mas_alloc_req(const struct ma_state *mas) * @node - the maple node * @type - the node type * + * In the event of a dead node, this array may be %NULL + * * Return: A pointer to the maple node pivots */ static inline unsigned long *ma_pivots(struct maple_node *node, @@ -1096,8 +1099,11 @@ static int mas_ascend(struct ma_state *mas) a_type = mas_parent_enum(mas, p_enode); a_node = mte_parent(p_enode); a_slot = mte_parent_slot(p_enode); - pivots = ma_pivots(a_node, a_type); a_enode = mt_mk_node(a_node, a_type); + pivots = ma_pivots(a_node, a_type); + + if (unlikely(ma_dead_node(a_node))) + return 1; if (!set_min && a_slot) { set_min = true; @@ -1401,6 +1407,9 @@ static inline unsigned char ma_data_end(struct maple_node *node, { unsigned char offset; + if (!pivots) + return 0; + if (type == maple_arange_64) return ma_meta_end(node, type); @@ -1436,6 +1445,9 @@ static inline unsigned char mas_data_end(struct ma_state *mas) return ma_meta_end(node, type); pivots = ma_pivots(node, type); + if (unlikely(ma_dead_node(node))) + return 0; + offset = mt_pivots[type] - 1; if (likely(!pivots[offset])) return ma_meta_end(node, type); @@ -4505,6 +4517,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) node = mas_mn(mas); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + mas->max = pivots[offset]; if (offset) mas->min = pivots[offset - 1] + 1; @@ -4526,6 +4541,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); offset = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + if (offset) mas->min = pivots[offset - 1] + 1; @@ -4574,6 +4592,7 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, struct maple_enode *enode; int level = 0; unsigned char offset; + unsigned char node_end; enum maple_type mt; void __rcu **slots; @@ -4597,7 +4616,11 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, node = mas_mn(mas); mt = mte_node_type(mas->node); pivots = ma_pivots(node, mt); - } while (unlikely(offset == ma_data_end(node, mt, pivots, mas->max))); + node_end = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + + } while (unlikely(offset == node_end)); slots = ma_slots(node, mt); pivot = mas_safe_pivot(mas, pivots, ++offset, mt); @@ -4613,6 +4636,9 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, mt = mte_node_type(mas->node); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + offset = 0; pivot = pivots[0]; } @@ -4659,11 +4685,14 @@ static inline void *mas_next_nentry(struct ma_state *mas, return NULL; } - pivots = ma_pivots(node, type); slots = ma_slots(node, type); - mas->index = mas_safe_min(mas, pivots, mas->offset); + pivots = ma_pivots(node, type); count = ma_data_end(node, type, pivots, mas->max); - if (ma_dead_node(node)) + if (unlikely(ma_dead_node(node))) + return NULL; + + mas->index = mas_safe_min(mas, pivots, mas->offset); + if (unlikely(ma_dead_node(node))) return NULL; if (mas->index > max) @@ -4817,6 +4846,11 @@ static inline void *mas_prev_nentry(struct ma_state *mas, unsigned long limit, slots = ma_slots(mn, mt); pivots = ma_pivots(mn, mt); + if (unlikely(ma_dead_node(mn))) { + mas_rewalk(mas, index); + goto retry; + } + if (offset == mt_pivots[mt]) pivot = mas->max; else @@ -6631,11 +6665,11 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, while (likely(!ma_is_leaf(mt))) { MT_BUG_ON(mas->tree, mte_dead_node(mas->node)); slots = ma_slots(mn, mt); - pivots = ma_pivots(mn, mt); - max = pivots[0]; entry = mas_slot(mas, slots, 0); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + max = pivots[0]; mas->node = entry; mn = mas_mn(mas); mt = mte_node_type(mas->node); @@ -6655,13 +6689,13 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, if (likely(entry)) return entry; - pivots = ma_pivots(mn, mt); - mas->index = pivots[0] + 1; mas->offset = 1; entry = mas_slot(mas, slots, 1); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + mas->index = pivots[0] + 1; if (mas->index > limit) goto none; From patchwork Thu Feb 16 05:17:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F91DC6379F for ; Thu, 16 Feb 2023 05:18:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 346D26B0078; Thu, 16 Feb 2023 00:18:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 318EE6B007B; Thu, 16 Feb 2023 00:18:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2070F6B007D; Thu, 16 Feb 2023 00:18:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0E0CA6B0078 for ; Thu, 16 Feb 2023 00:18:02 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D4FF81603AF for ; Thu, 16 Feb 2023 05:18:01 +0000 (UTC) X-FDA: 80471998362.10.41E0EBA Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 2F862180013 for ; Thu, 16 Feb 2023 05:18:00 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=gSbKQNPX; spf=pass (imf24.hostedemail.com: domain of 3hrztYwYKCNsPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3hrztYwYKCNsPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524680; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eiC+Iexfup+RTroKLCxxsKt1sRroZAPvjqioHV2wMiw=; b=zl+M+WYUz9izmxbpF8C+n0REjJVUIq8xILwNHnwyqiSVYU9q4ymzyhEPPNE1YyGVnSrvfH GHfcZZ6hbr9ruCJntvKSwtOwUb/ghl9c6jWOvLIeFBe6lZ9W2fruwOYerYUqRvMU9NA47a 5EHCGmiSyeGiaZa0GZ/VoIbJeKyNwqE= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=gSbKQNPX; spf=pass (imf24.hostedemail.com: domain of 3hrztYwYKCNsPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3hrztYwYKCNsPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524680; a=rsa-sha256; cv=none; b=yxDApfKA0cPBKvyPjiMAebxMSsiKXXvH4vGJVTs4gwTFiAwij+bQQMEBKLob2suQoXulmR aO4Lu7Mt5bJzLoUmjfbXI4IT1GXajhQxIgV1E8iAKgp1nykquUk6dwYo1pYcv7zW9ln/TY TcPf15huTzXrnPyIPNIuwzlXpkrnD/k= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-52ecbe73389so9085177b3.0 for ; Wed, 15 Feb 2023 21:17:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=eiC+Iexfup+RTroKLCxxsKt1sRroZAPvjqioHV2wMiw=; b=gSbKQNPX+j8W3tzxYZsRNAblkFTxWANtMyo2LSqGNkakVHWziV6JTyqlohrqDf6nLO 7F5N7WpIKnXsbFPEflJy25rhat2iqZIJRLZvV94iATncucILtc8zFlPzaxOQpEoApKoJ RWEtB6lTCeOcIkp8qbIxk4V683W1L3zeK3vTsHv9Yla4n9gJP1xuJQs99Ms5w5HF3olN Kli7prX+XczaCsAJ+8hVggWwSYtxp34C+A6/4yuDSchSQcAT4VW0lygmuaxa708qKR/5 zV/LK3WvWxvtgplCG5tRDu/7iPKMsNWXdzFiEgVvRQ3tE0ZzhzdCi73gDraIVPZ+cD1H 1RBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=eiC+Iexfup+RTroKLCxxsKt1sRroZAPvjqioHV2wMiw=; b=qYlGsU+q9CPHyYCzZjx0K5qLr4ST8a2Unu6fApKBZSG987M+FM2P/tRCbnGJgjeb8D 7HS7iMGIF2cHjEK4Dku6lnvT4zfcqLMjV/ID5bnYL0B1fd8vsAZu78qDfrAJ7ptkyi1J tYo5dcDZbbEow1s+5ABJ0TYZF693WOx/6cEGgumXqBqDLleyDr5YsVLcTDRcV7i+YefJ K6p1/Yt+0yiygRAh9ndWvhZBjMVQ668v79PqlJQk3FBXw+EFxiOC9VjHx6zHBsTtGK+b rtNEW0Wy1VF2FUBiBato4KjfoFDsVozinVXcAsUnWbnXfJclAWYM4emC6E2rywAOrdw5 FcpA== X-Gm-Message-State: AO0yUKW8PZXNVC1j8Isr+C7bDPyRJ/+rBUS1OS61MfSf0mSomoB1eiNZ AjSBgcUQaDYXlgAIm1/34sszYkj9F0s= X-Google-Smtp-Source: AK7set+h9VKChaGyyozNEu9Udu5uv7zPqiNZXg8WkDwG0C713PEP5x3O9LtVk8xkCPSk1ZyxP2XheyHDpnY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:10f:b0:855:fa17:4f5f with SMTP id o15-20020a056902010f00b00855fa174f5fmr78ybh.1.1676524678798; Wed, 15 Feb 2023 21:17:58 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:17 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-3-surenb@google.com> Subject: [PATCH v3 02/35] maple_tree: Detect dead nodes in mas_start() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Liam Howlett , Suren Baghdasaryan X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 2F862180013 X-Rspam-User: X-Stat-Signature: swnc9wse95isadipzqf8ag3dwsy9566h X-HE-Tag: 1676524680-804685 X-HE-Meta: U2FsdGVkX1+bLoUt2cNMUgdqiQMKJ2s8mJsjunkEWhtf2MpKR0JIfxzhanH7dipAvep0gMaEaTykgNLFvawj5EIHgIE+EOzxN40NlVatoETqBkaC5udmd37TOrQXbew9RRhJvmLaC2XYEM9ATN6bObrI4d76m7dJMcaDIkZ5rdQBzhGwPexTXc3H/1x2lsN/iJ7/u9+cQvpLaui9lrNORUIlbbJYyb61WMd/t16Vgu+KDYwL+ieo254/rnrb7ntFqdDr2N/4J7+cOaf4arXnlln96iWG/XCmDuo3ZosMMvIWgCzaIxy4m6G8Eq8GJKrvPawbMPef/jgtvDZEmTfIbccgelJJogOQl2S3tF6tBTVrrEyfxyA9lRqAVJQQ5GGAiSjbREShWVwGU6m8DtI7NK87IRENuvJr2QKZpDMQhYRyax1F2E1033OnBPqctTUcJtL6hIKJYizRovERsnGqiWveVKZCqETu0SjsUSFJG/MfKxoX1wvuPRf7Y1AOhfEoqJKwzpoMjypxr81Mn7mXhiqAU6kKn91gzqJHFb7BmkZQKEqyad8ca65oyrLk7m0e8tgRd05/WEbKjKH4mxBu7TcVHRvVM2frwG0rVckG0gJ15au7rrlD2gKOz3t40+QF7PckAsFSOqE+SgDnl9G4l80GAFUX5Cy/ih59VEKQR/Wu9u3M6Rbg37/1zZ8yN9O1MsjSM1Hmp+70syD6V8M7iqv6c1MlJRXDQDxzeA0fJdZ8LbY2CuueLVolyYlkco7SWJ2WKvrcDXzHwlqvsLuQ6Snble+05owDfeUS4fz4RsHVz6QG+sECN6L4/UBd+kJ6x3ezncKb9SfgxT00QWPN7Rl0Bg1ftl60Dpm2tcsqSNwdOm0NMdKv5vHE2kK4FHOccEwFxRtetgNNyMI7y+pby+8OyYFtrAQQiKRIOfvFUh7pkN5D539Z9xgk+zopm4JuZTKnw6aMl+ZGXfPChoq MPTgL0e5 W62QkcWcOB9Q08tLlPru1VZ/tdRY8Z9PlVSn8MWFXTjH65/uGtuUTcD8X/XDAh9WXkgQdcbJTcpzoYobbgb2cTy/EWjKL3jNAzz6mIcqjHmNgsQSfml9RusZAONPQBP4HS/MDGlbL+O+obZhkZyLDriIeJf8gcHHxgKf9ym5h3lfgNxE00VNoI8AIu67eTE55N7oaK4AMwC6JgGrWjY3cVGeIc0y9duVLvFPqeIVrLPRFD/IZyP7fCKJhLvXdlVLrbV6+XFA2qO8wGgUX9wGXDfUq+yc+cx2gnIr3/KXjXNyY4seiOa15+bKBMtEsfLn2QHCla8y4mmMyBgO0QzFqFQ0Y1rLbkT1HkvSDOpKtsifCY9OHxkzRXlUuANoL/E8D38iS1Km1+LsI/lpeXVN9Wn23vOy7zasimTwnycTs8bGzkdEwTkcEEx9RyV7Co3UcIwiCWBbfqi7vAicsM0LCPffJb6Bx/mjgikA1SH2po1XHcrz9/gfQdaznqRQa19Rim9WhteMFIdrPlDJlEZOtNWO7uH0LMn+uNdBeVkQC+1XZ1WZU095yOTf0ou+vmemnqrsqqvYwAdxPG+SrDnpW3eP5zk8Iqgq0lQFJ0LnKAMARzDZSfIUOV6j7DtEWwWgulIUZ0MyUnVo+hMTNxD0gkbdZ11LpRaCR+A/JHeaFXVvAcd8ju7GsovYR1FI+ndA4k33W X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett When initially starting a search, the root node may already be in the process of being replaced in RCU mode. Detect and restart the walk if this is the case. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index cc356b8369ad..089cd76ec379 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1360,12 +1360,16 @@ static inline struct maple_enode *mas_start(struct ma_state *mas) mas->max = ULONG_MAX; mas->depth = 0; +retry: root = mas_root(mas); /* Tree with nodes */ if (likely(xa_is_node(root))) { mas->depth = 1; mas->node = mte_safe_root(root); mas->offset = 0; + if (mte_dead_node(mas->node)) + goto retry; + return NULL; } From patchwork Thu Feb 16 05:17:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142496 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AE3DC61DA4 for ; Thu, 16 Feb 2023 05:18:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BCCF46B007B; Thu, 16 Feb 2023 00:18:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B7DF76B007D; Thu, 16 Feb 2023 00:18:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1D666B007E; Thu, 16 Feb 2023 00:18:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9098C6B007B for ; Thu, 16 Feb 2023 00:18:04 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5788C120B37 for ; Thu, 16 Feb 2023 05:18:04 +0000 (UTC) X-FDA: 80471998488.26.086F9DF Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf06.hostedemail.com (Postfix) with ESMTP id 730D3180004 for ; Thu, 16 Feb 2023 05:18:02 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=JaAJMQvk; spf=pass (imf06.hostedemail.com: domain of 3ibztYwYKCN4SURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ibztYwYKCN4SURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524682; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UHRaAOWRO45il7vCWYolP2vMSHNJY77rtpcEDPLreFo=; b=ae/mMLC6S3Ghg+RqXs0kDofxV332ul23JULr0ncp8t/UjsuexjQsZbMyIVd/4TzTElaj52 kF3Oq/FshT8Fef6jfGi3zyGfijmCoS2yhfj023pKtb+AEMfq27MDLvo0R6cVTnhQE/L6am EMOTkdOy8rIGNOPVCHh3xT+U72hk1MA= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=JaAJMQvk; spf=pass (imf06.hostedemail.com: domain of 3ibztYwYKCN4SURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ibztYwYKCN4SURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524682; a=rsa-sha256; cv=none; b=U12WXvFaG6ESZ2LpY/uAaKrswqD8QgJTjY7EF2JEr258o2POzN8IuFf/Toir13gIH8+DrK Bxb3+f9fImKEXSTkNtVLmWPowLjcKnZdWit+gHgZo0U74v4oyoKeHW2wSTe4PKLHgpfGPa VLWDyZklM5mCCd33jOZ0GO+iXaMh72w= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-52f233aff21so8590567b3.7 for ; Wed, 15 Feb 2023 21:18:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UHRaAOWRO45il7vCWYolP2vMSHNJY77rtpcEDPLreFo=; b=JaAJMQvkQYkAsRG/T0d+zxwkrl7mq+q9FZsokrXfiBXR+R1ZXxx2vVNR2lMSqf4OC/ WN5L5m6K8l41JtRKI2HZbVfjAY+cjnkF1BJGHx4Vp8UHSVK5UGOhTfSvkmRDjsxq+IrD 02V2mM9EdjYNQiXsOwUhDaiCYn5z8ljJQUJeFnF2XY5CF0WF7YFz2opQhOGyVfb59slI zMkZhBxatB6Jt2s9icMmk1QAWL7dqweRI7j0fEknZ3XaTRcFBLrS6UTE7S7Q4BJqI8Vo VXsm9/T42jqn+e5iri473tsxrbcsgusYZ9oNJzgxHuxj6F/D7b6ACgioO/upEa2Rro7t uyJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UHRaAOWRO45il7vCWYolP2vMSHNJY77rtpcEDPLreFo=; b=RBsPB0OnaI8r46s8WwyaFpCAAW3QjDMtR7BZBvUfRuZKYMEFr/0PUB/FQQU5IH5J69 eyswE8bNugdNNXwOT0bxHopoKmjWZePh2MAhpIhmK+9qc+6k4UkeBA6a/krVo4VvuQwv VDmphEOO3nKVYUG6Nmd+6dKigNFELrM4Aogf6Asw/0x0YjP1YQF3KPOlMcx1CIR3yC9O tql6u/8e6pwC+GN5iNgi0StmYXSrZXHAMXTp6eaATN9Y7yv2UUwYEYKWc1D95m8fDKKM tJ6ylBGG01ygz5Cn8VzShL9UK8wLgVXG9jiL1F/Hj7ja6dld27qXaidFYPBmBzjOnccc ILNA== X-Gm-Message-State: AO0yUKUTcoztn0PaqwNLxiPgSjdwjVECNR5tVw6UNpcAwb/d5fCy9Ply jEtBAxKkLGDRLVuMP+gzS+pvtYCpQ08= X-Google-Smtp-Source: AK7set8Uq/nZgRLZFfJuxWYQ4ij1Dw1PY56gBuOAhM9aSZ73J6nsbhc6qz5TmHaa+yP435YL3FPEkHcMzr0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:10f:b0:855:fa17:4f5f with SMTP id o15-20020a056902010f00b00855fa174f5fmr80ybh.1.1676524681354; Wed, 15 Feb 2023 21:18:01 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:18 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-4-surenb@google.com> Subject: [PATCH v3 03/35] maple_tree: Fix freeing of nodes in rcu mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Liam Howlett , Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 730D3180004 X-Stat-Signature: 538ax88de3xkccy7g7bf4w5zuwjfxzcc X-HE-Tag: 1676524682-374857 X-HE-Meta: U2FsdGVkX1/Iw9XXc6p3ZqPizH+CGj7mUiJtsJUF3aZuNyZNE0PsaW+AefknohmpVvWVYMW3jlSx0usP78Owg2MJDKMB8bcNkZjZkvIwlXlFdFZbFJnnKoM5mUAytSuxIynkeOSKq5zDm3rNeskcpwRlVTdHs5k0W1rpV0OeeJo341u3FaYOKaTHGe2qJAEUrc+/AhWyUR5etnMeGdcL3e1p+SfUhIh4FxPpCh+SHduFRujY3gpI8lbzdBDgI8PnjeZ/LBOpnvJKFOnQjAEYUznc1Ot6KehYdGjcOR2Na8LMiFkDURkWBtJSrfqv8xPJCoiJgPxXpYlnc1GKg1NjnpKO/etaPRS5403gsdOiWa7GuZ7PlpdzHVz5otZaBdS4YSVg/0Vksx04gdoEh/ZZH4e0ZPes9dmeTdz0WvKRSGDhTdxTTyRalPmQsbZj19hcb5EhQuhXny5PApLNiF38dyWLk8fO6pbbnQr645kJg0ZDlBK1gx2sWwcIZf++so75v52Q4wGQ1W032/cVEQUnGh16D6il8CWDvO9SNfgQcpMNrxPkPQadvRzEaMPzeBhtK3649EvvPednOWvLdf85Jl4EM/PC8bKcNGCcx+X9weY3rGnUZuKim6JUvOdJG51iSsDqx25Gz9OT4y9UBNsnzmL4TbejuJ4bp1FbYyo1zOMCZcgvijMt/Ei9BRYSiDtOuLJGMN5ldBf5KZCuw4VS8a+P3om278lAC2RfRxvvUXroXADa8jIAeOikl9zfHIfpOXsAzae636wRqLyDz0fybcFkxROjWxVs+hrbfk1keovGMv5tK/vPTLpiXdyUbWRIIWS0xuw0fZ647tKJCMfekJ8hCdTNXpxy/d7P6Vxv8DHX4sSVEwplfHVlOYOJzIR6lT3IkARKC414YVanGQMTD/3N244L/f7oBXeyZQ37nUnBRt3kJdaWBfiGX4IbxjbYis5AmO+ZRnEq8EnwbyK M5S7dsFb KqdU3Cpdw4qrfsPMp9u5oJx4/fL1zTuzLAkpTOyPdy0BIYPa6T3+nZS2ICa6qD/hNpr+DMLFGOQzIrWXFaq429Gn7+64VJBYkIsoVMVB7z6Ygmia1WNa+7NSfKfJV7ETb0LJWa0mK3mtHYpdIRq2/VUG0v8a8PuM6n2kkD1eOzUJgtP3hWjduTBdwjJeq8AmnlZY8QWULcBv2oF7Qy9znQ9knrxE89lzskzK8CYf04Ur5LIyRztAEdUwp3Arm7lcSziO29Lq0C9qtqeTlI+FCZOjUpuP1C0FFVCIb9xPxZGbUQESGcOG1ChAksML66xGv3T9tu8l0vDIiuQys+UT9fbGK0mh9vJg6rJeScqV//vd+XVoKIeeacpSdVTAWpC9C07S75KE87oCaO9dArtViXufMKP+GYRC4i/6q5QeyOUHDlj3Rh/ie1iH9KnApooq+9Kg0fVwvqbg7muMLy3v35iAi0+zRIf7Sey/LTl2+DbVN9ujQEoU4eAEx49BtPVmvJ2me4j0QoHC54kTiYcstpcnbyGVstcc/Lx7ZrTMZvn3cGgwM5fKmPgVomy3msAZx2Zs7mNbrF3oLKVOEo/JGoukWq1C0VA2PNHgPTqWaqe1Vfh7SgSieNrFGynB91ecYzOTBBWl4RmV6HUFSKw1q7vXBUvujG4HcklTtCnih+vUVvfz9CKDtmCdkAx15Ntp8cCpd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett The walk to destroy the nodes was not always setting the node type and would result in a destroy method potentially using the values as nodes. Avoid this by setting the correct node types. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 73 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 62 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 089cd76ec379..44d6ce30b28e 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -902,6 +902,44 @@ static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt, meta->end = end; } +/* + * mas_clear_meta() - clear the metadata information of a node, if it exists + * @mas: The maple state + * @mn: The maple node + * @mt: The maple node type + * @offset: The offset of the highest sub-gap in this node. + * @end: The end of the data in this node. + */ +static inline void mas_clear_meta(struct ma_state *mas, struct maple_node *mn, + enum maple_type mt) +{ + struct maple_metadata *meta; + unsigned long *pivots; + void __rcu **slots; + void *next; + + switch (mt) { + case maple_range_64: + pivots = mn->mr64.pivot; + if (unlikely(pivots[MAPLE_RANGE64_SLOTS - 2])) { + slots = mn->mr64.slot; + next = mas_slot_locked(mas, slots, + MAPLE_RANGE64_SLOTS - 1); + if (unlikely((mte_to_node(next) && mte_node_type(next)))) + return; /* The last slot is a node, no metadata */ + } + fallthrough; + case maple_arange_64: + meta = ma_meta(mn, mt); + break; + default: + return; + } + + meta->gap = 0; + meta->end = 0; +} + /* * ma_meta_end() - Get the data end of a node from the metadata * @mn: The maple node @@ -5455,20 +5493,22 @@ static inline int mas_rev_alloc(struct ma_state *mas, unsigned long min, * mas_dead_leaves() - Mark all leaves of a node as dead. * @mas: The maple state * @slots: Pointer to the slot array + * @type: The maple node type * * Must hold the write lock. * * Return: The number of leaves marked as dead. */ static inline -unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) +unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, + enum maple_type mt) { struct maple_node *node; enum maple_type type; void *entry; int offset; - for (offset = 0; offset < mt_slot_count(mas->node); offset++) { + for (offset = 0; offset < mt_slots[mt]; offset++) { entry = mas_slot_locked(mas, slots, offset); type = mte_node_type(entry); node = mte_to_node(entry); @@ -5487,14 +5527,13 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) static void __rcu **mas_dead_walk(struct ma_state *mas, unsigned char offset) { - struct maple_node *node, *next; + struct maple_node *next; void __rcu **slots = NULL; next = mas_mn(mas); do { - mas->node = ma_enode_ptr(next); - node = mas_mn(mas); - slots = ma_slots(node, node->type); + mas->node = mt_mk_node(next, next->type); + slots = ma_slots(next, next->type); next = mas_slot_locked(mas, slots, offset); offset = 0; } while (!ma_is_leaf(next->type)); @@ -5558,11 +5597,14 @@ static inline void __rcu **mas_destroy_descend(struct ma_state *mas, node = mas_mn(mas); slots = ma_slots(node, mte_node_type(mas->node)); next = mas_slot_locked(mas, slots, 0); - if ((mte_dead_node(next))) + if ((mte_dead_node(next))) { + mte_to_node(next)->type = mte_node_type(next); next = mas_slot_locked(mas, slots, 1); + } mte_set_node_dead(mas->node); node->type = mte_node_type(mas->node); + mas_clear_meta(mas, node, node->type); node->piv_parent = prev; node->parent_slot = offset; offset = 0; @@ -5582,13 +5624,18 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, MA_STATE(mas, &mt, 0, 0); - if (mte_is_leaf(enode)) + mas.node = enode; + if (mte_is_leaf(enode)) { + node->type = mte_node_type(enode); goto free_leaf; + } + ma_flags &= ~MT_FLAGS_LOCK_MASK; mt_init_flags(&mt, ma_flags); mas_lock(&mas); - mas.node = start = enode; + mte_to_node(enode)->ma_flags = ma_flags; + start = enode; slots = mas_destroy_descend(&mas, start, 0); node = mas_mn(&mas); do { @@ -5596,7 +5643,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, unsigned char offset; struct maple_enode *parent, *tmp; - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); offset = node->parent_slot + 1; @@ -5620,7 +5668,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, } while (start != mas.node); node = mas_mn(&mas); - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); @@ -5630,6 +5679,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, free_leaf: if (free) mt_free_rcu(&node->rcu); + else + mas_clear_meta(&mas, node, node->type); } /* From patchwork Thu Feb 16 05:17:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDE63C6379F for ; Thu, 16 Feb 2023 05:18:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5637C6B007D; Thu, 16 Feb 2023 00:18:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4ECA86B007E; Thu, 16 Feb 2023 00:18:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B4786B0080; Thu, 16 Feb 2023 00:18:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 29F2E6B007D for ; Thu, 16 Feb 2023 00:18:07 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id F32121A0FD8 for ; Thu, 16 Feb 2023 05:18:06 +0000 (UTC) X-FDA: 80471998572.14.73D327E Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf30.hostedemail.com (Postfix) with ESMTP id 25E058000F for ; Thu, 16 Feb 2023 05:18:04 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=WNV0WuL9; spf=pass (imf30.hostedemail.com: domain of 3i7ztYwYKCOAUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3i7ztYwYKCOAUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524685; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bYTu7nw04Y/eGAN3i9eCzY/Ke38OAVixcrHXlu+jNFw=; b=ZNjJs+Zo68GlPYsbGQC6Ljmj8a1/jMdlnwDIhZzV8tcrmT8BurHexs+MMQEbZU6/Jk21Gd Kp7Gm8Zl4vpjhxNuDG9/nvasTRc3/FipxHg/4xaW0qzQ0gxu6LlBkjPOrfGpAAbLHsXlAZ nvRaK2iVS62TO2dppSEO/aKBJ78w+kI= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=WNV0WuL9; spf=pass (imf30.hostedemail.com: domain of 3i7ztYwYKCOAUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3i7ztYwYKCOAUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524685; a=rsa-sha256; cv=none; b=7p45xFvGi9TqZv71wMRcPnk9bYGEvnPUV227Hpc9EGPAUBvGKjXQJzeD7nP68antH5gUIr yvLWMxIoCfKm6YDsQVgiRmkUuJ7PIqWaCiVrQ+YjAi+0MX1bwwsDa+Ae+HSXYbbfQiI8W8 GYzWbPaRw/W2ac/9ylNfK5fwdXVjzTY= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-52f1641b79aso8453637b3.16 for ; Wed, 15 Feb 2023 21:18:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bYTu7nw04Y/eGAN3i9eCzY/Ke38OAVixcrHXlu+jNFw=; b=WNV0WuL9vmXLIvOmuVEVNJHnS2Qfi+AtiXnEC8qLvBZk4nQfRysiQv+n6V2TYm/oOI 8PVrbtoCrNJAR827RYRUq+srgPH7ettT9CkewSYs2c20qjx3DSR6OCNLU8I9VtlWJ1vK BTnTnTmkn3I2d2LJu6019qpcNY3Rtyhwbcu+BTERQU4CpKL9XQ+m6KDQOlEen5oxaVLd mpn44+vsVlTTH8ejjgOPKl/5T26Yr75D1Q+Pk5P8P+e6MTdUqg4koF2LdV4PYx+3j32Q lvmdVkZra/AZy5zB6Ynbz9xE0KdiJ+yp/dRpPvAGFlnQDPvuVze/cOT0f1MBvOftG1xW MgmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bYTu7nw04Y/eGAN3i9eCzY/Ke38OAVixcrHXlu+jNFw=; b=okV8RNS7O6lggxUJmpXCZTXnaIajIlW/Jc8PzdcFbyEB12clThntukvBMQLGwtQY2K YbewXUUsz9tg+XBckTofE33eyn7o6jkNh0JGcJ7vjPrnIw9TmsXfpR5yxYWZjcK2yNa4 HOfiQ7nT4bhnn3G09jVTdHHyVQapTFe+V3XxxwC1T3d3lfVRwNObcCxIC6izxeqGTYc6 rOwSzfEOGHY6Bv81YHHGLKXzYJAfA4xN6x+q6Awz0XxZiPH79xOW0H33Dv2hVr3rq5/L PaSpDUqELTj26f+tXirbMcLqRmNf2PWqU0HU9n6PeTtjgn4C4vTXWGGv+Ll4TIHA0pNn nR2w== X-Gm-Message-State: AO0yUKVisvED6ZwoXgZc8/bNYz2CIz5MXiNoz8jo4Obqm6PAUXcxN7uc RXNY82XyZc9x4/12PRDCb0jz6Cc3IEY= X-Google-Smtp-Source: AK7set+6hui8S8GzrNNsQxXJRqNjwGdwQ7uNZMDuv5k3QQ7TT5mar5i4p4eV6/CeDDFvXWED43V4tYaI94g= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:1024:b0:8fc:686c:cf87 with SMTP id x4-20020a056902102400b008fc686ccf87mr15ybt.4.1676524683783; Wed, 15 Feb 2023 21:18:03 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:19 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-5-surenb@google.com> Subject: [PATCH v3 04/35] maple_tree: remove extra smp_wmb() from mas_dead_leaves() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Liam Howlett , Suren Baghdasaryan X-Stat-Signature: wzjrrh1jd9mdftadot446xxqa4xdr7j6 X-Rspam-User: X-Rspamd-Queue-Id: 25E058000F X-Rspamd-Server: rspam06 X-HE-Tag: 1676524684-164189 X-HE-Meta: U2FsdGVkX18q+FpQ9lZzIGV9oChserkHkzJ/17MXSg+laHmh1AnYvtNAP8Tu0FtloNGLaxsNDlJUIxCXQ9wWEJEgGOF8zu0K3Bi5fKHTzPdUPOwjhN5SYBRLc83su8FjjIza3gqTghyV0Ra096eIjcr+KuSBblWWXSnjm5dMb0NoOjfJevvWApPIHB114Oq+6ZASUkliOazuUdSDk9YtlNmr1d7A62rbpUL88merQhfQxjesZ9U7tct57etAl6dGo0Oqgac8xfw/MN7LWtxmIVY4gLycyrDUQCjno1BsSLmR7xQ6Xdm1DSIonTUPTjmvpi9Xm/frCx5oe5YPKqxJDox132Xp3kD7pjs9O5diqrbveMP14YoSswnFW64J3dea2k5cekVwer8d7ksrL+PpCOF0KEr3KQmfrQsOufQ0u1gyGMRSDcgqMiiZsSvj6glLhutw5/NWl9+JF+RO3cx2+J8VQzzJSiFvZYEbo9F+wtIjp77t3zBaPM/wJFI8fAD6j7x0pNaH4uWBKYM8Kqq45MgWqin70XswlFDQQ0Hbli0hV2kgPWe+TH2WNlHdcc+rc5sk/O34n++Ta2tw6z6rSIUXp8OxlSMaLAu1wdVz0RmFJg4WMWAyldGP31RM0btU7AM9JdO7HjjwzWp83Zh9JYHnon4FperP3wNm+OEUtiNM5pqfLnrXZZtA1rT8pkLH/B4UtyMnpX5OQEM/sNWIDSHnXliqQALt87TRoNn9WOfqoQNdNStQJMaDtNSg4IKTqe+WALcwflvtGsE6mInsjvMaR6sneD1p63KaWsWH/qizg+oz9iMDiB/YuTwdfn76fIqGk0FzuFWJCT/3d+M3J6R1f9YDFnEDAJHrUM2na8/sV6rWmOHtu+d6nfwGHUjd/Hh6Q8Nizb+gcUcSD0sWfyw98Jggxv4r+7VXC7giz0zfkHGtcdSAORqAgm0bwKasl5UsMB54HkMgvJ1TzJg 4LG4J88v l6d0i1zNL70L6x/u6vT/gbHdOQbEE0OgC8Dyvue8CcCcFt/g61SQ+H0bA5mxEoI36k0qdkp/uSpUjiWSWxac8pXeVP0lduD68lNSuzzEAhfVnk00iue2cXKZ+aLXhkegl0RIP6vSl+0gI6TuH0jMjkhbE++Umk8RVU9lhtsJqIJee9MpOkhPAxG/bpKDMnmPDJX8z/jWcOi63j0E3AJ7Q551oKbfSuQYfm8mD9umTxPT65DrrT5+L5k5uEeukl435jkUjdAwE99fkn6POJa3GPkbQoogv451dX0Lb4hbla6cSJjhHdpytVQi1OlUvtSPM9iYUmSr+yl7zAlOC7kRXJOmYh7e//JcdiR7RoxQqS2enOc3J+9ad6o1EqoYFTh4dQ0/QcZLZ7cwwuenwrHcUgtAhyJxGMTwkIoNbP9bUVoWzNkIXw3Lf6fB0xBU+/5fWLbinn8w+DCdqpnhOVLBcYkIxHwXppbtJGHClhUxhIWzTXg/eHJnLptr2P5HnRAHsUK/Ea26daxkccFmud6o1wAwyPlhfLoCgDPv07rlvhwJAomCQ1HdxHWtXxvSTucZBR7df0QRbjt060wE/DWgp7Hsi5sIpe04fd7Pj+eQYXUdleWE8hvbb6r4rLJxDg6TG2r0C4I95DOCFF5LCu3aIe8A6160GgMyc7kdjrRunhwi2jfQMnyrzTYyEkfKEDoWU0eJZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett The call to mte_set_dead_node() before the smp_wmb() already calls smp_wmb() so this is not needed. This is an optimization for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 44d6ce30b28e..3d5ab02f981a 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5517,7 +5517,6 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, break; mte_set_node_dead(entry); - smp_wmb(); /* Needed for RCU */ node->type = type; rcu_assign_pointer(slots[offset], node); } From patchwork Thu Feb 16 05:17:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142498 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53DADC636D7 for ; Thu, 16 Feb 2023 05:18:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF0556B0080; Thu, 16 Feb 2023 00:18:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DA14F6B0081; Thu, 16 Feb 2023 00:18:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C18EC6B0082; Thu, 16 Feb 2023 00:18:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B1EA86B0080 for ; Thu, 16 Feb 2023 00:18:09 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 81B5AA8E11 for ; Thu, 16 Feb 2023 05:18:09 +0000 (UTC) X-FDA: 80471998698.29.088307F Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf24.hostedemail.com (Postfix) with ESMTP id BBF8A180004 for ; Thu, 16 Feb 2023 05:18:07 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="nzJ7S/CJ"; spf=pass (imf24.hostedemail.com: domain of 3jrztYwYKCOMXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3jrztYwYKCOMXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524687; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rDMlMLTEmJU/Y3crV2atsbCBmpKXyma0yO0OAyc8vDw=; b=CiEE/MTwB64s35oDdAtQgJ/bL9d2woISs4brGfUMUF+jrdR5ssXJtgyjKTEYXPl0rog2PS 8Hvt6aLEpJ3FFRUruwfaNlrV2q5pIzCVq1csr691vkPtEmar3aaaFZqgXiBS49RIyF99px 8mYtfOtos9IuPhdET19QRkpDiWuTpaE= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="nzJ7S/CJ"; spf=pass (imf24.hostedemail.com: domain of 3jrztYwYKCOMXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3jrztYwYKCOMXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524687; a=rsa-sha256; cv=none; b=pg1vW61TB5aUPABssz94EVCBNKXRQwyt8y0JouXu6HdY+5/LtehIENffiKYzBlJAfvFvjn uWoVFK9y49U+SUh/fT83HvRIZYj6wILdeZ7jBj7Z1RiLfhXPNZOoRQcmMsuW/bZAH3qCNV x+9E8kT1Lu/+ZJx04dOPrksXl4uqCQw= Received: by mail-yb1-f202.google.com with SMTP id h14-20020a258a8e000000b00827819f87e5so917209ybl.0 for ; Wed, 15 Feb 2023 21:18:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rDMlMLTEmJU/Y3crV2atsbCBmpKXyma0yO0OAyc8vDw=; b=nzJ7S/CJB/US/rZvNC3fs8J8fXfgfseovx93tCDYzLZ8vkCeR9OXHCG0j7Ki073PZN rg58gpprBwFVBX+ke7DxYuYcgkrSQGYt8YR1LUHO8qrBrJHNZzYIw0fWkCpsF9iepuD5 iMfSL7cYbKTG2WSIKdzRhuGyAIwfPFbrqz4Ug3Wxq2wbPU6P8gVi3bWvRrz9dl4kYYU2 r7gzF5YCRjQDAYOvUYbM1k4rIOQJ+5ffpqKzLjZIEvFTLYyWs4Q62tIpz4Jjx53AOh2l MDbSaQOmeWEl4DEmE5lvlD2J9UkQB4Iedo6XxOd2amw2xyG9CIFDXUp1l5UWlLjXi78n VOeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rDMlMLTEmJU/Y3crV2atsbCBmpKXyma0yO0OAyc8vDw=; b=WdpvKt5Wl9MBWZ0/MDZdutKOIZPFqhnzbZQ4IomlUpyxNpxGG7TsdwAhiybkXIuiMy OhdS8QifwJnX8kbjLHp6M+/gPJm17f4uz5lg57I5RbBUmrkpFPfIRIdLhR9dED3SdOVu pIhyphhj5adgtqzAg6D7sc8a7/3a4azmERCYipzQgPdXVQ9vWTwi4d2cb+mjI5qyroAX CqKkJ3ISRG7gq56bTvtFxmlTadjDryL5NZcV8BZRIbd+zSPuzXbN9pvx400MTwMlY6A9 z00G58wBPU9LWjptY5ZF0VKaPJAQSHM7VQDHkGvdOaM+GGOuyvpBM3Oaa3OuoGdwRLG/ vVkw== X-Gm-Message-State: AO0yUKWrrF8orwjbKZXlvv6vFQdnAXKLRpi2k7OtFFS+vQO00h6qlr0F yBRD3o2h7Ou9uWsnSGghAMVHNEMytVU= X-Google-Smtp-Source: AK7set/lb47TNs4WZdV/lfneGa3vlSnF3eCz7HT/YLrrRFQlj7qxX88gXTKYg9jHepFjlMCGZaF+lFhq4LY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:4309:0:b0:52e:c2c5:b75e with SMTP id q9-20020a814309000000b0052ec2c5b75emr19ywa.2.1676524686391; Wed, 15 Feb 2023 21:18:06 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:20 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-6-surenb@google.com> Subject: [PATCH v3 05/35] maple_tree: Fix write memory barrier of nodes once dead for RCU mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, "Liam R. Howlett" , Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: BBF8A180004 X-Stat-Signature: w3pjs1378jj1sswyqkbaiw9z8zmaz4s6 X-HE-Tag: 1676524687-760182 X-HE-Meta: U2FsdGVkX19F2cpcDBYCfu2jlJ0U0aUrJ/F1llUid9WG+vc69jm/s2xHR2kosijQMNxc5cCy7kr7ffZZsbiygpf4iNN+OFlIaqqj6hxbWPBLXXzvDPf5BOC4IngECgm7sDJ4jk/45IIsEsU9d3ZzfB3y1QE5KSSfBt53FdUXZMqG26EKXpQgLGGO/96l4MeVPO9XJSk0IN6OpddH1D94GMFIsUwsXDUTxn0l7Hl0L8nJvGVhb7JeL9Dc+FvsfXAclHxZ3kX0KLKnrkmYdTeLhaY7+YG4aYxSGInXeyoU0wNhB3VTihJbyllfSN+0Lt42TMQyuiCOdIOv3FnZvCO38QA1qADmX13lYuYvN9Y9pz0rnDVnhhYXSRBz47vWB9QJ/vzxgwCaCboOWT+QYhsNXwetIPFF12Zdg926ZTwFucFRw+Xih/4srzcIITl2I9SZ4mt6LlPkQ4QKj5UnjRhFlexs2ftMmdGaB857dLsTJ6R40fl3nXh73ddQtHixSQ+8JPgeGOtK88hHOHF/xgiuWPoq7zsVFws2DdjleHKhvKhbt2VkFXCaMDpIfInMJKdQfVhgBzVYkOdBXUBuxlH3320ZyIbNOwfIqUF+06ovj78Tpnl7hqIJEnw5ruZ/i6O17T6Iyfdcncavesa2EJwfuM8na4Ar2k7g4o410g5uAGrZ08ooe+aHCTp1pTcOAj+dfDchHKo18y+SvrEXy619/FJlM0C+KcLDh/mnfdMRi8u+Rx0e8EOBQ2p6XTqecx4BWwICFXXVjdpQvcg/IMGfnqdFw5jAE8aHdcMhJEoTj7InmeCdDLKMggTrM7yt5eX0lVxOQkhEjb5/wOCPow99N1cXpNbAZiP+S7Wukni81nBf7hM6sPUfAMnNTjdvyOpWXMXFbRMqc23anfkGOt2ds5nJiuFAGzV4p0/HV23LAD9UgSPU9Rp9uzmop3Vcbzdsg4qAKtVw1JVd77hs3MR lAoSLPiR p76Mz2bob4QC10J2w7C+ZifUYfZ5tS5gM3MyUFP8L1UT8LX5aC4z6tf+CkDsTd2Ca0/B43qR45f3bIiY4FrfOHhqr0ncKfRj+Yeg/1KWeQFUSX+r+KVLL9U1eWxkB5U4smDsdua7wU02J+1rUBW5+I7x+6o+Wq4JUb7kzp6vMEdjqnzdlq3LwyuTeN6Z6lu5QFKRZhO7Xv/YFpMKTA7KY/9raTFSehkVN4KmDrNPZgbb+yoiDCPeIWsmMvWkHp3XtzfdHgAbuWujpcYdgGYLytNVr82CGt4ViyY3HMOIRWiDXtWjk2V6+w7r+uIICX/MSeFM4rwOUVo56zSrozNTdIn8lab2SBIurNHzOg2/32MhRAvc68Q/Wu4OgxIejYmtsy0T51W6Tla52EGkJsm8q/K3FFPT5Dcz8ng3tTsJNSwWJa51PNNMSxkOTE4MwY67eWSt3O9FHjwXIyOjO2eq9UxTQyTnTSmAftDlY+2v1PhFIR/EO2sX6pMIfeyqhTzDnPSRkURNpDpl7UMAO0NB9kiEoSB39EpnJW48kAFXbmmRgatejv8cOblG+ZW0mlRXh0Ry1lhy57D3SPg57cA87m2o4BWQw2OWLKThdzdO/UbEs+vQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" During the development of the maple tree, the strategy of freeing multiple nodes changed and, in the process, the pivots were reused to store pointers to dead nodes. To ensure the readers see accurate pivots, the writers need to mark the nodes as dead and call smp_wmb() to ensure any readers can identify the node as dead before using the pivot values. There were two places where the old method of marking the node as dead without smp_wmb() were being used, which resulted in RCU readers seeing the wrong pivot value before seeing the node was dead. Fix this race condition by using mte_set_node_dead() which has the smp_wmb() call to ensure the race is closed. Add a WARN_ON() to the ma_free_rcu() call to ensure all nodes being freed are marked as dead to ensure there are no other call paths besides the two updated paths. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 7 +++++-- tools/testing/radix-tree/maple.c | 16 ++++++++++++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 3d5ab02f981a..6b6eddadd9d2 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -185,7 +185,7 @@ static void mt_free_rcu(struct rcu_head *head) */ static void ma_free_rcu(struct maple_node *node) { - node->parent = ma_parent_ptr(node); + WARN_ON(node->parent != ma_parent_ptr(node)); call_rcu(&node->rcu, mt_free_rcu); } @@ -1778,8 +1778,10 @@ static inline void mas_replace(struct ma_state *mas, bool advanced) rcu_assign_pointer(slots[offset], mas->node); } - if (!advanced) + if (!advanced) { + mte_set_node_dead(old_enode); mas_free(mas, old_enode); + } } /* @@ -4218,6 +4220,7 @@ static inline bool mas_wr_node_store(struct ma_wr_state *wr_mas) done: mas_leaf_set_meta(mas, newnode, dst_pivots, maple_leaf_64, new_end); if (in_rcu) { + mte_set_node_dead(mas->node); mas->node = mt_mk_node(newnode, wr_mas->type); mas_replace(mas, false); } else { diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c index 958ee9bdb316..4c89ff333f6f 100644 --- a/tools/testing/radix-tree/maple.c +++ b/tools/testing/radix-tree/maple.c @@ -108,6 +108,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mn->slot[1] != NULL); MT_BUG_ON(mt, mas_allocated(&mas) != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas.node = MAS_START; mas_nomem(&mas, GFP_KERNEL); @@ -160,6 +161,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i); MT_BUG_ON(mt, !mn); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -192,6 +194,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != i - 1); MT_BUG_ON(mt, !mn); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -210,6 +213,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != j - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -233,6 +237,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i - j); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1); } @@ -269,6 +274,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -294,6 +300,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas2); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas2) != 0); @@ -334,10 +341,12 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); for (i = 1; i <= MAPLE_ALLOC_SLOTS + 1; i++) { mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -375,6 +384,7 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); @@ -382,10 +392,13 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); } @@ -35369,6 +35382,7 @@ static noinline void check_prealloc(struct maple_tree *mt) MT_BUG_ON(mt, allocated != 1 + height * 3); mn = mas_pop_node(&mas); MT_BUG_ON(mt, mas_allocated(&mas) != allocated - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, GFP_KERNEL) != 0); mas_destroy(&mas); @@ -35386,6 +35400,7 @@ static noinline void check_prealloc(struct maple_tree *mt) mas_destroy(&mas); allocated = mas_allocated(&mas); MT_BUG_ON(mt, allocated != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, GFP_KERNEL) != 0); @@ -35756,6 +35771,7 @@ void farmer_tests(void) tree.ma_root = mt_mk_node(node, maple_leaf_64); mt_dump(&tree); + node->parent = ma_parent_ptr(node); ma_free_rcu(node); /* Check things that will make lockdep angry */ From patchwork Thu Feb 16 05:17:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142499 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3CA1C61DA4 for ; Thu, 16 Feb 2023 05:18:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D48C6B0081; Thu, 16 Feb 2023 00:18:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 75EB66B0082; Thu, 16 Feb 2023 00:18:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64C7D6B0083; Thu, 16 Feb 2023 00:18:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 561F56B0081 for ; Thu, 16 Feb 2023 00:18:12 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3AAC6A0BD0 for ; Thu, 16 Feb 2023 05:18:12 +0000 (UTC) X-FDA: 80471998824.08.F4E0CDE Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf11.hostedemail.com (Postfix) with ESMTP id 73B7E4001A for ; Thu, 16 Feb 2023 05:18:10 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=WMbcZd2H; spf=pass (imf11.hostedemail.com: domain of 3kbztYwYKCOYacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3kbztYwYKCOYacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524690; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/TbDrnT4JTA4cMZmAjPRNE76LJNgcWEHHiVobOaA+Y8=; b=4WfA5llvjBujpe7NDyjF1kVmZnOBwDA/z2WZV1KViV/hW1LslylJQNx4A5iFQ6Y/fRnMDb CZF0/3Ya0S25l2juabcc2w0OLAGhvUA2le6i9aA/Lc+Fwm2KJBAi+BVNL3EFncjJ8afyUQ ODF9SI1EugGqshfk5WsxfXLvueG+ijc= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=WMbcZd2H; spf=pass (imf11.hostedemail.com: domain of 3kbztYwYKCOYacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3kbztYwYKCOYacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524690; a=rsa-sha256; cv=none; b=LXB7xtT2HFb3mdu9PG7Gvak+xkKuKRtFnXooCAgnBQ0hE5h0AYh+SoQNbZBu1p0ExUPM1f 0fKRXZs00d3B4KXEfEe7+vBi506jI68hVBGZeAampqKw83aPXlqKJq1Em5VoMGZiBRLTJg QzCv6238U54RIBfGSTfjD58u1sOub80= Received: by mail-yb1-f202.google.com with SMTP id p83-20020a25d856000000b0095d2ada3d26so831377ybg.5 for ; Wed, 15 Feb 2023 21:18:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/TbDrnT4JTA4cMZmAjPRNE76LJNgcWEHHiVobOaA+Y8=; b=WMbcZd2H6xEYgh8jPBjtr3c52H6Pfawu+33AP54ppxn4975bHziKOFepXARIS5jFg+ e6WwvlJoOPdxReXGpPCKc/NAAxfX4rbkO7DDbRqhMho2D2UtZ3EYOUrPrXvxo+yY0eM4 LMQxZ5khHMqm3h0sfllZRxbm/tuunDq1ItomZ38Vc5vIB77wslGrayg6KY3mbC66vptv Do13gHFViUDuIVWARk2CRzZ0n21J2khqCauKfFko2Yy6sWDAGtNFzpXO3m4fv/AYk9g9 lA127TPhorpqkZg7qKdsS4C88Qpbq0ryHT7Lx24g1uyKmp6VAfvqXa/maK1LoDRn6fAx 1DGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/TbDrnT4JTA4cMZmAjPRNE76LJNgcWEHHiVobOaA+Y8=; b=xrok2gtj/VJpzcEIe5GORJusOay5DMUeq0wMBN9Z8Bnppxs6bKHD1rRIDIe8AwYNSi wyuehuvP17mluZgB2tTXmrj61uV51Uo6Fk4NXw42UzbvS7geqTyMIauDgLP0EhUhbmih tRoipgYw28Y/T+0Z6kGP402TzWbBnqkBm8We9QBZxL4LKImOo0qsbjAgsf4kneiLXjP/ AHgIT8PE65eeDJY+ZfAbr0eUfBq8saKVa4dEtZQSzTHj/K5XEwtn2p7J8Hqtjd5OZ0Uc K+jT/rcA8Vg+HBB3Elbq0egHj7XlaAek7ipq4ExKeIV9ai4Z2fioObI3mbN+dFumSMb+ sVpQ== X-Gm-Message-State: AO0yUKWTyFDwkMLhpVN8CFqNFJp6ZOhVyOIQe/aH5CN0PUXeRJk+u3bW ib1C2W1LjlMdNvTuC7Uj1WSFdeKhXc0= X-Google-Smtp-Source: AK7set9oeDTHuLB1nfzndbZXzwVBlMHEMtdn96b6e8LnbJxiPKVQLZhG0+zpZpEt4JvzRmnu9oC+9ACRvag= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a25:7805:0:b0:8d7:9497:cbb3 with SMTP id t5-20020a257805000000b008d79497cbb3mr12ybc.5.1676524689233; Wed, 15 Feb 2023 21:18:09 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:21 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-7-surenb@google.com> Subject: [PATCH v3 06/35] maple_tree: Add smp_rmb() to dead node detection From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, "Liam R. Howlett" , Suren Baghdasaryan X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 73B7E4001A X-Rspam-User: X-Stat-Signature: p8hegfwycw74cqo3qr47ftj44ose6bs1 X-HE-Tag: 1676524690-10310 X-HE-Meta: U2FsdGVkX1/s2CjB8A9TgibPGqkIMeHdYf/nGVf5Ggm1MdKT0V+vGFWGo1e+abk8ITSPAnJL9nZY3WzlrT9iq7C9IUj+iwqq9/zJKt3J/j28vMvb0VkngCKXyVLUQGOgANXRyRUXlHsS10bFFq/c1UKDBWIHY8MiIgwibAAN2wCR50IhdhDXrxUNo396aRyUvf/DA7uFTqwc5tOfmiM5xxfgoo/Znj6huUkC5lVKN1nirFrbysv8e4pYZrQxywEMni2GUcs17r8A/acsDy29yKYOTYQ9w4e2oVbTr/ghV/H1EF3A1S6e5Hurf7FABeEF7tEt9+V9z9NM0fkaZAuHEnqIQuqdDqpwG4uXCk2Eqmh1pDv3ijIJL4AlO7f589L2WgegmoTz1sUETID+EDlmEDmMbr2ZJbTQ5AoKULCkJZUhvnFo9VYgNOKGy2xrhmXkdckS23MWP0MWgZrkUl4KeODSbLbXVNXPSXUCqiMjKVHJSCJjLufX7rlPL7Razq/vvbk58EkvH4QBFZ0YEugel5VeuIr2W5/Np6qwZDdjgJp8BUwbrPF3MckgmUDs5KgVghb/OC2VfOSYJFifXCjiWqd03VRUtr6Dy9IdB8M3PUOF7tleTHxJxT6qB6XNlD6Vr2HdumH8XsC0SeLPOVXyalewn8f7IYsuRjmV+C/PKkaiOOCrKfm8SJd5b5E8Kcb3T/EXG7hWeHiiUxx89E6Wvj59uW2a6IPSS+GBCcrcFIy36aG53WWgh4wzNg3+H/VA3GDRAlfRPe3zVqd1u3c/TXelUr3ir3zILhXl5fhVG7DEVMLb8D3UPDXBppZWMNWpoVg9/hZO9qT5dKGwDJGtAILYUFzhJFGv+79He55HA1mQ3b0hImG4jpoohZuBPr2gsYiBPCtuJN//A8T18KJd5xSbVSt8v1BzXOJOEU5e4/EMLJKG5R7GXsTYMparepQYzZeiCZJB3T+2jtVYe6A 3AyE2HEr Kk0VVuu/mpS9twQKgVeeBaJZ7mnuB0LtPYPKFRuXBCjzeBow0TTn/gZQoKTAYtil3FiHZHDSmGlU1FV9AgZDmACQF2JJ2ZhP+Sr6GJjSSEtpfL5BMwEJ3kHGpmSuWb5E3HzcVaMGzLUVXr0LuYkZ5uCP67QHmizj83f8PsDlP0W+/8R3CHt/6RPDfLFkPkoxf8OKR/B1iXL4koora1DGU9v/x9EItW0weONYwel74H6VecR1Yf9YURC4aAzwZZDKk6xemDtT85B9fHCv3uMj2RtNkEPQVlSK2cC38Nh4HAU/JWTTL9FL6XC5+QvPAUtmwjDiBDMrblSZwfhGQywZxlUMG5aYj7XXRuPQfJg6h1WEWAaWniYMbG4ZDGqmBqz5AU1ELzB5SWo9eforifIRcUIbd05D0hmdcCDFdPQHJQxf7mF7DWW35ury65/R+AipFFppyRqIWjsYeI+uHVuy5q4K9/i0IthHBlI1Yyp/SEDiiqL26ooLVw/U/Z16pNA4nPBIApSIhJQy8gq0oOKxKV0fhIA9lLYnv7Tst97/Kk0N9aCHvgbU4NJ68+CULgh+oX+aUXhlxrjevNokied/JQmvFt5FTK2i2+9xm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Add an smp_rmb() before reading the parent pointer to ensure that anything read from the node prior to the parent pointer hasn't been reordered ahead of this check. The is necessary for RCU mode. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 6b6eddadd9d2..8ad2d1669fad 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -539,9 +539,11 @@ static inline struct maple_node *mte_parent(const struct maple_enode *enode) */ static inline bool ma_dead_node(const struct maple_node *node) { - struct maple_node *parent = (void *)((unsigned long) - node->parent & ~MAPLE_NODE_MASK); + struct maple_node *parent; + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); + parent = (void *)((unsigned long) node->parent & ~MAPLE_NODE_MASK); return (parent == node); } @@ -556,6 +558,8 @@ static inline bool mte_dead_node(const struct maple_enode *enode) struct maple_node *parent, *node; node = mte_to_node(enode); + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); parent = mte_parent(enode); return (parent == node); } From patchwork Thu Feb 16 05:17:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142500 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B639C64ED8 for ; Thu, 16 Feb 2023 05:18:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8B116B0083; Thu, 16 Feb 2023 00:18:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E3BDC6B0085; Thu, 16 Feb 2023 00:18:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C66DD6B0087; Thu, 16 Feb 2023 00:18:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B16E56B0083 for ; Thu, 16 Feb 2023 00:18:14 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8A9F2A1132 for ; Thu, 16 Feb 2023 05:18:14 +0000 (UTC) X-FDA: 80471998908.23.0978BFC Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf06.hostedemail.com (Postfix) with ESMTP id BB170180004 for ; Thu, 16 Feb 2023 05:18:12 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZK5oAM6A; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3k7ztYwYKCOgcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3k7ztYwYKCOgcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524692; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hX1DUaFSHqHCHFWD5Crp9NnpH/+9VHFxYkECoHgya1E=; b=K2wpCC/ZukdEcCPe0UVwkewohW4XH3XdxmqbYX9pUfHpGDPRYYVU4v1lmCm3XNxBJArwnu FROTuyc6F7CHENqksZY43JNZTqhnGrkXjH1nMmawPYuOm0kjjMiZ81zzYsac/KhZ4IAYOv qr1Y0dIrQJ+KCzgypWIp25U/BcAcIW4= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZK5oAM6A; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3k7ztYwYKCOgcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3k7ztYwYKCOgcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524692; a=rsa-sha256; cv=none; b=fdqKNLtVXVLBWqx2U+gtfbtnRd7PmcMTVTUZiYZcdJh6XX+ovb8deyDgzFFIJvNoiDZVfR YOkzUTVQGXrKo7E/XULxxjP7WKAkz/x40weXBTVOg47uouiqFNIuyiOXlxoaj0GU6KfT/7 laCUSXZJRcflheqOi6KPh1RjeuBnl9Q= Received: by mail-yb1-f201.google.com with SMTP id k9-20020a25bec9000000b00944353b6a81so862503ybm.7 for ; Wed, 15 Feb 2023 21:18:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hX1DUaFSHqHCHFWD5Crp9NnpH/+9VHFxYkECoHgya1E=; b=ZK5oAM6AcKVtxFxzHW9PhD/PdA0+Kv3bSAspA28+ZPD1m3cnM9poVYSoqhonqkIeJU SshDVskllMZgIeSGhnCidLA6ddGXSMxz+S/xoEp08YDrkRg+7RXAXCoS6u4tI7rKulCi ggnzbiGToN5cME4XPhcHuI14WBO6+4EQsud/8F6AxE1deunG+n/R+uM3wq+OYW0Yw/xw 3Mt1GtYUQdIHi9t1kC0vmn2kgnR9+BG6rYf5dtABCjAt9PnVyIFEokbi9zalyZENG7ye R6/8svYYK3xPGp282avPu/RfQWQWw9vT/0uQU8vbTuKv0KxRDKx0B9gWO6ClqS6abquo +1jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hX1DUaFSHqHCHFWD5Crp9NnpH/+9VHFxYkECoHgya1E=; b=C+qhIAWmQFk/IsYTV+Dmf8VQTV7fxxjdOoehpkUUeOX6lMoSNrRxu0lbGm7M7pTBT/ PSjvAdG2JKBXhgalDH4wfeODwuEGhWhAbyzRYnu5GSVxrKkofUO7cKF7hrwc/V5efFJF c0PN/3Am4bWOaShCsZLQqyazrMIa7mN6WaIZeoModDFUUjsXUmrbq1HqZb37AG2lqorU AS/496gzGCAuKlAiQ8EIcnYk71EpZOH6OPxjeZGchHv3usS4COc+RSaexjdP5hSnI/+i yOi2lo8xGGA4ti23mMr+u7zgvVBoDKA/+0iNLz0Cda+QE2scz7icwkuBXj5NFPHEiuFC Z9bg== X-Gm-Message-State: AO0yUKWyUW89dYPVtVl9Har6jJa+dyFP76fM7eSiLvbahG9zu1wi9jO7 5nn/ab/QQXEP3AyKsKI5VKme4g9AQ5w= X-Google-Smtp-Source: AK7set+BUdfWyfJ4CpYtDrI1pKL/2vKMY8TRPzm0GzgZQonEWMOroSKkOakotgIDyZRMjNiR+HE0zfFqp3Q= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:4ea:b0:926:d02a:c18a with SMTP id w10-20020a05690204ea00b00926d02ac18amr43355ybs.233.1676524691938; Wed, 15 Feb 2023 21:18:11 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:22 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-8-surenb@google.com> Subject: [PATCH v3 07/35] maple_tree: Add RCU lock checking to rcu callback functions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, "Liam R. Howlett" , Suren Baghdasaryan X-Rspamd-Queue-Id: BB170180004 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: pzwxwzquz6x3wgazwiz8eqem1hoa17gs X-HE-Tag: 1676524692-346959 X-HE-Meta: U2FsdGVkX1+wxdCFxw0yGKJIwiosyp6pTGd/GPBWeNsS5OI8Ivr4l87fgONhync49wpNVjp41y9+tRz336DMPNqMs52lS3MAfBKgBoO+dvt8a8hb/PcfthNWHzeo/n0gqiBAPO2EH4beL06JqncTCgq9Yk/3XpdROvOkamhGr321vSesYsaDAzQR3OGJ8SFVcB+ilNojLvrYFODeGgnl/KirUUn1TXk0VKdIO5xcqKME+E9CBaEjgcuUhlVIQiqgkG/U8YVlsNjRIrLCDaWFaPt/OV4tJzWMsSUXvQDZTdV148RKs286YeiC8qWKJZryzm2o0w+zmrkTzvY1GtbOWDS00g7SfchXc/raZSPRaKEevTliVjj4cx7+3b6EZtVJg/haqMWcETqLy5SeqWGC5tpbEZKp8XzNczZJj5XVzM4/Uu1Nb0TijTCEP/GFCCLyKltErfCQWh9L3Pgg3e9nmetO5KMToYLDpHu8u/X8H44Y6E8JvqpFwHWHvc0o8aIR8LFqs4ttKaAPGtfHA7znKj7TiaXmMHgNV4/FiOaACJFHOXIRz6PbAz9veZAwSkcZX5JXn1wTIHnCOKsAPkf8i1N4fL14LyW4eM4va6OlAVs4HdH/ChMOJ4m7O0bMnKfnqNY/IlS29GiOJxe+sPFCuBNzmqA2FLGtA/odvFmdTGI+xl5FZLuws4JBre3ui2j7+xNYiOjFeKijSTioqHOGbHBktYx5Vr8suHq7IzzSqqOE7H6tu4MpGyuOXiP8AYpeY1T/L9RrIoTLFPiKV7xpbth9MijtUi/ZEfuBNWZAE5vFF8r59LmStLNHq3W1UbjKXnMzQ8XGBfKQBdq8J2nuyUUU9z7EHePqXDiq/K5trh4whtFRIezqcdMDw2NwT3h2+mV6R0h0kDBEiek31mraVvfWU9vUJxf/9ZPZQqxoo8k83FnuNl/6TKq/3tMgozZXWH3bjtK0/5fgU+Q5Gka 73yeCfQn mg9HJSQRjdQRdePHNymJKyCIkdnAlPADqZZy8QUpoagLSBS+tYm6o9br3G4WNlNOt21Oj6gP9rI2w2WUlo8j+ATdvWIk/pEyEZzE1CnR4YCcAigofgytvPPLiRgkg+XuyTFXpVZOL1eR3DDhWMO1DieDrgIw5kyJ2WRywUa11vY4emlwucUoBIzdC8nxzdkHw7tOLNOpGJbPbo/Emnk7ERW1YW2x5sObxZyhdSTRhMcGVWqrOtk4HK9RWrYN7HiUSWF9JZARdYWceJ2dMVRLj27w7KRZ9zx+M5Zzb9UEBiwuK+tAMTuBh5c/Xugep0sRhRevTXJmeiOZR3KvgYou/z3igLjhn/SXTdehP0apY8Jgdwd1eGc43YVH4EuVLOY7pN+EJiqqKFMMZ4XLcyh0La35LnGrkX5/l7d1KgwbrhYQagAoawTndAPB1hID7XaSkonktd3TSRzcFP9+nYImaMky3TzmUvoaWg1OmYpJ2TyapVTrKPZKIndtm2DwMMDBNGLnirPNYUh31SYbWOcXe7FK1DqGaYPuGLanI/r+o/eHbzfJQjSQLCYtJ0LVA2eB1Xsl78O9yxbHpO1G4O2UB/TzcLaQgzH/WlMbbFq2EAGeonsE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Dereferencing RCU objects within the RCU callback without the RCU check has caused lockdep to complain. Fix the RCU dereferencing by using the RCU callback lock to ensure the operation is safe. Also stop creating a new lock to use for dereferencing during destruction of the tree or subtree. Instead, pass through a pointer to the tree that has the lock that is held for RCU dereferencing checking. It also does not make sense to use the maple state in the freeing scenario as the tree walk is a special case where the tree no longer has the normal encodings and parent pointers. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Reported-by: Suren Baghdasaryan Signed-off-by: Liam R. Howlett --- lib/maple_tree.c | 188 ++++++++++++++++++++++++----------------------- 1 file changed, 96 insertions(+), 92 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 8ad2d1669fad..2be86368237d 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -824,6 +824,11 @@ static inline void *mt_slot(const struct maple_tree *mt, return rcu_dereference_check(slots[offset], mt_locked(mt)); } +static inline void *mt_slot_locked(struct maple_tree *mt, void __rcu **slots, + unsigned char offset) +{ + return rcu_dereference_protected(slots[offset], mt_locked(mt)); +} /* * mas_slot_locked() - Get the slot value when holding the maple tree lock. * @mas: The maple state @@ -835,7 +840,7 @@ static inline void *mt_slot(const struct maple_tree *mt, static inline void *mas_slot_locked(struct ma_state *mas, void __rcu **slots, unsigned char offset) { - return rcu_dereference_protected(slots[offset], mt_locked(mas->tree)); + return mt_slot_locked(mas->tree, slots, offset); } /* @@ -907,34 +912,35 @@ static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt, } /* - * mas_clear_meta() - clear the metadata information of a node, if it exists - * @mas: The maple state + * mt_clear_meta() - clear the metadata information of a node, if it exists + * @mt: The maple tree * @mn: The maple node - * @mt: The maple node type + * @type: The maple node type * @offset: The offset of the highest sub-gap in this node. * @end: The end of the data in this node. */ -static inline void mas_clear_meta(struct ma_state *mas, struct maple_node *mn, - enum maple_type mt) +static inline void mt_clear_meta(struct maple_tree *mt, struct maple_node *mn, + enum maple_type type) { struct maple_metadata *meta; unsigned long *pivots; void __rcu **slots; void *next; - switch (mt) { + switch (type) { case maple_range_64: pivots = mn->mr64.pivot; if (unlikely(pivots[MAPLE_RANGE64_SLOTS - 2])) { slots = mn->mr64.slot; - next = mas_slot_locked(mas, slots, - MAPLE_RANGE64_SLOTS - 1); - if (unlikely((mte_to_node(next) && mte_node_type(next)))) - return; /* The last slot is a node, no metadata */ + next = mt_slot_locked(mt, slots, + MAPLE_RANGE64_SLOTS - 1); + if (unlikely((mte_to_node(next) && + mte_node_type(next)))) + return; /* no metadata, could be node */ } fallthrough; case maple_arange_64: - meta = ma_meta(mn, mt); + meta = ma_meta(mn, type); break; default: return; @@ -5497,7 +5503,7 @@ static inline int mas_rev_alloc(struct ma_state *mas, unsigned long min, } /* - * mas_dead_leaves() - Mark all leaves of a node as dead. + * mte_dead_leaves() - Mark all leaves of a node as dead. * @mas: The maple state * @slots: Pointer to the slot array * @type: The maple node type @@ -5507,16 +5513,16 @@ static inline int mas_rev_alloc(struct ma_state *mas, unsigned long min, * Return: The number of leaves marked as dead. */ static inline -unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, - enum maple_type mt) +unsigned char mte_dead_leaves(struct maple_enode *enode, struct maple_tree *mt, + void __rcu **slots) { struct maple_node *node; enum maple_type type; void *entry; int offset; - for (offset = 0; offset < mt_slots[mt]; offset++) { - entry = mas_slot_locked(mas, slots, offset); + for (offset = 0; offset < mt_slot_count(enode); offset++) { + entry = mt_slot(mt, slots, offset); type = mte_node_type(entry); node = mte_to_node(entry); /* Use both node and type to catch LE & BE metadata */ @@ -5531,162 +5537,160 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, return offset; } -static void __rcu **mas_dead_walk(struct ma_state *mas, unsigned char offset) +/** + * mte_dead_walk() - Walk down a dead tree to just before the leaves + * @enode: The maple encoded node + * @offset: The starting offset + * + * Note: This can only be used from the RCU callback context. + */ +static void __rcu **mte_dead_walk(struct maple_enode **enode, unsigned char offset) { - struct maple_node *next; + struct maple_node *node, *next; void __rcu **slots = NULL; - next = mas_mn(mas); + next = mte_to_node(*enode); do { - mas->node = mt_mk_node(next, next->type); - slots = ma_slots(next, next->type); - next = mas_slot_locked(mas, slots, offset); + *enode = ma_enode_ptr(next); + node = mte_to_node(*enode); + slots = ma_slots(node, node->type); + next = rcu_dereference_protected(slots[offset], + lock_is_held(&rcu_callback_map)); offset = 0; } while (!ma_is_leaf(next->type)); return slots; } +/** + * mt_free_walk() - Walk & free a tree in the RCU callback context + * @head: The RCU head that's within the node. + * + * Note: This can only be used from the RCU callback context. + */ static void mt_free_walk(struct rcu_head *head) { void __rcu **slots; struct maple_node *node, *start; - struct maple_tree mt; + struct maple_enode *enode; unsigned char offset; enum maple_type type; - MA_STATE(mas, &mt, 0, 0); node = container_of(head, struct maple_node, rcu); if (ma_is_leaf(node->type)) goto free_leaf; - mt_init_flags(&mt, node->ma_flags); - mas_lock(&mas); start = node; - mas.node = mt_mk_node(node, node->type); - slots = mas_dead_walk(&mas, 0); - node = mas_mn(&mas); + enode = mt_mk_node(node, node->type); + slots = mte_dead_walk(&enode, 0); + node = mte_to_node(enode); do { mt_free_bulk(node->slot_len, slots); offset = node->parent_slot + 1; - mas.node = node->piv_parent; - if (mas_mn(&mas) == node) - goto start_slots_free; - - type = mte_node_type(mas.node); - slots = ma_slots(mte_to_node(mas.node), type); - if ((offset < mt_slots[type]) && (slots[offset])) - slots = mas_dead_walk(&mas, offset); - - node = mas_mn(&mas); + enode = node->piv_parent; + if (mte_to_node(enode) == node) + goto free_leaf; + + type = mte_node_type(enode); + slots = ma_slots(mte_to_node(enode), type); + if ((offset < mt_slots[type]) && + rcu_dereference_protected(slots[offset], + lock_is_held(&rcu_callback_map))) + slots = mte_dead_walk(&enode, offset); + node = mte_to_node(enode); } while ((node != start) || (node->slot_len < offset)); slots = ma_slots(node, node->type); mt_free_bulk(node->slot_len, slots); -start_slots_free: - mas_unlock(&mas); free_leaf: mt_free_rcu(&node->rcu); } -static inline void __rcu **mas_destroy_descend(struct ma_state *mas, - struct maple_enode *prev, unsigned char offset) +static inline void __rcu **mte_destroy_descend(struct maple_enode **enode, + struct maple_tree *mt, struct maple_enode *prev, unsigned char offset) { struct maple_node *node; - struct maple_enode *next = mas->node; + struct maple_enode *next = *enode; void __rcu **slots = NULL; + enum maple_type type; + unsigned char next_offset = 0; do { - mas->node = next; - node = mas_mn(mas); - slots = ma_slots(node, mte_node_type(mas->node)); - next = mas_slot_locked(mas, slots, 0); - if ((mte_dead_node(next))) { - mte_to_node(next)->type = mte_node_type(next); - next = mas_slot_locked(mas, slots, 1); - } + *enode = next; + node = mte_to_node(*enode); + type = mte_node_type(*enode); + slots = ma_slots(node, type); + next = mt_slot_locked(mt, slots, next_offset); + if ((mte_dead_node(next))) + next = mt_slot_locked(mt, slots, ++next_offset); - mte_set_node_dead(mas->node); - node->type = mte_node_type(mas->node); - mas_clear_meta(mas, node, node->type); + mte_set_node_dead(*enode); + node->type = type; node->piv_parent = prev; node->parent_slot = offset; - offset = 0; - prev = mas->node; + offset = next_offset; + next_offset = 0; + prev = *enode; } while (!mte_is_leaf(next)); return slots; } -static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, +static void mt_destroy_walk(struct maple_enode *enode, struct maple_tree *mt, bool free) { void __rcu **slots; struct maple_node *node = mte_to_node(enode); struct maple_enode *start; - struct maple_tree mt; - - MA_STATE(mas, &mt, 0, 0); - mas.node = enode; if (mte_is_leaf(enode)) { node->type = mte_node_type(enode); goto free_leaf; } - ma_flags &= ~MT_FLAGS_LOCK_MASK; - mt_init_flags(&mt, ma_flags); - mas_lock(&mas); - - mte_to_node(enode)->ma_flags = ma_flags; start = enode; - slots = mas_destroy_descend(&mas, start, 0); - node = mas_mn(&mas); + slots = mte_destroy_descend(&enode, mt, start, 0); + node = mte_to_node(enode); // Updated in the above call. do { enum maple_type type; unsigned char offset; struct maple_enode *parent, *tmp; - node->type = mte_node_type(mas.node); - node->slot_len = mas_dead_leaves(&mas, slots, node->type); + node->slot_len = mte_dead_leaves(enode, mt, slots); if (free) mt_free_bulk(node->slot_len, slots); offset = node->parent_slot + 1; - mas.node = node->piv_parent; - if (mas_mn(&mas) == node) - goto start_slots_free; + enode = node->piv_parent; + if (mte_to_node(enode) == node) + goto free_leaf; - type = mte_node_type(mas.node); - slots = ma_slots(mte_to_node(mas.node), type); + type = mte_node_type(enode); + slots = ma_slots(mte_to_node(enode), type); if (offset >= mt_slots[type]) goto next; - tmp = mas_slot_locked(&mas, slots, offset); + tmp = mt_slot_locked(mt, slots, offset); if (mte_node_type(tmp) && mte_to_node(tmp)) { - parent = mas.node; - mas.node = tmp; - slots = mas_destroy_descend(&mas, parent, offset); + parent = enode; + enode = tmp; + slots = mte_destroy_descend(&enode, mt, parent, offset); } next: - node = mas_mn(&mas); - } while (start != mas.node); + node = mte_to_node(enode); + } while (start != enode); - node = mas_mn(&mas); - node->type = mte_node_type(mas.node); - node->slot_len = mas_dead_leaves(&mas, slots, node->type); + node = mte_to_node(enode); + node->slot_len = mte_dead_leaves(enode, mt, slots); if (free) mt_free_bulk(node->slot_len, slots); -start_slots_free: - mas_unlock(&mas); - free_leaf: if (free) mt_free_rcu(&node->rcu); else - mas_clear_meta(&mas, node, node->type); + mt_clear_meta(mt, node, node->type); } /* @@ -5702,10 +5706,10 @@ static inline void mte_destroy_walk(struct maple_enode *enode, struct maple_node *node = mte_to_node(enode); if (mt_in_rcu(mt)) { - mt_destroy_walk(enode, mt->ma_flags, false); + mt_destroy_walk(enode, mt, false); call_rcu(&node->rcu, mt_free_walk); } else { - mt_destroy_walk(enode, mt->ma_flags, true); + mt_destroy_walk(enode, mt, true); } } From patchwork Thu Feb 16 05:17:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142501 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76388C61DA4 for ; Thu, 16 Feb 2023 05:18:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F8EE6B0085; Thu, 16 Feb 2023 00:18:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F264E6B0087; Thu, 16 Feb 2023 00:18:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D78A66B0088; Thu, 16 Feb 2023 00:18:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BBC2A6B0085 for ; Thu, 16 Feb 2023 00:18:17 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8B2751A09C7 for ; Thu, 16 Feb 2023 05:18:17 +0000 (UTC) X-FDA: 80471999034.07.4293B58 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf24.hostedemail.com (Postfix) with ESMTP id CBFD1180004 for ; Thu, 16 Feb 2023 05:18:15 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Xo1jDDVi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3lrztYwYKCOsfheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3lrztYwYKCOsfheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524695; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4opi4a+4wBSqIsfjyFGdOCf+RfFv13YZtxtXh5ocOUM=; b=tv0glaSS00KMP3R1lNggFDsXfmUM0ZOuCx8MnHFXFylI9Fay1qtEJhyn0Qw2PQf+byU14L Klh4fTmwRHq8GjWF0UiF2pliPv+JuIBdtdHCAlp1oBfW5SNkLd/eca4gP3//WJztin596b eIOBffTvCz0VVPEld73Zuf1M7G/gfCA= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Xo1jDDVi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3lrztYwYKCOsfheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3lrztYwYKCOsfheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524695; a=rsa-sha256; cv=none; b=k4keLIKc9RydGpz1Hl6UI9IZTn9wJdUWd1Gb1go9GfX7KpXYPqI28hZgE9358SWSVmWCFW YGBO90IAbEOAiOau4Hx/p/+EQBaBu7pKC/b92M6GdgiQM1I0y3JZro07STd6hKLNwExk2G fVx83HdStUjBClYmcl/UCt8aAUbziow= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-52f2ab6a942so8466397b3.14 for ; Wed, 15 Feb 2023 21:18:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4opi4a+4wBSqIsfjyFGdOCf+RfFv13YZtxtXh5ocOUM=; b=Xo1jDDViHSHN/alioaTDnw+zuoNPKrF9ZLXUf3ea0NE3bXPLl0z7WkGrQUW7AOG0Nr J8SsODBneQSeaMKO7nEqhtXELAqyVVzyHag3dRajS0MNlWpB1lyOtsy0oB8MM0CrQEnP Moe4iuqLi8ffYInMsJBu3lDYUcEPw1W16t68/pqdVg/PM7yHOfl4JRIe/AAV288dZ063 0OmpnGUH5/0w4iEIelRXBTS4gWduVmNFBkN/5ItORK5tiSjU8ab3rw3adCz9oFddo8Er mGSeo2MZqEM7cxpZTU0mXlOgRPyGjhgPgDiiUmvS//v9NEkdkZgUTyYfpCAeqd0avQoQ 0u1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4opi4a+4wBSqIsfjyFGdOCf+RfFv13YZtxtXh5ocOUM=; b=DctBjrBANyTnPTsyG1s7iaSmXfySh7mWxcYWFiMDUB7OVJBqceukHHmH2aAPNV64EH nm7nf/gvfraOdrNhzybdhZWWQYJYxaAwBr0qlsdbHlxinKEJHDUYNplvRhpmNv4P9GGQ PMWQ5/8wEsaqRpEHta5iHgubPKFHRvO1hQR19Wj15nw1rxGxBdr9O60y4HmfgUgiGCAx 1EqNp5pDqoztGDC7Z/TSaaZ++EoS6SUmOySPiv9baG1nWTzJj0ibcRgZh+ZDfKEKEpca UHn42V+nBw+RZbcg6w5akCrnDHouXAFitIIiHWO7NxPbhuIEexAPYTTog9Uz038ipSaz 5nHg== X-Gm-Message-State: AO0yUKXueiNgOxeU6lc2DYInySpU2L+++MHNjzmeJbVOjTPtj21sek5M XvZL0/kv/6ItwGzo3RgnVYsoqq9jz8E= X-Google-Smtp-Source: AK7set+/DgIBDY3hKjRHmWHVe7yByHJ3rZUqgnqz5mohiI9GciRP+9xmqTdnoEyOpn2/QPX/HT0hsnAfle8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:ca:b0:8f2:9e6:47a4 with SMTP id i10-20020a05690200ca00b008f209e647a4mr3ybs.7.1676524694561; Wed, 15 Feb 2023 21:18:14 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:23 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-9-surenb@google.com> Subject: [PATCH v3 08/35] mm: Enable maple tree RCU mode by default. From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, "Liam R. Howlett" , Suren Baghdasaryan X-Rspamd-Queue-Id: CBFD1180004 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: hit8smzbceajhhdhq1caoyx3fwb8hj8m X-HE-Tag: 1676524695-614475 X-HE-Meta: U2FsdGVkX19c8Y8m3GF+omTv+BqEHuLn1aEnhOyb+Mhz4+i3P8epm5xV38AQtqMvXYBhDVl53KuW2REUQXY1EqxfWmp4b5cBBQTqNckL0+HAdeP4WbdafJDlMk40FBLRxwwvttDjPtGYCFxDQnMHy4M1GTPgq98InVynTc5rI+g64MELCKUOHoFJNdwEnLLVC63L1Lunlqb/vnnvykk1d2iHJoefvLwLN2+UceOGojhnhL5SGqcC3tEE7OxNAHB5bfT7yQC4VajCeHEwPoAZJJc73dW/nWyu1wqAm1SPry/5jkXLgf7Djq6gQfKXq+ss5iEiVAllNg5ZRhoDoVAAtK68Oj3mL6PTwIEwOriebhrgvC7NSUY8MiwlTVnso4LcxQ3CCEBqdQELANnD2qkZ34AyviyE3HSLEZcxX0pukpb5za8cDrrMaJ0Bm4Y8oM0wgmbx3SNsQfXEcyOxUSDLHYm9g4oIf9Ze8lVXwoScdAdfXMf/2QjLdM0YdSVTmiq6HWhB16dLCqg39gbe9sm67v4QdGWj3fdGR1OgfxlnXcQk0q2NH9edCuaSDKN/FTLuW5plkNkXnjGpD4L/VC+Uz/GSKb1u+Lf3iSrCw7Ja+0rxTMR71UMyEvEEHmiLUnU/mm7tNc3LMZ2nDJiIHLlPuWSsWJhFhY+6PK+NtcScZp5cYlxGTZOLCZZ2jA4Te8qWdHy2HLTulETIiYZTlfl/mBu8MbuHPjkx44OR1Cdh4PTWmKPai6Yhgqf9j1V7ezXkzhg7+cDE32xNNtyEuWluoaO86s0K6veIkNl70tS75BLea4nOXb4KGRZI0K4Ad3sGrfgARQ0Bjjzctq8pxYh0ZM1NQaBcQvmqcyvvEodstuwnVy23z8cvEIxqGkliplrdFVi8qjvp6afkeGwkCWvmecTrzHV2Cwt/w13Z7hT0lHSYdKb2P+1QYn1Iz/C4kBZuTPKwDfQo/O018RoxWIm uSwNvoM4 dlAzqz1wZoqW3JME5eFT+nmC0fJ/O8hSwfYbcyrRH8aQyZlZImJuVVIG0o+18yQMJW40UQjr6RdvZpK+hHKluuuAGqnD6Mvf7ZCcPyzybG5R+CdxN2DHiwbHLUSWZ6+uebovZTbSMjWN7hR9e2ARKIPWzh0rUTOVe3cZVFMrir714hSPbWZmOoc60grv9wSllanAuCYMRTKfUo1RiVwXSMbcFHaVB4CwvTd7+oHzeap9PXHiKjF45WPnj611lWofxLRho7CaZfIMEP9oQdGimDhiqArAOJUCHaUm9MHP0q+hYpBhkokHkqYKZIuycRk/McYoOEQQkiK08oytOyFZpbXoy2Wr5M52DPUZ2dP3oeZW8vQam/TbCUZdCZbrBxKZPvKoVcX8knXYS/zVky3jZeTr/Xgciv/fDokyXHjxRxfnONjgkEH2eu9L5ioSdh5eBZCvq4nyh7nbuzsxiuiWRvsZPR98mKoclogJHd8xPqACt1hkBTFjZmIV/Egk0AR5IHrEEOw2q/XZ2kdoK0IskImy3sQIgCa6yF6tz6aQG7RXWSL3jg1JuMInzKWJa+hrEAx4OMQMQyHZCV4RCJPwUzjsmXzVIgmWMZ0CylR24orX07DtkJnfdI3efUCG8vWJe08j9gXrO1/Fwadv+YBCSuK8W9g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Use the maple tree in RCU mode for VMA tracking. This is necessary for the use of per-VMA locking. RCU mode is enabled by default but disabled when exiting an mm and for the new tree during a fork. Also enable RCU for the tree used in munmap operations to ensure the nodes remain valid for readers. Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 3 ++- kernel/fork.c | 3 +++ mm/mmap.c | 4 +++- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 3cd60243c625..3fd5305dbbf9 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -768,7 +768,8 @@ struct mm_struct { unsigned long cpu_bitmap[]; }; -#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN) +#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN | \ + MT_FLAGS_USE_RCU) extern struct mm_struct init_mm; /* Pointer magic because the dynamic array size confuses some compilers. */ diff --git a/kernel/fork.c b/kernel/fork.c index 5e3029ea8e1e..5f23d5e03362 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -617,6 +617,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, if (retval) goto out; + mt_clear_in_rcu(vmi.mas.tree); for_each_vma(old_vmi, mpnt) { struct file *file; @@ -700,6 +701,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, retval = arch_dup_mmap(oldmm, mm); loop_out: vma_iter_free(&vmi); + if (!retval) + mt_set_in_rcu(vmi.mas.tree); out: mmap_write_unlock(mm); flush_tlb_mm(oldmm); diff --git a/mm/mmap.c b/mm/mmap.c index 20f21f0949dd..06c0f6686de8 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2277,7 +2277,8 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, int count = 0; int error = -ENOMEM; MA_STATE(mas_detach, &mt_detach, 0, 0); - mt_init_flags(&mt_detach, MT_FLAGS_LOCK_EXTERN); + mt_init_flags(&mt_detach, vmi->mas.tree->ma_flags & + (MT_FLAGS_LOCK_MASK | MT_FLAGS_USE_RCU)); mt_set_external_lock(&mt_detach, &mm->mmap_lock); /* @@ -3042,6 +3043,7 @@ void exit_mmap(struct mm_struct *mm) */ set_bit(MMF_OOM_SKIP, &mm->flags); mmap_write_lock(mm); + mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); tlb_finish_mmu(&tlb); From patchwork Thu Feb 16 05:17:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142502 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAC2BC6379F for ; Thu, 16 Feb 2023 05:18:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 629076B0087; Thu, 16 Feb 2023 00:18:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D7F56B0088; Thu, 16 Feb 2023 00:18:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 403C26B0089; Thu, 16 Feb 2023 00:18:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 265EA6B0087 for ; Thu, 16 Feb 2023 00:18:20 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id EA047141090 for ; Thu, 16 Feb 2023 05:18:19 +0000 (UTC) X-FDA: 80471999118.22.2AD0760 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf28.hostedemail.com (Postfix) with ESMTP id 3B010C0010 for ; Thu, 16 Feb 2023 05:18:18 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=hDLR1pvR; spf=pass (imf28.hostedemail.com: domain of 3mbztYwYKCO4ikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3mbztYwYKCO4ikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524698; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=LvaVp1kRad5sSGE2tH9WqZF058TAx+MGjqAfO7BzRKanDi9XCbPZqlSqOryONvLB7LSHIR 7gHfcfQpF773njoIOVOyWwOtKRsvj75ETOxu8bLD1lbmCZL6kSRqCp0kZ5gn3B72+8Bpqq geMEu1UoIcmhHFuuJv9z4StYUzor5d8= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=hDLR1pvR; spf=pass (imf28.hostedemail.com: domain of 3mbztYwYKCO4ikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3mbztYwYKCO4ikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524698; a=rsa-sha256; cv=none; b=W++Mnk3hQRCh+yqGUrE1oJE7tL23vR89XGhNcF+PhaOiESshY08AW5OkAgDRBtPpq1Ty02 5TMyDKri3q5EBuABWDpnHkBR1GT1tow4bG9ThWjCuuJ2eOyRMjuBF1tdDf80nkU6MCB3DD vL+zace32OSoW6qP+J/9VGatNFO1FkQ= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-52ec2c6b694so8315877b3.19 for ; Wed, 15 Feb 2023 21:18:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=hDLR1pvRXBoNPEsd5rWO1uuL2xRvkViIuLxNT0AXT2es+wKEpveR0MayIpEY9kMwnX yKqoPf3/kr2W8U8aZxHHsI3MeVglv4Uuopqy/RZQvDUL+Iw/1b3yNPaW9UiWpFiktEjT IG17SwGNRtI3Tudjcg57P2Pp6pg3eItytQO0663ElsQFpDjPji34Qnzv66OgBFV/+yja uvKllpdQQMscfNlzsu7K4yE4HNqAZ53EkNxV25Op17ac2k06MkhohfLIKcbxFpec+BdV 695/dYeBJhC1TaWGjUDZda/yGXvfwWspsF3C1nfSkAJuqCo5le3F23yvLzWMlOVw2Nnw Ls0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=IiJgZ2q9QfdpYkevciC5jw8DO8AKQafnrGLXSMFYdaloOVw+5QuwRtut8v4xX0j+D6 t71U9ctd8/S402btQ4BpbFcEmCy7E+IEjjU7ZK0JVZLxGVaPTzNb70qKEdC1DcLMI4Vb ZlvDIBZQo0+cHv1ldeOXzid3QkDcKvh/MDoHfhaxpzaM51NtqOr0zo9dAr3126FABRfa a3oxtT7yIm+JrqoADww8Fvivlpf9HlktQmN6x3wIx/oREphCBj9WHgM+3TRxqvmv41BQ HafEk3ksBsAYWAQ1jRgwSWIJbR+qLnaCPtRWdxpZkG8rTP4Lz2vcGdRTnW0a08Pm8jbV PGHQ== X-Gm-Message-State: AO0yUKVhshljPDR9mimH3pT/lJ3fKoyHYrmDRL0dGLu7WImGd6E1MFBQ FJqnTsYvddci/puHnDokI/f+iIXqkSw= X-Google-Smtp-Source: AK7set/vOsUP0PVWaWbTlqFxALvIpJBlIbMQnTBHBu2Oot2RdkwCU0iTN4oS1VPXE/uz69lz5N2GmOvT6GU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:9e45:0:b0:50f:f163:7072 with SMTP id n5-20020a819e45000000b0050ff1637072mr442506ywj.285.1676524697322; Wed, 15 Feb 2023 21:18:17 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:24 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-10-surenb@google.com> Subject: [PATCH v3 09/35] mm: introduce CONFIG_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 3B010C0010 X-Rspam-User: X-Stat-Signature: zrqt6hf1rpehhmuwrg8hy93g3trjnw1k X-HE-Tag: 1676524698-370311 X-HE-Meta: U2FsdGVkX1/1YvGOvfLqC+F5GAg6iSkVmgOsjeklvDepgiSlwiJ1GJgzK66Q9WPRGQFgcmVCW7SgEoWAIe4yx4qi8YdHxecEboJu9SSrb9LPIG1g5fH5XnYaOoD61wx/QoNlamYh5QlEYxiVTs7H1hD6TaL6o4Yfw5cAfuWO6eoHuQfvdF7sMhM5g2iEApM8BR/uaTebyFTQAKBxsGNiBgXW4zRYfHfNv9FU/Rd2Q3ar6Uz/fBnyjkG3424cTBE0Pq1eclHitf90N6xwrV+JPa6e0noL0Z6DsLB/UJFHr7klGZ3hqZvaNU530rMMMk1SDicYXfIdDDzHejVfDOR9UQmmE30RShBexzeZgu6SJT+mFtXUDluVV/458lAdhD53UZ5pBNkDkTAVJl3mU3qB5BWQJUl0M7MrQxHGnPNeaoSXhGxbne1cSe6OTaE5BslNLu4o06eVXdkwh0sCD0J9Dj6ArbowOEz6dTVD+I7Mo2mXKxGYbVkqZ4TUg6RFWbeq/uNIDAJrd/ME9GXnTsQMO77b9bfR/u8AGfIVtUACmCKY7z1//cP9vILHtWNC8YCx9vtwdlAJzTCbK2jHKTmYLs//e7yCoXZ49LeDjtN6mnyy+zRDt5A1R7K5Id/OQyOGg6ujhvi3+v0tDXfbG1tG5/bLzTjyKxoVQeAeWE6t4DnQhYjg75pPM5YttmGZjEvmdOANyp5W35kEa2acrSDgG2QUQI6auSV6kOImvaAussIOI15TL/t3BiMd3ZIwZxN7h9akNpghWcD89cczAd0+UqotqHYQSrIfEiknC+HohFIJZUpzg88QSewXQ7Pc424Bb4OfJbcD72CjaggFEWnckfmsSPKHZCY4gfihnJCWi0sefMFeJAxMqoYLzAIa6E7K9XBxGtIQGPfql74BxJNO7dPcoTaMhVvTi47RTckzlolnt7b2EC9gvWDSVCuplXNeduf47Hjb8e2v/qhrp8c THDou+B3 R0/0pKVbnEqFvt1xYKPUZutnjjNj2wVSIZAfVEKBjzuWPtkJ865JO36+PsdZ8kOj70wkS8SBWfn1ZNEP1aD+OlFbb7mVcO+Z5eIjqi80q0G3+LpPPjXUuky3APN6+9LhW01FMBssk1TSMnq7a1119AJwkqhBUIoPseATrS/+kFG301LfmVmUl4ZslqQ2xp9OAM3K8l2+qhvcVb35kzKPtT3kE3kqze8ITu7cKDCw/BG9HXNLbna3bazrFkU6qhIt11qWsz5SKmYbGDbDlfXkhfnhrGVIR9wOI3NdaumXUAPhMGW2licho8Br8T+LR+Wvy/XoR7AkWcB7SDU+tcWD8eFcygYXniEQWggjvIMVeEheyAdFPICFfqfVbholziL324g2FxwpYLr+RKXkX4nKUuQaldaw9onz03QxcsyE7CYwuNAEja5gk4pZJewhBjCkJ8JluNiEGaSlO0w+xVU2oZSATcGeml7CH0+QWRjDeX08Hto6u4iYCtrZEXL6WOsDWzYEHNC4YPeghn6KxEJMUmCuAd2DpPpedBm5Ub0oIXymkOC7SZcgbLTLj/dZzA0v8rRCEHH5rLJPZ2azYJXxOzoHkjRu6KkBT05nmrXR6890zWM6yQN78zAhakf6QyMhfS/eIVVirGqb2MNeUie+T5ETqDMWNkMQzNrb4NKRVQKbJld0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000005, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This configuration variable will be used to build the support for VMA locking during page fault handling. This is enabled on supported architectures with SMP and MMU set. The architecture support is needed since the page fault handler is called from the architecture's page faulting code which needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index ca98b2072df5..2e4a7e61768a 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1211,6 +1211,18 @@ config LRU_GEN_STATS This option has a per-memcg and per-node memory overhead. # } +config ARCH_SUPPORTS_PER_VMA_LOCK + def_bool n + +config PER_VMA_LOCK + def_bool y + depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP + help + Allow per-vma locking during page fault handling. + + This feature allows locking each virtual memory area separately when + handling page faults instead of taking mmap_lock. + source "mm/damon/Kconfig" endmenu From patchwork Thu Feb 16 05:17:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C90D6C61DA4 for ; Thu, 16 Feb 2023 05:18:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6752F6B0088; Thu, 16 Feb 2023 00:18:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5FDF46B0089; Thu, 16 Feb 2023 00:18:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 452416B008A; Thu, 16 Feb 2023 00:18:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2FA966B0088 for ; Thu, 16 Feb 2023 00:18:22 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0B645407F8 for ; Thu, 16 Feb 2023 05:18:22 +0000 (UTC) X-FDA: 80471999244.13.46DE3EA Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf10.hostedemail.com (Postfix) with ESMTP id 41C2FC0007 for ; Thu, 16 Feb 2023 05:18:20 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=KnhAt7qu; spf=pass (imf10.hostedemail.com: domain of 3m7ztYwYKCPAkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3m7ztYwYKCPAkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524700; a=rsa-sha256; cv=none; b=GvV++fVV5ezZcjVnweTs9GJebM/QAuYUZ65HQgvyF+YFjv28GLKRKiTwrsEhG+uvxB6wyJ gehjmddWcZ8UdKQljFsD3umvH8pH/6LvC9G/FvoXpVkuMNsq5PmrzzIS7haqIfVJQIROsg mdfCvlVQbav/0jWwA5qcRkLM7fxOI0M= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=KnhAt7qu; spf=pass (imf10.hostedemail.com: domain of 3m7ztYwYKCPAkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3m7ztYwYKCPAkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524700; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nnxUMMQI6vT0M/Q5sa1IAaTl/WTbc27CfHmxL3/FviM=; b=4kD9dKn6hJEZvJZVtiuvyEMAdYFNYr0a2SKNPlTQzdLN98ZSKZVGf79JLGh86yfeyJ1ITp QItVmxZBgyi/f5pwudqD1TCsvM19+TJhchw6hiWN5PPXFa+u0qZRXW6n1X0XEyXT35gSnb lA6djHvuFUI3IZULA6IF/Rn4AUyKjYg= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-52ec2c6b694so8316627b3.19 for ; Wed, 15 Feb 2023 21:18:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nnxUMMQI6vT0M/Q5sa1IAaTl/WTbc27CfHmxL3/FviM=; b=KnhAt7quYBwXQRhfuSYaJDek4wttI7RDXOOACIL/F63Nz/58eVv03No3Zqh2UIwNdl TkKGisyfzRrXImR2jOkU2wcQekf+kt9/XxXIsRoXlqskuIeX6hOhHTNZXgdfaMplle7K SAATQyrMS3HXKQy7j0lc9Kc3dK3KueDsfgI8ryy5OQhnNBROeaCbU2Hmg2SLUQOwNbgt XwBV7zK7XX+1ClOK24Y2b0Ex8amo9ljLG0ZokYkKzEniw9mpgT07P+84GOQVR154zkPK 4YJcp2AEGd79KcxB1dsy4RuRGjRlYt9yxaLz/0Wa2WtK7Y2fqnfOErlG1kh3RHBuBBJK 58yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nnxUMMQI6vT0M/Q5sa1IAaTl/WTbc27CfHmxL3/FviM=; b=FQKJEO12CVRsb4Tycx441XXUl0sAOSJv+rs9EAnK98iRiN5k0shfubvYj5atdDw3dU u/KbZdmg1m7gcszDhmn3Ptyu3VkjwzNtNwmM2eLJfBzwaVN+qzirLMvH6k2tsRaCIil3 V9BqCwfbpbv7K5ERX8603zb5XfQgHr/ekDjmFH5BKTM/t/5i1rJVW8MxIU5hcXwT7A2w FmfGojb9akOEXRc4qqygJH/z5hj/Bjq7zZ/TFJhpqvVOTv40s5rP2Lxvd5pNdldmYy9E BpbruFCUdcHjaZPxLFuM8Lr2QLpqtkRdgfVGysqB9hUSiHCkf0NFxMydxSnoAyjnvCE7 VBBw== X-Gm-Message-State: AO0yUKWPImTxOS4XanYxt/el36nUZpwYc8VuOtxH0Rcnc5bp6MvcwDzq cV6LnFscZ6+A3e4XY8FW7v3u89jr17k= X-Google-Smtp-Source: AK7set+tlzm8noztFtNGPsWUqf3Z7zno+mYjAZ9UmWznIF5F5gJ5LrgX6pBfJhOwGepp15lWUy19j/IuAtc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:a20c:0:b0:52e:f79e:b639 with SMTP id w12-20020a81a20c000000b0052ef79eb639mr389199ywg.283.1676524699450; Wed, 15 Feb 2023 21:18:19 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:25 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-11-surenb@google.com> Subject: [PATCH v3 10/35] mm: rcu safe VMA freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Queue-Id: 41C2FC0007 X-Rspamd-Server: rspam01 X-Stat-Signature: 8p5eqyrgrna845p5nsiujfoj7oxr1ori X-HE-Tag: 1676524700-528578 X-HE-Meta: U2FsdGVkX19MaQ3BNlC8P/MnGDtHDYzLcGEuLCfo541+NKm++mj0RsgWKD211FGP7+5sRuup0VpkCm/YHKLM8Uf3s1OLMH9ky3KWhNdJwBOQNFTFYQwcpLffd/7/Y1KyITsclpFH7GaVjdtV1QBZwo9kXRW+KcHaEeyyPP1ziYRTGzbB4RodvU6q4zorwK640fPZec5Sl0pH1MSN2k8ZUnEpScWjCIQmqKUmcWJifzqwU5WpJ5f45v0SZQ3mpSgky8N9VKXXMTouAGr/jVrb1IgR+RhfRwGbVdOnJVb9MhWjq9qf1qf0rixmZ2TNA3906ll5VJ9rtoGjq+31JqQLi+nA31Xin43xku/RLFg6R94jMN4uNH2gaNOAY6ssfcIviS3ZekGrhSyVf+D5e5TYx7fecud5pWviBCYKRmonEutsIftonupPfGVvBx9IElJszNflbDalp/oaj2X7H73EhbxwdzzhiFtHgTY1LvoXWqi5q1K9cGxSC2lzCLJeDXH5DUlVTMYEGIlN3eSPGdt2yJY4a4gIbsqNYD5PGQWoSewoo3kX2giqPwWc0CSNvk0+lX2YstvNpGwp0TKPXjdKbOufmwTkcmvq9Pd+LtM30GACsMEf2umwggccSIUt0YYvcxhrGTW//NkNi/0BEctMrIi/rwdjYrHsOay4LcJhNimUzCi9Q/IasJQLmlsNhG8lSIWivdFHJ6WselNZAOaRURZhf/jhms0GnzCffrfEJPmkk9M0PbC6biyC+2T34sg2zRcOX4nbKdI3NVCwdLJwyMdxAXwlJsF4Fl6vnNErSVEGAqkHuX24Zq8tVpea5E2xiLP1GjkXe/AIeKkBFq0GtVsOdVqqRPkD1sXMfcSQmg0RhBcR9FNHHtBgc4hJb1sKXf/hiTybn2pNZYdQVyfRhA0XUU/TSpyfWuvAUcHNigQ41XdZLTWFf2FqxDmxLFs3P/94pjiUJVFI5Nd/zPx UWzDyjwo g/gSYC9RLt6juy2U0tqRLU431DFIXMVPKcnE3i1Pj7QF3t3K2RDRa01YbqAQE5Ob1cSgMMwD4VXK7Cn2QtZ8lHGie0b+SBaZJxtpwF+mXDyMForndTXmHUCScHC8Y5cJYtPkoSsFsc3tPX3dFIlR8M9IBI78wHBIdFXVRW/KpC+YZiQWxZOm9JSrSWotmznSmrcURYjvlxwXY266oa5N++CBkJIk0SPu4TQQ1Cd03qWfewtUmqxgLyn8fuRZwq8cwacpwTzKOdN+N4rTsvfmgU2gtcKEpr23iELfrkLUx1ktl8rW6nn62Uy9mImSxIWMuYYiu6kK6+yzZNns4viWhOer04VqNQEBouyxhK9xo+xIVyxh0Ywxe5mZ2RUj6ZIgRA1rLTod14GomyRhGIaMv2UL2NNwLlfvD5XMLEliKDc3zjw5jrq+xAWDTtBCiwsKuHa2KE5CaRiyQ/XTgL5eksyZw8slkvpr+fUr+RoK6C73IOiaa43RaN8vCXXIQUo7dtZMmUOT/j56N7HwD3G27dvgz4bIMvjp/813CiP29hgl10C2DEPHytj6Q6i8XjAXGoo6mRVd++gLWsecrYcgck41d2AxTefTfbWMKQYawRs/AvwT3baJG1TJZCFQXcvhF8QNYBLzgZ7n4+IHtMUDDp4NL0mt8nfhyCNO+ZnXn6WogKTezzPag7PMqWg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 13 ++++++++++--- kernel/fork.c | 20 +++++++++++++++++++- 2 files changed, 29 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 3fd5305dbbf9..fb4e2afad787 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -480,9 +480,16 @@ struct anon_vma_name { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start; + unsigned long vm_end; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct mm_struct *vm_mm; /* The address space we belong to. */ diff --git a/kernel/fork.c b/kernel/fork.c index 5f23d5e03362..314d51eb91da 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -479,12 +479,30 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -void vm_area_free(struct vm_area_struct *vma) +static void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); } +#ifdef CONFIG_PER_VMA_LOCK +static void vm_area_free_rcu_cb(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + __vm_area_free(vma); +} +#endif + +void vm_area_free(struct vm_area_struct *vma) +{ +#ifdef CONFIG_PER_VMA_LOCK + call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); +#else + __vm_area_free(vma); +#endif +} + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { From patchwork Thu Feb 16 05:17:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142504 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2DDEC636D7 for ; Thu, 16 Feb 2023 05:18:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 58A6A6B0089; Thu, 16 Feb 2023 00:18:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 513866B008A; Thu, 16 Feb 2023 00:18:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B4746B008C; Thu, 16 Feb 2023 00:18:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 23BCE6B0089 for ; Thu, 16 Feb 2023 00:18:25 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F20B6A95E3 for ; Thu, 16 Feb 2023 05:18:24 +0000 (UTC) X-FDA: 80471999328.27.D0286D4 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf29.hostedemail.com (Postfix) with ESMTP id 2A8FD120005 for ; Thu, 16 Feb 2023 05:18:22 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=MfUNnuRV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3nbztYwYKCPImolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3nbztYwYKCPImolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524703; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=SR6rQnzJYl3AIs7PSgbfP4pB8X2XW+dcE1Dof4x9AMtsr++S+P22qM3yd0S7Qyrw2vm88C mFBrwh+M4B/gqWMD9oSXuk8l7YZe/ndAh0vAGGzJIb1WvIcCE9GEV39PXH9GfcEtIqCkDJ 02dB9bsikjfMf+fS9pmAf8uahCvzKNQ= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=MfUNnuRV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3nbztYwYKCPImolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3nbztYwYKCPImolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524703; a=rsa-sha256; cv=none; b=uFjl/z7ODefBuAtfPB61lUkcyKUiInnXQCI1i+dWTA8Ab722RRwW4ViSxBVHYpHWAxjtOu QRIDPzAF+46Z3AUZEyl+8jnEJEJp9ETzrSCXRv377heCc3iQ2exkXYp60wCdD5rgbEWsgj z+Ju8q2sMgxeqOHGt0ZSN7wnh03nKAs= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-52f1641b79aso8460007b3.16 for ; Wed, 15 Feb 2023 21:18:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=MfUNnuRVqilTlETV75doNo5d3X2f1I64PkztavWwbIZWW2j+3a5GtwWp8jaeE2FAS7 MLS5o+W6QzlLw73Ic3jcbhKR5fcnDvpJtAm6NUi7rdrBvnG1z5U94zlj5p9Rf6AtiMXB /0WTQWR4hibDfKvJz8o49Ph/XAqUGXxJaWOev01ngzcVh9Xd6JHGOFVoBB6DDs9XGB0U jDPi/dJdwD6mt/I/Iobb8XMkCnfA0C9OXZWj96tLYiXqom79AFLMHYYopE8HfxvEiMuA aI5RyI6LPkvxen53dNb8XfFirBUFuNssxJItswg1uaEv8wS3br6uTFvWft0h1+Cz189K IN3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=hDnvax6we9gCw+hmEGMKdx7n3344u4Ye7fNhKllj+Q/Ovm7TtNSpuaVWfYLMqfNcDJ l8A8sYzE63kZAM+kyNsvZHAkO+bBH5CbtSe24fTi+M7sZQHD9Cgqc1a36Vlx1Bq8mxeM osgEJsRDltYn28UF4eBbEzoGvMU/tfl4XXGL+marGu1ppTRUevzFCykFfD0rHhSt9vKr ZsUJou7JR/BOnb7gc3qWM208+pMMeP+7qrKzzeXJsQN3BFqCtFYrtDAjgCbeA6W0Srbv 65BJHx16bvEbof6ksgBug45opi8qDch3GfkhL0b6MlaTeW2OmoahpwTAO66PGZKkxYL6 4H5g== X-Gm-Message-State: AO0yUKVeW4KzQRxQkvaWRz+kANjtjPwALkbKVT9gaOCRwy4HKD7XpSyX Compph20ei3xdblHc79P1kEqGTGTcBU= X-Google-Smtp-Source: AK7set/k5YICRmbO5PrlJaB9Zo139wu7UxHEXvq6ujpnvYnblxA9RcHNr8XeBP6uxC5C9xnIzXdi58p2CG0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a25:8e8c:0:b0:965:bac9:d458 with SMTP id q12-20020a258e8c000000b00965bac9d458mr0ybl.11.1676524701625; Wed, 15 Feb 2023 21:18:21 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:26 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-12-surenb@google.com> Subject: [PATCH v3 11/35] mm: move mmap_lock assert function definitions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Queue-Id: 2A8FD120005 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: kcma74wtapt1shs1phijh4rrjdwswf86 X-HE-Tag: 1676524702-443020 X-HE-Meta: U2FsdGVkX1+MhMaS01m6QYBqfbm9N9M20esZLJ/ZfkyD+JRAilX16/NpHgibaoaHIt9jcMSOjMyp24N2aMmExTyB2x7hIrQJEXqax0jwd4JxlLmNTPNmvByNF99xXRCC+Uf4vB3HXXNkT0JS9BXdjTUX6EtBhzzox1zSEA2aWMvRdreQ7ueu62fv9sILos3WUM346NpCWBy4L3iIB1KrFR5A5nPd1ijvuubIvvICxyum2Xs0ovMThwyGtN5Yh2JrbUX90GZthJqLSFRXUW/v9AYNtxWC0+kbR1m+VESFKTjUgpqkBf9p63jXmDXphiMpyjKvAeZcTJKcZhD+TOiYz4/cO7oIFDZnR+7JEFnYYw9C4UNxtUQMQgrmC24czdar3eXLK0cf6uaqLmsN8pO5kazy0DrCc7+1AO3o8WFoDBZGOagsKoWtovMn/k7slaMqX7EsiEgLPzl+ZBFNZcpTuMXlOwHNCpY1BrJ2sxvFJTmvBaHFfjZkQzP1+c0hmc7NbH6ui/U+fwD0xRP1aqDBgfTRR/pcy4HNRF1evvSe7QpJO0xz6HrB2TogiMhVVXvO3aFqCeg30RD41T/guky0WEEgLTQnPWxu7Dbzm/59EOpaW5p4k3z9wDxX14xOM/FhLDVnw0CFLS7sAYtufnuqF/mdPhGN1dDYMq18/C/+JQcXU/VmhIEtBg9jRxDIaKLEZIbRat/IE5GMS4i2wTYtDas+hrSQ8jgrewdE1EVU1WVlZWDzFlUr/Fk53S3Esk3hNbfmSdGtmtzLYhtiHw9UwZtdMWKvMIagLhV/Zw/wrHbwSgh8yJZ6r+vnYOfLhUmfclAY6OhKx4UtdMmhxo97y9LwRDV05E7dCbpZhtdx13fPL3tsnxs8ganV56gQDB7rYZetCX71E03WAybG3z7KphaYNtBbV4Mg5YqdtKU/6sE22rOgn9LmOJ6dspeXzXbRrLkqZ/Ujgsa1pKX5Tk7 wtlsm6k4 4YFXSuAd5dWsijHet5C5ITUD9/v71WBmgDLZNvbrVZuV2qQf1p96v54JCsQnoqoJNR+lAjXMKOwdCXxaBKZP8mVGlwC3maThNx1Vxfr0v+Lm9RqE6SCiYbOUiqfTeDMdP9Uj6e5ndj2JApEBxzix9tmNE+25PSIiBHOKBOVbFVTqif++fJp6ql9Vw9j56IghdC7WNZe24m9TeqrPkLWu9xiwlYZjO8mpr5LV75Ve16acDX73KF7s9n3ffG7AaM05X5RDLm3bvLUmo3lVT8jhC3HIM5+fjTE6SqJ/4fnFPOp+fdDC+yfajJKnliFrBlQmQMWnEbMIurHkurZjiGkQyI98BwpkM0MflsaurEico8NMnOwy8nYEi8kd9bjsqcQsozWhOu1003dhdAq+ZFpBzlUMN6rTq0P0LT1bOg6ep+zAquxrLK2gRdvVKTzWmfOREE6wBQiv40e2XmbNOuiowHK4fKNWPjCvP5/w/ClY+e0jJwKfGzBdkPazvyUHzb5udavQQb9adbwa3n1+Ia1mqxSOigQpMnyXxSgbSnAv8v8gNNVWcFe3NFyJ2ROkFVkBjzM+xNRvNFZDztQwHMhCDEfZZckrlEPTAL7Xn1IwK0mynE6zDC9tzWRkCtFKl4O3kedZIBDLmLitmT5UOAn7dyNwKVLyp9DsSUEOG0Q1I0zI8ZbM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 96e113e23d04..e49ba91bb1f0 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -60,6 +60,18 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write) #endif /* CONFIG_TRACING */ +static inline void mmap_assert_locked(struct mm_struct *mm) +{ + lockdep_assert_held(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + +static inline void mmap_assert_write_locked(struct mm_struct *mm) +{ + lockdep_assert_held_write(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -150,18 +162,6 @@ static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) up_read_non_owner(&mm->mmap_lock); } -static inline void mmap_assert_locked(struct mm_struct *mm) -{ - lockdep_assert_held(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - -static inline void mmap_assert_write_locked(struct mm_struct *mm) -{ - lockdep_assert_held_write(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - static inline int mmap_lock_is_contended(struct mm_struct *mm) { return rwsem_is_contended(&mm->mmap_lock); From patchwork Thu Feb 16 05:17:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142505 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58678C636CC for ; Thu, 16 Feb 2023 05:18:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3E866B008A; Thu, 16 Feb 2023 00:18:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DC7E56B008C; Thu, 16 Feb 2023 00:18:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C18DA6B0092; Thu, 16 Feb 2023 00:18:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A75696B008A for ; Thu, 16 Feb 2023 00:18:27 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7AACB1603AF for ; Thu, 16 Feb 2023 05:18:27 +0000 (UTC) X-FDA: 80471999454.14.DCA9F78 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf09.hostedemail.com (Postfix) with ESMTP id AF75B14000A for ; Thu, 16 Feb 2023 05:18:25 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=jg6fXXyN; spf=pass (imf09.hostedemail.com: domain of 3oLztYwYKCPUprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3oLztYwYKCPUprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524705; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=S0NkKUx4dkIelWS5aTsq7UgpXIDBIhGrMRdd8DrqTMg=; b=E8KpVZzf+aMTVIqCOc5SwqtUp/KA4unTvYWIWPXsjZjb//D3M0U6yoBtpLXEw1VQHTL0hd AzpvvhaqqstccsYH3O+a7rGmIGAngm2ppRAtVdFg+v4XSrY/O/d3V7Smjlo1Z+G1qj+ifO kVUQ52+6MU4Wcihpl5oEIxrvLq/8TGI= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=jg6fXXyN; spf=pass (imf09.hostedemail.com: domain of 3oLztYwYKCPUprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3oLztYwYKCPUprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524705; a=rsa-sha256; cv=none; b=bAIJVptX3P4EktuK+1lSCS/0CzuM4AwA01Hrhzcqq/AMAG0c6DRzxpSc+7x958HNLjJibC sVPV3FHIiHwfFwCqcaBYgaFdxZjiY+dXK6iYKPMegc83G6O+FhTii6MTi3bGC4Foaxsn3H lPwpy9/+Iup4Hy4QEZnniO6MQnSAmUs= Received: by mail-yb1-f201.google.com with SMTP id e191-20020a2537c8000000b009433a21be0dso836281yba.19 for ; Wed, 15 Feb 2023 21:18:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=S0NkKUx4dkIelWS5aTsq7UgpXIDBIhGrMRdd8DrqTMg=; b=jg6fXXyNPA7kGzHuabxhVQS/lUkV32r/2v7x3huSOdcQqflpdfqhoIlNaR0t9wMIZ5 tNsdAuEIvz4tV4UF6ouWYwPH6jjBst45X8ZS39R/or9bhVEvk/hAhXAhGiG6HODkz0Ti 4+W9CepIhu/sbVVhPdRUrFOaALk+XM5cteC/he2x1EN76QEzJ6hbvpDzGk804gLMUejh OhIx76lfAQ2Aj8i3vKfn29ePpUTgzAK6jOUxJ6v4seZ7ANgrIyb45IfAAVauV03+BjjR E1pd5h52L83bbDKmbPDtlCRxv6trm/z+Fa4Qs6PgKsiEdeHNP77Ta8uAxqqtUigosnhI vlvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=S0NkKUx4dkIelWS5aTsq7UgpXIDBIhGrMRdd8DrqTMg=; b=BDXJ6E4h89LorbCJ453+L9CAzv5VNwx8jp4pgdB+n8x2aWFChOyKsD8x99ci9ONfGs gByD1olpsthtCLGUJc7eYPm0FjqnmVr54UoismLocvYWMIlU3z9n9H/GKoAg4m1RnTai i0dGXT4cwAu/RzXgcr2k61vhAQE6cj/X8hG1m+VjhikzeZeQE9fzZFp4I4ZOFiC90vSi jftmP93OvME3ErRjUz6YJv6ywFx2vaKTHdNqoZaVzP1qAIMWXJDh7DKCoGXKI/BcobaA 0S//5gE/6FtEmDdGm+hZDxdXMqBPhRfaFJpSD42Xh6fW+xocafJi64tiGJueLWIFSXwk 5UQQ== X-Gm-Message-State: AO0yUKUyr1ODBqK3RUSJzhVsmIAvvPJ46LVh4Jd9DGmz6v10TXUUfE+P UjdHgf5VU71w8MuHUPue+lZ50VtlUNk= X-Google-Smtp-Source: AK7set9uXuFcAAjB5O0QV0Xb2ftT+HcRWdedCtf2z+W+4L/MaBisfzCjx3cFBAo6If411AuBWDXU7wCyXBg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:13c6:b0:8da:3163:224 with SMTP id y6-20020a05690213c600b008da31630224mr5ybu.0.1676524704439; Wed, 15 Feb 2023 21:18:24 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:27 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-13-surenb@google.com> Subject: [PATCH v3 12/35] mm: add per-VMA lock and helper functions to control it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: AF75B14000A X-Stat-Signature: pk7zwxsobpznj7tt868rysfe3xma1zc9 X-HE-Tag: 1676524705-521655 X-HE-Meta: U2FsdGVkX197UE/AspTew6UUg0beI3RJ9ibsLiwfwG2lTOzb2jEwWBd3qqIOhm2dJDMQpehWeBZRJTUBnJ4ny1IVe0HpRsnV2CaWfPF5THwjmkwjGrXVDRCTjdB2gSFPb9VzlodZLDt0k/7G+k6rX4b9c3HVke9rBx2LKIO/OYTRUOoKsrilfF5sV6mno7EUkC0c4ebXQDZA9l5RKXqioO1LkTYSDyaUtpRrfBxHVGHfnwBc5/tFMSbTbIhmU0tkjT+/Yzs3pX5J9XfcJTabtly0C4fbgixI/Hp2jnDm0mAaSMuxFyVpyY9BfckUohISDU9S65S8AadL4aYXbbe6yRpwqZMpDNKxQgUAveWKpsW5oRurzTU9QQgUWcqzKypTdJKhRybHW3W8KSNUwHCaBLgS/ALmj5SVA9cMq1OxomXdq0UcDRN7hTgoo9gJymElTPo8Ly552ZdkMzX3Rsna3Jeyr8Qcyd5VY0Qm78JU94cmouqoQsaWOpTZjze9e/Q5DRrBcAgUh+OqMFeLRG8sXAhs+somLRBTd2D4QeOqUW/VGxpOioE7NHyMky4VE64dyNkHjU95NURSBMrdCuYCMwp4+IEukOlIWMRWTJqd9nWQ+lHhOLRzTm0LR1nGSR57WqYdkfhYJOOB1Rvm3nz18ve3SVOFeP69NgTqrIXFLZspQ5fk2TOr9rbVH5cbxtI8XX4c7gbUYrz2Rqq+3oK5m5OHkHG7m7VyazaguSLIdd6DkXIHCefyE9NyT7cZjzLvFEcnXlVi4BI2x0XIns5r8KONaGUTUlGTwZ4XqyzL0kh2hjJRze+dr5b7X9U7cfvpVNUkeFUiLwiyaZcVuK7+fdBd3inTxhHDg6z1MkGOCsLQgFDYlzApl7w+8DiVxNgPe6iNw94SOhW/4wQb4UjTrNcV0o9LqTJXMFy+dbKRHmqbvKsYkqSx9eFLB2UQrODE77BMhKUjo8yIHPXPdMw IJ2acfWQ ctd5zucC5H97doSZeaVPk2lfkzrfSWEV2gYpvHhd9ITAwTmbR8yeUk2bSVHEXAPLMaaYVmsO6yx36oEu9QJBR5XqEMM2xFo0Y7laIDL+ZAjlIq5yK0f91mdoe6BtAyAuDmXdGu/qKFg5Oq4aBSNElRMIw2CB03SoYc0MezIGGxXAkAjNNMNOdYYWYE4ZX9ryEQgJVHBxlR2kt5A4NrHLZkB3t5fpfo/HB8PBonNb/sg6byfRuyXrrGAimqzhpbofApIcll/UqvsjkmPHkmWvRYd03Q1icOy6Jo+87pyXd53bDm/gO1ZAtTiMBEB7GV0F7v1fEroIk4fa1/xbeVYfqh71S29QsfQuWse2TrtWpCIx5MqFF4FYj3GBlVsy33C5OM7vZR3w3PTFeBA9Z+4GiGfMMsfPVJoyQ14Hyi6DMhmR40gyfeeHIU25RyGPEg2wELHfya+aj7YzLz9Pcb8EQT5vwK3YnDgJddQcApzsESbvdN64Q2/ECsCwjSwC++gVXKhgK+DLwY2SOIoT4ObrM4OZkECBOzCF94OB6H6JHw3rb1zZTbRirPsGd5A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce per-VMA locking. The lock implementation relies on a per-vma and per-mm sequence counters to note exclusive locking: - read lock - (implemented by vma_start_read) requires the vma (vm_lock_seq) and mm (mm_lock_seq) sequence counters to differ. If they match then there must be a vma exclusive lock held somewhere. - read unlock - (implemented by vma_end_read) is a trivial vma->lock unlock. - write lock - (vma_start_write) requires the mmap_lock to be held exclusively and the current mm counter is assigned to the vma counter. This will allow multiple vmas to be locked under a single mmap_lock write lock (e.g. during vma merging). The vma counter is modified under exclusive vma lock. - write unlock - (vma_end_write_all) is a batch release of all vma locks held. It doesn't pair with a specific vma_start_write! It is done before exclusive mmap_lock is released by incrementing mm sequence counter (mm_lock_seq). - write downgrade - if the mmap_lock is downgraded to the read lock, all vma write locks are released as well (effectivelly same as write unlock). Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 82 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 8 ++++ include/linux/mmap_lock.h | 13 +++++++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 files changed, 110 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2992a2d55aee..a056ee170e34 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -623,6 +623,87 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_init_lock(struct vm_area_struct *vma) +{ + init_rwsem(&vma->lock); + vma->vm_lock_seq = -1; +} + +/* + * Try to read-lock a vma. The function is allowed to occasionally yield false + * locked result to avoid performance overhead, in which case we fall back to + * using mmap_lock. The function should never yield false unlocked result. + */ +static inline bool vma_start_read(struct vm_area_struct *vma) +{ + /* Check before locking. A race might cause false locked result. */ + if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) + return false; + + if (unlikely(down_read_trylock(&vma->lock) == 0)) + return false; + + /* + * Overflow might produce false locked result. + * False unlocked result is impossible because we modify and check + * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * modification invalidates all existing locks. + */ + if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { + up_read(&vma->lock); + return false; + } + return true; +} + +static inline void vma_end_read(struct vm_area_struct *vma) +{ + rcu_read_lock(); /* keeps vma alive till the end of up_read */ + up_read(&vma->lock); + rcu_read_unlock(); +} + +static inline void vma_start_write(struct vm_area_struct *vma) +{ + int mm_lock_seq; + + mmap_assert_write_locked(vma->vm_mm); + + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + mm_lock_seq = READ_ONCE(vma->vm_mm->mm_lock_seq); + if (vma->vm_lock_seq == mm_lock_seq) + return; + + down_write(&vma->lock); + vma->vm_lock_seq = mm_lock_seq; + up_write(&vma->lock); +} + +static inline void vma_assert_write_locked(struct vm_area_struct *vma) +{ + mmap_assert_write_locked(vma->vm_mm); + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline void vma_init_lock(struct vm_area_struct *vma) {} +static inline bool vma_start_read(struct vm_area_struct *vma) + { return false; } +static inline void vma_end_read(struct vm_area_struct *vma) {} +static inline void vma_start_write(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -631,6 +712,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_init_lock(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index fb4e2afad787..e268723eaf44 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -508,6 +508,11 @@ struct vm_area_struct { vm_flags_t __private __vm_flags; }; +#ifdef CONFIG_PER_VMA_LOCK + int vm_lock_seq; + struct rw_semaphore lock; +#endif + /* * For areas with an address space and backing store, * linkage into the address_space->i_mmap interval tree. @@ -633,6 +638,9 @@ struct mm_struct { * init_mm.mmlist, and are protected * by mmlist_lock */ +#ifdef CONFIG_PER_VMA_LOCK + int mm_lock_seq; +#endif unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index e49ba91bb1f0..aab8f1b28d26 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -72,6 +72,17 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_end_write_all(struct mm_struct *mm) +{ + mmap_assert_write_locked(mm); + /* No races during update due to exclusive mmap_lock being held */ + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); +} +#else +static inline void vma_end_write_all(struct mm_struct *mm) {} +#endif + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -114,12 +125,14 @@ static inline bool mmap_write_trylock(struct mm_struct *mm) static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true); + vma_end_write_all(mm); up_write(&mm->mmap_lock); } static inline void mmap_write_downgrade(struct mm_struct *mm) { __mmap_lock_trace_acquire_returned(mm, false, true); + vma_end_write_all(mm); downgrade_write(&mm->mmap_lock); } diff --git a/kernel/fork.c b/kernel/fork.c index 314d51eb91da..9141427a98b2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -474,6 +474,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ data_race(memcpy(new, orig, sizeof(*new))); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_init_lock(new); dup_anon_vma_name(orig, new); } return new; @@ -1147,6 +1148,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); +#ifdef CONFIG_PER_VMA_LOCK + mm->mm_lock_seq = 0; +#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index c9327abb771c..33269314e060 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -37,6 +37,9 @@ struct mm_struct init_mm = { .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), +#ifdef CONFIG_PER_VMA_LOCK + .mm_lock_seq = 0, +#endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, #ifdef CONFIG_IOMMU_SVA From patchwork Thu Feb 16 05:17:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142506 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FDB3C636D7 for ; Thu, 16 Feb 2023 05:18:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 033766B008C; Thu, 16 Feb 2023 00:18:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EFEA46B0092; Thu, 16 Feb 2023 00:18:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA2756B0093; Thu, 16 Feb 2023 00:18:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BCCB26B008C for ; Thu, 16 Feb 2023 00:18:29 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9364B121048 for ; Thu, 16 Feb 2023 05:18:29 +0000 (UTC) X-FDA: 80471999538.01.D10D3BD Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf06.hostedemail.com (Postfix) with ESMTP id DA1F2180004 for ; Thu, 16 Feb 2023 05:18:27 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=QqHoT4KB; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3o7ztYwYKCPgxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3o7ztYwYKCPgxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524707; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lpf0g6jEgEDruwUAtDYeX37TOF9sV0bBdZYAp5OaPFw=; b=h4wtK60Um9rHu1QfjgK1x/1ZlKn97PbilqjBsV5MmvM3w4EuLwS6EPQTOUyjp3WcBDpc0F 7sA/oTXs1YZT4rEIHJSmoZzCj0UPmA4o0cGk5KUz00k+cjPvfzEL1NNFpqZDwYGA5h7s2W Kc96v5FA4D7UkDdFjPYm8CSIPV7MJcA= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=QqHoT4KB; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3o7ztYwYKCPgxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3o7ztYwYKCPgxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524707; a=rsa-sha256; cv=none; b=VySqIvw1mVmeQgGxIeoiSvqR8jFX/Et5VEGENwqjI0GGMyzY3qhtW2GDQKhxUgzbajAzsR PRehYKExwTfzNhKHgBU37v5a3I0QIkYfSPyKuUDF0r+zqLjk6ZIh2k+NUVmKiIq6W8CVzF S4PRzGNeg/1KpFV/GobSW5tUVcWG1o4= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-52ed582a847so8806127b3.1 for ; Wed, 15 Feb 2023 21:18:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lpf0g6jEgEDruwUAtDYeX37TOF9sV0bBdZYAp5OaPFw=; b=QqHoT4KBvKUoUdLlMqp51fx+DvEGbEeCQBAnfd6VVaWQ1iGMSVWdcYYT+IcmdoGW6k 3cvpI3DbxmqWzlsZbZtdKxchg+b34bnVwYQM28D7iM4wfdyptvE97biBiYefhiTugMIi 6QLuIfwXdU8feO6tOQjuROgghWTm3q6OIMngPhTxYx3FIBx/Q/vwNR9bBonlFJnVLRif jIgBrdCbBh8/u0HLrcAEynfXgVsHAUyrN5D1sUvBi7o9DTj4LnFYvOofA3wqcNiCIoKK +suiPaGBCd0kL8KKwbFLDtmEf/YUc3gt6F4GpOlLFqmXcdgxQWnJd0Ox4qoA9H1pNQ9Z aRAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lpf0g6jEgEDruwUAtDYeX37TOF9sV0bBdZYAp5OaPFw=; b=Q2Xh2qLAQN/dv0mYaEettPndmc0V7T2nWxOH03Oy25fn8MLt4Uh2/+3VdtaVvdkAyd GZNKQA6ObPkpbf1lVIy0a0W7aCiuUyY9GDsyXSmatixctVTXCz+fkce9Tkw6InUC37so ZVI4CkaghLcXyBBbgFQz171ybKtuiP1TC1G9kBp0zX3XTprsmackSF7GNkM+Q5Ku/uLq BMC+xc1KKrzj+sCsWZvJFrQaniMFoFQpHRCH9i5x/68oI05ZdcpdmPQAWDLJdPFkkzSZ oV+Tc3riQoALQfoxCQJsog5xisA3tmMvGjQnr9J8XTRXDrSSTVpIIGxi3OdikkCJPURk f9vg== X-Gm-Message-State: AO0yUKW0eFIzukp2xysvOYObiB8FbmfzyGIhqoMNCoWSIAb/VmLBjJVr Jpd4lRUnqLsSfbOesjahOaO1qr+adIs= X-Google-Smtp-Source: AK7set9lI2rlt8srxq1B77BNUY6DsMc7qDNM9tTWpZOotyLmczPTVi4pDHBbcvVuKtvP6t7k4zPbNPC6UU8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:5a03:0:b0:527:83f5:3f08 with SMTP id o3-20020a815a03000000b0052783f53f08mr494142ywb.425.1676524707124; Wed, 15 Feb 2023 21:18:27 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:28 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-14-surenb@google.com> Subject: [PATCH v3 13/35] mm: mark VMA as being written when changing vm_flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Queue-Id: DA1F2180004 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: he1arpo87s953wrowddes7um1xqzzasd X-HE-Tag: 1676524707-511824 X-HE-Meta: U2FsdGVkX19HoUlthxPAwXzncsM2bI4v0P6u7FzELRSL9oYfBZhjLdCfuAtTy7reuUoiuaEAxK5aHu4dNhDlikL6w0M9wIXBo7Ourc0Fx8ZN1OA+DrO43uvNFH/9C9fDAuMSWpD86r25y9705sUx5YV61LuWSQ6Vm3Oetx55+yPSWyTATyjn8EYPXTRCRPGJ2hQ1WrBwg3KonLN41Qv7uHGOYaUnekTNKYRprmrkM5tw1QW1TePdS93jZEkTcdG1YwCAPI/otHlmd6m8VEqDeNdkqIdp3XC3nxcqUGMD+iWldcQUOhxgU5Q0pzpHV0WLxPuVyeSYUnzF7Jc6SgQbPYqPdnh3nKRwJyItRbtDc3Mz2fZqkEUe/kvq2Sqa/1kwkkwkOmHYCmK8CkLpePdlW6xcihTl+PeztR78Fj3kfg0VpCEMJ/32BRyxUgF3s6texKcsd3lMx5XO+RxGSwOQENTrqA6PWEWBb5V44IuQJvBYQr++KfUEUMHsyfleh1C4jHmIdpKBiFOSi0mtK6PhGVUP2x20KPygiUAys4yjsn35MaE7uZlmaTF16eJbyY73uoV0ryx1SdlH+Oeds2OMFza0C+PdVdXmbz2/sDRCF0TDgHQlCBGYb397J/EB5o4Utntzdk+2JYGq8uolVwkF3387/CdmTi/ko/BAb8+a/Yq3UyqjxfDT5d1EkzDlzCqhBX30yoLg3AySssNu45U90fbCV5BoRrovpHnZbKum2Kyw4KYN0Xdg1Ue+NtOt8tszCkIwMNzNAccMqRmxDcQLlSSF8ftvRw4oSiEzdIbsuoTG85vHlSccRCy2ysWNecdLD7t7PfFQZ6x3v+waSCUXKM6Nc1P9YqN0uC8tVNLJdXnSughQ1gX/bt337onQpel2tueSECDUdspi1ZIZtmTLVqfCjOjE88Rc2gDvtqd9b999KJjrCSpXt88wEjGX1JS9n9Tci3yd/DHcdbqVrOy AXlFLRe0 QHeyHpDykcdOWkrUrhbO5Vv1sXIWpBscH8V0xULC/gZiTZw7WIrga0rEacF9OW+3Ov+sMxxxs/PI587IvZjVV0jNSzR4/adWk79uEUUhihKR2wEksVs5oExxq+/5cpU/NVcXsUro/2vp45FiGZgte15FmwS4VpVvl0q586/cB2lwY9SVIrvCVoJ0o7geaMR8oxkMuuQeFBcD05SnQHuwnSkvHuKN9xSdtTUk5/n+PoeHktmsX5JjKc5Tw/NUjMJOpLhiYB5090t5aof70YhnzbAI+JKLnMop2/TWHRy/DTeJ42l9uqDip6nEGn6KKzSSOFKUslTrPLi1dO+phqPU+53FeHFwPFrgrKytojvvyNxy2tsJg5nVfZFlda6fRz4PWVvUdiLX/xaVtBtB34yK1errU3bHK/pLrtz0qf2+hRLXZOd+e4U7ZliEY0j130S3pQhcbm1htI+4Y0x4JIudjrepxfaRKzB6lI2NwKUpecVbgSzP9mJSgm7V5/PCGHGJpqpBP+9xNgv1YLVJkPRw0tfIaHQ8WNxTih/2WznTRmCAQY2RVpD9mQJscCc2Z49Z/3/L/hdn2VuWm8TGu2FO1QyEY6wlfJxOCd/jiRZzVPjHkY/zIJ+xB9zCye0b79eufeUGgsQaOxVl5kdphvZPBmAt+T3PIb7xj6p7Hm+CtIalk6QM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Updates to vm_flags have to be done with VMA marked as being written for preventing concurrent page faults or other modifications. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a056ee170e34..f4f702224ec5 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -726,28 +726,28 @@ static inline void vm_flags_init(struct vm_area_struct *vma, static inline void vm_flags_reset(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); vm_flags_init(vma, flags); } static inline void vm_flags_reset_once(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); WRITE_ONCE(ACCESS_PRIVATE(vma, __vm_flags), flags); } static inline void vm_flags_set(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); ACCESS_PRIVATE(vma, __vm_flags) |= flags; } static inline void vm_flags_clear(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); ACCESS_PRIVATE(vma, __vm_flags) &= ~flags; } @@ -768,7 +768,7 @@ static inline void __vm_flags_mod(struct vm_area_struct *vma, static inline void vm_flags_mod(struct vm_area_struct *vma, vm_flags_t set, vm_flags_t clear) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); __vm_flags_mod(vma, set, clear); } From patchwork Thu Feb 16 05:17:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEFF4C636CC for ; Thu, 16 Feb 2023 05:18:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 390916B0092; Thu, 16 Feb 2023 00:18:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F0F46B0093; Thu, 16 Feb 2023 00:18:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16B1C6B0095; Thu, 16 Feb 2023 00:18:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EFBE46B0092 for ; Thu, 16 Feb 2023 00:18:31 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C182DAABF5 for ; Thu, 16 Feb 2023 05:18:31 +0000 (UTC) X-FDA: 80471999622.17.5559685 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf04.hostedemail.com (Postfix) with ESMTP id 0B11140006 for ; Thu, 16 Feb 2023 05:18:29 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Aww5hQNu; spf=pass (imf04.hostedemail.com: domain of 3pbztYwYKCPouwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3pbztYwYKCPouwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524710; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pCPLSRzUXQbjlFnRzNKKqjCLNtpc7ieGcBzd48Gb76w=; b=oyKk9ZqBc3Z26ol/DduVCOj5LN/SfoJHwiK9d5TTN3MjeNOVjRoC4K+NIsj+bZWguxJeoW ehr+BJFPralYXYncxC7D9W40TIFFQ1dAacnuwL/NAWhfzKrHGebLTYx2Uli52Fr79Uyvk9 3JgUq8aVYxGq6G98r4SI1vprvGW2g/Y= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Aww5hQNu; spf=pass (imf04.hostedemail.com: domain of 3pbztYwYKCPouwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3pbztYwYKCPouwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524710; a=rsa-sha256; cv=none; b=INLtCSACjZ1MKOImnwjY0D7vMEL6aLK65MmavSCC4KdmzTYzMXzxirCcKezv2cjWGkCmMa q9YB3jRzmlW2wQfrUwzMC83DySCp4n9jCP+uLOZLWpxXyGnk744JjWfGuWT0sTUYnRA0DQ noHGHubIvt7sIZKi0K17aDxWFV8JLXo= Received: by mail-yb1-f201.google.com with SMTP id a1-20020a258701000000b008e1de4c1e7dso847927ybl.17 for ; Wed, 15 Feb 2023 21:18:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pCPLSRzUXQbjlFnRzNKKqjCLNtpc7ieGcBzd48Gb76w=; b=Aww5hQNuikXrQshFEtykppDfMBP38yEIiv8YZeHijhJepfYOC8bhl72PA7W7euJsgq UTdE3Yop6yuQ+QJ/imxEBpOD963eWsY1ljkA0jBaqGGg7cF+cENyfejTL2VbGXvtI8Gp zk9pQRaFettmR6L6WtIha8F6CvO6a31a83bm3SVswwIqM5PcX+z7DGQBjncH/5EAGpUc AT/XTvZe03bL8P4vyUJmA5jKJVVP1mrUYpMWnBW8gSUZQLx6c9BBl7Qtd+VfpJh9LBNJ HXZBHaakiRojDQMlDwmDxwqFzGyn5/4IdZKihfypYjTTs63kUZWxAGQtxOSq8MgWRU/t uPxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pCPLSRzUXQbjlFnRzNKKqjCLNtpc7ieGcBzd48Gb76w=; b=A0961sFvxiATHdCxTp4CoYuiJCWhXWLtHDCZ5gkhuDkorPriSeyh2kyVKULZdx/z8T 1WOhKIX4qeS1afQ3bQndsKnDgOpRJr+oGhealJnAs1tDWHeHPyrmdzTZYBsbiMePGCQC d4Zt8F3xFcpSh4fKQVj70x8hipjm11dltVZzhPtHbkFBlze06cUZPM8RNB0DqfeuKf+/ L0zGjuyutWJXWTVFy6dPWS7sMNm3nSNJof3LuL0RYEtgGJcz8difZ0sbv1tcIWsRmlFm d3esooqPyHjNcU+oaY+TDaFuTr2xrW+PCVmtl21b1SiiflSGI72PALOA3IRBfnOTJ2Os Z6FQ== X-Gm-Message-State: AO0yUKWlM9ihThOD82I396LhmMBzRDui0BoJIJa9/S87zZF4m9yIkdAe mHdM2aRm9LqmDqb4KfR+sxM3rgSxbfg= X-Google-Smtp-Source: AK7set+U14wrD6Ap1lad6EVJBTJXnEdfFV5kP7qrWMf6wQhZaMdydVfue60wPwI1XeNtgIFp8aY/q1HqCcY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a0d:d5c6:0:b0:500:e6a3:a141 with SMTP id x189-20020a0dd5c6000000b00500e6a3a141mr575676ywd.470.1676524709201; Wed, 15 Feb 2023 21:18:29 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:29 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-15-surenb@google.com> Subject: [PATCH v3 14/35] mm/mmap: move VMA locking before vma_adjust_trans_huge call From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Queue-Id: 0B11140006 X-Stat-Signature: 9kdig5ju8ibkaac5budujdkarjf87b5h X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1676524709-542336 X-HE-Meta: U2FsdGVkX19hR+M/bQfvALyatpNxeVmlZBisswt4n7KdsjjzF8NhvVQ57a9EEcieco10hlJILp5AvzCWqAhYH7WZlpc+LuvmWaijuyBhDWh3R4CM1AH+NVR0MW92SsP+hg9xz4HKGRLbwzVeK9QQfvdFgv7q7Q/042Kqf7TIsi/An70BG90CVPAcW4FqIPNNIH0tDif7lfHUbEqZXsqUCkK0QVzRkfINI9lsKbvR260gyQ47Y5Wr8dlcX0HsUrfzvHe9z78n0hoeyt1P8SLfNd773Ux8Rw6waC3oLE+WifYu31ybdYGYABw1R3eWVtbiow1gCdDnkaza0dJRTZbCUxOCkvbykokjoQOGYHJ3GSZpBqgFRa4XNr0I7cn9YhIYvRaFV+A1VfoQZyy5Qh/PU0BrWZuANCc+07wp9QEC60P7K0JxqAJP6h+teGCKtBYc4L1MRaRFXoFeVcUVkaDlqGdyK5BTHbBH0a8pp1ZhY6vKOJNjqp5TelAo+sBjtD+M7GVWZwcvrGDzLYet7FqDdpgIrDG2rgBJbWuKf3CFg177BukSYaEPCRdIYkNyN/PVktAXCtJq/kUM3ziF+RlUaMHAECb6JJ4KTs0mTzVAwzNPTlL9/F6HAzu+IFMdAQytsGO4OHzkRWHY/DxjGg+rZcKxlYRQmPGIlzCu7pMdjBRF1w7JJsz5f5GMIblMbjcQkXRKI2pKdaVWKWGI/3sYGDqbHmt843KumKgjMBSgzYdRZ9LyhabRIE4urHBu1mipDIGPpn/za3xXgiyDn4+eq2jq9/2VrSrU3hRNMoYI5P3Bluikll9dpQwOFLNgkn0WR/9vimCtEbsElbDn0nhA1LJ1knAHLUVAx7QDZA3sj22FkINRhXp+p9uqyz4TgNEfv/58uawgTU4StSiIUVBTZpKtVVf08cb1/8ESyvFW4oSnBQt+bETOe5EncMfxLRLEkBtMB/3ie7M3m7T2LlA QMdLkJlc iWpyIhT7j45LtcKxCandomSKGIxe+EIww8jcLhT0XRTHF7IOwxqs5az90jFjpYanURWx3fK+AumieWQslTpiAxi2DUKlWPCPkRrmh6J0E+3UXAg8lYPAcBzhRGTmUa2ocnbmkfbybZpc90FRkwfELiT4dJBxv7Npe1DS2g+MnUvKpR2D85tTO0Q6tagT6RgHl6kqVyUPNzbkGI1CMXC6BWV182n4aPLWStarWsepf8wEN9n4Fac6EELU+lP5JxleVqiTndaW84pgfNTOY6iPOQbXQxlO6r3MzJVbYhvsS5un1UeoV7/Hbv/Eq64g4PLcSjRPA6SdV3xBfsuS8hk+yiakrIRPPkSOuYnXmoxnH34bK0sBBMmRJSMDjbcukpj3IwFOIpT8oTSFUplicMcocyGN/eDRSKLjto68n9ZrMmOHTASPDxhjsTKzg2eS9yimgIi3eqUkMPKT4s3z6m29Nho6RX0WrBobJIa6QpdnvmvC+RVBubK8CW8VAzQJt7rVXr81yncOuGK2rhMLwDWhwfnjgLqRaJiQbmJQGdQxohehBI5A1mGv1/Qs9YgYG6NR3NouhlCO7J2N5jto= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma_adjust_trans_huge() modifies the VMA and such modifications should be done after VMA is marked as being written. Therefore move VMA flag modifications before vma_adjust_trans_huge() so that VMA is marked before all these modifications. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index 06c0f6686de8..c5f2ddf17b87 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2910,11 +2910,12 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto unacct_fail; + /* Set flags first to implicitly lock the VMA before updates */ + vm_flags_set(vma, VM_SOFTDIRTY); vma_adjust_trans_huge(vma, vma->vm_start, addr + len, 0); init_vma_prep(&vp, vma); vma_prepare(&vp); vma->vm_end = addr + len; - vm_flags_set(vma, VM_SOFTDIRTY); vma_iter_store(vmi, vma); vma_complete(&vp, vmi, mm); From patchwork Thu Feb 16 05:17:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142508 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24328C636D7 for ; Thu, 16 Feb 2023 05:18:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B0CB86B0093; Thu, 16 Feb 2023 00:18:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A927F6B0095; Thu, 16 Feb 2023 00:18:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 932986B0096; Thu, 16 Feb 2023 00:18:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 787396B0093 for ; Thu, 16 Feb 2023 00:18:34 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5725616016A for ; Thu, 16 Feb 2023 05:18:34 +0000 (UTC) X-FDA: 80471999748.02.E6C5217 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf28.hostedemail.com (Postfix) with ESMTP id 96E56C000E for ; Thu, 16 Feb 2023 05:18:32 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=S2sXkiCN; spf=pass (imf28.hostedemail.com: domain of 3p7ztYwYKCPwwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3p7ztYwYKCPwwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524712; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4tPY63bpwhpT+tcOBCZmVWQ4EsQSsAiOumfwemoUSR4=; b=cfqlaibP5tVbDvuT/oWcTZ+mmeKOl4c/1yTzrV61kyvyY85xmp8g1sBolSzXoZRfEPDQwK zqxkmEtjdeWoAPXmM55oJF/z/8/cVBpyTddJOH0dZgiaS+r7lOtHgm8jFs9CejQQ2Lj9TA tsYx7ob0n+3XW+UA9B7E0vVxyI6QgpU= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=S2sXkiCN; spf=pass (imf28.hostedemail.com: domain of 3p7ztYwYKCPwwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3p7ztYwYKCPwwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524712; a=rsa-sha256; cv=none; b=T3Dekx+OIXGeqgqcAIaAB7DteT6B48ne0ATQO7utPh0H5iUcPzIYSPHgZiY1u5T1/I5EFs ggqmNiHMeAykYY+rGU3tlZCL8ytgdGQY2jKoAi5KzRUPjSOh81EOVgH9Kr/20rjQ04eevt yW0c7SBAazwhUWLhUsLlj/TDe/AbRG0= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-52ec987d600so8351387b3.21 for ; Wed, 15 Feb 2023 21:18:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4tPY63bpwhpT+tcOBCZmVWQ4EsQSsAiOumfwemoUSR4=; b=S2sXkiCN7LWOjXaeaMYvz0qNJtP37xcl3aD2Yh+x1ZEOEg3KLdTaCbat1bdYvpjsne ssUpFmsgMZn2ZzaQIQ1nCUMhum1BKo1kVXXpN4w0/hSPSa2H/9hW6SiJ4urWGDrWEDIg /lDLOzKc/0bGJk2Y7O7P1pt9zjSY2dn9UPLftjGBDJCogvybLOdBJ/pVV0l/nLyc8nbd 78hCVT3xeGJZ3GfClpuKXysNAZLItz5CqeYcRfuccmKUwUq5LhC6V5ar511CeqKShlzR g15CQtYg8YR8Te7+dSB+30LV6maBKvWp8iWZepEStLZa94uN9djYv3KGIMR2EUqABQ+Q /xVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4tPY63bpwhpT+tcOBCZmVWQ4EsQSsAiOumfwemoUSR4=; b=gKlPasPEZR/GJC2W/bF04iPHOcAMepEh9wHOH/P1aqlMAmU67dZ0EZ2DBr5yYHBafr j1LwuTC7/cGppZdzJ76LNwDr0OMam1bwMAXoq4pZbFT+2X76fy9KsE0WcpwcYhGiu+N3 3xRk2xAYqmSXTLIjPRVdLlNjWyj2/c6xSPIPRtdeIdSWx+xJ7f1DNT6tIPhzwop/HfTm UQnmIh1Pwiztz8OLooL7KCEFpvDgb2FgzOlOJJvPYIdLvAHMOUBSNp7u6yAJXf/zVqTA gSPT6OrNS34KUBFO34QQ5dBHhRKwXfNA4Zldwwq8XSS/Wd0ofSgGcQN06ZoXdnB9jNvH ZONw== X-Gm-Message-State: AO0yUKXbKQOIUssCzn5w7maGYQHlkcXjnKNYrP9iEA2rftptGuU5YpFR U587t7ApugJToUv9u2KDglDO4s51zuU= X-Google-Smtp-Source: AK7set+CCkBNuKyDmku/7wwfF8A9hgf/5Gdg5m5aj/wzSHdI7lsc4GkupJYUuH2rfCym/bLqx0hNud/JqYI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:10f:b0:855:fa17:4f5f with SMTP id o15-20020a056902010f00b00855fa174f5fmr82ybh.1.1676524711465; Wed, 15 Feb 2023 21:18:31 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:30 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-16-surenb@google.com> Subject: [PATCH v3 15/35] mm/khugepaged: write-lock VMA while collapsing a huge page From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 96E56C000E X-Stat-Signature: 48x8ypazzfebxt91rdsixm4yh5bo44tm X-HE-Tag: 1676524712-964213 X-HE-Meta: U2FsdGVkX19/qtn+op3TRefvuXEWoHWOcT1K/h8poTWnlNfQFkd5J6CJHhAlwuVqSHT+YBOVtq+zTP+n4PomsD6yairolgi524Ru3JVbeK3JDmB8Hpi0jF3GLd04LuOKF8Cv+kWPQkpH6xFh7aWaeVNuUrm1N4WACCzDXewUW8aY2QJkKA6YYiYUe07oE6g62eiy2VJ96/FM7J3V9vS4ZiCbQOQhfAB3sFpTz9STmcgJREEyfoqXL9aSc+bYa4v1m3Zh9JXH1sfqdsRd6m9KNOVakDPRzWyi7O1RKGnHHxBtiPrDBPcncbO89zro7xC8f7j9gFefJIRiH/ftuxmvPIjfl5IUTsdqtSZgUg7IBedj1ZOJOSeMXeYP2l9MEGeMDf8P5C1pmwka2Z+i2b5DauLAQY+shAUkt2NoZmKcz3vbQ/PPynrF+9NUTXAJKJoj0okdrUxxLGW8qlEr1rTWs4I1Ytyw9rIrK4kwNsMVEStLDb9dLpUrwBvEwFO2T8DXZjLOETbY8FZmpAT3pUcf2ozcthPa25D/eGJlbcLy/2cT+fiBN+VPaCWhCNcGcemQSUyGds3/OC7TwPSRl0/Rds430qUMznENbbAD3mK8u+Tv7mJ4SD1e5bMAPT9FI0+bEH/UNO05WglqiywDNV54lafwHvWZB5i4Xjq9ScWTWqVqd/D+UuFObObok1q9SPeM9oQLr/FSvKwrSxOqlUhr0tt56U5BObAxGNx2sLGwuMWIEadpRS6t2h6VWTKl8n6q8JWghSMf8XnSZznO3CbhBJHhz66NSQbau3Qgrg5MtZ6occiXFP3TmhDC+bZUbzt2KcMYuuMIRUfqXvlI7gdVVq9J8iNqQfi3zXiOqgmIWUs2W+DfsOgM/jgtdF+B1JAnag7xjp/zC0crf20wFciNcm18KMhM+S8kj014FrfYQBLB72gYC5Fv+bPbeaOQehPAh6wEYRbN85TkFWyit1U N9hs5gxU jI7pUycEbGMnJVDDerwVCwC2A9tfLJm9m0Actlsjn29edy2o7G6cp3tJdQYPG2pci4UI2LLGAOJurlEpcU7AxGnswEehFNumK8fKgLAnI1wWNIyqFYcrjm4dQfxvchFEUHny8GDPgAV1pCdlJpgTCJkgWNiOIndkBRywKgJRMFPtFUnL/msfJTh3NKwtTWY6Fh4yQ4e2AXOxWLEEsgnUDrcfxIlhgA+dUgeDaPrRH2WMrcYNaflkM6JmA0S+eJFZejWUmDF57MmiP6ORjvbEdXwPBNz13dSSwrvj2ZtlVhiJrhajF1nU0XMieds8iVPnZIgWHkpJh9Mf2URq2t3dYfOrHBQt7N7lllemkpN3EWrBWA+GUD8aIdvqA0StQhTn9TPDtKmgFx74UIMXxho/GElBW1OCEa6aIb0pA9XcWKH5sDlmnzQUb9L4+xFlgbG0OF94h83C3/r64NybkHxlbwdaCZZ+VHSFXgLLK2Z1NCr5S7cbZIQno+DJi4w4QHWJSQBNlUJkJX6+98YtD3B2x7BBG7l6460r4HU+B/lCVTjWSx2mPqWLFvDj5dMJTheLIiRtDSnSqz31z395ORyu1Wf7WRBxiAc7tQ+JMFjARx7DR6WqYI1XvzNCOTA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Protect VMA from concurrent page fault handler while collapsing a huge page. Page fault handler needs a stable PMD to use PTL and relies on per-VMA lock to prevent concurrent PMD changes. pmdp_collapse_flush(), set_huge_pmd() and collapse_and_free_pmd() can modify a PMD, which will not be detected by a page fault handler without proper locking. Before this patch, page tables can be walked under any one of the mmap_lock, the mapping lock, and the anon_vma lock; so when khugepaged unlinks and frees page tables, it must ensure that all of those either are locked or don't exist. This patch adds a fourth lock under which page tables can be traversed, and so khugepaged must also lock out that one. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 5 +++++ mm/rmap.c | 31 ++++++++++++++++--------------- 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 8dbc39896811..d9376acf1fbe 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1140,6 +1140,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, if (result != SCAN_SUCCEED) goto out_up_write; + vma_start_write(vma); anon_vma_lock_write(vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, address, @@ -1607,6 +1608,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } + /* Lock the vma before taking i_mmap and page table locks */ + vma_start_write(vma); + /* * We need to lock the mapping so that from here on, only GUP-fast and * hardware page walks can access the parts of the page tables that @@ -1812,6 +1816,7 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, result = SCAN_PTE_UFFD_WP; goto unlock_next; } + vma_start_write(vma); collapse_and_free_pmd(mm, vma, addr, pmd); if (!cc->is_khugepaged && is_target) result = set_huge_pmd(vma, addr, pmd, hpage); diff --git a/mm/rmap.c b/mm/rmap.c index 15ae24585fc4..8e1a2ad9ca53 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -25,21 +25,22 @@ * mapping->invalidate_lock (in filemap_fault) * page->flags PG_locked (lock_page) * hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share, see hugetlbfs below) - * mapping->i_mmap_rwsem - * anon_vma->rwsem - * mm->page_table_lock or pte_lock - * swap_lock (in swap_duplicate, swap_info_get) - * mmlist_lock (in mmput, drain_mmlist and others) - * mapping->private_lock (in block_dirty_folio) - * folio_lock_memcg move_lock (in block_dirty_folio) - * i_pages lock (widely used) - * lruvec->lru_lock (in folio_lruvec_lock_irq) - * inode->i_lock (in set_page_dirty's __mark_inode_dirty) - * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) - * sb_lock (within inode_lock in fs/fs-writeback.c) - * i_pages lock (widely used, in set_page_dirty, - * in arch-dependent flush_dcache_mmap_lock, - * within bdi.wb->list_lock in __sync_single_inode) + * vma_start_write + * mapping->i_mmap_rwsem + * anon_vma->rwsem + * mm->page_table_lock or pte_lock + * swap_lock (in swap_duplicate, swap_info_get) + * mmlist_lock (in mmput, drain_mmlist and others) + * mapping->private_lock (in block_dirty_folio) + * folio_lock_memcg move_lock (in block_dirty_folio) + * i_pages lock (widely used) + * lruvec->lru_lock (in folio_lruvec_lock_irq) + * inode->i_lock (in set_page_dirty's __mark_inode_dirty) + * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) + * sb_lock (within inode_lock in fs/fs-writeback.c) + * i_pages lock (widely used, in set_page_dirty, + * in arch-dependent flush_dcache_mmap_lock, + * within bdi.wb->list_lock in __sync_single_inode) * * anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon) * ->tasklist_lock From patchwork Thu Feb 16 05:17:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142509 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83554C636CC for ; Thu, 16 Feb 2023 05:18:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BA826B0095; Thu, 16 Feb 2023 00:18:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 143856B0096; Thu, 16 Feb 2023 00:18:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED8F86B0098; Thu, 16 Feb 2023 00:18:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D35D56B0095 for ; Thu, 16 Feb 2023 00:18:36 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A98F0141007 for ; Thu, 16 Feb 2023 05:18:36 +0000 (UTC) X-FDA: 80471999832.14.E0D1CDA Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf11.hostedemail.com (Postfix) with ESMTP id D59B340004 for ; Thu, 16 Feb 2023 05:18:34 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZPrkg6Ab; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3qbztYwYKCP4y0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3qbztYwYKCP4y0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524714; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=28uvo2IqceKOssyZjY7i8BHUgiXFo9Fj5hjouTFWFAM=; b=iUph8oP7tlBUFuNQdYJUcuMleymid33xvtCJV+ZCR/L2wYqA6QX7UVV45LKre097MGkHge LPhTodg7hhZ+Fss2eoSDM8+nXmwuBzNEV/jTtU1RXZxJs0Tbhin/1hpvEwNmZ/8n4UHshx HhJitNrq9LMF82Emxrp7sJZBBzGLv+c= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZPrkg6Ab; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3qbztYwYKCP4y0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3qbztYwYKCP4y0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524714; a=rsa-sha256; cv=none; b=eFZ6+ZFQ0VtqwMlSqC/dmUcF9X7yqwGKw5/FetOW7NUE4fLAcn+d/posh+8PbnGg2VqU5B Enh7FS/PeIIOYmHXTyRMqACIX3sZ7fmRNNB1ZtVnY1ADn+CEa1jLTXYx4mVeCFWOVDtlGW eanuO10pG2452bc/EIajO5cbv3On0Ts= Received: by mail-yb1-f201.google.com with SMTP id k15-20020a5b0a0f000000b007eba3f8e3baso875955ybq.4 for ; Wed, 15 Feb 2023 21:18:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=28uvo2IqceKOssyZjY7i8BHUgiXFo9Fj5hjouTFWFAM=; b=ZPrkg6AbbvOKBZi1SkD/IxKxJeFUcG0rgRJFt48c6bsRhM/elzC2zYFPNxVAX1Pb/y v8gEIUO6XL4wPMpxHAJU/jtJ7FTBRhHG6n1tq7RCpsfo3BvcbN99La419Vjdnmr4LZUg Gb+YjubKYslJ6dowNgphM8TvAV/LOX1cmszsmnXszpI4IFG2RXQTfj4jwXf+4wC/qln3 8/CHWsl7LtqeSj/ymA2+araRUo+8VrDVeXfGOtKcUcgJXUwZ/d5G40s9s5ABK3v0GdSA ZKM8IbDd/wXIDY67hjDQxG31fd3mnWXrl8BZPhPKFflZXpp+nOaLG/s24K3lMQ1hF8se I12A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=28uvo2IqceKOssyZjY7i8BHUgiXFo9Fj5hjouTFWFAM=; b=SccZNT2moctnsLvma66oEtJjtAJB+v+HXIcUsgjfV6djK8uT2llJ5xPelXPAvOKyZx i9kNx83cfu/dW8tEC1uQuVQajX2/szThEx1Ayje4DinYPSG1l5DkSD+PsymBhinuHkqL KVjH5olhR7hq/oJUTLhquOjUJRW1K2NYB62h5zdJldeomIzs4ZSwV+Cb4PVywWoiuRgm aorZhLMnp9gXG17lZBjBA0x4PEgnRcl+Uv6HNvq4JpLJjTmg+HNy9i1S+LmNwGwVDQof t5rCGECHkXndKayjC+aK+2/3LsQcVqEx/42dSDi4R0mk/4MiGFp6rIc1M1cBORIiUWth 0oUg== X-Gm-Message-State: AO0yUKVYujKTxg60vvVjI3/LEXNwAmhVsRlh3RV4OHtX85pzFfbrqErR 6WKBZ6R6nbSQGpVdTb7JlaqoVd/r0lM= X-Google-Smtp-Source: AK7set9S4UPuY0Di9CvIunrawIij5qvY5UYsfRSe+cxMHLWbfH6TBfZEYx+EaVk4FGZlSlGHqCNzztV/E7o= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a5b:90d:0:b0:94a:ebba:cba1 with SMTP id a13-20020a5b090d000000b0094aebbacba1mr584331ybq.179.1676524713985; Wed, 15 Feb 2023 21:18:33 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:31 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-17-surenb@google.com> Subject: [PATCH v3 16/35] mm/mmap: write-lock VMAs before merging, splitting or expanding them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D59B340004 X-Stat-Signature: gwrrr63p1fduakadh4rwqmaja1s8eabp X-HE-Tag: 1676524714-707954 X-HE-Meta: U2FsdGVkX19NCwf355RCQAjZM6eBlkDM7Ev7ouFTuaBWrC2dB715BgBjLAMY8EGbdoVlYJ+4fbP7cXD7b4TF/GxZzDC0e9AjTS7kyHESOHUwZKg6QPwoDAQ9RHUAbflFlOYyyfsT+DcBZCyXYDSEHkWf90QqD6KBB5FDL0QGN9DnFYQjivyH4DIU2SAh2m0FMR47JZBJhvIW8ljuhV+ecWpLRZEPByO8wDGyb1H7RLzSjk85fjmZ6ZWZ6wkXECqTVvDM3kmzkMvQLm7Iew7SikGFBqE6T+YdjRZry8XBPWrrOEL8m9qbQOdJU4b/6XPWRHKfGm23Y8gXHZwH73K5Bhhn/9/7CNNxV2n5kk5Q5rRpPBxK4IT71avziLVSZWXnSHaDSD4bgV0ONdcXCOwn4/bR8g8DCgFeEEwJ8bVVm1yG4mHl+82UxLZRFZnIkKuwMWJr0qAx/JRFUfBJkuWZo9/ZCL1F+0YDBPje4wgqZgWMnOhkloExtofly5mQXH12zxxYsT0kgtiru03ha29LhUdQr/IggZzSqDSqFNgrGAql6S7cbECWkg8qCfhtyQpg1BW8MbsKl9p4CmLYXK5lr8HOfT14si+tEJ4LwoOgsOx8v6O7zfyjMHZz00A5jlJDZN5Ys+Oq2QiYjtOHTXaQlthva5mx4i8yU3DSRnRPxsR4Mjo+UgbgtFtD1xiCgrO+7k+oWJwU6BD8a7jDReG4tkIlz1Dm9SPeXyI15Y5kqLypWzSergbdNWyRdtKP7P1UwfRX+BXRXG41PUeGArlH+YUDV+zg8yTJc1nFzifd6Mync6FUdINiHZrX2OAeAc50EtJO8nk5K5XpapUz0mUJ+bTK45ZJlJnSMYRZ78k39lRhSnApznHiZxMHlWiG5b+f3vEsfUYMJ3yeskTATE2zYIuSIgjm+hgJGOyLbaE+vW8YwtibL4AYOTCMiQPU1NmBRTvxJH15ggU5Jz8w5O0 Rn8dA0lE VjE5QZkTiEIJam8ooHUFDceOncIBJHFJPxfcHVxsWpX3O5Zm2HJfbnVVdhQP+0d9Udp3WliEpBi5PsDOsgP1TCAQJX7wRuEUGUlpHguWMHkNv+JuVogDkRUG3l12YKQh7B/b8/J1IfDJebCDVq73t0OvMVqV6qyjuxXnH3QljYZo5sJcdpH0ZYckvGCqpGxOBaciuh2ejKbCT6Ns3TEkTHwzVwR9OslwA+ira8wNYk/sMqlGF6ymNzSOwQytcbtNTrx5K4tfc5WMfqGj75Re5SvEVNCBWUdTCp0FaZz7h+lpZDoQtSJgsH7N8wGcz10h5/2UCCKMUs1ulfyUf4cKEPPc8qgCOpS7SjWX4wo8bsfhmhZck0yra217m/P5sCJlH8EsvWEUX9r2MAlIG6tAN7Q+QoyZMgBLtOcXMUHu9srX3Gjy1ZnkfoPAcJ66wV3r41NNrbTAtNsCzwPWe6wmS6xKo5Nd7/79ILCjsOk7yVQhDmw7Fe7yPFMM9onxPqUpdcBjS7c5+y+jjJmcw5669PIPy/Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Decisions about whether VMAs can be merged, split or expanded must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Write-lock all VMAs which might be affected by a merge or split operation before making decision how such operations should be performed. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index c5f2ddf17b87..ec2f8d0af280 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -269,8 +269,11 @@ SYSCALL_DEFINE1(brk, unsigned long, brk) */ vma_iter_init(&vmi, mm, oldbrk); next = vma_find(&vmi, newbrk + PAGE_SIZE + stack_guard_gap); - if (next && newbrk + PAGE_SIZE > vm_start_gap(next)) - goto out; + if (next) { + vma_start_write(next); + if (newbrk + PAGE_SIZE > vm_start_gap(next)) + goto out; + } brkvma = vma_prev_limit(&vmi, mm->start_brk); /* Ok, looks good - let it rip. */ @@ -912,10 +915,17 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + if (prev) + vma_start_write(prev); next = find_vma(mm, prev ? prev->vm_end : 0); + if (next) + vma_start_write(next); mid = next; - if (next && next->vm_end == end) /* cases 6, 7, 8 */ + if (next && next->vm_end == end) { /* cases 6, 7, 8 */ next = find_vma(mm, next->vm_end); + if (next) + vma_start_write(next); + } /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); @@ -2163,6 +2173,7 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, WARN_ON(vma->vm_start >= addr); WARN_ON(vma->vm_end <= addr); + vma_start_write(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split(vma, addr); if (err) @@ -2518,6 +2529,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, /* Attempt to expand an old mapping */ /* Check next */ + if (next) + vma_start_write(next); if (next && next->vm_start == end && !vma_policy(next) && can_vma_merge_before(next, vm_flags, NULL, file, pgoff+pglen, NULL_VM_UFFD_CTX, NULL)) { @@ -2527,6 +2540,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, } /* Check prev */ + if (prev) + vma_start_write(prev); if (prev && prev->vm_end == addr && !vma_policy(prev) && (vma ? can_vma_merge_after(prev, vm_flags, vma->anon_vma, file, pgoff, vma->vm_userfaultfd_ctx, NULL) : @@ -2900,6 +2915,8 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (security_vm_enough_memory_mm(mm, len >> PAGE_SHIFT)) return -ENOMEM; + if (vma) + vma_start_write(vma); /* * Expand the existing vma if possible; Note that singular lists do not * occur after forking, so the expand will only happen on new VMAs. From patchwork Thu Feb 16 05:17:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142510 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56480C636D7 for ; Thu, 16 Feb 2023 05:18:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3FBF6B0096; Thu, 16 Feb 2023 00:18:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D7A1C6B0098; Thu, 16 Feb 2023 00:18:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA4596B0099; Thu, 16 Feb 2023 00:18:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 925DC6B0096 for ; Thu, 16 Feb 2023 00:18:38 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 670E5C04CC for ; Thu, 16 Feb 2023 05:18:38 +0000 (UTC) X-FDA: 80471999916.30.C46871B Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf02.hostedemail.com (Postfix) with ESMTP id A36C780006 for ; Thu, 16 Feb 2023 05:18:36 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=NZSRPVmV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3q7ztYwYKCAIuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3q7ztYwYKCAIuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524716; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Gs5XDlSmqlz144wCEPg25vfe9kudkVyQE95J1Pewv8Q=; b=TfLWRYxEhSX+T1MwYw8Hid4Vod6DzywzdrNMB+Mmhk5bo3B6UK3pxAXcBu7dEd6BXC5T20 VoYBlr27uHBMcsXKF0Wv0kDxi2GElPHjObLIpgDsxtUrZ96DeGs+huHefV/IdNbxZdd/9G FnqmRhVe2Mh1YY4Ihm3YOhrGc7JgSfE= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=NZSRPVmV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3q7ztYwYKCAIuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3q7ztYwYKCAIuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524716; a=rsa-sha256; cv=none; b=F22ZSOMzqQyt//ZLAPCRjoG2PtJbcfSC8ImyFbrKIqwG8U8CdMZQtUYFRrLbCA4c5509Ux XSCLYlfuCcYRVSlTtHo1Uc8eQKPNVYXP1x/Ba8d/YU1IwChEId59wAW7M7Y+NkXkyg5sp6 JjZdQsVPAeHXLJR0BSAxd/sWdp50n3M= Received: by mail-yb1-f201.google.com with SMTP id f187-20020a251fc4000000b0087f69905709so867876ybf.10 for ; Wed, 15 Feb 2023 21:18:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Gs5XDlSmqlz144wCEPg25vfe9kudkVyQE95J1Pewv8Q=; b=NZSRPVmVQg07iGqxuaNSoyEyX/JUlE5074LqD7StZA8wPOZx80SXxNMt5E5KxxrtoC 5GmbXen1kftSSNBaHdmndb327iEU+JgvGlsjv+jrut0kw6bBF6uNimcKzz4dFOSEItYW Y4LEl4ERFvkTfh+pRssZBSprx0WHPyqr3iZkWvDeufJqDlNo5Dl9o53Xv9skjZ4WRCg9 pPw2zfwLYTP83BBCg7T3fM4u8tR26MmRDA1XMM3A/clmWDV/BI6rgpWB66aseND2aMHJ sak6MgPcdV5XEwJQs9ETMkutqiCQEhW6TAM4KAr4SAjK7KgYItsJ/rcb5RNyZGDs+f6q Q5lA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Gs5XDlSmqlz144wCEPg25vfe9kudkVyQE95J1Pewv8Q=; b=h0ZVX+EO4eJlF4WcdYNvpNWQtui6Z4yF655MVhT7EPSHHucobAbokZ2Z8KHuFV2knk tq59awNq7LE3XDGK3DvfDNhc/MzfBvhM1C3Exd3DKRMLN/JTvSnsCpADzDW3UsGJu2ZJ jhWvLtz7HnSre1NUDSz2Y4GaUQ3lQx4kjsEs0KcgmVZg1GAE+6L0I6W3pdZ3kPB7zUd3 nO96Ypc4A1SLSDZonXCU+/Jw6QQtdtoFWCmeATZMvOMuIDVIoUUpalgKuI+qyK/erPsq CXh5bL0QMC8csVTnoJbCplsyKUisJ4TsCSB614axzXZpK0ywM7NwaHjMQlWEoM34eAwJ VA7w== X-Gm-Message-State: AO0yUKVUeNlq38MZQKWCqqJqiOVWjYpUAHIw8FiR/9nGr0EPJRopjQ+p CIjElZkUxJU5/5PKHfrV6pXNcwG37GQ= X-Google-Smtp-Source: AK7set81sG3yc9X92pQzrCT4qIFN1iEujQCfNrT8GtjdXbA2djctnG24/XowjsXx2FP9/9bsOvEUQhAtEC0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:9345:0:b0:52e:e6ed:30bb with SMTP id k66-20020a819345000000b0052ee6ed30bbmr607996ywg.571.1676524715849; Wed, 15 Feb 2023 21:18:35 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:32 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-18-surenb@google.com> Subject: [PATCH v3 17/35] mm/mmap: write-lock VMA before shrinking or expanding it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A36C780006 X-Stat-Signature: y7x1scagruxp7ck8n4g48br8i957xx81 X-HE-Tag: 1676524716-734210 X-HE-Meta: U2FsdGVkX1/AzJmcHnPbpvb6XbrQWXHSFkwPPOntbETGUMJib79m0sMxwKFfwH/kWBT1lT4epcUcx435HvMXND7BwKTo/0bZzhs+0x2QRiqbhEOloPnsFnR9fuDmXxQZa39GOsjvfHKggkAffeuS6DVjL0/XBxVsCw6/oMMCdMlHwXJcRtH/UJdMIEWOF/nQBSMeDuuJJIVSNID6NvWM8WHhqXt1+b2KpKDm9YPXDfYGw7P+EPWPuMdejEF0/LpLEmiMTL/U9Mw7+7N/7XOchdFluSiSmkvyHl8GqZcAAVmaBnIf0T2sB9OW07BVUQArnRmR3yhu/YalJUvU+9Exywx3zRHbFAmSSeQEBUXoA7d8PU8tKAIzz4ULkEDZiuUDicc115sbeTX00S0r3h+ICxrFBsnT1+uxa6f4f62bSkzJK10e9/ko52PJek9d1nwANxgQakHvGSBD/8V/MCiczE6GsgxEU4VrFBG4d5aArAsVlSZqOhv9N2n/BrueXQx8cWgM9xwOpIyMsT7P7AcZqliWUKXrg60vvOAL3nFYH357cq+FTtu245QLc1pfUjxfCNHfk7RZ11Bi1EkFoS8Il3p97Ho9YBDgDhAb5oHZdovuZeaxqNyk+jal7W0bmk2MPFJi5g8xtVbZT2m5WNDAng1KBBSde23mgz1/33iE71LKRvfk/AxmLNbWxECPuMa69YW/R+SDdC8CwrbEJ/B3X1emW6a75cXwxdp4lQSGZq1B98Eabe8Tjmc8X2U8jL8pW5whFwAfMcdyvEcaxmgKULQdU9zvCLI4TgtN42takdfBw3jo2f3VeDsQK3jbA9Zi8H5qpHl/EIXZCk8UEiFzZBIB45TP2J8sI/dmTOpLl2JRBJ1JctQTf0D4Nf44Fw9xZ+JG853lVw6urR+9V9a9yEXuA07YgO7rPKMihRnZoGNB1thZTFmMIc0D21IUT+ImPfydxDh8pT1YLefR1d3 6u8bFhbD PDLA8cbVHDAgnEKMF16srTYhSmsV+g1ufmeniv2Fpd9xojnQ714Wv8G+90wN3NEZtWofEvCp/5E2b3QCxjSrQF2ceL5OTr/0Jprzu70jIGFbLlkoMKMVNM7e/Uz1dWi7LamTA7RYk4OQufq1OncxpYGXzMsIzllc+jjcgiKX2lMdDP3iMeBBIwBCh5a4ojR05byWIRJOCZIIWbsT1H3eyKU2MRSGZ0w/uUDTwlZfIFe2JYF7KZ0YuYyUxoqrIBZOnQQaUN2KAkSSePHWIrduY1J2lm7jeXgYn4bDfKQFcoiGBSD/4Ewmppyxzt9EFzC81yUPiWs8ymuKg3mnNuyy8dz6ndp4zN7bdIfB+Xz6oCFTOwchBNbn7gm0q9Q/IBKkiZyWIXk47+ALFT+5fFwW6yvgUBG0ZbJ81tTwvsU3pSt6lpoprLnxCHVyLXWrZEW9sIOzt+Gnk4rpuQUzMMsJ4SW3Y2uO0OVwqSx2Z6jdPypNtsr9dMyl4Qog0gVh4iXEzwKkR0yENCe104p5oY4F9aMWhoRZAt6i4z8hK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma_expand and vma_shrink change VMA boundaries. Expansion might also result in freeing of an adjacent VMA. Write-lock affected VMAs to prevent concurrent page faults. Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett --- mm/mmap.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index ec2f8d0af280..f079e5bbcd57 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -674,6 +674,9 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, ret = dup_anon_vma(vma, next); if (ret) return ret; + + /* Lock the VMA before removing it */ + vma_start_write(next); } init_multi_vma_prep(&vp, vma, NULL, remove_next ? next : NULL, NULL); @@ -686,6 +689,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto nomem; + vma_start_write(vma); vma_adjust_trans_huge(vma, start, end, 0); /* VMA iterator points to previous, so set to start if necessary */ if (vma_iter_addr(vmi) != start) @@ -725,6 +729,7 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) return -ENOMEM; + vma_start_write(vma); init_vma_prep(&vp, vma); vma_adjust_trans_huge(vma, start, end, 0); vma_prepare(&vp); From patchwork Thu Feb 16 05:17:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEF69C61DA4 for ; Thu, 16 Feb 2023 05:18:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8BE586B0098; Thu, 16 Feb 2023 00:18:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 847E46B0099; Thu, 16 Feb 2023 00:18:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E8576B009A; Thu, 16 Feb 2023 00:18:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5606C6B0098 for ; Thu, 16 Feb 2023 00:18:41 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 28E6514071F for ; Thu, 16 Feb 2023 05:18:41 +0000 (UTC) X-FDA: 80472000042.17.6763419 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf29.hostedemail.com (Postfix) with ESMTP id 68973120003 for ; Thu, 16 Feb 2023 05:18:39 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ClKHnTmR; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3rbztYwYKCAQwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3rbztYwYKCAQwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524719; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jGkERMphMFi5DTXO6AzCtASC+DQ2dhGXIGrkfltmA+k=; b=7gadOCXjRoG+rVmPvnL+cpcgJfZlgOxBzgs7/obHLyG6VS07UGRz2th1ptQAtuwqBe46a7 Zq8MqwfC9A+7yiBhKzTi8aZpC8EPkEDGTMjsZS650gcrUlwiK/m1apAfq1a/GFwO+Qo1IB de74gxuh7/UcnCAJCs0p9mXaptKEI+U= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ClKHnTmR; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3rbztYwYKCAQwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3rbztYwYKCAQwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524719; a=rsa-sha256; cv=none; b=7ePiLx3YohXDWwnn0Oxk01tKcKYCK0wxn0mG4AyOYIId8y2ZYb4k3tyLyCptKxgLHvfOlQ 72efRNcxo+2l04VFsAlsJgzx1KMgeKpdw3n/2Xv09hLFDBlUKpwFUL0OyVTla7iIAJfYLB OV45YbxmNNifHcPYCRvvqoG7/J1KWks= Received: by mail-yb1-f201.google.com with SMTP id a4-20020a5b0004000000b006fdc6aaec4fso843530ybp.20 for ; Wed, 15 Feb 2023 21:18:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jGkERMphMFi5DTXO6AzCtASC+DQ2dhGXIGrkfltmA+k=; b=ClKHnTmRqrwFsiwdvZoHbs8UrIxrOZcnAlLMKj0DmY4EeItAiZvM3xF/1XkOIILsHj t/ncp3cCJ608udelXLFUJi2J2Bkc4zBNFEFw7AtqnW62B+1ruBsojOMJ+l//Eveg6kMb 9mESDdWGEfVZo4wbqdchVNx7evryh4U8zH6lvFtZPteOd8Fw20lUCwc+q1jxjD4dRCfO fPscX4k3EE8nShVcC7jG6Imy8fv3tD4a5LzQcDPbvgxMAF9FYu61sb1sXCbMYbbYMnzw H0gcnPTwNkiDCSFpe5MW+pIUr82O7ftWGw19nAsT/pwTKfJQo1rJfp9QBdZ/qdttHeJW nqYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jGkERMphMFi5DTXO6AzCtASC+DQ2dhGXIGrkfltmA+k=; b=fYUi6X43BMFFsVGcjg6RcFdyhVt4RmhG0zTHmrUO5k3/cOzpdforoZ/2PC3e2NWudL 4bkg7ZlBgXuiZMJEPNT9p8qON11D+g0jskDYWPGWVRizRtyhnwgn/4S6KjMisyg9ek+Q ibX5ib3ZErrhB1oC7Kd3HjI6Ary1JPQlLiDGqhj6FJAzaM+2tW5mEptC9AQipJ9nlruf RmVzlrWfeyC/Vpld3mxvWnIlOfJGrHQEzAGRrTEjl/v0L28wJdA/1tToo0MnMuPYQ3ci j951H7swodi4yrFtMoV8CcytlmbyTMR8a+ENIZTibvky1kO/G5eWBWNSOiqEw7Jk/4+v Bhiw== X-Gm-Message-State: AO0yUKVP2UiHRBA6BbDYkhNc6wxUSCCBvonRmAoExSigzom8qfEH78/3 JRoeyY3iefg7ZJCNIPypgrugGcpWbq8= X-Google-Smtp-Source: AK7set+tNlKzQOXitoUESk90msiy2FCUMsrjKiA2Q/PTOw27FiKf0wEmZgQspJdrm7P++s7G3RI15C4Nebc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:ac54:0:b0:52f:45a:5b00 with SMTP id z20-20020a81ac54000000b0052f045a5b00mr11ywj.2.1676524717784; Wed, 15 Feb 2023 21:18:37 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:33 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-19-surenb@google.com> Subject: [PATCH v3 18/35] mm/mremap: write-lock VMA while remapping it to a new address range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , Laurent Dufour X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 68973120003 X-Stat-Signature: 7ku4xnp8aw6t4mhytg5itjotzzmwtt51 X-HE-Tag: 1676524719-941465 X-HE-Meta: U2FsdGVkX1/hF2OxMMRPHS9wBVrJdTvRdLFAPUaWWBrvWxqSdGMlYP9VARYJY0KW90i1WobzcKvMBlGWUX00ss9Nr7T1WDs5Vmc3SQqkB2+cnncFyykFOKOPEsRLvYvtvUwXTQtR65H68zrCyomyf2QaamwHD+WpcM1/l1oZmi/ugqfbH7hDRrwaSODORiuhohxCllEWr3j9t9PyHfHxI4dZrRdznVwjjlKX8PSm6PLb+MW8XN4nqNqu31Vv5I4SeJaveymSu2DgvXo8zqQvkf09oVY3pHE8Wd/psvcCxMYAoGoqwr2OAq8lb3aRGjxNsKs2EBszRWWYIcm0DKqwrsd9Z2IM1e3MVUuyl7PAGQO2sxuJ7PxF7ejgiG4p+7Q5NkfDvBqJzIckNBS1+BQaVjb+Lt6JcJLGXRdBvy7dscogxtkHp8959Cno216/8x1sf+vcBBhFLnei+IZ4IC4Ln3zx1HO46kMTnBtV3d3LECE64Sp4HUpDlCZXLjGWPfbliZYDPYCt6TEd1EYMwhZsyZnzYYFx/8EHmuz/3P+h32Gbherhhe296CAeQC6yMEhy/Dm9Iduaw/XtMuskX7puhBzRf+uQdSftrQLF399lQCLurn2XSCc+VYpUCSH1gAByjgu/8XUz9y+7ZXLpcpEPhgDlSZsv1t+yT3X2Lr6XalybuYDmuQkHOfb3kGXoKTjDF0No2TDEJvlZ5uxi5crbygAogEFeRekSU1yqajsYjtvOlphApZjxlZQlStaWLnTze6kLiTXHIc9yR5AZDnvkIkk40nL9mYGQ4QMvoidqeTwcnhmjniTBswMq5e+2q0SuawpO9g7ArlXBZ8uroNfcJh/QDwH7Q+CqNlxIQzYxLHDsyvr3YOD/D7uHg3o+sgRH9GuGEUrTS8R9K4e8Oj/9UCzYtG4dhfG02zuQbJibNb5jEa4kzloHRxolgZTNwyKVE0s9ssK9VaBA9M0Ykba 0hL1grP8 zgLvWro2ZHXHnctOxTNkb4N/cth5Es3e+e9i64LCC5xEY2dsgrJIxnLJNa+Y3fpKoYCmBWFHTJAHQvQWgYB79hTYPdNozy6iOlqMWm/TUTOA2pOe1v81AbKIB9F5vAzv6wx+Zyv2cv0NeNPc3wJvQYhBHL8Z3Lf2igjUn37da2EAucvJFxoG6nKAaKrshnlEJouhRrIl7XCH2/MP/YKDVl22pXtOYIW7tl18N3Jzv/D2w2odhgXEQ1p5yTHZwNaEn7usCLN7ozFZLe0/TfTmF8mm1iOQpzT3YHgJPjSLXBJ+GqXVc4sGYM6wSxqXq92cbu4Ynd/IO4LQmqt/TaJaDLAM2CRcenQo5hNHibTgu5A8WxrU4nd3MzrQLa8iX/uVTdpuRd47MvlzouT+24TN/v35afPjb8OW6H9xyZR9Z5eFSMlrj9fMiVIRlqpO+kCvx/9pW6xxYN1EcgTf6gvTdiogZjebkWMtwwTtPjejlV4H13/cNbBW29Y3xXl14dyGpxm7YPkpLhrGNpFW4yhvk4Jffi15L6dzvCsZuTlj1YsYfK4wv2W4YQrTKFw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Write-lock VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index f079e5bbcd57..0eaa3d1a6cd1 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3202,6 +3202,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); + vma_start_write(new_vma); if (vma_link(mm, new_vma)) goto out_vma_link; *need_rmap_locks = false; diff --git a/mm/mremap.c b/mm/mremap.c index 411a85682b58..dd541e59edda 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -623,6 +623,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, return -ENOMEM; } + vma_start_write(vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, &need_rmap_locks); From patchwork Thu Feb 16 05:17:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142512 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48741C636D7 for ; Thu, 16 Feb 2023 05:18:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D79476B0099; Thu, 16 Feb 2023 00:18:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D02A16B009A; Thu, 16 Feb 2023 00:18:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7C586B009B; Thu, 16 Feb 2023 00:18:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9E0F26B0099 for ; Thu, 16 Feb 2023 00:18:43 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7818214109C for ; Thu, 16 Feb 2023 05:18:43 +0000 (UTC) X-FDA: 80472000126.04.E733F80 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf09.hostedemail.com (Postfix) with ESMTP id BBA62140002 for ; Thu, 16 Feb 2023 05:18:41 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=LRD+lTaX; spf=pass (imf09.hostedemail.com: domain of 3sLztYwYKCAcz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3sLztYwYKCAcz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524721; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rL5PSjChr8wFTlKRWTO498bp0kCwADLZZe/SXHes2h8=; b=Wwtf00u8Up9VMd75jMxb9pArgD8TaxeUv7wkTYcHyVqdwQO8jX4uZt8PhlLRYBNkzwkEbd zZ3byL/Qi4LhxEhMKXq2pl39bb+1WXzxUSQXy2NE/vJeJNx/ekC58LNBXYUNhhpr+AHJky VZCADMfngvL7tm48JEos4Wc9u43nU0k= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=LRD+lTaX; spf=pass (imf09.hostedemail.com: domain of 3sLztYwYKCAcz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3sLztYwYKCAcz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524721; a=rsa-sha256; cv=none; b=YrsYou8IpG31YDRV9Ig1554BhPNGGW6gqwxA5yYEs2x7pOtmwNHLmm5MS9LTE0o4HY1smo +zLyM6Tm8v5KjF+kzLoBk5PIgmMTS/P8ggNf3ZkNG06cx5+CEr7qK64/0SZZMy2Wm2U7iH zuVfVBV+APEfIbgUqKRY0rXE19s3akU= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-4bdeb1bbeafso8699597b3.4 for ; Wed, 15 Feb 2023 21:18:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rL5PSjChr8wFTlKRWTO498bp0kCwADLZZe/SXHes2h8=; b=LRD+lTaXr6Pqw8LYMkej56Kfm91pJviFVpFaUDR17WEIFgUKOpU+ogLRfy6oidvw/F 4mOXwIhPJkh7gxaJzacQ5Q2xpA2MxMudctw4pQKYtuqshuSCMBeCw647b99z6JqUfMYU FnWI7aY3RIFvc0EuIZqwF4n5sAACqaxfMc5BUHOhpo7PrLkybX4Nw85DLStMxOu85Den S+x5aTXsnUr3Va/T4IBtQXPzTR4iMs6msv64zgKWL87HLYoK/sTJqZLT5fditOnzScG6 JUQkoM18jflZAeAGFMAkFQi7v0aeD9Evl42tB/J4ec9dW/Jbo4ktb05auTom3OB08iqz 18AA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rL5PSjChr8wFTlKRWTO498bp0kCwADLZZe/SXHes2h8=; b=yRqZmJyMttUTzOWgXambDvqpZ4l0hoOlxXE3SDXdsvOpn4aD2HH2yHkL1wg0iAS5SZ pVq9Hw1j1MES3qI1zad0yJhBU3AC/j4xqDhaA3vE3cpgtK20spGAq3Hkol7C+MOW+Hvo rpkstc6OEMkDk5E01rSKEgLayVzHR47V5ys4W4xyOBOguBiDNrAniWLWrRRSJVlkMWzL 7euoi82QlXqR9Lix9rcSS38xBYT7cE6nAoAVM+2xn3+N8YNb+idM0Nmlwdc/k/8AgpdL BY9LgALXLBe76GN0pxz3suuziSLaZREyi0QycG6IIIDoIMFowLn4d4FQkwChIWdhiRQL 9ApA== X-Gm-Message-State: AO0yUKVafKolIHsNhITeYM6Yw/Cmqlpi/76Wlj2OESQha3/tTvbtE9wE 7/QJOqfZMPCJxnCYnOzPTKAOWxXxqW0= X-Google-Smtp-Source: AK7set+44y5aeQf1zuFNW05XeMWyuIDsdI+r8pPqkHzR/in6WRVQ9dwNjFzyqkfy6m3wNLuusOCyHSVO/GU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a0d:e506:0:b0:52e:c531:20b3 with SMTP id o6-20020a0de506000000b0052ec53120b3mr448676ywe.147.1676524720915; Wed, 15 Feb 2023 21:18:40 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:34 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-20-surenb@google.com> Subject: [PATCH v3 19/35] mm: write-lock VMAs before removing them from VMA tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: BBA62140002 X-Stat-Signature: dh3anos38unxpn9gewzqpopg8z4hdice X-HE-Tag: 1676524721-788833 X-HE-Meta: U2FsdGVkX1/YUoGuDeKKm4fVTdvSDyVhKiJrUEg6prrtbJWSPb7hHD5cF6qeCbCU8WICfMw+kYl9eKcXgGeASFMNtUwnd6u66wdWWMt+q7UTXR+AJ+kNzmSZbnqlTVmgqHtBgkQmhfaHkDSpVuk35TltpKyUKJD1RMYYtv9jsCgORYcqUB3J8O9WeM+yQ64lSl9WBLQHJugy0kruSjqXcYHzfDnyRT36s2ozzNyYx7pWHA6Bk6TKdJKME7v2/B0ocfR5XI/muRETcgz1WXVAagqrlKhQCr2pLU423It454bkN0U+OtwvAXF+AOc1WI9SlAKXpn93QjiYPNRuIWcdal8ZkDiyzNahJZGzh3MYPPoereCli0/oAMapAQVmG4GVJDPj+IoC6NtsR4kGTVa5EQ0+ZsQ98Rrxi8N862ifRVegfFC2a623r1Mfe+5SIq6yHb+FgkhTDIw/HMfctl/JSRMrVWaYoFKz8EhLauQIbuqZjbsfKP2sKILYxX2Cf99zW33apo/NlTqM9um/Zx4IVdfGNUH8/WEmUtDOeiCZfHJSk/+C0Nu97E57vOiNirys7UgLwjO/MEBF5pnaKr81Tbw8zZHFMuljTFEJJcNAPrs0qrsbUX5Eoz1ewgI6Aa+tZimNN8gyp7Ggy8UWVk4zR6n5u7AvIkmTJDL5ah2k+XIR/wyDnQZ1qtalJBLgspKKqwWkKw2ieT8e9XgkU5+iVax4OszLRX75KjfosJYn1b8bjTUDZ49WDfkJ1SmxycIR76t6hdCuns1QVbqMyhxLO9lC54endpSKnqFO/OQXjaE/T8OecngAeGIjta1gDqxm+0lV6kJoMSJtHza0TEkeTfV0bj5o8oEwX4HxBhKeFvViMCBqyAgdTyi00faZKs3+2QLQJLEs33iHPDZj7uxn0w35WKIIgxx0KlLxFfw1xgZzPb2j7x0gt550Hrhfs3IDhDR103Q+iaUjK0tPX0y I17PMFUF FyFcj12BKRWPhl4mODuIO6Mg2X1Gc+ujCmrOFZHfw7xqO6PiZJ455zbUNrXFvE4e48qRS/ixKPUOKA182A41w6tpnMjMGmXmtsVBBsKDHmryQ16ie2WyDbR2vQLPHMHIwrLl5V1+GksoNDYba1AL1XF0Hxmc6PSeBkmQLnphGQzd9lP7m2pCossBORs94yCFUmGl2Mrzdp4LojVXTe551lCe/74pXzTGLaa3ziNOFzvhCYF0wOvjbA1FgnEB/vdeN6OiZ1/oTxPmZXn5vC9p4xUsfhwNiRoJyaQQJo55DSJKtYAZo4zdN/W+I1ufYauVQDGxyA0q2MBrbpz2xLdB1Q46LUHWD3L+ezGr5DxmdNqYq7TGeZyeSYYSytd2ixStejGI5U8F/pSyFmY+DGciG4oMHrZmuMruAhTAmW+NYT6TiEQKvoHbLiJgVOjbRIc7jXQh/aJ7V9vGYHWsgpz2z4Xp8fbz805qxozF3PhhNqkxIZeiSWwM0K96dnjrkyys4o+8LoOJfbG1DAfU4LlIvXyi4Ph1skdjllnt22levWpB9xNGBj6SgxqeEXAFJNUd4ASTI98p/ptUyO4S1lzU3Y3wYtUUOZaA03yim+O0cqMWYWje8Qq4b8BZ5rg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Write-locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 1 + mm/nommu.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 0eaa3d1a6cd1..aaa359a147b2 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2261,6 +2261,7 @@ int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, static inline int munmap_sidetree(struct vm_area_struct *vma, struct ma_state *mas_detach) { + vma_start_write(vma); mas_set_range(mas_detach, vma->vm_start, vma->vm_end - 1); if (mas_store_gfp(mas_detach, vma, GFP_KERNEL)) return -ENOMEM; diff --git a/mm/nommu.c b/mm/nommu.c index 57ba243c6a37..2ab162d773e2 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -588,6 +588,7 @@ static int delete_vma_from_mm(struct vm_area_struct *vma) current->pid); return -ENOMEM; } + vma_start_write(vma); cleanup_vma_from_mm(vma); /* remove from the MM's tree and list */ @@ -1519,6 +1520,10 @@ void exit_mmap(struct mm_struct *mm) */ mmap_write_lock(mm); for_each_vma(vmi, vma) { + /* + * No need to lock VMA because this is the only mm user and no + * page fault handled can race with it. + */ cleanup_vma_from_mm(vma); delete_vma(mm, vma); cond_resched(); From patchwork Thu Feb 16 05:17:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 081A0C636CC for ; Thu, 16 Feb 2023 05:18:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 981F66B009A; Thu, 16 Feb 2023 00:18:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 90BAE6B009B; Thu, 16 Feb 2023 00:18:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D2CC6B009C; Thu, 16 Feb 2023 00:18:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 692D26B009A for ; Thu, 16 Feb 2023 00:18:46 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 42252AAD24 for ; Thu, 16 Feb 2023 05:18:46 +0000 (UTC) X-FDA: 80472000252.20.9BD298C Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf08.hostedemail.com (Postfix) with ESMTP id 894B6160013 for ; Thu, 16 Feb 2023 05:18:44 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ngrSSnFG; spf=pass (imf08.hostedemail.com: domain of 3s7ztYwYKCAo241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3s7ztYwYKCAo241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524724; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=65rn53nYPCehqawxI5VHHUH5OXR7IFklQh11X90VOGk=; b=SRcerqiK8AV3phFT0pOmQ2yDprMEztQLMbu6gtoO6sY6DNk/5OZsp6c0xWhtciJvCwLHfB 2Y0TXDe4euyejXqD/YyMmBmryXcuFP8VDSdWmdI/NTDvKpI2agYMGLqoni4UhQw9kndhK2 dJ4E5BaZ1AQahL0zxXHpUhmrarB9AwY= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ngrSSnFG; spf=pass (imf08.hostedemail.com: domain of 3s7ztYwYKCAo241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3s7ztYwYKCAo241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524724; a=rsa-sha256; cv=none; b=Xq5atkh5C4z2qiOAU1aRrY6PtdviS/ce57KqxGOv5bb7X7vTrxW5AtEEQomAJmWSn76jfQ qfdRuQKdiL2HAZOf6W2rziceqUVzEvkxW5KJO05kmsvSftK4ES1DALUJ+BlOpzeI9lcWrM HS9fZhtRzDp0AIZi9eeDWRWphYeDOCk= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-527501b56ffso8392637b3.15 for ; Wed, 15 Feb 2023 21:18:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=65rn53nYPCehqawxI5VHHUH5OXR7IFklQh11X90VOGk=; b=ngrSSnFGdDPlYz5zuJYWd2ZcJdDDxwAT/RtnIL5xzVfAFVgHCKIj7A9xwEDcTyX2aB 3vsZSlkeyPVpcwi1aa9l93eu/icfxKL0DIaqQVOS2lNklMJwu2jpUpq532q1GD3mJcXZ OYyJWX25F7RdsRsm0bO+ULQDs6kU+hJx2yHcqT8VTisE8SVpLjiLy4deYJxyu/llv5yd uNDnBwf4OPlsNNTMONtuv4dnuvjdk8hbLw2jK3U2tc3OA2ZTvVJ3/lGJPyy4Jcy7H5Vt +ZqHtuuBA7mxmkHLm02ywaCux2tqtjmQCbr0/F6E0nSC73lfB2u+G7wX3tsYtQZiJlY+ ecqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=65rn53nYPCehqawxI5VHHUH5OXR7IFklQh11X90VOGk=; b=MV/fe/8utUhpbDHwR/leOlzwOs2WiR8KT18618jCv5OjiixuvOVyKgB8rJbMkJc9xI Dj4CmZewutPejLTB8Ca4VpDkfmwKLc1kgPjdM0xUeTBxGyaBLUM/wPVV2i1rr8Vd8gX1 8VpTAzBncwBZUPx/IgQ0flDdwUHrB8/tzL2UFUkvR+EPnxbAAPpb8e+VIFPxN7axq8wG 6yHi2Z3aHgB2mX6BYqsqT+F+WT8Et6mi0zqsVi5MWwZml9q7iQhS3KUtPpvZTdeJFnLA yr+cdCN3TZUkMN2ocuGEvyMQKFdP+edYfcCU0l4SiO2dnyCqnPXRHMriZKkVGSHqqjUB clSA== X-Gm-Message-State: AO0yUKW1tGbB5DmQVKR5jL2q07hL8vCoZoByxnZmJkM/GSAhlchLsk34 PfaG8K+8yLQ+0kzikeCqb8k1i+Vk/5o= X-Google-Smtp-Source: AK7set9eBfEal0C1jR4VjCmFFWO3/hvdavzvid6pMzzd4bvHowMVw43Kn34On8e4ZBDtcDdaQHGRoUI6tN4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:b704:0:b0:52b:fcf0:4fd1 with SMTP id v4-20020a81b704000000b0052bfcf04fd1mr5ywh.1.1676524723168; Wed, 15 Feb 2023 21:18:43 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:35 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-21-surenb@google.com> Subject: [PATCH v3 20/35] mm: conditionally write-lock VMA in free_pgtables From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 894B6160013 X-Rspam-User: X-Stat-Signature: g3z6qfyfsedhxq3kzhgbx8jjqdan41ww X-HE-Tag: 1676524724-791097 X-HE-Meta: U2FsdGVkX19qmn7GTjSpQGu/FlmWFg06vpSnIX3qotr9XTBSwFmxQbSEaZsnZlFsiIzI/9NLTGgznBlYVj+Afp0FGGjhVHjeTwH1lcbHPlbkjc+If0ZznukhuYfo7X/Dw3VdY1EUoTWDnAx5aP8T50/MeWZpBu5vmcuvhg2z7NC7LHLHVx68v71gIUF+A5QfJ180XN9dyEmhrVPyul7N9lUtQNNa++bs2xbOL2Vc0SaNWC/TVpHn0BN2Faz9sxH5492lOSkj/fDzprWjaFGwg2wmeTKo0PyjC8Qts0AAFxtQi7DKINEgY2r3K9B1Ql1n+sgjB/hnMnE450zb2YhX5Qq3CY2EK3A+Hw/UPebp/gYWBqEIgtdI/+kb04iNXFNPA7Dhb/Ouk1gXWfe9Ljmv4jiKDP9h5f0suKIoBD0OJy35hZkHFGPjjlgImjqDiZMHPl+Vd555UKdObDiSp4fmmpFp5yQ+GmTpIUYxHvOY/HnG/izhAo368F3VTgOhNuWBpjdF3Cc/bz/t4DL+AFJM5vsZWTO7gcCev6TsOSTql8DSDR/p8ToMyVAhHmtudKXpo98kOoV+bF8uNfxpxXpzVt+G5f6AlRhOMnvc4kPWeCSIpoWQg7zpChqLaJ8h9j65/qMSdj2OzTtWsR8AM+LhxhdEHDZrHi5jFLf+jxcNl6ZOJbBHyFZ3txai2ogdtKkciV7dhpeVXkIO7HDXgPmTlvgkQ+gS9UHdG9oCdLiJtpn0ZvjRpuPNqa8+wcqxXkaEpq391ld1DZjwAXL5hjfc6TeuIEPJeUSiDYkIuOoQqaFRO1Pu6NWmlAM5iJfPyl9EA/SlHrauyKtWCcZ5ygc2p14elb5iGCkzzhXgAUvPpHEr8EkXG8m98Gv5TCBpfpdjWuubq5H6P7acRtdeC9UYXfXgK8BoWyX5bfaaexRh4YL/C/BAk5dgkFidy5gZS6ulVKrf7vReOpZzaEBnmEx 2lKNMoo+ 1ICUoPpBe3b+vs75eF7FFPAvAa7h0GvkCPXHFNqelCQXMtrRT+WNbGwtZIuJkSI2VuJlSW3sza3kLz9KjqLrl3A1hDST/YRHH+HnYEesPD8/+agluf0LolU1xsmc+jiEjx2MCyDIGnRjP2INWbPrsxMsHcW6boqifExOQPC14n6RZxULuy+PmNbnYo3kqvvcTrkmfjLIct7KFZga0XaqKlF/m8TejsClW89buAmVslTpzGkA02hF3xwKEfi16gZri13+m9ARtfwYsfJgqRq1TlNV1gV8mr8Iarl91zVLZgk2BwQr0z00NMtCOxuSPbi93Y3HjR7mq17QB4+2MxjBYQqWy7KdjdoCKNWpWoAzhaM+2/yh6eF5YudeBmAtOjbarYZ+kPt76fOWCHcZdI66355b9tXczxPbloFeFAFNImImBvDbPTXrPcyoxAsq62zWzrjR2Akf93e2t9ZsMCFuXR4zYAwcAaGQKXCvuboVKY1DoCqkpvDQYQ2xtCk9yIqqVLoimC2sl0jAf7GmmPBbrQOfZcRF4OUJQJNofkiwHO4g7nZDpfKJZV7c7btRwZWK95h5n5gbMVQsYQwktDWk8yhZfV/42QeNlUcZJ6zFmjlekUimY0by62+WaK41lOF78dfogWK5ylfhg0PURkdkNx/Xk63mx8XK9k49aoX5R58veFh0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Normally free_pgtables needs to lock affected VMAs except for the case when VMAs were isolated under VMA write-lock. munmap() does just that, isolating while holding appropriate locks and then downgrading mmap_lock and dropping per-VMA locks before freeing page tables. Add a parameter to free_pgtables for such scenario. Signed-off-by: Suren Baghdasaryan --- mm/internal.h | 2 +- mm/memory.c | 6 +++++- mm/mmap.c | 5 +++-- 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index fc01fd092ea5..400c302fbf47 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -105,7 +105,7 @@ void folio_activate(struct folio *folio); void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *start_vma, unsigned long floor, - unsigned long ceiling); + unsigned long ceiling, bool mm_wr_locked); void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); struct zap_details; diff --git a/mm/memory.c b/mm/memory.c index f456f3b5049c..8177c59ffd2d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -348,7 +348,7 @@ void free_pgd_range(struct mmu_gather *tlb, void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *vma, unsigned long floor, - unsigned long ceiling) + unsigned long ceiling, bool mm_wr_locked) { MA_STATE(mas, mt, vma->vm_end, vma->vm_end); @@ -366,6 +366,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + if (mm_wr_locked) + vma_start_write(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); @@ -380,6 +382,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, && !is_vm_hugetlb_page(next)) { vma = next; next = mas_find(&mas, ceiling - 1); + if (mm_wr_locked) + vma_start_write(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); } diff --git a/mm/mmap.c b/mm/mmap.c index aaa359a147b2..118b2246bba9 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2157,7 +2157,8 @@ static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, update_hiwater_rss(mm); unmap_vmas(&tlb, mt, vma, start, end, mm_wr_locked); free_pgtables(&tlb, mt, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, - next ? next->vm_start : USER_PGTABLES_CEILING); + next ? next->vm_start : USER_PGTABLES_CEILING, + mm_wr_locked); tlb_finish_mmu(&tlb); } @@ -3069,7 +3070,7 @@ void exit_mmap(struct mm_struct *mm) mmap_write_lock(mm); mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, - USER_PGTABLES_CEILING); + USER_PGTABLES_CEILING, true); tlb_finish_mmu(&tlb); /* From patchwork Thu Feb 16 05:17:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142514 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53EE5C636D7 for ; Thu, 16 Feb 2023 05:18:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E15D16B009C; Thu, 16 Feb 2023 00:18:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DA1116B009E; Thu, 16 Feb 2023 00:18:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C18B76B009D; Thu, 16 Feb 2023 00:18:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A7A026B009B for ; Thu, 16 Feb 2023 00:18:48 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 82FDF141007 for ; Thu, 16 Feb 2023 05:18:48 +0000 (UTC) X-FDA: 80472000336.26.65EBBF9 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf07.hostedemail.com (Postfix) with ESMTP id BA39B40009 for ; Thu, 16 Feb 2023 05:18:46 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Hd3NZlon; spf=pass (imf07.hostedemail.com: domain of 3tbztYwYKCAw463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3tbztYwYKCAw463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524726; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zeEKIs6RouZ3NeHI7ZJ8m5Jmv1yLdTyFBRJo2pt5tCE=; b=ob6xidXd+WxxnR2HLVpcFdquFX65vZgiE2hayDhdQPiagrUzT6hqJvH+LuQ46LGNrpoCKX SD1XJzTBv+S0O3y3BHrLM56NqwsT+CXlrLZL5MV32p2t0YkavLYLMsnTzVv+CTgk/TcOK4 w3TG5xyrVcEVWcxyvPJ33PZIJ30u+Zs= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Hd3NZlon; spf=pass (imf07.hostedemail.com: domain of 3tbztYwYKCAw463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3tbztYwYKCAw463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524726; a=rsa-sha256; cv=none; b=FZhIDPBhltlEg1VyEQxhruhMcNz1x4hfeGPWrVhhPdfKzlMBe4fqc8rCplDugFogl4x5zh wldhAB7rA+ts1Qwmhz7gu7eIoWDvXvHH8rvMFFFPAcPRGzLnrmeov69jOI9KaGt1H+bMOa 0ubiyhRoeVw9kGCIa8K176+LE9FeWLE= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-52f2ab6a942so8475667b3.14 for ; Wed, 15 Feb 2023 21:18:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zeEKIs6RouZ3NeHI7ZJ8m5Jmv1yLdTyFBRJo2pt5tCE=; b=Hd3NZlonbpCzy5K7Z5feVeMneevLupNr2wIBk8xrF2F0FeVHuTW7v0p8Iwz465OudO Kz/cwziHnFST/NwsyYv9H1NpjXxfHSGLxO5ukc+3xsoPYXaSl8+RA3RdmvmM4fbLpXN9 gimJ3OxJZ4Ep64JmtioAuz7PGMRMavPtJuthlMhEIfRvRDzYwwXEt+kIY/UhQ/1ohnXV Mp7Yg73p6F/OdwNe1qmUkwSe8yYAoyBy/8c1r/0Zsdj45B+B6DEhDKlW7MvfqHG54uOt TWogzsTy9UemTUaeFAzTdpALJEgq+fwHHnoe+QVimQT+Dp4bg9S2kMJXbPj9+Gb2VnGT o48A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zeEKIs6RouZ3NeHI7ZJ8m5Jmv1yLdTyFBRJo2pt5tCE=; b=ss/hWwyybaKEvKrUab4PAsB5A2oc9WZfRGu+x75dqIqv5cex4hlI7eCuCDtRDkPA6w peWFdEyf36BirgheG2UTMxquGVy/D5NZDOX82EuC5a3bQMRD9EXl2dXyzyUTBJgf10aR RgRT/Vx7WbPCbfITWGXGCq8VX6LM6RoGPpAnS1FfvdzIMgOdK9MRgFpLaNl5zwNBd/oE xCwdDkDugiA1ltCwfhtstHmC7unkpKKLYMIi299DcmuAk6F5Y9D3IB/qHbOvX7PLVSvm 8H/ncnB2Uy+QxgWRn0wzJd1BJqZKp0TGWXglUEeVy8yAC8qxvXvDGSpfeTVCm+ObMfuB MaSw== X-Gm-Message-State: AO0yUKVW66pTML9c2iWdn47i1LFAaEL4Kl6ElGYBH4GuGfKU7vHKgoQR VGTDw4mInQEB+ZgNFwLGFDQjgnSaTcc= X-Google-Smtp-Source: AK7set9EG6pwKD6Qe8/WGqDgk7Um8RE2RvPxkguhsK1CKP8nzrHSAEE/Znnb7/ves5IDs/PTEla86+djibg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a5b:604:0:b0:859:2acc:deb6 with SMTP id d4-20020a5b0604000000b008592accdeb6mr561571ybq.79.1676524725671; Wed, 15 Feb 2023 21:18:45 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:36 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-22-surenb@google.com> Subject: [PATCH v3 21/35] mm/mmap: write-lock adjacent VMAs if they can grow into unmapped area From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Stat-Signature: f9kdzpxids4dhi7tisiepf136o1joymk X-Rspam-User: X-Rspamd-Queue-Id: BA39B40009 X-Rspamd-Server: rspam06 X-HE-Tag: 1676524726-184247 X-HE-Meta: U2FsdGVkX1+5kTV9jVOswGh4102IWTkhxcD9yFoOy9SjPUI7xukusqcgYDZHKskq6sGwdDsx3VehXXMcbqH4P0BE0+lUeIjLj7uXojwO6AVdRBFDcC8dnhlKwi3EnYZyy1b2L0HRKeObAX6xVR0dYI9NNzXPyAjLdT2xD9kDyYfFIWszdUFsP1piZPACUOw5d6D5QHtwDVv2TB2gwanbBwsmMo3xPqh7GoaZ7zg7l3fZbd7teqYKGbzIfKYdN45lExxINKQUXAXeJR8i2paz4hVMq9NpWYjop24Rb/s6NHHXLMFtA6W/BBswi1pBXHXHQpXFGScmy7b2TNRsqkFcQr5RsYEnoYy9A3U6qtl4YAOZnSjWrFQvTcz+SN8YTjgvXYIH++3cEpWtrZ23hr1pTSpYt3qgXgZjRbx7OAvlRQ4w1eDdyjbW2mRQlfVmH6w0ez3YBtr23JWgNXoPh8QpJJ1WUYh0I9AAkJDZ4OGO6RKOyFwgd9KyWZ2UbzkpAvqr32td8vtZTxBGm4ggKOcjaoESFTSrJc0Bsj1mfn5g2IkWriDnn7fR3SryyTjhc/ZMpr9P4pRNEJOM9LWqdBoOX6eIm+kB6jAxZpLZdJVsDGpRJdXVPNDWVE7RMc+nvhKsvJECuR5pVKUs6+545ZdcaTWfQZ8c8A1xwKJ2wrSvAivHYPo2jQ4ngViGlMu/E0bHKzhyc1rWaZxn4pIsS4nFtdd2cs3y5SvkKXHRu+8X0enOHW5ro2yfjZap/IRCIU6Zuk/J/6T3cvu4SA0f6nSQxNR+7YeaNGouVB7jOPd9RYAJcdTXt7fiMhHgPi6aQBqjRslsr4HIvWzyi/rwgcspexHfn5ENfUInAdqz88tG7Wh6SIvTmo+c55yp9WD5+YEzMueudwxFYh0DkEZPpUIOcdJ7MqRczPyVTLy+0HUfcSrMP+Y31MOGIpL+WrsVGEzXGEkIZDiEzfbYvTfnCqP 1yAywzGp QqyPkB+MKfLmnQqq885/01L9sm1PNuCaTpY0Mivao7Fk8nAUhO1/EPNj+1lUCSDOBzIeL00sg55wzLmrb3yqb+pORoAZnuIMblKk8CBQvZtBrySW99x4X/rpaTH1cJ3aCuhXTKDOwsqlt2Cnz6l993npCcjptuxOSTEkXr/T28COzjelAOIYaI1ozfc50ruiCrsfvaon0h4sIDVGQqxZxkc2x2f5lKaxPRWNXRJQb4nMH6HWQ0zg1e6bjPBo/4lVTRpXhJeMuxUZu2miTFHZl8UrWbEfRwf5qdMptd0GaP15KPnVSRpvfQfILBE9sSdKsS9m81Wk1sE+DP3DT//b9/vQDNNO6roBUC+eBS54WTGx6WTevLS4NdGmRaJqvi0yUa3G1YgZtNYDukv5lnN0P6R/Yko4wAY0e9er4O/nF2tFmwbWr5NaEGNXxCBsmEMnSxWsW2XA4EoAd+ytRYKWvsNJjfasGyX1sYstdtOkmlNjg7HyTl/0xCPSYnOpNCF0Xe/xGn0sVhWwrm3jTgc+YLUPekZcyeJr7hgrfSv9pYJibtUJgoyO1/iLoPvbKBEE0vajdD1ZhiScUgiae8CHebD1woA/9QONckU3PpGV+XxguBCjGeZ14J+Rjmw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: While unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases write-lock adjacent VMAs to prevent this growth. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 118b2246bba9..00f8c5798936 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2399,11 +2399,13 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, * down_read(mmap_lock) and collide with the VMA we are about to unmap. */ if (downgrade) { - if (next && (next->vm_flags & VM_GROWSDOWN)) + if (next && (next->vm_flags & VM_GROWSDOWN)) { + vma_start_write(next); downgrade = false; - else if (prev && (prev->vm_flags & VM_GROWSUP)) + } else if (prev && (prev->vm_flags & VM_GROWSUP)) { + vma_start_write(prev); downgrade = false; - else + } else mmap_write_downgrade(mm); } From patchwork Thu Feb 16 05:17:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142515 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E3B0C61DA4 for ; Thu, 16 Feb 2023 05:18:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 230166B009D; Thu, 16 Feb 2023 00:18:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B8D86B009E; Thu, 16 Feb 2023 00:18:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 032CF6B009F; Thu, 16 Feb 2023 00:18:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E52DA6B009D for ; Thu, 16 Feb 2023 00:18:50 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BFAC7C087E for ; Thu, 16 Feb 2023 05:18:50 +0000 (UTC) X-FDA: 80472000420.09.69F9DB2 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf27.hostedemail.com (Postfix) with ESMTP id D141640006 for ; Thu, 16 Feb 2023 05:18:48 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ee09VaaN; spf=pass (imf27.hostedemail.com: domain of 3t7ztYwYKCA4685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3t7ztYwYKCA4685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524728; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Cy3b4YdYe1JheypyM1qEqlntx+LNPOiVtlvqoTpRTGk=; b=oAQQe7CvMHz4JlfWsZcFStSU4hOSRIIc9tQ3gME2OQyIY/jmfSfsndfd+PzDuudoXuhDtC gmYmZuEPQcZTW5OyBHKtycmgZQtNdXu8OjobZCGchIMtZi9+OvzlE4oiZzn6kG0+i2MDq6 ZGImHE7iAUI8R3EgSuX2OEd3nIhEqBk= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ee09VaaN; spf=pass (imf27.hostedemail.com: domain of 3t7ztYwYKCA4685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3t7ztYwYKCA4685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524728; a=rsa-sha256; cv=none; b=ys9lsTlvpn3u6n9jBFaJ63KPOwlmFIhPCxtp6oObvFfH4vmdw14AhrqUnduYc0jVEmPWue eAzwUrzboeDsfuk9RdR9bdxb7VULh/rY/aK9+VrSN0gSQF0T+acMDZjuWxiObj0IWg7Egt dsTEHzskorwH2CHFRiBrx3Gu6ZoKZQY= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-52ee93d7863so8369147b3.18 for ; Wed, 15 Feb 2023 21:18:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Cy3b4YdYe1JheypyM1qEqlntx+LNPOiVtlvqoTpRTGk=; b=ee09VaaNfMCXdTYs7ho4FL1rz+7Jau0hVrpqtPJwR0LOjVJOOKMHcMbwd/rGxak63H BRVuTQ9tdXJzSR+6NhiK4KAEy3orTDG5A6ugcB1Mu/JZ3TtJ8rFlU/MHE4DwjmulfLqA 4lbez5vLr44VvmK0HXNKetjG61FmR5dtu/ffCFjZaMQfVdd6yPZ6Vqs7cSoCFmCWN2ej U7UJuLArjVJJvFA58VBbTFzUGAlbxAZVBV5EE2zVq6EhbmHffJxz6yeGBWtLnffH9zfX I0SKmeE3zn9vqZPVYyRXu1tk/VZ/LisV9OyoAUd6iOjxLewnK1RWOS7bKO/6E67varMh DYqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Cy3b4YdYe1JheypyM1qEqlntx+LNPOiVtlvqoTpRTGk=; b=cIGFsuhHQ7gM/LVzYydHgGKcmTavy/VFKswKor5+BnRGS4tIoRPtvi51qWRtRThsLE 6wZk6HpUdxeu3miGNkJhWf4M/FN2CcEcshQT9M5Ab4Q49ydoftu93dpBkR0Tt1PPpfSf CjhZLxnQpE0jzNJlZhjSbbv2jV1tgfgmuoSAsBRQ3h9nwPBYt1uhzyOC/iCKcRKx3vv2 NVBxL9Y4TqKVKNqc0DYzzKEmlGFJKZLOHyAVhfxNupZxgIFBtKO/P3R/Dk8kS8FKBQ3I RC8oJKO1yyLUcnnddyG6N1GZ3vqz06RTVHFjsoumqwikOzuqKkSjuaseAUeihUvFX0jI 6yOA== X-Gm-Message-State: AO0yUKVRTh+XBokNjEe5ct9DCYTtPrQ6tMO8TLmRuxGWrguT9xeq+5ub s3fRI83uBb6rSHXzj8nA+JkHlmSafs0= X-Google-Smtp-Source: AK7set98gd/BBmD2MrHMvmTSXIUmiTX+HY9u8tg6jgvhUw+D34Dv05igzcuyCWu8/RYb3F6ryIOQeEDmrWY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a0d:c347:0:b0:52f:1923:4f9d with SMTP id f68-20020a0dc347000000b0052f19234f9dmr562132ywd.366.1676524727987; Wed, 15 Feb 2023 21:18:47 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:37 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-23-surenb@google.com> Subject: [PATCH v3 22/35] kernel/fork: assert no VMA readers during its destruction From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D141640006 X-Rspam-User: X-Stat-Signature: 4rof18fp7n1u7awrhem583b59kam6gz1 X-HE-Tag: 1676524728-291776 X-HE-Meta: U2FsdGVkX1/3V9tsI8RWasobGE4u7RcUjx1IYzZubyHf4a+orUS1d86pMOeffha6bosam9mwiiI8SBlYNzsuEJJODY7/IXG+khnp3L8Dc920Chi/GSKZXkADIL/q7gKwDJbZxW2rxrbZGj2SFVWi4Hx2+2w9gV0PhY2uiEFQdC5btMnnDo7tu9xghTVzh0yQspC90IcbC8H+T3MRTcv6gvl9IT+n0tVbqTo5AtFGTXJncc+A5F3mtv866RfPyA7bKcVd1IRY8S6O8nfALUkVskLYmcyH1fFB9few9FxwkkbO9GLBVfvwep74c1xmutkqpTQ2MaSbe9z9vE6Srmh3R8Tt/RQlDpMBDkeDdcPROcB39WgAHhkQCcMbQ5a2pSI7idNP74EihWFGxbGX/eHhvf0mBME0CLYhg2Hy1EQ/O/P12wC6SbyXj+iKMjsPz1p+8OHQ3spz/hEb3C0zS7Ky0+5CKMo05fonfyKlsH1/akYN3he2I/Bl85f2GFNAKonFFDmydej58ZNvGYF6KMHU6JG48JhbfotdvB3bbEgH/zcWkYAwlJcYBLSlnTJpt/QqSEWS0zPglAsPnq+uA/n9btZulgDf9HtaU9wGy0jLFjTTZFv8NdgBWxdAMxOzFLodjfMqqKyh6djyOtEXJsGoZRJizTIPvxxFhJRmv9tLY+733XKACr0IoFGvJ/bsd7niuypQrDWEYuTal+5zlJsNDW5YT+Rc9covFCznZr+Q0+1aNZ2WdgQ265b0kWbSOrM2tQGgoDqTVG6fUcIxohSRVAuWdG6IUS2brIXQFRE01d4sctZqN/3Rlh5M7WS8WQ3ze/RBWxBXQuBMErxJL6d2QVYEBRCintvDHj2zBjKED6cwhNjVS3bmrclf0iRSAlIZMW0pTIbmxCCpYn+pR/aRZNtqGL6PuPT6KcP33aERmLRtYmFQBeMHlJQcmOwudJYs66BnN8cTotwSNUhQJiy khQeDP3u gJC442cl7WvcVsoWcC1432fUe8bTvMuSAC3g/m24/xmPirfDScfzmIyT7CiadNh5NFVmzHKvmfyoO5yCF1dBF/nmuQ69yrclBuZFlRZr5V+OOMJJecReahUV3AEr99Zwc+TlZPO2nJVQoJcTRLaRAVzCig2O6xbsLLWHLtxiZHaxBoil4DdB/QCHIJX26BQseMzFxgO/DXaztbWsMQS8JJENUJYVUrjlnLb1W4hs+5A/DzTKlbMWXOtR2Z6C7dbppCKczp2Kug1+Erxqd5+dd1n3vaGqz7uONk0JmU6NA/EUToB7jvCH9nAgcCGkGhrOscCXQQ0Hoe8QnEFEbUhbNcrYSAUQ7h7bdlhBp+ikX5bkruAD53DeXw725a9K8G5wN+ilHoK92cjskyHp/Ma2KYoazZ4sQDgeNha77y9ol03xpV2IPd8wJwsjRxBSNXPbJtV2jW4Ox7fwhJ8omsiChDDWluWrhFVLSwmd55ReLLMwKsp3ftu3teNae5H4NMKUIbx4BpLu5v651EICeuZ1aAJXG+dhdRz+cpDiBRzFD0GFmUU0rZxkLl8UIkR+BlinDfeviNjG/Bc1nAsggabfYbw+MjbaKKqJDiKnvdfz4NY5jXY3evNr8uDIGKQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/fork.c b/kernel/fork.c index 9141427a98b2..a08cc0e2bfde 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -491,6 +491,9 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); + + /* The vma should not be locked while being destroyed. */ + VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock), vma); __vm_area_free(vma); } #endif From patchwork Thu Feb 16 05:17:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142516 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8B33C636D7 for ; Thu, 16 Feb 2023 05:18:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 75FA66B00A0; Thu, 16 Feb 2023 00:18:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E96E6B00A1; Thu, 16 Feb 2023 00:18:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 513CE6B00A2; Thu, 16 Feb 2023 00:18:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 37DA26B00A0 for ; Thu, 16 Feb 2023 00:18:53 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1604F404B9 for ; Thu, 16 Feb 2023 05:18:53 +0000 (UTC) X-FDA: 80472000546.03.1C60063 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 50109A000D for ; Thu, 16 Feb 2023 05:18:51 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=lxQtFymG; spf=pass (imf25.hostedemail.com: domain of 3urztYwYKCBE9B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3urztYwYKCBE9B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524731; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GTmH+3zLa4DS1pm6XKzoFa9XiN/n/492GQmNm0F2u98=; b=3O2pzh8d6PcULleJVZLHBtKCjRktROp7qR50XZpPUS7p7z5IvxAio1YJJ5rheGVSMizVpJ ilrAGfKL+SSAXnVX3ZckwurCv0P2YHy6NO6JXI/iYQPwlwHFNXODGOyvf/rJ9iJ/1+V593 NcWcS/CQNkEnxeefnztZeRRUWapXQB4= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=lxQtFymG; spf=pass (imf25.hostedemail.com: domain of 3urztYwYKCBE9B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3urztYwYKCBE9B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524731; a=rsa-sha256; cv=none; b=2x1UwWF5gR0TRhLgDJ3tqKzrx8Xoqqi/zj+pOUd4ZdD6KFJutHpXzHXvQzYfR/vn6inN4T BZlq4FUUIO1KrjC00qVS3d0aLba0904sL3sDD2bMMtHRNAGZkuC8uoaCnMO/EFHNikl53A 1bZ5ke0IFp7f6bYp0lRwwESDktxWVlI= Received: by mail-yb1-f201.google.com with SMTP id 191-20020a2504c8000000b008e2fd9e34e6so854899ybe.9 for ; Wed, 15 Feb 2023 21:18:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GTmH+3zLa4DS1pm6XKzoFa9XiN/n/492GQmNm0F2u98=; b=lxQtFymGNoCMn73Q032qmSTTcj+ve7GubiXjG/S2/wZlI5A+wvms4mpLfeb6oKn81f gVVSU10rEKBXvaTQ4DI3+isYBuVvKtNDNUM4jT8VbxX9prWBulP37D7mcvfTTB0yv5Vd xLNBfAphNgvAxOXoC4bxmXgktQA4hGiUCCaPhABiIUzurEkqvA5yibf0xhi6vP6KsH63 2fMMNvtfnEstPxLz+UPzwP+9hfF+/M/equZBkR4oGxxGpJkcmKAi9IzofVMskpdF8tY7 tvoaCpVKwaaM8HBNFnnIuJry7ZsbTgV8SMkds4HojfN0Cbg+ccWTepT6dYOQ/gKplcjn qubQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GTmH+3zLa4DS1pm6XKzoFa9XiN/n/492GQmNm0F2u98=; b=rBR3v4eaY2uMI4YeeOWQ1h1F580iKaYq61R0sXXnlrBcTG9y7FK/lRbMRTo0FLzDUm cKnMfvsrju4R32SKG/2w0eHeAQ2pebINCnSaBBeqiXfxL95tGxdlk9lCUU7THj9p/lnL o1GQOIbrr9ArPxksmxyDrvgrMgAJ+wOy32ymSBN/5YHpKVuPaUqgNHH4F+L75U0FDLb6 1jFKdnU/yYO9/WDOgZry8nwuo8ZAfxteDza0GBh2e9zgkT5ap/23BykIBhP+DQqhf5w2 6wCUhKe6GeudeMu9fKHRgunLOPZuB3MsP6uFcQjjiHNwZhAqeLVIBitPsn4MXnxJseml QbsQ== X-Gm-Message-State: AO0yUKVycLyo3flwrfU6Xu6UHXnMOvNXOuSbB3vOJYiAczoOkAlPdaSj 0WuXQwxr5kbHGuz7iOLGqm4C6+QHTzI= X-Google-Smtp-Source: AK7set/Pm4WgUpt5iRaimc/pvYKHTFOy9juHCFu4sSQHLZZAabCINcoiFbttYYVnL3GbT+1smZzLxo4F4W0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:441c:0:b0:52e:c79a:cda with SMTP id r28-20020a81441c000000b0052ec79a0cdamr12ywa.10.1676524730090; Wed, 15 Feb 2023 21:18:50 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:38 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-24-surenb@google.com> Subject: [PATCH v3 23/35] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Stat-Signature: kst3dczn59g6kj4s65kxb5xfpyu56acu X-Rspam-User: X-Rspamd-Queue-Id: 50109A000D X-Rspamd-Server: rspam06 X-HE-Tag: 1676524731-961268 X-HE-Meta: U2FsdGVkX19ZpDLQ1pWmYqMgoDFbVwbK3fDmGQEcGtAwvKeE4+RZkVB7qWtat6oqjTYqLCkNizdrQj8VJh5Zki6aku4b8mpNyQdfdjd0d7uwctM2TxYoW3xkLxr8dLdakrUFGarINGb+3oM4RjCQl1BqO7K3YnEPXJOyAPj7Qafc8SWG8ybD2FMYDy8LgVmdPyCmw8xVuSI+3w6/nZwFQVHNe+HpxBgS1MNaGeq7wPAz/U/BEtEd85q8t80Z3rL5Z+79FaBbm+Ed+c4UfDh0+PbQC1yT6LfDMZtANDmrwkuTKn03wS5oeqVSB6n16j0PF06Foen9ZrBLe7tZ5UNcLnYB0OcyOPU3S4c8XQgRBR75degJZokCa12pPf7GF2Vbi/K9y1dPaKO6JsCJZfX/v01BwbUqDfO0emasJfjAqD5Nw68wXZi4Fz3JZBqVmlUeeTBQHuj+t32LdEzAyWtS+w6i5rtfZk0QrJdSN+UPo4TFC+7H1zsm7WQ7pSbkQxj/x3rI78JNNBBFTXJ/mSu0PeYSmTwlQ3PpQx0kr4WCc4Xtfs5KSQbOWZUpYyEJu4nBCfXXgRwHp5TAuLaWRvLHZqedTQsAT2zd8RXbQmetqOpGHqyB0rF3TdCw9oTEdVu9hOQ4z2oXaErTm6u9KwIrp85UjUvjTNsOUmwJ46azzu1+NWHl1aIGj8QJBlTdV5IQ5jwFPbIuPcYo5NH2PX8bihBXHYK1YpdIiNTReM2D3NMDba8fKTzvx+0ALNqP3fHGf9FapOm0i9BuAkPhjBRRP85jNK6FXYvUrnIwduKx/1xq96QEtviObfmZ0nq7tOhNx1w2iUo3Ti74BPeogfI9LTuUAYekPj+zifj58dBSR+BQwW7hhyj0m3el/uELlcJ1U1Z9hqeAtvM4djY+4omvVTAhMQaHpgj/MPB5J5p9RlXIiwO02P8GsTJJM5Q8kiJZGtiQBuO3wQU/YZrjqO9 8FZ8XcVe CSpqM6z4OKIbKIwHd9Lh82HUbKnH6OpwDwfcqA/6hNXO/Xa2E9IWYbVFrazgwlT0suxrFToBYJD2Q/PEfQen/uGx69cgl+vwjHdhZl+KStNGZwAjd0bYQCSGN+FWEsCS1Yq4ePkNSLD24Bz2aEDoTSkW+/g739V+8G/cTXy56hTd9RbSQRwf84iAvKIge+MAB7dCgox/afD+O+jELP1Nnrbylg85VytCZ95ARfVChDbPIxOce/tYdsYex3htAGvNFLyxcE2mUszTABrmhqe5nhCwQgvx+H6aWdHMqN9tPh5gUWKlzpVP9sZA4llaCKXetUNjANMiVe1y4MKf2glHPbyPU8WFbDNAPdl1JE15qTEd+P76gJbMGLRlXngHOw8jsRB3J7U2uUc0dPNcuhSScUP1Kzr55eYIGoUbES1PNPRxYvNAbySgA4xuwa51SwP7KejjRqnSxFY/pn7iGrtcgzolH75IHHqVewXn+VqyLrbi3LA4RSZ6f6AZx/7CCkn0WYzKENupVhfcBFSLUiNx3W6LMpfRWmc2WJyrZZA594+QRww06dpjmR2QKDDuwVjiZo7pdFPvgUheA1y4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Page fault handlers might need to fire MMU notifications while a new notifier is being registered. Modify mm_take_all_locks to write-lock all VMAs and prevent this race with page fault handlers that would hold VMA locks. VMAs are locked before i_mmap_rwsem and anon_vma to keep the same locking order as in page fault handlers. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 00f8c5798936..801608726be8 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3501,6 +3501,7 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping) * of mm/rmap.c: * - all hugetlbfs_i_mmap_rwsem_key locks (aka mapping->i_mmap_rwsem for * hugetlb mapping); + * - all vmas marked locked * - all i_mmap_rwsem locks; * - all anon_vma->rwseml * @@ -3523,6 +3524,13 @@ int mm_take_all_locks(struct mm_struct *mm) mutex_lock(&mm_all_locks_mutex); + mas_for_each(&mas, vma, ULONG_MAX) { + if (signal_pending(current)) + goto out_unlock; + vma_start_write(vma); + } + + mas_set(&mas, 0); mas_for_each(&mas, vma, ULONG_MAX) { if (signal_pending(current)) goto out_unlock; @@ -3612,6 +3620,7 @@ void mm_drop_all_locks(struct mm_struct *mm) if (vma->vm_file && vma->vm_file->f_mapping) vm_unlock_mapping(vma->vm_file->f_mapping); } + vma_end_write_all(mm); mutex_unlock(&mm_all_locks_mutex); } From patchwork Thu Feb 16 05:17:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C6F9C636D7 for ; Thu, 16 Feb 2023 05:18:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D7A956B0073; Thu, 16 Feb 2023 00:18:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CDB1C6B0074; Thu, 16 Feb 2023 00:18:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B555B6B00A1; Thu, 16 Feb 2023 00:18:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9D4F16B0073 for ; Thu, 16 Feb 2023 00:18:55 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8310EC087E for ; Thu, 16 Feb 2023 05:18:55 +0000 (UTC) X-FDA: 80472000630.15.02D60BA Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf08.hostedemail.com (Postfix) with ESMTP id C89F916000D for ; Thu, 16 Feb 2023 05:18:53 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=e8FcGNDH; spf=pass (imf08.hostedemail.com: domain of 3vLztYwYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3vLztYwYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524733; a=rsa-sha256; cv=none; b=m/qBxgvYFlopof28LAKubdxa2U2Z/ymVldGY8D1sfTwWjDZ2VtYYYAdwOFw+zqkqQoIHOC gcEUR2YScPCit8urHWp09ce99TgZt+fC2eO1SmgE+1jVnPUeHa3XhHcF1CAZUnqg7PmG6w e0L6bYszuj0Y0ZGhlVv/lxQbgTc6pV8= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=e8FcGNDH; spf=pass (imf08.hostedemail.com: domain of 3vLztYwYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3vLztYwYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524733; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=u26HcyQZUssNB/wvgmnc8FNLt5Bjt6C27EcCpLSHQ2I=; b=vFKYEZOxT5AeKFSe3MCqHum1TJ52GhTSkRUGsY8soMeWrg8N+w0UHcFyeIU1/5KWsN+6Mm i+DVLG/QG00ewYVQn/pvgh3f3BhaKYyv5KASerrM/P5ZitG26JMgh5pBEPlxk0esYEEWWm GDul6OBvNY4CDOxTffkw06slqYxbZhU= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5337e2cbc58so8652887b3.10 for ; Wed, 15 Feb 2023 21:18:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=u26HcyQZUssNB/wvgmnc8FNLt5Bjt6C27EcCpLSHQ2I=; b=e8FcGNDH5VkVzBTYRblbtAQ0+ZP2ugsPFuxmDPyoQXwVKDnKFUyAPJn/SJMrsaQ1Ia jo+Cd4OdGYV2adIa7Ps1WdQQvf3M0XkBbQCS2vH4hvx3mR4XIU6AeYNJ4fgFw+1soX5Z Pa1xM0MUFfbg6JS2S95T1SCUKSw+S8anZTC0et5KKeuEZZM6kRmGtxME3D+du+nzcvgG qzTaGZDic4CCJCQNQmsvjPxTw8HSPKOmRJrMlp7St/TdGbLlhdx8Vn9wPFTpNdXttF8Q VA64i9yGakw2MEWw1P/yGjLggzJzYv4qMjifZYLSh4iksjXsYnlNyRAbe89au2hayYpB yYBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=u26HcyQZUssNB/wvgmnc8FNLt5Bjt6C27EcCpLSHQ2I=; b=h9Ygnwyau8o6ta4ky4b+yzWH0W+igRkXvkXM/Vw0oF7OLLYnvAjJL3zz/H1Y1FevFG ZlTZDQu2lB7gFG0GrTYshR/08G/7I8L+EoCM8H4qw6NVlHQpMMpyV6Hr/ew6qfIHaCM4 wGBTqOHt4K+B7aEnsYo9CuW2eOeu+1B2eaLhBerWvFurdL978S3kdw/9mmC8JwNOGndt LiydAMm4yYvJRslfjxfAyc22YEL2/qPbNgKWi6aw/KgP3XkxQNDOgAZCOvGNLGwXNkiv CMBcFMETjrd+8mgzoZKqRI+ipCeHneQlgN/hRkcH+yYdzgCh8egXo16cC3jHCa8n8HsL 6GCQ== X-Gm-Message-State: AO0yUKWjNUUeYOtDDJ6c+FjUMC8TnWe35T3/U6T1m319eEQ5e18Wd9fr CIUzVznBDMQ3mCortD400Wic3Z6cytQ= X-Google-Smtp-Source: AK7set8LRdaZWnTNJXEEL+5OavvLJg+SJ7ELgpuDSzdI0PbEuUZjWdo/eOURR51UCivz1X36gI0P4ir+qmc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:114f:b0:855:fdcb:4467 with SMTP id p15-20020a056902114f00b00855fdcb4467mr31ybu.0.1676524732481; Wed, 15 Feb 2023 21:18:52 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:39 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-25-surenb@google.com> Subject: [PATCH v3 24/35] mm: introduce vma detached flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Queue-Id: C89F916000D X-Rspamd-Server: rspam01 X-Stat-Signature: rxmkjmwcw1mh8fcdohuanf1o4q8uoy6q X-HE-Tag: 1676524733-472738 X-HE-Meta: U2FsdGVkX184Ksct+tMyiKTjk9gUL+nN6sX8c7YIE9uvSDiioQ8+dPFC0ztIq0yXlFgSUYlZlEJfaIvbW3Q7D5L4IsNWX2XU4Qe/jxbphGv8WBn/q96c8nDcuxRJ2Ji7Bm3AeCYTDfMu6ogG5DAaV1sh+6MYXngHDdLSi0HpbRoK4aN6R6tLUpH45m7XuDR33asSkirvuN6ZMY1hUCDjUbDx7c+qthL8UfgvOMcBHvzNNtOg0CiHBUHRJRYCgl8FBtUbFZFsOZ3OmgznDRQ+kmnYaAg+/TBsETcCXLE5TZQiqvyTUeLTvBvwNqU0BfyRXIY8PuO/OhiX+ZVekG7cHQfVTGXudr1SOPvzJQACo+9r8XJRJt05lI4tTW8vv2jZysY3550RSNNrlXXwZ4CqPDqMUKRrh+ka+JmBDdT/hiYC6mqR/JxoYR7jfWEO/vvWoTkH01hapIjJ++HM0fu+dxuL5W7B95tgZr1ydnIcBG2TVp5t2nekk27MCiV0bXnZ+1G9u8NgDPZW/ViWPH4IT8oI3GRd9KvbckHWaBGxMouxxQUzZ0s14dcI/7T/VRua8PHDk9f+HKWuzBvJcpqge4ugBh0uPiGA6YeHBvpkUE0oFtXJ1P6X8Qbm9XwhJR9F0dYAav2ZhUAOAuCP4F/GpLX0BnVhVV2J5DMZs3g0Oj6LsZ8GldNkvh39JOZlUJRq4BlUUvvik9eFRFXQHn/trE+kXaOBgm65ENqEsKKOPBxplf9cTZUItqrHMlQ6kkK5OKbxAIrpFYmbTw5ZyoUqwYW0rhwdPI6nQvMeZfJZL3lYgWPkpj5TsgzJAIgKK+mAnifmgNGzlUlqhNFgm9I+RwXl8xZ8ttjFPHPuIQPt5OQ/WvbzdlzI3CI88sK9gbTzPbkWxY15VeqRiN/ARDt1HSexy0peBcTgwmBrvZ1tzVT643tdfNu5Bci/q8qVUaFfYXT1lqZQnjO39QtTls2 oVPzH7sS Q83ogPx5pSeLV+hkB/l/ArzgfMhsfvphLG4NNVOSHUhqwggpx+BjTd/c9T0Upg5FPKVD5jI+mr+b2NDRlgnPTmLoeu/eHAnK5IiEBqkjx+uMWeTrxNl1VtAPvABsUX0R1Y1OYpVy/+l1En3tbAeBpGkK5HsAr1IsHsFKYGWEpyWXTuyqy3jdBe9lj4Rol3TYN2m79KjxTaqWAlsdkFfZVezCBnn3AeqQiqovQcwBrTdYQNosFFVBt9qXWP8oHNgvRqly0ikqmmzbeLs05i7G6FEMRqBWlSuyKoML5lF3UHPAxfeBiXd4NzO5uayhis+Ln/C+c4cOlo7R32JewtnheWKqwljWuhLKvu1NfV+ftHVMy80417tYI2UMFrUPBzjcfpXaIEZFnJ/PMtAZGM7hbz/lqa3QOEZOx0uEyDBwig0VCGy6TWezYt+v5+/94eqOp9zHihcjrXyoZF2LtSwrAd/sdcyxkpyV+cyYybbSfV2XLKqLEFzvVxIJ0ai41jJ1YIUqZY01ChqJ9ysghawePO/m/rtPSb5ZKtA9mDlKOqyDmI3DgSOCNdK1wpt10PPxS5wqncDnSFhYWWfkGYFz+BjGa+Xr3BJfUOn39EIoyB15TydCe/KQrscsWUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Per-vma locking mechanism will search for VMA under RCU protection and then after locking it, has to ensure it was not removed from the VMA tree after we found it. To make this check efficient, introduce a vma->detached flag to mark VMAs which were removed from the VMA tree. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 11 +++++++++++ include/linux/mm_types.h | 3 +++ mm/mmap.c | 2 ++ 3 files changed, 16 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index f4f702224ec5..3f98344f829c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -693,6 +693,14 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +{ + /* When detaching vma should be write-locked */ + if (detached) + vma_assert_write_locked(vma); + vma->detached = detached; +} + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} @@ -701,6 +709,8 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma, + bool detached) {} #endif /* CONFIG_PER_VMA_LOCK */ @@ -712,6 +722,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_mark_detached(vma, false); vma_init_lock(vma); } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index e268723eaf44..939f4f5a1115 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -511,6 +511,9 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; struct rw_semaphore lock; + + /* Flag to indicate areas detached from the mm->mm_mt tree */ + bool detached; #endif /* diff --git a/mm/mmap.c b/mm/mmap.c index 801608726be8..adf40177e68f 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -593,6 +593,7 @@ static inline void vma_complete(struct vma_prepare *vp, if (vp->remove) { again: + vma_mark_detached(vp->remove, true); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -2267,6 +2268,7 @@ static inline int munmap_sidetree(struct vm_area_struct *vma, if (mas_store_gfp(mas_detach, vma, GFP_KERNEL)) return -ENOMEM; + vma_mark_detached(vma, true); if (vma->vm_flags & VM_LOCKED) vma->vm_mm->locked_vm -= vma_pages(vma); From patchwork Thu Feb 16 05:17:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142518 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAB4DC636CC for ; Thu, 16 Feb 2023 05:18:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C23E6B00A1; Thu, 16 Feb 2023 00:18:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 64C106B00A3; Thu, 16 Feb 2023 00:18:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C5516B00A4; Thu, 16 Feb 2023 00:18:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 327926B00A1 for ; Thu, 16 Feb 2023 00:18:58 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C0ED3120B37 for ; Thu, 16 Feb 2023 05:18:57 +0000 (UTC) X-FDA: 80472000714.01.64D76CE Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf30.hostedemail.com (Postfix) with ESMTP id E52628001B for ; Thu, 16 Feb 2023 05:18:55 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=hu+sMS9o; spf=pass (imf30.hostedemail.com: domain of 3v7ztYwYKCBYEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3v7ztYwYKCBYEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524735; a=rsa-sha256; cv=none; b=RdM5WaT1R6pe1Q2Qqw8f82yGk2hn/3ZQR3h3fCMwt/8+0S3qHn2wHpKm7yjFypSiFdnnWQ mkcTQC9HitbFD/sqWdXXNNOATEz5fFiKL5lxPQMArvsoyYr+7xg7W56QgSf/npdyqTKvkM oFZCJ/0U4fgbplyLTl56xsjmZMmjUS8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=hu+sMS9o; spf=pass (imf30.hostedemail.com: domain of 3v7ztYwYKCBYEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3v7ztYwYKCBYEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524735; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rll6a/VW4pEw1AhNBjbFoMRbi+XwYRGQoRwSBUD0bh4=; b=M1AR7A96fobkg0ywsqV6UemGJSjvsDq79sQtcrQ/Hrefmkqw3MmAvNqE0XhPISLjb+WuDk O9iWCBUJ5jx2nWYnD6ZZLAy5q5lT2eClpDEKEV2MnWMlSklqL8OE+8IXd98spOvFA02W8X BQnmZgwvJ16k+YTv70WqH4YhDtlVmIM= Received: by mail-yb1-f202.google.com with SMTP id n20-20020a25da14000000b008fa1d22bd55so840616ybf.21 for ; Wed, 15 Feb 2023 21:18:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rll6a/VW4pEw1AhNBjbFoMRbi+XwYRGQoRwSBUD0bh4=; b=hu+sMS9oSFHR1r7khxhf1UbXFcs2Z++5UsWdyoPEf98HLaKpCA/5NqLhTjKtFbNcAI bq4IC94z3LeKr+N8fZ27HaBeoB9eV7FJUK2PbLxLqtWEgkZI6C3sySZgRKxnS+2i/7yz RFGxaAOIl8opKirVI8bfAZAj2T7sVtyi7tZlmgl6KCk8dFxbKL8va87izUWyHNnCWm+i wFd5WvNjbROGmsVfKcufeNfiLR4xBKkikebAtyhw//UrLzCCkecCcI/c1WZAFUB1nZ5g ONtU8uW/bb5j1wyQf5MWCYTwREIrfwMyez4h+y24cgMUk3OrfInLJF7DZne80Aw4Hh2T oQOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rll6a/VW4pEw1AhNBjbFoMRbi+XwYRGQoRwSBUD0bh4=; b=qyqK37WHagZn82q2GYDwTG7PvL9uW25bXwGvvp8LX+Q98mXO6d8tFZ6a7/30srG5T+ 9ksQRWFITde77a6SoPBc+WsY7bmTQpu7X67MlQx2Tfc3TsrIxcUqZBZzRc4Svuvg/HqS 2vr7VE346wg5NJiNsH315EgCkA1/46cgG4iD1aT++5JyPokx4vLTYBrZDNSlOSFrVO5p UkrWL0lHUzpTi66PqTmDmuxzrIhjd7WmOq6vjEFhEL1qvvIJAHOjPF/E1mwhG4We+cAv +9Z3jQyOsacqKsFYz/UIlkBKvSwu5z9hGofLrudUBZPZH7qbZCg0BR62ADat0BzVxAqV BwqA== X-Gm-Message-State: AO0yUKVI4EbVK8WSPanhuL8NCRPRaaT4koiObXlFzH4TqLBo/PFQb+sb msb0P8yFjfQSUCwPW2nYnc+iBpq0E5c= X-Google-Smtp-Source: AK7set/t52sA7P5kGXoASmYABiID5pIEjMuKzn7Z6yq6NOdnfI/iNhk3E99tS5Gm4PH95uk7/Rdn/fKqkhY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a25:9012:0:b0:874:380b:887e with SMTP id s18-20020a259012000000b00874380b887emr545965ybl.239.1676524735071; Wed, 15 Feb 2023 21:18:55 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:40 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-26-surenb@google.com> Subject: [PATCH v3 25/35] mm: introduce lock_vma_under_rcu to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Queue-Id: E52628001B X-Rspamd-Server: rspam01 X-Stat-Signature: jy43eqbwrbmdnppt6apaozygfnmwnjfe X-HE-Tag: 1676524735-475091 X-HE-Meta: U2FsdGVkX1+lP0M3SdfCo71L0h3J71RA9tsQ9CB+/1vlV5Qy8MWbI8O5eiHLg5/7B7eYBCWSOIgUrdJaEQjUYWkGho+moeKiCD1t4ZJOlhI+1Ii/9hUGulaCHMGgvXun7x4wxL/do7u0wztOwDGJIjSx5iYB7vSszRaE8vPpWE6V9wV5lNFqP6d2Tizfq80pGCnlLdizgWuHU/LLaHLueH3C/z81m8hRh7QyqxHSVsvaCJARyOJB695j0DxM+DA+Lof/tN8/547agNEvheYOIFkdTj8c4oHIdHymFM6atqaTvJOdpNCU7wtKAHRyRvb3V7dcWMM+dr6DpEYW9nmIIwQmOyrQyWPNxhWGg/CHXeYv7z8Y8pnRE6zY5xnNtifCtYb6t+AvBQIXJjzDRKSieZPmGxBAo7vBGugD9bHR/NExrsOAMCE7xCUvRb4lq9qzv+Sy1XZvtz7kO4zxK+cJkf/1u+6v2hPA3d5fThJCepXqWDbU+M7ZPsboKUucfdY2icSw9/rma4A8WBLefh7fIojSkPSXzcI05C+AxypmieEQuogYakcEmGlw6nS+6EKzEyDkfNEhFrFHh+wT9KCgLJR8WGZtBVJe81HlTznqqK4MxR/Yu3roBLRNRAALvbUBnxSfgFoptesrh8JNIlBs7fTLqWMNVPpiio2KuTZq3CKvJYK6bO2KOARmQCMRtOnI/srVby+/fW0gzzQ7CpcvISb9k4osEnAMupEPB4zmlHhnVggFn+Xof8oNdTE1FWDPklPNyg08sDJAUt1Bd/hre79XaH6rRsxDI3YLF3rW1LQDx9GnkxbG+FcP//ok3ixVf6bWbkoWok4gl/f1THfjfRSceFYXH7sIr/G/B5QasnPH75aou7J5Mshd8mY+7MTNFBY8SE7J5R6q7BSNuwLkqQle5xJjjTvkIWTCq3iTwFyciq/RKtzzUbiSh7pGvpqikBtMxB/jejSzf8Z1ZIL YOQQXgDd /+XLgdi7eW322+PV8zPOsNG59xRFWbWlTvN+JX8eED2P8PenNbKjyeffXY2POQmWNLaplx1N6HUf2/t1SLsmF5nzjxcy7M+MIupcJup/bD3M0dfN1YarWnVKlgeT1x7BNsiw7U5GWZSon82y8GlmmkA/RIfA2dy6dtctqu/9NBgH0U1pCbonCxl+p3cnYW/uW49PUYlfVzbAOjy3A/wFnaxtZFLQy5xA3phMTmQQ4LMDbP09zUyJsE7HbmfENubCWAFFY/jy2pllfOijwIZFFu+ErCFK+v0cyUohWYpBqKuoTxT2fKE2TtybrISo5zPVWswShv54kcuNTf4P2NK48GK2oH44obkPMhbSuPBeENoNggoj3lq24otZhF7Dj78nE20b6NtS6CNAlOt2/eIkx7QErY5WWJvXsfwPBNH5vaoQ2mAL+psnjJLqPqA4+rySjZgJ+Uz8wSMVcVQHLYygCl537VJw0TIa0McsqUHRV6TA4T0h2EvtvEIModo87tIwrHq4MnTMKs/QAjLvzkIGqNwkg8vrcu1cxPxuijWmJnJ0vQaGtmXvbdXS1kXUCEyLqFLAS20pe9AiQbkw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce lock_vma_under_rcu function to lookup and lock a VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. For now only anonymous VMAs can be searched this way. In other cases the function returns NULL. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3f98344f829c..36172bb38e6b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -701,6 +701,9 @@ static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) vma->detached = detached; } +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 8177c59ffd2d..5e1c124552a1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5220,6 +5220,52 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +/* + * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be + * stable and not isolated. If the VMA is not found or is being modified the + * function returns NULL. + */ +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + MA_STATE(mas, &mm->mm_mt, address, address); + struct vm_area_struct *vma; + + rcu_read_lock(); +retry: + vma = mas_walk(&mas); + if (!vma) + goto inval; + + /* Only anonymous vmas are supported for now */ + if (!vma_is_anonymous(vma)) + goto inval; + + if (!vma_start_read(vma)) + goto inval; + + /* Check since vm_start/vm_end might change before we lock the VMA */ + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { + vma_end_read(vma); + goto inval; + } + + /* Check if the VMA got isolated after we found it */ + if (vma->detached) { + vma_end_read(vma); + /* The area was replaced with another one */ + goto retry; + } + + rcu_read_unlock(); + return vma; +inval: + rcu_read_unlock(); + return NULL; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. From patchwork Thu Feb 16 05:17:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142519 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE3A5C636D7 for ; Thu, 16 Feb 2023 05:19:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A0966B00A4; Thu, 16 Feb 2023 00:19:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 402D16B00A5; Thu, 16 Feb 2023 00:19:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27DC96B00A6; Thu, 16 Feb 2023 00:19:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0D9D36B00A4 for ; Thu, 16 Feb 2023 00:19:00 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E4A6616083B for ; Thu, 16 Feb 2023 05:18:59 +0000 (UTC) X-FDA: 80472000798.22.4B90BCB Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf13.hostedemail.com (Postfix) with ESMTP id 3C93520008 for ; Thu, 16 Feb 2023 05:18:58 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FCDmrKEx; spf=pass (imf13.hostedemail.com: domain of 3wbztYwYKCBgGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3wbztYwYKCBgGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524738; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=76w8C0WtscQXhYR05yNYHvPFGtZWYYoigmdRBKNsAGs=; b=cdCgS7Lds2cEnn8c09b7B6nGApoAcVN2urZ9vEDrqXLcP3X6b9EJDJEnFtPnmZLxdWyIWW ucIq7OTuJ1sDjccTUNxDPxkZmOezIcJyN/9Ncep+Z8l8myT8/6zNDfwTmPufXVYKqAktBJ Ekl2p/pJFcGAI+tsWwZOQNzjA+3bZl0= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FCDmrKEx; spf=pass (imf13.hostedemail.com: domain of 3wbztYwYKCBgGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3wbztYwYKCBgGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524738; a=rsa-sha256; cv=none; b=tUZyIEr3cZy8AfEAZRDq/1NFrSVG3CQpTvyG7HoKvxNzsPr39JDyQ1BIoM2NRpM/NE2slN j4Czq5t9A2qiHPWhZg5w4NMTn+irJqJNE0+gc7dwC0oDXuLKARJYsNDn//qoAFE580r8aL h7RFxQ1REMI0g7AjAKch/mHxhQINJPw= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-507aac99fdfso8559787b3.11 for ; Wed, 15 Feb 2023 21:18:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=76w8C0WtscQXhYR05yNYHvPFGtZWYYoigmdRBKNsAGs=; b=FCDmrKExzLuCArz312mMsT1GG2XBAJpY0IRk4Dudkq8tacxOtpStMHAB4mCqF7cHzr HxmYKbJzh0e/OlcI5tze/pJirqiUzOseaOUlDEtDH80qm7FP3CxiVmnvUlVVgBXp61ej ykGRdx137mYbxOiW4ntwVshjUhNYXqvxbpsd7tR4+U5K8xmf3z6sAC5BmIr5igjBFbE2 Lhh7YXKnyKDhial8w7C+wSOyLz2qDj3wAEjKVPIMU3N4BzLNr3q2F7Jzk8qKKMPeJp0i mhDW6Px8X9de0wrc9+taZBLP7xnnbG62Qe2RXE2u3O0X+3PidgLM2eknvp7ECPYQAyqI 26OQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=76w8C0WtscQXhYR05yNYHvPFGtZWYYoigmdRBKNsAGs=; b=5WjqietA4ZynmtHK2awOZYi3Efmmk3VQFNY7M4ZP5h6Ndm8VQXlDI1e2BvPAN8EMgW ovTFXyITT8J4GIFJojNwtyYh7O4oN1c61ZEVUG2U8tTTmi6Rm9Ven+icqgvaYkGaa522 d7wxUcN/2YrAeEYCOSuFCiPx9TGr8oBJaR8/IR1cJXyrWsxDZ/BtJvG1MhI0u/nPtzSU QzI0tu65bQKONRxulUS0VgtwNM0JFODHTC5zL8Ts/b/nT3xtOgCCAn5RlFcWFbTMu9Zq +bApslMVTF3a74wwsC//APd1wgBimxQAKRM9cAPOYCa+G85uD/3L5I8UvlFlngvIpuf3 lPhQ== X-Gm-Message-State: AO0yUKV3wgftWWk2/6901wKXO10m+BLO4du9K/mIAhleBU2Kb3zCZP/t +bmsVf3CnBEp8MJ8YTo9hTj4gxDVwaw= X-Google-Smtp-Source: AK7set/7Y/K8KWVB4K6J3b/XMgTvXO7u2Z/Kc+o++p3RcwHAcWZWf0oYrOVDXbq6GKc7ezqD+HuFayT02ok= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:9e0a:0:b0:532:a8a0:8d76 with SMTP id m10-20020a819e0a000000b00532a8a08d76mr235300ywj.85.1676524737340; Wed, 15 Feb 2023 21:18:57 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:41 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-27-surenb@google.com> Subject: [PATCH v3 26/35] mm: fall back to mmap_lock if vma->anon_vma is not yet set From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: ecnnjyddueckiy7a6qix3fhqrncjkcf3 X-Rspamd-Queue-Id: 3C93520008 X-HE-Tag: 1676524737-734853 X-HE-Meta: U2FsdGVkX1/Nzsp96aqwxM5TdW6e/NkPPThw3T4Cn/cuN5h8BW5cUJUMEkL0T/D6UQKXlWz45LvSoKQ0culHHKpX8QOl5H4boe3kyg+QbY4j4OOz5ZyBmgaL1zmo9M2sTcLlGwzYxIGeOhfUs0+MDs1wvrf4Iwxq5c1kbwgiO7FKle+Xc3AGihygWgxxvVEblzmT+NWH6LybCIGh6rwq78N5RFLn59Gt+DIlO5QQavb2d1kf+wtSa5UlbvMmCT9R6gCud2WKSWlqhvd+lY1GPjVfY/KdjB+c6/HB7fRw08FPhJZWgvO7Fk9asiKrmmnqORTiwNJ5zIbZuMT/G2qchCEvhNp3u1mwZfzUqqBWXowOm4kX29ylf1GvMN5luHsbpJGEL06zaGYItHeD654pTLnp57Iw/Xyd5nQu7wW1c9Fu6/ufrvnmm6VEBVo7V1i0ha7eQbLLE1OsmeME4IrsF+T9WgELCHm61f6gRQo44fDS1SLU+WV6kxo8hpfBLEGLJ+axvrpd/zAsC9SNPG5KOuFM8ilfaWLCnfc5GfKm0Wfpu8K6ZRDKjY/xoiofoY3PZO6mIPVsUjnJLf8eWQAl7g5HGPJQ/MSqimtj0TFP+o4llINOhWgxkM1+rCnrcUnXDKcZrOuWnwHevt6QeqseOEOVdu2Wad438o5wGlH1J+Ij3ZE6tN1UWznaXWo+PKA0VSUUMDDgya7Dwp4fuwuaMorTa9OpQCfQf5uBYLK5a/JkXkxa5vUy40DgP8iWuLoPwA12Dhsc6fpiRkECEg85b4MUrLYt1hP/Ahr995fPrrQd9p1E20XUspQ5Ay4pj1hTOdnQYH4JesgcZqzMbyyPSm+rFJxzphoN4GFoe8bvaSWDlEhCc3okLRFSE+eHMi+rHPwh5GtoY1c4StOp3jbWNKhf/0+egRLetHhyteI+vzKvZJ+54PlOulONgzscyBHrdMUHD56TXG1t2TEUAVb Wlct7xHu C37B8knC27GfZkazqD1wPVpzoD1BpOPuSvovxxmzA00FaJqPYGnG8EVUjkYScI41bJhP48NadLvFgfKMrVnu4OJuOvxJa2gPnyGaKJtsPWHjG9blSxDowQWcd/ZuQBRAFllOAyDW17Wrs/FZiiFKMfOYeNzy1E2h2/J65xUAGYbG2/n88lGydtMPMrHYt43n0J1ukiDJllS3zD6i5NjMI0ZCYTjjy9AkUpcem1pKVss5tAnxI6pdTZwYxkZ0skcyR9nv/8volIYT7/d2VdpRfkyKGHmiEuk0DBNlc6n44UUcfYC+aAqfu/5Iwzg3ohui1+tEG2yEluFuDo3pqJFCO3ad2S4GTHFj7uBVDWNqjBgg9G+B+s2LXPj390yZhmxTXaO+0COl1iD3AIT8zDYR/ITzG9BvMqNf/zDF8sSHLLnRDU59U1ivRZ7KGK2Zer2rc+UuMKlof5e3ZAb5CfQPHLqKHTlh+gZWBQSXo4UIUA5fli/evXmWQTb/fBtITtGAftjHrqrNd3/qQF1Faj1fCOJdnSQ2rdmP3Kell3k2IcoGr5d3Zumgfl9zpOCP/uEb94D/I3qfziT/UDBSxVuaoRhC7L7vhSOE/ooVgzttjXnm3MPi48kW79rKglMzmo2+MVv/ttCsT+/Rm4MRhDfGndCYc/6aCCKfLzkEYAXyCbz5muvw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When vma->anon_vma is not set, page fault handler will set it by either reusing anon_vma of an adjacent VMA if VMAs are compatible or by allocating a new one. find_mergeable_anon_vma() walks VMA tree to find a compatible adjacent VMA and that requires not only the faulting VMA to be stable but also the tree structure and other VMAs inside that tree. Therefore locking just the faulting VMA is not enough for this search. Fall back to taking mmap_lock when vma->anon_vma is not set. This situation happens only on the first page fault and should not affect overall performance. Signed-off-by: Suren Baghdasaryan --- mm/memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 5e1c124552a1..13369ff15ec1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5242,6 +5242,10 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_is_anonymous(vma)) goto inval; + /* find_mergeable_anon_vma uses adjacent vmas which are not locked */ + if (!vma->anon_vma) + goto inval; + if (!vma_start_read(vma)) goto inval; From patchwork Thu Feb 16 05:17:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142520 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0B65C6379F for ; Thu, 16 Feb 2023 05:19:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A1D16B00A6; Thu, 16 Feb 2023 00:19:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3DC7E6B00A7; Thu, 16 Feb 2023 00:19:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 255D46B00A8; Thu, 16 Feb 2023 00:19:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id F3FB86B00A6 for ; Thu, 16 Feb 2023 00:19:01 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CE1A8C0264 for ; Thu, 16 Feb 2023 05:19:01 +0000 (UTC) X-FDA: 80472000882.09.2EA05F5 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf08.hostedemail.com (Postfix) with ESMTP id 1D2E9160008 for ; Thu, 16 Feb 2023 05:18:59 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GJ5BxW0Q; spf=pass (imf08.hostedemail.com: domain of 3w7ztYwYKCBoIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3w7ztYwYKCBoIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524740; a=rsa-sha256; cv=none; b=o04/E4ZSsEBZTbudt1LkNTOpqKmKDO+jtMu1FB8Ita2fHBpsIEYRvmLLvFuoxt/pRdoCfL SnWhGMK3UYaYq0PQYm2Zfs3qU6lOIdnIGFlsZ63PzWLZKShtmPiQAF9W22GvyIU8KXrk4R tQt8AeLMpLLsoVdKIfElxb8HaBwdRRU= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GJ5BxW0Q; spf=pass (imf08.hostedemail.com: domain of 3w7ztYwYKCBoIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3w7ztYwYKCBoIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524740; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g+2R9wHJik/ujG7pbEIR7UB7BFuMmi7T4GQjwfs+Slo=; b=nTm878GgaK+fPWUpv8UzSrEgLuMGAlIUSpyAMghi1LNEn8OKi2tUIspec7pA+kUcNdNKVR 9bqwkCUYTbEb8b0dApMHQ1FoKctgYxsjZ7QRzjk+n5APu6y/AZdm9ewyhWDdmZO4u8x2SZ 7+RwFeNydWqn6iU5QMfyTa+NXmMUQWI= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5337e2cbc58so8655187b3.10 for ; Wed, 15 Feb 2023 21:18:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=g+2R9wHJik/ujG7pbEIR7UB7BFuMmi7T4GQjwfs+Slo=; b=GJ5BxW0Q1u88snEvipBftLmc3+Jf1QItAPirJyrJgB8Mp9ylRIIALZlXmJQ4wNiSwP o865zgbeTpwGE0/miDmLSBWepWhJN06egqwo+YGBO2p834t1tPcrBoNtCdfrXD1jQzaN e7lrVFe77phFTZAotKwLYsW/LOb59LyIPFjYvDx4mPrMSRvD2iWSSQr/TeKwF3SManJP 7zOmcrgCnGJ5t0wPXfGvFFU5Wepw7LHd2B3wtwv+KMRgdeCQNTeRgpwm3UNl2CvWi/nE 3FOidEUJKnpNbe60Or5Um47yum4t5Cm2ARYxCb7pet6vv6oqpLv+fwdHpYVdfug+gwv9 oixA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=g+2R9wHJik/ujG7pbEIR7UB7BFuMmi7T4GQjwfs+Slo=; b=ywAh28McrlWe50r3zgXEap+Z/GIMV6CeTWA29dhMdc4O08SinEEx/lku0j1h+HpvK2 3/kwiePWQE41CXkaamAfmBKmMb+s+pPu6y0Ypk9SSWGc31lRPrTJ2C+2rYtsF15F+mq4 fdJbHWJYgOC5/DQ4qlahaV0ZtyPk3Z7RnpUIP9O+BgxkgCFPCrHZ4tw9NZIfc8wSm9h1 bvgBo36jITG5z245QXMc2PJPq+wBZs55kgho0p7zmkNfA/ST0hWwa6RhViVfdjmlI2WV boi3RziNy9rQJsVJ1txw+rYTU9uQtEUruV7CNsHXn1oD6Ri/6iK8DFBcs15ySgXq2soC Etfw== X-Gm-Message-State: AO0yUKW6uGcSkgb+C4L5vagjIrv0ffvq4lxfI2/moHrBFq0UH3b287aV 094x+XEmTecXqnaBk1Ihqvtq1a+o5xk= X-Google-Smtp-Source: AK7set/8zrN2bEgVNx1m2BtSP2crHaoN6LxODGu2trr1UY0Ec3uUEJoRM+mde4AC+sE3ZWg+Wi9w81iJ8kk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a5b:144:0:b0:8ec:cc9b:7333 with SMTP id c4-20020a5b0144000000b008eccc9b7333mr463396ybp.205.1676524739514; Wed, 15 Feb 2023 21:18:59 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:42 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-28-surenb@google.com> Subject: [PATCH v3 27/35] mm: add FAULT_FLAG_VMA_LOCK flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , Laurent Dufour X-Rspam-User: X-Rspamd-Queue-Id: 1D2E9160008 X-Rspamd-Server: rspam01 X-Stat-Signature: b3keky1omrf36ne913uz6iq7uuoqhxwm X-HE-Tag: 1676524739-520312 X-HE-Meta: U2FsdGVkX1+DIz8K9owbm0YWBSlo+1curq5IVE1R4aa1Rky0SRqEBx1Ex2G4yDPfFbplAR7i0l51J7kXYRCUcfA0tFQLG7ET4jjLBCanarZx44SeK6zKU62so9RZuEHYmUegbcO7/YOn2HbqKJLtDBf7VcmmjU0S0QQ3iRKPeJQz6xc9I/xlPs5L8GArEORGgNbL59bhfltvCEXP8Na11nR16dJtcbal2grzKDOWyRHVBMkuOlhEN88face0LQNTOT1KOe4aIRkpebr7OrhbCD2bI54iG+qTPY1Xwv5qGJez1oFgZaY7SDklvwitI8UaP+EsWkO7f3xVxpm/r9gd2KzztjGmq9Kdk14BEMKb+6g6UdFz4FErysThFH3lldZlgJfcU2KxDtiGcZbYR9oU2kiR5LF3TvSK0CViTNdKdNXaBS/PTP8Ic/DvAbhK1bZ5UALfnBFR+ayIWgxdOySpJEWz9/uhZyUWJQJVuKh1MRh2jRGqQJH/iUu4d6+EZhOdLbjfMboNzoCUONX5gbE0nnwit+E5aNwo234J9H0Y4Rmn8hhHHK7Z/cGKRkHAusblvQwQSc1kXB/mVQ81kMzU977B2xTDf6GMBiLDiczrRZmSR4DLrNN/cJbDKAQ3ShAtf71tg3Kwb65tCIBJ2lMvr7UpyioGGT0IKCBSt37ohEd6D6oZLCmYPLqNCz08VU9jr16xceCGH57TkjrvQZCv7LcEQlIXts5ovZ4QZwCi7WYgIwYi6z3ehFeIazrgBysfAQBbIZQ+M4XHj5v8srZ6v7bhpNc81S5al8ZX+NOB5yQOMUfoGCA/VMsKkMoivsYxzI43fm4COoKLh2i2B8iwcxzGUU9Sj+NEqNpTje8BjSBzfUyOqHs8NIZjshKVnZON8I1p1Bk1UNaEfBjtADO1Vi2M5TyzDeIWZ+jjz1xJ+leEqrmIELThzaTqBWtELXUjXgqsL1GOLKJR9EgXeBZ SV2Mu2Xm ivgVdQMXkpSddJVhPA8MndvrTS1sCwIIQAS4WTwyEfLXTllBhOMJuWadGwVzLdcm934XM9s9dTr5hS+ksOrBAiFD6EpFYpTIDyHLzezYXsGQZau5klFGKzJCh2qdZIkVHVaDgJEf9eIIDArHjSqzY2YX4MdmZcnfdFpxq30nMOrCI6OfcItjOtMS352eSvpZdQT7ndKD9xnuO1GeXY/73QtcbiPHLmqQ+8NxbeiUnWAtAk+vAUhFbgEu0nEJn7PaXUw+gC+YchvKarOOoMze/qcXu9eCNaluqrxHY3IpP3ZFDLEZFB3/iQIQkVN6i+UcfefFL2qcgcQis8Nae2WvKtAMJeNZnPYwYEgN8rCvKSLF++Iej8cVMbmHniTJtyCT4iyx+U9jE7BJ3MM1IAD5QXfZlGBOePWvqf3qxmeu3KtHAyxEHPS0JMORztn/ZEC25tfynXXnhda8yYV0slwo5qf78MIihsof4E6E7z5CYh+9GT1W5LtKqUM1BZk1LYA8ipYeiYovUcn5f1IEY1u0PAvzGsiZ5wtTYaT4tdtXWp3249xYM1S5n8h+/gr45gyxwnsQRNOMXB0M3wPDSDgZ9hI9IJUU9aUmtgChkOdA3QML45yVY9kiYzQ6xlgoetutHiYUwqzB/JKCdfHekkLpHfQP6MlwEyM4qtxtHdqLtj9ounI97gUwJdMjnQrodWZrOJLpR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000011, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 36172bb38e6b..3c9167529417 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -478,7 +478,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_VMA_LOCK, "VMA_LOCK" } /* * vm_fault is filled by the pagefault handler and passed to the vma's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 939f4f5a1115..212e7f923a69 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1056,6 +1056,7 @@ enum fault_flag { FAULT_FLAG_INTERRUPTIBLE = 1 << 9, FAULT_FLAG_UNSHARE = 1 << 10, FAULT_FLAG_ORIG_PTE_VALID = 1 << 11, + FAULT_FLAG_VMA_LOCK = 1 << 12, }; typedef unsigned int __bitwise zap_flags_t; From patchwork Thu Feb 16 05:17:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E444C636CC for ; Thu, 16 Feb 2023 05:19:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A930C6B00A9; Thu, 16 Feb 2023 00:19:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A1C3E6B00AA; Thu, 16 Feb 2023 00:19:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C6C76B00AB; Thu, 16 Feb 2023 00:19:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 72CAF6B00A9 for ; Thu, 16 Feb 2023 00:19:04 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 505F0AAEE3 for ; Thu, 16 Feb 2023 05:19:04 +0000 (UTC) X-FDA: 80472001008.28.329C45D Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf19.hostedemail.com (Postfix) with ESMTP id 9723C1A0019 for ; Thu, 16 Feb 2023 05:19:02 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aDErRd7k; spf=pass (imf19.hostedemail.com: domain of 3xbztYwYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3xbztYwYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524742; a=rsa-sha256; cv=none; b=JAYxtZsdDu/1i+QDo6g74AN1I5yeTXE6qD5j8RwOHwcnivMIGxA1NDZ8ynKE86V62GjjLp wm4Zyk6cxS5BlelT/lJt6kbmgj62hFXvxlA2jq1vnmyXTItVXNde21NZgABtwLXwd0bA56 KMxKRj3QfaawG4W6L4oMUb/F0MQQzX4= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aDErRd7k; spf=pass (imf19.hostedemail.com: domain of 3xbztYwYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3xbztYwYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524742; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+haucx/0aUjNKP4B5JYWnKqFhX48IZcct98M2E1/jbw=; b=CRfbf0GH93yMo0QQ6dFw2O2CZAtSx53OsUIOP5vEWsmz6KP9v42Ulq/R46p/oN0zLwyCRv oAEZDUTdGW7SGSD4yTJbTSMNr9ydWZToKDdxM+F92TX6QFvWUKj58NSsf1udgRu1n2hANI mEPWdp9CjYyd7PQiuHkW2GU08B3UI00= Received: by mail-yb1-f202.google.com with SMTP id a1-20020a258701000000b008e1de4c1e7dso848908ybl.17 for ; Wed, 15 Feb 2023 21:19:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+haucx/0aUjNKP4B5JYWnKqFhX48IZcct98M2E1/jbw=; b=aDErRd7kkY+Kp4AaU6gjM3F503YHpBTPjjocmaNoc4zXVCAuMI2RAQKjzcRAbLtJge uySjaOCoHcGS53W7o3RYdOQoeaAfeHvH08PuG1fh1bxXWOGZDVGDBlzIfGuJz8kR8m0P 4LCu9IW21ZFA5+XCq7hXP9c4LIZtK/mDOJpY3r8EhzjVEo7Ti1XlFspuAz3gf99Vo08U GQhyt4xy4/c9ROtLln2G8n+ixCeeiCqz1U0tO8VqF1k+R16fVXyhOw17lx8zHct64i1Z f89mO/K5qgtX3vlh4Aty7U18axNicxnQBp8nhqlTIla4iaOnphrb06qKzWIf+oeLlfcz N8gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+haucx/0aUjNKP4B5JYWnKqFhX48IZcct98M2E1/jbw=; b=d9ANfJMnaGszjZaC7rhx56Ip0YAiKJwa5CQ59pTpSlkAzC3uptAMXH5N5ReZ7Svh1A JanLom0vm24OqthYoyxlt/URFCJsExRnE7p0rlKrFC77y+STAC4hQJtAVofapl+oMPTj fm8bYSuHDMsnmEjv7NgY3SMVPZQl4qHMuyWmTCOyTFe1GRmx/RWG2JP/usgsAlqiePZo wC3S0dHgni5UOV2VzpI+AZXo40/d4bKQOeMPXSTzQb+Q8rY/0PGyvriGKwzVHgzEh4ax Y47JLGHFKnMaiFqAQHGbxl8NvxHAqeGVEe0TqCSTV7GMM5LmCcFwPxRm4U3+EPkojUZb /zUQ== X-Gm-Message-State: AO0yUKUvz9/1pd1RCmqPp4bJoHClh1+UGkqWkViaDPJyvNHthUDabTPH BijmkHFzRDD5o6DsvDBAWumah/fYtSY= X-Google-Smtp-Source: AK7set9hR8JRk82RlKZHeYCMxCYi1ok7+igo70HcdJvHzw/Nn50rv4GE5Ev7Sgo0D41w7o23DTD5b6roJhQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a5b:c41:0:b0:91d:ddda:665f with SMTP id d1-20020a5b0c41000000b0091dddda665fmr396601ybr.589.1676524741694; Wed, 15 Feb 2023 21:19:01 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:43 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-29-surenb@google.com> Subject: [PATCH v3 28/35] mm: prevent do_swap_page from handling page faults under VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , Laurent Dufour X-Rspam-User: X-Rspamd-Queue-Id: 9723C1A0019 X-Rspamd-Server: rspam01 X-Stat-Signature: jnyt6fn43gfmda5ns1x9it5wkdee4qoz X-HE-Tag: 1676524742-715039 X-HE-Meta: U2FsdGVkX1/WS27gKsYd6Tg3xMY5KFLDup3NnzM23j2FiHBOxYvpfNba+UGUX7Oz7aH+7b6JZrBC6eZupkGg7XB1c+h8fO8tbmYTKk1Mrvhsm79jMSa+gkDTklWtVsZeXQ4m4O4xtay2J3keofG/5rLqGzztnwN2/8OMszFd3RBQV3VapF22tyJNB9WvOXMJOKztrkcdmKBAYqBUWpJyAv3vCyhG7ptLeIKTMNMTi6z4b/p+IR7VxrheYwPRbP4+Gv7u4ykPVoK7NNRyqzxJutV6AHsZer+Tcs20FWl1BhjL8GOIj37icRPGPzsxT6WA0+06GtbO1QeSbrtFgfIo6+3xlM885hx9ZA5qq79rwM8xq2bz4SL9nXMXctmJSF9Dn6KoDCuL4YaWjfxGV3HqHsQYiaxGDYEctg+yAT2qUQ4FreK5CuiRlwM438Uu9m1c+2M/zKMLF1W1CKUjRybDIcUAVjxMiY3B3Rdik9yr++W25nhZz2um1rwo29hDaDkt/9Ll9HLYFHNnpjPvoPiOkvJgc56AwtDu461crPLo/JHHf/pEEDjWgP6kC6XCNr8lp+qosQ6G4AHepJrKMt2BCDKDmipgK9TcJIgPrmAYs32qIUDozkfusVHOOISwOMB88xiL8i1YIzdZuuerDNqzsRL+sPqfylEyta+Z9wvmipz5GcoSP09JoLUoL6UnbMzmfqjDQtrSU12g5LU84S+jQO3HHDnxMoFrPrTYEjQcWdQ/TctbwZ9+HKVA1qurjKqhGK6iw2ArchERx2YO29lXnEsxkWpIMvAeOdF56K+rXp9p2yThWpJAyYywbMZv3akkjLklcWJH6S4q2Q0LTh+QylMOKfyRa9VgteOdcyXxAbvytOb7hXvqhIx+tZ7X6nOPCNU/GKg9+M874asPDJgQOdjnAV2bNk58Er9dsZnz0fT6fNzLIfqVuJCdaVOzZT2hfY7DCpJEWJ5o6H9o3fY LTbUHfOw /O/mULbB+TsZXpSQ2Q4Y6OYzNw9BXvl25kWcj+68wJyoj1D0FAsqjyIcMzp4lCHFJNXZTvuNIPKSDf22tciL21w45zstQBz3Quz7NsiwoCuLqxGZNNQExTO7mlhLdx/cXbGOAf2Zn8pNSX3t8szBVdHF4KG3LFTIAejaspuoypnjKm4ycRAB+sZK1vZCPK9Zb8Oevkpo+y5vig/1YxKRTEtcjbWcLg546PpvSdFZay3Tn4C2wrgZ5CQROTvyZ2OthSgYMe6F0YAQD+PX13vhYboTMEee4RYV4/NLwe1wyWy61pl0nBuLOisLJMt27CLdo54hnF6sG0uYhbL9V89gC+KgANFAqDh+/3+Ycrpq0CPJPhnHSbW5GqcW+v81g/qPOKql1tqN4reApua9JWQdlGc7LzQBeBfEh5D4YXR7okLNJ+5ea4Db6FOe4qZh8lulwUVE09PzuP3cw/vNtjE2NQbr0ekNKyj9FJhET2+JnQrBEXe6UlGgcBN3sv14eYLh7aKPeL7GRRBrjzG/KxGXFIH5KfRNZwbazAnvYJZaECdMPmIMnsBaFgSApdkbOy05TsZ7dNuMN/rQ2+tOYpsaqwJt5h1/gjYOOimrMApbbzGhLqmM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 13369ff15ec1..555612d153ad 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3688,6 +3688,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!pte_unmap_same(vmf)) goto out; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + ret = VM_FAULT_RETRY; + goto out; + } + entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (is_migration_entry(entry)) { From patchwork Thu Feb 16 05:17:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142522 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8536FC636D7 for ; Thu, 16 Feb 2023 05:19:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B9B56B0078; Thu, 16 Feb 2023 00:19:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 169AD6B00AB; Thu, 16 Feb 2023 00:19:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EFE426B00AC; Thu, 16 Feb 2023 00:19:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D9BAE6B00AA for ; Thu, 16 Feb 2023 00:19:06 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id BD8E214108E for ; Thu, 16 Feb 2023 05:19:06 +0000 (UTC) X-FDA: 80472001092.09.D72D1A3 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf23.hostedemail.com (Postfix) with ESMTP id 130B4140006 for ; Thu, 16 Feb 2023 05:19:04 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=sqrDU+BC; spf=pass (imf23.hostedemail.com: domain of 3yLztYwYKCB8NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3yLztYwYKCB8NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524745; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2G5AxmmOVrsRnXlCrW6/hvh5FUxSjD+0mS6LSWzYfho=; b=vhZP/K31DCApeMaZzHGTj4FRP2p9m4dS1jTKb/zp6M1RpnnFq0c3Iyve+jfV7LnUlxnWhJ /Miy3pMbtbVF3togZOG4jboVTZtihrdvpN46g9vLNau56iSdx52CPhbXoikHNJGUOyOChW PeeXfnp2V2KXQCxROxwZIL48xAXuJdg= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=sqrDU+BC; spf=pass (imf23.hostedemail.com: domain of 3yLztYwYKCB8NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3yLztYwYKCB8NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524745; a=rsa-sha256; cv=none; b=vmxhQslFBg1meuq91nH1p7eKa1s3Oi3tp7D/KbDoO3YJPRj3tIe6JoTan+S+TOHeS6T7wz LcRma6QUnmbzRh8bs7l7NN3qPRjPnUAt5r9v+tzi3DFQkoQTO1nPi6K/ldZYv7ENsp7l2k yhJFMMnB+pMLxDu8rceZocB0m4H2QmU= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-52f233aff21so8610067b3.7 for ; Wed, 15 Feb 2023 21:19:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2G5AxmmOVrsRnXlCrW6/hvh5FUxSjD+0mS6LSWzYfho=; b=sqrDU+BCO7AIuj2EKpC5JjJjchpfs4e7nICzmha43sQtLyfoiEZwLUOwsCD209H5Ui lu0l/EwT0lWrDRHYoglaYWRXNYPJ1g8O+GpgkaLO5UTCsF2sShD98xQNq7PDWtH66r6g ZsB+oua1JfA0KymwRm2/PgdgPfreEDDqnwQyHLX1ndmfndQgCQh8neVqDDo8QNluQIit psaYv9QythR+X2eyBbc1tAjHl/fZRS54lIuiB1Ux7odRjp5k2KlmRL+IIzajgpgWcCV1 agKdJl/mJ1QtI4UVDWAjmp+dEFlVMZYnML3DG7HfRey4EeaADk1hZvmemdIk3Yb/sCoK 4MXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2G5AxmmOVrsRnXlCrW6/hvh5FUxSjD+0mS6LSWzYfho=; b=Z+aMus8eghWmXTSp3n4UJXOL5G0Aapzg/g+7eteaFyIn4dxSj3rzra0DvRySYToyH4 IoUz3Mwc7ooQ/y98t2QcGssOAHy9QoIEghgUgt0KicSqvGx/jnOljc2iuBXeeVXfahLx gC1zs9bNnpY92qos61R7dkYMHBDQrhJ8wANCGdzFgOE/mDx9v6ryOFfcBGbalQeQd7y8 Ls2wFGWVXVOtOV876/I+OkbbRZdscQc9urvge78eQsm2Grpkn+rDCw6CA+iy55cMjb6X DUBhnShxkWcrKnIapChR0c3DLovgI1HIPI83GxxYZbNP+75MOw36LVK4kHGo7+2SU67I Nxrw== X-Gm-Message-State: AO0yUKVvVd7PtICQv4smo/hghR+uzo/S9MwMnC1ZCRHcRnf2Y+XkeBuF WpIh415U8rZt2DrdPYXDUTOkO4TYqF0= X-Google-Smtp-Source: AK7set/QuR9OxnmNolAau/sd0Kj8g/BkdB7enEU88ehNvfonJ32Y6Yzdt5dUleuzBy4yluRtxlQTgtsiJAU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:13c6:b0:8da:3163:224 with SMTP id y6-20020a05690213c600b008da31630224mr7ybu.0.1676524744013; Wed, 15 Feb 2023 21:19:04 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:44 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-30-surenb@google.com> Subject: [PATCH v3 29/35] mm: prevent userfaults to be handled under per-vma lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 130B4140006 X-Stat-Signature: nhkfzr1tn5oim315wa3yut91j65ac9f4 X-HE-Tag: 1676524744-757590 X-HE-Meta: U2FsdGVkX1+eITS6pEPWiR0ppfAA1C/ofkzo7cgPrvEK+IieaOzDAXJ7lytPXCPJcO7TAJDxeBF/WJgruY4Owx7XUuSAu6/sLwEZ/egiBLPCY7PiPc6gWazcZM+u7+NKUz0Zl1Kz1QAat+9rZE6oWVNjvAcP2XdjLbd7oxv8oM62Lvxu2tAVkW1/1ve6b2eDrO4XJWJE3HFgC/6M9T0GG6Dlku8qyOg/hVcq7jYTGls9+RMSuooEmGse1NtGSFrPoauTgJTaEStYAYk/9Lvq85MLkvGCPWTQctemZj8Zhfrd1S5p/vwdmfNIEkzLo5xlke/oPwNNHu4L6qGy2Yv1mkWIcfzBzVRh30yNWika11j6oG5AmhFoV5msHDjZXMQf8ekkJKvDLUnta0PpmWFfceduydtRXTnT0k/3dv9ZLVLb9NnXQnqoefxwpRz59qXeas/bAVU846W1hz86Rq2a5b8yGfU5KlLDd1gGHOenpqY6Mr/WmN+lvNsvV7LEvyVDZvpMjipcmuorLMx+3imnEwaGK7Z3JsdkgqNMKU9BeoyD+XoQlkdgOMNzKbb4GnH/ymonobvu7tx0H5maOubquF6rPVchb7P41lEzenMr3S6bLBdKNum0cQ7o6ZAm2PwzWPyFtl6akRnIV1j6PuTQgFjsoKDYncslYuWlMFEhymHpSZEu36b4tvHCbeI8mtk5go6AkAA3D0E5CBpof/oozvwxz59gOGdVWlrGmEqhLNYaAuWoRFtxdwmxmeWMMHkAYW2rE0netrbj0bykjKCdWyBGOC3SNYdkRKDvogpSBxy4YNfDR8Axa8MdLaOlj7fn+NJVm3oZJQ3ZqK25ewJE70UPRdP3xNABxbuNYMVAdY0AcdPiX5Q0lm5nXMKTQl9Oq2OxcK5IConeB+/I10vb2YtAQw1O/rIpCowvRwrHqFIcXIEKBOiFCjz1xYo2CZ27Yh5TYvYpWXy+URyPmBV MSn5qyq2 75Ryfb6ofKDaVaBj17ZuLepqOa8BymwRs9se/2iHGKBwphHlm/XsiAbHCwWFpKp9U5ODiLKwM2gf3IlCt80kl2kxYw9w+c/c6Wsrg4wQBSEhFXdYsMCFWr8BDgLJdaUR3mi2XcVhkKhY90Xj8L/dq71j8HRaO1il8s6TZroDUkpxpIwAl6JAqA0DbpvpCCVcCiwKMIERwLU9NnBDUHxMQN/bPblr6sqqJR1LCh/CmgAwzhS0F9B3y7d5mA9z4RMZ+KlbxZGdqSZPiXdMjxbW9x6EGBxky2nrXPqEeJemjj/Se9HN2PsRYX2uJC/28fkIUUjn3BzezhYLuaVFKoSNBKOmULWtfi49xLl+uRgAnIYWGJHGXtVNeoYRJJku2cqRvIR/6wT9NSt8UJuzmbPx/u51/UfkBgC0msRG86UbZtmiLjwmRIgre9HHSGIuHqP++ikeMjD8HgRLbrn+1/VkSJ2eS8esTJPrmCkCQfSb3zzQ8VnJJM9EsE42CJ8vc6DgpDnea8Nn0wfwpwvNwOEu/0q4f5mlaNaZ2CFXeTSnJTnB5qQHqjCKdX4jxuqqqGMkFtntcJKxn2oQo/NuHmZLDIqaKUhpa64krc0VF/q7l/VjiQ9k9stkVW80H3tXU6GGpmp3Su5ys7A27uWiB9MpPv12Dog== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Due to the possibility of handle_userfault dropping mmap_lock, avoid fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Suggested-by: Peter Xu --- mm/memory.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 555612d153ad..751aebc1b29f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5254,6 +5254,15 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_start_read(vma)) goto inval; + /* + * Due to the possibility of userfault handler dropping mmap_lock, avoid + * it for now and fall back to page fault handling under mmap_lock. + */ + if (userfaultfd_armed(vma)) { + vma_end_read(vma); + goto inval; + } + /* Check since vm_start/vm_end might change before we lock the VMA */ if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { vma_end_read(vma); From patchwork Thu Feb 16 05:17:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 126F7C61DA4 for ; Thu, 16 Feb 2023 05:19:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F6F06B007B; Thu, 16 Feb 2023 00:19:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 959046B00AC; Thu, 16 Feb 2023 00:19:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F9666B00AD; Thu, 16 Feb 2023 00:19:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 665496B00AB for ; Thu, 16 Feb 2023 00:19:09 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4333A121010 for ; Thu, 16 Feb 2023 05:19:09 +0000 (UTC) X-FDA: 80472001218.27.B92948C Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf27.hostedemail.com (Postfix) with ESMTP id 7AD4C40008 for ; Thu, 16 Feb 2023 05:19:07 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=a3wN6yhQ; spf=pass (imf27.hostedemail.com: domain of 3yrztYwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3yrztYwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524747; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nAnyYB3Nw50vBlFaI5emapvq5zyyV08CcfGYNR6eiBY=; b=yj2G5AuW+jMluraaq03OPKvPk86YSEpcRxNakJIXzeejhUr6jrlYa6O/BpEm8EB2mqpvig IdA1lbhfFQF4FnCPobF0G4depq7feGYRRboLE8ajLe86tCBAp7yfX4yjJf2ajptYbA6hM9 ssjT87I9C9s2Ie8rNb148uNSewwGELE= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=a3wN6yhQ; spf=pass (imf27.hostedemail.com: domain of 3yrztYwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3yrztYwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524747; a=rsa-sha256; cv=none; b=shQ3N0tqYEGoYMg8pLbTcKLmMNar1DUPUCz/6omUZXuwe+P8lIdXPDG/RgLwazaAsHGoJJ obERASAkPghUw8vHhpCWn6Up+qotT1Ihz55xdvXzxBvwo2U2MNaI4iFo0u0/+2Hd8+lDPv IELz5L/YYJjoUfmyyJTTFbDbtzsWQAI= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-4bdeb1bbeafso8708277b3.4 for ; Wed, 15 Feb 2023 21:19:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nAnyYB3Nw50vBlFaI5emapvq5zyyV08CcfGYNR6eiBY=; b=a3wN6yhQzn6ykzBbjResLcztZyUFHxuHfZtRwk9INETFTCK3R+0qznDveEPm+o/T6G wJ3wEiieeEUcjgJFPl0Q1WJhoSjHld23FZEQoY2Aq8Bhu2X6QB+bcNMvcqIxTt4gpYGo 62C0Ezu+RzKHJNHqe8a2WWwzq3BWBtjKwlH9vropbhb+eXqNxVp6cua3PyUJuPb2PSda pEBwm4EDV55dFWt5TRT7+9jNt1+V85COwPZnyzclXtLSHEwvb2nMc9rLXuPiU2TfsCi1 1h3mje1L/1Q/DM34hGjfe9SSEWf4rBhdCLw8Ngy6TcviKT73/jmv3Lf0iRWhfdK1hSLg WjQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nAnyYB3Nw50vBlFaI5emapvq5zyyV08CcfGYNR6eiBY=; b=24+r9PwAb8upiElXPYwXsxM3Sx5N5CUXLmbY2ETJHz9N+QYFJnsI8SgGvYRL9EqRkk msoKNrpSGpZuah+Wsh7w6VlfwlNAr78LkTAIMaRcLHemMxHDzaXQCyvXMlLb+NbPPw1M kRw8WcqXNJWTqZN0YzS+gVN8cNe4xENvo8peEQS53HzTSB6i98hwJ5TAMtKXr/Rj596I wEGV0YF6oqP7O3cP3Sy9EBt7TTdGVJcNPu9e/ydmflmoNsj86WVasEib3lbjKvnlqctH gADirxm+BBO9aeYM/6DhWnDxUgljBerCJIVZn83h31PcW/sM2HdAgWImLELdDbhw2uKu 6g1w== X-Gm-Message-State: AO0yUKVarJgXCbYqFBbIhKTuBweRThbjgkIwovDkVrhNAdx8kFpBWybK MLgQUVal4qgxFf0T4MvZdGxVzRtsntU= X-Google-Smtp-Source: AK7set9XJXC8LdbYcGXxryYVlcRlcBZQcF1FMhZd/N4oCcpN8tIB1OMR+nLJJI9Pn7NEIXJu3Q5fjvT0dR0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:18cc:b0:8a3:d147:280b with SMTP id ck12-20020a05690218cc00b008a3d147280bmr5ybb.3.1676524746144; Wed, 15 Feb 2023 21:19:06 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:45 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-31-surenb@google.com> Subject: [PATCH v3 30/35] mm: introduce per-VMA lock statistics From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 7AD4C40008 X-Rspam-User: X-Stat-Signature: e66pgekci5dfe6ozgkbao1todkecsr5x X-HE-Tag: 1676524747-745479 X-HE-Meta: U2FsdGVkX1+hblfPoE9HeQsNyfzipjm/TepYUuffMNw4KCrk1o7CXDBsi64+fzR9CJTXSoXxGm0tnSEZUtKXdDKbpD2feHeXhOajK2wEE0eGZldFjUdJef/u/aamn9PAwuCAajZOSfEIL2YQUk85gZ+Rssl8JCDrkpPaxLPIkRdqCw2DOYdFtzu/rNRp/itSftgGgxzN2DEGpFCrESrp1zx/D+oa/zACUJ2hqODn5z2o5nP8brDN87gtPAmwHDXX9RZwcZ0jSaVQfi2neDuudc13QFS0EY4nCE6kG1ygb+rTDZ5ycHtmdPL85RoqhqeX5HJNLwhWPhZzMrIONLLv5tplgOELgrpV4Cpq5aa7zneGRVPP40s+03/5EczUHpNe/TuvKEy0TFEZK8ibcTpolb+ocyY/2iePN5KFUCuEKxQxSTdvek54ID665coblehIvTNOS5iIjfmYDteP3kJnx/gnj+bCHZmz5FG8QHYGEejyWuVOYOTi79LNgxccTVo1rs9eYeKJGHCNHKgKlTmDzU4dzPo0Pcv9KkF3mOUgKzrxeWUyghz5CYbImdH/uYAmE7H+GvjA7AX+af8g4JBjCa9Qp1wOIz+Yi3sPwx9CKYPcwjO+IrhNqpkBTRPmkShkgje5vy0sxCzRsNRu71JD8WDB9EXpxGjfMn5eRt+LyAQ4odZCMAZaF00qK69h/8bXM7bPgi6SiM/Qqpl0YEGUfIkerWhnvUwspkuKuUKBgkWng+9WvTjLvWQr85g1hVfJ9l5zxi9QWN6NjjKQBCDftD9oi4MkIFTdCrahkVjPOclC1Spkm/bMlx+bT67ZGv96VNbu9IYmahcSEmIkoEDbJ3sTif3UFo1Phipi5AofWRYXWiFGa+5TtlYDY9hk/6WFCw9R142Vgg6A0wPp6EM/ch5oDNtlk2whl7BOCdnDOunjpuqNzFbvlr2/ftinrc9SdeqqueVgt/s70wNnAuR 4eo8UA1h vo8UiE4V62TAYr8zD41iFOUtjaPzPQViwYnfzO8j4Vqv2ehrrca9NXXv30Y7nIrUeklmE1hw2ZCk6HISPE5Zl0ZyrT5i41hUkckbZctsk64sP0RppBfigh0cnDipckOo8/sc166feYbge9R3BZNLe2We9Vdv7vIsqQM6OFPr3wd9/HqmUx9Kl2VbqsHDPf6erA8Z1r4p+DRSmsN1nJ9XfR6Ju3HkZsdRw+CwicGKqhKGEddAawooVgmPTVNaScLrVc1vVoIi7tqNhghhemycI3w9SD0pLIa02wchZsRgPlaGuQEpdjOnqiB+fEYhkOGtXq6ocWM2hEIc/+4PExOdl0yfq1u/U5M2KHHul1ainFZI/ANedeVyIZa+tS0HQyccVWD6haDf6/9cXhopkNbpF8UnJIO9KOEe5Grbce8h6DpyYRsSvXh7hbO6HckAAfZlSiTCrwJGov9TDo3XVcHIsbk8u04EIyLmP2OA2PyIt+yHV53D6yLH/FDg3OD2eqJSj2U4rUHPLi0s23uD3euSPrTLODzG0ql0LwAzGy9qOejh/wuNHaPkKTX5iEbn1mmvKSQ8ymf381vEcojW3WKQCvIoj8HTRFS4XRC5Rwns+xy7YoA0BJtOpI55LuA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++++++ include/linux/vmstat.h | 6 ++++++ mm/Kconfig.debug | 6 ++++++ mm/memory.c | 2 ++ mm/vmstat.c | 6 ++++++ 5 files changed, 26 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 7f5d1caf5890..8abfa1240040 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -149,6 +149,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, DIRECT_MAP_LEVEL3_SPLIT, +#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + VMA_LOCK_SUCCESS, + VMA_LOCK_ABORT, + VMA_LOCK_RETRY, + VMA_LOCK_MISS, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 19cf5b6892ce..fed855bae6d8 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -125,6 +125,12 @@ static inline void vm_events_fold_cpu(int cpu) #define count_vm_tlb_events(x, y) do { (void)(y); } while (0) #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS +#define count_vm_vma_lock_event(x) count_vm_event(x) +#else +#define count_vm_vma_lock_event(x) do {} while (0) +#endif + #define __count_zid_vm_events(item, zid, delta) \ __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index c3547a373c9c..4965a7333a3f 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -279,3 +279,9 @@ config DEBUG_KMEMLEAK_AUTO_SCAN If unsure, say Y. +config PER_VMA_LOCK_STATS + bool "Statistics for per-vma locks" + depends on PER_VMA_LOCK + default y + help + Statistics for per-vma locks. diff --git a/mm/memory.c b/mm/memory.c index 751aebc1b29f..94194a45ffa7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5272,6 +5272,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, /* Check if the VMA got isolated after we found it */ if (vma->detached) { vma_end_read(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ goto retry; } @@ -5280,6 +5281,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, return vma; inval: rcu_read_unlock(); + count_vm_vma_lock_event(VMA_LOCK_ABORT); return NULL; } #endif /* CONFIG_PER_VMA_LOCK */ diff --git a/mm/vmstat.c b/mm/vmstat.c index 1ea6a5ce1c41..4f1089a1860e 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1399,6 +1399,12 @@ const char * const vmstat_text[] = { "direct_map_level2_splits", "direct_map_level3_splits", #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + "vma_lock_success", + "vma_lock_abort", + "vma_lock_retry", + "vma_lock_miss", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Thu Feb 16 05:17:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142524 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89EAEC636D7 for ; Thu, 16 Feb 2023 05:19:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 207976B00AB; Thu, 16 Feb 2023 00:19:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 16A1A6B00AC; Thu, 16 Feb 2023 00:19:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F25596B00AD; Thu, 16 Feb 2023 00:19:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DF5DC6B00AB for ; Thu, 16 Feb 2023 00:19:11 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9F65440D09 for ; Thu, 16 Feb 2023 05:19:11 +0000 (UTC) X-FDA: 80472001302.15.12AA9DD Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf28.hostedemail.com (Postfix) with ESMTP id DDEF9C000E for ; Thu, 16 Feb 2023 05:19:09 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="VYBN/Sx6"; spf=pass (imf28.hostedemail.com: domain of 3zLztYwYKCCMRTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3zLztYwYKCCMRTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524749; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IMepWjMdRmbly/mfPnmeELsKhzAKjn/e+MmdnnCshxo=; b=ziWgRfgcoAiSYiKlqBT6p0T4ghy0OCfGGGofG3NakyAqVkwN2LeqS1tsp/2O+pHNzO/ZwH tETWvfbvADHUQKKhH+ZVrB8/XaG4VcBuWzfJ1Ea5/UBkHD+TWWvZw7mXFUr8SLJkjee8bI Q7Xmp9o7BfUeP0CKWxpYn5zRCFVxLHg= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="VYBN/Sx6"; spf=pass (imf28.hostedemail.com: domain of 3zLztYwYKCCMRTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3zLztYwYKCCMRTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524749; a=rsa-sha256; cv=none; b=dlrT3JVlXEZEVD+yI4TYQAOdYIBWZCbma5b3cHhpZKea0SbWOWEpB9oooOYIooY7Ohm6lW lHvVFoGijzfsT5HnXzVk31NeraUlpCGsW9vWMWvvQWjHJhgdwmKiIKX5thEY66jgE8WwWs 3f9me7IiJO/j6Wk7EVvrTLBHsJb06Lg= Received: by mail-yb1-f201.google.com with SMTP id f187-20020a251fc4000000b0087f69905709so868942ybf.10 for ; Wed, 15 Feb 2023 21:19:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IMepWjMdRmbly/mfPnmeELsKhzAKjn/e+MmdnnCshxo=; b=VYBN/Sx6H1zkhlZQfZvD0Liynh1SzBUBQ9VOXzPL7TQljh9ke7q1U13nsJvyL8jhTT VJdOiIwIjVau+ArgVKuyZBGkikb4wPVpg6tKpQ0o6b7A2xVbgU2vETmFbJMmm/kSYK3W jjQeULRVDfrD7eqjNl25w/HYQSurUi+SuUv5YnVVfpqkT7+a9RL2gs3g5vQZ1Iey2WKK za8Zu9gxbyi43cwHcYzE3Uj7rIHDEJE8sD5soBil1CQae5YChuN7lRgwr3AT9vKkAkhX nNMop/PJ3OYQ+4sMUjA9lss931diz+voYeLd0iyVQtxE8G6e+SWaxHgo0zdZNYDoa0CR el+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IMepWjMdRmbly/mfPnmeELsKhzAKjn/e+MmdnnCshxo=; b=6Vad1CEBspTzcW0T21AHZwj69HBtl5/7IZcuk2bLi692s26OrwiLrrknPx+8vq+1un Oasar0MvOnj1vM0rdCx79coSj44qeIq6qHitHOmBkVvNbsIHutB/VG+Sb3yV5ZV/36gA 4pjm0h0fSfx3NRDJCLBb/bdELXW4p+PgwGJ3SOntiyHJqv+r1pCWMIpoeLhkEHQbk2MA 6NTt4d28RnK+0bSQU2zRx77NwSQ6j49ZFpyal+OgvPphYXopiZV4cwUUsQL3Pcn/POQk Xs4BPL5erkD8Rk+XqKhfCL0IQdIjte8PxX4NbXsXJoqYTKYy/q1OcAqKQ2DfUSoOqJun EEFA== X-Gm-Message-State: AO0yUKX6A/07LJkgbr3gGIpo1rnIKORf/+g7mbgJ6oqkuv+1v2mo6YvM xWEgwcZSWmIrGDyQzUz41BRqs6h6XjA= X-Google-Smtp-Source: AK7set9HCvy2lOclXRQNI1sLIIJkNYQ8CImwbHHTA3gwi641v9vkQcIdLDEBqeWHm5koRinED1uR5T68Tno= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:4516:0:b0:52e:e718:9641 with SMTP id s22-20020a814516000000b0052ee7189641mr16ywa.9.1676524748625; Wed, 15 Feb 2023 21:19:08 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:46 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-32-surenb@google.com> Subject: [PATCH v3 31/35] x86/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Queue-Id: DDEF9C000E X-Stat-Signature: qjd98wgjq6ptkc4xkho5yn9ewbb41soe X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1676524749-38054 X-HE-Meta: U2FsdGVkX18R8V6i36IRK59wtfl71ZlIaiDwAk3XHX1bv0HHts4QakpLFtp2Jweq93zDBAbCa4kVO3kF2m0eIhMUYUhnjdPY4ZH/mG+/co/nI/qZHonlZG1F9k0yU5SMZrCJGa3ke1mg6jwIGvS+7SOenRw0cKEBmBuExy0cIGK10zt878tWD05nM4MCr7kz+Qmt8YSkjes52vO9ylqDWLBSgLS3tiERqFkA2R+0FG/djWDeRhOQvuQR7VCwXaXGO1hY1Z49q0tM0fNHJUDMfsSgPV/cP+B/iAPGpGbnKjGejC4RsGLDARDX0deqJ68rD3whY98YLFNb24YF1uvxQYReHtozvDTji2qrfymJ4OIVnZmGNTPI1s9ehPYoiBAxqYROi/7u1bhcpNJpA/+aX0D/lopzKRH/c5duboDjtQOvzJvElUjqHwlwngcLC9w81rdSLt49Urik9g82q5hdvmG5/CDKQuxvEphccmfTwpE9+s/9Yl4p8ZPqE52beE/n+1d7ZPOnzjf25lNBkLjARLeL7F2/vy28SnjF5Yl+La/QmIZzhBrFRRCnXfV4FU44SApz07+h4+ifyWDow2aAD/EAOfjHWT4+czMJ4PZhYyycPWm/8M6UXbVR/S0yZdCxpHtVKb/3jQT1LJRzWtsHpDb96cs63RZH8QLkzSMGMJMF/rbkuwnVXF99YK+gYi17Vng/4QvVTebt/mgiFJPeNadvAEar2F72rRyPbO4EHupHnZNKLUqc8ojWPgcMFcOXk+sC77sxGf2eKVcsGVxEFsbPhU6WYq5kX95FjHrIkFmmjqGwW6C8cuixD4y7q5DpqC95BFB0YAj3rNSgRTNCVfRi4xh6N6KW9eu1IYIbKphyoFZjKCKE7CzfjMAOCKYQheYEvnyfC2uH+XfJVLU68LJ2j5Mz26G9CJ/T4cCMWQVNo+ulIaVxU6L0Ekp4UtXGWxnqKUqxI6axdEa+Wzc bijdRabn Bs2u2BEHzqutV0hktTBnra0ErGzmuIiJo0/WEabim7uOiPIoZUZnWRpwNgBEa5hlkjrVJWyXKGChLq+Cx8h8rmMvIXaYJ5Eo2tT56EsxJ3/ZY4DbWRwCyoRMLjVibb5OgAjjXqNnspbQsbMhsU3ZAUSUMBBj1Z62FVxuTlzsOQrRVOR8ArHr2lcXCukC5sUnn6ueTb4LOgDKgjMWl6RzUQOy0dPJi9xDsIbjb1vpRYATMN92bHYqCh72rlrTt5Qj7NSMhjDuyZgCCFLFPyGqzme42FqYiDArKNeufme8CdzB62fKf/9RQseORdEMbfkThh4uIzdzJrzonHVbojb18aMm9REShFWqywxBJC+OIlYP8nhF6QudOnsi+JSK7XmMlfbl6EZ9k/bzbD2C8WtZp9blH7nUARKBZKGWqM0zYzIEmCAYCNRwWK8p54Rdbb1wSEUmTHVTA1qMSMP5koa7jc+wicqZWQSqgLfKthIv4cEfST3bNMpEohF39dsKfFIq6ph9JSdLqUP7JAnjLggUat4uRCTJ2mfay6joRa1B6sI5/24BDhGpumGU+AFiNsdAOOiG1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig | 1 + arch/x86/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 3604074a878b..3647f7bdb110 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -27,6 +27,7 @@ config X86_64 # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index a498ae1fbe66..e4399983c50c 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -19,6 +19,7 @@ #include /* faulthandler_disabled() */ #include /* efi_crash_gracefully_on_page_fault()*/ #include +#include /* find_and_lock_vma() */ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1333,6 +1334,38 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER)) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_error(error_code, vma))) { + vma_end_read(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR, + ARCH_DEFAULT_PKEY); + return; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1433,6 +1466,9 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (likely(!(fault & VM_FAULT_ERROR))) return; From patchwork Thu Feb 16 05:17:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B900C61DA4 for ; Thu, 16 Feb 2023 05:19:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A8256B00AC; Thu, 16 Feb 2023 00:19:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 030D86B00AD; Thu, 16 Feb 2023 00:19:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DECD36B00AE; Thu, 16 Feb 2023 00:19:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C8DF06B00AC for ; Thu, 16 Feb 2023 00:19:14 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A424BA1132 for ; Thu, 16 Feb 2023 05:19:14 +0000 (UTC) X-FDA: 80472001428.18.4A2BDA2 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf30.hostedemail.com (Postfix) with ESMTP id E6C5080018 for ; Thu, 16 Feb 2023 05:19:12 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aft7yDPM; spf=pass (imf30.hostedemail.com: domain of 3z7ztYwYKCCYUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3z7ztYwYKCCYUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524752; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=REPTBg+i2LhaxaiXpF8lrF3nvI6zBhAH5JQxrJ1l/Gg=; b=PnYhyNS6ZjZTaz8YX7x55rUk4Kknqigg9qaPzMbjRxgPvl327WQQf6d98SmhYkQX8mMb8d nhlA0dZrunjft6lNi6RHQ+BTbRHYuraipW2yKBu/tu9YCi8QgJ1t3YYTAQjexTh32CrTPG ETioiGkZkWYcC9Q/7+g3mB2gnyMc8/c= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aft7yDPM; spf=pass (imf30.hostedemail.com: domain of 3z7ztYwYKCCYUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3z7ztYwYKCCYUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524752; a=rsa-sha256; cv=none; b=Znp1zxUdaPi6HZYZU8TP39LQTkKi1hNbyqVrY63nMT8xjtGeLChImTd38pQXc6/oHnmGGp kvwpTgY5sdYryJRhy5V9Od3nzZGrHIoAdA8e4pu7ZOzuBJQNG44Xu5pkUO9qGiV8BLNRnK e45tGxoIeJ6mYeyrOjE2YssOvg5VggU= Received: by mail-yb1-f201.google.com with SMTP id q127-20020a25d985000000b009362f0368aeso884010ybg.3 for ; Wed, 15 Feb 2023 21:19:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=REPTBg+i2LhaxaiXpF8lrF3nvI6zBhAH5JQxrJ1l/Gg=; b=aft7yDPM8GMTHB06MFPSYtKh6oWhZ8a8LMFo6PcEI9pnNLntVOlXFNQs/1Yvf52/5v q7Mb9j3IbqGydOQagHUnsSeX/sfiJUssC8Nuv1RApw1dHW5T8YzKas/LnJ1orI0tqnNj 0nSP5pSWAhACQCuzYc7axzYrfhmKcyRBbpYSaW4sNEVU5J76PVU7lF/u3PAFDwnMoS0w 449nI+g12YKzlYuLkXvI0hBRvnRJV+xrScZAqKlTKhgGwG9sA/5/r9CeoVq4yrFt/yH+ v8L5/t9gLYQIZy7UnUIKS96KONHOM4/bkOuXUpcfaqGPz0nskdK5CUeyDODt2k3AI1C9 6jaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=REPTBg+i2LhaxaiXpF8lrF3nvI6zBhAH5JQxrJ1l/Gg=; b=hzIEWkuE+KSZGeW0lyovp+kVpbYuXUY6EWR/+U+e+Kg0a462bLbbw8uJyuH/FeG/HC FsUtFhq/e2IFl+nlvVecvFIpMQP4B+M32uROVo4ggh6lkG37CIoavESyC7VGi6P51L2a EtwVayxbDBzxBFte7H9lR1856Q/JXnukBhUEZwBDa8kfi1HZ45/9Ln7O8v9nO9ZwW1N2 TrC9Ewt5HPCrijoq93R5fvGSp6fp2pYFePcMShlTzSF4EwFGANzbvJarsB3gTsYpgqZm ELXYWbXtPzTvbzKfrQ1B4n+DLB8z+52Vnr0YdKp4RxIZ2TYNCnvB3fBWksRFnBrPpPmL iX2w== X-Gm-Message-State: AO0yUKWtuCUCPwSKXzhTJNCzKFeeK+o8yb15ZA50crJLzNmyioTH1FcD bcBNq1cd9/10YriWjNLeFSCgJUE4CMg= X-Google-Smtp-Source: AK7set+v0FR2W9n0VmO9fC7d0HyR1AW4QGVl+/6/uhiM7Ww0w+Lt/AynlEHt7RfffNYQztU45jTBm0A6vi0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a5b:d49:0:b0:90e:37e9:1e01 with SMTP id f9-20020a5b0d49000000b0090e37e91e01mr1ybr.1.1676524751194; Wed, 15 Feb 2023 21:19:11 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:47 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-33-surenb@google.com> Subject: [PATCH v3 32/35] arm64/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: m6qhuy1i1u7uyzsjqtttwjbicpidwu6t X-Rspamd-Queue-Id: E6C5080018 X-HE-Tag: 1676524752-117143 X-HE-Meta: U2FsdGVkX1/zno6gOWwe+KSWfWLm5fBH4BY1Ghklz6HXZ95hA/pbdjW6R3NER6GZVFipRFxazB+PyLdf1Cs/Er2qbfwGbl7lk4+HJHHklq18jRVe9LldF/uC0guN1Yv1N8+82/PxbKQby8TR/sw/GXwy9n+reY6zRaFLhAbSpiUv6r4yMyegnGObsOX1ndRMFYCsiyTkm4QTL60P57yqASPdf5dalY94TOcoKCIs6/xy3qjPqjMHKgwzdYjEMYpeBJW2P/933Jkuz5e6sr6haR9BOEtdl43gnLzChHFZ+RNmLykakiD5ewSGDQvkOeO5yPmFux1O2uxQafdM2uyEi/H4qmrXAy4UuEiWZUXQwYhYpx3tugbdWzSrVR92z2gaLCFqsr3dEivaZYtHjc3QbzqjLy4XYVHvWkvJjdr4nh89Ebc+y4FtNQvSHyNSCMUnqVd/ariWZEO5bD8cZQFlc/2Vp1eHqYK8YkPXJFjvDhQx4Elp1oK+7aQ04VZkuO2/Lsb7ipLmLIproiQLgzqUpHYTq8lZNrTBUD6gWADCNtr4Ugn5AwDd60aorIsJAYLxKnlcjoEgY5nWJ1T3ow6SnGcn9bWKJRn5by6JflyAJfYGue6vRoWINctcuoU2/drlMBQCcsPhknUMAVAuIh4ztH5ahRer4eOGoao7w6Q5uZ91YEcxOSW7B7VmHlwG8uQWFPniBPGFHIkv3vO9yFCM8+McQdJOkD7nnoJ7KEjnwmXEKWV8nq03HINOR4c8BndKhY8aeolkGzjJCyil4KaNzX2cXonmqbAysfQOaoLWyQZYzLqkE81D4jhJwybtiHn/NJakHti7/pn12Vw2354N1AAGeDVwYelLlbAIyCk98xtcagXnBGRrzFyCpF5HBBN/pjH0yz2r4hho0qwWGYLe3Yb8gVm4GElA8RzpsoF7pY+7hH9d9O66+OsYiw+8p3rD/dtRZEi0GLwppzRmP6x LBx2KZNM dakMvv7dpBS0Egv0I1x6M5mKNNnej+Ex60hf+664Gi8esWZW7JqoY4pYNDaCRRvRABZg6NmyYufNreYzSknzSWwmEytEXshdL7vvfKnc85I8meOwWoDiZhIBK/uIUrFvm41rvZfvaeRTWpGSOesdQhEkLo1rEAjmhlpTe3e6yru6HPXPBFZ0y8PJX8TofZyVTE/u/dQG2t1RySBF0knlWHf27krGhTlv/thsH3B9vSiQaoOzd2LrnEXbIXGwo63BXq+5XnoEZ9Sigur81UbI63xm2OAiMe8CgpHboHdZTnQhmPqHMoHfZfa+ohjzbx8AIJxdexUQA9ZElO3rlVNYtFr7sQaUFf2y+EAwWElLu/l2YuP3hjsOV9zIRfM0oT+knY27usebTbYjnL3X7TzhvXpPkQjlz0bKuR8CJMYLuy1D+6G5ox2givsjP0J8zdWTgk+5/Ij3Cmtk7HF7NRfa0TmN4iqaaLzfYaebMXQDVD90hqFOOThRcfItyfpqNbEpwhChmMX5AIKFLmt8Zw5Yfu2yP4w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig | 1 + arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index c5ccca26a408..9f2c0e352da3 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -95,6 +95,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index f4cb0f85ccf4..9e0db5c387e3 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -535,6 +535,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); +#ifdef CONFIG_PER_VMA_LOCK + struct vm_area_struct *vma; +#endif if (kprobe_page_fault(regs, esr)) return 0; @@ -585,6 +588,36 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); +#ifdef CONFIG_PER_VMA_LOCK + if (!(mm_flags & FAULT_FLAG_USER)) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, addr); + if (!vma) + goto lock_mmap; + + if (!(vma->vm_flags & vm_flags)) { + vma_end_read(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -628,6 +661,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */ From patchwork Thu Feb 16 05:17:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142526 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9F65C61DA4 for ; Thu, 16 Feb 2023 05:19:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 760316B00AD; Thu, 16 Feb 2023 00:19:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E8226B00AE; Thu, 16 Feb 2023 00:19:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53A696B00AF; Thu, 16 Feb 2023 00:19:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3DF346B00AD for ; Thu, 16 Feb 2023 00:19:18 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3F458141007 for ; Thu, 16 Feb 2023 05:19:17 +0000 (UTC) X-FDA: 80472001554.07.0CF16D1 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf16.hostedemail.com (Postfix) with ESMTP id 75EE3180013 for ; Thu, 16 Feb 2023 05:19:15 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=KGDJpcNA; spf=pass (imf16.hostedemail.com: domain of 30rztYwYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=30rztYwYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524755; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UQ3ySmwJxDlPKOGcVlpRpfqxPlYDfL6UGC9Ax4HzzOw=; b=noAZhO6FjQV4bzpqTVqgBzp7lTSf1ioiRi76x59aw7LchIThyAPnnRkJVzsS8O6BHy9QCk UZruMYN7sXll+lFXXk5l3V6GHeqRN7pUp/AgKpgLq4rnb4UXk7zNKSiKdh7KjpAACPZxgf R3RZZ4+CnqqczjTdWZQ/7Alz7K8qr9A= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=KGDJpcNA; spf=pass (imf16.hostedemail.com: domain of 30rztYwYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=30rztYwYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524755; a=rsa-sha256; cv=none; b=GfS9uRJKfcpaKT52RTF+t79xfZXna11tUrapbnQh+yxgjWwAAMAl5h0HILklbW3hozK6SH Y7LZnyKdzgqJ5LJzS5XumKdY4qBGQXV98EguG+nOXEXw/YOW/xL558BNxhjKmpHmIDPdwK a+JVuYsoNCbWdCm3Dz7fsqAYICBc6DU= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-527501b56ffso8402517b3.15 for ; Wed, 15 Feb 2023 21:19:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UQ3ySmwJxDlPKOGcVlpRpfqxPlYDfL6UGC9Ax4HzzOw=; b=KGDJpcNA4oBtC5xQjxkPPkTW0kNBhMrSybgGLHrbBGSwSGEq+rySxMS3rk0dGM+Qo/ JpvUuwvGZIqjAVk+FVRsZ45D9AgvgQzyseylKUSLhPKRWFnmJExh3o/n5uArNAg9pZmc szoMYmExx5uq8bOtsbN9zvU5WJmapQpCTW9YfX5yCKinl4hK2fSO0QEupE6sIVAF6mis 0qvHvTjkF2AHE/zcyOYTuhnORntosRwKL2siaVlbrxc1aSWTq08YWLJWNhdZghaVxqan vIZPE33VZDXNYNH6tC5BE6Aqx0uXP/yBJaWCtfFr0Yvy7z7UrJ4q+Ys5HfXN3+WVCYvy Y1vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UQ3ySmwJxDlPKOGcVlpRpfqxPlYDfL6UGC9Ax4HzzOw=; b=wnwByVb+m/hwG3fBgoYZpdR9BkTBzuCiWc8RTg5fWzSEh8afSedYzE7N8vc8cNAn2A QTVTvTIlxAv5pmLrGAKrWOwcHQGzcQ+nNRDcXloWdA3x12FOPHzLcyqgr6y7yRMfuaBl UBQdtG+pmA4Ia9hIYWfODrIaNFyIGPlEkr6YEl77GiIsF0ESsproJth6tw3T9GL1W7+b DikLZHpC8H8SsaN/L+vd9EMNs+uSycBaMRCvLWZCIVkm6oiD7V+c921wtVHX2GI4TOMq Ryy4ONpzOyKgOw7E/JzSRC1c/poHNDKLOel222Xd9caddPvcgjCcCEEYk2AaUKYLANVV fagw== X-Gm-Message-State: AO0yUKWJC1pV7QKP++m3l+SYntmd2lOJIaBF7uWiK2h77kIQAWXzLeEs 2NTkIEMTzpWGgTVd49tVa0rzscdgsGs= X-Google-Smtp-Source: AK7set8KqluIZbArnLHWo7hLv5kb+HlUQLsv3CceFcKXduycZudkxodptdsBc1Ay4P83RAwvWnVWegqhxxw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:114f:b0:855:fdcb:4467 with SMTP id p15-20020a056902114f00b00855fdcb4467mr33ybu.0.1676524754468; Wed, 15 Feb 2023 21:19:14 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:48 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-34-surenb@google.com> Subject: [PATCH v3 33/35] powerc/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Stat-Signature: pe5493mxbuaaxjzyi3doq8w8bn9pworf X-Rspam-User: X-Rspamd-Queue-Id: 75EE3180013 X-Rspamd-Server: rspam06 X-HE-Tag: 1676524755-869857 X-HE-Meta: U2FsdGVkX1+cDR7nJxZQE3Jc1ty9+Ax0vm5CpW3sef09xLMPr9Jzs8A872M+Eof7DsPDkdnuarSKbZNpZ9P6bFvicaYkc0mz5R46+qylJxBR9OgDUtVyxk4HHU1VqYv71ID/D1/n8Nr0Q8UkVRKo3qvnaSSxitClFjkrhO9uONpxwBEcRDNdJqdEHoE2as+xolWwz23JlD+z9uy2qRqN++AdEVf2ve71dxKkReY6A4AQ1rrL8YfnMs3ApEcJrG/nE0zgxQYAJKhxy6FgwaCNdRnWTD9oxSpCAM1jEF5eutlsF9uqVepLkbbIPPe68WjUGqfkEicmnRWFdDnINnchApleISAV4vQrVY8BPp8a7rNr2zLuuM50o+tEj9K7sUmOKfuf5S2jrahVApbWhdTm8tpuH9OwpMq0X6iSiMc2hmKBFLplbLFD/fPLlI0j4ZGK0Hq8aNd9tvBopXoSwUMJAj/BCBOLNpQwFLhduDxq7aQMapki6CZHRIuBTN8X9Bq1HUTWxdk5G5o5cOSO/j2CjcJSl7OawsDYmKhAXD/9p3WlD6wjKO81YHkKNi6N/vZfr5IIxQ5uVe2Z/KWphb0klufZjExJ+PB6i47b4TvIzM0uhOmhG0kmfGEORZZzfkfjqsSOEQERz+wMKlVf2CaRc9m+A7jpTpBAr7joqmWWAxXJjSP2SzgoND729EPrmm/72ncZec+JqXZzdDkxyUyEXebbtf7tsg58iO5yR7O9n/BzaPbW6N/b/WXztUgpvPyqpxCjJvIcCpkefMmb+by3W1IIkfsy74NHf60VD3NBs+jZOLWLG40Z3tFv9R0pYzhUGYwZZJGWuQciawQ3R1w2NQFUtc9e/9nwlxcRvWvsD3jaQb2QhNPvo5ZvIrtSPYrWWZQtrRQQQG9Mi4o1pNH5uf6SbIFYZZGZtoq3U/A1Eel2QPPchovqbOgPpA4DdjFwzeMP6GGCgeaYhOd29AQ eIXLudIq NnAvxGr0OKS6gp3FSxZkORv4dQvMdwfV1UXmGZN8yJgEpRRzgCUzgGlT2abKvOXTWi+XGJzPrirf08+1S3BbahfuYlpBox3PdIdY4NHgsOroF6ztyroYrJU315EsJVOTIeUsZvH7mppFi3/LvauDdDeMNPbJj+ADiQtJ3LVFZxp00AU9g2qqecO619SHxDyWycdjvUEeWfIcTvOXf3UZliBV7Kf60A61zeuyw1aphgj40R7pTPvaaeFEdMmODYEE9eegDZGrQoQQ7A3n+Wm43+Vldum+I3I2rE/B6CA1sfKUaQfWEe2SLVi8s0FWJAdfu8HaDXFIOCTBO//ZDM+MqgBrS98pfWi9U1tBadW/7bPL8v0T66vlsgNVsZuW6IzWcC6ndyWphzSQuJMxwZIbq7tFA5EY1jyhWK1nWSlXduDFSOcuQlfmqCsIfS8weCB0222N66G3TBg1NFk9sbNoLICK8klRO2gFg/4D+HP9zpnwTvqiuNAOPg+g3ExokQfGL9NpOi13cObJ6eaL8l6O0YR8cDbz2NYUYbvBNTwTgJ35DD0Q7InJQDoI2Zx/DsjfMDnTUcN8XMSpvWzF1Ag00Zn7utpNACuoZ7aPpA9zDUFCEng7lTBPcs4iQ4EziyLNUxhtpbhZQIr4qofS/Nd5D644T2A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/powerpc/mm/fault.c | 41 ++++++++++++++++++++++++++ arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + 3 files changed, 43 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 2bef19cc1b98..c7ae86b04b8a 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -469,6 +469,44 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER)) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + int rc = bad_access_pkey(regs, address, vma); + + vma_end_read(vma); + return rc; + } + + if (unlikely(access_error(is_write, is_exec, vma))) { + int rc = bad_access(regs, address); + + vma_end_read(vma); + return rc; + } + + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -545,6 +583,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, mmap_read_unlock(current->mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig index ae248a161b43..70a46acc70d6 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -16,6 +16,7 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP + select ARCH_SUPPORTS_PER_VMA_LOCK default y config OPAL_PRD diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig index a3b4d99567cb..e036a04ff1ca 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -21,6 +21,7 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB + select ARCH_SUPPORTS_PER_VMA_LOCK default y config PARAVIRT From patchwork Thu Feb 16 05:17:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 448C0C636D7 for ; Thu, 16 Feb 2023 05:19:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB5CB6B0095; Thu, 16 Feb 2023 00:19:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C3EFA6B00B0; Thu, 16 Feb 2023 00:19:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6A656B00B1; Thu, 16 Feb 2023 00:19:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8E3E36B0095 for ; Thu, 16 Feb 2023 00:19:19 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 76BE31C629D for ; Thu, 16 Feb 2023 05:19:19 +0000 (UTC) X-FDA: 80472001638.23.7CF5A6F Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf19.hostedemail.com (Postfix) with ESMTP id BDF3E1A000A for ; Thu, 16 Feb 2023 05:19:17 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=JAL2Aq1j; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 31LztYwYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=31LztYwYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524757; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gCR40Nt3TWmXm3CiMW8diiOUlGd9n6tZbm3lsLizpE8=; b=3UataMOu4NKWjyVFxLUppp10ejDmXS6cgwBHVgBAEiODFlwXhsUCjfc6XYGvaJWnjor/l3 HlkXhZKRKHauUZ9kOtjmTXx2wRXXEGCiNZnBqAdEgKbRUaBCgYpjt8I5wKh/D6oWqJlbz+ u9Jx97g1IWSOYui8t9DDu/FVUXG96rw= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=JAL2Aq1j; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 31LztYwYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=31LztYwYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524757; a=rsa-sha256; cv=none; b=uDtPqKb+4K3g3/5VF5HOtYbS8JQWakvhFkFRsr9YkDeS2c/qEr/7S+rjcCn/n4e4A54qfv JF5MLe1gzXph7lUahDg3KFeZfRJZfukbChPSO9MD7Awq6nAuZeE9c+SB3NXI4mChc9NVqx NSwGYly91v+2kmBHiNyoKz12FSdp+oE= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-52ed582a847so8822177b3.1 for ; Wed, 15 Feb 2023 21:19:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gCR40Nt3TWmXm3CiMW8diiOUlGd9n6tZbm3lsLizpE8=; b=JAL2Aq1jm/4r1JqeZYEWBrue/W3MPTrJmtqCzmWLkH/WIKCq/WNMS7CZcuyalyxOey IosycXB+MxcwT4QBmmHaciSLx6z1YK/qs/xuZHAZ6O8LZPapUolphd/QYky6xf7HXG4m TIajJoSTXZu0Yk9TmYuKl91OepBHI9VMsFb9wtZhCPHchjEtdJSeWxPPYnIrHgMp6AHH zReb74rJ9HRrecOXakA/GD9/YkepqNwyoeE0Te0FwV1uNkov7IGEqryUYtJWv/bTKuMJ k46shKzXYlPWW+imaaqXvyzMwcruK2EkfBqoUiODxJ9SCK0t0Bhmm1Z4Ww2xAI1rN1gq fZJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gCR40Nt3TWmXm3CiMW8diiOUlGd9n6tZbm3lsLizpE8=; b=7HtrQudQ78Jozzub41vJSmLKdUay7ICQtR40NMXBmFTqoPEF7aESrwjtuGX3qU0Dhq D2lzlhGBWHEeq+Jg5gefhi7lUbYol+YFu+uYHkj0/eAyP8yaVq+1f0O7+VIwZojlSbSc 6bon6Jvg1MJCd8y0B9nfj4wGo+kvjs7X/keBIxPwUkiHy6wmPEYF2ldmrtGyhpgJ1HYJ s3dECOBICLCQ4BzPqkOSiainPtsdJNXtDntLu6D8kVE/SofWjeVsaJwsij2yBsoTL5vg WjftYth0H8b9YoqtFQ2b+aZkCKB2XEA9OAPIiaXrGqmWrdzdJh3gh+x9N0fQ4jfJtAsC iUpA== X-Gm-Message-State: AO0yUKV4J6DV7LWcNisGf4n+ph7XO4W3bNhyIQw0MjH1kdYMIKiNvkDQ Rk0E9JlpJbBd7X/L1jLF1iuHGk8NTD4= X-Google-Smtp-Source: AK7set+zfCXRRrabneP7jbhvgAuIkh3ByJMNAQijkS9aLyX/yWFovh9nbJLYkoOZH5rfO4SiFvvO8qqtNyU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a5b:609:0:b0:80b:adf8:4f9b with SMTP id d9-20020a5b0609000000b0080badf84f9bmr575565ybq.33.1676524756940; Wed, 15 Feb 2023 21:19:16 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:49 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-35-surenb@google.com> Subject: [PATCH v3 34/35] mm/mmap: free vm_area_struct without call_rcu in exit_mmap From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: BDF3E1A000A X-Stat-Signature: qq5atu47fabhcjmhohkr6tfzhagw37wq X-HE-Tag: 1676524757-759187 X-HE-Meta: U2FsdGVkX1/EzujQ9Z4h96KSdtYj0buxetQPH9ElFQ9fvjL/5NEWX4YEDCEcY2HEZlw7G/GHQa+RzJVTx6cT7AHe3DATo4oaOaEq3CTwC6Lt52ccLY7tocw+zwLL5b6axXYUuMgjFqWVB/xlq/hpInpYpqA8til/PiLRVSvIfGpX8/KAB3M82Z28HBIrGEWonmlPBGIkp/8HpEZaozoDsfgnK3PJ0Wr4rpEvxqST1La9pbUfN9aFBwrjaSjS2zaZh5XrPcJwOnzeHyZaImyIv0bF6VSlT/EH+/cxSgFP0RVdJ2+ZiqYb4dWxbLRS3/HWq/9zl77ixT2hE7mU3EJhxvNrY39R7RZ6Lhs/Odr+sj7RB6P6Ww1db8oNRM5EnN48b89GeXUOkt0nt59vyeSsx6XMYeDNlnhnukV5DmGpSot3y5COO0jnNmv4t99JJbVBcCgs6CHMqAHMNg2otlayzWz3SZ/gNDGyEMeid0HlTygYoq8WyYJjGCe+L2xqD6E4NVZzIvhfZY8MIAVLC3QF+i7zhwl7qmWFAmldAhreQuDY1qmRWok2xUAOLcffgLGaG1M2YFgmaCiABV1CIGxjvHfRb3tFZGRjsu5dSdAGTJ/62cA99EE1lrIvkyUzLszCRMGknQWyB37T0zM/woAXGf1UQIovIT6PRzjNLVTnTjCl6NkdqfoufpO+QiareFnPVwaFKtV9KM1zkmgIPyFmd2QqUfrl4lgVx6hcVqsh51FUxXbCwu1UFB94SArVB3GM7Y6reFcVwmSwbJFVa62WLKMlNPtQ3Qg3GPOcRVAX8hUb3sHjmD5IxVYQxNCQg6oF2OkdeBinkmsT5PRCKxyKrwbE4WzrxHUikDxn2ZZOLaIfuuYKZ0Y/s3sS5p6nB8u6bVfWvJn9iaCX/tWZo58/4ubPNy5WpjZ+lNVWpoR3P0a2PAHlyi9HfdroAUJRIAUarErCln0ydRarSvqqyH2 0uomQKhE 6i8LbpYEtnS/hdGw3ohzU0+xfW4IVHkm3e/bhPHYL/WGhEU5nH4Lmjbhe0gjOzd7i8rWpSbL3fDCG2sTDaAYYzUyZ4NZKEKc06StMX5+b8pmQEZb65OVJpFL60xMHe2ZsObN1fNGgR7OZZXZdxkPVm6pDlXKcVQJQvU0HNIOntK17OFO5OfVawZw3vjjOebHVoah5qfPoWpW6YfLhvtMXCBo3dtSoiL9Zu/M6TlmIofcJJnLDIH4qjHW3FYQGjySC4C05g/fRcjGqk4UWDyWHiWOQdbVUK/o93cxAKwgMOCokpm7iBfjLZm7l1PHVGLA8w/Udz2U0wBJyL4yN3g5N3X09h99ixNdulROisdAIy/a4/qEjLNlGO0ETidER5IJ4NguAUScoUdNcPug5izC2o4QH+xxgY3RW55EIwsHNtLPuc9uYkYBUKx05ryryJMB1d99iPagPItmF7lJGwd3fv+ANCKyYvS2ITXHd9Y+MsYpgbezHfaic9b6XxHet/FtLnwKoea9NFCGNYUI+w7IYRpuauRqi+rTIp0ZXi/wWVLi8UdssL2lUqWCzHvqmV3EVRyPKryxUm3vwzjBWyHPofHH3EMgjZm0cI//pTWAbdix2GAJ/NgHCAL3eHDgbKDvK7aawis25l2/jVs3P+49TeH3Y0uVQw9JoP3rnM7DlYpHjMjE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. Because exit_mmap() is called only after the last mm user drops its refcount, the page fault handlers can't be racing with it. Any other possible user like oom-reaper or process_mrelease are already synchronized using mmap_lock. Therefore exit_mmap() can free VMAs directly, without the use of call_rcu(). Expose __vm_area_free() and use it from exit_mmap() to avoid possible call_rcu() floods and performance regressions caused by it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 ++ kernel/fork.c | 2 +- mm/mmap.c | 11 +++++++---- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3c9167529417..cedef02dfd2b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -256,6 +256,8 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +/* Use only if VMA has no other users */ +void __vm_area_free(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/kernel/fork.c b/kernel/fork.c index a08cc0e2bfde..d0999de82f94 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -480,7 +480,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -static void __vm_area_free(struct vm_area_struct *vma) +void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); diff --git a/mm/mmap.c b/mm/mmap.c index adf40177e68f..d847be615720 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -133,7 +133,7 @@ void unlink_file_vma(struct vm_area_struct *vma) /* * Close a vm structure and free it. */ -static void remove_vma(struct vm_area_struct *vma) +static void remove_vma(struct vm_area_struct *vma, bool unreachable) { might_sleep(); if (vma->vm_ops && vma->vm_ops->close) @@ -141,7 +141,10 @@ static void remove_vma(struct vm_area_struct *vma) if (vma->vm_file) fput(vma->vm_file); mpol_put(vma_policy(vma)); - vm_area_free(vma); + if (unreachable) + __vm_area_free(vma); + else + vm_area_free(vma); } static inline struct vm_area_struct *vma_prev_limit(struct vma_iterator *vmi, @@ -2135,7 +2138,7 @@ static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) if (vma->vm_flags & VM_ACCOUNT) nr_accounted += nrpages; vm_stat_account(mm, vma->vm_flags, -nrpages); - remove_vma(vma); + remove_vma(vma, false); } vm_unacct_memory(nr_accounted); validate_mm(mm); @@ -3085,7 +3088,7 @@ void exit_mmap(struct mm_struct *mm) do { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); - remove_vma(vma); + remove_vma(vma, true); count++; cond_resched(); } while ((vma = mas_find(&mas, ULONG_MAX)) != NULL); From patchwork Thu Feb 16 05:17:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142528 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3047C636CC for ; Thu, 16 Feb 2023 05:19:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3B5936B00B0; Thu, 16 Feb 2023 00:19:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 33E666B00B1; Thu, 16 Feb 2023 00:19:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16B4B6B00B2; Thu, 16 Feb 2023 00:19:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id F3F026B00B0 for ; Thu, 16 Feb 2023 00:19:21 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CEECD40ADB for ; Thu, 16 Feb 2023 05:19:21 +0000 (UTC) X-FDA: 80472001722.12.7C96150 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf21.hostedemail.com (Postfix) with ESMTP id 19E0A1C0007 for ; Thu, 16 Feb 2023 05:19:19 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kcrkMPk5; spf=pass (imf21.hostedemail.com: domain of 317ztYwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=317ztYwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524760; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/bIj4YrLTVPG3QwseSVPiqG0bxdZfORy7sIu/6u8O1k=; b=iLmZX6fhLqpzZm9q3EX5JblTsp6PZJm9c50qc3IH9YEkVUjef8g5hEKId9ieDLuW1W9mJX dLWNgxzgSNQ/JPAA9yP3YlVwepwQ2IFmJcBXi0aBB+JwD8U26vblswsOPMdrz5x7Wnu4Xs NpHPa+sxexzbeCOh4NTzyETzRTAC62A= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kcrkMPk5; spf=pass (imf21.hostedemail.com: domain of 317ztYwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=317ztYwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524760; a=rsa-sha256; cv=none; b=P169qZqV6WdwxzvJ4zx37lpGVE45fXSUOCH2a5OQzi4d9rE0+iL3UlGcxK0C1FhyXqujRd iKrOn8/SvIpurYN0UVILVlGTJU4MBXrVkfUDWXYtBK91dKpqsesrjSfai6Zp51RHudntjW c6RoGWBXHuZ63GZvwc/pNu5RkBcD6l8= Received: by mail-yb1-f201.google.com with SMTP id v63-20020a254842000000b008f257b16d71so843018yba.15 for ; Wed, 15 Feb 2023 21:19:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/bIj4YrLTVPG3QwseSVPiqG0bxdZfORy7sIu/6u8O1k=; b=kcrkMPk51QPn4U+Qy9PCMBVvz0+hngRjWyUVg3mDY5eCPo5kdNdGQXXAF5nSKjvtxI ovQfwzu/2jGBzDo4t58Z0TFwD6xaqMMee+NhMptwfny7jSZBnt/+oKNA3aeQgGljd8sS akXety944Mg3EUnDXa5SJO4Zw7aGeTPZtwigMyHbadec4ytdul9kOPOFJ5Xa+9QmQgoM XeWGGTE6ollJyfe3sBlUvV2EFNH4AWaioDHTISUd5H/KMkPUAsjYGzb18bayjRFOkrs2 88EaIEp+Df0ZGeekcwiDAUNprBAPetgMpUou+tgl+Oy9Z03QrXgN/piqimqVkufqqcTw xlTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/bIj4YrLTVPG3QwseSVPiqG0bxdZfORy7sIu/6u8O1k=; b=ppOcfe2HCMPdtaUvLG8saO/mEXMfL4ROSXBt4kTd1f5Yni1fjDMS7jQ5EZnhJUIby8 Afcc+7uUoHsapFKyeoR0Chxfmi+EUOZ4+izMTW5Z1vzgpT1uR3+tTmivRjWCJTZSEk5t Is4ErNg2EAzmvJdNuTuCXobN12pZx7N21oSjG8yDFXaNUl9G8u1ssc7hvXBp/4Ge9/lT g9pItx8iBmOvilmjUUsp4z/qp5Ki7ZCwllMD6AvmIrg9FTltWlQT2gSbYUefjI5fpcOl vdKPtZa1K3oLYavKLkOOaNn94g2lPNE4O3NfBPb9GANIP0LKf3LQ5Nzs8WzQ6KWtf7ww xEQQ== X-Gm-Message-State: AO0yUKWQEfYUlnt7tl1CE0yubooAC1jq2DktWwD+wl4KJudb7Z+BfUpd 5XyTnKKnOKQ4xQIeT2qWpDq63VUijW4= X-Google-Smtp-Source: AK7set8myT7mAaWBlS7bx8SxyKn+Tp8Tz7m7loe5CFurrliPBtK4ZSRcKwf3y/QxM2DkFEVl7Wxm2+r0UC8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a0d:ea05:0:b0:52e:fbcd:e1b4 with SMTP id t5-20020a0dea05000000b0052efbcde1b4mr590898ywe.0.1676524759267; Wed, 15 Feb 2023 21:19:19 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:50 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-36-surenb@google.com> Subject: [PATCH v3 35/35] mm: separate vma->lock from vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 19E0A1C0007 X-Stat-Signature: hc7dodjbbuhu98bib4k9gmpwmiwfy6r8 X-Rspam-User: X-HE-Tag: 1676524759-763225 X-HE-Meta: U2FsdGVkX1/QHxAIdW6UVmGqDThJtcI8cmrGptpGVbWtUskvq5SUZMWhZPtEWsFzziHhAje+XIzXEkagPVjyelhZbW0m8gXVSAotgCs/4iq9B4HRYmZgdHJyIhjC7ydMHLMv0U3Kfm2daJ9CKHqZNT/IjWwRiPL//90AuxvYcwf6cCiEyaE/BoGF7VgQ/zvwJzbpq2q/uy8Ffnt6LGROj1FSSn+YCcsA7bkAeggoSaFCMkbNqKEcKQR++6b5p1ZPrz/MZj0AA6FM8UZx0zBXwlvEexULyBe+dKrFYGhdw1jKKJ207doMtUcjlrkPbcSJhANRLci99qd6tJQh12tVtLe8HVN8Lg9/Rx4tqa1+gY/2ePO6jku3JBYISMsSjYuSt+0R0eoPz8jP2aAJTO6o+4NXTKHk4OLJzxkWJF01vrlXXII8UaUIDzRNyHOuAcvTUsfboXfuxxKZ39+E0gt8t1z0DXyKrenhsGR55nav/l9QEcnDYed77crU3/qvDplsyAqW4pHNWqJKTiZk1RUD/2pCyPNfvhw8e5Jzut6KdhIMtQ+uFlKuGCFxgt2z1JQMLgBsvMnC4CmV17r88At+mURVUFGcKQq3YfMweQbsc6IW48gkZkHOV5iI6xABgLUIrgw84AEwLeV76zqjr6aQlnx61lvaEA3N57gRLq1Ode4kpmRQliWba87NMNVdWS7LuZQccMfGbi0dFbE4Z4/8+44nuRQPzOFz0cPljc1X3agFXNULuxEoitQUgXsZsOCDdbjbAkf3DcgB1df90xaUSunTcD0PYLpSkI/7ZmEVWG4rHJoJ5rFOj273c0OvWRj03z16K4PMZfV5Z9L3mFTfWR3pouGCAMBN22qL9hamrdM+uFUff868uqml0v+65++Tep8huOWzwmxor+4BUCtm17TmCV++OLnJhTx1NRqIoH+hHmI0INO6SU93DwOjKWJFmbguj4toLF5Bjhd+ZHo E8Fzj0fC woODmKkuN7MKmqvkxnDO5RdyzNaSAGnLb+Nw5U70TRav86yRmQhcpbKF++eOdnoLjBOmOMi4xnM0aW351svoJxcctUmV7904/n06l4eZaiDZ1Fok7sJ4IfzH0iQ5Fu4hOPtYjmjXpT0l/8WNbsYV7iaT80wNm3Bz87UU+weQISKwIDVWngRO//FRYgSfEHTp+TvttSk9ZULyXmO9A0YJZFwitpEUMhZX/tv40EPyMgBU3Yir/SD4bRMvWhELm2mUwMAU7zsI4UThwNdqhjZBBkvtZJIsuCf0AVVbD3tkfo7BjX7EGm+CsoxDcz67hOifInlybXjIeyUrydBmewCmtttdzS7amoAzewSkCX9rHYPCCyQ6lygW6V+fFOnE3zHU/I+vHJN13Obue5+Z4/ptHJUd20i822x80Rh0bsF0MOZkiG/ti2lfzsSLXOyCSkrLY88kvlLVNO4q2VtazrD3zxVBt6IGa0SuLqPelSCG6fzyXiMppyEgmIBIAZes+YfCHMl8rd3FIzCws9wUVe9PS/pGAZdp2gPJvWtV188deumdtdxwPBdhPgHNnYsl0K2gSBvJb5molrzJ5n1Y= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma->lock being part of the vm_area_struct causes performance regression during page faults because during contention its count and owner fields are constantly updated and having other parts of vm_area_struct used during page fault handling next to them causes constant cache line bouncing. Fix that by moving the lock outside of the vm_area_struct. All attempts to keep vma->lock inside vm_area_struct in a separate cache line still produce performance regression especially on NUMA machines. Smallest regression was achieved when lock is placed in the fourth cache line but that bloats vm_area_struct to 256 bytes. Considering performance and memory impact, separate lock looks like the best option. It increases memory footprint of each VMA but that can be optimized later if the new size causes issues. Note that after this change vma_init() does not allocate or initialize vma->lock anymore. A number of drivers allocate a pseudo VMA on the stack but they never use the VMA's lock, therefore it does not need to be allocated. The future drivers which might need the VMA lock should use vm_area_alloc()/vm_area_free() to allocate the VMA. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 23 ++++++------- include/linux/mm_types.h | 6 +++- kernel/fork.c | 73 ++++++++++++++++++++++++++++++++-------- 3 files changed, 74 insertions(+), 28 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index cedef02dfd2b..96b18ef3bfa3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -627,12 +627,6 @@ struct vm_operations_struct { }; #ifdef CONFIG_PER_VMA_LOCK -static inline void vma_init_lock(struct vm_area_struct *vma) -{ - init_rwsem(&vma->lock); - vma->vm_lock_seq = -1; -} - /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -644,17 +638,17 @@ static inline bool vma_start_read(struct vm_area_struct *vma) if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) return false; - if (unlikely(down_read_trylock(&vma->lock) == 0)) + if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) return false; /* * Overflow might produce false locked result. * False unlocked result is impossible because we modify and check - * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * vma->vm_lock_seq under vma->vm_lock protection and mm->mm_lock_seq * modification invalidates all existing locks. */ if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); return false; } return true; @@ -663,7 +657,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); rcu_read_unlock(); } @@ -681,9 +675,9 @@ static inline void vma_start_write(struct vm_area_struct *vma) if (vma->vm_lock_seq == mm_lock_seq) return; - down_write(&vma->lock); + down_write(&vma->vm_lock->lock); vma->vm_lock_seq = mm_lock_seq; - up_write(&vma->lock); + up_write(&vma->vm_lock->lock); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -720,6 +714,10 @@ static inline void vma_mark_detached(struct vm_area_struct *vma, #endif /* CONFIG_PER_VMA_LOCK */ +/* + * WARNING: vma_init does not initialize vma->vm_lock. + * Use vm_area_alloc()/vm_area_free() if vma needs locking. + */ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -729,7 +727,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); vma_mark_detached(vma, false); - vma_init_lock(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 212e7f923a69..30d4f867ae56 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -471,6 +471,10 @@ struct anon_vma_name { char name[]; }; +struct vma_lock { + struct rw_semaphore lock; +}; + /* * This struct describes a virtual memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory @@ -510,7 +514,7 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; - struct rw_semaphore lock; + struct vma_lock *vm_lock; /* Flag to indicate areas detached from the mm->mm_mt tree */ bool detached; diff --git a/kernel/fork.c b/kernel/fork.c index d0999de82f94..a152804faa14 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -451,13 +451,49 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; +#ifdef CONFIG_PER_VMA_LOCK + +/* SLAB cache for vm_area_struct.lock */ +static struct kmem_cache *vma_lock_cachep; + +static bool vma_lock_alloc(struct vm_area_struct *vma) +{ + vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); + if (!vma->vm_lock) + return false; + + init_rwsem(&vma->vm_lock->lock); + vma->vm_lock_seq = -1; + + return true; +} + +static inline void vma_lock_free(struct vm_area_struct *vma) +{ + kmem_cache_free(vma_lock_cachep, vma->vm_lock); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline bool vma_lock_alloc(struct vm_area_struct *vma) { return true; } +static inline void vma_lock_free(struct vm_area_struct *vma) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) { struct vm_area_struct *vma; vma = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); - if (vma) - vma_init(vma, mm); + if (!vma) + return NULL; + + vma_init(vma, mm); + if (!vma_lock_alloc(vma)) { + kmem_cache_free(vm_area_cachep, vma); + return NULL; + } + return vma; } @@ -465,24 +501,30 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) { struct vm_area_struct *new = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); - if (new) { - ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); - ASSERT_EXCLUSIVE_WRITER(orig->vm_file); - /* - * orig->shared.rb may be modified concurrently, but the clone - * will be reinitialized. - */ - data_race(memcpy(new, orig, sizeof(*new))); - INIT_LIST_HEAD(&new->anon_vma_chain); - vma_init_lock(new); - dup_anon_vma_name(orig, new); + if (!new) + return NULL; + + ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); + ASSERT_EXCLUSIVE_WRITER(orig->vm_file); + /* + * orig->shared.rb may be modified concurrently, but the clone + * will be reinitialized. + */ + data_race(memcpy(new, orig, sizeof(*new))); + if (!vma_lock_alloc(new)) { + kmem_cache_free(vm_area_cachep, new); + return NULL; } + INIT_LIST_HEAD(&new->anon_vma_chain); + dup_anon_vma_name(orig, new); + return new; } void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); + vma_lock_free(vma); kmem_cache_free(vm_area_cachep, vma); } @@ -493,7 +535,7 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) vm_rcu); /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock), vma); + VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock), vma); __vm_area_free(vma); } #endif @@ -3089,6 +3131,9 @@ void __init proc_caches_init(void) NULL); vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT); +#ifdef CONFIG_PER_VMA_LOCK + vma_lock_cachep = KMEM_CACHE(vma_lock, SLAB_PANIC|SLAB_ACCOUNT); +#endif mmap_init(); nsproxy_cache_init(); }