From patchwork Fri Jan 27 19:40:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119276 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 83EE0C38142 for ; Fri, 27 Jan 2023 19:43:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=8RTpDB1/A2ayrfbfUE+ciuFYTDYUdxyvIIYsoTFAoVE=; b=H4ayrjhrcnR3oYlPetwGYK5WWB pura881I57epRDfLUEs7l7n9q7cezzA+0Q1/D/aJsOEErZtud0DthOtBEYLnxjVVcR0m0jYx+W/aA WItYmHFRUIwlfflJNYgKFl3tFueJXCzvhq/GJmxe4VDC25eitq3GYKc/nwsD4g3sB1PPztMPS6wC4 VNFvb9ypbHme94ZpAEXS0WZqXKE2bmuNm6s/SRacSdpa07deesA5wyv6WWdpyXbcqg0JLhPwBzhYM 7Jxb6R+nNMLRHmwg3kzz5IpkAfgy5SwmVkbodPBcGoTOGZqnOeVSqroZIQCdxJSIpXgw8sTNMahtp 9XThKr0w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUcI-00G9Rz-NT; Fri, 27 Jan 2023 19:42:30 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUb8-00G8yC-00 for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:20 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-506466c484fso65952947b3.13 for ; Fri, 27 Jan 2023 11:41:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SRJKiHKrq/SvrK23nJ6WvAwYz6ftOZ3c7xtMrR/Zkmo=; b=q++c4JbFs1cULNmXHJ6C3DczcQ0hXzOKIr6+9Qx2yz39vZWA+rjLc18PKWJkWy6B+C UeBdui8K3sk0l0OIGcxNzLRt494xbxI7WVRAtP+VLTOL1/q+UNWJf0WDLHuQPthduco2 hKKj52t3wKu54Vbflyx5DWFngCga+0YCh2aWf3wXfSgRE76VHyLLWka1QvKlb5EEsVq+ yKsc5SnLc4E9cZlzwXYLALtVVR/qLY2NGEjlF8VIz3gJJ8VVrEVxdVCt2bqpvNVXSzLO bCSv1unz+7C8JA4tTmfPKXKgKgaiBEIUDtx7XKU+dM+xg3yKNZdzkdQzW0TW1MecPnG9 7xHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SRJKiHKrq/SvrK23nJ6WvAwYz6ftOZ3c7xtMrR/Zkmo=; b=E2dWfP6xooAG1/jxbplK1o6eSYzIbPFvgWQIbKx1ivp+H9qnJVnlw9/jcNHSvTC8L3 JUwROKTXtmLLrjV/hW9TPH26+IC3TAHaIx01sArotDNcTWy1dLyuC9jzDpWHVM4myK9/ th4IEc7P+YK35N/7NVelezXMfT2/EdEA1oPa2RbKlTVBd1uUwPX8U4OjFNGl45Vb+YFU Ecwsx0/FH3Y+es0rFZRh+JthT8HER3grIX9wcjRRgQH2LS1PKkBoipF9Db/eRbXOlC9m KEyGt4kseylQjGLWuDhjIjPvkVWugHpMjkhdhr7MP8LvS9s2l6XAd1Rhlsxzc4rxYJcS hlcQ== X-Gm-Message-State: AFqh2kq+3JNUPKtpWza3CUQN/pJ3jwPPRr0fSu3dNWOjXBkofW4N/jVu 7//qXgbw/8VOWQq499I+umqAIQv8YTk= X-Google-Smtp-Source: AMrXdXvEqVwWOYAYjpqjq4VZxkQLftXilDeGRL6KyIgBgBs2rP6fdUqH1kwtp8PmmItyT8YCkU5DjbIMLFI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:4e54:0:b0:7fe:6d35:1a28 with SMTP id c81-20020a254e54000000b007fe6d351a28mr4351236ybb.534.1674848476456; Fri, 27 Jan 2023 11:41:16 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:38 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-2-surenb@google.com> Subject: [PATCH v2 01/33] maple_tree: Be more cautious about dead nodes From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_114118_074706_B11AA12D X-CRM114-Status: GOOD ( 16.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Liam Howlett ma_pivots() and ma_data_end() may be called with a dead node. Ensure to that the node isn't dead before using the returned values. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 52 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 5e9703189259..e5eeecd14eee 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -537,6 +537,7 @@ static inline bool ma_dead_node(const struct maple_node *node) return (parent == node); } + /* * mte_dead_node() - check if the @enode is dead. * @enode: The encoded maple node @@ -618,6 +619,8 @@ static inline unsigned int mas_alloc_req(const struct ma_state *mas) * @node - the maple node * @type - the node type * + * In the event of a dead node, this array may be %NULL + * * Return: A pointer to the maple node pivots */ static inline unsigned long *ma_pivots(struct maple_node *node, @@ -1089,8 +1092,11 @@ static int mas_ascend(struct ma_state *mas) a_type = mas_parent_enum(mas, p_enode); a_node = mte_parent(p_enode); a_slot = mte_parent_slot(p_enode); - pivots = ma_pivots(a_node, a_type); a_enode = mt_mk_node(a_node, a_type); + pivots = ma_pivots(a_node, a_type); + + if (unlikely(ma_dead_node(a_node))) + return 1; if (!set_min && a_slot) { set_min = true; @@ -1394,6 +1400,9 @@ static inline unsigned char ma_data_end(struct maple_node *node, { unsigned char offset; + if (!pivots) + return 0; + if (type == maple_arange_64) return ma_meta_end(node, type); @@ -1429,6 +1438,9 @@ static inline unsigned char mas_data_end(struct ma_state *mas) return ma_meta_end(node, type); pivots = ma_pivots(node, type); + if (unlikely(ma_dead_node(node))) + return 0; + offset = mt_pivots[type] - 1; if (likely(!pivots[offset])) return ma_meta_end(node, type); @@ -4498,6 +4510,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) node = mas_mn(mas); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + mas->max = pivots[offset]; if (offset) mas->min = pivots[offset - 1] + 1; @@ -4519,6 +4534,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); offset = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + if (offset) mas->min = pivots[offset - 1] + 1; @@ -4567,6 +4585,7 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, struct maple_enode *enode; int level = 0; unsigned char offset; + unsigned char node_end; enum maple_type mt; void __rcu **slots; @@ -4590,7 +4609,11 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, node = mas_mn(mas); mt = mte_node_type(mas->node); pivots = ma_pivots(node, mt); - } while (unlikely(offset == ma_data_end(node, mt, pivots, mas->max))); + node_end = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + + } while (unlikely(offset == node_end)); slots = ma_slots(node, mt); pivot = mas_safe_pivot(mas, pivots, ++offset, mt); @@ -4606,6 +4629,9 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, mt = mte_node_type(mas->node); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + offset = 0; pivot = pivots[0]; } @@ -4652,11 +4678,14 @@ static inline void *mas_next_nentry(struct ma_state *mas, return NULL; } - pivots = ma_pivots(node, type); slots = ma_slots(node, type); - mas->index = mas_safe_min(mas, pivots, mas->offset); + pivots = ma_pivots(node, type); count = ma_data_end(node, type, pivots, mas->max); - if (ma_dead_node(node)) + if (unlikely(ma_dead_node(node))) + return NULL; + + mas->index = mas_safe_min(mas, pivots, mas->offset); + if (unlikely(ma_dead_node(node))) return NULL; if (mas->index > max) @@ -4810,6 +4839,11 @@ static inline void *mas_prev_nentry(struct ma_state *mas, unsigned long limit, slots = ma_slots(mn, mt); pivots = ma_pivots(mn, mt); + if (unlikely(ma_dead_node(mn))) { + mas_rewalk(mas, index); + goto retry; + } + if (offset == mt_pivots[mt]) pivot = mas->max; else @@ -6624,11 +6658,11 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, while (likely(!ma_is_leaf(mt))) { MT_BUG_ON(mas->tree, mte_dead_node(mas->node)); slots = ma_slots(mn, mt); - pivots = ma_pivots(mn, mt); - max = pivots[0]; entry = mas_slot(mas, slots, 0); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + max = pivots[0]; mas->node = entry; mn = mas_mn(mas); mt = mte_node_type(mas->node); @@ -6648,13 +6682,13 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, if (likely(entry)) return entry; - pivots = ma_pivots(mn, mt); - mas->index = pivots[0] + 1; mas->offset = 1; entry = mas_slot(mas, slots, 1); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + mas->index = pivots[0] + 1; if (mas->index > limit) goto none; From patchwork Fri Jan 27 19:40:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E0793C54EAA for ; Fri, 27 Jan 2023 20:09:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=UQb3IYPLVQUwnlIJLhll6pUBv/BDThTtDW9VWA0c+Xw=; b=fz1RKLtWCIeyqEoD3hZWXZdKt4 d3o2bUdnYWZoAOlzrhtmWA2Y7C/GKIcuUmNVz8ta1pBQq7raynFyvTFRerIxugnGrbcr05DSv8wa3 F0tyOAIMgfp6hwu++OGvAQVh4FDgs2SBAMkhyvn3mKNzYntEb3WcfaI5X+cBMSrQY+p1T/X+V+aIQ S8RXvvQ9okY1QrciWuU1VNtYGuiCNqYGqjWvctxG639ajYSjsH7LMxc5TvfCO4usuyKrMCIwEXM4q XuJ4FMhzgRPPx8TItQ0vPw7i1u74mvt6fUmBZ+F/htTV78oUxJ9dqetj/+X5G45/W8At/QppTkaEh 8IcHjipQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLV19-00GJDS-6E; Fri, 27 Jan 2023 20:08:12 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUx4-00GHQr-HY for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:03:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=6D7AVbhEFxAo5EKdbJ2HQfTq9cKDp9C6gMDUsIDn2rU=; b=mLCTEPekuf1l+NqF0eb3VVUJ1N NIcz54E41/iUKDIw7Q7UmCaBYFqcrY6AXTb9l+h0HRAewasY6q5wgO00/EJJOQwW7sh/1YC474eRL vc5hk6Ql1Y1tryZ36laPPKwYxZ6K61ovXkEJ4Chocrxm496S01CHCHjTRUKJGF23m071nRE1Rgetv Qb2gSaDJeHnb864FTWJrMJN974GXwpG/6CNACx6Rr1BRv68rvLZdCjzZHMnJ91JHu+czd3l/ZQutB PlNdfgZbZ7SNfpfH5bPttwf3aeysry64yD2n2L29vGc98Ir8db/EXNcvgqxso5I4TvTQr1C7vjRow beqMbnxw==; Received: from mail-pj1-x104a.google.com ([2607:f8b0:4864:20::104a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUan-002xhb-0C for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:40:59 +0000 Received: by mail-pj1-x104a.google.com with SMTP id gn18-20020a17090ac79200b0022bef1f49c9so4968052pjb.0 for ; Fri, 27 Jan 2023 11:41:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6D7AVbhEFxAo5EKdbJ2HQfTq9cKDp9C6gMDUsIDn2rU=; b=ij3G6594G9QMVIcaC7y7IotZlrRGSd461+VUEA/BMKS/DqdJRnleVk5IzJ5gil69bZ nXwMrZpHBfopZhY+dBOM8TntT+xgyMzx0lhn0c4XjFlDdq8hUc51rQLAJayIjAlxU0Vs tSDUCQts0+fWuoHVHrLcPeZOnjtBdS6uOzMiFPrkU4AVT++YMj0+eDXitszbX73gjVet Uj7on6Z68KT42I30IFAHZ8MnQyKVo+KH97KXkIJWbVMmYdISzeeP/HtVHdR/3Pzjnpiz r0YfwFvkG3mRoMMu6hvaQ0O0QqVpO+Mcl7G8itnKrK/kLoHBd5Ey6CcVhBTBp6nBJTd/ 24fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6D7AVbhEFxAo5EKdbJ2HQfTq9cKDp9C6gMDUsIDn2rU=; b=NM0/32xcbb9qpb5/ZtqVGx0F6z7zO0AEURf1n4V8PE5lAmnCt1Q0EYjfLvpdoQGHBK e8HSKZlNk/n7EjrJtKJNgi74Dbxp33scP7MQtUOAH9imEGyApHV2Je8TbcnaBlDxLI9u AuJILS/EHJJio55aEuZSEw65K9lzdvlkjc14Cpd14ANL583HIG1vEyjp+qd/mmKCRYd/ s6eW/4QhUbolINwridL0tejt9B2m7txP20mGLbV1frCMidZhPmdZIWkyTpJdRlgyRFKK sQg1+NIoDYWCJ9tVx/hKVyk0QYeyej12YbuF9Rr9/mMDRccQIm5ozwI76xfmtkNH5mGy yILA== X-Gm-Message-State: AO0yUKU1xf2t6wdYs8DQI606i6l0zEQ6uh4vBDkzPSYmPDMU4pBJQkhl uQXPS2OFZf3lsRn3561YePdPpqEw8q4= X-Google-Smtp-Source: AK7set+8nPPCBQ/ToCHxQhHQnspD+CZKcqljSV2Aj5HMLvXZA8Wu8rpp2+AujJr6dncfulY9izqabQa2p3o= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a05:6a00:1990:b0:593:909f:ed45 with SMTP id d16-20020a056a00199000b00593909fed45mr72889pfl.0.1674848478930; Fri, 27 Jan 2023 11:41:18 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:39 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-3-surenb@google.com> Subject: [PATCH v2 02/33] maple_tree: Detect dead nodes in mas_start() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194057_163160_A46A23BC X-CRM114-Status: GOOD ( 11.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Liam Howlett When initially starting a search, the root node may already be in the process of being replaced in RCU mode. Detect and restart the walk if this is the case. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index e5eeecd14eee..482e17a460cb 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1353,12 +1353,16 @@ static inline struct maple_enode *mas_start(struct ma_state *mas) mas->max = ULONG_MAX; mas->depth = 0; +retry: root = mas_root(mas); /* Tree with nodes */ if (likely(xa_is_node(root))) { mas->depth = 1; mas->node = mte_safe_root(root); mas->offset = 0; + if (mte_dead_node(mas->node)) + goto retry; + return NULL; } From patchwork Fri Jan 27 19:40:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 31AAFC61DA4 for ; Fri, 27 Jan 2023 20:05:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=6wbf8GJt0yhF6km3kBg3BpIyw5Ji87gPzxPyexbpMRI=; b=eQaZ4Ji/HSLdgVOQBSn0bwPZjB nJqBKKSUY3hFjUqBh2SDKmcQ3BSn5dLtW8tHuJIfEifC4wOjQxJ2jQzFyt5hQyN0niewprmtMVlk2 nd3W1GPFp1HWb+ypaFAay4tv90ojQNv7W9UGBNSU+9DfSAFUMXDxxTlCmqXugyUoDyp2fdbwbPh+1 7EbxXbhSf4iE9I/ffWojNrnECsQrlRReKQDiCVqxkvkjJJ21/b32l0hSb4RyEeQgmhKSA36MDTXfF HGemhG/krl9pE7byNUrA3vJHZvuJ3NnffNRQoeqgf03km70zcvd/blkQQuu8DXfFiutHvYqV9wJYZ vS3hnxIw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUxA-00GHTy-HS; Fri, 27 Jan 2023 20:04:05 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUvs-00GH0U-Rv for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:02:45 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=FUzzHzCQcxOurh9YuvudjXpNdZx02NIZtoLw3Jde/PE=; b=qaWBgqI26vhkkiPBZw1LWKIiZB M6dVVckCB6C7Vw1PT50x8ArMbk7kluVfaWGjfHG3/XW3Fs0KFAX5ia1SZ559QHRBFulJVuRoGA4iC fiJ9Luk/duyqXCDCp/ziw91oSHVEhjozAtU3aS9qrzajTUkXuMxEl0tWrooIm+rO+0Sh1D3SLgXeJ DuMR08eoH2bCZhBSfNhpjfWdYczqzqGcYvBCh3/ZyyUDXk56jMFDTFHP+Km9Yf2YbprYKDrzaZucC vJcC+CJ0T/Q9tCrJHON0u8LxtmDnTSwD+8W6cY8O8E7BwCCJESnaRePY9/llbMLTIUqsyDm4nYB1M XbKy6pGg==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUal-002xhn-03 for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:40:57 +0000 Received: by mail-yb1-xb49.google.com with SMTP id c2-20020a25a2c2000000b008016611ca77so6286509ybn.9 for ; Fri, 27 Jan 2023 11:41:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FUzzHzCQcxOurh9YuvudjXpNdZx02NIZtoLw3Jde/PE=; b=S2HR++ybc/ns4vYecXBlgBA5lh7rKCR5RBxCn2d6nLoproYmP4Y/TuK8A2ykL1TZCy PQi2lsVKc//h5m3TPGFUZRCScYCxf/iBRsPO4Le2L6kyLee01De7/gBctPL5x+AFYdbM +GjlXUavIluJZluW6t/CA9goOPJfDGVnM+Mgt+aMp9WxpRPbD/Tewkr6wZubL9JQbZ/Q D22ZQZJELsrUxamrbo349CKHJL642zBXbi0BO/uzmOjlGEGSeG1j/mwEO9F4yWD/yh+R Z4EUx0fbMYpLh0Co5d8CYSkewP6ymMJoH7lGt/nPXvwtHvtN4R6vwQVewCoOaVtJA9nQ 6Lhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FUzzHzCQcxOurh9YuvudjXpNdZx02NIZtoLw3Jde/PE=; b=bZRKGh6PTZCr8Ri9sqf6VHsOd48o6WMoUuzj/3TDzh3ydGhxxdBo7B/igwhQeLxXrJ oKer7UkvgAaCCcOzJoZQSk3otQrRo7VGM7jwwU8xCgBZ4LTrwK/71UdA3TAByjBodw3O fBU3AgNFy1uIMhrDNsgHTTrTumBPo9LGBUuQgsLt47IAu5YNYs3MJDoF1XeK1884PXQm TFSH2Tfe/6SqGo2R48BQwK0RFgneo9rNxg+ZNfnDG+6zR0p+O/EQSRU+JhLQOdASlJzj XH9MFiqwv7o88FMSVE5LJBcLHc2k2VtgCm36+DPo4B0AL8HIC5w98N4jJP0wI951Hs90 UHIg== X-Gm-Message-State: AO0yUKXhm0ZmAewlKbjF2yw5soP/I+nFof9zCG2vrf7+Afndk9fFjOsO A7Val3PWygfW8n1k+IUwjbFBmqAW1pk= X-Google-Smtp-Source: AK7set8Zoaf7xb4TZLOBvvU/fOAKWYAOQ/jwdJrqulf4473/2u2hKvPvmXN7ahB+v/eEMKhFLKA99DnsrP0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:7e4b:0:b0:506:4f19:740c with SMTP id p11-20020a817e4b000000b005064f19740cmr1518489ywn.383.1674848481189; Fri, 27 Jan 2023 11:41:21 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:40 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-4-surenb@google.com> Subject: [PATCH v2 03/33] maple_tree: Fix freeing of nodes in rcu mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194055_321869_5B7E6EB6 X-CRM114-Status: GOOD ( 18.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Liam Howlett The walk to destroy the nodes was not always setting the node type and would result in a destroy method potentially using the values as nodes. Avoid this by setting the correct node types. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 73 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 62 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 482e17a460cb..73917dd2c608 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -895,6 +895,44 @@ static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt, meta->end = end; } +/* + * mas_clear_meta() - clear the metadata information of a node, if it exists + * @mas: The maple state + * @mn: The maple node + * @mt: The maple node type + * @offset: The offset of the highest sub-gap in this node. + * @end: The end of the data in this node. + */ +static inline void mas_clear_meta(struct ma_state *mas, struct maple_node *mn, + enum maple_type mt) +{ + struct maple_metadata *meta; + unsigned long *pivots; + void __rcu **slots; + void *next; + + switch (mt) { + case maple_range_64: + pivots = mn->mr64.pivot; + if (unlikely(pivots[MAPLE_RANGE64_SLOTS - 2])) { + slots = mn->mr64.slot; + next = mas_slot_locked(mas, slots, + MAPLE_RANGE64_SLOTS - 1); + if (unlikely((mte_to_node(next) && mte_node_type(next)))) + return; /* The last slot is a node, no metadata */ + } + fallthrough; + case maple_arange_64: + meta = ma_meta(mn, mt); + break; + default: + return; + } + + meta->gap = 0; + meta->end = 0; +} + /* * ma_meta_end() - Get the data end of a node from the metadata * @mn: The maple node @@ -5448,20 +5486,22 @@ static inline int mas_rev_alloc(struct ma_state *mas, unsigned long min, * mas_dead_leaves() - Mark all leaves of a node as dead. * @mas: The maple state * @slots: Pointer to the slot array + * @type: The maple node type * * Must hold the write lock. * * Return: The number of leaves marked as dead. */ static inline -unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) +unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, + enum maple_type mt) { struct maple_node *node; enum maple_type type; void *entry; int offset; - for (offset = 0; offset < mt_slot_count(mas->node); offset++) { + for (offset = 0; offset < mt_slots[mt]; offset++) { entry = mas_slot_locked(mas, slots, offset); type = mte_node_type(entry); node = mte_to_node(entry); @@ -5480,14 +5520,13 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) static void __rcu **mas_dead_walk(struct ma_state *mas, unsigned char offset) { - struct maple_node *node, *next; + struct maple_node *next; void __rcu **slots = NULL; next = mas_mn(mas); do { - mas->node = ma_enode_ptr(next); - node = mas_mn(mas); - slots = ma_slots(node, node->type); + mas->node = mt_mk_node(next, next->type); + slots = ma_slots(next, next->type); next = mas_slot_locked(mas, slots, offset); offset = 0; } while (!ma_is_leaf(next->type)); @@ -5551,11 +5590,14 @@ static inline void __rcu **mas_destroy_descend(struct ma_state *mas, node = mas_mn(mas); slots = ma_slots(node, mte_node_type(mas->node)); next = mas_slot_locked(mas, slots, 0); - if ((mte_dead_node(next))) + if ((mte_dead_node(next))) { + mte_to_node(next)->type = mte_node_type(next); next = mas_slot_locked(mas, slots, 1); + } mte_set_node_dead(mas->node); node->type = mte_node_type(mas->node); + mas_clear_meta(mas, node, node->type); node->piv_parent = prev; node->parent_slot = offset; offset = 0; @@ -5575,13 +5617,18 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, MA_STATE(mas, &mt, 0, 0); - if (mte_is_leaf(enode)) + mas.node = enode; + if (mte_is_leaf(enode)) { + node->type = mte_node_type(enode); goto free_leaf; + } + ma_flags &= ~MT_FLAGS_LOCK_MASK; mt_init_flags(&mt, ma_flags); mas_lock(&mas); - mas.node = start = enode; + mte_to_node(enode)->ma_flags = ma_flags; + start = enode; slots = mas_destroy_descend(&mas, start, 0); node = mas_mn(&mas); do { @@ -5589,7 +5636,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, unsigned char offset; struct maple_enode *parent, *tmp; - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); offset = node->parent_slot + 1; @@ -5613,7 +5661,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, } while (start != mas.node); node = mas_mn(&mas); - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); @@ -5623,6 +5672,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, free_leaf: if (free) mt_free_rcu(&node->rcu); + else + mas_clear_meta(&mas, node, node->type); } /* From patchwork Fri Jan 27 19:40:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D3FF4C38142 for ; Fri, 27 Jan 2023 20:04:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=nF+3azVPwckTIJ46OfGl5KVbIesN/BTaxUegcAPf6iY=; b=2abWaGeieslXu1iC1HwkMG56Bx DOnxVEmFLgQrUbpOmFC7aKVMo5JCzDbCNZu0YDU8+/bMtzHguWBUDKq2FaEGiQ4FP2CAk+FwzhsgX r7dJ029YdfVEM8+FR0Jn15J9MGC41CbgPnLdMOzqHmQFmuuH2lt2WFqfdENztzmgxatdt36U6BvAJ tBpP2r7F8HiqddXMdDFqMFl8cuwBdc9XzhioplyXk+SmIse8GqCRcKWjlvvoK8xjOReeVv+lam9y2 FMwD00Amd4BdeV1RN3JMgWfjugOeAy+YJDfZj4ihLTqIk/GySDtLgbgFBgzNV7c4rOfGKxM2EIAc2 MaSQ4cYQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUwb-00GHE9-LB; Fri, 27 Jan 2023 20:03:31 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUvr-00GH0U-Qt for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:02:43 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=DF9zqRO1prEDm4IKfYXh+Z69pFj8RdYSRvmvJpFQZKM=; b=qh9aC6sA59KzU5a211cWiT7h2Y akI4s+Utx1uberasnljm9M3F2dnC65VanasgSlgtahMhB0EW4RusJ9Urp1431+kydUpmnmcS4ZKQh p5BDMOT4DZFlDLbQpWaT501Avg+D52lELNbg/9HndG3PgUkl5f9fulI91quxce5I9xBGeYhaZ29VY uipjKA73NlYvgdxKihhagbb1TTk/9sr8U4WFi02zJse7ekkD3LYJI20t1rmB2KfzIZwqTy0h57UV9 L7JEqeH39wjUHP+rxSaG9BQfObCM+jor8z5X7GJhggz7FAf6UtkR0Cn1Wx/vbB1DuQbT5vZZ73fT5 sn5Wcvdg==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUal-002xi0-1o for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:40:58 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id t13-20020a056902018d00b0074747131938so6388214ybh.12 for ; Fri, 27 Jan 2023 11:41:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DF9zqRO1prEDm4IKfYXh+Z69pFj8RdYSRvmvJpFQZKM=; b=rnpUdmPcBnY7plbcdmS2pwvwnctQIRIqHumfjs2guCuPsY+Dw6YQ2tfutal8HnrgXz edy5Yrgey9QhoRSbyw/SmRVhoDEfIcnSP0t0LCmLUom2gJSxBSb6RoRoQ6R6E1A4+8T0 XcTb3Fhtcz5YcundAJZb+WYoiQCLWWkiPl+/L7XZyu1tWh8b/ENfJXXDc0COCB4H0HVg yuAJnUBiF1z/T9ta5KOn1AHf3cQZUH/oUuoUg6oOo5mcl+MifhrO/XwhHeojlZjv/4Mm lccC2wtCBVxTksQLP5lorO6u35MCkxakWEmMvAgnG98JEeGI6fsdBcznmI1LtXlDE24U yV4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DF9zqRO1prEDm4IKfYXh+Z69pFj8RdYSRvmvJpFQZKM=; b=eroUpj4l73NoHa4Lvs1uXnWyEzTsPz9H1Z+KvugKusmBqj8SeOI7CccNOeHNpa3gVv LLmpZzW+//1z9t6/l5BRfQGE+kY++MJDb00sh+cuXJl8+YKmecnfjcIt9cnQDf0dvtcD 3MffsgE9DHI6trgi045toVhPibMIyHmfgUcT/vxWE0PPbShPcIzBnZKMcDh7GZzdmZZM vMkbeB6nePpKoIgiBoEVBU4+393qODArjctwRe+2xJs0AV5WpkrsuPMhqTKz27qL0JIT buUB+N8rCxUTfroUAqw4okzY4vsQpYaYHOBEDN3/zXmaxWeq3SuHPmhL3eNQ2R1IDpOK mRdA== X-Gm-Message-State: AFqh2kq7keGtSK7T58QrXGIPNS5hL6Kkt+oAHTCXluSy8u/v7U1UH6Fv Ax788ZDOSQK71isC3ZgmrRv9bSt6iEI= X-Google-Smtp-Source: AMrXdXs3xZaVcR3p65iqnfy8qe0w+PmLMIhbeI9n6IEvkAzMLp6jSQOzi5T/oBy2kRvdJXFWY9xcFs4GkSg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:c8c5:0:b0:4f3:10b1:1351 with SMTP id k188-20020a0dc8c5000000b004f310b11351mr5009568ywd.516.1674848483749; Fri, 27 Jan 2023 11:41:23 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:41 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-5-surenb@google.com> Subject: [PATCH v2 04/33] maple_tree: remove extra smp_wmb() from mas_dead_leaves() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194055_688426_25D458EC X-CRM114-Status: GOOD ( 10.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Liam Howlett The call to mte_set_dead_node() before the smp_wmb() already calls smp_wmb() so this is not needed. This is an optimization for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 73917dd2c608..75cce2d4d5da 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5510,7 +5510,6 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, break; mte_set_node_dead(entry); - smp_wmb(); /* Needed for RCU */ node->type = type; rcu_assign_pointer(slots[offset], node); } From patchwork Fri Jan 27 19:40:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DC381C54EAA for ; Fri, 27 Jan 2023 20:11:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=asEhbACSFKOk5RkMCvyXgCfNTNWuZgrDWVmKKMFCkmU=; b=xgq8jhMEL9aqkYnoZaQh/wGlFi ejgOvpQ3aCNkYVqjHtvRmmEGxS+zn3mwy5/TD0cl34uKmkhNECa9gnEQt6w5ciLgHxXdlVAvKhozJ hU3u+IYlOsFxMbSYPNAqAWekh9ycrpA5zx1Dq89lE15E+9lTaJYwsaAt4e31nSSZZvtGXOCvA4rTC sueVKPuZxbz5121lXZm3akMLK/duJC8Mr+2ISDDPfRtIt6en/QIgUAPzT/DHnN5yiXw5pvm10JBFi EHVOJKAJzwr+Oo40F+LhQm5N3rz4+yzYJnr50VCLvgz19AeE1wa+Y6UsacB/+JKVIb6YoYryjLPkY +Dr7Df6g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLV2h-00GJoI-Nr; Fri, 27 Jan 2023 20:09:48 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUxJ-00GHXe-Ec for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:04:13 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=fluAUEldqgo6Mr/4Su/gpwJVTd/f+QKHo7F0iCGeqOg=; b=giqqYSB/LUi1xMe7nHEzDNIRjg RW4+Gmx4cyNUo3ysPZySoUGcLYV93Tt1sVwxvLcR+MaMIwRgQwCVNdzsrfOINiKkfCEtmBAooElLq XQfxuA3Pj6Ykk0ymRw3uAMViMGJuOIKUtV+SEz3ddRImEkwHE7g1IE1zUSe370cSjNf4Vn4n19ltT bndv7qX8a0GDElbIN0/OgtwYyL0gkKClAlmL/1cNOqJuEobErmp8wpiOEXslWRZ2c+o7zmlOVGfkr pHC5DQxh8cRJ9M9OJgG+6yEV6OO/w2X8xSIzePGTsjIiSkyJmwXGcqd5hQZ5d4B4mX79V7k0PkH56 iRrK5cBg==; Received: from mail-pg1-x549.google.com ([2607:f8b0:4864:20::549]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUaq-002xiy-07 for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:02 +0000 Received: by mail-pg1-x549.google.com with SMTP id f132-20020a636a8a000000b00473d0b600ebso2619543pgc.14 for ; Fri, 27 Jan 2023 11:41:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=fluAUEldqgo6Mr/4Su/gpwJVTd/f+QKHo7F0iCGeqOg=; b=B0QFA0zTTAgJFhNgrdDgKRfLtInL5VmAHE1KFwNYXYIlcWvaJJq1PKzYA8ckdy7Kyn xAwTf5YPOblK6+TJj1tGvdojweaIBaHFiTxRDHhkDnK6qDnyvgtYbq5iiXWxBxOCwUuh WaET5h8CLSuaMj6adU6fJWH6EpE5SqyIKnzFJ/pir1JZwDIco5M5jQ2a/9VWhgwD3iWL QgWDwglZ0wa8AC1rJh0bk86XOm5Sb2wt2duuTU/yqZcdsgUl6ZFLCYWIeCgAmHx4ObRd hHIbD6pxz2ke/VukXpDhoYuOo0GEGrcZQn8DqZwlpC0FiE7wMl7MPAjekSdb7LenTMW0 255w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fluAUEldqgo6Mr/4Su/gpwJVTd/f+QKHo7F0iCGeqOg=; b=P8HXISEH0xMLQM5ejb4fs6noXfWSN1fIHUojINlLi22j0a0oKBEv8D5DN0h+yEAKzU x8P8UQoKnuhzTNdt+wR/eJWTAfTs7UaL1i9lBluo7xnVdpmAlXtefmoraZ02q5zyKL0P 2mJdJ0XwnXkerQeP7G9Qbxtp456WZzCr6wCosS6M0sxVpM6QpdDgkB2sk8zNkojPK9Jg fx5K/8amn0oKj4CwAXZlyJ3AEya+PXRKDKPNlRXPQ3jYUTsE7QXVfr8rLR+HKtlt/0nm X2SCnStJbiTisUz8BcNTDHFvTp+ThNfl5cH1PSS/Ucvybn1Et3HddXjd1dOO2xmGRvus jpJw== X-Gm-Message-State: AO0yUKVcdpoY9oggF5Ub0iIBFilxa5zP+hL1lPL6BnZ/lqDM5lAzusFX ZXh+28uiQnnHxSGldlRbjdtq+W2EaSc= X-Google-Smtp-Source: AK7set+gvzIFHouo+ELhD7kNJY3eRZqYYIzPpjJr9+paWQDRkxxsh7BJRXlaLIfiaGLvKexw3FDEYkSiOQU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a17:90b:a16:b0:225:eaa2:3f5d with SMTP id gg22-20020a17090b0a1600b00225eaa23f5dmr12984pjb.2.1674848485998; Fri, 27 Jan 2023 11:41:25 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:42 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-6-surenb@google.com> Subject: [PATCH v2 05/33] maple_tree: Fix write memory barrier of nodes once dead for RCU mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194100_195080_68589B62 X-CRM114-Status: GOOD ( 15.77 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: "Liam R. Howlett" During the development of the maple tree, the strategy of freeing multiple nodes changed and, in the process, the pivots were reused to store pointers to dead nodes. To ensure the readers see accurate pivots, the writers need to mark the nodes as dead and call smp_wmb() to ensure any readers can identify the node as dead before using the pivot values. There were two places where the old method of marking the node as dead without smp_wmb() were being used, which resulted in RCU readers seeing the wrong pivot value before seeing the node was dead. Fix this race condition by using mte_set_node_dead() which has the smp_wmb() call to ensure the race is closed. Add a WARN_ON() to the ma_free_rcu() call to ensure all nodes being freed are marked as dead to ensure there are no other call paths besides the two updated paths. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 7 +++++-- tools/testing/radix-tree/maple.c | 16 ++++++++++++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 75cce2d4d5da..49e399e8afaa 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -178,7 +178,7 @@ static void mt_free_rcu(struct rcu_head *head) */ static void ma_free_rcu(struct maple_node *node) { - node->parent = ma_parent_ptr(node); + WARN_ON(node->parent != ma_parent_ptr(node)); call_rcu(&node->rcu, mt_free_rcu); } @@ -1771,8 +1771,10 @@ static inline void mas_replace(struct ma_state *mas, bool advanced) rcu_assign_pointer(slots[offset], mas->node); } - if (!advanced) + if (!advanced) { + mte_set_node_dead(old_enode); mas_free(mas, old_enode); + } } /* @@ -4211,6 +4213,7 @@ static inline bool mas_wr_node_store(struct ma_wr_state *wr_mas) done: mas_leaf_set_meta(mas, newnode, dst_pivots, maple_leaf_64, new_end); if (in_rcu) { + mte_set_node_dead(mas->node); mas->node = mt_mk_node(newnode, wr_mas->type); mas_replace(mas, false); } else { diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c index 958ee9bdb316..4c89ff333f6f 100644 --- a/tools/testing/radix-tree/maple.c +++ b/tools/testing/radix-tree/maple.c @@ -108,6 +108,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mn->slot[1] != NULL); MT_BUG_ON(mt, mas_allocated(&mas) != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas.node = MAS_START; mas_nomem(&mas, GFP_KERNEL); @@ -160,6 +161,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i); MT_BUG_ON(mt, !mn); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -192,6 +194,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != i - 1); MT_BUG_ON(mt, !mn); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -210,6 +213,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != j - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -233,6 +237,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i - j); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1); } @@ -269,6 +274,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -294,6 +300,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas2); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas2) != 0); @@ -334,10 +341,12 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); for (i = 1; i <= MAPLE_ALLOC_SLOTS + 1; i++) { mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -375,6 +384,7 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); @@ -382,10 +392,13 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); } @@ -35369,6 +35382,7 @@ static noinline void check_prealloc(struct maple_tree *mt) MT_BUG_ON(mt, allocated != 1 + height * 3); mn = mas_pop_node(&mas); MT_BUG_ON(mt, mas_allocated(&mas) != allocated - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, GFP_KERNEL) != 0); mas_destroy(&mas); @@ -35386,6 +35400,7 @@ static noinline void check_prealloc(struct maple_tree *mt) mas_destroy(&mas); allocated = mas_allocated(&mas); MT_BUG_ON(mt, allocated != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, GFP_KERNEL) != 0); @@ -35756,6 +35771,7 @@ void farmer_tests(void) tree.ma_root = mt_mk_node(node, maple_leaf_64); mt_dump(&tree); + node->parent = ma_parent_ptr(node); ma_free_rcu(node); /* Check things that will make lockdep angry */ From patchwork Fri Jan 27 19:40:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 86E80C61DA4 for ; Fri, 27 Jan 2023 20:12:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=OkMuoWY0UthFWiCFaseZ83FCFa0yxrllYqDAiVdZ224=; b=q/jtzsvj53Mas6iACXA/MyW4y0 k97QPmIsxuLN0Vh33Dnhs0YyX7PqHQB76GUubhpQNhZCURAK+apVDloQ+6sU/3ksuLabxHTl4Tvk/ DMU478J1KXNqEg4VS9ro85FvH5Tqr6jfnnCB7JyIPK5j6bdiybzMJCWsf1JKscZ+dLJUkhVTGpO9e JoodTBeqOfpyEASRtG0IWjKF7NGHMdXa6UxGWK8+IpPO9O5poxi4jGJtI57GjX0yCdT1BVqRlv5le sB4TB/qlf/+IvbjS0q7Hd4ELGOOv5w1o9HUtWtNKcppjkoxkVtBkEOcASd99UlP/oNV2VJdM0Q6pR tls8Pvtg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLV4O-00GKQv-Nf; Fri, 27 Jan 2023 20:11:33 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUxM-00GHXe-1k for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:04:16 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=A97N+ZL4sr7Moa7VA3MaIfeszXjKUidgl5sbUl4VE3M=; b=DH0lY7HDK6m9RPAGpVSm5oUhET khzBoZ+KyJvOaPi5pwCoETnIMonIo8wxfibKXjP2ci6iqsuPFUdQ70fFfJiO05KOsevwAElI3RwDX zB3cC7YoISj9ttG7F14hnE+YutC3mGKw+vT9OKrdc7HsQJ70YXWG98opVMUribBe/zrZksC7CyYQA q6dLmcl21JL88goVjnwrcsL2pVjdCz5bqb5DTw+KQBxewCFLX8yVMK+GS23ch+t80MhcSDh+mvham g2P5dp8t9IpPK5qlTCX4P80/7659c8zQDqH2XWmwK9NoIGDJSGj6OEyPmcy8XxmgcXBxDnQ2xKRNi wy9acLdA==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUaq-002xjk-0z for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:04 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4b34cf67fb6so65239297b3.6 for ; Fri, 27 Jan 2023 11:41:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=A97N+ZL4sr7Moa7VA3MaIfeszXjKUidgl5sbUl4VE3M=; b=h4z95aWqlTijLuw2+pa/O521iwctvYJ/JE9Gk94Om8JNdm1J4qRBJ1EfNZDOaurfUb h0xAr5ZA79m4jhsd5KmptV8L5YTVXkJEqIIo06fTcAgDWyf2Iw1unXEy8R6f3F+h7n0/ eKAno27phPvhqEsCndY+cAQm1TD4BuL7gONpQNAXYFn8Kf2q+ZhvCxmCxJsT4hK4blKF lDmG8J5RutFJ9mwfq4DqWgfTIN8+TeptwgRlAF+qB5k0K+pf2NR8dkIUNhEjxCEKvwc/ GAm3buayP+cfA/HX9e16HQfg7j+WzzAIFSLNgZD3Zm/44+VjSwN/Bplr8AdUjUNKklkq c5Vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=A97N+ZL4sr7Moa7VA3MaIfeszXjKUidgl5sbUl4VE3M=; b=CPhVMa+c9Zybk6b2HoLrBYJ0yk2wxL38wpnfgu2scB1VO21BxFuUm7lpJZ2cUn8H3V vnB8TKFDv+0ixaGJyCdm6Xj30wXi3ELGM3PTnhWP2XkDPAbdTN5riR7coKp56F0iWgec anoWIp+nob3zKnqWL/9lK0XhoBExJ/ca5LWQC04K+5y+2JyJQ8YhS1ZiQu1hIMCUMKmS dRv1MvwQuIeToxzf0d7S070IYbywtKjXy0nT3O3GkP7+7FBYwy9xogrD965ZnVzFGgYi KcQPW80XIAYGI0I01kJflHCRbFu97AHTCyANEG2LT9skt/wD4QDUzsyzddzFVT38gDqK prUw== X-Gm-Message-State: AO0yUKXYADRDjsmyt2tkXPTMVh16qVppAf13LfNMtxzsijVjrtYwdJbd uehPoGDOBRl+O5/jNZ6xml+MeCVr1NA= X-Google-Smtp-Source: AK7set8r1BYS7Wv/oAXgdJ6+dIiQhgZ/QiambSorQvdFTePlCpS2jC0PmTUUP3dVFKS1ORR7IfLO9gpuOYg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:9bd6:0:b0:509:5557:c194 with SMTP id s205-20020a819bd6000000b005095557c194mr892323ywg.449.1674848488954; Fri, 27 Jan 2023 11:41:28 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:43 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-7-surenb@google.com> Subject: [PATCH v2 06/33] maple_tree: Add smp_rmb() to dead node detection From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194100_445555_FC2D2C48 X-CRM114-Status: GOOD ( 12.28 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: "Liam R. Howlett" Add an smp_rmb() before reading the parent pointer to ensure that anything read from the node prior to the parent pointer hasn't been reordered ahead of this check. The is necessary for RCU mode. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 49e399e8afaa..859303d2da90 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -532,9 +532,11 @@ static inline struct maple_node *mte_parent(const struct maple_enode *enode) */ static inline bool ma_dead_node(const struct maple_node *node) { - struct maple_node *parent = (void *)((unsigned long) - node->parent & ~MAPLE_NODE_MASK); + struct maple_node *parent; + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); + parent = (void *)((unsigned long) node->parent & ~MAPLE_NODE_MASK); return (parent == node); } @@ -549,6 +551,8 @@ static inline bool mte_dead_node(const struct maple_enode *enode) struct maple_node *parent, *node; node = mte_to_node(enode); + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); parent = mte_parent(enode); return (parent == node); } From patchwork Fri Jan 27 19:40:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4FC4BC61DA4 for ; Fri, 27 Jan 2023 19:46:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Q6/7LlAELguCqGR3Qoq1/OQQogrTQ8kZyH4aymBMGz4=; b=mSKvSPWkRBZh/kWMh2vi19rf2x qBA1qIWxqk/heCg5zQ5G8auTMQrjcW5Ynz/xH7Xqvwzgwsy6xaOGHhZuzdLucIjcMBWZcJqda31/1 oLgpfpEptV6hvxlfvEiuUiEx3r8u5TecG4V6q7DKOiB+lXEOkaMHxaYujE6vpcYzx4zxq2OG2mRJi QKC0pOTx+zQHUldTwdnEU/BY7je4ERTjE4VgIRxd6e6NvEB5BKIdFGjqomBF2RvmYIEZiDL2oH6SC sg5QAJzt1dvinVPSuMpHori1prBClG86In1DjIySGIqPIlI1+hMDe1Guk5aCEC9GW/Ftg99KszH7Z jY3dK5Ig==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUeZ-00GAXU-Nc; Fri, 27 Jan 2023 19:44:52 +0000 Received: from casper.infradead.org ([90.155.50.34]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUbe-00G97M-Qo for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 19:41:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=mDt1g22Fa9gYu2Kb7/F2UNWz7p2znEfhohgEMazkZAE=; b=PSov22FsMV1DSgXgMKtBuXvTw3 LCKvsR6LRIuVIPRqXB2Blx5rooUeGoDVfv3NAe8113oigj/YzFRwO+bygjcAVbHVHTUw5FVoOMCdS Qv6Bh2dFwJMXBut5fOHZGlKkqwfK7MynbSESVCt57InBB0aAz3P/4NwUPBxE2Hx3B6GwXYQkHqcq/ 1It/wTHA/DHQJRBkxbSrv0gutyA7z0PNP6Eczcui4dw7IK7DkYNJMCM9hywWfqi1OX7fyVYauM8I2 QPOBlAkcpbtcbZoRy6kdR2HRnYTDawclU25iq8De0ueB+CNkPwUhRJhkJBVlzqxwavZGBQvKFR2C1 vGed+ujQ==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUbV-007zFU-S1 for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:48 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-50e79ffba49so13647117b3.9 for ; Fri, 27 Jan 2023 11:41:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mDt1g22Fa9gYu2Kb7/F2UNWz7p2znEfhohgEMazkZAE=; b=n77zu0oRsJL7OvjwFNjS7T1FAOtu4hdvxpK7kq3XthnUipBv5szcKq0gW6Eh2qWZ+N VqA54npBtuI1r6BsRB8yCAVHYO39SuVCqNfhqCm/Ya4OMopwGqTAxzjzqXFTaLEmWefn G7d9A2dlsE9NEs9J39+IVmn9jXa2H7iuxd7eUv1r4SdiVN/BsV9E4rX7Fx/7SJnqIKhp 5pljM+LwjnmUAESpDExSAXXjwcvSQjwdpBA+epV9jtEBYonrLt6ilkcmvKbIgLrFIqhL dWwHvu2PRt1rquBCjgEA2IpAdLCPilrM/O5s952uzrtLUUUIX7VUyMYpZs8xK3jyFgDu 2ztw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mDt1g22Fa9gYu2Kb7/F2UNWz7p2znEfhohgEMazkZAE=; b=mtXiEE2vFi/A7K38+dqCOwal5AWAA+Oe7Qh+WFWWRA+HhSiDJY9IAuMx6qzChogIVI DyKtGFwFGkDoEYDRue58DuGJOvyj6W0/zdWzPAd9j2QoKML3tZ9iRhHhLL6QFSMWqu3v sdQg1DeWPdymZ8Ax6pmjuu1+gP5yECMBpneyzxFIh7G8fcqTY7zyxoowyuxqsdGZOzUD tfm+GWUBDRgVNX+ruvWD0mXVoej1DWoGOJ9XGoMiH0QfuRSBxv0aJpQtuBD2YmX+5qxx SsbpuaNHqPE+w50TJLgNa7GiHHSotNB9+2FJrm4mb/mLiO3haMXXpfZr+XebsZBG4FRZ bDlQ== X-Gm-Message-State: AO0yUKVoBCYbj2nCNUVFYKIhddmU/P+Nja2tM/AoF8ghEtMyd3UO7I6C +cMV7mY8xxQDqL1b59raiG7JY7eOmio= X-Google-Smtp-Source: AK7set9vKZUOyTcPurWe/bFPJyhfVHuV/nLiUj+ueaBvjKNu+svIA1blUVC8f6GWjxXLSfMUdNgMQcXmD6M= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:354:0:b0:80b:543d:2106 with SMTP id 81-20020a250354000000b0080b543d2106mr1234365ybd.10.1674848491227; Fri, 27 Jan 2023 11:41:31 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:44 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-8-surenb@google.com> Subject: [PATCH v2 07/33] mm: Enable maple tree RCU mode by default. From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194142_397105_0FA683A0 X-CRM114-Status: GOOD ( 13.09 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: "Liam R. Howlett" Use the maple tree in RCU mode for VMA tracking. This is necessary for the use of per-VMA locking. RCU mode is enabled by default but disabled when exiting an mm and for the new tree during a fork. Also enable RCU for the tree used in munmap operations to ensure the nodes remain valid for readers. Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 3 ++- kernel/fork.c | 3 +++ mm/mmap.c | 4 +++- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index da983aedb741..8410c3052148 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -768,7 +768,8 @@ struct mm_struct { unsigned long cpu_bitmap[]; }; -#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN) +#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN | \ + MT_FLAGS_USE_RCU) extern struct mm_struct init_mm; /* Pointer magic because the dynamic array size confuses some compilers. */ diff --git a/kernel/fork.c b/kernel/fork.c index 5e3029ea8e1e..5f23d5e03362 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -617,6 +617,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, if (retval) goto out; + mt_clear_in_rcu(vmi.mas.tree); for_each_vma(old_vmi, mpnt) { struct file *file; @@ -700,6 +701,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, retval = arch_dup_mmap(oldmm, mm); loop_out: vma_iter_free(&vmi); + if (!retval) + mt_set_in_rcu(vmi.mas.tree); out: mmap_write_unlock(mm); flush_tlb_mm(oldmm); diff --git a/mm/mmap.c b/mm/mmap.c index 9d0dadb55e9a..9efe13d36df7 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2277,7 +2277,8 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, int count = 0; int error = -ENOMEM; MA_STATE(mas_detach, &mt_detach, 0, 0); - mt_init_flags(&mt_detach, MT_FLAGS_LOCK_EXTERN); + mt_init_flags(&mt_detach, vmi->mas.tree->ma_flags & + (MT_FLAGS_LOCK_MASK | MT_FLAGS_USE_RCU)); mt_set_external_lock(&mt_detach, &mm->mmap_lock); /* @@ -3042,6 +3043,7 @@ void exit_mmap(struct mm_struct *mm) */ set_bit(MMF_OOM_SKIP, &mm->flags); mmap_write_lock(mm); + mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); tlb_finish_mmu(&tlb); From patchwork Fri Jan 27 19:40:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119292 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2F461C38142 for ; Fri, 27 Jan 2023 19:45:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ygiPalslccWMLEhoh8/uqREvT1CtunyYMTVQJRnz/M8=; b=ZvmBi162JUofJ2X+H72b3mCdI9 7TPJl+UXt6y/IuCIeLdN3KwVPRzADhUe67fmWlzQTzpZbLEebs1z68NiLoEZe0o4kIhPBLz9nTss+ lN3xcsdQvWkc7YXYCZ6HojLiHAOta93OBDrrP8QX4m95txHAxg+iZ7WxmV/rsekPTCtUCj8SA4+38 WFFGuxo7jtXzj2+tuOlfcKImAdx0W1j++MpC1DMiIfPNHWqOzlA6W/EZH1Y7+AZIDHOgdFeNRPPgq PsXNsH7uGQ+/a1FRTgcCPd7WeLrvQ6cinwJB472LgmrWcmWitdyW/bq5xBVJzzpHtQtb8gbYiVtbW sa5xcynQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUeG-00GANj-RK; Fri, 27 Jan 2023 19:44:33 +0000 Received: from casper.infradead.org ([90.155.50.34]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUbe-00G97R-FB for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 19:41:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=Z3PGaqKdxS8i+UY8qe7U5+e5Uq 4wKT0OW1+cFKwQoSyeahFmCXGIPQOrqf17IXJuL1TMU4NWzVtrtxpbndxPNQLWu6mbSqMeWsPKsjR hby8QHmCoTpbUNAf6dG2sS4soWecvD/J4MuyKrTWuYbtvasK7bGc2PDK4mehs92bCZDgRE6HuYBE1 XxfGLXHMe3c4DDGMDWMzvWkQyvXAIhCIyCG8qmoER6gb/G+naT0zz0a15V+jTFIXxf5RRzXc6BaVg SWTSXv+RvmjCa2018+vDyQnSXXAh6hXzD3qrdosRxbyENMb2CvM1JW571bgs/YMjCS0pakcKqn0/7 nAyF0wrA==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUbW-007zFT-9S for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:48 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5073cf66299so66249197b3.17 for ; Fri, 27 Jan 2023 11:41:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=m4Wtor8vj3J0ycRC6QGu22LySA+n2MGlbf/sH76T8N8Pnp/d8SuPXb/kR+Hyj37+ad 48NkH1zhtbcV3yGPaiQ5KPXnB06jsyj37BRmFZaU4H+LJ2YSwMP7BoYjL006YrtxIZWR VsSyGQhSW7cL19AuEXNwVKmULWycfkr7kU4POPAgtbT6hDzlZniUKpxVu1Hu98jAvRgz bcxq0t9OKK2J7/3O0lRl4dNTg073Sp+ojK3rnWcxuyGW0RvPh9anMnsNc+W5/rLzSMsJ Ho/St/2m0p0zsIWZF5QsNOVQeuQXkfsBP43+XpjCQzZI9euWanUB/WzPnytDjlkSAgXd IXOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=VtvBM9DMfGmETnWX1wzxnRTSKv/qQtxGj2Y0EKXuxbqz98Ckccuh4yMgPkkIOgemno WCZa1FLtngnRw2bbhj+qruxBFYz8xytvF1YaC/Be6UG6cqNJydWW7RLagFnovy29yZ+T JRYqb/g5NvynNgSCqFgs9JxsdcH9LyZiMyKJFafg8DFCB09tMvMpUR4SyQEzqFejDX7Q S73tWAg7MgPkP5CN8BKky5yVY6Tm9dDkavKutyY9YVHhR0ZMxKNMSw0Hn5uMP132lset WRApnaY7xdlkdNQQvIXd2OAikpC6PKjQhXuNW0K60V3O4hiA3FwaZbyBLEHw487RB5WO w0IA== X-Gm-Message-State: AFqh2kpa1fe0IMJI7d/Rg19zF4/lkTZgeunk2h5M1lWQYsScfsG0MAhb hg5f3Loru0L677V4jJs5/GQvskJ2+Wc= X-Google-Smtp-Source: AMrXdXt7ky1YKmCHrEvdSmppryHuxdZmy/ExGSeixqws3R/QJJ6NNyUiC3cubRusLU6Amdx/dq1XzZif1cI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:1d47:0:b0:3f4:6c62:18d1 with SMTP id d68-20020a811d47000000b003f46c6218d1mr3383963ywd.491.1674848493581; Fri, 27 Jan 2023 11:41:33 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:45 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-9-surenb@google.com> Subject: [PATCH v2 08/33] mm: introduce CONFIG_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194142_779656_98450049 X-CRM114-Status: UNSURE ( 9.38 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This configuration variable will be used to build the support for VMA locking during page fault handling. This is enabled on supported architectures with SMP and MMU set. The architecture support is needed since the page fault handler is called from the architecture's page faulting code which needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index ca98b2072df5..2e4a7e61768a 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1211,6 +1211,18 @@ config LRU_GEN_STATS This option has a per-memcg and per-node memory overhead. # } +config ARCH_SUPPORTS_PER_VMA_LOCK + def_bool n + +config PER_VMA_LOCK + def_bool y + depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP + help + Allow per-vma locking during page fault handling. + + This feature allows locking each virtual memory area separately when + handling page faults instead of taking mmap_lock. + source "mm/damon/Kconfig" endmenu From patchwork Fri Jan 27 19:40:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 198FAC61DA4 for ; Fri, 27 Jan 2023 19:45:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=h+Q/eTuA/USe+zl0568TCCwMacsTIXvMFBYHfw2BkwY=; b=K9eDak+UTOdIhbrEbpB1yIenAM PQR9EYk8SOn2oJsInuG+BQ1su70PP3R7kKXo1yobhoOBJXlh/HQqF0F1tt3HKGG8B0rptwnXfQF+h 67iLRySnIjUJJ9DBtFjDmmeziro41o0oCW+7eIy0WtGeEMVHDRUh4tfz+74bqfp1G5brtU/Cvatqc KEhhvD0NirYv/jtgwktW7XrVDiQHy4y2dp1tMX9zxbun3qB2Z8cNLBUW0AJrreRyCRqHzRM7PHBOu KXLkhdJtQdVHmzJtMoA4tVFPRamfFXIZ8q1F8IEJs4M24klGGKPoLgq1cdjAHQRa3egyYbZtozvrz 2HRL7fpA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUdo-00GAAO-SR; Fri, 27 Jan 2023 19:44:05 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUbd-00G97C-Du for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 19:41:49 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=Nu9/mdu38G6WtBjXb6Do7xpUyt6CdrDbmAUDhoyp47k=; b=HKl5dcsfIMPqFWUcXhXKAO38g5 vaN/N4c6mUUuNb7kWyYklX3T4RWDKeTip+qnES+N5wa/P1ZDNhNxXP62LLwBaLSEhmdncB3/dwvc9 fEY7sL8bm836VB9ioN5g+ekaBIcAOP96cC7ZG59eRrnpbRTGNcrQo3z4dXXsKFTGlv2bTYaiz06oj eqRuavUb2Q9wyuKH+qC288Ien/Ew6XcNJz1zTSWgi7STKSxyBsQw0z2xH5drd5jyZ5epJp1iCn+8b 4hRzmmt+HhEZmeZJe6AoAd5WB1eta2nnANWOac5cVIIxf87xsPFiZ7edR23VtTfgcSTA6+ksJMtAd oFZHqAEQ==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUbW-007zFd-G2 for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:47 +0000 Received: by mail-yb1-xb49.google.com with SMTP id k15-20020a5b0a0f000000b007eba3f8e3baso6326840ybq.4 for ; Fri, 27 Jan 2023 11:41:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Nu9/mdu38G6WtBjXb6Do7xpUyt6CdrDbmAUDhoyp47k=; b=oFyOjlhEa6jPD1d+pKzu8xqMu9sShcBJM0zkcdXAO9bWZHfs5hDegJ8sEKWFr4AQNl iGPCIH7iC8aR5RiIjRoH9W2KUncZPWl0b5vsJXQEB2UE60mYiP5ldYuNlEW3dM96CgJp +2KdvoCyVN7G6APzAMKWAfXHTUhAA+JaJZhJgfnGwlr68Zr0GLZrYj2YnOQkKo96sWGj Ng0DOWnRANnP+XQns5edhVeWyYl+9EdfDgkHbYKOkVF++Ku/0IaWanDTI7HqCQuL/w5R edw8ZGqmSQJCCmVgs5lV30HhGrOZqx2K3qMJgNbHCqjkT1CdH2dJNnKk0QEKLZZ9jCPS y7/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Nu9/mdu38G6WtBjXb6Do7xpUyt6CdrDbmAUDhoyp47k=; b=75E3FbKgfT20602nj66EZjUQN76Pu7B3nE5QhHvPUxbqWuLuMwOgY8iIWhgZbrZNaQ mDJ9NbKxsRDJPxqJUs9FqsNFnLS45CT7Nkomxq0lKAAWWmja8u6D5H54pArFywlf22CG erePGraGf2IZU03qBf3+O6M3skj7/iAIytrMXhh3GDT5KIWjp0doWT4dA8M6NJ2IBezF Zc9sQBjhlG08/sKh+sk6pzWYXFYJlXANyJgqDBqGPyZoVt7P/JxsQg3Ss08+zF8j/bu9 a1Je0hETX87n1Y1FTUS8wZTKLYSp8r0gPitBYCKvF3XH35++mxns5HyuwCZs8KdyaCj5 k5Lg== X-Gm-Message-State: AFqh2krdJcRmkYdErFKB/xvJOtKln/pcv6trLgF0rEUFP02IItiU138U lMJNPiX7LJnfjomztDmj6Iu2wcInLfw= X-Google-Smtp-Source: AMrXdXtMMT5r56SGjlxKGvDbQirEmfUIu1ilIQcv1/3wi/4TSk/ascWeVtYTcuwkoFwuriGWfFUnQoKWHlg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:cdc7:0:b0:4df:f712:5d57 with SMTP id p190-20020a0dcdc7000000b004dff7125d57mr3818080ywd.303.1674848495850; Fri, 27 Jan 2023 11:41:35 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:46 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-10-surenb@google.com> Subject: [PATCH v2 09/33] mm: rcu safe VMA freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194142_972578_DC7D6E63 X-CRM114-Status: GOOD ( 11.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 13 ++++++++++--- kernel/fork.c | 20 +++++++++++++++++++- 2 files changed, 29 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8410c3052148..62e413f84011 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -480,9 +480,16 @@ struct anon_vma_name { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start; + unsigned long vm_end; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct mm_struct *vm_mm; /* The address space we belong to. */ diff --git a/kernel/fork.c b/kernel/fork.c index 5f23d5e03362..314d51eb91da 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -479,12 +479,30 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -void vm_area_free(struct vm_area_struct *vma) +static void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); } +#ifdef CONFIG_PER_VMA_LOCK +static void vm_area_free_rcu_cb(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + __vm_area_free(vma); +} +#endif + +void vm_area_free(struct vm_area_struct *vma) +{ +#ifdef CONFIG_PER_VMA_LOCK + call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); +#else + __vm_area_free(vma); +#endif +} + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { From patchwork Fri Jan 27 19:40:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119294 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 43820C38142 for ; Fri, 27 Jan 2023 19:46:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=p23McDkSB/91fspNv/277rMIezCczqPQQTrSHXEX0Ss=; b=TEOuc4B4d2LCawUpk/5fuMd9vb jrVdeE/ImCKd8TnMhd+6j7qWwD5Iya98sCksaW3sPS/M6rFWnr4k5Ap/ppA2zg8Nq2V5ZtRlLtYus KxkqvkCXvdZF3Nuf4KUoNfnjOO77kICpZhkus3fbmOy8Ab1JEZynkXQMHZ671LhNLIwtxsenly+R9 k5Zxku2yMMFyaQK6wNPdw8m4/nnPtBmqrQac+cAgtHyYcXzFYeiUCAO/UAsxtpVEzqBDYVVNQ9wYI QlWfeZ7srUeDR0qv7m9jPA4334HH7U+ssJRMa81NrveBdNu3ek+Y8M5L59QONtG4fXlfbfRUMn3Ol X2X0b3ow==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUey-00GAiM-1U; Fri, 27 Jan 2023 19:45:16 +0000 Received: from casper.infradead.org ([90.155.50.34]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUbh-00G98l-6O for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 19:41:53 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=NwAvtCipHsAYlHTRgh0o0tNIOc uFL37LEQcw8ufWaIHK0qbjyodMRN+bmMR7HEZjF6O3OqJ0YhDo/40iXhNTa+UvtbLrfgk6yB/cIyt GbfPUQ3854cKA9bLAI+GqG9bPlV87980pEzFatCxVUkZrikFKb1vChPcmFVQXaERo9KLv81kjeea8 +vXDyM2ZQhtG2S6dUDaszWlQNcAICFaz+BYvMWF4/toneZqSocbrWdzIg1gf3WUYFGQZSAI1uURkz hQUXZD+Qewbyar7P4aXN+9VG5l4BS9Y4kP01IMsEXJPtgyoM54AhKZfTTXZQWfxEcr99sQv8Kpm34 TdgzhkdQ==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUbX-007zFj-KH for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:51 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id x188-20020a2531c5000000b00716de19d76bso6361387ybx.19 for ; Fri, 27 Jan 2023 11:41:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=Q7jceUjhtfhNXIUFz0/MzwmS5ZgtWEAK+cBK0blt69F8+KIFs/AwOKnAqPB+IEDKT/ FyLbx1gLVqD34Rlv7gRilV5FNygR87+qF/41EC627UqwDLTg6NJEFnAj/1s6LMME1gBy h9D9urWoN0e32uyucIiiCDtt59fZ91UJGpShxrAxprHccHpeDkPGS432b78Iuw1rAi+Y PGxS0P/Csnmusy4TpNpOjbr38NhV2uRvkkx/I7vAEJVPhb/fpP3E3BB7l9N5Jq8Ddy4K G0deOaLN7bYZJlZOXJL3xM94KhFiD/71EdmjsD4THVYmj7eB4g9ZYLPfdvfcp6yLyVR8 Oqkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=X1PGNm9/5Vp96jz5AKnoHIdC7ncZbogBD4ySliHR32jVnvP4kVPUBGvzrxhnQTCuny 1/S8R8sa9vuDNizx1Kh6bu8b3BZUu6hWLJKceqKgzaTt6DV0JBuEgNDj1BOOt2R02Y9+ mSRFS8qdEvREkEeD8SIqh8zoWzvQLcSsuMn55hBY69acJ66IJypiYl/SOrVumTrLIpfP BQldgxxSVSAjmk3ShziYvEZrC7fiqdSKwjKHTZZNAYdP2PBYiQJyxOhicIxgOZb9tyly yk1fdprEGlId5EVNDWe9obk7M94EJWWhaaacJIFrr1EeA2U09NvncykNhfgv6RQqQy8G M6lA== X-Gm-Message-State: AO0yUKURMUYYsUyIT6HUMrsUtKr8bc/tiTFrc2UpA8tc+SsBH8mlmOmb mLrmp/J6SRY76T1Ic5N8rZChw2alw7I= X-Google-Smtp-Source: AK7set9LIYXWd5UhR4gu2cf1OYz0XpBzDlrTVxbQWEhDgYvTk1OPOqMByF3iTmnXKTsrl+3RDXXXUtjCRT8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:f2c3:0:b0:50a:468d:48d8 with SMTP id b186-20020a0df2c3000000b0050a468d48d8mr731943ywf.104.1674848498315; Fri, 27 Jan 2023 11:41:38 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:47 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-11-surenb@google.com> Subject: [PATCH v2 10/33] mm: move mmap_lock assert function definitions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194144_116691_B2EEC9C2 X-CRM114-Status: UNSURE ( 9.21 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 96e113e23d04..e49ba91bb1f0 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -60,6 +60,18 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write) #endif /* CONFIG_TRACING */ +static inline void mmap_assert_locked(struct mm_struct *mm) +{ + lockdep_assert_held(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + +static inline void mmap_assert_write_locked(struct mm_struct *mm) +{ + lockdep_assert_held_write(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -150,18 +162,6 @@ static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) up_read_non_owner(&mm->mmap_lock); } -static inline void mmap_assert_locked(struct mm_struct *mm) -{ - lockdep_assert_held(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - -static inline void mmap_assert_write_locked(struct mm_struct *mm) -{ - lockdep_assert_held_write(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - static inline int mmap_lock_is_contended(struct mm_struct *mm) { return rwsem_is_contended(&mm->mmap_lock); From patchwork Fri Jan 27 19:40:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EDC64C54EAA for ; Fri, 27 Jan 2023 19:47:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Fkec8cESq28/ElDQXhDeCZoSBPcmuEnnkYcfUlsE1IQ=; b=pE3X+MCxhlKjIuJeHAAm2Bddxf fg5UDnhF1mz0VPDiYlEV0Ja1rTWtmYJkU+MKYs9h2Opio/VZq2bOMbtudDA5mp/hBWdZrcO8IfdtV NoRhqw5LbHOFDNGUTSs22BHdfmIg6AKpJNsCiewU5+JPxKBWTkk/42dZp1Qyd27JQH2d6Nph4zgiR EojrzBtrOlbrR8xc1+rr3un5C15oXIkelYu/b9ycmJ1eggXfGEixhj+XaVZ32ZE1tnpDGEbbPMxqr /60ctJFsza41dTOnExaOfsZa/JnJGMH754PsVckV8LHcyTfp8a/iiUd8MbRH5AmfJ2M4lS+RSTdXL z2isGykw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUfS-00GAxM-Ie; Fri, 27 Jan 2023 19:45:47 +0000 Received: from casper.infradead.org ([90.155.50.34]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUbg-00G98k-Hd for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 19:41:52 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=XXZYVjn5Y9alAXdawo6LSRZijnPMVBvcI/Ib+FHEz28=; b=sVnRgIIqtMUqfjcLTv5SdX6asO 1dCNf29eJcVzO/QprSirWwYXU5Fj06KLkykWfS1nG71SLuMcxWGJbuYCdmGMnk4oVPwsBphp50wgs zBmTLlnKXs8Z5jNAQglXWKlAkvtHiyg2p9vpfi9qF1EdfaGNYXClyDYk7KLvWjT43zdrMpddvL6Y0 aTRcjZgjjtleqEyJWQ3i4A7zm2T/SU8VzFZ3U/aDkuePPrYYmRPyEC5KpJE/p2n/hVNYiD4fhoz7I v1Q5D33ejMI59+2k8OIgLGIDIhSxOP6TH16A/J8/7/xMEzWEbwb45eLknz17fjjg1GWO209h8/y1e jgOiJ63A==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUbX-007zFr-3u for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:51 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id a14-20020a5b0ace000000b007bf99065fcbso6360493ybr.2 for ; Fri, 27 Jan 2023 11:41:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XXZYVjn5Y9alAXdawo6LSRZijnPMVBvcI/Ib+FHEz28=; b=dBDLlmFtxDGaSIxnsKi7TuOlfLmJDoa6COq6/jSvBgh64Qerq1YmsLWwhnAAxCyVf/ g582eX0Eo4U8zmrxOFGYnkJmijlE67J9D9xBXGrS1NtU3RqxGXmBh2f947WBs0NX/Zll emlcCuJwSkFMTmaE18K/oCpx+uIqpIqH/tqes3ZD00h73pebtBtpRPZksR0g19LqKl4v xmDEdLHNhpj3CDXqX8SiQjapJ6l/dJ1IUEAjdkc/2LeqJcIWhkIf+A2Qxlo3WsGI6Yzg srHRwUq16G0bEIrEimCIRBcTpWLvsmh9iUkFM5cCzPZIfcbsD+d957wvwGuV97WpJ3D1 vEJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XXZYVjn5Y9alAXdawo6LSRZijnPMVBvcI/Ib+FHEz28=; b=DKVnbPlM81uH2wU3LJpNigW2nLLkbAojZZTNxQD6Wlp769wS3mOVnAl+YKOdOjydtx KwVxoq2UiFUzDvEgQRg12b4H1GIN7o7y8DhOntUtzbAISMvQhdJvOPuOtD6p5boc7KKN ayaUAkGoyYwW0Pz+5cF/8avTD4RufYndZ+GCIsUZlBpuVs6DyDTZgerSJAEjNONU2d8R hm+WLUTIuZ0IjB6OR+d5DI1+0PdSqeFR2r2ij3ee1yBynem+nCIlW/apQX5JDBafipPv TTbUCopkP5B7GFeu6Dy7sl607FF0F1LE8cJp0G8/25nJHGDjmScbD8kQFmXft0w/RPNg EHow== X-Gm-Message-State: AFqh2koe3wM8oR+MCI3a9gC+mWXpN+Xo0EcI4oKAA6MJdPFu9yZyDJMf XPpD+0ePgMMGuOInjxKFoUri5uH7SK0= X-Google-Smtp-Source: AMrXdXuJ9Lmn9omVCHOpi5PBakoG2f5tpLAKUWWlsexTwKMDDrEOz6rknp2w/KZIAkkkuCmWH0rkraaKink= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a05:6902:10cb:b0:7b8:16c:e66d with SMTP id w11-20020a05690210cb00b007b8016ce66dmr5496306ybu.85.1674848500886; Fri, 27 Jan 2023 11:41:40 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:48 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-12-surenb@google.com> Subject: [PATCH v2 11/33] mm: add per-VMA lock and helper functions to control it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194143_611464_58E95A27 X-CRM114-Status: GOOD ( 19.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce per-VMA locking. The lock implementation relies on a per-vma and per-mm sequence counters to note exclusive locking: - read lock - (implemented by vma_start_read) requires the vma (vm_lock_seq) and mm (mm_lock_seq) sequence counters to differ. If they match then there must be a vma exclusive lock held somewhere. - read unlock - (implemented by vma_end_read) is a trivial vma->lock unlock. - write lock - (vma_start_write) requires the mmap_lock to be held exclusively and the current mm counter is assigned to the vma counter. This will allow multiple vmas to be locked under a single mmap_lock write lock (e.g. during vma merging). The vma counter is modified under exclusive vma lock. - write unlock - (vma_end_write_all) is a batch release of all vma locks held. It doesn't pair with a specific vma_start_write! It is done before exclusive mmap_lock is released by incrementing mm sequence counter (mm_lock_seq). - write downgrade - if the mmap_lock is downgraded to the read lock, all vma write locks are released as well (effectivelly same as write unlock). Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 82 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 8 ++++ include/linux/mmap_lock.h | 13 +++++++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 files changed, 110 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index dd295c020e85..fee08e8fdce7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -617,6 +617,87 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_init_lock(struct vm_area_struct *vma) +{ + init_rwsem(&vma->lock); + vma->vm_lock_seq = -1; +} + +/* + * Try to read-lock a vma. The function is allowed to occasionally yield false + * locked result to avoid performance overhead, in which case we fall back to + * using mmap_lock. The function should never yield false unlocked result. + */ +static inline bool vma_start_read(struct vm_area_struct *vma) +{ + /* Check before locking. A race might cause false locked result. */ + if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) + return false; + + if (unlikely(down_read_trylock(&vma->lock) == 0)) + return false; + + /* + * Overflow might produce false locked result. + * False unlocked result is impossible because we modify and check + * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * modification invalidates all existing locks. + */ + if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { + up_read(&vma->lock); + return false; + } + return true; +} + +static inline void vma_end_read(struct vm_area_struct *vma) +{ + rcu_read_lock(); /* keeps vma alive till the end of up_read */ + up_read(&vma->lock); + rcu_read_unlock(); +} + +static inline void vma_start_write(struct vm_area_struct *vma) +{ + int mm_lock_seq; + + mmap_assert_write_locked(vma->vm_mm); + + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + mm_lock_seq = READ_ONCE(vma->vm_mm->mm_lock_seq); + if (vma->vm_lock_seq == mm_lock_seq) + return; + + down_write(&vma->lock); + vma->vm_lock_seq = mm_lock_seq; + up_write(&vma->lock); +} + +static inline void vma_assert_write_locked(struct vm_area_struct *vma) +{ + mmap_assert_write_locked(vma->vm_mm); + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline void vma_init_lock(struct vm_area_struct *vma) {} +static inline bool vma_start_read(struct vm_area_struct *vma) + { return false; } +static inline void vma_end_read(struct vm_area_struct *vma) {} +static inline void vma_start_write(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -625,6 +706,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_init_lock(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 62e413f84011..88619c6a29a3 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -508,6 +508,11 @@ struct vm_area_struct { vm_flags_t __private __vm_flags; }; +#ifdef CONFIG_PER_VMA_LOCK + int vm_lock_seq; + struct rw_semaphore lock; +#endif + /* * For areas with an address space and backing store, * linkage into the address_space->i_mmap interval tree. @@ -633,6 +638,9 @@ struct mm_struct { * init_mm.mmlist, and are protected * by mmlist_lock */ +#ifdef CONFIG_PER_VMA_LOCK + int mm_lock_seq; +#endif unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index e49ba91bb1f0..aab8f1b28d26 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -72,6 +72,17 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_end_write_all(struct mm_struct *mm) +{ + mmap_assert_write_locked(mm); + /* No races during update due to exclusive mmap_lock being held */ + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); +} +#else +static inline void vma_end_write_all(struct mm_struct *mm) {} +#endif + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -114,12 +125,14 @@ static inline bool mmap_write_trylock(struct mm_struct *mm) static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true); + vma_end_write_all(mm); up_write(&mm->mmap_lock); } static inline void mmap_write_downgrade(struct mm_struct *mm) { __mmap_lock_trace_acquire_returned(mm, false, true); + vma_end_write_all(mm); downgrade_write(&mm->mmap_lock); } diff --git a/kernel/fork.c b/kernel/fork.c index 314d51eb91da..9141427a98b2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -474,6 +474,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ data_race(memcpy(new, orig, sizeof(*new))); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_init_lock(new); dup_anon_vma_name(orig, new); } return new; @@ -1147,6 +1148,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); +#ifdef CONFIG_PER_VMA_LOCK + mm->mm_lock_seq = 0; +#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index c9327abb771c..33269314e060 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -37,6 +37,9 @@ struct mm_struct init_mm = { .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), +#ifdef CONFIG_PER_VMA_LOCK + .mm_lock_seq = 0, +#endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, #ifdef CONFIG_IOMMU_SVA From patchwork Fri Jan 27 19:40:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DEFABC38142 for ; Fri, 27 Jan 2023 19:52:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=UP47d8UwhsS1fZ6SdliCZnZ9wWHK25/UNfMqzlM9M/I=; b=TLPNHr1DgRCsvFXmaJ1/B3Zeql SC3YcADaVIkEhBWBo3uWR4wBbjNVbtlStmZLV+dUE5BGr1jDerq6tw87mGOUCAmzziZ48VvpCGCw3 isX8llIlyQrYdJ7CO8AQ2ehRq81U+lrLOyggT355nL+2DiXUTzxUnxz8iG08aWm8ncdErxwRJSALO e9gwWrACyD9cWtiPFfo8aUxCv8/9eTzrWd0ujOZJ2v0K1s12YPuQRPdapTUGJwXhsa6M7H2lLwlrw aw4qZo92fRd9KXzX/TRI0YeC+HPJErKv5VADAtlciy7C8ZPZr+axlRgZfRIBowSwLJovxJR4bf4Qt 3nbAc+tg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUkb-00GD1q-NS; Fri, 27 Jan 2023 19:51:08 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUh6-00GBj5-15 for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:47:29 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-507aac99fdfso65654917b3.11 for ; Fri, 27 Jan 2023 11:47:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Hq20e7KS4f4C48PKmYYknVSNimtVVHYzyV8VwmVTJ0Y=; b=SD5yhaj9WNDyBmH2zVXxIQSAPvkdOOdT/eZZRKlGP77z2J9nVnn6Obfbxv5sauUEV/ LCtaz+vX4qip57892cEocWjVTNmQS4/TdyxbxT8xLbIWgkt6lFG5ZxopwUuxBJEnTkjr vQID9RMKTIQGW2sRQmcGmPfXjoMh0D1eDPyuL/b6rqwKxYdkfEYgEgHjQL0Tw8ZlMbjX ZH2/uoF+k+VRzud7oIym//RQdtO+lmKAHdfjocpDVWhkn9piFYOBRbFkKQo8sM+osjEQ FxFQ0vmeBJ9JfrQeSA2WMn51BintgFmdPrgL1EZjLw25vLNDWAlkic6I+BYJ/3xqK9CW +GQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Hq20e7KS4f4C48PKmYYknVSNimtVVHYzyV8VwmVTJ0Y=; b=pV4MI+5DU8yoXloGH0D751CzUZdEcjiNRvcf1LcY3Ka/yuAM3ra2zBFtoNTkW4gnQk TOAEyyi2BqomUIgM79vWqe7zHrWFpZbEAxgbxN5CeCDHIMZ+lycuK1oojlctDPhPL2b1 oXssca5TiiMsR9Hz8hgzIynZjmXD4G4v1NBUqiOsdR8aiXcxojtEtTz+Y2GWhzjwif9u viAGyJFnqnJX9EDI3DyMWQ1ERjdaG17X3XdQtAmU7k7+qfu32HwqIT8LQA2WQ1btfNun Mg3ZMxlmQQ7tzjcwj2ix9vOrt5H56MU1rDC/en+K550nU3koUQpW42Q8uThONoC5iZZo MoIg== X-Gm-Message-State: AO0yUKU1/Jn/gp9IDWB1/IvkE/yJ9QP4r2aLm3T6A3KRHBxhLZd+w6Ch lWqvToRu3TuCOPFGnXvUNOBCDtBHY/g= X-Google-Smtp-Source: AK7set9CI7/3kZjfJSWqixfLkcUok8XHsx1YNSreUFU581ZlI+jbpFxEiZ86BfATPj+NjPN8bnYvtim468k= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:b283:0:b0:80b:c7cb:6d1f with SMTP id k3-20020a25b283000000b0080bc7cb6d1fmr1156521ybj.298.1674848502930; Fri, 27 Jan 2023 11:41:42 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:49 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-13-surenb@google.com> Subject: [PATCH v2 12/33] mm: mark VMA as being written when changing vm_flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_114728_091111_CCB89C3A X-CRM114-Status: UNSURE ( 9.44 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Updates to vm_flags have to be done with VMA marked as being written for preventing concurrent page faults or other modifications. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fee08e8fdce7..66dca140695e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -720,21 +720,21 @@ static inline void vm_flags_init(struct vm_area_struct *vma, static inline void vm_flags_reset(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); vm_flags_init(vma, flags); } static inline void vm_flags_set(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); ACCESS_PRIVATE(vma, __vm_flags) |= flags; } static inline void vm_flags_clear(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); ACCESS_PRIVATE(vma, __vm_flags) &= ~flags; } @@ -755,7 +755,7 @@ static inline void __vm_flags_mod(struct vm_area_struct *vma, static inline void vm_flags_mod(struct vm_area_struct *vma, vm_flags_t set, vm_flags_t clear) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); __vm_flags_mod(vma, set, clear); } From patchwork Fri Jan 27 19:40:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119314 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0C9AFC54EAA for ; Fri, 27 Jan 2023 19:55:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=VgIYx4yRJ2VtRTOOimmOl7cJN3/DUtoVp0+y8/Yho8M=; b=NRqmjAGDkmBO7NX6b+vMSREp+G mfv+J7r0gDr08raME0/ib21hmAuxblxcTqGJLm7WMtKTZzI/el67H60ID2WEbFIXH6/87qe8K3GXY vyfjtbybwmzeJTm5MbTHDCTyIYFuLTuHspC3sSveORBkBrxe2xoja1pkTksz1ufl36nJnFtvmxcf8 0PQGhc24Z/6J8GAU0Ai3jWgq3decxlsIEs8U01BF9BOpnRdwaFBMZ8tgHpZdyVK9qF9XM5cU2T9Ha FiKIOPtNVxoInj34JIPyuOgB+YtgOqLpma4jfad5IG2mzjkZKgmG9bm1s2MwsHeI66WImEzbvTOdj Vz175BlQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUnr-00GE9p-B5; Fri, 27 Jan 2023 19:54:28 +0000 Received: from mail-oa1-x4a.google.com ([2001:4860:4864:20::4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUjT-00GCct-VB for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:49:59 +0000 Received: by mail-oa1-x4a.google.com with SMTP id 586e51a60fabf-15fd4dc0ee3so2306195fac.13 for ; Fri, 27 Jan 2023 11:49:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=icGEQDhVNg8AQkGt0Dau1cb43VWZWeb/kUxMjHSAwGA=; b=nhjpt9VbHDPsHYg33OlJfuq5Iz35VmBZ3kiMxPcXRXZZYdOboFlR0spKtxfGWFFcLn eBkIzKgB8QHaANcEzDHfkdLN1fRlxg2tqUBsMTv/7fNevBLbu/43ueowkRgVsLx1vLRo uHdRFKLtGeWmcojM55WkW4Vbk+IWqqj37nvYWjKvQOLDRhcdlXynMiveqJDQteuGpWZz ETc6Sup2efRPDAlMWE74qe3dlQIJeV8JWIsag7a7AlDm28r/breAdd9LIL/XkRDxD2X3 yud8bBPMeadGbrZ7ftbCoegvJj4wUFszXo3BfVZSLdIr/XgKY9tYqvkkj2bzILo8Fvtn ItZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=icGEQDhVNg8AQkGt0Dau1cb43VWZWeb/kUxMjHSAwGA=; b=YZ3V5seGt3wL1psut3wcxwXH3j07FByjPMCOmgV5r1/x8yBHzGdXNZQMBAkruw/Dby DOCsK7J7myyb+5pUdbjIc3UxWzjkqjWG3zReHc6zGDyUJPz4nNopRrRdHTsAV6Ylr3EY H1zHFf9lGcBRc0MZmMcfB57FWtUCJahZyDw2TIE+uGNBaEw2pQ0pDyvaskfhTNrA4lgi Zs651TkAjyuk0ZDEugTQ3mPR/kmgEM1dPfU/GMnRQHaEfOl5Sa8ryTfI+B7V6ysm+GWR tIlYyA05EzrCAUHKkAS7zBaEQd+f7tn1CfnpY44f316UVauP+ckUCWyEWC3VBu2DwWif X1Xw== X-Gm-Message-State: AFqh2kpOf45aUtWOyRBbGtnHFrUKRK+dxlTLqtsFUpX5N37UgxD8OZwX BvmHWkTSe7yNuxe6jMY1hXFqyHHjP3c= X-Google-Smtp-Source: AK7set+R/81FxpFmvdAm43AdDVGV+lMO8EmoQcyFeqmQ4bL2GAdYwzTWfE7122Mwrx91JEcHzv5APn9iCmY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:48cb:0:b0:506:4c8b:231e with SMTP id v194-20020a8148cb000000b005064c8b231emr1848392ywa.80.1674848505193; Fri, 27 Jan 2023 11:41:45 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:50 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-14-surenb@google.com> Subject: [PATCH v2 13/33] mm/mmap: move VMA locking before vma_adjust_trans_huge call From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_114956_027048_1AC3E1D7 X-CRM114-Status: GOOD ( 10.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org vma_adjust_trans_huge() modifies the VMA and such modifications should be done after VMA is marked as being written. Therefore move VMA flag modifications before vma_adjust_trans_huge() so that VMA is marked before all these modifications. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index 9efe13d36df7..7467d691e357 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2910,11 +2910,12 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto unacct_fail; + /* Set flags first to implicitly lock the VMA before updates */ + vm_flags_set(vma, VM_SOFTDIRTY); vma_adjust_trans_huge(vma, vma->vm_start, addr + len, 0); init_vma_prep(&vp, vma); vma_prepare(&vp); vma->vm_end = addr + len; - vm_flags_set(vma, VM_SOFTDIRTY); vma_iter_store(vmi, vma); vma_complete(&vp, vmi, mm); From patchwork Fri Jan 27 19:40:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119312 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2DEE5C54EAA for ; Fri, 27 Jan 2023 19:53:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=EjNVuaT9RGCEGyqJiwE4p4oqsngw2OlnyWVyAwKZ1io=; b=Kk10u+mLh0E21zV8TIXqN5eBOL KB1WX63VtK17v2WpbzYvq26+NhSVWSHN7Y9sH79RE2aTq+oaLULMsc5kY1E+aChrac3TBiGh+ltnL 5cr1/SQK+fZ4uykrZgyGp6IIimOi0qghR2GFnP6HP1f7rzo9Fpfv4Xd41TBKe8Nhl1ihOFZrq3FM9 Vzja6fFuSPU9u8ZzdnJV9muOsS+kNlw2PpLzFY1GZbAUP9W8TPan875ccyjxzKHwd8jnBMkv4awJo h1fF1XcZoeeosIFMtxhLMfEskJS0zZjmdjv2p6VL/Buahj6ub/cnci0s+JEiPTIVDoWpozpuBSMdc 7vlRCyGw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUls-00GDUT-O8; Fri, 27 Jan 2023 19:52:25 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUhb-00GBwc-Gl for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:48:02 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-506466c484fso66095557b3.13 for ; Fri, 27 Jan 2023 11:47:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7Tx+qt9ZwCfzeziH+AxLsfIEHPwRgvIaxjFEfdwiAVs=; b=MwDUDcWNdtFYfmhAuC43tEYj+bx0/7zv78dJ0hpiWDqIAfrnPySCh31pwGtES+llPo g9aJ1BaJcPmj6QIFKuJtNB4+OKnKX4w74YBlWwFtrSbaxVsErI4QpqtlUvrQKvku3x0Y fv8bSpOOjUv/1+/JuPfz5M6ciXa9EK4OZHTaoyyN5QHozfr1J0btP+Cb6s+q8s6DTvgx Eb/YBgAB71iKrit1/2Z+f0K+0VaK6nNNkqEz24OPAMCTiIMwYOn1N+e1hh+02mLm2qhR X0SB7IZ09+W7stQU0DgYwMtwYdWCgN82KCjbLOZQQ4WgzpAt4BTc4SEqK+AY1F70NhR7 prsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7Tx+qt9ZwCfzeziH+AxLsfIEHPwRgvIaxjFEfdwiAVs=; b=oQS0jp2nxHFOebNWX2ljAJo+38GW1vUY6O9HdfB9K2hC0WZVb32JNYPsRBSWGHVcjq 5FUtnYgbhf+H/+hLgtSGROJ0fOLYP9JO07fLdszPVvO/9rdBVdFUy2FiEUmWDTC00Ura ULYfSeBRTob3I3MoemX9rvKzbu7kLqCuY1FRMXYs2Ok/TmlW2GOAe2qClDRQXyybkYf9 WpU0Ccv1TXO86FUMIbdXh1TKH8Ct6e7uS0XvIEEHlgmtDbl2JPgKZqD+eIP3ECgqUWL6 caZSlBQaVAx5YNnOd/NiIW8cnwq8DuZ18AYtw6kZn5IlrJrNzeyMzib+/vzy1S8dYHSq g5jw== X-Gm-Message-State: AO0yUKV+dlfzpaxXiP5a04doOtmhs4yMAh//t+GKHRbzdTfvkb2zsbC+ MYjTIbq0dXmt1IpwdwIeYx4YWMNFVts= X-Google-Smtp-Source: AK7set+eTaGDRvBxQnOso6u8cp6LCvB6g6dz+di36ExTAmlNkOKvBeltyp9kLCTqVY9tlqnocHmCi479DpA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:6008:0:b0:506:833a:c32e with SMTP id u8-20020a816008000000b00506833ac32emr1175163ywb.50.1674848507730; Fri, 27 Jan 2023 11:41:47 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:51 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-15-surenb@google.com> Subject: [PATCH v2 14/33] mm/khugepaged: write-lock VMA while collapsing a huge page From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_114759_626756_203CA72D X-CRM114-Status: GOOD ( 13.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Protect VMA from concurrent page fault handler while collapsing a huge page. Page fault handler needs a stable PMD to use PTL and relies on per-VMA lock to prevent concurrent PMD changes. pmdp_collapse_flush(), set_huge_pmd() and collapse_and_free_pmd() can modify a PMD, which will not be detected by a page fault handler without proper locking. Before this patch, page tables can be walked under any one of the mmap_lock, the mapping lock, and the anon_vma lock; so when khugepaged unlinks and frees page tables, it must ensure that all of those either are locked or don't exist. This patch adds a fourth lock under which page tables can be traversed, and so khugepaged must also lock out that one. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 5 +++++ mm/rmap.c | 31 ++++++++++++++++--------------- 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index eb38bd1b1b2f..b938b286cdc3 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1134,6 +1134,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, if (result != SCAN_SUCCEED) goto out_up_write; + vma_start_write(vma); anon_vma_lock_write(vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, address, @@ -1601,6 +1602,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } + /* Lock the vma before taking i_mmap and page table locks */ + vma_start_write(vma); + /* * We need to lock the mapping so that from here on, only GUP-fast and * hardware page walks can access the parts of the page tables that @@ -1806,6 +1810,7 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, result = SCAN_PTE_UFFD_WP; goto unlock_next; } + vma_start_write(vma); collapse_and_free_pmd(mm, vma, addr, pmd); if (!cc->is_khugepaged && is_target) result = set_huge_pmd(vma, addr, pmd, hpage); diff --git a/mm/rmap.c b/mm/rmap.c index 15ae24585fc4..8e1a2ad9ca53 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -25,21 +25,22 @@ * mapping->invalidate_lock (in filemap_fault) * page->flags PG_locked (lock_page) * hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share, see hugetlbfs below) - * mapping->i_mmap_rwsem - * anon_vma->rwsem - * mm->page_table_lock or pte_lock - * swap_lock (in swap_duplicate, swap_info_get) - * mmlist_lock (in mmput, drain_mmlist and others) - * mapping->private_lock (in block_dirty_folio) - * folio_lock_memcg move_lock (in block_dirty_folio) - * i_pages lock (widely used) - * lruvec->lru_lock (in folio_lruvec_lock_irq) - * inode->i_lock (in set_page_dirty's __mark_inode_dirty) - * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) - * sb_lock (within inode_lock in fs/fs-writeback.c) - * i_pages lock (widely used, in set_page_dirty, - * in arch-dependent flush_dcache_mmap_lock, - * within bdi.wb->list_lock in __sync_single_inode) + * vma_start_write + * mapping->i_mmap_rwsem + * anon_vma->rwsem + * mm->page_table_lock or pte_lock + * swap_lock (in swap_duplicate, swap_info_get) + * mmlist_lock (in mmput, drain_mmlist and others) + * mapping->private_lock (in block_dirty_folio) + * folio_lock_memcg move_lock (in block_dirty_folio) + * i_pages lock (widely used) + * lruvec->lru_lock (in folio_lruvec_lock_irq) + * inode->i_lock (in set_page_dirty's __mark_inode_dirty) + * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) + * sb_lock (within inode_lock in fs/fs-writeback.c) + * i_pages lock (widely used, in set_page_dirty, + * in arch-dependent flush_dcache_mmap_lock, + * within bdi.wb->list_lock in __sync_single_inode) * * anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon) * ->tasklist_lock From patchwork Fri Jan 27 19:40:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 28C90C61DA4 for ; Fri, 27 Jan 2023 20:07:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=eO9zR3xMt6avep8T0+5Mpk0SfAUsqPn1+aZcncu2A5A=; b=I0S59HwnK5UmZ99QEoDRtKu8i5 ps4sGJc6jm+tXevcF2QGnEjFnm82SonKT37BcnMVGMu+3oV43f+Wk2BglDuiDLUW7st16ACcXLec3 S3IQznE3zqAWQdA9fILbRHuHOOeSuvzd/fILmlf62VPaaBZsZeqV1xfhd25EMVlPfmbIVCvtFCjVL q4IDezvjjM9ofyChnxhGN4lqSvnTs9tE5Eo/ias7WxhFi9PNtRfXFYnAMLKo9QvAaaNJPV44OZRUk BicEhkPEw9DVPFpp777+s6w3pJXxq26d2a9RGkYXsQQJqzb1FAjxJFp1eaZQ5czYXFEqyXsUrw68h 4wGAAqPw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUzh-00GIcI-BL; Fri, 27 Jan 2023 20:06:41 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUwg-00GHGG-OD for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:03:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=xM1gV9tZESxmDxiysmAmLZq38ZaKZXSlDI1GeX9HKic=; b=R1jQiRFbFyzBWZOn+Cj4k0pPAY zgqKz7OSa1fAex+YTRBc1YEBdizsIFI3f5ciRtDW93705NwvlBFsQpQqvMNJR1SnIp1mmRT7XhixY YS76LY1TsCdJwDKD6Z95/bLB3PPCAwoCFNgVPRHSfd03CEjaxIQ45bTFAh1PyI0yGvv2jxeY7GzcV oYVm3QdpKPXO+BR06n+QalbkhZyGRM4Wue/exswKtmi+ARF4hMje93t2Q/MhGkkUh0DF3q9Qfa2W4 aFNo7V7r9qg4k6+HzEwrpV+GnrxAr1GZiU4dsQi28nRB+upXyVxh/tkc0us4DFaMD2hD6bUQUwSdA dYFnepRw==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUbA-002xqc-3A for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:23 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-50660dd3263so65573927b3.19 for ; Fri, 27 Jan 2023 11:41:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xM1gV9tZESxmDxiysmAmLZq38ZaKZXSlDI1GeX9HKic=; b=lE9qRR84U5+rYrRmzpPAw7Ud6uFSOEezLerrdXzskk/7N7IWEzIS+P8h33l5/GHLwm pzf0b8p0OfVWGzGLb/dMUWWv/4+TG3PN/VW/HgmJDf4GTYQxz4qhZvzGmSe6iwYXQbEv pmRWin5z3kTr/dQHGmOmQouL9X7gNxEvHMfX/Mw4+j3xs8bwN3ke4zv6/7Jpk/cXGw8a hnXfsAcVwtJnLNS//9cNJnJKu2XJduYEuopAyOxXqZTZsrcp23rvXM55Jk6t7ntayedf QFH2QchxeUajMjFYvRPLkdcrU5vfcJYPLkQ5Wb/mKP36Ds4BBv5V6MGo7zAGbp1SePYq xw1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xM1gV9tZESxmDxiysmAmLZq38ZaKZXSlDI1GeX9HKic=; b=UN0p+Q8cpLmIDNfqFCKvt75WdfeyvqwJSN0mLAkNcl7keotq7l5BAgO1uFn1J8mJqO Jv7pm10N32cuMJSVE5xlW8H2AO3XolLNnSpmCCksnZH6OTA+ENIjbUIgmBZbvLUc6xSd v/GDLVJYyiQ+5eAHbWHkISIumKBMMPCTOU1pvBMq9KeP9XNzRJW6+higbr8gZsa8OPET lnOXGbNg94mPuEIcUO7WNbA1W2pzpcGqDrknJ7RJbZX7UH5yvACItLxijDN4q+cgLVka aIWV68oFOLK24Doma5059VD6qAoEi15aNNvuvC82uHaqrnDgMNkUkqGSgHv+L7MbmEZ4 qsVQ== X-Gm-Message-State: AFqh2koTa4hEnxoX3uA5GVwWRRbSoJt1OYm5tOOPFis1tJ+M2AD6ylu3 S6jhvhPiFUHNkhraH3PfXBPI5UhOrVs= X-Google-Smtp-Source: AMrXdXsibCiIco7YlC7GUbshkOFJQAk8PpTkup371cElxWyegDeUsxNSE/XyOxE1JUpmHxGWPZOyG9ioimM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:b94d:0:b0:801:e503:dd0c with SMTP id s13-20020a25b94d000000b00801e503dd0cmr2559229ybm.384.1674848510225; Fri, 27 Jan 2023 11:41:50 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:52 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-16-surenb@google.com> Subject: [PATCH v2 15/33] mm/mmap: write-lock VMAs before merging, splitting or expanding them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194121_131641_97931DA0 X-CRM114-Status: GOOD ( 16.21 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Decisions about whether VMAs can be merged, split or expanded must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Write-lock all VMAs which might be affected by a merge or split operation before making decision how such operations should be performed. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 7467d691e357..6fff76334177 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -269,8 +269,11 @@ SYSCALL_DEFINE1(brk, unsigned long, brk) */ vma_iter_init(&vmi, mm, oldbrk); next = vma_find(&vmi, newbrk + PAGE_SIZE + stack_guard_gap); - if (next && newbrk + PAGE_SIZE > vm_start_gap(next)) - goto out; + if (next) { + vma_start_write(next); + if (newbrk + PAGE_SIZE > vm_start_gap(next)) + goto out; + } brkvma = vma_prev_limit(&vmi, mm->start_brk); /* Ok, looks good - let it rip. */ @@ -912,10 +915,17 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + if (prev) + vma_start_write(prev); next = find_vma(mm, prev ? prev->vm_end : 0); + if (next) + vma_start_write(next); mid = next; - if (next && next->vm_end == end) /* cases 6, 7, 8 */ + if (next && next->vm_end == end) { /* cases 6, 7, 8 */ next = find_vma(mm, next->vm_end); + if (next) + vma_start_write(next); + } /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); @@ -2163,6 +2173,7 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, WARN_ON(vma->vm_start >= addr); WARN_ON(vma->vm_end <= addr); + vma_start_write(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split(vma, addr); if (err) @@ -2518,6 +2529,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, /* Attempt to expand an old mapping */ /* Check next */ + if (next) + vma_start_write(next); if (next && next->vm_start == end && !vma_policy(next) && can_vma_merge_before(next, vm_flags, NULL, file, pgoff+pglen, NULL_VM_UFFD_CTX, NULL)) { @@ -2527,6 +2540,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, } /* Check prev */ + if (prev) + vma_start_write(prev); if (prev && prev->vm_end == addr && !vma_policy(prev) && (vma ? can_vma_merge_after(prev, vm_flags, vma->anon_vma, file, pgoff, vma->vm_userfaultfd_ctx, NULL) : @@ -2900,6 +2915,8 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (security_vm_enough_memory_mm(mm, len >> PAGE_SHIFT)) return -ENOMEM; + if (vma) + vma_start_write(vma); /* * Expand the existing vma if possible; Note that singular lists do not * occur after forking, so the expand will only happen on new VMAs. From patchwork Fri Jan 27 19:40:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119296 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 64E1CC38142 for ; Fri, 27 Jan 2023 19:47:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=KzOY7n7V4snnXHAzTTDG97PQrbOGTB2PLxK+BpBs8Po=; b=Xl6KDIkwp0X4mJi1oO/3OJ3MRR WW+RaJsR4gn45XPzzIfce+2naN5WX2+25azQi57HLK7Bqi47f+9ZlxqOPf/7QWymvRxVJ8PlWBisV Ke9DOLvNv3NAA+JNRuZbw+rMgqNGW23NU1OzwhjC5tnIuQ1VUFHs4rGhRRywKhVOi/ukNmANYg+/K cLxpqzIyD1HlwG9xgb2trUXQPeEQwfkEhjovtXcfDyvPvG0OZMB1TBLGQno6jG27cXuYyuGLNKhFj PNcp1q+KwK8i5O/cSa+GH+SsNBvTkWfv0O5m2eeFCOQQMz/xtsa8JvGldXtoTtzp2zD9xsOimlw8A w8VoJDIg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUg2-00GBGV-LO; Fri, 27 Jan 2023 19:46:24 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUbj-00G98u-HK for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:58 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5063c0b909eso64913087b3.7 for ; Fri, 27 Jan 2023 11:41:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=z2/pTj5C3cgEFiOJp2CeCn46OI8ibDfme0ccGGe2ykQ=; b=Cb3p2QcNsUnk+wL6/K6P4ED35dG3ZUcB734zcAsddvwr/PVSkrRYzepHFsLMU82wqr DCByjFxSWTXuXxKpPzrEyhOes5tD65SfDW58DVPGzZ4bl/GM62EXvV72BYyh+FkQftRL 9ik0iporDuduJD370bXV4KniJvgp40Hs/4xlWBjuzB4w/OEbzQnMEDOcEM26lHODokML 801aFmGgsonG3Ea3SKObTIkdYEG/WUj/siwNQ/Elnoh6/xVR3ypDahdEx2v1JUXFD+l6 cbwz8ftG3Ry5zhC62QLkQy0TB2bZZjts/vu8EdbF1X6krxdoifCb7qgLuiaEVqlGXvIi PdsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=z2/pTj5C3cgEFiOJp2CeCn46OI8ibDfme0ccGGe2ykQ=; b=MLitAcZg4e5Yj17gA9QyoTm8Bw1JZRw5I0577A+CGNBWZ07emwB1590q12Gm9gYWLe MSgYFMvUgJA3iG2Fyaa6ZWXelQx8QM9PCUX6J+q72a6GKMb3me2FNKxAd4I8/UDH8oKp T7fZ1vHD2cIfmjWp+6q2Az2JgaWeJ55HxVXvYRxWTohZnpgTXw1ry0TOS3MFfmqfJFUd RxBIgJCkZLAcMnq6ZZWeSz8nncevFk+XAGFjrEgPu73mL5vMm3jc+TZUEIY+HRCPz6fI fBeZKmVhCks6EcWVpPxHerUoOnQI8Kjr7vSZrlfgZaBQIxvIYMTYI8M3x488vT03zynR lbjA== X-Gm-Message-State: AFqh2krSfnK3Lo3STgEEOs35GMwMMdulr89JX3MjoH5p1KnyKNzJAbUr R4omDXVdkpZxShikO9EI1GxH64Xx0Fk= X-Google-Smtp-Source: AMrXdXvc0ioDltykhYGthi7quhcG5gFdvKFRYGkIf9fgUkYhxxt/rX6VwG8N7AhqKPLty1w08hNt6tf3mAM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:f6c2:0:b0:4dc:1241:cc0a with SMTP id g185-20020a0df6c2000000b004dc1241cc0amr6165876ywf.499.1674848512603; Fri, 27 Jan 2023 11:41:52 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:53 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-17-surenb@google.com> Subject: [PATCH v2 16/33] mm/mmap: write-lock VMA before shrinking or expanding it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_114155_770163_3AB0337A X-CRM114-Status: UNSURE ( 9.79 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org vma_expand and vma_shrink change VMA boundaries. Expansion might also result in freeing of an adjacent VMA. Write-lock affected VMAs to prevent concurrent page faults. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 6fff76334177..60038c24d836 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -674,6 +674,9 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, ret = dup_anon_vma(vma, next); if (ret) return ret; + + /* Lock the VMA before removing it */ + vma_start_write(next); } init_multi_vma_prep(&vp, vma, NULL, remove_next ? next : NULL, NULL); @@ -686,6 +689,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto nomem; + vma_start_write(vma); vma_adjust_trans_huge(vma, start, end, 0); /* VMA iterator points to previous, so set to start if necessary */ if (vma_iter_addr(vmi) != start) @@ -725,6 +729,7 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) return -ENOMEM; + vma_start_write(vma); init_vma_prep(&vp, vma); vma_adjust_trans_huge(vma, start, end, 0); vma_prepare(&vp); From patchwork Fri Jan 27 19:40:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E7522C54EAA for ; Fri, 27 Jan 2023 19:54:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=PezS2bp3K7KL/uTmJhdeak3axydyA1gF0lebRQL6lK0=; b=XbUnqzQH8MUY8mijBCHiJryJp7 J+sLzoaWmuV6SobwXlyEfiridgTLDush8TbM/diu06kMi9L0XrBeouqkwXXxLgK0yKQC99Igdfv4w qJkZYrZbP16k1gl75hzoBmrNK7htjz7MkA9SnMvILUNhd1biy0ZFxoSEm6vjAQgic2TfyALCPZGaT azu8JkNZe/JiabVXq4wZSujRdLP5W8JFTuO9WXN1bfiE9Bn/JFu0baBfja0hEjkM3xN/gVrOuY+OO qDVVpBgX77g3nnhf6igAGA/h3oAeVSKEbj98uJMDrc5UtEXJR2ZzjkBjpt4qb8/K29Iy40IVwOdvb 82uWwJPw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUmn-00GDn2-95; Fri, 27 Jan 2023 19:53:22 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUiz-00GCTH-VN for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:49:27 +0000 Received: by mail-yb1-xb49.google.com with SMTP id n8-20020a258d08000000b007facaf67acfso6410320ybl.0 for ; Fri, 27 Jan 2023 11:49:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=L4ezA/DdKPqYrQ5cS+Ke2gtcDFDEWQ5+mTqJ9joAzRk=; b=GKKENfMfMG4N0pZCcl9OmPXOIcATJjfZsJmDxgzcwEK7KGqwCsad+AWtcOXvaoC0br qvO01w86MwzjSygGqi8PA6cojX9dXaSV7GQ7IPy4D+7hoAEyyoJoE4vESGbmz//KocjS XInPKVcNcxUU/7v9tYz8Eq3N+worhIjUrPeLT3iGbzfEm71MChhnFLrlat60IF4TyfFg cT/sx9cN16Dq7Csnqgd8AQwbf9+2MwGgrR42iD3xwMMpBpotxh+QKt5ySOsx98KoL0iq FNYiUMi+Hv+PIq3dZA5NAAwxArkgkUL/euE7T+pJZglO6ldDOiv8hb2KxyDrVWXCBL/Q MOaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=L4ezA/DdKPqYrQ5cS+Ke2gtcDFDEWQ5+mTqJ9joAzRk=; b=DdCnhngQcxj65DP4BlN+qsrPVPgvET6vwhy9Zwf8aA7kGMsFzvIBAuMJoV+GoJ/p1a 4fi3IjRp87JtdnTiw0tQRXJC1CVE4H1mJfu2AeOGpw7o8WdrcQyDHzDFl2/W2ImndVLi 4cgXcIHp4JeZLhQvwyfCMxqBa55dfT5tlhY/LfaO+UVw2e/Q64qpy7QeKXjk510u34xP SCHT3d9m3e1dGtiVn7dO0Mz6NheY71kQAZ0HjlgSafeLktAkrRpus6euYCkOTByR0xVe qKxDIN9I81XhSp18JapnBtPDnr9fnKIxQKv1t9OAlMG2AJbg5mcTq2mzYqmpqKCyJxag jlEw== X-Gm-Message-State: AO0yUKWH/iqwgpPFEHO6DfLMVxHe/yUmLjGc/EuUhkTNxMuvHdkPKYoy oJu7pYEXaghmO5UDoeJfW7bc3EwxXbI= X-Google-Smtp-Source: AMrXdXtaMEzriXRGDDaPyWp6cI2qFr9C55J9V7x8prQAmgawr911NF2wOHqTeINFz8M7FkUN1SViGFe49Jc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:3c9:0:b0:803:3696:c473 with SMTP id 192-20020a2503c9000000b008033696c473mr2168756ybd.224.1674848514947; Fri, 27 Jan 2023 11:41:54 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:54 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-18-surenb@google.com> Subject: [PATCH v2 17/33] mm/mremap: write-lock VMA while remapping it to a new address range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Laurent Dufour X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_114926_033348_1C7A397C X-CRM114-Status: GOOD ( 10.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Write-lock VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 60038c24d836..b3c247073aa0 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3202,6 +3202,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); + vma_start_write(new_vma); if (vma_link(mm, new_vma)) goto out_vma_link; *need_rmap_locks = false; diff --git a/mm/mremap.c b/mm/mremap.c index 411a85682b58..dd541e59edda 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -623,6 +623,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, return -ENOMEM; } + vma_start_write(vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, &need_rmap_locks); From patchwork Fri Jan 27 19:40:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B1041C61DA4 for ; Fri, 27 Jan 2023 19:48:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=qtE8XfBk6Oq7R1GOrtHMHlbG+aD8Ln9L7cpY0OU/opw=; b=euH0YkDNLx8zK2rX6gkMcLyq74 48v7m3euJJU0UeQ0SVCPYfYRBcivL5Z+uN8Y+nxBWXusD5KQZ9l2vWhmIj08U9/TyfLHYf8QWuVNG 09bRuW7yWN5ZLInavStrVBNSViZTdg4dgO0dileMBB2AMlpc0J8sCCi6/MMt2FVYXBuRrSGP0ybba oTZqBAdTkuQqBavUjG6zv7zTMttIcEZiSFoD67fx1mWufjwsuQkdgHABXjbTNpbQZiZwaK1DrAoGd gEkS3GO1kBmA3quaDpObaNJzVxNM9WST0LSqrISqxYd7ikQQCaY3OAYT5r04ZPN/3YiEs7gxsW7vh 4bZ5aM1A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUh7-00GBjo-SI; Fri, 27 Jan 2023 19:47:31 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUbn-00G9C5-2T for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:42:00 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id n194-20020a2540cb000000b008038647d9ebso6355855yba.5 for ; Fri, 27 Jan 2023 11:41:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=vIuOo+WD7hhMM8dbfJD4mGHBD08/Cqf6l0kIi97gWpI=; b=iI2a7MepfOBNDqB7wu+/83gKbYOOOUW8grtwa3CiYxFM9JfwE6yM70Ne3Bb3nWcBVw Yu75uY4jTmGekedOn8ngeIwpB83dCuwgdGAmo3PP8/5H9WQZ6zUPSiR4iOLBbFN7215j t5fY8PD6baQLdLtkjh6u6ca2lGsT7fHu5oxzuRWhyEjLoCc6wJIYtxHPj4OPz1R/r154 +v9JTIHiwr1S1nQG1EzjVKOQzPqMVCb1Q8Qi6vYQuYU/8I9rd1kQnHGC5kdzCbnlOWk7 FPBIbkrQ5+p0qEBCYmhl8CWtOia+KC/VA/z2TgOLyA8KXzLYreAkUsa906GcFZG1qYyr SCmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vIuOo+WD7hhMM8dbfJD4mGHBD08/Cqf6l0kIi97gWpI=; b=EGOSGcbx7rOcqi8HN4eXliu1Jq/eO4yBlEtgToWngN5m1Zz5G1W4BFLc6Sz+x/TQbL eKde7SgirIvwhp9nY5d7lxHENCyMWULQMBxFaEtRa+ilBzU4LW4OqBI8Wtd7QptXJW1+ fZtY2g0/6UXqhKswR6CHN/MA/AX2B11PzqZ2Doo/MTI3iBr3SdUpq4n8kR9WiK2gZBeh g3KJl640vqf85Nzab/xdC8Bip7xSoUsEOLexjijI+aGVDd9yoVUftO46rafSxYbEcv8l HI+i7C1Hl+8RlgooLyXjxGxwggSiO3IDDmfyew2QJtInp91fHxJFWWq+ru1acnWnKSFJ n8Vg== X-Gm-Message-State: AO0yUKVaNBTdq40kdInC0+dnOjqVUoaK0IU9+dPBjtlk2rJzA5cZRqqa y0Lz0lXjvim5A4EQ3zO4WGnjxZMn3yk= X-Google-Smtp-Source: AK7set91XCOXqw0Dg5UJpzZX/mqq3ueYFNiPcfvNNp2V4Tl7t8riuyYgZdoB/8yFEdiYxfiOzkRBBcWpyqU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:3e15:0:b0:506:653c:db97 with SMTP id l21-20020a813e15000000b00506653cdb97mr1675918ywa.201.1674848517274; Fri, 27 Jan 2023 11:41:57 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:55 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-19-surenb@google.com> Subject: [PATCH v2 18/33] mm: write-lock VMAs before removing them from VMA tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_114159_216377_ABA5331A X-CRM114-Status: GOOD ( 11.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Write-locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 1 + mm/nommu.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index b3c247073aa0..5bdfd087b632 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2261,6 +2261,7 @@ int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, static inline int munmap_sidetree(struct vm_area_struct *vma, struct ma_state *mas_detach) { + vma_start_write(vma); mas_set_range(mas_detach, vma->vm_start, vma->vm_end - 1); if (mas_store_gfp(mas_detach, vma, GFP_KERNEL)) return -ENOMEM; diff --git a/mm/nommu.c b/mm/nommu.c index 57ba243c6a37..2ab162d773e2 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -588,6 +588,7 @@ static int delete_vma_from_mm(struct vm_area_struct *vma) current->pid); return -ENOMEM; } + vma_start_write(vma); cleanup_vma_from_mm(vma); /* remove from the MM's tree and list */ @@ -1519,6 +1520,10 @@ void exit_mmap(struct mm_struct *mm) */ mmap_write_lock(mm); for_each_vma(vmi, vma) { + /* + * No need to lock VMA because this is the only mm user and no + * page fault handled can race with it. + */ cleanup_vma_from_mm(vma); delete_vma(mm, vma); cond_resched(); From patchwork Fri Jan 27 19:40:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A186AC38142 for ; Fri, 27 Jan 2023 19:49:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=+5BRMXxsTT+DFTGMWGMokMYxWMjDCxfJ5Y/KZ/cPyhY=; b=sfnJNFvAFGJuBOoXMiOuFPfQYn r+zSDcwFUMYq3amYJvBxDxJGvAYZLU1tgfW6oT270qYIG/r5PiKG7OED8IC7EAmeUFLhDfaggRoIj qB+2jdHtvv54xUc4kXlMkPzviTMspYbEboEArgRRs2ydmdaOtrFOrciDMjurlW//oRU4zO4Q9A6NE HZ+EJQ0HLuKBROnfgd6vrtJ68M+Fj+DXFj4adQqFUUy41/YTSaP0vEPEhVhsjsikENJt4uA5KUKop 2LoX4fySWn0F/kcCR0INq5gMHLd1xIBR6gssckDLlC2B4L9u1/IFsXKc9AvtIQwHtcL4bge6xCKRW qb/Y0pBw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUi1-00GC5s-SI; Fri, 27 Jan 2023 19:48:26 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUbp-00G9Dx-3w for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:42:02 +0000 Received: by mail-yb1-xb49.google.com with SMTP id g138-20020a25db90000000b0080c27bde887so2634769ybf.18 for ; Fri, 27 Jan 2023 11:42:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PbZvlTacNRMtTYgLZEFyk5N8KO4rHMcD13osXtjRKQc=; b=dIXUHHJyt3zldvcJ25OA8M/N2ekAkSadxrM4Vc86G1vAxHUN7FQNbSzlWzbH3EF3Ay Np/mC7lgd8yNrbdx+fJZdOiPs5Ax7Y+isaOSeA0W3slHJVunqRE0R5gvKOYJUJLOV6vy eusS7uxTBSUHthbBE/M+AcF029iuITRiXxkeDAE1bHTcC4qSxwHlMi32ImCbpO9h9EMV +wvjzOskJitBDhg128jcOLxlukATYCWhVPwB6m3ynKWMcyTvnD5SZLfuK2oGD5H+Rl4/ hAABUVAQozzvBCIG2lhb5atfzqUlxx9w7Rjl6rs3/T9zVK0hATJevSe9VLcJ9uJd0aXA HxXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PbZvlTacNRMtTYgLZEFyk5N8KO4rHMcD13osXtjRKQc=; b=F7LQnePDbvPenL39DC0EdQYbPK/9+tcdqmO1ePKXcxrD6o2VNf1HOkInRPvn+4Zbh3 5fXZnh679jxEtpMSF5RjC3B3UL7z+PkBLp74QTofnplWd823AetCclB2ViIIKgvN8VCD NRFqrAUYhG07MOMNetURwCNvAqa8iy69/3SFkDxzfI+aoah6onxS4cw/gheko5Wi/1RU 5D0quNxZmo+MreTkVAXPsUljeqSvlYEHX87Z7elefhPMY3yCsAUNU2r5wYwiQvxdBvFP 7MIGwi9Q2dFA5ddH4JGwO7OP6JMCr5SWndWaiI/Gn+rvuORcb8KG8O9ygPygjhffWdh9 eksA== X-Gm-Message-State: AFqh2koKt/33yGgjWGlwyu5XxJR3S6ZoPVhAucjA4UDYvmS3fB/dQJXB mKJSrraUcldbMLHOzHnt61oOERGpXSw= X-Google-Smtp-Source: AMrXdXtxOfNby/WJjyD0PPf8WyFmfovtGs47EqfediusuKgjOEAtqZl1SXyt3bJWMQsRbQEhpFz5LQpAf70= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:cd4:0:b0:4dc:e929:40fa with SMTP id 203-20020a810cd4000000b004dce92940famr5633708ywm.331.1674848519684; Fri, 27 Jan 2023 11:41:59 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:56 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-20-surenb@google.com> Subject: [PATCH v2 19/33] mm: conditionally write-lock VMA in free_pgtables From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_114201_211578_97C5944F X-CRM114-Status: GOOD ( 11.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Normally free_pgtables needs to lock affected VMAs except for the case when VMAs were isolated under VMA write-lock. munmap() does just that, isolating while holding appropriate locks and then downgrading mmap_lock and dropping per-VMA locks before freeing page tables. Add a parameter to free_pgtables for such scenario. Signed-off-by: Suren Baghdasaryan --- mm/internal.h | 2 +- mm/memory.c | 6 +++++- mm/mmap.c | 5 +++-- 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 90bb2078444c..52d7e9c2e58f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -105,7 +105,7 @@ void folio_activate(struct folio *folio); void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *start_vma, unsigned long floor, - unsigned long ceiling); + unsigned long ceiling, bool mm_wr_locked); void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); struct zap_details; diff --git a/mm/memory.c b/mm/memory.c index 7a04a1130ec1..d48c76e9fa57 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -348,7 +348,7 @@ void free_pgd_range(struct mmu_gather *tlb, void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *vma, unsigned long floor, - unsigned long ceiling) + unsigned long ceiling, bool mm_wr_locked) { MA_STATE(mas, mt, vma->vm_end, vma->vm_end); @@ -366,6 +366,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + if (mm_wr_locked) + vma_start_write(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); @@ -380,6 +382,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, && !is_vm_hugetlb_page(next)) { vma = next; next = mas_find(&mas, ceiling - 1); + if (mm_wr_locked) + vma_start_write(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); } diff --git a/mm/mmap.c b/mm/mmap.c index 5bdfd087b632..57cb3a2ac9b1 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2157,7 +2157,8 @@ static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, update_hiwater_rss(mm); unmap_vmas(&tlb, mt, vma, start, end, mm_wr_locked); free_pgtables(&tlb, mt, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, - next ? next->vm_start : USER_PGTABLES_CEILING); + next ? next->vm_start : USER_PGTABLES_CEILING, + mm_wr_locked); tlb_finish_mmu(&tlb); } @@ -3069,7 +3070,7 @@ void exit_mmap(struct mm_struct *mm) mmap_write_lock(mm); mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, - USER_PGTABLES_CEILING); + USER_PGTABLES_CEILING, true); tlb_finish_mmu(&tlb); /* From patchwork Fri Jan 27 19:40:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1D503C54EAA for ; Fri, 27 Jan 2023 20:06:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ylOu+y0FI50CwtS8jvlM1nDV5Hq7PkkZ37iDnUR4iak=; b=R9rtqZPgS/sxh6qnEmERr035UM VqrN0xA5vuSwCUBF4iebDwb/FRtWqAx0L0Yo+tz0QuQyuKLlaofvwzH97Q9K63/0tC91Saxcymw6y s2zyS9HD5rsRiOcj4vXRyLsqkemSUff2KQURFjOY+qpcQhnuT4Mw26ywMeD0T+evKUeRBZ93mfORn Q4P9FwEeeUwdhthsuZ+7XYcI2GUbD1ZnU0AGAKGmuPLh1121fcjya/b30zpMeJXIaKOvH7ZmerExg oWfuL4x5j/rflJn6i+zEX1vB7ywUZqtEl9qicPaIu2BJuxxRaDsYEDkWVHVjRVGF7cDFodSaf/F+A 63wRjlCg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUym-00GIDw-EC; Fri, 27 Jan 2023 20:05:45 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUwa-00GHEi-HV for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:03:28 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=1VGgj1TmnjNrGwceq1Y9mwtgHdomU/ZJ3zMBBp7twvM=; b=A7k9zyih7JsVKzTt+n99S5FuZh hszKU4TSHVb/TVKIjsZdQlx9ArjQXZkz13wzlfrord3alN1YEQFSApaPQPwMwFl/2Wdc9SOF4Dbap BVvJxF7VVIaGzjd6ASP+h8jdquec/2F+yjdNmY2dAV6EpUBRfuysc8tyyjMgCTa91u0OYWxwxsosV 7tSjrDIDzpyMYS2n8ausKuo5IckmSCokO33DtMsZeQ2AeWUi8DGMOT6qGKxJziRJ/MG7eLu6J7rOl KC9xlYr7jPCUIJIPdVSIZBHMA5ILGrAMMOzIvLQ+lnDQGk7fc2/oq4FriEOrFrd9EoGWQ1Z7CKES0 kETusMzQ==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUbQ-002xu6-0v for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:40 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id i17-20020a25bc11000000b007b59a5b74aaso6309471ybh.7 for ; Fri, 27 Jan 2023 11:42:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1VGgj1TmnjNrGwceq1Y9mwtgHdomU/ZJ3zMBBp7twvM=; b=kcxorgjDvieL4H0gsZ4o6ev4StdVYsVDA7GZVrIORQdjZKoFiGI1KoZQdzzvMshMrE snI0AzaYSvDLC9zEn2IBQumHO6CKLkrL/i3wJ9ueET6bm8oRn7QEPBX+BM7UODPO3hzZ U4MGRqXmViJMsS4HDNU6vnAmwplZNvnqe2D+qu3fGFBMwOJOM+QBc0aicDuQnfczJKX+ vPW/+j1Kiez+csRZc1jn4fVsV2fgBwTNZpRX4guJZVX8gsy3dFik2ggDGPcziixMOIIZ ZlSkjqSVG2I0FEWcD7z9JqxBzo/zAlN1m8/Qkt+nlr6qzU3SYltCmGn1Rp59psmpvaAD Fzjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1VGgj1TmnjNrGwceq1Y9mwtgHdomU/ZJ3zMBBp7twvM=; b=VFqcvSJmvHWKJbDkLh4t7zbJ8nZWrjun/GyaSicD/6rJw3j9PLNY4l1iuFfPpncwG3 HAhC6NizKj3EpC8f5H6L1DZXyXdlfKc0uE4XxpoPdeb3hEedvX1YYB0Bs0ODH6rLVeC6 oXsg3HraV20YJvou/gBc55+2du0ihX4F5ghDBMb86fmbEU/Gvwsftx8OK1rQszl8yvTl +Hn45YriE3ObONrxgOwR/43lxq1LoeuVAI2akx5u+70ilkc1RA8be8ClKGYd9OVspZxj NW3ViW2lQ1fdnTvNIunD17PMoLhYxcORccIsyKQsJIpZ4Mqv3uvld2f5Oho6EvY1LUEO KqRQ== X-Gm-Message-State: AO0yUKXhyDiYsUd9V/S0oyK5FkuziVdalRUaccDw8Hm6a+BwQHMei7mL m3C06lF18PnbT+9wtEdJIx5atQ/MGIc= X-Google-Smtp-Source: AK7set/jPsOwzlm+kUPz+CABbySieLCP3cL/8jypOFocWiEQOE8fcbH4Z91B+4ltIKP12TkxYenF+FhrAT0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:cb8b:0:b0:50b:1b97:f19a with SMTP id n133-20020a0dcb8b000000b0050b1b97f19amr434148ywd.164.1674848522027; Fri, 27 Jan 2023 11:42:02 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:57 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-21-surenb@google.com> Subject: [PATCH v2 20/33] mm/mmap: write-lock adjacent VMAs if they can grow into unmapped area From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194137_835552_60D2204B X-CRM114-Status: GOOD ( 11.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org While unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases write-lock adjacent VMAs to prevent this growth. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 57cb3a2ac9b1..3baf218836bb 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2399,11 +2399,13 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, * down_read(mmap_lock) and collide with the VMA we are about to unmap. */ if (downgrade) { - if (next && (next->vm_flags & VM_GROWSDOWN)) + if (next && (next->vm_flags & VM_GROWSDOWN)) { + vma_start_write(next); downgrade = false; - else if (prev && (prev->vm_flags & VM_GROWSUP)) + } else if (prev && (prev->vm_flags & VM_GROWSUP)) { + vma_start_write(prev); downgrade = false; - else + } else mmap_write_downgrade(mm); } From patchwork Fri Jan 27 19:40:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119304 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BBED3C54EAA for ; Fri, 27 Jan 2023 19:51:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=0broeHEjhQvtkMuvJxJ8QF9LuJDuiVxlz9H7Ya84dsE=; b=gUw1a33GD5sXTopTO1LzSrAVlJ OVg8LVJzmqG/3X1tEnyH0UCSfC/q3SfGcLUx9LL2jU8ijmPnGCk+aHucUalBrzXhBr096IX8gBEvz TZStssj/aZx8chBXtlrsT3dCywtUcGjQsK4NA7GmQbLhU+N3Ep9/gyQBs/iyJep0dx8SJRLh1upBy AMvadXTaxeeOKfan4smoJnDxtGBHxucQ0yW4HPByl6Ae3R6M1vNEMdbMxz0uNhDNubrF+xVZWAdLr OGNsKvQwo0yIM9eVeOMylddYBM0nU0N0Z2PfGTuRH0ENOQGbaE/1KFH3rozlomrXDU05fce4LX13I KGhDNKag==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUjX-00GCeF-2I; Fri, 27 Jan 2023 19:49:59 +0000 Received: from mail-pl1-x64a.google.com ([2607:f8b0:4864:20::64a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUcC-00G9Gi-5v for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:42:26 +0000 Received: by mail-pl1-x64a.google.com with SMTP id y8-20020a170902b48800b00192a600df83so3327001plr.15 for ; Fri, 27 Jan 2023 11:42:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Cy3b4YdYe1JheypyM1qEqlntx+LNPOiVtlvqoTpRTGk=; b=oI0LJkxWJWb4FwrADuA4kk3Z1VuG/PqmB+wfudzVRf1FutZSssy7j1ZDpJRsJeC2x6 AK+2fBcgvA6Z1yebvss3n+ODruOq7UlamXlS7DfsbRapPdytBdlVYqH5VQCwBS0bW5Cs 14NyRgqd5c0dS96F3WtaSc6UYIsY2Q866sZsgDc9kakHc8RR5OiJX50DOvF/naV3sW35 K5NQwDrUGeBVt7kutcZKVO5xLo4MSKoGaFWDyWGZ3caTDR6ZBpyZR8N/O+LnlyPsEjZa UEkxm7VSV5agQqzuF3YgYMg0fZBeMdeXvS4WgRk9vqB/9OKyGK8SEZXO7u7cVN0V9JOR wUhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Cy3b4YdYe1JheypyM1qEqlntx+LNPOiVtlvqoTpRTGk=; b=vwzkMgodWjaTjBDKvC1hVepTmTg+gp75akLXBqEcZQeKaZ0l5ZuXzy1pLi6c6M4ID+ QnRgpx0ahHcAN5cH64l7FeoL1JHwDESCRGE0AjfX9Tduvfxqs91YgBLEM8I9Toc2B+tr 2eNp2IUTXYaI3oeZGJktq36EYrnTzoyHzDdMlG+xlU0LWrQX1xkPNtrc+bUm3pOjyCR1 rbQD981FcxGSTDHqv420lhdDvuJhzq4n7bj2Pq8psq6Bv+p2Roh+Snhb77utmPlGExw9 l43t0/1PhG127Gmbf4IMampRHF9DElymnqBZTG8YL+LzGJiLAKLFpdfO5T7uqgFL9kq8 ylgg== X-Gm-Message-State: AO0yUKUQ26RA82JXfYTW2WYQ/cMamAcZy1iOSYR56dc8dww1Mvprud81 yRsxVhJv2v9NvukFoXMnzXkVkTrxOrU= X-Google-Smtp-Source: AK7set/yd1hcYRvbMqSOfXS0bVRoPLBLaG1qWLcI1OwZ+QZcbDYXkFqGvC1aaqenThgkEbN36zZqWHoF630= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a63:ba1a:0:b0:4dc:3b99:6f67 with SMTP id k26-20020a63ba1a000000b004dc3b996f67mr1091417pgf.47.1674848524282; Fri, 27 Jan 2023 11:42:04 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:58 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-22-surenb@google.com> Subject: [PATCH v2 21/33] kernel/fork: assert no VMA readers during its destruction From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_114224_266663_9ADF227F X-CRM114-Status: GOOD ( 10.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/fork.c b/kernel/fork.c index 9141427a98b2..a08cc0e2bfde 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -491,6 +491,9 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); + + /* The vma should not be locked while being destroyed. */ + VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock), vma); __vm_area_free(vma); } #endif From patchwork Fri Jan 27 19:40:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A36D9C54EAA for ; Fri, 27 Jan 2023 20:06:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=83GtTLDeJH5X0UJYTPvnH+03exd8T9sB8D0AtDO5Id4=; b=3xZ+iV2AJVJjun8jcId9oIQgYb 5Scy0fEAsnAh25QupvnWGAnLau8lZEvD5R3r0gh3HXlu5/R0UAfneDTZEucpGcji4YWqCm4AXdafq uqBBmSI+Nr8dUdGXdzTkMYw+P8XOngdD0R5d2ePhgqffuKcmuhWfSEWucbQKIHpCjykosvDETgoAt hs3bcGfNL1+OuyEkCOwRF4a0QkLr9ZUa+3DTsTkuJdDZJueIEehVyCToSno+jybURqBzhFJW5REqU 5p3TTXR69pDUCVUTugE31jWEXCSPva3GHHoZ0clm8G6FqCLdj/y3SqBkWM/8BeKex0H4eCICNRIeN QHGjRedQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUyD-00GHxM-NO; Fri, 27 Jan 2023 20:05:10 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUw3-00GH3s-Cm for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:02:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=mMW5WtXlwQRVw5jLv0uMPkgP2YoyRT4XQf8L1G2g0sM=; b=jDUHLDHPIIl3GYvZtDk/Onwb7g Szcx+0MP1QHpL0Qai3sYT4rEzsud0iC/lrxykvvEUYaX12VIrQfU78IjaCrfHcLgIWLVbfyfTA7D5 bCyuNvm2591mp14vLXy+rnj5LeZeXy/tF0fxZRWYjQHJ8zjcdfg9U07cBb6IWFHlCmbhIcWzmGwWj xQJCF97GfJDx7uJqzaeka/3FH1OrrX4ey7zrnU7WL9Dz+lUojACPjCbl87Jn9teRmXIdtgsIk3IfP E2TMCK/5IxlnRRaJeHL7Mn/SlJzjnSx1j9EC5HkOEGKH0uHoUFTlVrGT/8xviG8DEDZbtQB0wfYaz L8RTeHlw==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUbR-002xug-2K for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:40 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4fa63c84621so66292447b3.20 for ; Fri, 27 Jan 2023 11:42:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mMW5WtXlwQRVw5jLv0uMPkgP2YoyRT4XQf8L1G2g0sM=; b=mdQhCODmER/WcuXnOwjGOPGsluXP611R3K8yAAqGfZO+eojPR0iLkDWsNYoGq20Pxc pLGndC0gNc0k/+qBk7J/uFwYPbCjsW7XmlHDcZCSFndQyDIXmYjY6yxKamOAoeP6UD5R 10GuF5qfONyUFHlhJRc27yht8CtL7+xZMg96kBN6doFn/8DSebIrGsIvYM1XfrVRdrIT Vzd4VicNtC2Hx0ebeUSEC751adwD7ZH4dP9Tg/rHSx32WXBUDV2TcmMYgTc4y3CEq8j+ 2IuM945ICEGJ6/3bquYB/9nBx6UvS9o2Tl6iob7NJEFRIs2YxCbcwKtUsh6O7/MdtiEH lFnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mMW5WtXlwQRVw5jLv0uMPkgP2YoyRT4XQf8L1G2g0sM=; b=sJd60Lyh8GAVNia6BCh1Mq/irUmQgagPVu68pAYHdkHz1I8aXXFlJVPMZuL8ZmnMJh okjZB6GNHYR6BKIkMPnQj+UpEBAtTQhvPPK3nI5XeQQTpOymFl+D8KByB/7IhqzmuIpD v6iS+maRVhyAPwW87R5/UJCouA/zjAIoTTin7VGY6+q1vYAoVf1iAmz7XtIsqfvId+pY DXiEM6DU4F/QKlNpXmFQJlWLQzWvttDb9FELCfBLSxiqfNxUSJMYnYBoc9i24jF8mJp+ CEuXOo7S9n1EEapwQRRqFTJngAN0v44KAJidd6g6eiVd4SNsjQeuTqW11IhxKQzhrMOj Vjzg== X-Gm-Message-State: AFqh2kohF5heylZVs6PJOLNvMy4V0hZ5PrzpRNknVJyoGQ6gux2Jjxs+ +Zh4DfjS/Z7Xg+t+EHTwqm+6Dgym1Bk= X-Google-Smtp-Source: AMrXdXvm/6CR5hD1SSLODW5O6UqAFkTGTyEvJF4A+5Su+1PwVS4k4WIa/asYlNCGXFd3Q9zHSTloGdPuiKg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:1fd4:0:b0:7d4:712e:2852 with SMTP id f203-20020a251fd4000000b007d4712e2852mr4289060ybf.628.1674848526673; Fri, 27 Jan 2023 11:42:06 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:59 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-23-surenb@google.com> Subject: [PATCH v2 22/33] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194138_043285_740B74AF X-CRM114-Status: GOOD ( 10.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Page fault handlers might need to fire MMU notifications while a new notifier is being registered. Modify mm_take_all_locks to write-lock all VMAs and prevent this race with page fault handlers that would hold VMA locks. VMAs are locked before i_mmap_rwsem and anon_vma to keep the same locking order as in page fault handlers. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 3baf218836bb..3d0cfbc92745 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3501,6 +3501,7 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping) * of mm/rmap.c: * - all hugetlbfs_i_mmap_rwsem_key locks (aka mapping->i_mmap_rwsem for * hugetlb mapping); + * - all vmas marked locked * - all i_mmap_rwsem locks; * - all anon_vma->rwseml * @@ -3523,6 +3524,13 @@ int mm_take_all_locks(struct mm_struct *mm) mutex_lock(&mm_all_locks_mutex); + mas_for_each(&mas, vma, ULONG_MAX) { + if (signal_pending(current)) + goto out_unlock; + vma_start_write(vma); + } + + mas_set(&mas, 0); mas_for_each(&mas, vma, ULONG_MAX) { if (signal_pending(current)) goto out_unlock; @@ -3612,6 +3620,7 @@ void mm_drop_all_locks(struct mm_struct *mm) if (vma->vm_file && vma->vm_file->f_mapping) vm_unlock_mapping(vma->vm_file->f_mapping); } + vma_end_write_all(mm); mutex_unlock(&mm_all_locks_mutex); } From patchwork Fri Jan 27 19:41:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119352 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C8AA5C38142 for ; Fri, 27 Jan 2023 20:10:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=93iic7GRySi6zY3Ravq6SYselxzBumMscU+/0pKFyUM=; b=MucQwTbbFgEHIHHeT3taTqBulG 7808l7V1e0SUKhvKjY1W5ReFPlH1b9QWUs28eparG7uR7ObIkdHc65pusIKQpHRxkywP72UgNlj17 YO4YlZmsK7R5VmtJLS4A/uZfxLVgXKOqHMeqNDqvRz1Y41w2mlC24N9k44ZQB+kn3TPDYUKZlfZDI /4kIYGehMmCg8Zjq1P6JA7Otwf7s0sRW//EsHVCf/QJ1m/fsYyFsmtGIzUBnlO71N17/8+DH+h+v9 yFGnopjhtlzX9c2iKXdM/W1jV8LvixItf6DH4lUsaJtHPQ7qQIBxoczKx6nwwtcB3YmH3S4KrOAWS CSktqdng==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLV1m-00GJT6-Fa; Fri, 27 Jan 2023 20:08:51 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUxF-00GHVx-Gp for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:04:09 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=5PvsAEkfbXSvdKN+lTJ9M9+vqfLQVBlihpxSKUw/t/s=; b=n/R6qhnpdR6E/qHLzmZAS9CFsN 1GZegfnNNoGQeha7dtyXJpf5THxn4uuzP5ZorYIQ/2g3bkoG18xrn5/UZN+wz3CXLv1SYKXBpWyz8 HRO/bdZIx5N/cgQWphumEIF5XK+nSKahCpaxdem2vIR5ofD2LUtKYVWkNsTYb9rz4rHmxYs6yho7K sxPkMkZgPhi+eciGS3YwPDVZw2+toFhczFkq6tsWCzbX3C5r6bfNr3OYB+WPanaHWXdUS7a+LW+8g uNbL372x4dz82Ds76qj9S4G3sa/RswyVKz922HzJ64h6T4L2vQPtuMHWZVAEIzyJvVslamgw3OJ9r LUCAj/FQ==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUbT-002xv0-2k for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:43 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-507aac99fdfso65542267b3.11 for ; Fri, 27 Jan 2023 11:42:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5PvsAEkfbXSvdKN+lTJ9M9+vqfLQVBlihpxSKUw/t/s=; b=kp92vFUO2N4rCYUenom2PgB/dU7qqaylHAcH6HVVMYT5QMpL/oqQ/DdOm4bVNlYLNw 9rNSkse+dfgOJKwj09yKdpzV2qc7upA8mjHmEpxt3RKsaa3pDRks1QGh3Nb2hG4bp/jU 3eSArvNsdqtxNb6zpVi6gl2vf0iZr8y+PtaslVXMeV9Pu9xS/EmrXvCiajKJa9iCZe4S LpK/JO6vbmew4JxbSzbZQyffZL4sOt9sBlPWbT9WmPY7+pp+Ff0i+9BT7l4WdUaIrrgA btwhF2XkX8IUP7VpoyJ9OAYm06ddrZftAFhV0gBzofVRmYYEJ8dT0O9gMwk7XMh0b2Ct qw3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5PvsAEkfbXSvdKN+lTJ9M9+vqfLQVBlihpxSKUw/t/s=; b=GS/F0+8U0C8uLvi/3T+fQOYjC87rPTd11bYZ7Ov1FKW42IQ2dCl8miUEvY1AXMAvNn uVxqxKvJnhwtFj0Kqkp7rGhXdKKcanuvGhqPB9fsq3aTu+OVrHqJyr4FjTDTpwJPKpXH gBbbgxUg9n7KekeC/fkkMvtRVSxAikMKaiMfIlGce3Bq/95HVA0a9rLYZxlkg2HvShhm o8y9y1bdeoxEeODO8kUDCd1cdBidV9iZ9XoqjHK24UY2DOxiHmfifqxEODWrvUxFdvjo Bk+pCgbRj2SRsc2p7ol49ZAFhxwfSInFhxFjLefOynRUV8NRyEix7pLHkL0Itj2nsPxC b1EA== X-Gm-Message-State: AO0yUKXC62bJ0Pkqxh06wXiHEntbbiejQP0CdiAlAK21yhQ0/uhuHr2/ uu0penRl1j3EPh26XPn1EqDIQNy9PkA= X-Google-Smtp-Source: AK7set8zsvT5wkP/7moSYA7dIHSCpHQad2RzXlV5c0Wh0zEZrq5yaOQZsQ7iykW9DnqHdXZz8PLHMfkZLc8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:6d2:0:b0:50e:e91d:f47 with SMTP id 201-20020a8106d2000000b0050ee91d0f47mr27189ywg.61.1674848529047; Fri, 27 Jan 2023 11:42:09 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:00 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-24-surenb@google.com> Subject: [PATCH v2 23/33] mm: introduce lock_vma_under_rcu to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194139_958234_E9B0ED44 X-CRM114-Status: GOOD ( 16.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce lock_vma_under_rcu function to lookup and lock a VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. For now only anonymous VMAs can be searched this way. In other cases the function returns NULL. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 66dca140695e..fa2b9d6e665e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -687,6 +687,9 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index d48c76e9fa57..5568fcb0a46b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5222,6 +5222,57 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +/* + * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be + * stable and not isolated. If the VMA is not found or is being modified the + * function returns NULL. + */ +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + MA_STATE(mas, &mm->mm_mt, address, address); + struct vm_area_struct *vma, *validate; + + rcu_read_lock(); + vma = mas_walk(&mas); +retry: + if (!vma) + goto inval; + + /* Only anonymous vmas are supported for now */ + if (!vma_is_anonymous(vma)) + goto inval; + + if (!vma_start_read(vma)) + goto inval; + + /* Check since vm_start/vm_end might change before we lock the VMA */ + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { + vma_end_read(vma); + goto inval; + } + + /* Check if the VMA got isolated after we found it */ + mas.index = address; + validate = mas_walk(&mas); + if (validate != vma) { + vma_end_read(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + /* The area was replaced with another one. */ + vma = validate; + goto retry; + } + + rcu_read_unlock(); + return vma; +inval: + rcu_read_unlock(); + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. From patchwork Fri Jan 27 19:41:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EB4B5C61DA4 for ; Fri, 27 Jan 2023 20:05:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=UU6h4fYfPACl6t9mQ0xsvtTi3/oys8yI34viZBG42Zk=; b=mPT2tCnZBNwjJor+tUpY+852nH o/uGPPm7fX8FpZqhtqdS4bXOuNsor6Br5TI8cGvX+RwJiPGKpWp9sNvq5AYkeqEzwkfHSkixI8eUB uwNPiLzUnPwEmyfkJqZ1oFRIdi0XWCkYS/5i9ho71D0C73dPLMA/AX8Ob9qhL4Z+vK4h/38BmCWhn hB9SwlpbBV9MdTxZe495eMGkH1cR4KcRKV04/qBIIDxziKI0iN8kVe95tzRNKonuI68c/9t2fne/h t7/RUM7pW9tuDdkK/PnP8e6xn6dDpdKL0IYNJUnOEJsjpdixfFg6Wg9g+f97dVzYVB1x3aNHjEQHb b0qrWdrA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUxf-00GHiS-B4; Fri, 27 Jan 2023 20:04:35 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUw1-00GH3R-Lj for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:02:53 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=gSaEMik01Wf1+Bdt2RmdUMjgYY7lDp9hLxJUYqXXyyM=; b=E0jwcxZeXwY6QUUi6XD8GhDfiz hhPUPFb27cXgNX3F1+mQS0upc/yAReVV6MfwVVbslPc1/OsU1Q9VM+G9ywjVk1AUpBuc7XGKkBVjd 2z6T+ZZSvC18syjkAB1W4abecdPCsodsOfAxQcKE/JHTwSsFS+rmjWgodsJAueeFqW9a8EG5yz1on 5rl6iUut9LF7sU8GHd9uXptX37ciA4OlfJ5F6GJIWSICTv63ExnEIOU3ZMBUN6tTyfbqzHa0re6R7 5l4XKCT9hEcv//ROujPRYB39YGAHuEcccTk7UXn/Pudpsg4MeVcUSBVxNGAkkO+QWskbuMD1sX8Cr fNjlzzXQ==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUbW-002xvi-2R for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:44 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5005ef73cf3so65349257b3.2 for ; Fri, 27 Jan 2023 11:42:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gSaEMik01Wf1+Bdt2RmdUMjgYY7lDp9hLxJUYqXXyyM=; b=rcBVm8w95my/f0ebkBHOBSdk/g3dXIBnrvyKhE+0lfp8SBBL9UDohndGfm9prz69Nb 1VsGdk5hVjbZdd99cfs1kRzqndkhM72YtTrleRtcoKwDWYV7T956W4hi9P14DNVnfcTE 4dzd8voX/MwzX+niWcYh6zC2gtPII1RsQu2W+o1SzbH/VUzyZkZ+K0mbi6bCuvIO2r/9 q8JUw2FMeYJjbLjibdrizaV+CI77RGRlLfF03fxNpzuIRE1ymkIpYCAH7a7Vh5p4LTWM djOGRW9GCrBrxw2Z+lpBSKPiwjL/b62fESDSi0yRTpPNMfCaBkk0gvqmlpNjM1674+ZN MkCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gSaEMik01Wf1+Bdt2RmdUMjgYY7lDp9hLxJUYqXXyyM=; b=zC0C5gCEmCm3Ii+OQq2h5OrnAQKnU+TAUtW17fhF0oS5jlCsHdEBqIwZnNaEX8wnCh oFR17lmXZU+Z6cF+wbl7Q+Gm2Bzww2TKur90vKNwFOgPptKCzdHNxdY+lIuvFyy4dmXK zCtsUyedsTpcddbKl9jmUJoI0YV/ODmlLN3BaUtc46gKtczgisQKfkpfrhRTWtPDCOYp a6oekp68PPX0upjrA9+IQYTNS1K36fLndGqwbb9gLfE8kZ2zTjob/lBg0NZmMuNGYy1w wsuYwC6JnyGnwCX0c1L5rw4wDL7pu3799s/x6ztXcjFcjPauaY3/GgIeAgaHfCw8KHBW RTYQ== X-Gm-Message-State: AO0yUKX1ACeyigOTIHhldtgkfcRTdBzV33e2ayz81eT/Iq8MdtQuIL/x 0XR2KB+C+lXic0NrdPjWOZ95TyHwj4s= X-Google-Smtp-Source: AK7set9ZuZbDGpVLmgenU71+ffHh46VvnF49YmmPOU0QZTkBBursiC5h0isQQWJJC0fjjZLxRphOeOdStg0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:d8d5:0:b0:508:1348:3e54 with SMTP id a204-20020a0dd8d5000000b0050813483e54mr1003857ywe.440.1674848531551; Fri, 27 Jan 2023 11:42:11 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:01 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-25-surenb@google.com> Subject: [PATCH v2 24/33] mm: fall back to mmap_lock if vma->anon_vma is not yet set From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194143_000816_7AC1DF9A X-CRM114-Status: GOOD ( 12.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When vma->anon_vma is not set, page fault handler will set it by either reusing anon_vma of an adjacent VMA if VMAs are compatible or by allocating a new one. find_mergeable_anon_vma() walks VMA tree to find a compatible adjacent VMA and that requires not only the faulting VMA to be stable but also the tree structure and other VMAs inside that tree. Therefore locking just the faulting VMA is not enough for this search. Fall back to taking mmap_lock when vma->anon_vma is not set. This situation happens only on the first page fault and should not affect overall performance. Signed-off-by: Suren Baghdasaryan --- mm/memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 5568fcb0a46b..593548f24007 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5244,6 +5244,10 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_is_anonymous(vma)) goto inval; + /* find_mergeable_anon_vma uses adjacent vmas which are not locked */ + if (!vma->anon_vma) + goto inval; + if (!vma_start_read(vma)) goto inval; From patchwork Fri Jan 27 19:41:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119326 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 87250C54EAA for ; Fri, 27 Jan 2023 20:07:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=L+wqZqxpBn93Jn+SCVa4eIuz9r3Kn8SNuC0oL1Ok3nI=; b=14bI60WSlU4IWzXb/G+u6tzQFJ SjPOOd/bzmTBlLsmslvYIDQCA9yPiU05c65PYNoJXSExvwFO/FlT7++gY0Ti3FzNAAi4Agczfx/HR MQu0jy9KodqtFWUCTEgPeyFAbatssXAzmuBsD367G7KkRYhCuEwYUTT+lWEEV51aR8rS6Gb+qvzNv hIlSVDSmAyFazmohmq6VX7gHOmzk9ddwjSsinh1y5bblH+GdvSAlGhFaso/y9HLA1tKXDUOBWAv9A 6WgQW/FN4EsJWz4emN4SXd8njMqJklG9K1wGcWrLlzxtDUgJIUyX5odHNIo04DUtVQr6fez+bjR68 0Oowfc9w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUzJ-00GIQO-DP; Fri, 27 Jan 2023 20:06:18 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUwc-00GHFC-GT for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:03:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=2v/Evza9dkU4Ne98Lm6HAWxL8/YazIwD41pmx7joRkE=; b=PbzTBDesT7YV7MkpiM/bwVot4L WLdtzns6cJejiOs6YOFE0FMxOQEc2oUHRGEWK05cCk7FS8sT17poo//77pB64byuMRM1bU21jJvbk AiToPSjzBMJkh4MBZBetQ8gymEQFQQ1029CUqrEs7ss4LOJX3NT9t0VP3+4dce6sLQECZbAXrk7Sh V4iUR+CjPkW+HRwsFCZtpNhZa6ZRHunWsQYHO65/OTAKxQizwZRxTD5oF9rRj2PlMyJVurmY6JuOI fB7CtQkiZulPm1SZJjpqg0KqUxzzHgy+s+zpom2LUJ8dFG/zrlMTdMIB/BZ87ofM4+JgOzr4liIUG /PmmB/Tg==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUbY-002xwb-32 for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:47 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4c8e781bc0aso66311697b3.22 for ; Fri, 27 Jan 2023 11:42:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2v/Evza9dkU4Ne98Lm6HAWxL8/YazIwD41pmx7joRkE=; b=mLwUPJ0nc0ulmmwFoyR9amwtEYn4GV75pogWN7ijAK31MqrZCBgfhijsONjhepzopP 2wYY2SkQatHOd+vKpDSxYxHFT1IvhzZTwDWuTcrFWygojUbcZe7xPlEiO9BQnq0w9fKe z4cHjJJqSwoele+PetnaYF8lewRv+be+h1Bgt+SNNsmNqcyEHhiyYbNRieovHRIQnY7Q KgzOHSrbk4QlZX3GRmHUucd8wrsetfG2ey7u2AbUsjqg+O7jZuWXERmenSwIYYomTRZT 0AcxNZ8Sif0L8fvqzJHXb63zST/Gg9+vMF9oOa1KQZBohhw7ZSFfQ9eLw7FWbw/nE4Qf wodg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2v/Evza9dkU4Ne98Lm6HAWxL8/YazIwD41pmx7joRkE=; b=HdGT5ETRKSwYkXsjlIUDVja+nx8UghUX9CD0H3qJMQ3LF3wd6/pO4RpLM6+OFLRSDA 8NFLj101tnh5yLsc8EWD+4+pv0khmox73huu42xBkd4P/v9UYBCh+8AiI2IGJzpoM/tS X6H4NU/2Fi+MNV3KQufKg9OqFkoLt87huPAJSIXeMsdG01O2+ch0sZOdHXaCh0Ub+4PQ w0IkPifp8oNU9F0PAC8A3iVd/YjmlU4S1a5BAGUECzegfnh90CkcatO+4mjpF+GeAOz5 ZhnC81zA6TAaAlJphquvWe1i/qNjipXSQfcByBmQRTj3o44UZbeqCMM34YkWSCaAyMnV Z60Q== X-Gm-Message-State: AO0yUKXmG0PTc3Espdca8cig7+QjcJFzG9kQOX6xqrK+SwVRxLQYE/YA /U+KVUQYroJtb9lYaxx8dWL05xqvTpM= X-Google-Smtp-Source: AK7set9kKqDiJe5U7EOGuYIrwgRntRqTHzt+zr2t5qOdc5+YeQS/lL3tZ/xnNoww1dZdAcqE9B1Pln9KCVk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:b2c6:0:b0:509:5b20:ff22 with SMTP id q189-20020a81b2c6000000b005095b20ff22mr917640ywh.328.1674848534136; Fri, 27 Jan 2023 11:42:14 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:02 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-26-surenb@google.com> Subject: [PATCH v2 25/33] mm: add FAULT_FLAG_VMA_LOCK flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Laurent Dufour X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194145_317005_31B596E4 X-CRM114-Status: GOOD ( 11.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fa2b9d6e665e..a6de58bb40c7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -472,7 +472,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_VMA_LOCK, "VMA_LOCK" } /* * vm_fault is filled by the pagefault handler and passed to the vma's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 88619c6a29a3..c4c43f10344a 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1053,6 +1053,7 @@ enum fault_flag { FAULT_FLAG_INTERRUPTIBLE = 1 << 9, FAULT_FLAG_UNSHARE = 1 << 10, FAULT_FLAG_ORIG_PTE_VALID = 1 << 11, + FAULT_FLAG_VMA_LOCK = 1 << 12, }; typedef unsigned int __bitwise zap_flags_t; From patchwork Fri Jan 27 19:41:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DA777C38142 for ; Fri, 27 Jan 2023 19:50:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=F8cDOiIcz+HfzfD7H8pTVstvLP1yG6G6xj5NbcYn5Xs=; b=DFHU83Fp1AnPM+4pKTDTJM2MvU jqxmq2i0mvdz8nRbhWXhFp4pWgrdAqjn4r81deH8CD48wLf/COtCgB9FHsxrjlr5+9zDngItagYWk PFubnzAHptSH3TngExnIDM4+0M63/kNTrdZAC02eQojpErgjQnGcSGBcaE9RTSgXZz4j0uzGmGdf0 06fhjJxBkOnRN6yIMT8DM7Ctt5WtycA099xtheHZLYPfPCWA638KtWKPK6IR7aigR4caZ2gcHgqja UzZAe57ec2mwlBr98dWZVyspVf7ttbiMENJ/tr0eiirHz/Brjkh4VUW8U9JMdhu0pPKRSZxs+PUPY rKYFk2VA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUil-00GCMS-QE; Fri, 27 Jan 2023 19:49:12 +0000 Received: from casper.infradead.org ([90.155.50.34]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUc9-00G9Ni-9K for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 19:42:21 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=sBLmwmiYmTj+OGGVUcLxhx0PU/jBUoUEJpANG8soAfU=; b=od/vsmlSuYaTBanHHZceUNJiAT 2yVDjtBw8ruBI2wnVfgBbxZFY620nnPrcVpE+/Dy/NmP56sQ+olFlvjZlTSvL9r1XB3zg02RQbqu+ gP6jI7fAuJT1F/ToyPyr9i4AEbxJSf+B7SetZswQ0wJTygB3YNLM6QNsUut4I78iflfMum5lsPI9W DHP3n3z4d6IQDXNJ9L7s00qyKvhvetzwnRxhQ+eLkxEIFKhaK6qgnxr5/kALI8qC4JGzXBrOybEIC R3Bj5YJxVmfjevYpxL0cTKLgMDYiVZLkyDyHs9rYhH90kZEBLnYKYz0B9NUBrBsxmc9fCgCqBpgvN faLhfudQ==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUc5-007zFj-Er for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:42:19 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id x188-20020a2531c5000000b00716de19d76bso6362959ybx.19 for ; Fri, 27 Jan 2023 11:42:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sBLmwmiYmTj+OGGVUcLxhx0PU/jBUoUEJpANG8soAfU=; b=KWtfJ/7xIlPNyMbJgMly5Sn2+77Pi3kT3bIKcXhSEYdAbccLPUNTncstNEX89dvAhW NM9ImIflxrHOdJ1Mj5fWozRK69MXYGw+crQDfh9Oftap/iRU1kAVPiBZhr/+UuvLj4+U 4LLcR3mdcWgS75oxY+Q2vGwsrXQ6jEpwNWpRZ0m4MKHrCT8iGAtEZyFhrSxdE7CjPr88 djL5aWWS3cMBdV/JlcmaRg0LM4Y33l8LwLfoSQoWCllE6I7RGCcpBFSNwKknGwVlLQii i5R/yajGZAULuwac7zP2AeGWc0Ux4Tm1rMqU81PAQU3nWhlQVK4py4JD6fzjGBwa6RFE gFWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sBLmwmiYmTj+OGGVUcLxhx0PU/jBUoUEJpANG8soAfU=; b=NBvKzvaFtpt7nL7Z3CLIF7WuvvzcNZxqbnS/z2GmqG7hHutAsrYr1Nx8B78mncyCUQ jolNnMygAF9+XPh9Gxra6i+oQu8V2xmaVSZuawfkcf6GxjyaHmkMOnWO004h33T/Iyka fT/glmsb3dLpW1eXhu0dhNIxW3vp4iyCBMnhlN1xTBuWsA4APSP/rlvxP3nqFepAjtb9 OMNhtsovixMdbdSGMycbFJ+ttro8lLwkW6fgxS1TpaEuuU6c5fI8wsgDyvb1q5zrs+cY NKtkrvI0GuUva0E/pUnjaqnIIG2Zd0jAdogp4zFW41vmismlzW3X7uaBRFh7f7LQwyQQ bX8w== X-Gm-Message-State: AFqh2koGjrBujaZOJo04hybQHKw79OwAb3AHd2LWxnGGq5Bq/BTLWSTw p4N4YO14xr+S//xjGINGkW01kjklktM= X-Google-Smtp-Source: AMrXdXtj44VmRN6kQLB49pXglCfjpmg9iXCqYgU9gl6+Dl4v+hAma4/0PMDtIg4t8IMM3q4b9OalQWjETEg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:cd7:0:b0:4dd:9ed1:fba0 with SMTP id 206-20020a810cd7000000b004dd9ed1fba0mr5011323ywm.265.1674848536283; Fri, 27 Jan 2023 11:42:16 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:03 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-27-surenb@google.com> Subject: [PATCH v2 26/33] mm: prevent do_swap_page from handling page faults under VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Laurent Dufour X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194217_978653_94EFE743 X-CRM114-Status: UNSURE ( 8.56 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 593548f24007..33ecc850d3cb 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3690,6 +3690,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!pte_unmap_same(vmf)) goto out; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + ret = VM_FAULT_RETRY; + goto out; + } + entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (is_migration_entry(entry)) { From patchwork Fri Jan 27 19:41:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119328 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 173DBC61DA7 for ; Fri, 27 Jan 2023 20:08:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=jHWbf45SGv4Lop3UNslrYB2drm/CODLWAqQhWu4e4GY=; b=t68FBp0ZGtg9IRu3QNlWGxLkZN VECMBaqA8ED26AknoLsTvJskgVyKqC0x38AEnj6oWAOIafZ6l57Y6aPofP/aJyB97LtNst6D7rtIg IDZprJc9hGpEBOEPaElSU6wdYTpa0T5oFE3MAtJ++v7+gp794qfGxkr7zDVGgXJSg6Wbc/3w8iZyV r1cSUhc9nPi6+pELAQ6ilu4cXTaHvAHH9Bm3YtvKVAVs/Arp4Du9aSOik97AMC87v42r7Z8HQaFgQ XvXIm1ooHi89pOEL6KJguwJ3iODgwBgWXdgvRX5XlBc5a4s5i6zgDgtGD9Hz3oyTfasct9HhIM1pg dYhjW5eQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLV07-00GInt-MH; Fri, 27 Jan 2023 20:07:08 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUwm-00GHJ6-29 for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:03:40 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=dtbb/rgJDVPG5hb8ydKzjEL35WMZYcFrpyL4j03oF8s=; b=C9gWAmJfBw4xwu7zVxMivuls0x ExHsHZWiHUxjaqMCwh19K062OMmHHJfwZeIUQHN+ObbWUSTCvxFzeioac1z2p+SRaN5oiYdbrtDwt TJPomcgNKHlklYiatNumDOsjN3lsP7lR3mrPVt7GoVGEpFIe0ie0DUXc0N2L/3aw/SyvprrpLbLYj 35Dij8k2SmYmR3OnTLEfhuSXHyengOoTiymbJqk/0xhQoPAHG8QpN1TIf9+jCQXTYAAIHEIDgeU79 lVhQdTNLOVAdzcMgmpIcq0uCeMRGAUICoZOPtJuEw2RC/GpUstv4NavRQVjnffBnwNarzwmP/sg6H p8uuwS7A==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUbe-002xyO-1V for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:53 +0000 Received: by mail-yb1-xb49.google.com with SMTP id t13-20020a056902018d00b0074747131938so6390476ybh.12 for ; Fri, 27 Jan 2023 11:42:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=dtbb/rgJDVPG5hb8ydKzjEL35WMZYcFrpyL4j03oF8s=; b=GSbfUJd90yi2xMLpxJwkNIQoO7AYaRuo1p0W3egBrsgFnTlql/QObCOWtQyWcyCJri L9IkPEewPotUja2mT2dWXCpgA9vVmlCVPbr2KFEVViPcTyej+SHFX8+fhAEaOP6xtrM/ x5eCKOKsLW06Glc5jxntDrg/hIgANLyaiKO4UzofC0FOHh/fDzJT0Gg9GuJf/nEgmEZV hix6GjSSYXSWndo4QypjFazRowiaazjUscTvm2uHMbtPJDABgUnEX51b4PI7vURt3LgP X4xLvpH0mH+5Sv/7D+0hQE/rAXv1nGKdGusu8zdXbWRFOMqJp4iPgA6p21w37iQT/eTP bctw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dtbb/rgJDVPG5hb8ydKzjEL35WMZYcFrpyL4j03oF8s=; b=wc1cmxu/EuBXKqdx2O/Snr3JW93+Mr14CJ7MJo5DchcaVDMbykpdaVoISBjqb0Nb+r THmxzMcBQ4thps11WGAqqEIVL93WCNsJvxKwxb4sX9BgRyDvzUmxq6zsYCGAf/+bjcej kV6qOhJ+zKFYXYIuoPNdUPDxItpHptOxEp/l4D8mtN01k2KSHVLxPC/4Scxu86kXEeEX RRitrdrZsTdibDUdlNDk3/9gpyLFkktNWihUw5CBX5RXmi/jeJdzLFmGTHXGuzs7ezgp 0t91NwKjjA3h5TKSKdGimrAllOS1/cXrDnxPfEG2InriWN925xzrwd459ca5IZ+riAmw ppWA== X-Gm-Message-State: AFqh2kr7jImDH4dKcz365vZXuLX3UYUSQSl6ll9T1SSmdNkEajoKTQ9s dprcNrstzJND96MYP+Eos7txPvxvKck= X-Google-Smtp-Source: AMrXdXszwUWrk3+ezT3vpCagEl0mvWZa9sEHax26SYGf7QapJl4fBzpxwL0/6TcVCB7ktvgIaczCkSx6r1s= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:fc86:0:b0:4db:3ac0:78a3 with SMTP id m128-20020a0dfc86000000b004db3ac078a3mr5702708ywf.266.1674848538548; Fri, 27 Jan 2023 11:42:18 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:04 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-28-surenb@google.com> Subject: [PATCH v2 27/33] mm: prevent userfaults to be handled under per-vma lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194150_605806_4C3A11B4 X-CRM114-Status: GOOD ( 11.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Due to the possibility of handle_userfault dropping mmap_lock, avoid fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Suggested-by: Peter Xu --- mm/memory.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 33ecc850d3cb..55582c6fa2fd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5256,6 +5256,15 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_start_read(vma)) goto inval; + /* + * Due to the possibility of userfault handler dropping mmap_lock, avoid + * it for now and fall back to page fault handling under mmap_lock. + */ + if (userfaultfd_armed(vma)) { + vma_end_read(vma); + goto inval; + } + /* Check since vm_start/vm_end might change before we lock the VMA */ if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { vma_end_read(vma); From patchwork Fri Jan 27 19:41:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1EAAEC54EAA for ; Fri, 27 Jan 2023 20:04:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=7zdiiIxkrObb/gwlzbsHjZpV2uEBfd+znJA1y2K+quw=; b=34o4VA5M0N/QjewAyhXACDUKZ7 wnQUqTFEyEa6w3FDQel9mBovHk8st/7GG9fR0Mb22cfIZ6sFfPdjKeSXenFiIYhIaZ9hFhAsvkV/3 GI000b3LBvGYdeekx7nD5UcbV/wBZ9Yv4Pp0dMFY1J9DZa8nX+wA1UJdiaSJoZEwqiU4kTsGdaHxq wPJnOVsrY8rNXjpC5aZwrWOIkf4g7Y3hEC4LFoq1i5CslsufQDt9XAYniKoslF5Zohdh/yY224Eq/ 70Z/+7DNG8zHklOfV4bhaBWVki3L5lZeq2TuQ1HQYkVwFlRx/nXmZYCmaBr/y0gnS7gpruDXc470n nkC//Y/g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUvw-00GH2J-O6; Fri, 27 Jan 2023 20:02:49 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUvc-00GGui-Tt for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:02:29 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=QTC+ocJD/ZK0d8AWah1kQLEh3QM8J68abwl+vLpX/MY=; b=dl94TIFGImdjZCzvA5nQ1f9f0m nJTvb6P+zLuiOpLXHOpqSEc3TNcqMpWf5cPjGcwWaqBBzmF+uHl1Yh8Fb1fgTcnJmoQgDq11N0kPn Iw7ciD7HBOGkbkA/BIrMilglZXBxY1aKECMMET7894jBmMgCJ7bz8qYI5mpYqXywwXX7aXhFXyupU OUeyZIWGAy3e9EkdM/j2o9CEeVonw+P4g3iPgK7ls8MpX7HNF9L8CONg6YCaoNjctjGwRZu41b6sQ ElZ3j81CH5GSqremFKpvcaod0dfdoTY506sSo4iNvB901nuW7XuaEfL55TFlFYJKwwhIiN5+BHO0P NeLh93aQ==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUcD-007zPV-Ma for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:42:28 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-4fee82718afso65299487b3.5 for ; Fri, 27 Jan 2023 11:42:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QTC+ocJD/ZK0d8AWah1kQLEh3QM8J68abwl+vLpX/MY=; b=mU8gu/pWf6zIsQFo9qZJmzO+C9Rz2zywIzJAt0hiLsBb0sfMScn0ga1WT0lJP4+cuU jnlE49/hYrnAMYh/RLEYzocL5Jy7mfCziMb33lJyjegHt186LkbV/WCwUSBxzV0FGK1I xrqhc33jx9MJLtdss02+APSdeRf2XUnFG09IUKfXCIr+hkTGBQo7qLvcD8/EZZarD0c6 9RiTDx0vtERha59paJtaKyAUCno0Vzl4l7lvqpJzE2nTjhpBkjOXcTvShnV68JU8Wm5n IoRxYa+gg9A7UVOaBxUJpIYlD0HzhsBk7iegMRGVNF5IQVYNK1FSZGz91UjWgsue8k5N kYLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QTC+ocJD/ZK0d8AWah1kQLEh3QM8J68abwl+vLpX/MY=; b=SGwBWxUn7VhDCnNbf9AZlffNo31N0Ig5ihJt1bHaKcHL59oS6t7cdPP1PykA5a0Bdr m3bqiWsgfvbghgOP6v5A1m7S0ft0ZUKv4+S3y7+WjyvWjSCvXmaCfioRRqoqrkf5cwkk NNgyIZN1VYc/BF5C+NENBm4DOz4kQRdV2XdCaSs8H68czYactS9dWucqyb67/zcdKAn8 PAgPq+STlzhNb6wTVzI7fEtFtF0H738+XFFKAjJgFO+BhntYRaJ4UIbT48xWKo0uava3 KS1WKN2c2eWfDhMOKTy0Po9dSqYxJzUSmsGBNydGAiTX8oxMBYYb7JK/4oEsjXvyU8yS nXPg== X-Gm-Message-State: AO0yUKWuwzX9pk7yRP0VxnXjkTfdsrJfzgFZ0ZOEbm7Ah0cK9whNRWdc PUbMM9AVJVo0YdsAnqpDy0mcQ1k8TDc= X-Google-Smtp-Source: AK7set93uG4w1kFwIB1zyvlDAWxZcHXpTslQnEHpzoDHLwlIEt0PkMVZGLSp9rtobNPjvtDDJk1UuyFV7ts= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:e705:0:b0:80b:bdc5:5060 with SMTP id e5-20020a25e705000000b0080bbdc55060mr930533ybh.27.1674848540634; Fri, 27 Jan 2023 11:42:20 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:05 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-29-surenb@google.com> Subject: [PATCH v2 28/33] mm: introduce per-VMA lock statistics From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194226_195297_CE81C09D X-CRM114-Status: UNSURE ( 8.81 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++++++ include/linux/vmstat.h | 6 ++++++ mm/Kconfig.debug | 7 +++++++ mm/vmstat.c | 6 ++++++ 4 files changed, 25 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 7f5d1caf5890..8abfa1240040 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -149,6 +149,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, DIRECT_MAP_LEVEL3_SPLIT, +#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + VMA_LOCK_SUCCESS, + VMA_LOCK_ABORT, + VMA_LOCK_RETRY, + VMA_LOCK_MISS, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 19cf5b6892ce..fed855bae6d8 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -125,6 +125,12 @@ static inline void vm_events_fold_cpu(int cpu) #define count_vm_tlb_events(x, y) do { (void)(y); } while (0) #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS +#define count_vm_vma_lock_event(x) count_vm_event(x) +#else +#define count_vm_vma_lock_event(x) do {} while (0) +#endif + #define __count_zid_vm_events(item, zid, delta) \ __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index 9191897e76af..4965a7333a3f 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -278,3 +278,10 @@ config DEBUG_KMEMLEAK_AUTO_SCAN memory leaks. If unsure, say Y. + +config PER_VMA_LOCK_STATS + bool "Statistics for per-vma locks" + depends on PER_VMA_LOCK + default y + help + Statistics for per-vma locks. diff --git a/mm/vmstat.c b/mm/vmstat.c index 1ea6a5ce1c41..4f1089a1860e 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1399,6 +1399,12 @@ const char * const vmstat_text[] = { "direct_map_level2_splits", "direct_map_level3_splits", #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + "vma_lock_success", + "vma_lock_abort", + "vma_lock_retry", + "vma_lock_miss", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Fri Jan 27 19:41:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119320 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8BD3BC38142 for ; Fri, 27 Jan 2023 20:04:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=9M0Y2GudgtY4UhjWaaqRUfEOjWso2SiosZqg9b38PsM=; b=Lwzjl1rH8rWXxF6+nLXf4r/xWL 8kdzk3nR+77N7IMAVtghClmtCIzt3CiKCzHjzWxlGtqvU5sjnXq3nuaE+NM7VDvVk1BnaDvV9spe9 cPnhP8k/j3BG3Lk+FCfozZqa4pbJpzULzwIYHrIIQEVl0Uxl1v2DBJIa9q3FCJVcBOAmTvOikMvwY 6ua7t/OXLESaR2GWgguU9a7tYtBzLAsgPzmjO6P4fvJ8tlmDR5CCJm9FTUpx6Xd0DtZfnEBO5Xuew wfz6UYwz4fT4U0U9k94OCz56Xqa4QZ+1Dq8g/+0Tnra0/XaLighjbhyXuwESf0rV/eFX2O2OTqgj1 G3M4Fr/Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUwE-00GH6p-Pd; Fri, 27 Jan 2023 20:03:07 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUvd-00GGui-Aq for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:02:29 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=zr26Tmb2M01sb1afZfnklHdnYeWn0/HWZzrpjXJYooI=; b=wYlelOvZsxbN6j0IzvNAKN4eg1 gsPUmBwC9mcWSkaBXm6tVmzsVbWHiJBKtAYqhH+zDSOXwRZGg49EBA55JTPIx8+b7CRxEBh3utB93 FVjr42M7PplTlnDgINy7LbzKaEpEZEpUCeM40eMijnrJXCSb0EoIA/oGM3S/KfI1mo5dvRyvvdnaO C1nCHAUR0XRBpnIBGF+OPZQxIXZEGXG8VERi3ODfZEpPmYw4o93hyuckJ5IUhESaVfp3s3hm7Vzep EQLsVGZYkRdvja2B6Uf6Avdpy4Y40Vq811b0K4SJrLrjqobgGE2FSbvSm/8Zqaofb4Msq/y8EHj8U 6xzPpdxQ==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUcD-007zPW-5R for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:42:27 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id b201-20020a2534d2000000b0080b87d910a2so6437096yba.21 for ; Fri, 27 Jan 2023 11:42:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zr26Tmb2M01sb1afZfnklHdnYeWn0/HWZzrpjXJYooI=; b=I2P3olZBrDoCFDP1jHcaHXqtsXyUzxQr/ws4wX6bBu3CUipyuxA31yRpWa3MnFa/SU pcf5+CCx+WnMbozlzDdIg2ALFwq/OFj8s/q24FIJwL7Tizh8I6eeGtbBFib4zJzLB5md /zjFFlwtWXGghsSwviKTLUSRcwBRmcJaXn6b7nEXFweXpvU3ZyCKtcIxikLC8+UQ9pUQ JujyWegWeeB7qz6qsENh+jKg1foSfeMqF01FuGu+Yb/GyFYoIS6vq3ynrguPkj0pOoLx BYJ8cK1I9Z+jOMo/TqTeh+dlSH9FVfZfZ84FkijhY572Ij8GlcP0ii8VORVWe0sgl8Ck Cp6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zr26Tmb2M01sb1afZfnklHdnYeWn0/HWZzrpjXJYooI=; b=1AwWLFHYwC4y41wszjTTjOcnoeA43n2TGkQUTbvidqfeFailObO2wFy/MDTIm78Fcw lg5guXeTEXBzre1/qP/qOR1iOwwaaw5t4EzxdOyactU9aM3HrE+2r0mGgG29ecttrz23 c5iAXwfQz/lfrTjxhprKt/F1qXI1NN7qkD4aiUsg0a76LzGseBZL7vview7RsnnDSidP HwpnuoTr5mbSujs0Izre919GZvRroMw55eoHnR9LFs/PClvzt06sgjPbmM/MyBkj8jVr akTFhS6M/KMJWLk7c70BAo+0IipDzJdf6z9eUgINMcYYVcK8Cc1wciSGpnotFj7dQF9z 45hw== X-Gm-Message-State: AO0yUKUXUAgKXvOVvn75WRAqzAz8dpvHKXMjAsZt+qrMWdQ7qItCp1RD 6BrGBwqG77vZP3VkrdhaW8tSXGqqnbY= X-Google-Smtp-Source: AK7set8yRf6g7ktuBKP0CQzBeFyUCv/mUJgTqrv00u2WfcKZqCCOgg0GVy3Vw6WGmXU8JBT03ehy/weY7OA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:eb09:0:b0:506:391c:6d37 with SMTP id u9-20020a0deb09000000b00506391c6d37mr2021270ywe.88.1674848542935; Fri, 27 Jan 2023 11:42:22 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:06 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-30-surenb@google.com> Subject: [PATCH v2 29/33] x86/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194225_665105_BA6A5F53 X-CRM114-Status: GOOD ( 13.16 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig | 1 + arch/x86/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 3604074a878b..3647f7bdb110 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -27,6 +27,7 @@ config X86_64 # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index a498ae1fbe66..122c8fe4af12 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -19,6 +19,7 @@ #include /* faulthandler_disabled() */ #include /* efi_crash_gracefully_on_page_fault()*/ #include +#include /* find_and_lock_vma() */ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1333,6 +1334,38 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_error(error_code, vma))) { + vma_end_read(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR, + ARCH_DEFAULT_PKEY); + return; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1433,6 +1466,9 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (likely(!(fault & VM_FAULT_ERROR))) return; From patchwork Fri Jan 27 19:41:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119350 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0F004C54EAA for ; Fri, 27 Jan 2023 20:09:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=dcZ6NXBiAlT8nIy9eNipvJyKC9Pb4jd4bRbbsjC34Qg=; b=RaZ6N0/vfLdFdzofQl+SAcPzTF mce1r5GTx12nMPXYEdGS7vvYghojh2w6NVSxmGgJ0M4uHyUpNl//FF8um/9SeVuUtLNBaYlc8I3sR 6w0qkYgCefBeFg/bOcBlG7zXZuB67TD/6Fu9Rk8bqLfOMXnZDb3VBhWOThV1GCrKxllB7VO4shc+0 PS3AjyBTMLR3nBNg8qhLAOl4TP9fmC0C6JVsSJXR0i+jw0O38WHz0P1vB+Y7rAuJRH5jYJJ4lIN2y WHoIJuA9fULnra2Cj9XXTRAV/fojL1+8TC4MN0Lz8hbGExZeu2m4PK2WthfRtelxZvXuRsMgTwxWi 7gykQL+Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLV0e-00GJ0f-3G; Fri, 27 Jan 2023 20:07:40 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUww-00GHN6-8u for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:03:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=MNwo0rhG4y8C+tQG7LPM4nLJWTbIHprc8uVQsZA2GSc=; b=gCTim006WAKMJNQPiakn8XDWjS PVzXyM/lTD+nG+qggteCicdfrpfqEE/Rskn78EC1Z4vCB7DRH20EeLvC/k6oS5o/OADOLJHBihhMj QgoDBA7HmQSFVDaIlqEuhF0s/oxUJtS6ZWTfGsFGhPJsH4F9eATMf+cINSNLBbTxxFO7GMFN2KHBA dHZTJCdp8QHi+9wBBT0d5BzWQjv5YBvOJmpL822DtotBDi5RFdne8/zbGIJHjfv+RbuLKJW2fdSvx 0CQHO5PX+BQDIS9kTOr2PNj60tRepTk2ULYqsHd8R+Y8zYT7/DzJ3Fb90ciECACtmWfpiTZHVIQmw qyIQUsgw==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUbi-002xug-3B for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:41:57 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4fa63c84621so66299667b3.20 for ; Fri, 27 Jan 2023 11:42:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MNwo0rhG4y8C+tQG7LPM4nLJWTbIHprc8uVQsZA2GSc=; b=JxxlO5586M0z4sVDSxHugqEEADB1cAjEHo2xluVRFI46WqHTOUsszCaJjysGoPqCq8 ztKkJGmntymAGuEGxdM2A7EX8Sykw/reCFrQFVsgm7kaRoHFsG9BNF/PmmjyzPElNA3A u2zVx465PKwxuuDkWTqvf70dFDsT4sE76+Q1i+nyNMyErq//frz9WqLOph65UMZVM1Q3 1JnBBBOhnZpIDf2DrbbDSdhADbiUH2oov3flacw7sejkvu/YDNPNLxViqgxSV8aAiDpN WEe8RN23e0S32q4tATcZ7ZhVfhZ7+NOeU1bzW/fjTQ6HzW6TJlQUp4mHpwvwuOr4rhfO QkKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MNwo0rhG4y8C+tQG7LPM4nLJWTbIHprc8uVQsZA2GSc=; b=RGrHfU84er87o8TU6d475Ky1anYgbGk/HWNf6d8YepCSNreD+3X9K26jcnfvgbliK7 /s1u173SgN6gA9QyQps3iP592bnb+S5LORIZLIXNcsq8y7q0qKeK4cYodG8RMpdp+i1w 1VE4MhqxsVtkTwewEh0QqiuNwexewkNQqkop90YvkJhpsLORVrLjtyQrsfH2cwNA5RU7 IVZKHFUaBhyknpQaSBNZNUBWSh8i1t6V6XCx8HwPlPTN+I/std80rhICiAQ73WkrN+pT PKMFK1pJ7fR7VW3tGGqaXtjZlQi5G9ULZvRakcwlU/t6UGB8nPAvW5TJ1qglvvRfhaSk YtIA== X-Gm-Message-State: AFqh2krNNZG5FQ8Or/E26ueh0tYavcl9bTZ5uA5a0N4OzxUnF3mDcLjh vDkI7FTS1hzhYbQLPrxcF8R7L85Ds88= X-Google-Smtp-Source: AMrXdXv6cPUTMY06wj7LHpX7SofIKh9xmcnJ1P5+6wR9m3ToK9ikAh4HiQMQbcnL8It7MYI0W5hAcMnIURQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:5b88:0:b0:462:e278:1a08 with SMTP id p130-20020a815b88000000b00462e2781a08mr5361156ywb.243.1674848545116; Fri, 27 Jan 2023 11:42:25 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:07 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-31-surenb@google.com> Subject: [PATCH v2 30/33] arm64/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194155_091037_26627F9A X-CRM114-Status: GOOD ( 14.07 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig | 1 + arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index c5ccca26a408..9f2c0e352da3 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -95,6 +95,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index f4cb0f85ccf4..16588dda9258 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -535,6 +535,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); +#ifdef CONFIG_PER_VMA_LOCK + struct vm_area_struct *vma; +#endif if (kprobe_page_fault(regs, esr)) return 0; @@ -585,6 +588,36 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); +#ifdef CONFIG_PER_VMA_LOCK + if (!(mm_flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, addr); + if (!vma) + goto lock_mmap; + + if (!(vma->vm_flags & vm_flags)) { + vma_end_read(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -628,6 +661,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */ From patchwork Fri Jan 27 19:41:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3086CC54EAA for ; Fri, 27 Jan 2023 20:03:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=vqVwsdtt0Y761LqGRSioLjRZjclFE8m296e30FqdZH8=; b=V611MfBu9n/SKVEdn3r+1eTyA8 Aed9GHD+LzSdK1N1o9Bv90LKdtDsBEnNfvjpcPJxw5J0zYoZwNSQ6frVnQJ8kyZkqgBUpG1nIgwgH jY5eaFKkC3DEUajv5kYDYYpo5cv6Ij6y1httDBIsmnEIsyzauH5qwviLTR/xwEYYQ2hIVwkmkvEh6 FiXAMrfoQiZOjbbPVtEBfsk7a4TksYVo/LtoFrF6YbYoxVgoFxFkadUKhpHWGtnHo9+pUHnPEf+hk xcW67fa2To3h8BB/7MG69UAlAYj9pifMu8iC0rJX8PZT/mFBUV76hzWdDNhgplXUR5d5e7pDBJeYx kh65r5+Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUvd-00GGwn-TJ; Fri, 27 Jan 2023 20:02:30 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUvb-00GGui-R0 for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:02:28 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=kzxeHm8qCtyYeQADp+m05TsFOqarDXtVrIERGy2x4bE=; b=HzpLwR61iYZxqZzIQGFlNvPQmx HmNhNOi8jeTETFKP8087D0BG35GgunHf71B59y9M3fJ3cJwnExnZG2+8fnh9NO+uzahqEwBUPp2U1 m17xURi48ktaQcE2efWofwi7Cerb7Cl1dSsko8tLooQd0qLZW/0+yjGFSiSBWdyKAs2yQggwME13L Bsbp8G0zXJ8a02qs7PbvJl8TF0DPD9PQ/zso2CY1+2JQvbqCgTiYwk72QWE8ZA19o6Jn/48JgoI15 FL1zIVzucA6JA99rFdCVNfxEf61R3HvbdKkr1zNBDPzGK+4rmDbMlCZFN8+0WGDFIg/RX57YYXDSE /3dcRixQ==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUcI-007zTn-Aa for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:42:32 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-4c11ae6ab25so65260057b3.8 for ; Fri, 27 Jan 2023 11:42:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kzxeHm8qCtyYeQADp+m05TsFOqarDXtVrIERGy2x4bE=; b=Qw/TzqObuxKUrxtkILJdFMJrnRXDl7IhHsfqWYsg15Fht0zJymOStyW6umwGn696mL /vz0L3L+AxNuyNMgLOdykFyo3ffxujr1NJ9PZ/3TCzeykyM3g1PjhFRGoo/e9FoO8fmJ 0gJOei12QFUXaa/Oa6hhy8ENMFQY5aEUa+SKaoSJiwpGU68x3ruM/ipRGHnS2jScV2XA EPWUY+rZwtRmlqA25G6oa8LHcJDQM3bZQ4diO+PFB7fofYZtcMVMGn5EMXtqF1YwdUnI Yur83PIWndlmZU2J2xMV4at0clCg0WXiP4a0wm/tFrIYOxaiWzueFhV10wBQ62TvKtYz 5Efg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kzxeHm8qCtyYeQADp+m05TsFOqarDXtVrIERGy2x4bE=; b=bjhWW+gGx1h8JmqsapwCGX3stqy/Q4b5xLq3JDSDAPBE1AZKrS6B/e9AI6MdlIgJ5T NbYlbfdZjMUqfKTlTWQPQNJ1FmRWDeAYF20dQtML5+y1y+N2lLpgoEBLHCpgA7L3xdwM c2G8zUnmeNRGCpaECd8ZtHfc0Zyd6XjZ3soWuyHE3TKnaD51vANXHx+yGaFT7UxO1KAg 3ionFHNaRf+6tEeeyY3U/Z7lQNp5lU7I6kZCa7+ySAm+A1sWbh8Wfi8TwVHL7uTQ1b8t eanLtsuoAXLIz0UeZRla6sRu26gfVlnGSEuWyUOoHcqu98iEreUuVAvBxlIeO4cdiAJA Uzhg== X-Gm-Message-State: AFqh2krInOJ4ydRsQoFQgR7dImk0V2yBGACiM3Bd26izvhTOFyClfVNa VOXmYmUvu61SUIonl5dwsPltlUl4PbQ= X-Google-Smtp-Source: AMrXdXvuIF+ftjjZVhAq6LLQlzBTotqsaAzARNS7Nb+n90q796c/W7VGIXChRGDMsXog66D2ETjcV07ygFs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:1754:0:b0:4fd:ff18:de6c with SMTP id 81-20020a811754000000b004fdff18de6cmr2568712ywx.227.1674848547503; Fri, 27 Jan 2023 11:42:27 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:08 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-32-surenb@google.com> Subject: [PATCH v2 31/33] powerc/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194230_820696_7C172F93 X-CRM114-Status: GOOD ( 13.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/powerpc/mm/fault.c | 41 ++++++++++++++++++++++++++ arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + 3 files changed, 43 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 2bef19cc1b98..cab229e75018 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -469,6 +469,44 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + int rc = bad_access_pkey(regs, address, vma); + + vma_end_read(vma); + return rc; + } + + if (unlikely(access_error(is_write, is_exec, vma))) { + int rc = bad_access(regs, address); + + vma_end_read(vma); + return rc; + } + + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -545,6 +583,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, mmap_read_unlock(current->mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig index ae248a161b43..70a46acc70d6 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -16,6 +16,7 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP + select ARCH_SUPPORTS_PER_VMA_LOCK default y config OPAL_PRD diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig index a3b4d99567cb..e036a04ff1ca 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -21,6 +21,7 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB + select ARCH_SUPPORTS_PER_VMA_LOCK default y config PARAVIRT From patchwork Fri Jan 27 19:41:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119354 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0AC13C54EAA for ; Fri, 27 Jan 2023 20:11:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=0MdB8p0WsqbDXRriE2mO5dGtTCvv9HqjjZ7OvovrcSk=; b=xEf5TFJSmOZlU1N8Ov2FRXzKk9 FqR6Kx1kidIKqjxYqWOZ+f6yNXgexNtKyVJxgVxXujqxxUEk8ARQ7ke/YGbWk1632QJkOmLKtdNWA VxzJQJdu3e988xhAfTbr0QFYeCsXk4zvfh/RiulDMeekUolIFtPCDpTPyfEMTveHgk+GsXrNfnq9k W7UFGO8acj87rk46WyqsaZJSc5+WXT2C0yARgpyU6i5NS+FgVzIAqJnXqd5GLKbh1ARSzMRsfjs65 y3jp49lqnxNkjEBIx6h+vCnQoZb30/LFR1h/IJKIi+gM1/CLdJnrEODaUgwpEZiwhibdDojA1q565 2WbEM6Eg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLV3V-00GK5s-Ri; Fri, 27 Jan 2023 20:10:38 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUxK-00GHXe-V8 for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:04:15 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=garDV9GEiqmooU30u22JnQrNH9 VRPKo27fv0V2Mpa0dttnWJuu6wWw2YNd+n9yU3lKS8tbQTaOfG6sXs2+F0cOkSb0BPavuPTUsxf13 iYIloxN3YrzYnsLf0lEfmv2JOjlpaV8H/PdVNp3D1YwV4sPjUW6LmJ2pQawzQuozbpDUXZMtfNBlD 9h0+8xwK8GCQRXzskplni1ZAOfOnnoXz6ZMDtnESVIXA4WL0htT3BIi/Op4/VIfXj8ACrfHgNpJt4 DgSQNos6Zp+ahRs71Bic16ATCqVGBdXKaow1U3zGc/iVVh0i4rM+iPGMlRHeIeTnwCIbqMZIwvx51 rwHGnCxw==; Received: from mail-yb1-f201.google.com ([209.85.219.201]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUbq-002y1z-0i for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:42:04 +0000 Received: by mail-yb1-f201.google.com with SMTP id z9-20020a25ba49000000b007d4416e3667so6421556ybj.23 for ; Fri, 27 Jan 2023 11:42:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=R8vm5sGePmLU1Pe4mKnKd7BNiKDqaMQRSE2Uxaf9jHVe/UX+8wZaZnli/NAPrvHh5D Gp2YVdh1UhOUz23uxWg1VsM9bUD5cup3/ZbXShp0DX2DQS4OYSwXrDDdrwmKhIAFH0Hw 74ENSW8vtZuV6jUaLsmkcaEy+yDT+z+95V7siTwW55EWIBuns5aniQb/TRczPRcRq3g4 VhtuB5yMK58BxiemGZ2CqsONqT2FuthDA4119d3zQhl7jpXryXulujWcH7o4rSRWH8Tl w610sqrsMHevwvr509uliYrTdDC7Uzh2VKUBma4LjNdxi7kycc6HPZnQZbUV0crFM3wd Zbgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=sKl9u03BZ3LNnbAce7KqT04GJ/BzJ1Otdnh4lP7HYSlJ0C5762uqBp5r0IyPKvYXGW rNYLqubiiAIAX1wEWnKvvR1sBjv9bOJMk9Gu1IimaZVgobNIErIu3uIJvyWwR1PwTJTn /XkH/c+rqQ5mCFVNJgJ4dfhnCsTBNhr0/5QrfdGa7APygdWEA8hAhvSc5Ku+lGgESlPA LurvlAAqB4HgRDT7cybRwcy+qCRoSCz9PANXS0jkHAbCK/KERrcMIhRNZQgx8IKzF2e+ PxrOMl3HypZJsLpXwibUUu6mRpC8ZqVLoDrcG76wJ28Uu9733FtygJhhyFmYSOy52R02 gsnA== X-Gm-Message-State: AO0yUKWGo1zM4hEG7IPJe8dV34hAqHis05J8AwnVjcO9d+KCeT+yWZaI qtMjg2tN5ZqAb6Qde/OEdh+Ox2Djvz4= X-Google-Smtp-Source: AK7set9ZwBhHaiq8PgHnHZpi30bVFoqemTYpH4CqkWTR3ntjpG4ZYW0UY93Gsvx11KtEAuMPsZpv9FPCPgs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:db07:0:b0:80b:8dd0:7b35 with SMTP id g7-20020a25db07000000b0080b8dd07b35mr1614607ybf.322.1674848549978; Fri, 27 Jan 2023 11:42:29 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:09 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-33-surenb@google.com> Subject: [PATCH v2 32/33] mm/mmap: free vm_area_struct without call_rcu in exit_mmap From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194202_276207_333C2D42 X-CRM114-Status: GOOD ( 13.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. Because exit_mmap() is called only after the last mm user drops its refcount, the page fault handlers can't be racing with it. Any other possible user like oom-reaper or process_mrelease are already synchronized using mmap_lock. Therefore exit_mmap() can free VMAs directly, without the use of call_rcu(). Expose __vm_area_free() and use it from exit_mmap() to avoid possible call_rcu() floods and performance regressions caused by it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 ++ kernel/fork.c | 2 +- mm/mmap.c | 11 +++++++---- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a6de58bb40c7..1c4ddcd6fd84 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -250,6 +250,8 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +/* Use only if VMA has no other users */ +void __vm_area_free(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/kernel/fork.c b/kernel/fork.c index a08cc0e2bfde..d0999de82f94 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -480,7 +480,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -static void __vm_area_free(struct vm_area_struct *vma) +void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); diff --git a/mm/mmap.c b/mm/mmap.c index 3d0cfbc92745..1028fe131bb7 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -133,7 +133,7 @@ void unlink_file_vma(struct vm_area_struct *vma) /* * Close a vm structure and free it. */ -static void remove_vma(struct vm_area_struct *vma) +static void remove_vma(struct vm_area_struct *vma, bool unreachable) { might_sleep(); if (vma->vm_ops && vma->vm_ops->close) @@ -141,7 +141,10 @@ static void remove_vma(struct vm_area_struct *vma) if (vma->vm_file) fput(vma->vm_file); mpol_put(vma_policy(vma)); - vm_area_free(vma); + if (unreachable) + __vm_area_free(vma); + else + vm_area_free(vma); } static inline struct vm_area_struct *vma_prev_limit(struct vma_iterator *vmi, @@ -2134,7 +2137,7 @@ static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) if (vma->vm_flags & VM_ACCOUNT) nr_accounted += nrpages; vm_stat_account(mm, vma->vm_flags, -nrpages); - remove_vma(vma); + remove_vma(vma, false); } vm_unacct_memory(nr_accounted); validate_mm(mm); @@ -3083,7 +3086,7 @@ void exit_mmap(struct mm_struct *mm) do { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); - remove_vma(vma); + remove_vma(vma, true); count++; cond_resched(); } while ((vma = mas_find(&mas, ULONG_MAX)) != NULL); From patchwork Fri Jan 27 19:41:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13119356 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 78934C54EAA for ; Fri, 27 Jan 2023 20:14:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1f2f35402167CP0r0T9mDDhixnJsYavkeNgyQV6986Q=; b=vR24w5yMw6WYtL2GXntIPYxaUZ J1TM7R8bwkWk17fNwCBstBj/+hXQh2td0isbm4sBQuH/BEB1rMRpW0Ic63vLhgveltQxFFJLwMrM/ ipyqRt/LVQzkEPm0DTAcg0EpgtqkkA/ro1XNB4BE4amatSd5hA/Gc5od0i/71u4W1cqalGcekiAED 3hDfQ0Ytzd+7QM5UXgdvfi33SIN0bctZkkIkawYMHEfAPpZvcnXxiZrLABRIXT2R7vRuNq2eCuXyu UO5FotfQ5+n8ZDFrUJAuU0aojbAsI99URnpU5tFxrWKz9KHcqlTg4RXZ9VDeVESecQIkt3ywqWJAY MBsqIZ4g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLV5N-00GKkd-Rk; Fri, 27 Jan 2023 20:12:36 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLUxQ-00GHZb-GV for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Jan 2023 20:04:20 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=Ud3T37IGHP0g1lmdTa/tq8X3xMhLxr1AXmnGvd1U4CI=; b=PwLPBQWgYgFYRvZOyF0h0wmnPv HdtX0m6ZNHqbrAP6nam7rI8YJMjj0n3Qmc6HwuF4Pfi8qE8S+1eIpylCB91DvUaMlnUIXkAWIaA41 94P7nuqtDrtzm0YTdtiJLxZIQNcQErl37Mj1aIIyv1lMgo0NroAe6wtCLeTifIFZO9go8oW/pr/dP cFZrvg7vWhRiKw1kd4Y7xUUdcvPyxDlcUafpYkH/YMUKqs6CFD1lbttHJOpqrepBoVhDjIGb326Sh 2NnLyEb8rJ6o3y4UiLC+Ytn5hAQCNeucPuDmx9DQMe0RaAIvXAmh44RBpnQiE0xU/jIDl5WdbYi4K m6Xz22fA==; Received: from mail-yw1-f202.google.com ([209.85.128.202]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pLUbr-002y2W-2F for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 19:42:06 +0000 Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-4bdeb1bbeafso65158917b3.4 for ; Fri, 27 Jan 2023 11:42:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Ud3T37IGHP0g1lmdTa/tq8X3xMhLxr1AXmnGvd1U4CI=; b=FNDN2+9L1Vmxr+/mMF6eML5JcVJbNLvQW3fVgvwdePzy3lehe5YuSr9HWsNaMC4a8J FUhI3Xak6vLO90aUXXpIFs6adKFN80x7s3bJgLZLYfmaGEKhkkZLbtXJLX0N/ZyPrSOD vT5YWUMa6D2YWUL+XcKLQxhxuZ8Qre6a0DkbEZ1ct6ULdZqWxPdE1UEo8jeDeLLPMaSB b3W+W/qkByIa+a5SR1bxuJVPYAEoRSrded/1J6aiwU0I/kursScnVvXrI1MYVCgC0Qld xz7h6VUcKpIFgOOniSgMFSpFb+kGVYjmhq6wkneKC/yuMYuw9ACcdFQ59agxDWycdAy0 DJpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ud3T37IGHP0g1lmdTa/tq8X3xMhLxr1AXmnGvd1U4CI=; b=OKRmBCuxy7yD798jkF++kt9ZPvHpY9/OWTVYT+oHHjfuKOH9/XpqDi4QGxoovdL+Bg oHhbuHVXP4aNKMBL2leTsJNlE6lvWWL//taKWp29NIY0QKsAzPHlJuXu+VKwKGZzvzP7 Fo98102xtlLlWWUZDRhEYMYJ1FD5rgNh9I89aZ9DaULK3hkInNRPGvB0wq7KEQDLvWUC W1ONvwSCVDCDXKQrQkg3n8keydy6MlZk8pKuF0mZyob3pETa9c4XVB1MDaUdBRtIqv9+ Nb5mw6RfiibG80mCuG9ggffWPRqB5HoFpRo4JVfWR2jN9uYwA3EkM13zg8/2jsOp4dK6 9Vyw== X-Gm-Message-State: AFqh2krvqCRzHc887aLou0nvfHaHZiGrOY4REui/shvbReNTQ9ame5E2 AuZVwklIWWlZmq9q4a5CtHwEFkMaAlA= X-Google-Smtp-Source: AMrXdXuWd7O6+z302DT+NWEpQuMGblny5VWrJhlz+cYeowewBMKBJvug6bhWSAlG5JyuNtY5PTSWQDkfixM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:be06:0:b0:4b2:fa7c:8836 with SMTP id i6-20020a81be06000000b004b2fa7c8836mr5004644ywn.195.1674848552353; Fri, 27 Jan 2023 11:42:32 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:10 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-34-surenb@google.com> Subject: [PATCH v2 33/33] mm: separate vma->lock from vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_194203_817835_FFB3076C X-CRM114-Status: GOOD ( 24.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org vma->lock being part of the vm_area_struct causes performance regression during page faults because during contention its count and owner fields are constantly updated and having other parts of vm_area_struct used during page fault handling next to them causes constant cache line bouncing. Fix that by moving the lock outside of the vm_area_struct. All attempts to keep vma->lock inside vm_area_struct in a separate cache line still produce performance regression especially on NUMA machines. Smallest regression was achieved when lock is placed in the fourth cache line but that bloats vm_area_struct to 256 bytes. Considering performance and memory impact, separate lock looks like the best option. It increases memory footprint of each VMA but that can be optimized later if the new size causes issues. Note that after this change vma_init() does not allocate or initialize vma->lock anymore. A number of drivers allocate a pseudo VMA on the stack but they never use the VMA's lock, therefore it does not need to be allocated. The future drivers which might need the VMA lock should use vm_area_alloc()/vm_area_free() to allocate the VMA. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 23 ++++++------- include/linux/mm_types.h | 6 +++- kernel/fork.c | 73 ++++++++++++++++++++++++++++++++-------- 3 files changed, 74 insertions(+), 28 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1c4ddcd6fd84..52e048c31239 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -621,12 +621,6 @@ struct vm_operations_struct { }; #ifdef CONFIG_PER_VMA_LOCK -static inline void vma_init_lock(struct vm_area_struct *vma) -{ - init_rwsem(&vma->lock); - vma->vm_lock_seq = -1; -} - /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -638,17 +632,17 @@ static inline bool vma_start_read(struct vm_area_struct *vma) if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) return false; - if (unlikely(down_read_trylock(&vma->lock) == 0)) + if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) return false; /* * Overflow might produce false locked result. * False unlocked result is impossible because we modify and check - * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * vma->vm_lock_seq under vma->vm_lock protection and mm->mm_lock_seq * modification invalidates all existing locks. */ if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); return false; } return true; @@ -657,7 +651,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); rcu_read_unlock(); } @@ -675,9 +669,9 @@ static inline void vma_start_write(struct vm_area_struct *vma) if (vma->vm_lock_seq == mm_lock_seq) return; - down_write(&vma->lock); + down_write(&vma->vm_lock->lock); vma->vm_lock_seq = mm_lock_seq; - up_write(&vma->lock); + up_write(&vma->vm_lock->lock); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -704,6 +698,10 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} #endif /* CONFIG_PER_VMA_LOCK */ +/* + * WARNING: vma_init does not initialize vma->vm_lock. + * Use vm_area_alloc()/vm_area_free() if vma needs locking. + */ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -712,7 +710,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_init_lock(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index c4c43f10344a..1e97bb98197c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -471,6 +471,10 @@ struct anon_vma_name { char name[]; }; +struct vma_lock { + struct rw_semaphore lock; +}; + /* * This struct describes a virtual memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory @@ -510,7 +514,7 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; - struct rw_semaphore lock; + struct vma_lock *vm_lock; #endif /* diff --git a/kernel/fork.c b/kernel/fork.c index d0999de82f94..a152804faa14 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -451,13 +451,49 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; +#ifdef CONFIG_PER_VMA_LOCK + +/* SLAB cache for vm_area_struct.lock */ +static struct kmem_cache *vma_lock_cachep; + +static bool vma_lock_alloc(struct vm_area_struct *vma) +{ + vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); + if (!vma->vm_lock) + return false; + + init_rwsem(&vma->vm_lock->lock); + vma->vm_lock_seq = -1; + + return true; +} + +static inline void vma_lock_free(struct vm_area_struct *vma) +{ + kmem_cache_free(vma_lock_cachep, vma->vm_lock); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline bool vma_lock_alloc(struct vm_area_struct *vma) { return true; } +static inline void vma_lock_free(struct vm_area_struct *vma) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) { struct vm_area_struct *vma; vma = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); - if (vma) - vma_init(vma, mm); + if (!vma) + return NULL; + + vma_init(vma, mm); + if (!vma_lock_alloc(vma)) { + kmem_cache_free(vm_area_cachep, vma); + return NULL; + } + return vma; } @@ -465,24 +501,30 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) { struct vm_area_struct *new = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); - if (new) { - ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); - ASSERT_EXCLUSIVE_WRITER(orig->vm_file); - /* - * orig->shared.rb may be modified concurrently, but the clone - * will be reinitialized. - */ - data_race(memcpy(new, orig, sizeof(*new))); - INIT_LIST_HEAD(&new->anon_vma_chain); - vma_init_lock(new); - dup_anon_vma_name(orig, new); + if (!new) + return NULL; + + ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); + ASSERT_EXCLUSIVE_WRITER(orig->vm_file); + /* + * orig->shared.rb may be modified concurrently, but the clone + * will be reinitialized. + */ + data_race(memcpy(new, orig, sizeof(*new))); + if (!vma_lock_alloc(new)) { + kmem_cache_free(vm_area_cachep, new); + return NULL; } + INIT_LIST_HEAD(&new->anon_vma_chain); + dup_anon_vma_name(orig, new); + return new; } void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); + vma_lock_free(vma); kmem_cache_free(vm_area_cachep, vma); } @@ -493,7 +535,7 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) vm_rcu); /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock), vma); + VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock), vma); __vm_area_free(vma); } #endif @@ -3089,6 +3131,9 @@ void __init proc_caches_init(void) NULL); vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT); +#ifdef CONFIG_PER_VMA_LOCK + vma_lock_cachep = KMEM_CACHE(vma_lock, SLAB_PANIC|SLAB_ACCOUNT); +#endif mmap_init(); nsproxy_cache_init(); }