From patchwork Mon Jan 9 20:52:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 47A61C54EBD for ; Mon, 9 Jan 2023 20:55:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=uFZbM/rLgczzAF6bqP6nXKwW/+wunVqk33FGbQfzG3o=; b=A+ZjnI8gq99/6pRMLp3JVaMsPe bRqPeuVx0dOO6qc968EF31TCkfRHD9ZlXCbPcBdu1mpLIXwn3fnITf9EUNdm4gPOkWTKWoLY2uVyi QWI0AnPAsAD4Ni6oaozt5ejcGtYl/yDmmmNQ5yccJJlB54T4XUHWVpqhLH3jx0QcAy72zqnq9J3L4 A4wtdHCSvrLKQFsZSqxGeDm9nvzhR+cLvi3q3vZrRjQSuZakhjLm/grlMSMXhh0hHvWZ544Vz6JtN iHtYce4ZwFEBVL8fZeTyf8K+bupYaiOqugrN+A3ArD5A4+6pT248TaIoqUKBygQOa2rQITNcqjZDF bHcPDfwg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEz9Z-003zsE-GL; Mon, 09 Jan 2023 20:53:57 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEz9N-003zjL-1f for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:53:47 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id z17-20020a256651000000b007907852ca4dso10289820ybm.16 for ; Mon, 09 Jan 2023 12:53:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NqXJ+ZomBW9TS34c/jGDj4kdswhUlT56T9j0TeoADNk=; b=hV1eYOET6T2afidyHxuU+TDalPYH7SoLHvYl4BYwO2+JPtGGpug5jlQUoy9K4emxnp wkRJchtDj68k/F+2XFNh7yn/DcVU/S6b5BfTlEf/HmlwKUP7B8xL+hIWlClC+stgjn0Q yjEqU4w0bHW24Tkvl4HAvFsvboqjzGC/OqOppLTxN2NoinKdWAqCYIk7Wp05H1K9TySk fMjEc/Qrcjc9huA2llMuqioo8fVnZzgiNBMYUce/o/EkMO+/Qtv30xUsop9ErVhi9UD1 18fRcEvtN38NaeuSdMu7DJkgG+1MjHvnJbjraNd54FiH0BDnP0iYXASfyl6JxE09JvbK zNVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NqXJ+ZomBW9TS34c/jGDj4kdswhUlT56T9j0TeoADNk=; b=LAvJlGuGXeIm8KE5oI2rnsF4WYrjQ1KZbAUttTNOu9fGi+QXe7//w7bSgXSfO1Cp10 NEKM7qCVVgP855M/2bS8q+9hl+G5vyhWDkXZ6K2whKPL8GepAHBVVsLDG4mdWTONyV7k aaxFpUN8+qGKKOlkaB/oUl1jQc7AlfeCRB9d4WuBOyOBF8BkC92aDgO2Gs2LjR3UTNsz 4r6vNlFDD2200CkMcP82/tTgHV/XDkAhllrm5QpT/SDD7SfuVXgWgGNPX2rbU05i4A7v 8yshvfKf4srW7UV2siuf1FYyDevG8halOukpOkJU3Zi3zPKpTNepl2d6t41uLOdJnVya 1HrQ== X-Gm-Message-State: AFqh2krFlAtjUIzHPgOnpZMGl0VdJXUvwnLP7lL5Ss4x7LR87FB+VO8z 5wEcpUIrK5kHj2h9sddV77y8L203lWE= X-Google-Smtp-Source: AMrXdXsc+lRNNMUj/grXDfNh64QUbjHDxIxAaFKRrkqMjOPH6SvLaEBts0d+5j6B8x6v1ECq7EPEJR82hgY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:7189:0:b0:707:18f:7226 with SMTP id m131-20020a257189000000b00707018f7226mr210074ybc.505.1673297623117; Mon, 09 Jan 2023 12:53:43 -0800 (PST) Date: Mon, 9 Jan 2023 12:52:56 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-2-surenb@google.com> Subject: [PATCH 01/41] maple_tree: Be more cautious about dead nodes From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_125345_113514_F19E10EC X-CRM114-Status: GOOD ( 16.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Liam Howlett ma_pivots() and ma_data_end() may be called with a dead node. Ensure to that the node isn't dead before using the returned values. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 53 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 10 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 26e2045d3cda..ff9f04e0150d 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -540,6 +540,7 @@ static inline bool ma_dead_node(const struct maple_node *node) return (parent == node); } + /* * mte_dead_node() - check if the @enode is dead. * @enode: The encoded maple node @@ -621,6 +622,8 @@ static inline unsigned int mas_alloc_req(const struct ma_state *mas) * @node - the maple node * @type - the node type * + * In the event of a dead node, this array may be %NULL + * * Return: A pointer to the maple node pivots */ static inline unsigned long *ma_pivots(struct maple_node *node, @@ -1091,8 +1094,11 @@ static int mas_ascend(struct ma_state *mas) a_type = mas_parent_enum(mas, p_enode); a_node = mte_parent(p_enode); a_slot = mte_parent_slot(p_enode); - pivots = ma_pivots(a_node, a_type); a_enode = mt_mk_node(a_node, a_type); + pivots = ma_pivots(a_node, a_type); + + if (unlikely(ma_dead_node(a_node))) + return 1; if (!set_min && a_slot) { set_min = true; @@ -1398,6 +1404,9 @@ static inline unsigned char ma_data_end(struct maple_node *node, { unsigned char offset; + if (!pivots) + return 0; + if (type == maple_arange_64) return ma_meta_end(node, type); @@ -1433,6 +1442,9 @@ static inline unsigned char mas_data_end(struct ma_state *mas) return ma_meta_end(node, type); pivots = ma_pivots(node, type); + if (unlikely(ma_dead_node(node))) + return 0; + offset = mt_pivots[type] - 1; if (likely(!pivots[offset])) return ma_meta_end(node, type); @@ -4504,6 +4516,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) node = mas_mn(mas); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + mas->max = pivots[offset]; if (offset) mas->min = pivots[offset - 1] + 1; @@ -4525,6 +4540,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); offset = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + if (offset) mas->min = pivots[offset - 1] + 1; @@ -4573,6 +4591,7 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, struct maple_enode *enode; int level = 0; unsigned char offset; + unsigned char node_end; enum maple_type mt; void __rcu **slots; @@ -4596,7 +4615,11 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, node = mas_mn(mas); mt = mte_node_type(mas->node); pivots = ma_pivots(node, mt); - } while (unlikely(offset == ma_data_end(node, mt, pivots, mas->max))); + node_end = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + + } while (unlikely(offset == node_end)); slots = ma_slots(node, mt); pivot = mas_safe_pivot(mas, pivots, ++offset, mt); @@ -4612,6 +4635,9 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, mt = mte_node_type(mas->node); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + offset = 0; pivot = pivots[0]; } @@ -4658,16 +4684,18 @@ static inline void *mas_next_nentry(struct ma_state *mas, return NULL; } - pivots = ma_pivots(node, type); slots = ma_slots(node, type); - mas->index = mas_safe_min(mas, pivots, mas->offset); - if (ma_dead_node(node)) + pivots = ma_pivots(node, type); + count = ma_data_end(node, type, pivots, mas->max); + if (unlikely(ma_dead_node(node))) return NULL; + mas->index = mas_safe_min(mas, pivots, mas->offset); + if (unlikely(ma_dead_node(node))) + return NULL; if (mas->index > max) return NULL; - count = ma_data_end(node, type, pivots, mas->max); if (mas->offset > count) return NULL; @@ -4815,6 +4843,11 @@ static inline void *mas_prev_nentry(struct ma_state *mas, unsigned long limit, slots = ma_slots(mn, mt); pivots = ma_pivots(mn, mt); + if (unlikely(ma_dead_node(mn))) { + mas_rewalk(mas, index); + goto retry; + } + if (offset == mt_pivots[mt]) pivot = mas->max; else @@ -6613,11 +6646,11 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, while (likely(!ma_is_leaf(mt))) { MT_BUG_ON(mas->tree, mte_dead_node(mas->node)); slots = ma_slots(mn, mt); - pivots = ma_pivots(mn, mt); - max = pivots[0]; entry = mas_slot(mas, slots, 0); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + max = pivots[0]; mas->node = entry; mn = mas_mn(mas); mt = mte_node_type(mas->node); @@ -6637,13 +6670,13 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, if (likely(entry)) return entry; - pivots = ma_pivots(mn, mt); - mas->index = pivots[0] + 1; mas->offset = 1; entry = mas_slot(mas, slots, 1); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + mas->index = pivots[0] + 1; if (mas->index > limit) goto none; From patchwork Mon Jan 9 20:52:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9A49DC6379F for ; Mon, 9 Jan 2023 20:55:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=UqbgGdZqEvr/KdkDnCBvswvryo7SX0OTGMQuuh3yljE=; b=lFwfb7/qkLZiXaW6Oy27nXU6ca NOAef3jW5qxGgwp9dNOdOMPV4vWjSA+594qJgf3ERkWDpyr0PXLH/rKvxjYR5wXp/komqaZ+QOyU6 DFTipLBfK2AefLLhNNOumIbuhDpTqq1zGk1hFTPQ+gWuP7q8gYav/BJhJVX5WsYnc1+aH6nOXOGmP 8Vd8GN2CEE7uXqJ//WdXQX4P+OY0dSTisTqSRE8ewmJ/z3YNXhfH2FKkd3o6UGuO1NpxC3/y18nRp R2tpkXGwwIjsEFUn01q6KQUHyIFjOKxdzrVkxMF1VBtMiK0sziLe3hXQZy8sOALEyvFp3Z+6AATQR 5O1AY+2Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEz9i-003zwA-7W; Mon, 09 Jan 2023 20:54:06 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEz9P-003zkl-DX for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:53:49 +0000 Received: by mail-pj1-x1049.google.com with SMTP id 31-20020a17090a0fa200b00226e43409c2so3541595pjz.9 for ; Mon, 09 Jan 2023 12:53:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GU7SFbwLvAPdD9ofPJY8ZC3AFrLWlaqOvHr/TBsM1Ak=; b=cDmwrsTnYnxMRe2M80E4L23YmooQix5wB2D0eUb+oN2Hp3bbyg7HFQB/PgVNLoHyKT pK0aWtF4CTk14CmlPxG5u4iqhHXIU+wNQHLdGD94tpFP5D5/s9ok9CC5dVKy7ZTVeaNC t3/Bp5EGifXzniBaXl20JHyzpYwXEFHr6vzZDBUr27QQthKbHcJnIlWiGRlMM6UQg0ND gIbAYHo5+gFS5Z1JXyH/8Q0hICWyuNm77sn6kp0Bt2Cy3fEKAr803t0EnzzgxBkOELQu v2Q/SfqHP/lVu/l9N+v6WarwdrrXQ25UydUZpK32X5OyRK1NLw7E4rGZtvAQp6+5paD/ Xoog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GU7SFbwLvAPdD9ofPJY8ZC3AFrLWlaqOvHr/TBsM1Ak=; b=qoUX2SRa97lcfilpZxl+EMStXxA6t5/VqNUgmNByko3I84+C/X7KcUcBvpSdeelF4b bUMTnlmdEp83ev3Z6tQ7vJ9J0qy/cDm1Mzpr1UgIjBc3+Kw2CTJndKcpG+rPdEgh9IKT BIXNw7uBh4IPMHT75NoNMRmAKiFXiGrst4cva5Q/PMMr51gVdgfzcCjjWeqbP9URs42Y kyUWfur7L4auv22MAJ5WkscjapUP3LqOrClU4lBCJweYcJ1Cb+BfPj46gBXg/gMuyidE fs4XBHbn7YDw+R5dvWRQvjKDXOtAgIhVEFCVwrDrzUBkhza3BbmWfZJbM4GByu8AcMXt mbPA== X-Gm-Message-State: AFqh2krb273w48KxXYrXSrYn+jytNrYbURNkak1voBXn6iuduyDWWFmd Jo150fRs4ftZ0Wu4dS+015wHz0uTOlc= X-Google-Smtp-Source: AMrXdXsx8jOEqF8lBOHYbim74eUIuqk/BpJXbormK1OrkMgDb8c5pT4kOfmHApmrR7R22KnvvmwfjroHm1I= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a17:902:c209:b0:177:faf5:58c5 with SMTP id 9-20020a170902c20900b00177faf558c5mr4095626pll.166.1673297625715; Mon, 09 Jan 2023 12:53:45 -0800 (PST) Date: Mon, 9 Jan 2023 12:52:57 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-3-surenb@google.com> Subject: [PATCH 02/41] maple_tree: Detect dead nodes in mas_start() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_125347_524626_1BE5DC16 X-CRM114-Status: GOOD ( 11.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Liam Howlett When initially starting a search, the root node may already be in the process of being replaced in RCU mode. Detect and restart the walk if this is the case. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index ff9f04e0150d..a748938ad2e9 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1359,11 +1359,15 @@ static inline struct maple_enode *mas_start(struct ma_state *mas) mas->depth = 0; mas->offset = 0; +retry: root = mas_root(mas); /* Tree with nodes */ if (likely(xa_is_node(root))) { mas->depth = 1; mas->node = mte_safe_root(root); + if (mte_dead_node(mas->node)) + goto retry; + return NULL; } From patchwork Mon Jan 9 20:52:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094312 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0A46BC5479D for ; Mon, 9 Jan 2023 20:55:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=l5tPQGjEQg/59sF8T/yknO77HkLAqr5Tmc/pH3m2JG8=; b=Z97Rv0nryM1G8Fpeox0Iv9NQau w82VVauzflo0gF1h4foAu6CnsySp8NvFPie1hAO+LoCf1SuT4ZkGTb9worTtfnjrcP+WkIvZxVtru ZqQYhiz1UqaorJ77b8Ytp+c+nv4XNqP6cTRJ4mHWzB/kNcdnNobKmfCEuc2HOEfUcpxJcubd/VjmX 2PFs363GcdmoMC3+HJiigxFZ+m7/BokpYh9FegkFUF3QrydhbzSK5dohipOMwJeNMSnw2Pgid5+XS YUrkgKucWfiXgf0TMEdplv0SDFD1akRZEVNHDbxQ754BCjGDls1GcpVnjnEWK4BK9MdCYt9A9ex9a YWoeq6QQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzA2-00404v-J4; Mon, 09 Jan 2023 20:54:26 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEz9R-003zmU-LT for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:53:51 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id k18-20020a25e812000000b0077cc9ab9dd9so10215971ybd.8 for ; Mon, 09 Jan 2023 12:53:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tYS2Mdxn+jwv2Pf7FZ7D7lMnoZhtkSPqAfmT0KRte/A=; b=nFEf7/SuzvGfFfBp8alBHLcE/S20BVhkxOFCKJJxDEEpkBCqnPrEjXLQRPhpajIyrV J/MheNxTRWGkew5780i2M8uqtnyvNe0zoFPa24hnhtsW7xvdNOQ3PiEyoeTmA3WWPiUh QK2oHtldPphM+TCue7rjkADeiZ4F/aOO5f9+1p0Hswrqrn/9/CHXJyrN6vRAUq5GDqS7 Axk6TqH0bcf1b4jM3d3LsItdyw7HXfIeFcgC6ZYFCu5leTcjZd3mHlVr8yfm4xVeOVbq OV6UpEzwnJ3oSH64LbnS9rS633sT7MF5+6p/WT9/4pbYtB4HJqqA4EN1qZDYnvdMkGJo Aq4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tYS2Mdxn+jwv2Pf7FZ7D7lMnoZhtkSPqAfmT0KRte/A=; b=8RsJopj9DOkjQCcPXeelRBU+aap86tO8XGi9LbcLPBTukLuAxl/3VsEYFgnG0N5Ju2 IxihBKRKNzalsHIVLGx1sfAMCJ8MEE+BCrq8SKKDPlKaWdUMiW0epxy7H/gxfB4MH7Yc jbzycKE6Qyrg2mJT+7txsqShUpCjpoZZFBk+lDojsltstdN8j0IZKTYMJQCEYy74JOsO +AGX1AA1CreZx6UurHBDgdKapabG/HCpc/DwMDn42zzZJcdADvt4VdCu2Bg3Qg614ITD /0Avc2y+MeSKB6GLSBiVMyQNoEQKAnZh9udllJ9+8n2TAcXzgXfmZjAkJpCcHYIEgzBO OexA== X-Gm-Message-State: AFqh2koFWEmBAsGh9WXTGhanoRsgoJHkKvY+XlFx7rXGKFpDYz08FCdH LY25dJGvrV4OlVC5yahjy0ibSLWfFoQ= X-Google-Smtp-Source: AMrXdXs7OcQqihjg4pdS+IdnJHYFJuvOEzRgXy7ctwOn05/Em3c821T3ojLTZvqk1LIxMtAuoTuisCHM2Gs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:dd07:0:b0:7b4:db9a:48ae with SMTP id u7-20020a25dd07000000b007b4db9a48aemr1234542ybg.207.1673297628223; Mon, 09 Jan 2023 12:53:48 -0800 (PST) Date: Mon, 9 Jan 2023 12:52:58 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-4-surenb@google.com> Subject: [PATCH 03/41] maple_tree: Fix freeing of nodes in rcu mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_125349_758414_C4883981 X-CRM114-Status: GOOD ( 18.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Liam Howlett The walk to destroy the nodes was not always setting the node type and would result in a destroy method potentially using the values as nodes. Avoid this by setting the correct node types. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 73 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 62 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index a748938ad2e9..a11eea943f8d 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -897,6 +897,44 @@ static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt, meta->end = end; } +/* + * mas_clear_meta() - clear the metadata information of a node, if it exists + * @mas: The maple state + * @mn: The maple node + * @mt: The maple node type + * @offset: The offset of the highest sub-gap in this node. + * @end: The end of the data in this node. + */ +static inline void mas_clear_meta(struct ma_state *mas, struct maple_node *mn, + enum maple_type mt) +{ + struct maple_metadata *meta; + unsigned long *pivots; + void __rcu **slots; + void *next; + + switch (mt) { + case maple_range_64: + pivots = mn->mr64.pivot; + if (unlikely(pivots[MAPLE_RANGE64_SLOTS - 2])) { + slots = mn->mr64.slot; + next = mas_slot_locked(mas, slots, + MAPLE_RANGE64_SLOTS - 1); + if (unlikely((mte_to_node(next) && mte_node_type(next)))) + return; /* The last slot is a node, no metadata */ + } + fallthrough; + case maple_arange_64: + meta = ma_meta(mn, mt); + break; + default: + return; + } + + meta->gap = 0; + meta->end = 0; +} + /* * ma_meta_end() - Get the data end of a node from the metadata * @mn: The maple node @@ -5448,20 +5486,22 @@ static inline int mas_rev_alloc(struct ma_state *mas, unsigned long min, * mas_dead_leaves() - Mark all leaves of a node as dead. * @mas: The maple state * @slots: Pointer to the slot array + * @type: The maple node type * * Must hold the write lock. * * Return: The number of leaves marked as dead. */ static inline -unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) +unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, + enum maple_type mt) { struct maple_node *node; enum maple_type type; void *entry; int offset; - for (offset = 0; offset < mt_slot_count(mas->node); offset++) { + for (offset = 0; offset < mt_slots[mt]; offset++) { entry = mas_slot_locked(mas, slots, offset); type = mte_node_type(entry); node = mte_to_node(entry); @@ -5480,14 +5520,13 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) static void __rcu **mas_dead_walk(struct ma_state *mas, unsigned char offset) { - struct maple_node *node, *next; + struct maple_node *next; void __rcu **slots = NULL; next = mas_mn(mas); do { - mas->node = ma_enode_ptr(next); - node = mas_mn(mas); - slots = ma_slots(node, node->type); + mas->node = mt_mk_node(next, next->type); + slots = ma_slots(next, next->type); next = mas_slot_locked(mas, slots, offset); offset = 0; } while (!ma_is_leaf(next->type)); @@ -5551,11 +5590,14 @@ static inline void __rcu **mas_destroy_descend(struct ma_state *mas, node = mas_mn(mas); slots = ma_slots(node, mte_node_type(mas->node)); next = mas_slot_locked(mas, slots, 0); - if ((mte_dead_node(next))) + if ((mte_dead_node(next))) { + mte_to_node(next)->type = mte_node_type(next); next = mas_slot_locked(mas, slots, 1); + } mte_set_node_dead(mas->node); node->type = mte_node_type(mas->node); + mas_clear_meta(mas, node, node->type); node->piv_parent = prev; node->parent_slot = offset; offset = 0; @@ -5575,13 +5617,18 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, MA_STATE(mas, &mt, 0, 0); - if (mte_is_leaf(enode)) + mas.node = enode; + if (mte_is_leaf(enode)) { + node->type = mte_node_type(enode); goto free_leaf; + } + ma_flags &= ~MT_FLAGS_LOCK_MASK; mt_init_flags(&mt, ma_flags); mas_lock(&mas); - mas.node = start = enode; + mte_to_node(enode)->ma_flags = ma_flags; + start = enode; slots = mas_destroy_descend(&mas, start, 0); node = mas_mn(&mas); do { @@ -5589,7 +5636,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, unsigned char offset; struct maple_enode *parent, *tmp; - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); offset = node->parent_slot + 1; @@ -5613,7 +5661,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, } while (start != mas.node); node = mas_mn(&mas); - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); @@ -5623,6 +5672,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, free_leaf: if (free) mt_free_rcu(&node->rcu); + else + mas_clear_meta(&mas, node, node->type); } /* From patchwork Mon Jan 9 20:52:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B541DC54EBE for ; Mon, 9 Jan 2023 21:19:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=olrbZ7JlAuHc9F7lPAEq7whWMsWJggUO71dOO3TFzMs=; b=RAs9pEt/IW9XgdvIn0jZYEFsKM yQsAdG7Axs9U54B8fU1WnFrf7L51ex32bd1xAp1RP0HeSXHUS+5cojhEcxyTbtaa4g691ltnEvT/y KB8wZ+4j3qDhWPskG+DVV2xFZvoRrfUg98A44WHsOEwx4Z+MX/2GlEG/XJX8d/297pW8Uvbg0+SOb 59XDPN/V4CE9hxaQ1Uzim2ZpRBkxkjAHzHZDw+oQhNWpc+RvhGOSFfLiBDaXll61nyRbFMjWWMDpo sll36Xg99vBIrUEYuRmInvIBqbZ0gQmxPeeKFdyAf3Q8YthSvKpBlreswLwGTU7nq06oCRtxbH8wA uMZPsWKg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzX3-0049z7-AU; Mon, 09 Jan 2023 21:18:14 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzTV-0048ei-UV for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:14:33 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=xyMpIuQX3h8xEF8sgRu6qp18jXndG+tVBpDaAHqFUcE=; b=HTutA7Etk8glXUZnBiFVvKDQC9 podtjEiYFq5Orxh4YTMCjRijlwOBrmbFoo0TvIQz/OaqAfdtE8ct6SzKG92RoJgBC3cEz+eDRaz0g a1O2EanGpZixUqahSaZc++jOMq4XIyHqrjAUN7UcTKrgHe7QPtrEXfMAYFc+f5/Trjlw15PkZsQ+Z HU51JaGT92Ybw0b+zCfOvRka7HXUSbzyg9CdKg+HhaVu2TTGBWvGr5jY2o12HATq8uL7FLPOTCOOd wAmlK16f4kcT0Zfl0l44H6GKJUcCNuL8omRbzDkduSwAC+ddKzGhSZlVYeIkl0ZH/n9N0OEycUgJA jJf9iyBA==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pEz9c-002sq4-26 for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:04 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4ad7a1bd6f4so105751007b3.21 for ; Mon, 09 Jan 2023 12:53:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xyMpIuQX3h8xEF8sgRu6qp18jXndG+tVBpDaAHqFUcE=; b=fcpOTy1E+3ThlAgpg3hHomgF7jysjdpHFVliFHHbaTXw696G+d4UafDfpkipyR4nuU tlOJBreetd3v3dCXyb/Rdewxh5p1khH4cqHcd7IJy4L2zy/b/nqItrk1BUqrfoTrE/cO 7u4Zqnf/OqJvpx/Forf/3GSWw4rGhAxROXbd0Dn4cjHGhWxv2F6/FkGrRQlT4BSVn9QB Gq9sxTxT4dzmvld2wEbl8PkQKZUyemG56T9+2h4MZ3hLbr2uHjNUXIbifPrLJAMZf98P qyO5BkbAlxHe07UMVtRr4iqChj93uPwUR0TOsMr/XaxeajQHkPY5F6oCUcd/VOMyVzz5 1yLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xyMpIuQX3h8xEF8sgRu6qp18jXndG+tVBpDaAHqFUcE=; b=egYUEalL313gKe6GrOQ2kECenOp1mlvSU0ZuO/K39JX9gwRFQjV1G+tqKkIsB9mbNz XAsnVWui3w67XVoiqHIRTCQ6mMtDzDb/WqoLkFM9/sdmxUfYDSxbx0FRxBqxDTbMvueF hg34YHXT7cy5NI3oD03KJyfRHXOmocLwbJHaiBWxte40s/QrKtyAHqFEInLRAPyjRnef hMPS1BEHYT21Er77AfP24NG/aPiPf6t5pFs5A8fKwQ0yCAd8pFwEBrnJW9o3hI4bZNcr Z5LnX9XM+TlINZayHLuODbEyuJiT25W2HJ9xGK1Gc+zahbQjGK2PzygQXHzRw0D1fySi auLA== X-Gm-Message-State: AFqh2kp4gUtMEuRbd1CvrJMvuUrCyVs4lnQiFRlWW6y9367nfRZKYdQ/ DLMjBTf1vGxykeU9R62QP2aIL40bjJc= X-Google-Smtp-Source: AMrXdXs29F0iyIf9MRCODAQbmPmP2d9pU1gWPBdnTE2PRXcNw1d8zc+w15TjL10LubrEOMnWhSTePMV2FM0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a05:690c:299:b0:370:2d8c:8193 with SMTP id bf25-20020a05690c029900b003702d8c8193mr1225043ywb.221.1673297630870; Mon, 09 Jan 2023 12:53:50 -0800 (PST) Date: Mon, 9 Jan 2023 12:52:59 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-5-surenb@google.com> Subject: [PATCH 04/41] maple_tree: remove extra smp_wmb() from mas_dead_leaves() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205400_964093_B3E7CB2E X-CRM114-Status: GOOD ( 10.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Liam Howlett The call to mte_set_dead_node() before the smp_wmb() already calls smp_wmb() so this is not needed. This is an optimization for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index a11eea943f8d..d85291b19f86 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5510,7 +5510,6 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, break; mte_set_node_dead(entry); - smp_wmb(); /* Needed for RCU */ node->type = type; rcu_assign_pointer(slots[offset], node); } From patchwork Mon Jan 9 20:53:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 909CAC5479D for ; Mon, 9 Jan 2023 21:19:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=nsZcMRHPXR4+CZIG1VzbmfzweO1/arqeKMkUeK0pHJM=; b=1DwqHZY5VEtXVy79S3YgLiiv/O EBr2h8bxN+lOk3JGdphb3tTn9ZxFUaAVILzSTaA/d4FBO3HUmNqop3TdrGP1u1WnQZofF0xxIm7On RFBhOZqIew8RzUEezA5hGXn2/QCeVgiYofga1Nm6bEpiBmZZfplMhmZAP+HIT3iOj4ZPl2+bd+MUL bpfeoyTg6k5PTjZ3hjVjFmBPedOl/mX7M9w3254prWsR3Nv+OhNlh2d92qHPZ35QuPGligdre3Us+ 2lk+L4q7hU1FgqR0NwBaVsuSd2o4fPJcuFmyeZZ/7nQSiuvhqTUw4UlW4CinbIOK59BbOR3uWzNnq 5VHGO7sg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzWW-0049nC-D0; Mon, 09 Jan 2023 21:17:40 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzTV-0048ei-4a for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:14:33 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=r/ByKXL5/TL05Cni5RZ8/3QBk5f4BKrzjgLm3Nsczuc=; b=fbjrNjIZj7CIUq+MmJu56IIYe8 uPRDkclbanbvtoKuHlI0XwH/6FIC6mL0ipI2GpJQ1HHld+RI/Fe7NIGmnaJoochPBf5KMOzy2SvjN jeUvqO55YBGpe3oP7eup4R+cMGwV/tvM3OYPWef7Uh4CaRUiFWF7O79FrENzQAnyZ6xMn8bKWznSM 2yteU2OwAnfsD+sAxhUeGQZMBSP5LisruB2qnM80MJ60jjfJ/d4u7ajHSLQel2Xdhf8tSmNK0tgDL vaqodPQkLatTwOqGJxmwvQSD5+vvF2uivW6G7fc1p3NG7VB8qZ45tHOBzkEe4vNoxiH5nxFsGb+N5 ske1DLzQ==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pEz9c-002sqJ-1d for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:04 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4c2d68b6969so94620177b3.7 for ; Mon, 09 Jan 2023 12:53:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=r/ByKXL5/TL05Cni5RZ8/3QBk5f4BKrzjgLm3Nsczuc=; b=XZhB0MurOIdgvWrQwu+hJ2lSxOQT+CHAxznZAlATY9tL6yBOIdJSyl09j7qs++7K6+ +noh5D5/c9QmPEOhqfvJHyQwq3kACUyssJheSfMwcYViJIFHS1/y1fiRpO9p3lukpSo8 zurDxJt/kmbPIHQRSNmtoT5iYyxem6gKfY8wpBL8JKkSoFXDKg0JciA3ABN5+4Pa/8FZ Svy0a58Amowv53Vk76UKApMAQ9ZprfLfSHlZL0fWzl8fmhDrZgAmkrOVH9/RFI/nNlnR WsAxkAJv9sUdL+8FTIMmDBza+16v8vXL8ENtWQu+olGHSLyyl/6A++OsnTnGdWDPMYeD K8RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=r/ByKXL5/TL05Cni5RZ8/3QBk5f4BKrzjgLm3Nsczuc=; b=BaC25w0L0UnTa54WBj/CHjHTOjeXfgTHhBTePLtGVOQxW4rrxBryJrWnHTRMSQOwh8 vJPXByAt0U30hPqI5dgQe+tGQNg43xPFAsKOYtpaNy24d/oeoZ+r5rAuENOe8qOYXSA9 YJrP3CZ8bC3xGUUAvS0QLGMeQbSCr29h0eWgAc8ndj5JdcSHYboTCdzxcWaooV+659sH 6QheO5brocdD0OuzO7oaP0vPnnQpyh8LeK9hUehnGgjJbog4XOpkZd3jTqmcL+TRKdE1 NfvplUulpsB7VWVGFdHEoScZSZd3uQJX03T94eRSeHQqXb/Goj/uz3ZiOp95hoRJ2DyO 7EBQ== X-Gm-Message-State: AFqh2kram4ZN8wLMxbUo8bLIwAm3B8DlxGsm1AqdW1y1ZmTCprZegZcP Sc+ly/8KQaMYP9dIcMr/UZ2IqgCu5e4= X-Google-Smtp-Source: AMrXdXv6pxAJ0Ez+UVi7qfPaeqrQmlVnO2LvHLtbQu4X0J/l7RkUZy3CJtaYn7PIXuCjX2fZCm6e/eZrYK0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a0d:f007:0:b0:388:941:23a8 with SMTP id z7-20020a0df007000000b00388094123a8mr1505536ywe.152.1673297633444; Mon, 09 Jan 2023 12:53:53 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:00 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-6-surenb@google.com> Subject: [PATCH 05/41] maple_tree: Fix write memory barrier of nodes once dead for RCU mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205400_944897_9C7E3A5A X-CRM114-Status: GOOD ( 16.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: "Liam R. Howlett" During the development of the maple tree, the strategy of freeing multiple nodes changed and, in the process, the pivots were reused to store pointers to dead nodes. To ensure the readers see accurate pivots, the writers need to mark the nodes as dead and call smp_wmb() to ensure any readers can identify the node as dead before using the pivot values. There were two places where the old method of marking the node as dead without smp_wmb() were being used, which resulted in RCU readers seeing the wrong pivot value before seeing the node was dead. Fix this race condition by using mte_set_node_dead() which has the smp_wmb() call to ensure the race is closed. Add a WARN_ON() to the ma_free_rcu() call to ensure all nodes being freed are marked as dead to ensure there are no other call paths besides the two updated paths. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 7 +++++-- tools/testing/radix-tree/maple.c | 16 ++++++++++++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index d85291b19f86..8066fb1e8ec9 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -179,7 +179,7 @@ static void mt_free_rcu(struct rcu_head *head) */ static void ma_free_rcu(struct maple_node *node) { - node->parent = ma_parent_ptr(node); + WARN_ON(node->parent != ma_parent_ptr(node)); call_rcu(&node->rcu, mt_free_rcu); } @@ -1775,8 +1775,10 @@ static inline void mas_replace(struct ma_state *mas, bool advanced) rcu_assign_pointer(slots[offset], mas->node); } - if (!advanced) + if (!advanced) { + mte_set_node_dead(old_enode); mas_free(mas, old_enode); + } } /* @@ -4217,6 +4219,7 @@ static inline bool mas_wr_node_store(struct ma_wr_state *wr_mas) done: mas_leaf_set_meta(mas, newnode, dst_pivots, maple_leaf_64, new_end); if (in_rcu) { + mte_set_node_dead(mas->node); mas->node = mt_mk_node(newnode, wr_mas->type); mas_replace(mas, false); } else { diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c index 81fa7ec2e66a..2539ad6c4777 100644 --- a/tools/testing/radix-tree/maple.c +++ b/tools/testing/radix-tree/maple.c @@ -108,6 +108,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mn->slot[1] != NULL); MT_BUG_ON(mt, mas_allocated(&mas) != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas.node = MAS_START; mas_nomem(&mas, GFP_KERNEL); @@ -160,6 +161,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i); MT_BUG_ON(mt, !mn); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -192,6 +194,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != i - 1); MT_BUG_ON(mt, !mn); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -210,6 +213,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != j - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -233,6 +237,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i - j); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1); } @@ -269,6 +274,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -294,6 +300,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas2); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas2) != 0); @@ -334,10 +341,12 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); for (i = 1; i <= MAPLE_ALLOC_SLOTS + 1; i++) { mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -375,6 +384,7 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); @@ -382,10 +392,13 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); } @@ -35369,6 +35382,7 @@ static noinline void check_prealloc(struct maple_tree *mt) MT_BUG_ON(mt, allocated != 1 + height * 3); mn = mas_pop_node(&mas); MT_BUG_ON(mt, mas_allocated(&mas) != allocated - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0); mas_destroy(&mas); @@ -35386,6 +35400,7 @@ static noinline void check_prealloc(struct maple_tree *mt) mas_destroy(&mas); allocated = mas_allocated(&mas); MT_BUG_ON(mt, allocated != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0); @@ -35756,6 +35771,7 @@ void farmer_tests(void) tree.ma_root = mt_mk_node(node, maple_leaf_64); mt_dump(&tree); + node->parent = ma_parent_ptr(node); ma_free_rcu(node); /* Check things that will make lockdep angry */ From patchwork Mon Jan 9 20:53:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E32D4C54EBD for ; Mon, 9 Jan 2023 21:16:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=fKXpuiPV9ogXjhah4ndKGq2UvnwS7E+Gt/SNAEpWsQ0=; b=gp29u2JgGgouolxQmdb5Wl69HO boXugG+k2uC3r5LUeghQ9OH8rdGSx9AqM5bfXGQGepbbOMa2m5pAxxAGHCT8VoJHfTpD74ea5Nytk Z23waFr1KPzYemLD2NcKCfbq4nk+cIAdrgjwjjvDIP2X5GMLl3gS+pOKHZIDKdUCKCDCPgALPInmE AibUn09lpbthoeiwf8LOV6q2FjngiUWLvArihp3AhxZewWKKLaS2o9FnyiXuhVXZC0wr13Jx7nihW nZI++kT6sjhrcCNZrSLF3Ic1beiHFS3FsJkWN9cRelyB6lxQSIpbnRLNrpd/8bYrxY1Eck2iFUp14 8w3Pl+vg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzTc-0048hc-Ga; Mon, 09 Jan 2023 21:14:40 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzSj-0048O0-76 for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:13:45 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=tpY24U65dxDevik5396IsQMXmVZXiLzmW2j6b7ptC2o=; b=KZPmVtvk6IzAe9kGOCjOH9znry qcFmUu9cVb+bcXqQDM8yEerKoMDxhckxveNDfdBjuIVu5rxRVeH+otPchDyFHpkn95fYhEO7oybsf tchCWlA+sDrjxCSu1fBk4oMe1NTzpEwbW9gdi6vu/zFr62NV5mHoBF3BHfVufPYHz2O/n0wyJTW+T ZQPP9bTeWbGohV0db2E9IFJk0g0/gDblklGzlINj+P7INslsn+xuoyt4HfSmREXmecTZmGmylqYyw IsNCxW5ulU49N9MR0vYRjHoHhzVSLKdXQ/gD7gma2PijMGGu83FnigtD2k+M9DcZYnsQyTx3e3VC5 piYI676A==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pEz9g-002sqL-3C for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:21 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id l194-20020a2525cb000000b007b411fbdc13so10388724ybl.23 for ; Mon, 09 Jan 2023 12:53:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tpY24U65dxDevik5396IsQMXmVZXiLzmW2j6b7ptC2o=; b=FGVUXzbMKsAgHtVe2H+ZGPByy9vTxmIYvrPV7ZuF76ZDV4PCOupJdRY2uHLiI7i+8R 4Fjm3rnOC/ry4DX8g5MN/0YQ4aHPdCQZYFwG7UcSZ/43mvYNjuR6DmjlzCEnl8acF7KD ERVaFO4iv0zgtEZ4MmYzhpcYCiFLTkatnznfDTOltPoYyw5CvEIb1yptgTPwDbjWQDix pGxXZ1tFpWIKQUAKj6j+gMffNX2JeuA3u7s8cWrwnHHzy9T6apLvHv6bApzCw3vZvb6B a2xiK595oO97JhT8ZQYkTmP/j11qoIyiGxI24X5jHCYtpJjinnDlcXs5UPFQKhdZ/Kdz DK0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tpY24U65dxDevik5396IsQMXmVZXiLzmW2j6b7ptC2o=; b=kwdekhXsliBVFmjkW/BHsztw2e0bKd0hltCBZDY5d8oFI16Lp5MKhQ9bdQCj3UNBKK UOJ661TbC79vt7JAMgk+ACXnuAnTc1uQud2z/glHpwcYrqmpPQLHU9NR3ez3BI6WICDz GsVlI+rnflS1fttZZvvz3kVoYFAy4M/FZ+GlaaGBdz13nj4+1qapHWvU3UwKy+EyJo/z w5VQpVBQF2qMywImmNePps0jrV7VKL+Gewa5dG9s3O2CdKuGFRuiQMl+avSJLuMlqpPa cFu1twTdEJa8mYgcDTMX6mXEzORpwxQ9w1dPEYkIuZGCfrCNdejZLP558ANOHvwQMbm6 cWNA== X-Gm-Message-State: AFqh2kpjeenJnUYlVa55ZNQt/QijlpoKTXHp2QC2UbxZjwsFU3nw/dfc iOPCt1rMkNGRYRSNEa6Fr7ruq97SB/4= X-Google-Smtp-Source: AMrXdXs5jnkdpf0WLGjSqHJja91Z23ecX2Os+zrAvpCFIdrfHWytTqhc2UPXUEbTUa59RZZK2SYuIAkg8gM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:2491:0:b0:7ba:cd98:5bed with SMTP id k139-20020a252491000000b007bacd985bedmr646916ybk.69.1673297636065; Mon, 09 Jan 2023 12:53:56 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:01 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-7-surenb@google.com> Subject: [PATCH 06/41] maple_tree: Add smp_rmb() to dead node detection From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205405_388754_A533AA87 X-CRM114-Status: GOOD ( 12.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: "Liam R. Howlett" Add an smp_rmb() before reading the parent pointer to ensure that anything read from the node prior to the parent pointer hasn't been reordered ahead of this check. The is necessary for RCU mode. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 8066fb1e8ec9..80ca28b656d3 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -535,9 +535,11 @@ static inline struct maple_node *mte_parent(const struct maple_enode *enode) */ static inline bool ma_dead_node(const struct maple_node *node) { - struct maple_node *parent = (void *)((unsigned long) - node->parent & ~MAPLE_NODE_MASK); + struct maple_node *parent; + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); + parent = (void *)((unsigned long) node->parent & ~MAPLE_NODE_MASK); return (parent == node); } @@ -552,6 +554,8 @@ static inline bool mte_dead_node(const struct maple_enode *enode) struct maple_node *parent, *node; node = mte_to_node(enode); + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); parent = mte_parent(enode); return (parent == node); } From patchwork Mon Jan 9 20:53:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2EC4BC54EBD for ; Mon, 9 Jan 2023 21:21:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=xj1+/BO0lyB4MDXdXoOOub8vp9sWQKBU3DcnVAVea4U=; b=mDT+NVFBs6a+jPdViSAHKNYYn7 wojlbX43zan7PGOnQZ0E8jJ+VGEmj4Z9xId1sUY928GLBQ4EvsUoKDwlUR9U1r3bj+Mpa0cbs13nn /aumtxjoN2o6m6pM6bvOvP/l+zcDKKgLj51fAboDBiXogMaL8c3UijIErPBpN2Hd01vwn4cVCsIou UTeKW06CMhJsylu+WrgtY1IE9GtpM7sLM6eOgdoW+fJDzmxr/wsHY/OnKEg88u74LHhScdMBF7PkH ASpuSddaSi4Svl3bw7yovEv1uO0QH2HiWzCKggmK8OC88tOjFBA6fA7q2WsyMUoQbuhisteKgGWfL cUYW8r1A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzYk-004Add-Qx; Mon, 09 Jan 2023 21:19:59 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzTs-0048nw-VA for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:14:57 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=TT+2XsH00KwTX585kTrU5AriHJozMAXcs0iFxeAd0aM=; b=gcnFwE01k+1aIXt/8N5Aqr+aqi ge/XD2xd1tsGlKL/fSn6hAuqqm6zkMX31OuqO+PjMW5VyOpqVIvJSlJeBvP+M/ZZOXuAAUcUXwukA IZBdvDfAuxpAUivYiqic8CxYGjlSdYJAmER79Ptl2UHKJTCth7uxTEcbJsQejG78WmFcVHOW8Tm23 jXuDF8FoxxsHDruzg9deLHaw+AU6P/teMnUeDE1USVKAuR83GNnRE3cnushQ422oLibk9LF5WkXZf 0AT9MxcPT17VYUUXCUElB5mOgipPQILbgMLpMfBIO5+8hkfeDsTkQOLENYIayexvIzCIfEcPq35sP +gE7bEdw==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pEz9h-002sqW-20 for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:21 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id n203-20020a2572d4000000b0078f09db9888so10252638ybc.18 for ; Mon, 09 Jan 2023 12:53:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TT+2XsH00KwTX585kTrU5AriHJozMAXcs0iFxeAd0aM=; b=Czdnd0b+5bkyGxVk+9aViJOW1TJ5m6k5XpyXdj4R+Q3dGGEBK2eP5deQvySgSh0iz/ GZ1iYooFrZ5CipGLrqMEQsrY/Zn4kUyQA8m4xIFN7mfEerPKCDkU1AubZLdIfDfth4hu d8dG+6QDcTUs7A9Y052NxeL9TA5a3QdfxlEt4CBoSlbUsLAUBEORkFyS1sdo7qsKHieD EXHjCZRBRMhYEWoMQiXcySJyrWxVf33iALFZt6xsCbFcNo15dZaOnJt+J48Z17ETAj7O 45DKwd6DY6kCK0yPohqOb3E3TN0P5VYKNR4qSRkuKInUqGltfhkGbi1MmOGm2N8pUB+H +TEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TT+2XsH00KwTX585kTrU5AriHJozMAXcs0iFxeAd0aM=; b=WPbRGiinx5Z/10h4tYwAGrFXxlfu+9YEcyy3GSdq9ChlZ0HPpwYewqD3ovzxmIRWwM dUHm+nSRGsc6EgM6fT7HrmT627avbD+tLe1AdFLaZQAbZOOqmAnbDRiALffhW/NYGwjl NDtaXGhacO0/x8/k2/j3/aooGC9evkTkr332ocRrTjDGBO8TdpC232DsSlzAfhs6lNV6 L+C8Sq31j/DRnQHFlFxXgkoRuKnj72UozhsHxa5L87I4UGYQub1t1C3V9xfBUViGKj3Q Eboe/JQBAGYhIUAtTkVn50fbZdS5bEpEl8/a+kxczfUUhq77OG5uKyryF5lsgMHH5mwU n58A== X-Gm-Message-State: AFqh2kpo8nOcn0xibwoumhcggEeBGaxj7tRDRujstG7mQh1XrUo2VYJL 8yTebJm82PZUQm8q4BXvLkCr1JY2zbg= X-Google-Smtp-Source: AMrXdXtUN3SiDpbjRFPvhoNJBXB5uM1KkpI6Zcz0Naqt9m0yZDl1B9u4vz5wIeviZulbrZU/bcJYjep5MGE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:abea:0:b0:762:b86:e82e with SMTP id v97-20020a25abea000000b007620b86e82emr7288109ybi.407.1673297638462; Mon, 09 Jan 2023 12:53:58 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:02 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-8-surenb@google.com> Subject: [PATCH 07/41] mm: Enable maple tree RCU mode by default. From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205405_925060_09E3BB5B X-CRM114-Status: GOOD ( 14.68 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: "Liam R. Howlett" Use the maple tree in RCU mode for VMA tracking. This is necessary for the use of per-VMA locking. RCU mode is enabled by default but disabled when exiting an mm and for the new tree during a fork. Also enable RCU for the tree used in munmap operations to ensure the nodes remain valid for readers. Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 3 ++- kernel/fork.c | 3 +++ mm/mmap.c | 4 +++- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 3b8475007734..4b6bce73fbb4 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -810,7 +810,8 @@ struct mm_struct { unsigned long cpu_bitmap[]; }; -#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN) +#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN | \ + MT_FLAGS_USE_RCU) extern struct mm_struct init_mm; /* Pointer magic because the dynamic array size confuses some compilers. */ diff --git a/kernel/fork.c b/kernel/fork.c index 9f7fe3541897..58aab6c889a4 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -617,6 +617,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, if (retval) goto out; + mt_clear_in_rcu(mas.tree); mas_for_each(&old_mas, mpnt, ULONG_MAX) { struct file *file; @@ -703,6 +704,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, retval = arch_dup_mmap(oldmm, mm); loop_out: mas_destroy(&mas); + if (!retval) + mt_set_in_rcu(mas.tree); out: mmap_write_unlock(mm); flush_tlb_mm(oldmm); diff --git a/mm/mmap.c b/mm/mmap.c index 87d929316d57..9db37adfc00a 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2304,7 +2304,8 @@ do_mas_align_munmap(struct ma_state *mas, struct vm_area_struct *vma, int count = 0; int error = -ENOMEM; MA_STATE(mas_detach, &mt_detach, 0, 0); - mt_init_flags(&mt_detach, MT_FLAGS_LOCK_EXTERN); + mt_init_flags(&mt_detach, mas->tree->ma_flags & + (MT_FLAGS_LOCK_MASK | MT_FLAGS_USE_RCU)); mt_set_external_lock(&mt_detach, &mm->mmap_lock); if (mas_preallocate(mas, vma, GFP_KERNEL)) @@ -3091,6 +3092,7 @@ void exit_mmap(struct mm_struct *mm) */ set_bit(MMF_OOM_SKIP, &mm->flags); mmap_write_lock(mm); + mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); tlb_finish_mmu(&tlb); From patchwork Mon Jan 9 20:53:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6DB71C54EBD for ; Mon, 9 Jan 2023 21:16:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=PuZDowQv3KpwJMNfHTElUl0UcwcObLWGHT3YF38nS9M=; b=DVfp6LBpFFHFB4rXkVM/wKxo/i +LKNIIEDO9Fmfml6YAPj+Aorc5C7wpgkMi6LPSNeiW/O/n4400ngeSlxjrP+IR/Idsh8HxLL/Z5D+ W/pxsiGVyczxOaaWWjz4gLkbSbN9zKfHF1mipL0Bwe+roExl6YMf96B5a2ZbfOSZYia6pLzWYwjoN yIut/nT/aSGtW7ezoQZ2dgvb0Whd17TltGT6nDySoq77FSGH1XO8eyHSC16PL3ldRNu6kSpnkTeFD QGPtS25A6ohohmYhvEykZ1gDg9BRwsEuaYOmmG08fun7zKiO5IMfMvaUiRDpj28Jw6fTVOSDpfyGQ N/Gpvy3w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzUR-00492n-SH; Mon, 09 Jan 2023 21:15:32 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzSo-0048PJ-8Z for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:13:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=bU2TYXI/IQ+ahWPrvtMjvIvPP35LcWsiAJFQ60YWV8o=; b=UL9cjJho/KkVji9a8Is4ewXMib Pr0GEHuBg6sHbG7uNWMFRp5mxhhuZymWBOJkgS2NmpK1BF4bX+WZqgfldGOHLSaPksoYSL63zBWOx tXRhTLWaS7aD+syd6UtuFJr4mgIJD49ESuCpGXIuQ3jFMBXUGBTLmdcpUt12lif89IrbxXEZTHwVI GnSyw2uriXrZgV824xGOLORvwL4Ckh4tf5eGSQa4oV2bVH3ApdH7UwUB78vhaMYqLeOldjPaysnNE oAF+W2BZMUHfNswEn8NKnWbHTbEs4+VNimL4VYI3xXyT4tvdYCqBhOWaw0sse8aeP+NejGxb5EqbQ 5WOP6ZTA==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pEz9d-002sqb-0G for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:06 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4755eb8a57bso104725297b3.12 for ; Mon, 09 Jan 2023 12:54:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bU2TYXI/IQ+ahWPrvtMjvIvPP35LcWsiAJFQ60YWV8o=; b=hJQlgWQOwjt9bdNS4iNCFQHGlLFypCJo9NibsA7lH+Cj66pCbZCU+CxVh9F7A2fm7T vmlH3bAYg1kSOc6vTNCehxfbrsiSwMkUwfklPcXaZBTBOgsQNuQ1lURfneHqEpASj7ak 72wsJeHj+71ppFuT3bry8xcf97dMVhsy87IzDxduellVAKg7BiJc4vIK8HA7ompTnbjq +gURuiw11LvdH0ad+L6DkctzinvpOQbzYQIPe9DQz0Et4Siymb/UJSUp4isr8x/TUfSb z+/96aen7RzYU/fA94YBNLJOwTLnQ/SBQCaJx176+XVEeqKe1Aaxg+5hL1pkiSz0xMHc S9Jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bU2TYXI/IQ+ahWPrvtMjvIvPP35LcWsiAJFQ60YWV8o=; b=u41ARgMuTK2oeaEykHgs6m5jiNBSrQqZexZfkF6Ugksc0QVLJEBneat3GLfkbJ5NFR gPJcGL0WwxUKQbaO02CRE3BM4igiS+swPpanrAarZVWNH18Kd99MyRvK9eL20GD+Wg7x py+0PMT0Mi3soT785WyVw3icY8kAG4QMy2NM4MyPzxWeu5rZTRFYjS2oxGZyvwjA7kPR IXrcy2WR4Pl1rP4ehUZUWsUM8Q6/9fITiwlyrjNEOuwj9CzHBBV1H8MXx1QOWZZwg1J6 xFQO4QqdODZa/qgSIOmDCnJ90pj4n+OIe9WJg5GD5lU2+wh/79tL6rxHQJDnA/41joWw 94Jg== X-Gm-Message-State: AFqh2kqULnxp65mIVTK/pwCwmj9hOjpfwEkkyHjsxV8VmMjSptTTgI/k wJTv2D4kSaGAFmA+Kp9TryB2j2RT2lA= X-Google-Smtp-Source: AMrXdXt3ngSzeTd4BjcWtYkKZUmKlJ+coRJjsu8zn30hseAAXL6r6J79qyIJjR/uVl5OkeApgAgECG74aLI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:9f8b:0:b0:705:cde7:2363 with SMTP id u11-20020a259f8b000000b00705cde72363mr8137319ybq.81.1673297640913; Mon, 09 Jan 2023 12:54:00 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:03 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-9-surenb@google.com> Subject: [PATCH 08/41] mm: introduce CONFIG_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205401_344402_9611D7D9 X-CRM114-Status: GOOD ( 11.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This configuration variable will be used to build the support for VMA locking during page fault handling. This is enabled by default on supported architectures with SMP and MMU set. The architecture support is needed since the page fault handler is called from the architecture's page faulting code which needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index ff7b209dec05..0aeca3794972 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1183,6 +1183,19 @@ config LRU_GEN_STATS This option has a per-memcg and per-node memory overhead. # } +config ARCH_SUPPORTS_PER_VMA_LOCK + def_bool n + +config PER_VMA_LOCK + bool "Per-vma locking support" + default y + depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP + help + Allow per-vma locking during page fault handling. + + This feature allows locking each virtual memory area separately when + handling page faults instead of taking mmap_lock. + source "mm/damon/Kconfig" endmenu From patchwork Mon Jan 9 20:53:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094386 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2E02BC5479D for ; Mon, 9 Jan 2023 21:36:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=/bX1erbKfOpws8woHM0edM5OO2aXCWSfdHcXlLMEwvQ=; b=BnLbtTVro+AA5czU7gClLNoMhc Qa88mfrSo8P5uO6vkVFKmJLL9MnvugAdBTgoXwjT+Tyvbo7pTJxcUPTASK+34uwgJPA8ZgVVhGRji m+wrORkN6SBNHWLDCdG7R8C68KbME2NMlJ/p8L5xoRyWQBu/2ICHctTmorCELScScAXqong3g4/CS dqQZm7ZhhsQadGcySfrJY2Inxj2l8U8/mINeib2RHtl+8DVqIkcLCBSsga67+YVAGF1mzJN8vFzpM vSGk1hKNKgd+YX5HU8RllVS893L4xw1Iam7r7Ilyby5FXeRWXmES1dw+3zpXhe3f3HYE0M9OFypGW QhctLgAg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzna-004G34-VA; Mon, 09 Jan 2023 21:35:19 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzUA-0048n5-Jh for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:15:14 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=D1pZqRdf8jzuIQZRLrMCJmLqIEvbyFYNYfXzudRZz6Y=; b=SKph7SF4BcVdvT+6chgqtCSbfm uPGeCCIElnGuX3me1TjroUTIEXzBcKpevzN3D5YZ5S0LnF3cK+dlsIhB6J5tr7m6Z7cD9pPXpBcKU +Ql6EmJGvu4St+wljk2Kwfz9H55B7UJ3mAGBsytP3AJzOJBy7lPh+rYOnGgjGOD324gYRnytbHxD2 dKkI9O1KHJycfLnKEM5xA67k110lclVoj1a4PzPicwpWFf8WapjC8ayP5EKQ87SkS//FRHYXS2Gx2 /zDq72pS5JiNxNnNxoy59liy6UvV2g/lJ4Q8m5t60ATk2w2CYGXZsqcPI8N8l6BqXCAKdrW0O5d7f aHM9n+EA==; Received: from mail-oi1-x24a.google.com ([2607:f8b0:4864:20::24a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzA2-002cL2-76 for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:31 +0000 Received: by mail-oi1-x24a.google.com with SMTP id 4-20020aca0b04000000b0035e9d1b2188so3058951oil.17 for ; Mon, 09 Jan 2023 12:54:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=D1pZqRdf8jzuIQZRLrMCJmLqIEvbyFYNYfXzudRZz6Y=; b=plHU/zWrfFxaQ0yt/JPDRB18eEsMPdd+ZJ4E4yT8T5NRkdKBZYZu2kA11sksr3NXzt jDgkb12AloyToG1qepTp/9j3c1S/nL9/sMLfiUGj3EbCeKSxNDyG2Qbnp2R8KUfjKQNu 1hUjiCYEdhUY2xqldYVyznhCdRudSDrsopmij1IaAdQfZY5giCy+PasuJa7G7ZNci7e/ L6oXIEMKihljCOjvvq+Mko/UGqlkYlK33Q2p9mFgU8soW7h5CIdjJbM9Bn0dub4CcI+s 5hJtucAeeWPqjUkI2FPiSFxFnJ3u2uppPiS920i95ViALBCSjuucJs6UET/oQp6p8GLF Q6Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=D1pZqRdf8jzuIQZRLrMCJmLqIEvbyFYNYfXzudRZz6Y=; b=WtYAvlUPVoQHZ4KCqbjbcKlUqMqI938rG8brBmtBuLQZ8C7PMHdk9WYhQ3CQQg7Hb5 VezcrnyVE2/sln7D644EplJ/HR7YhQZq5ai1l/aJ1sFVt5q/m9Q3E/mMX9FwXNLryo5a p6jw53ZOh/P8YxZFKX7tGGlKraCpJb5/6EQe5M6K/m1nk2f+eI5F7Xo7MNdQ5YlpZlex 1jesE7QGDUgnjj6Rv6pAl6OS6GtboraIUhFKT7iBG2COeRdsXdqcDMzLTNBMP4taPZc/ ebOHAVjTTTSf18gSZw6JEI5XzU/DElJv+1XR+PN+JAewLTJfVB/kfn++0qRNIIu6vXJN 1F/A== X-Gm-Message-State: AFqh2kq4NQ9pZRpwzhAFXpPwHd+LzYajqoHkfyQ0GiGn5Y4pZJm0qF32 ix6mQMOzlD4MWl1+k9BLMhiOO2cp0Ws= X-Google-Smtp-Source: AMrXdXv95OTi9We7BklH2FBX3pkox2UVBRMtva8Mm7ne7fTGkQBBnTiOs3huBoDZBy+tfDGkNugi4GYzEh4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a05:6870:649e:b0:143:509f:17f0 with SMTP id cz30-20020a056870649e00b00143509f17f0mr4940900oab.211.1673297643629; Mon, 09 Jan 2023 12:54:03 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:04 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-10-surenb@google.com> Subject: [PATCH 09/41] mm: rcu safe VMA freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205426_284097_AD630167 X-CRM114-Status: GOOD ( 10.54 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 13 ++++++++++--- kernel/fork.c | 13 +++++++++++++ 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 4b6bce73fbb4..d5cdec1314fe 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -535,9 +535,16 @@ struct anon_vma_name { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start; + unsigned long vm_end; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct mm_struct *vm_mm; /* The address space we belong to. */ diff --git a/kernel/fork.c b/kernel/fork.c index 58aab6c889a4..5986817f393c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -479,10 +479,23 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } +#ifdef CONFIG_PER_VMA_LOCK +static void __vm_area_free(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + kmem_cache_free(vm_area_cachep, vma); +} +#endif + void vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); +#ifdef CONFIG_PER_VMA_LOCK + call_rcu(&vma->vm_rcu, __vm_area_free); +#else kmem_cache_free(vm_area_cachep, vma); +#endif } static void account_kernel_stack(struct task_struct *tsk, int account) From patchwork Mon Jan 9 20:53:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094388 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C35B0C5479D for ; Mon, 9 Jan 2023 21:37:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=oDzGE7XmKvfLfZIHDxkpQkEaXiAERmYYXuQzMBlx//g=; b=23a3KgY/um74A6Ax8p38yGYhuQ LZs4pLEFJIkqESQ4cIhC60epW8ueERl0ew8pSL2vgPTQBqtpuRIzZ2WEH/mww12HpFXMSHSU4adMW YtnNKoSw7FpwbRlZr81QLHccmdtwXoaqciq2wnhxPlBpJgoDxYRPj/n6rMF0ZTkAMlLVI3BAp6eCA sIwSryYsRBKsrHfei825P8n9k+5eq1YP0uiZop1yx9+6+K6uMtYEqv1qZ0u2/9w1qKeBN3i9lvYc4 rDpJYAx2xdVmEntgZEdjgJY6DX3ZTwH+o4g5kkRSmUzNJoPIWm5GBMzXRI4EoQ/kjYf/QuJR9acNi TMv8XWIA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzol-004GQu-B1; Mon, 09 Jan 2023 21:36:31 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzUF-0048n5-On for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:15:21 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=S7Z9LsPdf1X6x51hAUi2Nm1VlEONIlFgbEepvCuHuj0=; b=C931wFf1HndJ8uUFSKcIIW0lcZ peUo3c06x2Cq8eseSj+RzX9BWkL0yUpTDjn2w5SFpVW/rz6gl9vJI+zIp+tjEbNsAif9fGjJg9Cf7 9aPOmEFjFAWy0k6orBfNe0ZwhQZNCZezQYuG/l9+ZnETWA+Wto/PW6qs9/fdwSnko9ZxhlZMTSEWo q8bULRfzPCvtSDsntJeuI/PlI1Y1CaCslhXRMi3Ba+BK89//zOdgC3xrTH2I4I37jG+Io8CTTIYKj rZCCfr80+I1rLjRcbgEA6y0wgyvtLDEywXxOpEQqCFIwRtNG1i521qZOLKMfn6h/FRfMhPa1y1RTj 7d9F1qJQ==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEz9w-002cKC-IJ for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:25 +0000 Received: by mail-yb1-xb49.google.com with SMTP id y66-20020a25c845000000b00733b5049b6fso10231615ybf.3 for ; Mon, 09 Jan 2023 12:54:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=S7Z9LsPdf1X6x51hAUi2Nm1VlEONIlFgbEepvCuHuj0=; b=lU92C8Jcfmul8qFoqZuzCOb83nRDo2DOHABHtbUeff7KpjabsQdodVMIjgjNegm4Ui gxUyuAPRcETsiRyXJ5mliOj6g1oYM6B1qnPvTAT1LA3dK82Ds4hQfkrM9RtYlbde0cbx i9uLWTSQCuYKj1fxgTqem2OIhsK37AMmmELynCklGF8pD//eviNoFDjhffy4KqRRdQJC MSzyVPeVUGI3nBlo70SJ15d8SfW3UoSbvzQXr0uydmfUdXg4Fl9jpjR3RSUiLlvME5Kv Z62tI5fReQIg+DMNczYR66QAZbTIG8pGdwhVnCRBmPZaWsdfRkDIb3vX4UMeFSba2+4A TcUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=S7Z9LsPdf1X6x51hAUi2Nm1VlEONIlFgbEepvCuHuj0=; b=1MsWPIWDOHNZyRJE0bomrvHK4rYCuSnL2hoqk1hzf93F/uAD/n24ZGpM31G0o+avjN 5sYSex6hQ06eivXgjih5lVDZIfxLqPElKpGicNG+lGT7NPH1Hqpsf0cB+L07J/AXLVos 3+1yWpBEoYIzhloZz3+c4lr/83h0az4qiJLXKjnEkuCo6JU8EPU0mY4hTg62PNMzAeRm Ep5Mb48g2mBOWROHSD0daaX0jNKNdvu4ziyIsw++8PHbgFPwnaf13Gt6+QJ2U3s3TjdT OPojWMw61/SQzRHE+RKi7cWZipwJkUG0phhwSy3DZP1S/lwg1r7FoWBlpL8+ulhSDUUq bszA== X-Gm-Message-State: AFqh2kruyMx1EbQoBGDEkgMNyAciVZ2QTawvZ8i6w2gnHJBvyVnDZH4O GLY8jv0KtSSrsiAFfaKxRbLjXtPfkrY= X-Google-Smtp-Source: AMrXdXt5VtatAvHvgu6DDkpLTQLv6tox4QHYgsFVwcGGt6mUkf2zF94ad2ULjvwhvY9gJJrtYWAmbQIwTHs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a81:4fd0:0:b0:468:5f78:265a with SMTP id d199-20020a814fd0000000b004685f78265amr1632391ywb.87.1673297645909; Mon, 09 Jan 2023 12:54:05 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:05 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-11-surenb@google.com> Subject: [PATCH 10/41] mm: move mmap_lock assert function definitions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205420_647379_54AAED03 X-CRM114-Status: UNSURE ( 8.83 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 96e113e23d04..e49ba91bb1f0 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -60,6 +60,18 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write) #endif /* CONFIG_TRACING */ +static inline void mmap_assert_locked(struct mm_struct *mm) +{ + lockdep_assert_held(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + +static inline void mmap_assert_write_locked(struct mm_struct *mm) +{ + lockdep_assert_held_write(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -150,18 +162,6 @@ static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) up_read_non_owner(&mm->mmap_lock); } -static inline void mmap_assert_locked(struct mm_struct *mm) -{ - lockdep_assert_held(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - -static inline void mmap_assert_write_locked(struct mm_struct *mm) -{ - lockdep_assert_held_write(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - static inline int mmap_lock_is_contended(struct mm_struct *mm) { return rwsem_is_contended(&mm->mmap_lock); From patchwork Mon Jan 9 20:53:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 26114C54EBD for ; Mon, 9 Jan 2023 21:34:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=9HCNlFVLW/mc4oKsjYgv7b+wRKAYdaI4RhBJa69IiYo=; b=eAErCiwgCz2SSC45Sm/4JZTXha /zyBHZsIU52MXpmBdYNvQK7QeeFFY9HNQ0r6D8jQqg77RcA4PFDN/7sdbTDuyYKR7ADQQ/3GluuX9 AurFiE/8UT5y7qIdjIbEe++dX7svwI1FpS76CcrbdtF8Q+1c+K3rssEHZ6n6o7HhTq3xWPs1k15iC B+Dkvn1X7/OyefHH1vgL5lCgOQxqC85BcLyzJ0P8km+oNOIq7mRFutQrQ9u7476zfp6IJRWGH7rLw 3rxKU3GBRn8q6rDwoiU5z4/8w0xnXzxP1EPZlMaWMWYiQ81kM8usyxk99pVkQz7HVASfOJjmDszkw HJjFHnzQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzl1-004F43-KK; Mon, 09 Jan 2023 21:32:40 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzUB-0048n5-0G for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:15:15 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=VKcEnbcUm4yNPmJ0/UBWAk4YOoOktsWvie1GPNgM1C8=; b=gMESGmFMXl7bDUt52gcXjmtUmU GJy+Lkz4BWvhHtVkqm2MHjLngkge2O9eQrt/3qKIrtFsD48V9L+hr6+C3D1471kPWivY7S1re2l0O 5QxQzL1eucKYr2lEp2nvmMx6Hh6WSS8kzrQY68B8fJkkvihw0Pd6+owMQQP2vLuaI51BciKwRy4gN edj7ftm92dhiZ3c1Y5c+Ywo3v/M+0FhY+GwEefHMxe56aTlFL28EY77LIHdAwTKP4sqox5wezZDnk JnMlyVWID1WL4KcsxIi6OA2xT17ctHC7ZdP9xwffGbGDum7WGT4mDf2cZ9QmZx2/e8yLR3C66yFpe LNkVGR/Q==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzA1-002cL3-9u for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:29 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4bdeb1bbeafso103862317b3.4 for ; Mon, 09 Jan 2023 12:54:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VKcEnbcUm4yNPmJ0/UBWAk4YOoOktsWvie1GPNgM1C8=; b=GGWOlVFNZmpsv4EqX6fJ0IzU46mRQdwbpB1woCBvprw4wrB1Px0W3aw0R3eA4F/+xV kXuBRDFymMvypEzFr/D4iD7og2WAH2DpqMoKxtl2yVGo8HrvBlCIzI0TYQCav4I1owI5 QhJAem5dniKSevuuTKEF0EwC81wZP9AQ1hwH/aOaMvCSNfv3FcYZaW7nXgrr4EsyFNJE AJznojM8o2oQLC4YAgQdb7K64EDnrtRV2QjiyePckmOOYm4vpeviEyBwwk5GM5l0mBi5 NKaa6vy3INpTQuuxZ0BRwKOg99o3TKTitzAEr6jS2DiN0O+IImG4Tiab8jZ8nkO0QP6Q MsPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VKcEnbcUm4yNPmJ0/UBWAk4YOoOktsWvie1GPNgM1C8=; b=NslmbJBqgsFLs7a7AhwM057t24FQiwn2dBG73U3L/qJBWnN+uRwHFw0w0gXlDym1Lw axcdBoIb44wwaQScks4EydH7g2vj5ijyrbzS6nyw84ow0WgKNAC6vo1eMq/5g6pKDZXu tYch2D7PSHFKlMdFRHXnwgMVb8eV6WwiWtNwp0paxhyTDTnF6mzd2Y5lhdi1w0YRAHpZ DIWhknxgq3B6UOwzvGgZ1LJI8pSscG0p5nKungnN1fwBoULajNOw6KLJguLJcdLQ6eyW 1e/UonH+H5kwxllTWqmXMOVXqZzXotTDG1+2U1iVKoJrxI4aoYex63jRFuTF3N5cYCyo UzQA== X-Gm-Message-State: AFqh2kqp9mPtjAYbqJrWUAlZqmHRX5zfT75TBOVXNvr4mSxwBjx+HLPK qnTNhoPtrbb+DpsClTwO6v1KFuXbkWM= X-Google-Smtp-Source: AMrXdXuuklmw1U5OSS04yToFIKYXQnoMsaTWu76HfgvyMhLNzfJxTKm5wgWz/6XkfwWL03n+CBAO8y2WRQ4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a0d:d48a:0:b0:46a:f5ee:a2fe with SMTP id w132-20020a0dd48a000000b0046af5eea2femr1229443ywd.207.1673297647974; Mon, 09 Jan 2023 12:54:07 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:06 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-12-surenb@google.com> Subject: [PATCH 11/41] mm: export dump_mm() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205425_377298_A350EDAF X-CRM114-Status: UNSURE ( 9.04 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org mmap_assert_write_locked() will be used in the next patch to ensure vma write lock is taken only under mmap_lock exclusive lock. Because mmap_assert_write_locked() uses dump_mm() and there are cases when vma write lock is taken from inside a module, it's necessary to export dump_mm() function. Signed-off-by: Suren Baghdasaryan --- mm/debug.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/debug.c b/mm/debug.c index 7f8e5f744e42..b6e9e53469d1 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -215,6 +215,7 @@ void dump_mm(const struct mm_struct *mm) mm->def_flags, &mm->def_flags ); } +EXPORT_SYMBOL(dump_mm); static bool page_init_poisoning __read_mostly = true; From patchwork Mon Jan 9 20:53:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094376 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 11E66C63797 for ; Mon, 9 Jan 2023 21:35:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=8fEIufpEwby6r1n0Kw07lWnSVXNx03IT3UL6JDTFwss=; b=nP92rszmbRAxK91ohz0ENj/3IN JcgSU8ouupXEsy6p1RNnv6eEpGlIJQ+vKqp9kPIyIhrOZr+15doECwLPUa5fbt6u/XsBULT3ugPsC 3CpWy3q4kiPQiy74WOEKKMvK5RFScvvKTj8RqMjgFmXOf3eZ71hEDk3Uxl5KDukHiCiqXSVhyN+iy W6Qv3EjVZW+ez/U9NyockYRk6dTGUVk77ElotFUZPRQWmvPvCCU4NRTMzVgw0fMgRtXecSrjjwcmD +4qCAvy1p90KLpkGGD5H0ytDrB15AwQDd4JSxo4gzzoK5nty8cSjiDOMGdmhzMmfLQnEZ/huRGV0u 4GDbScdQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzlw-004FMj-Iv; Mon, 09 Jan 2023 21:33:37 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzUA-0048n5-1d for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:15:14 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=79bvrukKvLAS13wwuEGJsW+4aNck296i4OOdN8AZrSw=; b=dWIp+O+8u2kmRQdm02YE+UX1dF VA+BgWCni7k9gkMBuO6cC7B7w27XPerQWusz6/pHzJS/1kJsfeq3G28HZvdI6pEpRFvPUvGhEDPHL u3yYsLQnbnBGSccc9counkLpaUdO9bUP+25l5RchT5d10x5bkbvODoazNLiwzjrTlccQIN4i60bty yCwB4AZWl6Un3G1UY9efAGA+PMOu4AUSKaFaR09u6J60M8sukcgGWgAQJMLSxKzphXN8OrQ3RN2Pt n0bCnaTOxfHYySpGiG6AOD5Cia7Ib7yPkVSoXQHF5ZJ+HoyGg7/4C4kYYa9HoJDwh0N6hRUvFy9mM OBuKH7+g==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzA4-002cMq-RC for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:32 +0000 Received: by mail-yb1-xb49.google.com with SMTP id i10-20020a25f20a000000b006ea4f43c0ddso10380263ybe.21 for ; Mon, 09 Jan 2023 12:54:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=79bvrukKvLAS13wwuEGJsW+4aNck296i4OOdN8AZrSw=; b=TUshLIkna8lEowgf738l64K6OhURrpjePFxYxdKUuWhyUGH2D62dGB+OGRki5oXp4C xTYuRmR7XzQ7I/28ieatH6mJigev2F3K0HIzC6hflnBGI34WrRdJkJZ4AZfQFOQ8gGav 9UNpZpO2F0owMoX4KYmO5GlLvBicz0eIhvCyoFVAlBVkJOLooeK67cBYtsiWvskUp6uE tE8t31WtIP9ueuHL/u/lXs1s/RC2HGn9FvMQfHmx5Itoy4F3Nj/5Ls77RI1e7g50JCXC 9+/P6cV1602qOnEnSC6h6SSpk6hPN1ukv5Ai24vZZWfGTttvD7F17H6z929cULqI8Sm7 oWIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=79bvrukKvLAS13wwuEGJsW+4aNck296i4OOdN8AZrSw=; b=qa6s3XZQc+FJRdv1ztPikpHUumWedAggbRmQ7jxV6Lvd48wBvX4iyuNtM590L+E860 zrfYDnF53Q37ODoS7rTJjFnvm3Uqx3/6kHvVQ8Gt74JNJDgCem4bwcZG50BAUo37tM9L amSQi4njGT1aOF64W4sHS+TThAb/vLcVQYdnlkFKxbE8tUUsXEUvabEf1tVB/CwGIG5U 8GGNXd0sqDjqvqt1SYLjP0utFt5iPkMSHKHWd8p+K8wcF+w8z4ukC1UiJEAtU3VAgxFH YQ5PTZ5PjiznRpkOaDlpjiy+bjvferqqPZ9wmoYAwVBsdG3fBQS+h9SefB1eoNnlRWVL boqg== X-Gm-Message-State: AFqh2kpWCEsfebEj9COxvISRCKnBWCX/bwfjuJWXribeG8OOFrCyDSr2 7Ge2IxhoPwjI6dNN6LDb3wnJYxT0ZfI= X-Google-Smtp-Source: AMrXdXukETqaSxCpG7iZuCRz7oEpYHbdWSzbf1+vDAfX24FIika8R4beORIYrEIIAWwV84R+bp3+gXy0ol4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a0d:c481:0:b0:4ad:7104:1f66 with SMTP id g123-20020a0dc481000000b004ad71041f66mr3418570ywd.53.1673297651084; Mon, 09 Jan 2023 12:54:11 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:07 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-13-surenb@google.com> Subject: [PATCH 12/41] mm: add per-VMA lock and helper functions to control it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205428_967835_83DDB559 X-CRM114-Status: GOOD ( 18.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce a per-VMA rw_semaphore to be used during page fault handling instead of mmap_lock. Because there are cases when multiple VMAs need to be exclusively locked during VMA tree modifications, instead of the usual lock/unlock patter we mark a VMA as locked by taking per-VMA lock exclusively and setting vma->lock_seq to the current mm->lock_seq. When mmap_write_lock holder is done with all modifications and drops mmap_lock, it will increment mm->lock_seq, effectively unlocking all VMAs marked as locked. VMA lock is placed on the cache line boundary so that its 'count' field falls into the first cache line while the rest of the fields fall into the second cache line. This lets the 'count' field to be cached with other frequently accessed fields and used quickly in uncontended case while 'owner' and other fields used in the contended case will not invalidate the first cache line while waiting on the lock. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 80 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 8 ++++ include/linux/mmap_lock.h | 13 +++++++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 files changed, 108 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index f3f196e4d66d..ec2c4c227d51 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -612,6 +612,85 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_init_lock(struct vm_area_struct *vma) +{ + init_rwsem(&vma->lock); + vma->vm_lock_seq = -1; +} + +static inline void vma_write_lock(struct vm_area_struct *vma) +{ + int mm_lock_seq; + + mmap_assert_write_locked(vma->vm_mm); + + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + mm_lock_seq = READ_ONCE(vma->vm_mm->mm_lock_seq); + if (vma->vm_lock_seq == mm_lock_seq) + return; + + down_write(&vma->lock); + vma->vm_lock_seq = mm_lock_seq; + up_write(&vma->lock); +} + +/* + * Try to read-lock a vma. The function is allowed to occasionally yield false + * locked result to avoid performance overhead, in which case we fall back to + * using mmap_lock. The function should never yield false unlocked result. + */ +static inline bool vma_read_trylock(struct vm_area_struct *vma) +{ + /* Check before locking. A race might cause false locked result. */ + if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) + return false; + + if (unlikely(down_read_trylock(&vma->lock) == 0)) + return false; + + /* + * Overflow might produce false locked result. + * False unlocked result is impossible because we modify and check + * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * modification invalidates all existing locks. + */ + if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { + up_read(&vma->lock); + return false; + } + return true; +} + +static inline void vma_read_unlock(struct vm_area_struct *vma) +{ + up_read(&vma->lock); +} + +static inline void vma_assert_write_locked(struct vm_area_struct *vma) +{ + mmap_assert_write_locked(vma->vm_mm); + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline void vma_init_lock(struct vm_area_struct *vma) {} +static inline void vma_write_lock(struct vm_area_struct *vma) {} +static inline bool vma_read_trylock(struct vm_area_struct *vma) + { return false; } +static inline void vma_read_unlock(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -620,6 +699,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_init_lock(vma); } static inline void vma_set_anonymous(struct vm_area_struct *vma) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index d5cdec1314fe..5f7c5ca89931 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -555,6 +555,11 @@ struct vm_area_struct { pgprot_t vm_page_prot; unsigned long vm_flags; /* Flags, see mm.h. */ +#ifdef CONFIG_PER_VMA_LOCK + int vm_lock_seq; + struct rw_semaphore lock; +#endif + /* * For areas with an address space and backing store, * linkage into the address_space->i_mmap interval tree. @@ -680,6 +685,9 @@ struct mm_struct { * init_mm.mmlist, and are protected * by mmlist_lock */ +#ifdef CONFIG_PER_VMA_LOCK + int mm_lock_seq; +#endif unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index e49ba91bb1f0..40facd4c398b 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -72,6 +72,17 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_write_unlock_mm(struct mm_struct *mm) +{ + mmap_assert_write_locked(mm); + /* No races during update due to exclusive mmap_lock being held */ + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); +} +#else +static inline void vma_write_unlock_mm(struct mm_struct *mm) {} +#endif + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -114,12 +125,14 @@ static inline bool mmap_write_trylock(struct mm_struct *mm) static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true); + vma_write_unlock_mm(mm); up_write(&mm->mmap_lock); } static inline void mmap_write_downgrade(struct mm_struct *mm) { __mmap_lock_trace_acquire_returned(mm, false, true); + vma_write_unlock_mm(mm); downgrade_write(&mm->mmap_lock); } diff --git a/kernel/fork.c b/kernel/fork.c index 5986817f393c..c026d75108b3 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -474,6 +474,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ *new = data_race(*orig); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_init_lock(new); dup_anon_vma_name(orig, new); } return new; @@ -1145,6 +1146,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); +#ifdef CONFIG_PER_VMA_LOCK + WRITE_ONCE(mm->mm_lock_seq, 0); +#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index c9327abb771c..33269314e060 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -37,6 +37,9 @@ struct mm_struct init_mm = { .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), +#ifdef CONFIG_PER_VMA_LOCK + .mm_lock_seq = 0, +#endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, #ifdef CONFIG_IOMMU_SVA From patchwork Mon Jan 9 20:53:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AA112C5479D for ; Mon, 9 Jan 2023 21:03:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=KfMhvIQe/iDpJJpFx9nJama39Bu53ddsh2FSoQMQ9tA=; b=HNQ4YYL5eNccAmFU7Jc2wSafy5 X7ZfgtKTMwXR4Is4z+x5Mf9boylLU45fa4QkbBEsBKAMRarBQfA6CYIvQxFbNZYpQDGFleBvocM19 XM4xFaevx3c9JiwluE/xY+MvaRfX7S09A2PDRROJHNrlrRqgIYCHhMHyWI3t4e53MFwYYJGFAmFEr ZHfVW1StPherU9QAy9yeHAti9bvNw19p+R890eleAXVT7DCuGYasQekuh0xtMaw0fTMCvxU3ey2wV ZWEsAdBveCerx4LyNWwG289IRINU4sYAMORHn1UInFkwzxKts62wPeofpo73k1UYURkkL5ZtK9ZDv VMc9KNgA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzHM-0043xa-M5; Mon, 09 Jan 2023 21:02:01 +0000 Received: from mail-ot1-x34a.google.com ([2607:f8b0:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzH4-0043sK-LJ for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 21:01:50 +0000 Received: by mail-ot1-x34a.google.com with SMTP id cr28-20020a056830671c00b006774d5923ddso4920254otb.4 for ; Mon, 09 Jan 2023 13:01:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7ThjMRKbwnKxVGB3eylsidymuk6Qi/iLA6nYtJEm4MU=; b=RKn/zVLZEJYWNukTXjRCu4nz81zaytlRkR7rakIR2uB/aIfeElp0yCIRzuM8+pvZ3x BQOElK8tbS+rhMmhhEhBtfHOWGZOyvH/HCUdLENpEngLYbZsqjJ9OUXahFZPUHh+XaxU YMwkVhfHQ73TasfAveK5V4w8pX7O6IKpRzfg+l1/+SN9l2l60gXMRIOKbNPAPi3/90FE v/HorT/klJnq8Lcd3XX+VkW89j017v/vCfpCQTBG4XFodaDoUu6obKl8t1y+jpshNDo0 91Vfmkl3EHo0Elu4jQ9efLprq8Gy+cE160O08y2PdqRoSx1lsEPlDSvoNxeAyXeTk2x0 qpkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7ThjMRKbwnKxVGB3eylsidymuk6Qi/iLA6nYtJEm4MU=; b=EBCEMYrmfo/ClzeHkpmBJ0cNYIdN6SVIDESrQo6j1RI7qDa+UXcEkVkYz11C8sFIO7 veeDV8MG9PL/9RdUWyKdi52rEXuOREK49tslao+IJetHVjgB/ULt7gOALWNLSQXQKo6q 7dXgEt9qE7yDecH7LBZbkqTZqmfYujETRGoo/+wMeCDlLzhdQWtoaA2Pejj1PP+c5aAn V5fKavHj2dd/eypUw9mdKoCKua5UyURILCIbaz2lcCqqSNRp1i8MyMqf2qJPbVi4YI4M /6jdAoXzVUKYaTc6pdoUExg7gRbQ0L+3uiKQU1E1z3xHLEZ+dUHBXVrKk0l5QsMAO9GB 6n0A== X-Gm-Message-State: AFqh2kqBcjxEvekW5/OUw+hHUKX1BdUXa5dJ8smmUDrNHSnvlWU+3lmO iNZXnc/bkQhKUFfHF1OgtHFwp3t9Jr4= X-Google-Smtp-Source: AMrXdXvxnut8BEB8WYWMFHzY9PnfOwu60Xd9fTsfu57bMChvZCCQkV4pslftmJOaEGGsy1IoVxboiXkLZWU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a17:90a:5c86:b0:219:c1fb:5da8 with SMTP id r6-20020a17090a5c8600b00219c1fb5da8mr5399379pji.221.1673297653600; Mon, 09 Jan 2023 12:54:13 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:08 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-14-surenb@google.com> Subject: [PATCH 13/41] mm: introduce vma->vm_flags modifier functions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_130148_759587_0EEA58C6 X-CRM114-Status: GOOD ( 12.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To keep vma locking correctness when vm_flags are modified, add modifier functions to be used whenever flags are updated. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 38 ++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 8 +++++++- 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ec2c4c227d51..35cf0a6cbcc2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -702,6 +702,44 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma_init_lock(vma); } +/* Use when VMA is not part of the VMA tree and needs no locking */ +static inline +void init_vm_flags(struct vm_area_struct *vma, unsigned long flags) +{ + WRITE_ONCE(vma->vm_flags, flags); +} + +/* Use when VMA is part of the VMA tree and needs appropriate locking */ +static inline +void reset_vm_flags(struct vm_area_struct *vma, unsigned long flags) +{ + vma_write_lock(vma); + init_vm_flags(vma, flags); +} + +static inline +void set_vm_flags(struct vm_area_struct *vma, unsigned long flags) +{ + vma_write_lock(vma); + vma->vm_flags |= flags; +} + +static inline +void clear_vm_flags(struct vm_area_struct *vma, unsigned long flags) +{ + vma_write_lock(vma); + vma->vm_flags &= ~flags; +} + +static inline +void mod_vm_flags(struct vm_area_struct *vma, + unsigned long set, unsigned long clear) +{ + vma_write_lock(vma); + vma->vm_flags |= set; + vma->vm_flags &= ~clear; +} + static inline void vma_set_anonymous(struct vm_area_struct *vma) { vma->vm_ops = NULL; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5f7c5ca89931..0d27edd3e63a 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -553,7 +553,13 @@ struct vm_area_struct { * See vmf_insert_mixed_prot() for discussion. */ pgprot_t vm_page_prot; - unsigned long vm_flags; /* Flags, see mm.h. */ + + /* + * Flags, see mm.h. + * WARNING! Do not modify directly to keep correct VMA locking. + * Use {init|reset|set|clear|mod}_vm_flags() functions instead. + */ + unsigned long vm_flags; #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; From patchwork Mon Jan 9 20:53:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094332 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E6B07C5479D for ; Mon, 9 Jan 2023 21:15:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=kg/52yzFSqQuw57Ckdkt2Ad1EDbZjA5tJVdi2sBtcEY=; b=EzXLIoxHWjnIYGJGwPVmSUrjkE 6u4bBP1SA9jj9/LPm95ON2lpungfhwesUZgN48aOUf44a/h4mJmcunbVv2CE/G8MbHyJk/iDpnEzf mgdKVyea3idDxIdX5ddj+xms+OX1eY2DBizsPe2FpvR5ZGcdYuApAK3AS1Oln0KCd0Mh3yixLhOXq 01QiD3/nQrXr9OG9auZKaXDCRmLxgDEcUj6a9NPxr5nImd3HztSNCz9weUSpsh1R64IycYspbXb7K zYw3q0g84/PH+YkI+LYzSfp2CBcEkLLccYM6zKS7dqdGhedxo7abn2k723MRhw3crg8HCRsPys+Qf J32Sa4/w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzTA-0048Xq-Uj; Mon, 09 Jan 2023 21:14:13 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzSc-0048MK-CA for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:13:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=q5vG3v1PIcAm/ERdB03D9/j7bAwMWdEUWbQM9OP8Y4c=; b=I0uPCE6yZw5+MPNnoS/mS4IJvL bOBzzAMP0uqbSGxHSicdjggK9eg7qDhBwWveZFp7uFAc+/nrzM8rrUYnbqRPV7Qic5kmyZ+aUkw4V KppaGu7+U/Z80cDtvFR4cPPluDdHcNgFqqkVvSgNPrkrPvG5tpT0HTe+SmBSb195bgTV6ARtZzW7x I3bwAiygyXvzx07Lr0CKeFrCT6K+WniTaR+89VgBOcx9hlssZPx64ZbcpesUMp0IJJOWTlHXiOHCe DhIpQVPLmJvzLNyLJGoiWsa6xs5pkfpxHlpuqdegGRutuwduTU/GyIBgw3cRclUt0yk5jzGsAt6Bw qya1Yu8A==; Received: from mail-yw1-f202.google.com ([209.85.128.202]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pEzA3-002sw4-1B for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:31 +0000 Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-4597b0ff5e9so105262947b3.10 for ; Mon, 09 Jan 2023 12:54:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=q5vG3v1PIcAm/ERdB03D9/j7bAwMWdEUWbQM9OP8Y4c=; b=jGV2twit8eMqqlAwxSI2kIyw+GdbQsmn963oqlT3GN/ehTsmuPHj865pM8TWLhKQHd jTIMTXUStd0tYuBt7C6hb5YIOb4/wBppGWfSzDrzDKFeyuMa9uP2aNs8o441PZXFnjCo UTpNeX3UnEavX2v58G4+rqPOBU9aUnr6H/ChTcAvQIipWCjxL9GlTdM7oqLGXY19DPFQ q7bt4dBbBloqbwmYh9ylKkKgLjVpvssPEYXAiDYbhfBukg85oeSOt+1VHZNlGA7zFiYp 8fJla0RzQ4AZwln2gfdXjRilgUN42dHF3aEjUxK8uZvM/sfKPj53v6EzOYOtMmGPRMLa fuFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=q5vG3v1PIcAm/ERdB03D9/j7bAwMWdEUWbQM9OP8Y4c=; b=kz2IFKmZ+VENaMwSRW+py4Y31CRgQctaJxcOwzlZinuRNT+YbeKrr7zygGcOLPaEWI dEvNL69+5AZZ6Gjr5/eHLSvRCYE2LweuA+e6ZLZV2s2PTuv+qe19GzylozFAQaUVLCBr xFHWGqKshDhwgAGpkX60bQDEAQOmWm9V/PVsluFIKJJTvpu2eQARv/6emxsOzXCYG3M6 DLr3mDDVWOqP5nlxcvGgXEbtWS5/EO2Kz3N3Orry+ce2txlKwEtYTMjV5v/aSfaKEHJ5 gHwyCrI7ntjK9PSkP/VCnICyFpG8q+oIDdK3mmobUhSpK1lRbsuGa2oVPl0iKB7bOvGw HJUw== X-Gm-Message-State: AFqh2krb1sC4FU/wFjIl4zbhm/yH9t9QXv4/7rswFeNRJBx2QpnNI2nK msspRBpHa0CiJeAyyT02NcfT9bJhkNs= X-Google-Smtp-Source: AMrXdXvsQzJo8PSoW1ccRcrlqwtefe5KwIQ7Te6dX0P6/V4WPjQWsck4Zjq+DJXzzvFNYEa755Hhn8p6kg0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:ca88:0:b0:7bf:24c7:997e with SMTP id a130-20020a25ca88000000b007bf24c7997emr393826ybg.217.1673297656291; Mon, 09 Jan 2023 12:54:16 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:09 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-15-surenb@google.com> Subject: [PATCH 14/41] mm: replace VM_LOCKED_CLEAR_MASK with VM_LOCKED_MASK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205428_343571_1177544D X-CRM114-Status: GOOD ( 18.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To simplify the usage of VM_LOCKED_CLEAR_MASK in clear_vm_flags(), replace it with VM_LOCKED_MASK bitmask and convert all users. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 4 ++-- kernel/fork.c | 2 +- mm/hugetlb.c | 4 ++-- mm/mlock.c | 6 +++--- mm/mmap.c | 6 +++--- mm/mremap.c | 2 +- 6 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 35cf0a6cbcc2..2b16d45b75a6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -416,8 +416,8 @@ extern unsigned int kobjsize(const void *objp); /* This mask defines which mm->def_flags a process can inherit its parent */ #define VM_INIT_DEF_MASK VM_NOHUGEPAGE -/* This mask is used to clear all the VMA flags used by mlock */ -#define VM_LOCKED_CLEAR_MASK (~(VM_LOCKED | VM_LOCKONFAULT)) +/* This mask represents all the VMA flag bits used by mlock */ +#define VM_LOCKED_MASK (VM_LOCKED | VM_LOCKONFAULT) /* Arch-specific flags to clear when updating VM flags on protection change */ #ifndef VM_ARCH_CLEAR diff --git a/kernel/fork.c b/kernel/fork.c index c026d75108b3..1591dd8a0745 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -674,7 +674,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, tmp->anon_vma = NULL; } else if (anon_vma_fork(tmp, mpnt)) goto fail_nomem_anon_vma_fork; - tmp->vm_flags &= ~(VM_LOCKED | VM_LOCKONFAULT); + clear_vm_flags(tmp, VM_LOCKED_MASK); file = tmp->vm_file; if (file) { struct address_space *mapping = file->f_mapping; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index db895230ee7e..24861cbfa2b1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6950,8 +6950,8 @@ static unsigned long page_table_shareable(struct vm_area_struct *svma, unsigned long s_end = sbase + PUD_SIZE; /* Allow segments to share if only one is marked locked */ - unsigned long vm_flags = vma->vm_flags & VM_LOCKED_CLEAR_MASK; - unsigned long svm_flags = svma->vm_flags & VM_LOCKED_CLEAR_MASK; + unsigned long vm_flags = vma->vm_flags & ~VM_LOCKED_MASK; + unsigned long svm_flags = svma->vm_flags & ~VM_LOCKED_MASK; /* * match the virtual addresses, permission and the alignment of the diff --git a/mm/mlock.c b/mm/mlock.c index 7032f6dd0ce1..06aa9e204fac 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -490,7 +490,7 @@ static int apply_vma_lock_flags(unsigned long start, size_t len, prev = mas_prev(&mas, 0); for (nstart = start ; ; ) { - vm_flags_t newflags = vma->vm_flags & VM_LOCKED_CLEAR_MASK; + vm_flags_t newflags = vma->vm_flags & ~VM_LOCKED_MASK; newflags |= flags; @@ -662,7 +662,7 @@ static int apply_mlockall_flags(int flags) struct vm_area_struct *vma, *prev = NULL; vm_flags_t to_add = 0; - current->mm->def_flags &= VM_LOCKED_CLEAR_MASK; + current->mm->def_flags &= ~VM_LOCKED_MASK; if (flags & MCL_FUTURE) { current->mm->def_flags |= VM_LOCKED; @@ -682,7 +682,7 @@ static int apply_mlockall_flags(int flags) mas_for_each(&mas, vma, ULONG_MAX) { vm_flags_t newflags; - newflags = vma->vm_flags & VM_LOCKED_CLEAR_MASK; + newflags = vma->vm_flags & ~VM_LOCKED_MASK; newflags |= to_add; /* Ignore errors */ diff --git a/mm/mmap.c b/mm/mmap.c index 9db37adfc00a..5c4b608edde9 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2721,7 +2721,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, if ((vm_flags & VM_SPECIAL) || vma_is_dax(vma) || is_vm_hugetlb_page(vma) || vma == get_gate_vma(current->mm)) - vma->vm_flags &= VM_LOCKED_CLEAR_MASK; + clear_vm_flags(vma, VM_LOCKED_MASK); else mm->locked_vm += (len >> PAGE_SHIFT); } @@ -3392,8 +3392,8 @@ static struct vm_area_struct *__install_special_mapping( vma->vm_start = addr; vma->vm_end = addr + len; - vma->vm_flags = vm_flags | mm->def_flags | VM_DONTEXPAND | VM_SOFTDIRTY; - vma->vm_flags &= VM_LOCKED_CLEAR_MASK; + init_vm_flags(vma, (vm_flags | mm->def_flags | + VM_DONTEXPAND | VM_SOFTDIRTY) & ~VM_LOCKED_MASK); vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); vma->vm_ops = ops; diff --git a/mm/mremap.c b/mm/mremap.c index fe587c5d6591..5f6f9931bff1 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -686,7 +686,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, if (unlikely(!err && (flags & MREMAP_DONTUNMAP))) { /* We always clear VM_LOCKED[ONFAULT] on the old vma */ - vma->vm_flags &= VM_LOCKED_CLEAR_MASK; + clear_vm_flags(vma, VM_LOCKED_MASK); /* * anon_vma links of the old vma is no longer needed after its page From patchwork Mon Jan 9 20:53:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094314 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A30C7C5479D for ; Mon, 9 Jan 2023 20:56:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Y+AY/UN48PfJb7lZswQibU+7If9AD8qNsc608g/gGg0=; b=Gxul+UXCXd4vFLiV49cMqHhCGs lGADhSiDvwbqsQ8+PJXRFwq7KIlQYHIC6EoCnYp4If6BXjnT98OACTdSCI0xfqWjF28VqurP0kFuV hVeJDaK31mLFOtLI+/Jboae6lRzQR5tX25CX5ZQu9u5w/fTRxqdrezIxKPMACSdsXhMx6iwCm6The QeXd6k/CyDYa+AqhdxTigRdl2Z35+r3p9yjhsHMkI5faiZ7qiE0c445+sGumqRr3qNaNlQMoy3eDS Ynnse+BbNxQHkBmyY0DXL/its0W/GiTz44F3Y4jUfM9QanifeT6devH4SMg9HQJiq2tH6ITbfoo9+ z3L79AWg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAY-0040Sm-IB; Mon, 09 Jan 2023 20:54:58 +0000 Received: from mail-yw1-f202.google.com ([209.85.128.202]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEz9z-00401I-EY for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:33 +0000 Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-4ce566db73eso35372887b3.11 for ; Mon, 09 Jan 2023 12:54:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6Ardk86ZWr85sWOJatKIPgo5mLXp8YCAbjCFaUECHgw=; b=U85xBLsLgZCrqvXfEAg+BaZGz/oCE54YcPmN31yIJksHoum3GUFpvX6o/xRYHrKVyd E4a8yls32cSVlFzqD8P875zWgC/tJSi2iWYjbH9dUZozVCd+b4WdQPTCi9gGl7+7zj7A bQSa+/lZgMkuENk9nuHI7gV3Bhxr7XKQMhvJd8f0UiXEXL2tce0fkIW+hBTEhRWZnOgt /2tL4nNxLnQgCAC2A3SpLPpA792wJNiga2ra8JsoQUeBZO8Q3z22sDcpkTZ9PKzrpIa/ lk8fMvausMKZA9FfvsgHWY3VFM3qx2b48rKCPwhE5oifYwxJxANcNQsndjFCM9F7eiVG iPKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6Ardk86ZWr85sWOJatKIPgo5mLXp8YCAbjCFaUECHgw=; b=CLrZTAjyr3lcs4Ht/VtS6lu91Ip6S3IK0PqUWm3ShCWgXLbMeAFk3/QbTO4xGuagaI oW2N7hYJUtWeIMLf8tuIFArCqPKd8UxJlkBybvGMTFEtA11YhA0P9LX/z4e9p0pTAfr2 CnYKRJ5baRyfFJ/IADZc43fPYKoUgqK5Yx3oCTmWj9/YvxXROELNoEtgGsoliqOhmQzh ZxQVoYo9B3d5lzcBsMY3GsdtiYBVU7DOGbEbU5pF1y3iOSN3ym8U6V4IoJb6pfldzMiL C4geT4qqizlYUOyohorfZE/sDMQOyAVFBnLYgzoEnHgaN9+NTTJhYS5N1UxCpqX6XJHV UClQ== X-Gm-Message-State: AFqh2koqT3zDza6hzA+eTa2M/CQxl1X+cZRi5E+mBffoUct81bCMfaAV ZRsr89CAiA3NhB7UA8BhPVWHWHNQJ8g= X-Google-Smtp-Source: AMrXdXtHh3sXAXvU3ipTAaTFjEARVHgNgaZ/Vo1lplSDPhEcoTa76U/QAOd7/9m4ZYA8KiqxLFoWIR57kX0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:ef4b:0:b0:7b9:3895:a1e6 with SMTP id w11-20020a25ef4b000000b007b93895a1e6mr1365847ybm.335.1673297658938; Mon, 09 Jan 2023 12:54:18 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:10 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-16-surenb@google.com> Subject: [PATCH 15/41] mm: replace vma->vm_flags direct modifications with modifier calls From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_125423_781098_E5196EE4 X-CRM114-Status: GOOD ( 16.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Replace direct modifications to vma->vm_flags with calls to modifier functions to be able to track flag changes and to keep vma locking correctness. Signed-off-by: Suren Baghdasaryan --- arch/arm/kernel/process.c | 2 +- arch/ia64/mm/init.c | 8 ++++---- arch/loongarch/include/asm/tlb.h | 2 +- arch/powerpc/kvm/book3s_xive_native.c | 2 +- arch/powerpc/mm/book3s64/subpage_prot.c | 2 +- arch/powerpc/platforms/book3s/vas-api.c | 2 +- arch/powerpc/platforms/cell/spufs/file.c | 14 +++++++------- arch/s390/mm/gmap.c | 3 +-- arch/x86/entry/vsyscall/vsyscall_64.c | 2 +- arch/x86/kernel/cpu/sgx/driver.c | 2 +- arch/x86/kernel/cpu/sgx/virt.c | 2 +- arch/x86/mm/pat/memtype.c | 6 +++--- arch/x86/um/mem_32.c | 2 +- drivers/acpi/pfr_telemetry.c | 2 +- drivers/android/binder.c | 3 +-- drivers/char/mspec.c | 2 +- drivers/crypto/hisilicon/qm.c | 2 +- drivers/dax/device.c | 2 +- drivers/dma/idxd/cdev.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 +- drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 4 ++-- drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c | 4 ++-- drivers/gpu/drm/amd/amdkfd/kfd_events.c | 4 ++-- drivers/gpu/drm/amd/amdkfd/kfd_process.c | 4 ++-- drivers/gpu/drm/drm_gem.c | 2 +- drivers/gpu/drm/drm_gem_dma_helper.c | 3 +-- drivers/gpu/drm/drm_gem_shmem_helper.c | 2 +- drivers/gpu/drm/drm_vm.c | 8 ++++---- drivers/gpu/drm/etnaviv/etnaviv_gem.c | 2 +- drivers/gpu/drm/exynos/exynos_drm_gem.c | 4 ++-- drivers/gpu/drm/gma500/framebuffer.c | 2 +- drivers/gpu/drm/i810/i810_dma.c | 2 +- drivers/gpu/drm/i915/gem/i915_gem_mman.c | 4 ++-- drivers/gpu/drm/mediatek/mtk_drm_gem.c | 2 +- drivers/gpu/drm/msm/msm_gem.c | 2 +- drivers/gpu/drm/omapdrm/omap_gem.c | 3 +-- drivers/gpu/drm/rockchip/rockchip_drm_gem.c | 3 +-- drivers/gpu/drm/tegra/gem.c | 5 ++--- drivers/gpu/drm/ttm/ttm_bo_vm.c | 3 +-- drivers/gpu/drm/virtio/virtgpu_vram.c | 2 +- drivers/gpu/drm/vmwgfx/vmwgfx_ttm_glue.c | 2 +- drivers/gpu/drm/xen/xen_drm_front_gem.c | 3 +-- drivers/hsi/clients/cmt_speech.c | 2 +- drivers/hwtracing/intel_th/msu.c | 2 +- drivers/hwtracing/stm/core.c | 2 +- drivers/infiniband/hw/hfi1/file_ops.c | 4 ++-- drivers/infiniband/hw/mlx5/main.c | 4 ++-- drivers/infiniband/hw/qib/qib_file_ops.c | 13 ++++++------- drivers/infiniband/hw/usnic/usnic_ib_verbs.c | 2 +- drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c | 2 +- .../media/common/videobuf2/videobuf2-dma-contig.c | 2 +- drivers/media/common/videobuf2/videobuf2-vmalloc.c | 2 +- drivers/media/v4l2-core/videobuf-dma-contig.c | 2 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 4 ++-- drivers/media/v4l2-core/videobuf-vmalloc.c | 2 +- drivers/misc/cxl/context.c | 2 +- drivers/misc/habanalabs/common/memory.c | 2 +- drivers/misc/habanalabs/gaudi/gaudi.c | 4 ++-- drivers/misc/habanalabs/gaudi2/gaudi2.c | 8 ++++---- drivers/misc/habanalabs/goya/goya.c | 4 ++-- drivers/misc/ocxl/context.c | 4 ++-- drivers/misc/ocxl/sysfs.c | 2 +- drivers/misc/open-dice.c | 6 +++--- drivers/misc/sgi-gru/grufile.c | 4 ++-- drivers/misc/uacce/uacce.c | 2 +- drivers/sbus/char/oradax.c | 2 +- drivers/scsi/cxlflash/ocxl_hw.c | 2 +- drivers/scsi/sg.c | 2 +- drivers/staging/media/atomisp/pci/hmm/hmm_bo.c | 2 +- drivers/staging/media/deprecated/meye/meye.c | 4 ++-- .../media/deprecated/stkwebcam/stk-webcam.c | 2 +- drivers/target/target_core_user.c | 2 +- drivers/uio/uio.c | 2 +- drivers/usb/core/devio.c | 3 +-- drivers/usb/mon/mon_bin.c | 3 +-- drivers/vdpa/vdpa_user/iova_domain.c | 2 +- drivers/vfio/pci/vfio_pci_core.c | 2 +- drivers/vhost/vdpa.c | 2 +- drivers/video/fbdev/68328fb.c | 2 +- drivers/video/fbdev/core/fb_defio.c | 4 ++-- drivers/xen/gntalloc.c | 2 +- drivers/xen/gntdev.c | 4 ++-- drivers/xen/privcmd-buf.c | 2 +- drivers/xen/privcmd.c | 4 ++-- fs/aio.c | 2 +- fs/cramfs/inode.c | 2 +- fs/erofs/data.c | 2 +- fs/exec.c | 4 ++-- fs/ext4/file.c | 2 +- fs/fuse/dax.c | 2 +- fs/hugetlbfs/inode.c | 4 ++-- fs/orangefs/file.c | 3 +-- fs/proc/task_mmu.c | 2 +- fs/proc/vmcore.c | 3 +-- fs/userfaultfd.c | 12 ++++++------ fs/xfs/xfs_file.c | 2 +- include/linux/mm.h | 2 +- kernel/bpf/ringbuf.c | 4 ++-- kernel/bpf/syscall.c | 4 ++-- kernel/events/core.c | 2 +- kernel/kcov.c | 2 +- kernel/relay.c | 2 +- mm/madvise.c | 2 +- mm/memory.c | 6 +++--- mm/mlock.c | 6 +++--- mm/mmap.c | 10 +++++----- mm/mprotect.c | 2 +- mm/mremap.c | 6 +++--- mm/nommu.c | 11 ++++++----- mm/secretmem.c | 2 +- mm/shmem.c | 2 +- mm/vmalloc.c | 2 +- net/ipv4/tcp.c | 4 ++-- security/selinux/selinuxfs.c | 6 +++--- sound/core/oss/pcm_oss.c | 2 +- sound/core/pcm_native.c | 9 +++++---- sound/soc/pxa/mmp-sspa.c | 2 +- sound/usb/usx2y/us122l.c | 4 ++-- sound/usb/usx2y/usX2Yhwdep.c | 2 +- sound/usb/usx2y/usx2yhwdeppcm.c | 2 +- 120 files changed, 194 insertions(+), 205 deletions(-) diff --git a/arch/arm/kernel/process.c b/arch/arm/kernel/process.c index f811733a8fc5..ec65f3ea3150 100644 --- a/arch/arm/kernel/process.c +++ b/arch/arm/kernel/process.c @@ -316,7 +316,7 @@ static int __init gate_vma_init(void) gate_vma.vm_page_prot = PAGE_READONLY_EXEC; gate_vma.vm_start = 0xffff0000; gate_vma.vm_end = 0xffff0000 + PAGE_SIZE; - gate_vma.vm_flags = VM_READ | VM_EXEC | VM_MAYREAD | VM_MAYEXEC; + init_vm_flags(&gate_vma, VM_READ | VM_EXEC | VM_MAYREAD | VM_MAYEXEC); return 0; } arch_initcall(gate_vma_init); diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c index fc4e4217e87f..d355e0ce28ab 100644 --- a/arch/ia64/mm/init.c +++ b/arch/ia64/mm/init.c @@ -109,7 +109,7 @@ ia64_init_addr_space (void) vma_set_anonymous(vma); vma->vm_start = current->thread.rbs_bot & PAGE_MASK; vma->vm_end = vma->vm_start + PAGE_SIZE; - vma->vm_flags = VM_DATA_DEFAULT_FLAGS|VM_GROWSUP|VM_ACCOUNT; + init_vm_flags(vma, VM_DATA_DEFAULT_FLAGS|VM_GROWSUP|VM_ACCOUNT); vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); mmap_write_lock(current->mm); if (insert_vm_struct(current->mm, vma)) { @@ -127,8 +127,8 @@ ia64_init_addr_space (void) vma_set_anonymous(vma); vma->vm_end = PAGE_SIZE; vma->vm_page_prot = __pgprot(pgprot_val(PAGE_READONLY) | _PAGE_MA_NAT); - vma->vm_flags = VM_READ | VM_MAYREAD | VM_IO | - VM_DONTEXPAND | VM_DONTDUMP; + init_vm_flags(vma, VM_READ | VM_MAYREAD | VM_IO | + VM_DONTEXPAND | VM_DONTDUMP); mmap_write_lock(current->mm); if (insert_vm_struct(current->mm, vma)) { mmap_write_unlock(current->mm); @@ -272,7 +272,7 @@ static int __init gate_vma_init(void) vma_init(&gate_vma, NULL); gate_vma.vm_start = FIXADDR_USER_START; gate_vma.vm_end = FIXADDR_USER_END; - gate_vma.vm_flags = VM_READ | VM_MAYREAD | VM_EXEC | VM_MAYEXEC; + init_vm_flags(&gate_vma, VM_READ | VM_MAYREAD | VM_EXEC | VM_MAYEXEC); gate_vma.vm_page_prot = __pgprot(__ACCESS_BITS | _PAGE_PL_3 | _PAGE_AR_RX); return 0; diff --git a/arch/loongarch/include/asm/tlb.h b/arch/loongarch/include/asm/tlb.h index dd24f5898f65..51e35b44d105 100644 --- a/arch/loongarch/include/asm/tlb.h +++ b/arch/loongarch/include/asm/tlb.h @@ -149,7 +149,7 @@ static inline void tlb_flush(struct mmu_gather *tlb) struct vm_area_struct vma; vma.vm_mm = tlb->mm; - vma.vm_flags = 0; + init_vm_flags(&vma, 0); if (tlb->fullmm) { flush_tlb_mm(tlb->mm); return; diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c index 4f566bea5e10..7976af0f5ff8 100644 --- a/arch/powerpc/kvm/book3s_xive_native.c +++ b/arch/powerpc/kvm/book3s_xive_native.c @@ -324,7 +324,7 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev, return -EINVAL; } - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot); /* diff --git a/arch/powerpc/mm/book3s64/subpage_prot.c b/arch/powerpc/mm/book3s64/subpage_prot.c index d73b3b4176e8..72948cdb1911 100644 --- a/arch/powerpc/mm/book3s64/subpage_prot.c +++ b/arch/powerpc/mm/book3s64/subpage_prot.c @@ -156,7 +156,7 @@ static void subpage_mark_vma_nohuge(struct mm_struct *mm, unsigned long addr, * VM_NOHUGEPAGE and split them. */ for_each_vma_range(vmi, vma, addr + len) { - vma->vm_flags |= VM_NOHUGEPAGE; + set_vm_flags(vma, VM_NOHUGEPAGE); walk_page_vma(vma, &subpage_walk_ops, NULL); } } diff --git a/arch/powerpc/platforms/book3s/vas-api.c b/arch/powerpc/platforms/book3s/vas-api.c index eb5bed333750..a81615768fff 100644 --- a/arch/powerpc/platforms/book3s/vas-api.c +++ b/arch/powerpc/platforms/book3s/vas-api.c @@ -525,7 +525,7 @@ static int coproc_mmap(struct file *fp, struct vm_area_struct *vma) pfn = paste_addr >> PAGE_SHIFT; /* flags, page_prot from cxl_mmap(), except we want cachable */ - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_cached(vma->vm_page_prot); prot = __pgprot(pgprot_val(vma->vm_page_prot) | _PAGE_DIRTY); diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c index 62d90a5e23d1..784fa39a484a 100644 --- a/arch/powerpc/platforms/cell/spufs/file.c +++ b/arch/powerpc/platforms/cell/spufs/file.c @@ -291,7 +291,7 @@ static int spufs_mem_mmap(struct file *file, struct vm_area_struct *vma) if (!(vma->vm_flags & VM_SHARED)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot); vma->vm_ops = &spufs_mem_mmap_vmops; @@ -381,7 +381,7 @@ static int spufs_cntl_mmap(struct file *file, struct vm_area_struct *vma) if (!(vma->vm_flags & VM_SHARED)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &spufs_cntl_mmap_vmops; @@ -1043,7 +1043,7 @@ static int spufs_signal1_mmap(struct file *file, struct vm_area_struct *vma) if (!(vma->vm_flags & VM_SHARED)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &spufs_signal1_mmap_vmops; @@ -1179,7 +1179,7 @@ static int spufs_signal2_mmap(struct file *file, struct vm_area_struct *vma) if (!(vma->vm_flags & VM_SHARED)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &spufs_signal2_mmap_vmops; @@ -1302,7 +1302,7 @@ static int spufs_mss_mmap(struct file *file, struct vm_area_struct *vma) if (!(vma->vm_flags & VM_SHARED)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &spufs_mss_mmap_vmops; @@ -1364,7 +1364,7 @@ static int spufs_psmap_mmap(struct file *file, struct vm_area_struct *vma) if (!(vma->vm_flags & VM_SHARED)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &spufs_psmap_mmap_vmops; @@ -1424,7 +1424,7 @@ static int spufs_mfc_mmap(struct file *file, struct vm_area_struct *vma) if (!(vma->vm_flags & VM_SHARED)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &spufs_mfc_mmap_vmops; diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 74e1d873dce0..3811d6c86d09 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -2522,8 +2522,7 @@ static inline void thp_split_mm(struct mm_struct *mm) VMA_ITERATOR(vmi, mm, 0); for_each_vma(vmi, vma) { - vma->vm_flags &= ~VM_HUGEPAGE; - vma->vm_flags |= VM_NOHUGEPAGE; + mod_vm_flags(vma, VM_NOHUGEPAGE, VM_HUGEPAGE); walk_page_vma(vma, &thp_split_walk_ops, NULL); } mm->def_flags |= VM_NOHUGEPAGE; diff --git a/arch/x86/entry/vsyscall/vsyscall_64.c b/arch/x86/entry/vsyscall/vsyscall_64.c index 4af81df133ee..e2a1626d86d8 100644 --- a/arch/x86/entry/vsyscall/vsyscall_64.c +++ b/arch/x86/entry/vsyscall/vsyscall_64.c @@ -391,7 +391,7 @@ void __init map_vsyscall(void) } if (vsyscall_mode == XONLY) - gate_vma.vm_flags = VM_EXEC; + init_vm_flags(&gate_vma, VM_EXEC); BUILD_BUG_ON((unsigned long)__fix_to_virt(VSYSCALL_PAGE) != (unsigned long)VSYSCALL_ADDR); diff --git a/arch/x86/kernel/cpu/sgx/driver.c b/arch/x86/kernel/cpu/sgx/driver.c index aa9b8b868867..42c0bded93b6 100644 --- a/arch/x86/kernel/cpu/sgx/driver.c +++ b/arch/x86/kernel/cpu/sgx/driver.c @@ -95,7 +95,7 @@ static int sgx_mmap(struct file *file, struct vm_area_struct *vma) return ret; vma->vm_ops = &sgx_vm_ops; - vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | VM_IO; + set_vm_flags(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | VM_IO); vma->vm_private_data = encl; return 0; diff --git a/arch/x86/kernel/cpu/sgx/virt.c b/arch/x86/kernel/cpu/sgx/virt.c index 6a77a14eee38..0774a0bfeb28 100644 --- a/arch/x86/kernel/cpu/sgx/virt.c +++ b/arch/x86/kernel/cpu/sgx/virt.c @@ -105,7 +105,7 @@ static int sgx_vepc_mmap(struct file *file, struct vm_area_struct *vma) vma->vm_ops = &sgx_vepc_vm_ops; /* Don't copy VMA in fork() */ - vma->vm_flags |= VM_PFNMAP | VM_IO | VM_DONTDUMP | VM_DONTCOPY; + set_vm_flags(vma, VM_PFNMAP | VM_IO | VM_DONTDUMP | VM_DONTCOPY); vma->vm_private_data = vepc; return 0; diff --git a/arch/x86/mm/pat/memtype.c b/arch/x86/mm/pat/memtype.c index 46de9cf5c91d..9e490a372896 100644 --- a/arch/x86/mm/pat/memtype.c +++ b/arch/x86/mm/pat/memtype.c @@ -999,7 +999,7 @@ int track_pfn_remap(struct vm_area_struct *vma, pgprot_t *prot, ret = reserve_pfn_range(paddr, size, prot, 0); if (ret == 0 && vma) - vma->vm_flags |= VM_PAT; + set_vm_flags(vma, VM_PAT); return ret; } @@ -1065,7 +1065,7 @@ void untrack_pfn(struct vm_area_struct *vma, unsigned long pfn, } free_pfn_range(paddr, size); if (vma) - vma->vm_flags &= ~VM_PAT; + clear_vm_flags(vma, VM_PAT); } /* @@ -1075,7 +1075,7 @@ void untrack_pfn(struct vm_area_struct *vma, unsigned long pfn, */ void untrack_pfn_moved(struct vm_area_struct *vma) { - vma->vm_flags &= ~VM_PAT; + clear_vm_flags(vma, VM_PAT); } pgprot_t pgprot_writecombine(pgprot_t prot) diff --git a/arch/x86/um/mem_32.c b/arch/x86/um/mem_32.c index cafd01f730da..bfd2c320ad25 100644 --- a/arch/x86/um/mem_32.c +++ b/arch/x86/um/mem_32.c @@ -16,7 +16,7 @@ static int __init gate_vma_init(void) vma_init(&gate_vma, NULL); gate_vma.vm_start = FIXADDR_USER_START; gate_vma.vm_end = FIXADDR_USER_END; - gate_vma.vm_flags = VM_READ | VM_MAYREAD | VM_EXEC | VM_MAYEXEC; + init_vm_flags(&gate_vma, VM_READ | VM_MAYREAD | VM_EXEC | VM_MAYEXEC); gate_vma.vm_page_prot = PAGE_READONLY; return 0; diff --git a/drivers/acpi/pfr_telemetry.c b/drivers/acpi/pfr_telemetry.c index 27fb6cdad75f..9e339c705b5b 100644 --- a/drivers/acpi/pfr_telemetry.c +++ b/drivers/acpi/pfr_telemetry.c @@ -310,7 +310,7 @@ pfrt_log_mmap(struct file *file, struct vm_area_struct *vma) return -EROFS; /* changing from read to write with mprotect is not allowed */ - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); pfrt_log_dev = to_pfrt_log_dev(file); diff --git a/drivers/android/binder.c b/drivers/android/binder.c index 880224ec6abb..dd6c99223b8c 100644 --- a/drivers/android/binder.c +++ b/drivers/android/binder.c @@ -5572,8 +5572,7 @@ static int binder_mmap(struct file *filp, struct vm_area_struct *vma) proc->pid, vma->vm_start, vma->vm_end, "bad vm_flags", -EPERM); return -EPERM; } - vma->vm_flags |= VM_DONTCOPY | VM_MIXEDMAP; - vma->vm_flags &= ~VM_MAYWRITE; + mod_vm_flags(vma, VM_DONTCOPY | VM_MIXEDMAP, VM_MAYWRITE); vma->vm_ops = &binder_vm_ops; vma->vm_private_data = proc; diff --git a/drivers/char/mspec.c b/drivers/char/mspec.c index f8231e2e84be..57bd36a28f95 100644 --- a/drivers/char/mspec.c +++ b/drivers/char/mspec.c @@ -206,7 +206,7 @@ mspec_mmap(struct file *file, struct vm_area_struct *vma, refcount_set(&vdata->refcnt, 1); vma->vm_private_data = vdata; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); if (vdata->type == MSPEC_UNCACHED) vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &mspec_vm_ops; diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c index 007ac7a69ce7..57ecdb5c97fb 100644 --- a/drivers/crypto/hisilicon/qm.c +++ b/drivers/crypto/hisilicon/qm.c @@ -2363,7 +2363,7 @@ static int hisi_qm_uacce_mmap(struct uacce_queue *q, return -EINVAL; } - vma->vm_flags |= VM_IO; + set_vm_flags(vma, VM_IO); return remap_pfn_range(vma, vma->vm_start, phys_base >> PAGE_SHIFT, diff --git a/drivers/dax/device.c b/drivers/dax/device.c index 5494d745ced5..6e9726dfaa7e 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -308,7 +308,7 @@ static int dax_mmap(struct file *filp, struct vm_area_struct *vma) return rc; vma->vm_ops = &dax_vm_ops; - vma->vm_flags |= VM_HUGEPAGE; + set_vm_flags(vma, VM_HUGEPAGE); return 0; } diff --git a/drivers/dma/idxd/cdev.c b/drivers/dma/idxd/cdev.c index e13e92609943..51cf836cf329 100644 --- a/drivers/dma/idxd/cdev.c +++ b/drivers/dma/idxd/cdev.c @@ -201,7 +201,7 @@ static int idxd_cdev_mmap(struct file *filp, struct vm_area_struct *vma) if (rc < 0) return rc; - vma->vm_flags |= VM_DONTCOPY; + set_vm_flags(vma, VM_DONTCOPY); pfn = (base + idxd_get_wq_portal_full_offset(wq->id, IDXD_PORTAL_LIMITED)) >> PAGE_SHIFT; vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c index bb7350ea1d75..70b08a0d13cd 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c @@ -257,7 +257,7 @@ static int amdgpu_gem_object_mmap(struct drm_gem_object *obj, struct vm_area_str */ if (is_cow_mapping(vma->vm_flags) && !(vma->vm_flags & VM_ACCESS_FLAGS)) - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); return drm_gem_ttm_mmap(obj, vma); } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c index 6d291aa6386b..7beb8dd6a5e6 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c @@ -2879,8 +2879,8 @@ static int kfd_mmio_mmap(struct kfd_dev *dev, struct kfd_process *process, address = dev->adev->rmmio_remap.bus_addr; - vma->vm_flags |= VM_IO | VM_DONTCOPY | VM_DONTEXPAND | VM_NORESERVE | - VM_DONTDUMP | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_DONTCOPY | VM_DONTEXPAND | VM_NORESERVE | + VM_DONTDUMP | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c b/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c index cd4e61bf0493..6cbe47cf9be5 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c @@ -159,8 +159,8 @@ int kfd_doorbell_mmap(struct kfd_dev *dev, struct kfd_process *process, address = kfd_get_process_doorbells(pdd); if (!address) return -ENOMEM; - vma->vm_flags |= VM_IO | VM_DONTCOPY | VM_DONTEXPAND | VM_NORESERVE | - VM_DONTDUMP | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_DONTCOPY | VM_DONTEXPAND | VM_NORESERVE | + VM_DONTDUMP | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_events.c index 729d26d648af..95cd20056cea 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_events.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_events.c @@ -1052,8 +1052,8 @@ int kfd_event_mmap(struct kfd_process *p, struct vm_area_struct *vma) pfn = __pa(page->kernel_address); pfn >>= PAGE_SHIFT; - vma->vm_flags |= VM_IO | VM_DONTCOPY | VM_DONTEXPAND | VM_NORESERVE - | VM_DONTDUMP | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_DONTCOPY | VM_DONTEXPAND | VM_NORESERVE + | VM_DONTDUMP | VM_PFNMAP); pr_debug("Mapping signal page\n"); pr_debug(" start user address == 0x%08lx\n", vma->vm_start); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c index 51b1683ac5c1..b40f4b122918 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c @@ -1978,8 +1978,8 @@ int kfd_reserved_mem_mmap(struct kfd_dev *dev, struct kfd_process *process, return -ENOMEM; } - vma->vm_flags |= VM_IO | VM_DONTCOPY | VM_DONTEXPAND - | VM_NORESERVE | VM_DONTDUMP | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_DONTCOPY | VM_DONTEXPAND + | VM_NORESERVE | VM_DONTDUMP | VM_PFNMAP); /* Mapping pages to user process */ return remap_pfn_range(vma, vma->vm_start, PFN_DOWN(__pa(qpd->cwsr_kaddr)), diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c index b8db675e7fb5..6ea7bcaa592b 100644 --- a/drivers/gpu/drm/drm_gem.c +++ b/drivers/gpu/drm/drm_gem.c @@ -1047,7 +1047,7 @@ int drm_gem_mmap_obj(struct drm_gem_object *obj, unsigned long obj_size, goto err_drm_gem_object_put; } - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_page_prot = pgprot_writecombine(vm_get_page_prot(vma->vm_flags)); vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot); } diff --git a/drivers/gpu/drm/drm_gem_dma_helper.c b/drivers/gpu/drm/drm_gem_dma_helper.c index 1e658c448366..41f241b9a581 100644 --- a/drivers/gpu/drm/drm_gem_dma_helper.c +++ b/drivers/gpu/drm/drm_gem_dma_helper.c @@ -530,8 +530,7 @@ int drm_gem_dma_mmap(struct drm_gem_dma_object *dma_obj, struct vm_area_struct * * the whole buffer. */ vma->vm_pgoff -= drm_vma_node_start(&obj->vma_node); - vma->vm_flags &= ~VM_PFNMAP; - vma->vm_flags |= VM_DONTEXPAND; + mod_vm_flags(vma, VM_DONTEXPAND, VM_PFNMAP); if (dma_obj->map_noncoherent) { vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c b/drivers/gpu/drm/drm_gem_shmem_helper.c index b602cd72a120..a5032dfac492 100644 --- a/drivers/gpu/drm/drm_gem_shmem_helper.c +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c @@ -633,7 +633,7 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_struct if (ret) return ret; - vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); if (shmem->map_wc) vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot); diff --git a/drivers/gpu/drm/drm_vm.c b/drivers/gpu/drm/drm_vm.c index f024dc93939e..8867bb6c40e3 100644 --- a/drivers/gpu/drm/drm_vm.c +++ b/drivers/gpu/drm/drm_vm.c @@ -476,7 +476,7 @@ static int drm_mmap_dma(struct file *filp, struct vm_area_struct *vma) if (!capable(CAP_SYS_ADMIN) && (dma->flags & _DRM_DMA_USE_PCI_RO)) { - vma->vm_flags &= ~(VM_WRITE | VM_MAYWRITE); + clear_vm_flags(vma, VM_WRITE | VM_MAYWRITE); #if defined(__i386__) || defined(__x86_64__) pgprot_val(vma->vm_page_prot) &= ~_PAGE_RW; #else @@ -492,7 +492,7 @@ static int drm_mmap_dma(struct file *filp, struct vm_area_struct *vma) vma->vm_ops = &drm_vm_dma_ops; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); drm_vm_open_locked(dev, vma); return 0; @@ -560,7 +560,7 @@ static int drm_mmap_locked(struct file *filp, struct vm_area_struct *vma) return -EINVAL; if (!capable(CAP_SYS_ADMIN) && (map->flags & _DRM_READ_ONLY)) { - vma->vm_flags &= ~(VM_WRITE | VM_MAYWRITE); + clear_vm_flags(vma, VM_WRITE | VM_MAYWRITE); #if defined(__i386__) || defined(__x86_64__) pgprot_val(vma->vm_page_prot) &= ~_PAGE_RW; #else @@ -628,7 +628,7 @@ static int drm_mmap_locked(struct file *filp, struct vm_area_struct *vma) default: return -EINVAL; /* This should never happen. */ } - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); drm_vm_open_locked(dev, vma); return 0; diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem.c b/drivers/gpu/drm/etnaviv/etnaviv_gem.c index c5ae5492e1af..9a5a317038a4 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gem.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem.c @@ -130,7 +130,7 @@ static int etnaviv_gem_mmap_obj(struct etnaviv_gem_object *etnaviv_obj, { pgprot_t vm_page_prot; - vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); vm_page_prot = vm_get_page_prot(vma->vm_flags); diff --git a/drivers/gpu/drm/exynos/exynos_drm_gem.c b/drivers/gpu/drm/exynos/exynos_drm_gem.c index 3e493f48e0d4..c330d415729c 100644 --- a/drivers/gpu/drm/exynos/exynos_drm_gem.c +++ b/drivers/gpu/drm/exynos/exynos_drm_gem.c @@ -274,7 +274,7 @@ static int exynos_drm_gem_mmap_buffer(struct exynos_drm_gem *exynos_gem, unsigned long vm_size; int ret; - vma->vm_flags &= ~VM_PFNMAP; + clear_vm_flags(vma, VM_PFNMAP); vma->vm_pgoff = 0; vm_size = vma->vm_end - vma->vm_start; @@ -368,7 +368,7 @@ static int exynos_drm_gem_mmap(struct drm_gem_object *obj, struct vm_area_struct if (obj->import_attach) return dma_buf_mmap(obj->dma_buf, vma, 0); - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); DRM_DEV_DEBUG_KMS(to_dma_dev(obj->dev), "flags = 0x%x\n", exynos_gem->flags); diff --git a/drivers/gpu/drm/gma500/framebuffer.c b/drivers/gpu/drm/gma500/framebuffer.c index 8d5a37b8f110..471d5b3c1535 100644 --- a/drivers/gpu/drm/gma500/framebuffer.c +++ b/drivers/gpu/drm/gma500/framebuffer.c @@ -139,7 +139,7 @@ static int psbfb_mmap(struct fb_info *info, struct vm_area_struct *vma) */ vma->vm_ops = &psbfb_vm_ops; vma->vm_private_data = (void *)fb; - vma->vm_flags |= VM_IO | VM_MIXEDMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_MIXEDMAP | VM_DONTEXPAND | VM_DONTDUMP); return 0; } diff --git a/drivers/gpu/drm/i810/i810_dma.c b/drivers/gpu/drm/i810/i810_dma.c index 9fb4dd63342f..bced8c30709e 100644 --- a/drivers/gpu/drm/i810/i810_dma.c +++ b/drivers/gpu/drm/i810/i810_dma.c @@ -102,7 +102,7 @@ static int i810_mmap_buffers(struct file *filp, struct vm_area_struct *vma) buf = dev_priv->mmap_buffer; buf_priv = buf->dev_private; - vma->vm_flags |= VM_DONTCOPY; + set_vm_flags(vma, VM_DONTCOPY); buf_priv->currently_mapped = I810_BUF_MAPPED; diff --git a/drivers/gpu/drm/i915/gem/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/i915_gem_mman.c index 0ad44f3868de..71b9e0485cb9 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c @@ -979,7 +979,7 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma) i915_gem_object_put(obj); return -EINVAL; } - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); } anon = mmap_singleton(to_i915(dev)); @@ -988,7 +988,7 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma) return PTR_ERR(anon); } - vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | VM_IO; + set_vm_flags(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | VM_IO); /* * We keep the ref on mmo->obj, not vm_file, but we require diff --git a/drivers/gpu/drm/mediatek/mtk_drm_gem.c b/drivers/gpu/drm/mediatek/mtk_drm_gem.c index 47e96b0289f9..427089733b87 100644 --- a/drivers/gpu/drm/mediatek/mtk_drm_gem.c +++ b/drivers/gpu/drm/mediatek/mtk_drm_gem.c @@ -158,7 +158,7 @@ static int mtk_drm_gem_object_mmap(struct drm_gem_object *obj, * dma_alloc_attrs() allocated a struct page table for mtk_gem, so clear * VM_PFNMAP flag that was set by drm_gem_mmap_obj()/drm_gem_mmap(). */ - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_page_prot = pgprot_writecombine(vm_get_page_prot(vma->vm_flags)); vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot); diff --git a/drivers/gpu/drm/msm/msm_gem.c b/drivers/gpu/drm/msm/msm_gem.c index 1dee0d18abbb..8aff3ae909af 100644 --- a/drivers/gpu/drm/msm/msm_gem.c +++ b/drivers/gpu/drm/msm/msm_gem.c @@ -1012,7 +1012,7 @@ static int msm_gem_object_mmap(struct drm_gem_object *obj, struct vm_area_struct { struct msm_gem_object *msm_obj = to_msm_bo(obj); - vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_page_prot = msm_gem_pgprot(msm_obj, vm_get_page_prot(vma->vm_flags)); return 0; diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c index cf571796fd26..9c0e7d6a3784 100644 --- a/drivers/gpu/drm/omapdrm/omap_gem.c +++ b/drivers/gpu/drm/omapdrm/omap_gem.c @@ -543,8 +543,7 @@ int omap_gem_mmap_obj(struct drm_gem_object *obj, { struct omap_gem_object *omap_obj = to_omap_bo(obj); - vma->vm_flags &= ~VM_PFNMAP; - vma->vm_flags |= VM_MIXEDMAP; + mod_vm_flags(vma, VM_MIXEDMAP, VM_PFNMAP); if (omap_obj->flags & OMAP_BO_WC) { vma->vm_page_prot = pgprot_writecombine(vm_get_page_prot(vma->vm_flags)); diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_gem.c b/drivers/gpu/drm/rockchip/rockchip_drm_gem.c index 6edb7c52cb3d..735b64bbdcf2 100644 --- a/drivers/gpu/drm/rockchip/rockchip_drm_gem.c +++ b/drivers/gpu/drm/rockchip/rockchip_drm_gem.c @@ -251,8 +251,7 @@ static int rockchip_drm_gem_object_mmap(struct drm_gem_object *obj, * We allocated a struct page table for rk_obj, so clear * VM_PFNMAP flag that was set by drm_gem_mmap_obj()/drm_gem_mmap(). */ - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; - vma->vm_flags &= ~VM_PFNMAP; + mod_vm_flags(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP, VM_PFNMAP); vma->vm_page_prot = pgprot_writecombine(vm_get_page_prot(vma->vm_flags)); vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot); diff --git a/drivers/gpu/drm/tegra/gem.c b/drivers/gpu/drm/tegra/gem.c index 979e7bc902f6..6cdc6c45ef27 100644 --- a/drivers/gpu/drm/tegra/gem.c +++ b/drivers/gpu/drm/tegra/gem.c @@ -574,7 +574,7 @@ int __tegra_gem_mmap(struct drm_gem_object *gem, struct vm_area_struct *vma) * and set the vm_pgoff (used as a fake buffer offset by DRM) * to 0 as we want to map the whole buffer. */ - vma->vm_flags &= ~VM_PFNMAP; + clear_vm_flags(vma, VM_PFNMAP); vma->vm_pgoff = 0; err = dma_mmap_wc(gem->dev->dev, vma, bo->vaddr, bo->iova, @@ -588,8 +588,7 @@ int __tegra_gem_mmap(struct drm_gem_object *gem, struct vm_area_struct *vma) } else { pgprot_t prot = vm_get_page_prot(vma->vm_flags); - vma->vm_flags |= VM_MIXEDMAP; - vma->vm_flags &= ~VM_PFNMAP; + mod_vm_flags(vma, VM_MIXEDMAP, VM_PFNMAP); vma->vm_page_prot = pgprot_writecombine(prot); } diff --git a/drivers/gpu/drm/ttm/ttm_bo_vm.c b/drivers/gpu/drm/ttm/ttm_bo_vm.c index 5a3e4b891377..0861e6e33964 100644 --- a/drivers/gpu/drm/ttm/ttm_bo_vm.c +++ b/drivers/gpu/drm/ttm/ttm_bo_vm.c @@ -468,8 +468,7 @@ int ttm_bo_mmap_obj(struct vm_area_struct *vma, struct ttm_buffer_object *bo) vma->vm_private_data = bo; - vma->vm_flags |= VM_PFNMAP; - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_PFNMAP | VM_IO | VM_DONTEXPAND | VM_DONTDUMP); return 0; } EXPORT_SYMBOL(ttm_bo_mmap_obj); diff --git a/drivers/gpu/drm/virtio/virtgpu_vram.c b/drivers/gpu/drm/virtio/virtgpu_vram.c index 6b45b0429fef..5498a1dbef63 100644 --- a/drivers/gpu/drm/virtio/virtgpu_vram.c +++ b/drivers/gpu/drm/virtio/virtgpu_vram.c @@ -46,7 +46,7 @@ static int virtio_gpu_vram_mmap(struct drm_gem_object *obj, return -EINVAL; vma->vm_pgoff -= drm_vma_node_start(&obj->vma_node); - vma->vm_flags |= VM_MIXEDMAP | VM_DONTEXPAND; + set_vm_flags(vma, VM_MIXEDMAP | VM_DONTEXPAND); vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot); vma->vm_ops = &virtio_gpu_vram_vm_ops; diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_ttm_glue.c b/drivers/gpu/drm/vmwgfx/vmwgfx_ttm_glue.c index 265f7c48d856..8c8015528b6f 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_ttm_glue.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_ttm_glue.c @@ -97,7 +97,7 @@ int vmw_mmap(struct file *filp, struct vm_area_struct *vma) /* Use VM_PFNMAP rather than VM_MIXEDMAP if not a COW mapping */ if (!is_cow_mapping(vma->vm_flags)) - vma->vm_flags = (vma->vm_flags & ~VM_MIXEDMAP) | VM_PFNMAP; + mod_vm_flags(vma, VM_PFNMAP, VM_MIXEDMAP); ttm_bo_put(bo); /* release extra ref taken by ttm_bo_mmap_obj() */ diff --git a/drivers/gpu/drm/xen/xen_drm_front_gem.c b/drivers/gpu/drm/xen/xen_drm_front_gem.c index 4c95ebcdcc2d..18a93ad4aa1f 100644 --- a/drivers/gpu/drm/xen/xen_drm_front_gem.c +++ b/drivers/gpu/drm/xen/xen_drm_front_gem.c @@ -69,8 +69,7 @@ static int xen_drm_front_gem_object_mmap(struct drm_gem_object *gem_obj, * vm_pgoff (used as a fake buffer offset by DRM) to 0 as we want to map * the whole buffer. */ - vma->vm_flags &= ~VM_PFNMAP; - vma->vm_flags |= VM_MIXEDMAP | VM_DONTEXPAND; + mod_vm_flags(vma, VM_MIXEDMAP | VM_DONTEXPAND, VM_PFNMAP); vma->vm_pgoff = 0; /* diff --git a/drivers/hsi/clients/cmt_speech.c b/drivers/hsi/clients/cmt_speech.c index 8069f795c864..952a31e742a1 100644 --- a/drivers/hsi/clients/cmt_speech.c +++ b/drivers/hsi/clients/cmt_speech.c @@ -1264,7 +1264,7 @@ static int cs_char_mmap(struct file *file, struct vm_area_struct *vma) if (vma_pages(vma) != 1) return -EINVAL; - vma->vm_flags |= VM_IO | VM_DONTDUMP | VM_DONTEXPAND; + set_vm_flags(vma, VM_IO | VM_DONTDUMP | VM_DONTEXPAND); vma->vm_ops = &cs_char_vm_ops; vma->vm_private_data = file->private_data; diff --git a/drivers/hwtracing/intel_th/msu.c b/drivers/hwtracing/intel_th/msu.c index 6c8215a47a60..a6f178bf3ded 100644 --- a/drivers/hwtracing/intel_th/msu.c +++ b/drivers/hwtracing/intel_th/msu.c @@ -1659,7 +1659,7 @@ static int intel_th_msc_mmap(struct file *file, struct vm_area_struct *vma) atomic_dec(&msc->user_count); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); - vma->vm_flags |= VM_DONTEXPAND | VM_DONTCOPY; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTCOPY); vma->vm_ops = &msc_mmap_ops; return ret; } diff --git a/drivers/hwtracing/stm/core.c b/drivers/hwtracing/stm/core.c index 2712e699ba08..9a59e61c4194 100644 --- a/drivers/hwtracing/stm/core.c +++ b/drivers/hwtracing/stm/core.c @@ -715,7 +715,7 @@ static int stm_char_mmap(struct file *file, struct vm_area_struct *vma) pm_runtime_get_sync(&stm->dev); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &stm_mmap_vmops; vm_iomap_memory(vma, phys, size); diff --git a/drivers/infiniband/hw/hfi1/file_ops.c b/drivers/infiniband/hw/hfi1/file_ops.c index f5f9269fdc16..7294f2d33bc6 100644 --- a/drivers/infiniband/hw/hfi1/file_ops.c +++ b/drivers/infiniband/hw/hfi1/file_ops.c @@ -403,7 +403,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) ret = -EPERM; goto done; } - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); addr = vma->vm_start; for (i = 0 ; i < uctxt->egrbufs.numbufs; i++) { memlen = uctxt->egrbufs.buffers[i].len; @@ -528,7 +528,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) goto done; } - vma->vm_flags = flags; + reset_vm_flags(vma, flags); hfi1_cdbg(PROC, "%u:%u type:%u io/vf:%d/%d, addr:0x%llx, len:%lu(%lu), flags:0x%lx\n", ctxt, subctxt, type, mapio, vmf, memaddr, memlen, diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index c669ef6e47e7..538318c809b3 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -2087,7 +2087,7 @@ static int mlx5_ib_mmap_clock_info_page(struct mlx5_ib_dev *dev, if (vma->vm_flags & (VM_WRITE | VM_EXEC)) return -EPERM; - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); if (!dev->mdev->clock_info) return -EOPNOTSUPP; @@ -2311,7 +2311,7 @@ static int mlx5_ib_mmap(struct ib_ucontext *ibcontext, struct vm_area_struct *vm if (vma->vm_flags & VM_WRITE) return -EPERM; - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); /* Don't expose to user-space information it shouldn't have */ if (PAGE_SIZE > 4096) diff --git a/drivers/infiniband/hw/qib/qib_file_ops.c b/drivers/infiniband/hw/qib/qib_file_ops.c index 3937144b2ae5..16ef80df4b7f 100644 --- a/drivers/infiniband/hw/qib/qib_file_ops.c +++ b/drivers/infiniband/hw/qib/qib_file_ops.c @@ -733,7 +733,7 @@ static int qib_mmap_mem(struct vm_area_struct *vma, struct qib_ctxtdata *rcd, } /* don't allow them to later change with mprotect */ - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); } pfn = virt_to_phys(kvaddr) >> PAGE_SHIFT; @@ -769,7 +769,7 @@ static int mmap_ureg(struct vm_area_struct *vma, struct qib_devdata *dd, phys = dd->physaddr + ureg; vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); - vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTCOPY | VM_DONTEXPAND); ret = io_remap_pfn_range(vma, vma->vm_start, phys >> PAGE_SHIFT, vma->vm_end - vma->vm_start, @@ -810,8 +810,7 @@ static int mmap_piobufs(struct vm_area_struct *vma, * don't allow them to later change to readable with mprotect (for when * not initially mapped readable, as is normally the case) */ - vma->vm_flags &= ~VM_MAYREAD; - vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND; + mod_vm_flags(vma, VM_DONTCOPY | VM_DONTEXPAND, VM_MAYREAD); /* We used PAT if wc_cookie == 0 */ if (!dd->wc_cookie) @@ -852,7 +851,7 @@ static int mmap_rcvegrbufs(struct vm_area_struct *vma, goto bail; } /* don't allow them to later change to writable with mprotect */ - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); start = vma->vm_start; @@ -944,7 +943,7 @@ static int mmap_kvaddr(struct vm_area_struct *vma, u64 pgaddr, * Don't allow permission to later change to writable * with mprotect. */ - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); } else goto bail; len = vma->vm_end - vma->vm_start; @@ -955,7 +954,7 @@ static int mmap_kvaddr(struct vm_area_struct *vma, u64 pgaddr, vma->vm_pgoff = (unsigned long) addr >> PAGE_SHIFT; vma->vm_ops = &qib_file_vm_ops; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); ret = 1; bail: diff --git a/drivers/infiniband/hw/usnic/usnic_ib_verbs.c b/drivers/infiniband/hw/usnic/usnic_ib_verbs.c index 6e8c4fbb8083..6f9237c2a26b 100644 --- a/drivers/infiniband/hw/usnic/usnic_ib_verbs.c +++ b/drivers/infiniband/hw/usnic/usnic_ib_verbs.c @@ -672,7 +672,7 @@ int usnic_ib_mmap(struct ib_ucontext *context, usnic_dbg("\n"); us_ibdev = to_usdev(context->device); - vma->vm_flags |= VM_IO; + set_vm_flags(vma, VM_IO); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vfid = vma->vm_pgoff; usnic_dbg("Page Offset %lu PAGE_SHIFT %u VFID %u\n", diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c index 19176583dbde..7f1b7b5dd3f4 100644 --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c @@ -408,7 +408,7 @@ int pvrdma_mmap(struct ib_ucontext *ibcontext, struct vm_area_struct *vma) } /* Map UAR to kernel space, VM_LOCKED? */ - vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTCOPY | VM_DONTEXPAND); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); if (io_remap_pfn_range(vma, start, context->uar.pfn, size, vma->vm_page_prot)) diff --git a/drivers/media/common/videobuf2/videobuf2-dma-contig.c b/drivers/media/common/videobuf2/videobuf2-dma-contig.c index 5f1175f8b349..e66ae399749e 100644 --- a/drivers/media/common/videobuf2/videobuf2-dma-contig.c +++ b/drivers/media/common/videobuf2/videobuf2-dma-contig.c @@ -293,7 +293,7 @@ static int vb2_dc_mmap(void *buf_priv, struct vm_area_struct *vma) return ret; } - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_private_data = &buf->handler; vma->vm_ops = &vb2_common_vm_ops; diff --git a/drivers/media/common/videobuf2/videobuf2-vmalloc.c b/drivers/media/common/videobuf2/videobuf2-vmalloc.c index 959b45beb1f3..edb47240ec17 100644 --- a/drivers/media/common/videobuf2/videobuf2-vmalloc.c +++ b/drivers/media/common/videobuf2/videobuf2-vmalloc.c @@ -185,7 +185,7 @@ static int vb2_vmalloc_mmap(void *buf_priv, struct vm_area_struct *vma) /* * Make sure that vm_areas for 2 buffers won't be merged together */ - vma->vm_flags |= VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTEXPAND); /* * Use common vm_area operations to track buffer refcount. diff --git a/drivers/media/v4l2-core/videobuf-dma-contig.c b/drivers/media/v4l2-core/videobuf-dma-contig.c index f2c439359557..c030823185ba 100644 --- a/drivers/media/v4l2-core/videobuf-dma-contig.c +++ b/drivers/media/v4l2-core/videobuf-dma-contig.c @@ -314,7 +314,7 @@ static int __videobuf_mmap_mapper(struct videobuf_queue *q, } vma->vm_ops = &videobuf_vm_ops; - vma->vm_flags |= VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTEXPAND); vma->vm_private_data = map; dev_dbg(q->dev, "mmap %p: q=%p %08lx-%08lx (%lx) pgoff %08lx buf %d\n", diff --git a/drivers/media/v4l2-core/videobuf-dma-sg.c b/drivers/media/v4l2-core/videobuf-dma-sg.c index 234e9f647c96..9adac4875f29 100644 --- a/drivers/media/v4l2-core/videobuf-dma-sg.c +++ b/drivers/media/v4l2-core/videobuf-dma-sg.c @@ -630,8 +630,8 @@ static int __videobuf_mmap_mapper(struct videobuf_queue *q, map->count = 1; map->q = q; vma->vm_ops = &videobuf_vm_ops; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; - vma->vm_flags &= ~VM_IO; /* using shared anonymous pages */ + /* using shared anonymous pages */ + mod_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP, VM_IO); vma->vm_private_data = map; dprintk(1, "mmap %p: q=%p %08lx-%08lx pgoff %08lx bufs %d-%d\n", map, q, vma->vm_start, vma->vm_end, vma->vm_pgoff, first, last); diff --git a/drivers/media/v4l2-core/videobuf-vmalloc.c b/drivers/media/v4l2-core/videobuf-vmalloc.c index 9b2443720ab0..48d439ccd414 100644 --- a/drivers/media/v4l2-core/videobuf-vmalloc.c +++ b/drivers/media/v4l2-core/videobuf-vmalloc.c @@ -247,7 +247,7 @@ static int __videobuf_mmap_mapper(struct videobuf_queue *q, } vma->vm_ops = &videobuf_vm_ops; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_private_data = map; dprintk(1, "mmap %p: q=%p %08lx-%08lx (%lx) pgoff %08lx buf %d\n", diff --git a/drivers/misc/cxl/context.c b/drivers/misc/cxl/context.c index acaa44809c58..17562e4efcb2 100644 --- a/drivers/misc/cxl/context.c +++ b/drivers/misc/cxl/context.c @@ -220,7 +220,7 @@ int cxl_context_iomap(struct cxl_context *ctx, struct vm_area_struct *vma) pr_devel("%s: mmio physical: %llx pe: %i master:%i\n", __func__, ctx->psn_phys, ctx->pe , ctx->master); - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &cxl_mmap_vmops; return 0; diff --git a/drivers/misc/habanalabs/common/memory.c b/drivers/misc/habanalabs/common/memory.c index 5e9ae7600d75..ad8eae764b9b 100644 --- a/drivers/misc/habanalabs/common/memory.c +++ b/drivers/misc/habanalabs/common/memory.c @@ -2082,7 +2082,7 @@ static int hl_ts_mmap(struct hl_mmap_mem_buf *buf, struct vm_area_struct *vma, v { struct hl_ts_buff *ts_buff = buf->private; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP | VM_DONTCOPY | VM_NORESERVE; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP | VM_DONTCOPY | VM_NORESERVE); return remap_vmalloc_range(vma, ts_buff->user_buff_address, 0); } diff --git a/drivers/misc/habanalabs/gaudi/gaudi.c b/drivers/misc/habanalabs/gaudi/gaudi.c index 9f5e208701ba..4186f04da224 100644 --- a/drivers/misc/habanalabs/gaudi/gaudi.c +++ b/drivers/misc/habanalabs/gaudi/gaudi.c @@ -4236,8 +4236,8 @@ static int gaudi_mmap(struct hl_device *hdev, struct vm_area_struct *vma, { int rc; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | - VM_DONTCOPY | VM_NORESERVE; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | + VM_DONTCOPY | VM_NORESERVE); rc = dma_mmap_coherent(hdev->dev, vma, cpu_addr, (dma_addr - HOST_PHYS_BASE), size); diff --git a/drivers/misc/habanalabs/gaudi2/gaudi2.c b/drivers/misc/habanalabs/gaudi2/gaudi2.c index e793fb2bdcbe..7311c3053944 100644 --- a/drivers/misc/habanalabs/gaudi2/gaudi2.c +++ b/drivers/misc/habanalabs/gaudi2/gaudi2.c @@ -5538,8 +5538,8 @@ static int gaudi2_mmap(struct hl_device *hdev, struct vm_area_struct *vma, { int rc; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | - VM_DONTCOPY | VM_NORESERVE; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | + VM_DONTCOPY | VM_NORESERVE); #ifdef _HAS_DMA_MMAP_COHERENT @@ -10116,8 +10116,8 @@ static int gaudi2_block_mmap(struct hl_device *hdev, struct vm_area_struct *vma, address = pci_resource_start(hdev->pdev, SRAM_CFG_BAR_ID) + offset_in_bar; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | - VM_DONTCOPY | VM_NORESERVE; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | + VM_DONTCOPY | VM_NORESERVE); rc = remap_pfn_range(vma, vma->vm_start, address >> PAGE_SHIFT, block_size, vma->vm_page_prot); diff --git a/drivers/misc/habanalabs/goya/goya.c b/drivers/misc/habanalabs/goya/goya.c index 0f083fcf81a6..5e2aaa26ea29 100644 --- a/drivers/misc/habanalabs/goya/goya.c +++ b/drivers/misc/habanalabs/goya/goya.c @@ -2880,8 +2880,8 @@ static int goya_mmap(struct hl_device *hdev, struct vm_area_struct *vma, { int rc; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | - VM_DONTCOPY | VM_NORESERVE; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | + VM_DONTCOPY | VM_NORESERVE); rc = dma_mmap_coherent(hdev->dev, vma, cpu_addr, (dma_addr - HOST_PHYS_BASE), size); diff --git a/drivers/misc/ocxl/context.c b/drivers/misc/ocxl/context.c index 9eb0d93b01c6..e6f941248e93 100644 --- a/drivers/misc/ocxl/context.c +++ b/drivers/misc/ocxl/context.c @@ -180,7 +180,7 @@ static int check_mmap_afu_irq(struct ocxl_context *ctx, if ((vma->vm_flags & VM_READ) || (vma->vm_flags & VM_EXEC) || !(vma->vm_flags & VM_WRITE)) return -EINVAL; - vma->vm_flags &= ~(VM_MAYREAD | VM_MAYEXEC); + clear_vm_flags(vma, VM_MAYREAD | VM_MAYEXEC); return 0; } @@ -204,7 +204,7 @@ int ocxl_context_mmap(struct ocxl_context *ctx, struct vm_area_struct *vma) if (rc) return rc; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &ocxl_vmops; return 0; diff --git a/drivers/misc/ocxl/sysfs.c b/drivers/misc/ocxl/sysfs.c index 25c78df8055d..9398246cac79 100644 --- a/drivers/misc/ocxl/sysfs.c +++ b/drivers/misc/ocxl/sysfs.c @@ -134,7 +134,7 @@ static int global_mmio_mmap(struct file *filp, struct kobject *kobj, (afu->config.global_mmio_size >> PAGE_SHIFT)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &global_mmio_vmops; vma->vm_private_data = afu; diff --git a/drivers/misc/open-dice.c b/drivers/misc/open-dice.c index c61be3404c6f..9f9438b5b075 100644 --- a/drivers/misc/open-dice.c +++ b/drivers/misc/open-dice.c @@ -96,13 +96,13 @@ static int open_dice_mmap(struct file *filp, struct vm_area_struct *vma) /* Ensure userspace cannot acquire VM_WRITE + VM_SHARED later. */ if (vma->vm_flags & VM_WRITE) - vma->vm_flags &= ~VM_MAYSHARE; + clear_vm_flags(vma, VM_MAYSHARE); else if (vma->vm_flags & VM_SHARED) - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); /* Create write-combine mapping so all clients observe a wipe. */ vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot); - vma->vm_flags |= VM_DONTCOPY | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTCOPY | VM_DONTDUMP); return vm_iomap_memory(vma, drvdata->rmem->base, drvdata->rmem->size); } diff --git a/drivers/misc/sgi-gru/grufile.c b/drivers/misc/sgi-gru/grufile.c index 7ffcfc0bb587..8b777286d3b2 100644 --- a/drivers/misc/sgi-gru/grufile.c +++ b/drivers/misc/sgi-gru/grufile.c @@ -101,8 +101,8 @@ static int gru_file_mmap(struct file *file, struct vm_area_struct *vma) vma->vm_end & (GRU_GSEG_PAGESIZE - 1)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_LOCKED | - VM_DONTCOPY | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_LOCKED | + VM_DONTCOPY | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_page_prot = PAGE_SHARED; vma->vm_ops = &gru_vm_ops; diff --git a/drivers/misc/uacce/uacce.c b/drivers/misc/uacce/uacce.c index 905eff1f840e..f57e91cdb0f6 100644 --- a/drivers/misc/uacce/uacce.c +++ b/drivers/misc/uacce/uacce.c @@ -229,7 +229,7 @@ static int uacce_fops_mmap(struct file *filep, struct vm_area_struct *vma) if (!qfr) return -ENOMEM; - vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND | VM_WIPEONFORK; + set_vm_flags(vma, VM_DONTCOPY | VM_DONTEXPAND | VM_WIPEONFORK); vma->vm_ops = &uacce_vm_ops; vma->vm_private_data = q; qfr->type = type; diff --git a/drivers/sbus/char/oradax.c b/drivers/sbus/char/oradax.c index 21b7cb6e7e70..a096734daad0 100644 --- a/drivers/sbus/char/oradax.c +++ b/drivers/sbus/char/oradax.c @@ -389,7 +389,7 @@ static int dax_devmap(struct file *f, struct vm_area_struct *vma) /* completion area is mapped read-only for user */ if (vma->vm_flags & VM_WRITE) return -EPERM; - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); if (remap_pfn_range(vma, vma->vm_start, ctx->ca_buf_ra >> PAGE_SHIFT, len, vma->vm_page_prot)) diff --git a/drivers/scsi/cxlflash/ocxl_hw.c b/drivers/scsi/cxlflash/ocxl_hw.c index 631eda2d467e..d386c25c2699 100644 --- a/drivers/scsi/cxlflash/ocxl_hw.c +++ b/drivers/scsi/cxlflash/ocxl_hw.c @@ -1167,7 +1167,7 @@ static int afu_mmap(struct file *file, struct vm_area_struct *vma) (ctx->psn_size >> PAGE_SHIFT)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &ocxlflash_vmops; return 0; diff --git a/drivers/scsi/sg.c b/drivers/scsi/sg.c index ff9854f59964..7438adfe3bdc 100644 --- a/drivers/scsi/sg.c +++ b/drivers/scsi/sg.c @@ -1288,7 +1288,7 @@ sg_mmap(struct file *filp, struct vm_area_struct *vma) } sfp->mmap_called = 1; - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_private_data = sfp; vma->vm_ops = &sg_mmap_vm_ops; out: diff --git a/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c b/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c index 5e53eed8ae95..df1c944e5058 100644 --- a/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c +++ b/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c @@ -1072,7 +1072,7 @@ int hmm_bo_mmap(struct vm_area_struct *vma, struct hmm_buffer_object *bo) vma->vm_private_data = bo; vma->vm_ops = &hmm_bo_vm_ops; - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); /* * call hmm_bo_vm_open explicitly. diff --git a/drivers/staging/media/deprecated/meye/meye.c b/drivers/staging/media/deprecated/meye/meye.c index 5d87efd9b95c..2505e64d7119 100644 --- a/drivers/staging/media/deprecated/meye/meye.c +++ b/drivers/staging/media/deprecated/meye/meye.c @@ -1476,8 +1476,8 @@ static int meye_mmap(struct file *file, struct vm_area_struct *vma) } vma->vm_ops = &meye_vm_ops; - vma->vm_flags &= ~VM_IO; /* not I/O memory */ - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + /* not I/O memory */ + mod_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP, VM_IO); vma->vm_private_data = (void *) (offset / gbufsize); meye_vm_open(vma); diff --git a/drivers/staging/media/deprecated/stkwebcam/stk-webcam.c b/drivers/staging/media/deprecated/stkwebcam/stk-webcam.c index 787edb3d47c2..196d1034f104 100644 --- a/drivers/staging/media/deprecated/stkwebcam/stk-webcam.c +++ b/drivers/staging/media/deprecated/stkwebcam/stk-webcam.c @@ -779,7 +779,7 @@ static int v4l_stk_mmap(struct file *fp, struct vm_area_struct *vma) ret = remap_vmalloc_range(vma, sbuf->buffer, 0); if (ret) return ret; - vma->vm_flags |= VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTEXPAND); vma->vm_private_data = sbuf; vma->vm_ops = &stk_v4l_vm_ops; sbuf->v4lbuf.flags |= V4L2_BUF_FLAG_MAPPED; diff --git a/drivers/target/target_core_user.c b/drivers/target/target_core_user.c index 2940559c3086..9fd64259904c 100644 --- a/drivers/target/target_core_user.c +++ b/drivers/target/target_core_user.c @@ -1928,7 +1928,7 @@ static int tcmu_mmap(struct uio_info *info, struct vm_area_struct *vma) { struct tcmu_dev *udev = container_of(info, struct tcmu_dev, uio_info); - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &tcmu_vm_ops; vma->vm_private_data = udev; diff --git a/drivers/uio/uio.c b/drivers/uio/uio.c index 43afbb7c5ab9..08802744f3b7 100644 --- a/drivers/uio/uio.c +++ b/drivers/uio/uio.c @@ -713,7 +713,7 @@ static const struct vm_operations_struct uio_logical_vm_ops = { static int uio_mmap_logical(struct vm_area_struct *vma) { - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &uio_logical_vm_ops; return 0; } diff --git a/drivers/usb/core/devio.c b/drivers/usb/core/devio.c index 837f3e57f580..d9aefa259883 100644 --- a/drivers/usb/core/devio.c +++ b/drivers/usb/core/devio.c @@ -279,8 +279,7 @@ static int usbdev_mmap(struct file *file, struct vm_area_struct *vma) } } - vma->vm_flags |= VM_IO; - vma->vm_flags |= (VM_DONTEXPAND | VM_DONTDUMP); + set_vm_flags(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &usbdev_vm_ops; vma->vm_private_data = usbm; diff --git a/drivers/usb/mon/mon_bin.c b/drivers/usb/mon/mon_bin.c index 094e812e9e69..9b2d48a65fdf 100644 --- a/drivers/usb/mon/mon_bin.c +++ b/drivers/usb/mon/mon_bin.c @@ -1272,8 +1272,7 @@ static int mon_bin_mmap(struct file *filp, struct vm_area_struct *vma) if (vma->vm_flags & VM_WRITE) return -EPERM; - vma->vm_flags &= ~VM_MAYWRITE; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + mod_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP, VM_MAYWRITE); vma->vm_private_data = filp->private_data; mon_bin_vma_open(vma); return 0; diff --git a/drivers/vdpa/vdpa_user/iova_domain.c b/drivers/vdpa/vdpa_user/iova_domain.c index e682bc7ee6c9..39dcce2e455b 100644 --- a/drivers/vdpa/vdpa_user/iova_domain.c +++ b/drivers/vdpa/vdpa_user/iova_domain.c @@ -512,7 +512,7 @@ static int vduse_domain_mmap(struct file *file, struct vm_area_struct *vma) { struct vduse_iova_domain *domain = file->private_data; - vma->vm_flags |= VM_DONTDUMP | VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTDUMP | VM_DONTEXPAND); vma->vm_private_data = domain; vma->vm_ops = &vduse_domain_mmap_ops; diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 26a541cc64d1..86eb3fc9ffb4 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1799,7 +1799,7 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma * See remap_pfn_range(), called from vfio_pci_fault() but we can't * change vm_flags within the fault handler. Set them now. */ - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &vfio_pci_mmap_ops; return 0; diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index ec32f785dfde..7b81994a7d02 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -1315,7 +1315,7 @@ static int vhost_vdpa_mmap(struct file *file, struct vm_area_struct *vma) if (vma->vm_end - vma->vm_start != notify.size) return -ENOTSUPP; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &vhost_vdpa_vm_ops; return 0; } diff --git a/drivers/video/fbdev/68328fb.c b/drivers/video/fbdev/68328fb.c index 7db03ed77c76..a794a740af10 100644 --- a/drivers/video/fbdev/68328fb.c +++ b/drivers/video/fbdev/68328fb.c @@ -391,7 +391,7 @@ static int mc68x328fb_mmap(struct fb_info *info, struct vm_area_struct *vma) #ifndef MMU /* this is uClinux (no MMU) specific code */ - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_start = videomemory; return 0; diff --git a/drivers/video/fbdev/core/fb_defio.c b/drivers/video/fbdev/core/fb_defio.c index c730253ab85c..af0bfaa2d014 100644 --- a/drivers/video/fbdev/core/fb_defio.c +++ b/drivers/video/fbdev/core/fb_defio.c @@ -232,9 +232,9 @@ static const struct address_space_operations fb_deferred_io_aops = { int fb_deferred_io_mmap(struct fb_info *info, struct vm_area_struct *vma) { vma->vm_ops = &fb_deferred_io_vm_ops; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); if (!(info->flags & FBINFO_VIRTFB)) - vma->vm_flags |= VM_IO; + set_vm_flags(vma, VM_IO); vma->vm_private_data = info; return 0; } diff --git a/drivers/xen/gntalloc.c b/drivers/xen/gntalloc.c index a15729beb9d1..ee4a8958dc68 100644 --- a/drivers/xen/gntalloc.c +++ b/drivers/xen/gntalloc.c @@ -525,7 +525,7 @@ static int gntalloc_mmap(struct file *filp, struct vm_area_struct *vma) vma->vm_private_data = vm_priv; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &gntalloc_vmops; diff --git a/drivers/xen/gntdev.c b/drivers/xen/gntdev.c index 4d9a3050de6a..6d5bb1ebb661 100644 --- a/drivers/xen/gntdev.c +++ b/drivers/xen/gntdev.c @@ -1055,10 +1055,10 @@ static int gntdev_mmap(struct file *flip, struct vm_area_struct *vma) vma->vm_ops = &gntdev_vmops; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP | VM_MIXEDMAP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP | VM_MIXEDMAP); if (use_ptemod) - vma->vm_flags |= VM_DONTCOPY; + set_vm_flags(vma, VM_DONTCOPY); vma->vm_private_data = map; if (map->flags) { diff --git a/drivers/xen/privcmd-buf.c b/drivers/xen/privcmd-buf.c index dd5bbb6e1b6b..037547918630 100644 --- a/drivers/xen/privcmd-buf.c +++ b/drivers/xen/privcmd-buf.c @@ -156,7 +156,7 @@ static int privcmd_buf_mmap(struct file *file, struct vm_area_struct *vma) vma_priv->file_priv = file_priv; vma_priv->users = 1; - vma->vm_flags |= VM_IO | VM_DONTEXPAND; + set_vm_flags(vma, VM_IO | VM_DONTEXPAND); vma->vm_ops = &privcmd_buf_vm_ops; vma->vm_private_data = vma_priv; diff --git a/drivers/xen/privcmd.c b/drivers/xen/privcmd.c index 1edf45ee9890..4c8cfc6f86d8 100644 --- a/drivers/xen/privcmd.c +++ b/drivers/xen/privcmd.c @@ -934,8 +934,8 @@ static int privcmd_mmap(struct file *file, struct vm_area_struct *vma) { /* DONTCOPY is essential for Xen because copy_page_range doesn't know * how to recreate these mappings */ - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTCOPY | - VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTCOPY | + VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &privcmd_vm_ops; vma->vm_private_data = NULL; diff --git a/fs/aio.c b/fs/aio.c index 562916d85cba..db821fb1e92d 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -390,7 +390,7 @@ static const struct vm_operations_struct aio_ring_vm_ops = { static int aio_ring_mmap(struct file *file, struct vm_area_struct *vma) { - vma->vm_flags |= VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTEXPAND); vma->vm_ops = &aio_ring_vm_ops; return 0; } diff --git a/fs/cramfs/inode.c b/fs/cramfs/inode.c index 61ccf7722fc3..874a17a1b8d9 100644 --- a/fs/cramfs/inode.c +++ b/fs/cramfs/inode.c @@ -408,7 +408,7 @@ static int cramfs_physmem_mmap(struct file *file, struct vm_area_struct *vma) * unpopulated ptes via cramfs_read_folio(). */ int i; - vma->vm_flags |= VM_MIXEDMAP; + set_vm_flags(vma, VM_MIXEDMAP); for (i = 0; i < pages && !ret; i++) { vm_fault_t vmf; unsigned long off = i * PAGE_SIZE; diff --git a/fs/erofs/data.c b/fs/erofs/data.c index f57f921683d7..e6413ced2bb1 100644 --- a/fs/erofs/data.c +++ b/fs/erofs/data.c @@ -429,7 +429,7 @@ static int erofs_file_mmap(struct file *file, struct vm_area_struct *vma) return -EINVAL; vma->vm_ops = &erofs_dax_vm_ops; - vma->vm_flags |= VM_HUGEPAGE; + set_vm_flags(vma, VM_HUGEPAGE); return 0; } #else diff --git a/fs/exec.c b/fs/exec.c index ab913243a367..5e1631e109a8 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -270,7 +270,7 @@ static int __bprm_mm_init(struct linux_binprm *bprm) BUILD_BUG_ON(VM_STACK_FLAGS & VM_STACK_INCOMPLETE_SETUP); vma->vm_end = STACK_TOP_MAX; vma->vm_start = vma->vm_end - PAGE_SIZE; - vma->vm_flags = VM_SOFTDIRTY | VM_STACK_FLAGS | VM_STACK_INCOMPLETE_SETUP; + init_vm_flags(vma, VM_SOFTDIRTY | VM_STACK_FLAGS | VM_STACK_INCOMPLETE_SETUP); vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); err = insert_vm_struct(mm, vma); @@ -834,7 +834,7 @@ int setup_arg_pages(struct linux_binprm *bprm, } /* mprotect_fixup is overkill to remove the temporary stack flags */ - vma->vm_flags &= ~VM_STACK_INCOMPLETE_SETUP; + clear_vm_flags(vma, VM_STACK_INCOMPLETE_SETUP); stack_expand = 131072UL; /* randomly 32*4k (or 2*64k) pages */ stack_size = vma->vm_end - vma->vm_start; diff --git a/fs/ext4/file.c b/fs/ext4/file.c index 7ac0a81bd371..baeb385b07c7 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -801,7 +801,7 @@ static int ext4_file_mmap(struct file *file, struct vm_area_struct *vma) file_accessed(file); if (IS_DAX(file_inode(file))) { vma->vm_ops = &ext4_dax_vm_ops; - vma->vm_flags |= VM_HUGEPAGE; + set_vm_flags(vma, VM_HUGEPAGE); } else { vma->vm_ops = &ext4_file_vm_ops; } diff --git a/fs/fuse/dax.c b/fs/fuse/dax.c index e23e802a8013..599969edc869 100644 --- a/fs/fuse/dax.c +++ b/fs/fuse/dax.c @@ -860,7 +860,7 @@ int fuse_dax_mmap(struct file *file, struct vm_area_struct *vma) { file_accessed(file); vma->vm_ops = &fuse_dax_vm_ops; - vma->vm_flags |= VM_MIXEDMAP | VM_HUGEPAGE; + set_vm_flags(vma, VM_MIXEDMAP | VM_HUGEPAGE); return 0; } diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 790d2727141a..d63a392985a7 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -132,7 +132,7 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma) * way when do_mmap unwinds (may be important on powerpc * and ia64). */ - vma->vm_flags |= VM_HUGETLB | VM_DONTEXPAND; + set_vm_flags(vma, VM_HUGETLB | VM_DONTEXPAND); vma->vm_ops = &hugetlb_vm_ops; ret = seal_check_future_write(info->seals, vma); @@ -813,7 +813,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, * as input to create an allocation policy. */ vma_init(&pseudo_vma, mm); - pseudo_vma.vm_flags = (VM_HUGETLB | VM_MAYSHARE | VM_SHARED); + init_vm_flags(&pseudo_vma, VM_HUGETLB | VM_MAYSHARE | VM_SHARED); pseudo_vma.vm_file = file; for (index = start; index < end; index++) { diff --git a/fs/orangefs/file.c b/fs/orangefs/file.c index 167fa43b24f9..0f668db6bcf3 100644 --- a/fs/orangefs/file.c +++ b/fs/orangefs/file.c @@ -389,8 +389,7 @@ static int orangefs_file_mmap(struct file *file, struct vm_area_struct *vma) "orangefs_file_mmap: called on %pD\n", file); /* set the sequential readahead hint */ - vma->vm_flags |= VM_SEQ_READ; - vma->vm_flags &= ~VM_RAND_READ; + mod_vm_flags(vma, VM_SEQ_READ, VM_RAND_READ); file_accessed(file); vma->vm_ops = &orangefs_file_vm_ops; diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index e35a0398db63..4d651777c8a5 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1302,7 +1302,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, mas_for_each(&mas, vma, ULONG_MAX) { if (!(vma->vm_flags & VM_SOFTDIRTY)) continue; - vma->vm_flags &= ~VM_SOFTDIRTY; + clear_vm_flags(vma, VM_SOFTDIRTY); vma_set_page_prot(vma); } diff --git a/fs/proc/vmcore.c b/fs/proc/vmcore.c index 09a81e4b1273..858e4e804f85 100644 --- a/fs/proc/vmcore.c +++ b/fs/proc/vmcore.c @@ -582,8 +582,7 @@ static int mmap_vmcore(struct file *file, struct vm_area_struct *vma) if (vma->vm_flags & (VM_WRITE | VM_EXEC)) return -EPERM; - vma->vm_flags &= ~(VM_MAYWRITE | VM_MAYEXEC); - vma->vm_flags |= VM_MIXEDMAP; + mod_vm_flags(vma, VM_MIXEDMAP, VM_MAYWRITE | VM_MAYEXEC); vma->vm_ops = &vmcore_mmap_ops; len = 0; diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 98ac37e34e3d..f46252544924 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -618,7 +618,7 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx, for_each_vma(vmi, vma) { if (vma->vm_userfaultfd_ctx.ctx == release_new_ctx) { vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; - vma->vm_flags &= ~__VM_UFFD_FLAGS; + clear_vm_flags(vma, __VM_UFFD_FLAGS); } } mmap_write_unlock(mm); @@ -652,7 +652,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs) octx = vma->vm_userfaultfd_ctx.ctx; if (!octx || !(octx->features & UFFD_FEATURE_EVENT_FORK)) { vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; - vma->vm_flags &= ~__VM_UFFD_FLAGS; + clear_vm_flags(vma, __VM_UFFD_FLAGS); return 0; } @@ -733,7 +733,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma, } else { /* Drop uffd context if remap feature not enabled */ vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; - vma->vm_flags &= ~__VM_UFFD_FLAGS; + clear_vm_flags(vma, __VM_UFFD_FLAGS); } } @@ -895,7 +895,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) prev = vma; } - vma->vm_flags = new_flags; + reset_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; } mmap_write_unlock(mm); @@ -1463,7 +1463,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ - vma->vm_flags = new_flags; + reset_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx.ctx = ctx; if (is_vm_hugetlb_page(vma) && uffd_disable_huge_pmd_share(vma)) @@ -1651,7 +1651,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ - vma->vm_flags = new_flags; + reset_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; skip: diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index 595a5bcf46b9..bf777fed0dd4 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -1429,7 +1429,7 @@ xfs_file_mmap( file_accessed(file); vma->vm_ops = &xfs_file_vm_ops; if (IS_DAX(inode)) - vma->vm_flags |= VM_HUGEPAGE; + set_vm_flags(vma, VM_HUGEPAGE); return 0; } diff --git a/include/linux/mm.h b/include/linux/mm.h index 2b16d45b75a6..594e835bad9c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3756,7 +3756,7 @@ static inline int seal_check_future_write(int seals, struct vm_area_struct *vma) * VM_MAYWRITE as we still want them to be COW-writable. */ if (vma->vm_flags & VM_SHARED) - vma->vm_flags &= ~(VM_MAYWRITE); + clear_vm_flags(vma, VM_MAYWRITE); } return 0; diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 80f4b4d88aaf..d2c967cc2873 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -269,7 +269,7 @@ static int ringbuf_map_mmap_kern(struct bpf_map *map, struct vm_area_struct *vma if (vma->vm_pgoff != 0 || vma->vm_end - vma->vm_start != PAGE_SIZE) return -EPERM; } else { - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); } /* remap_vmalloc_range() checks size and offset constraints */ return remap_vmalloc_range(vma, rb_map->rb, @@ -290,7 +290,7 @@ static int ringbuf_map_mmap_user(struct bpf_map *map, struct vm_area_struct *vma */ return -EPERM; } else { - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); } /* remap_vmalloc_range() checks size and offset constraints */ return remap_vmalloc_range(vma, rb_map->rb, vma->vm_pgoff + RINGBUF_PGOFF); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 64131f88c553..db19094c7ac7 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -882,10 +882,10 @@ static int bpf_map_mmap(struct file *filp, struct vm_area_struct *vma) /* set default open/close callbacks */ vma->vm_ops = &bpf_map_default_vmops; vma->vm_private_data = map; - vma->vm_flags &= ~VM_MAYEXEC; + clear_vm_flags(vma, VM_MAYEXEC); if (!(vma->vm_flags & VM_WRITE)) /* disallow re-mapping with PROT_WRITE */ - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); err = map->ops->map_mmap(map, vma); if (err) diff --git a/kernel/events/core.c b/kernel/events/core.c index d56328e5080e..6745460dcf49 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6573,7 +6573,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) * Since pinned accounting is per vm we cannot allow fork() to copy our * vma. */ - vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTCOPY | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &perf_mmap_vmops; if (event->pmu->event_mapped) diff --git a/kernel/kcov.c b/kernel/kcov.c index e5cd09fd8a05..27fc1e26e1e1 100644 --- a/kernel/kcov.c +++ b/kernel/kcov.c @@ -489,7 +489,7 @@ static int kcov_mmap(struct file *filep, struct vm_area_struct *vma) goto exit; } spin_unlock_irqrestore(&kcov->lock, flags); - vma->vm_flags |= VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTEXPAND); for (off = 0; off < size; off += PAGE_SIZE) { page = vmalloc_to_page(kcov->area + off); res = vm_insert_page(vma, vma->vm_start + off, page); diff --git a/kernel/relay.c b/kernel/relay.c index ef12532168d9..085aa8707bc2 100644 --- a/kernel/relay.c +++ b/kernel/relay.c @@ -91,7 +91,7 @@ static int relay_mmap_buf(struct rchan_buf *buf, struct vm_area_struct *vma) return -EINVAL; vma->vm_ops = &relay_file_mmap_ops; - vma->vm_flags |= VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTEXPAND); vma->vm_private_data = buf; return 0; diff --git a/mm/madvise.c b/mm/madvise.c index a56a6d17e201..5b74321bcac9 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -179,7 +179,7 @@ static int madvise_update_vma(struct vm_area_struct *vma, /* * vm_flags is protected by the mmap_lock held in write mode. */ - vma->vm_flags = new_flags; + reset_vm_flags(vma, new_flags); if (!vma->vm_file || vma_is_anon_shmem(vma)) { error = replace_anon_vma_name(vma, anon_name); if (error) diff --git a/mm/memory.c b/mm/memory.c index aad226daf41b..2fabf89b2be9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1951,7 +1951,7 @@ int vm_insert_pages(struct vm_area_struct *vma, unsigned long addr, if (!(vma->vm_flags & VM_MIXEDMAP)) { BUG_ON(mmap_read_trylock(vma->vm_mm)); BUG_ON(vma->vm_flags & VM_PFNMAP); - vma->vm_flags |= VM_MIXEDMAP; + set_vm_flags(vma, VM_MIXEDMAP); } /* Defer page refcount checking till we're about to map that page. */ return insert_pages(vma, addr, pages, num, vma->vm_page_prot); @@ -2009,7 +2009,7 @@ int vm_insert_page(struct vm_area_struct *vma, unsigned long addr, if (!(vma->vm_flags & VM_MIXEDMAP)) { BUG_ON(mmap_read_trylock(vma->vm_mm)); BUG_ON(vma->vm_flags & VM_PFNMAP); - vma->vm_flags |= VM_MIXEDMAP; + set_vm_flags(vma, VM_MIXEDMAP); } return insert_page(vma, addr, page, vma->vm_page_prot); } @@ -2475,7 +2475,7 @@ int remap_pfn_range_notrack(struct vm_area_struct *vma, unsigned long addr, vma->vm_pgoff = pfn; } - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); BUG_ON(addr >= end); pfn -= addr >> PAGE_SHIFT; diff --git a/mm/mlock.c b/mm/mlock.c index 06aa9e204fac..4807e91aaa8b 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -380,7 +380,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, */ if (newflags & VM_LOCKED) newflags |= VM_IO; - WRITE_ONCE(vma->vm_flags, newflags); + reset_vm_flags(vma, newflags); lru_add_drain(); walk_page_range(vma->vm_mm, start, end, &mlock_walk_ops, NULL); @@ -388,7 +388,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, if (newflags & VM_IO) { newflags &= ~VM_IO; - WRITE_ONCE(vma->vm_flags, newflags); + reset_vm_flags(vma, newflags); } } @@ -456,7 +456,7 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev, if ((newflags & VM_LOCKED) && (oldflags & VM_LOCKED)) { /* No work to do, and mlocking twice would be wrong */ - vma->vm_flags = newflags; + reset_vm_flags(vma, newflags); } else { mlock_vma_pages_range(vma, start, end, newflags); } diff --git a/mm/mmap.c b/mm/mmap.c index 5c4b608edde9..fa994ae903d9 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2607,7 +2607,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vma->vm_start = addr; vma->vm_end = end; - vma->vm_flags = vm_flags; + init_vm_flags(vma, vm_flags); vma->vm_page_prot = vm_get_page_prot(vm_flags); vma->vm_pgoff = pgoff; @@ -2736,7 +2736,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, * then new mapped in-place (which must be aimed as * a completely new data area). */ - vma->vm_flags |= VM_SOFTDIRTY; + set_vm_flags(vma, VM_SOFTDIRTY); vma_set_page_prot(vma); @@ -2959,7 +2959,7 @@ static int do_brk_flags(struct ma_state *mas, struct vm_area_struct *vma, anon_vma_interval_tree_pre_update_vma(vma); } vma->vm_end = addr + len; - vma->vm_flags |= VM_SOFTDIRTY; + set_vm_flags(vma, VM_SOFTDIRTY); mas_store_prealloc(mas, vma); if (vma->anon_vma) { @@ -2979,7 +2979,7 @@ static int do_brk_flags(struct ma_state *mas, struct vm_area_struct *vma, vma->vm_start = addr; vma->vm_end = addr + len; vma->vm_pgoff = addr >> PAGE_SHIFT; - vma->vm_flags = flags; + init_vm_flags(vma, flags); vma->vm_page_prot = vm_get_page_prot(flags); mas_set_range(mas, vma->vm_start, addr + len - 1); if (mas_store_gfp(mas, vma, GFP_KERNEL)) @@ -2992,7 +2992,7 @@ static int do_brk_flags(struct ma_state *mas, struct vm_area_struct *vma, mm->data_vm += len >> PAGE_SHIFT; if (flags & VM_LOCKED) mm->locked_vm += (len >> PAGE_SHIFT); - vma->vm_flags |= VM_SOFTDIRTY; + set_vm_flags(vma, VM_SOFTDIRTY); validate_mm(mm); return 0; diff --git a/mm/mprotect.c b/mm/mprotect.c index 908df12caa26..79adae74c094 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -633,7 +633,7 @@ mprotect_fixup(struct mmu_gather *tlb, struct vm_area_struct *vma, * vm_flags and vm_page_prot are protected by the mmap_lock * held in write mode. */ - vma->vm_flags = newflags; + reset_vm_flags(vma, newflags); if (vma_wants_manual_pte_write_upgrade(vma)) mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE; vma_set_page_prot(vma); diff --git a/mm/mremap.c b/mm/mremap.c index 5f6f9931bff1..2ccdd1561f5b 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -661,7 +661,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, /* Conceal VM_ACCOUNT so old reservation is not undone */ if (vm_flags & VM_ACCOUNT && !(flags & MREMAP_DONTUNMAP)) { - vma->vm_flags &= ~VM_ACCOUNT; + clear_vm_flags(vma, VM_ACCOUNT); excess = vma->vm_end - vma->vm_start - old_len; if (old_addr > vma->vm_start && old_addr + old_len < vma->vm_end) @@ -716,9 +716,9 @@ static unsigned long move_vma(struct vm_area_struct *vma, /* Restore VM_ACCOUNT if one or two pieces of vma left */ if (excess) { - vma->vm_flags |= VM_ACCOUNT; + set_vm_flags(vma, VM_ACCOUNT); if (split) - find_vma(mm, vma->vm_end)->vm_flags |= VM_ACCOUNT; + set_vm_flags(find_vma(mm, vma->vm_end), VM_ACCOUNT); } return new_addr; diff --git a/mm/nommu.c b/mm/nommu.c index 214c70e1d059..b3154357ced5 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -173,7 +173,7 @@ static void *__vmalloc_user_flags(unsigned long size, gfp_t flags) mmap_write_lock(current->mm); vma = find_vma(current->mm, (unsigned long)ret); if (vma) - vma->vm_flags |= VM_USERMAP; + set_vm_flags(vma, VM_USERMAP); mmap_write_unlock(current->mm); } @@ -991,7 +991,8 @@ static int do_mmap_private(struct vm_area_struct *vma, atomic_long_add(total, &mmap_pages_allocated); - region->vm_flags = vma->vm_flags |= VM_MAPPED_COPY; + set_vm_flags(vma, VM_MAPPED_COPY); + region->vm_flags = vma->flags; region->vm_start = (unsigned long) base; region->vm_end = region->vm_start + len; region->vm_top = region->vm_start + (total << PAGE_SHIFT); @@ -1088,7 +1089,7 @@ unsigned long do_mmap(struct file *file, region->vm_flags = vm_flags; region->vm_pgoff = pgoff; - vma->vm_flags = vm_flags; + init_vm_flags(vma, vm_flags); vma->vm_pgoff = pgoff; if (file) { @@ -1152,7 +1153,7 @@ unsigned long do_mmap(struct file *file, vma->vm_end = start + len; if (pregion->vm_flags & VM_MAPPED_COPY) - vma->vm_flags |= VM_MAPPED_COPY; + set_vm_flags(vma, VM_MAPPED_COPY); else { ret = do_mmap_shared_file(vma); if (ret < 0) { @@ -1632,7 +1633,7 @@ int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr, if (addr != (pfn << PAGE_SHIFT)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); return 0; } EXPORT_SYMBOL(remap_pfn_range); diff --git a/mm/secretmem.c b/mm/secretmem.c index 04c3ac9448a1..334b85714bd7 100644 --- a/mm/secretmem.c +++ b/mm/secretmem.c @@ -128,7 +128,7 @@ static int secretmem_mmap(struct file *file, struct vm_area_struct *vma) if (mlock_future_check(vma->vm_mm, vma->vm_flags | VM_LOCKED, len)) return -EAGAIN; - vma->vm_flags |= VM_LOCKED | VM_DONTDUMP; + set_vm_flags(vma, VM_LOCKED | VM_DONTDUMP); vma->vm_ops = &secretmem_vm_ops; return 0; diff --git a/mm/shmem.c b/mm/shmem.c index c301487be5fb..2096bbdc955f 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2289,7 +2289,7 @@ static int shmem_mmap(struct file *file, struct vm_area_struct *vma) return ret; /* arm64 - allow memory tagging on RAM-based files */ - vma->vm_flags |= VM_MTE_ALLOWED; + set_vm_flags(vma, VM_MTE_ALLOWED); file_accessed(file); /* This is anonymous shared memory if it is unlinked at the time of mmap */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index ca71de7c9d77..da02ec9c650f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3657,7 +3657,7 @@ int remap_vmalloc_range_partial(struct vm_area_struct *vma, unsigned long uaddr, size -= PAGE_SIZE; } while (size > 0); - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); return 0; } diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index c567d5e8053e..30158585c688 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1890,10 +1890,10 @@ int tcp_mmap(struct file *file, struct socket *sock, { if (vma->vm_flags & (VM_WRITE | VM_EXEC)) return -EPERM; - vma->vm_flags &= ~(VM_MAYWRITE | VM_MAYEXEC); + clear_vm_flags(vma, VM_MAYWRITE | VM_MAYEXEC); /* Instruct vm_insert_page() to not mmap_read_lock(mm) */ - vma->vm_flags |= VM_MIXEDMAP; + set_vm_flags(vma, VM_MIXEDMAP); vma->vm_ops = &tcp_vm_ops; return 0; diff --git a/security/selinux/selinuxfs.c b/security/selinux/selinuxfs.c index 0a6894cdc54d..9037deb5979e 100644 --- a/security/selinux/selinuxfs.c +++ b/security/selinux/selinuxfs.c @@ -262,7 +262,7 @@ static int sel_mmap_handle_status(struct file *filp, if (vma->vm_flags & VM_WRITE) return -EPERM; /* disallow mprotect() turns it into writable */ - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); return remap_pfn_range(vma, vma->vm_start, page_to_pfn(status), @@ -506,13 +506,13 @@ static int sel_mmap_policy(struct file *filp, struct vm_area_struct *vma) { if (vma->vm_flags & VM_SHARED) { /* do not allow mprotect to make mapping writable */ - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); if (vma->vm_flags & VM_WRITE) return -EACCES; } - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &sel_mmap_policy_ops; return 0; diff --git a/sound/core/oss/pcm_oss.c b/sound/core/oss/pcm_oss.c index ac2efeb63a39..52473e2acd07 100644 --- a/sound/core/oss/pcm_oss.c +++ b/sound/core/oss/pcm_oss.c @@ -2910,7 +2910,7 @@ static int snd_pcm_oss_mmap(struct file *file, struct vm_area_struct *area) } /* set VM_READ access as well to fix memset() routines that do reads before writes (to improve performance) */ - area->vm_flags |= VM_READ; + set_vm_flags(area, VM_READ); if (substream == NULL) return -ENXIO; runtime = substream->runtime; diff --git a/sound/core/pcm_native.c b/sound/core/pcm_native.c index 9c122e757efe..f716bdb70afe 100644 --- a/sound/core/pcm_native.c +++ b/sound/core/pcm_native.c @@ -3675,8 +3675,9 @@ static int snd_pcm_mmap_status(struct snd_pcm_substream *substream, struct file return -EINVAL; area->vm_ops = &snd_pcm_vm_ops_status; area->vm_private_data = substream; - area->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; - area->vm_flags &= ~(VM_WRITE | VM_MAYWRITE); + mod_vm_flags(area, VM_DONTEXPAND | VM_DONTDUMP, + VM_WRITE | VM_MAYWRITE); + return 0; } @@ -3712,7 +3713,7 @@ static int snd_pcm_mmap_control(struct snd_pcm_substream *substream, struct file return -EINVAL; area->vm_ops = &snd_pcm_vm_ops_control; area->vm_private_data = substream; - area->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(area, VM_DONTEXPAND | VM_DONTDUMP); return 0; } @@ -3828,7 +3829,7 @@ static const struct vm_operations_struct snd_pcm_vm_ops_data_fault = { int snd_pcm_lib_default_mmap(struct snd_pcm_substream *substream, struct vm_area_struct *area) { - area->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(area, VM_DONTEXPAND | VM_DONTDUMP); if (!substream->ops->page && !snd_dma_buffer_mmap(snd_pcm_get_dma_buf(substream), area)) return 0; diff --git a/sound/soc/pxa/mmp-sspa.c b/sound/soc/pxa/mmp-sspa.c index fb5a4390443f..fdd72d9bb46c 100644 --- a/sound/soc/pxa/mmp-sspa.c +++ b/sound/soc/pxa/mmp-sspa.c @@ -404,7 +404,7 @@ static int mmp_pcm_mmap(struct snd_soc_component *component, struct snd_pcm_substream *substream, struct vm_area_struct *vma) { - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); return remap_pfn_range(vma, vma->vm_start, substream->dma_buffer.addr >> PAGE_SHIFT, diff --git a/sound/usb/usx2y/us122l.c b/sound/usb/usx2y/us122l.c index e558931cce16..b51db622a69b 100644 --- a/sound/usb/usx2y/us122l.c +++ b/sound/usb/usx2y/us122l.c @@ -224,9 +224,9 @@ static int usb_stream_hwdep_mmap(struct snd_hwdep *hw, } area->vm_ops = &usb_stream_hwdep_vm_ops; - area->vm_flags |= VM_DONTDUMP; + set_vm_flags(area, VM_DONTDUMP); if (!read) - area->vm_flags |= VM_DONTEXPAND; + set_vm_flags(area, VM_DONTEXPAND); area->vm_private_data = us122l; atomic_inc(&us122l->mmap_count); out: diff --git a/sound/usb/usx2y/usX2Yhwdep.c b/sound/usb/usx2y/usX2Yhwdep.c index c29da0341bc5..3abe6d891f98 100644 --- a/sound/usb/usx2y/usX2Yhwdep.c +++ b/sound/usb/usx2y/usX2Yhwdep.c @@ -61,7 +61,7 @@ static int snd_us428ctls_mmap(struct snd_hwdep *hw, struct file *filp, struct vm } area->vm_ops = &us428ctls_vm_ops; - area->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(area, VM_DONTEXPAND | VM_DONTDUMP); area->vm_private_data = hw->private_data; return 0; } diff --git a/sound/usb/usx2y/usx2yhwdeppcm.c b/sound/usb/usx2y/usx2yhwdeppcm.c index 767a227d54da..22ce93b2fb24 100644 --- a/sound/usb/usx2y/usx2yhwdeppcm.c +++ b/sound/usb/usx2y/usx2yhwdeppcm.c @@ -706,7 +706,7 @@ static int snd_usx2y_hwdep_pcm_mmap(struct snd_hwdep *hw, struct file *filp, str return -ENODEV; area->vm_ops = &snd_usx2y_hwdep_pcm_vm_ops; - area->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(area, VM_DONTEXPAND | VM_DONTDUMP); area->vm_private_data = hw->private_data; return 0; } From patchwork Mon Jan 9 20:53:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094374 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 26CF4C54EBD for ; Mon, 9 Jan 2023 21:32:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=bRZCOxbxPXw2aJQ8LMEKtoQi817P+H5jebVZSbDX9r4=; b=2EL47akHJ9kn0Wa/sf8QB7lNL6 WIPrgsPbgUnRzxK90wLgc2J5ZnJQEUPELwx4YnXongonIAfbMyDCVfIblXxgY9mfyvvWj1bxNq4gt yrr358nL/MeVNwUpClDJd8s7s3l35+35CU9g/9OqUD9TJrIQ83bhyMVxeFAZojZHNaIAwvD6ADScr adUTt5zOy0YCnJCuJAtki9RWauvgOGz5fui/ZjdH3O9/PuHJoGUDH2EH9aTwtg7GfuoNeizbbjWMd eWsFN8kFfNWHtMfkdshC2WV0Nw6+tUvnUEet5EQHzOU6knw04jah6LqDNKlJSbAFqQqHmXd8xmTF1 G41akJmg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzjq-004EfQ-NY; Mon, 09 Jan 2023 21:31:28 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzU8-0048n5-Cq for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:15:12 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=N7mRvZOR9OSs0kq6Wz0vbjqUGqop/n2M19/OmQJBzkI=; b=fIO1zSs4wTD9fIrN34AWyK1K6x aWoP8cbFD+1rn1ytCNx/m5fl9rvtaCz9euNS2XlrkqMS3+zXLeLVV8KJ7Y0c/lujUzvMPgg0ulmVL 7Q6Xzu9FvDzcPXl7T+Z8we1drJbHXxJBgTWYognjJtzR9V2QnZumqilm3b4i8qp4Wiv4GzFjRLJJo suqhbcQeasyQ/kv9uQRc6HPGNvsG8mXprwtq/+YFrm5m7N/nI0VuTypCWyzj0V6p/+ArQIX4UODnp 7UpHDFSvz6kgNGwmnZLVlPo/6rXv3Td7hzHyJSmIuk9B7UBUadWSToNlUVNi7n3z3HWbt7RDMhh5W JlPMiA5Q==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAB-002cL3-KD for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:37 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4bdeb1bbeafso103869497b3.4 for ; Mon, 09 Jan 2023 12:54:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=N7mRvZOR9OSs0kq6Wz0vbjqUGqop/n2M19/OmQJBzkI=; b=kK6VGqQrZF8SG+/SjIkBNYvs6AD0XP75PAW8W4oCooA8s3XRGaAuXa1jS1HIr83jUq PfI+R4O3x61YrRtYzJolldiJm8hvHC8SxT9psAXnN9HRUclq2Ip2r4U5z9IkvWwPkq1G R8EmUXB5Ci5VhDTGcvMKuR9ocC5TZKh6cUC/3KU3F2jjjEENLe5/WyQt1N1DazjgpxS4 yljAKeS+kCfpqMArc2lBS+96rZHSjRTZ1IN5B+phw/v3GaAeI+j/XkkH3A++TUlQdWQv Z4M4yuHxGDrl8gjpdpiZsATGjMq3MZe3RurXo4cwDj7NtnBXJ6X3cbtci2E7Lsi2JjQl 1UiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=N7mRvZOR9OSs0kq6Wz0vbjqUGqop/n2M19/OmQJBzkI=; b=B22IHqRwL46GC9EmgfFPnQpZIdsZo64HAfVoAE8+m1NkQgo/hDs578Q7kWtr/Wdg/4 nbGsEUBdfvhQBcMC3tifhopGzVf4KN9/L079txG0/ttxalCCqRFocl45XFz4ZT4v2yqg Cj2ePcATA3gxDV+kyCic2rQtLZvFzy57IZ3OwZXYnXLM8JEoImTAa1xePd45vmBeZHUS zIU59/hn5RFIA13GS50t9vsbyviYwQA1MT2J+KpmUJ+XAaUPb5Lnl1DTze99iWCklLkj jr8kvhZyZxBR51A5aPYrMoNG18FtWC9HNhkjMXtWxSx+A9FB2WzWa3ePANIrNg3xuJBe hZ3w== X-Gm-Message-State: AFqh2krO1yp7gq3lEj39EwsrN+O3y66CbUpRGvna2vckugIvnLidLdnO qXvedPqKxlrbxlr9IUQbjUrt605eR0A= X-Google-Smtp-Source: AMrXdXshiyJrN6EY9CYZGhpA9uWntRCGsbXmXt4bSfqvBPsR2AoGwmZNzha8yCXqkiCEpIRfWXrumcAFYxY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a81:65d7:0:b0:39a:afeb:f519 with SMTP id z206-20020a8165d7000000b0039aafebf519mr799450ywb.146.1673297662050; Mon, 09 Jan 2023 12:54:22 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:11 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-17-surenb@google.com> Subject: [PATCH 16/41] mm: replace vma->vm_flags indirect modification in ksm_madvise From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205435_705585_3C0F9B5A X-CRM114-Status: GOOD ( 12.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Replace indirect modifications to vma->vm_flags with calls to modifier functions to be able to track flag changes and to keep vma locking correctness. Add a BUG_ON check in ksm_madvise() to catch indirect vm_flags modification attempts. Signed-off-by: Suren Baghdasaryan --- arch/powerpc/kvm/book3s_hv_uvmem.c | 5 ++++- arch/s390/mm/gmap.c | 5 ++++- mm/khugepaged.c | 2 ++ mm/ksm.c | 2 ++ 4 files changed, 12 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/kvm/book3s_hv_uvmem.c b/arch/powerpc/kvm/book3s_hv_uvmem.c index 1d67baa5557a..325a7a47d348 100644 --- a/arch/powerpc/kvm/book3s_hv_uvmem.c +++ b/arch/powerpc/kvm/book3s_hv_uvmem.c @@ -393,6 +393,7 @@ static int kvmppc_memslot_page_merge(struct kvm *kvm, { unsigned long gfn = memslot->base_gfn; unsigned long end, start = gfn_to_hva(kvm, gfn); + unsigned long vm_flags; int ret = 0; struct vm_area_struct *vma; int merge_flag = (merge) ? MADV_MERGEABLE : MADV_UNMERGEABLE; @@ -409,12 +410,14 @@ static int kvmppc_memslot_page_merge(struct kvm *kvm, ret = H_STATE; break; } + vm_flags = vma->vm_flags; ret = ksm_madvise(vma, vma->vm_start, vma->vm_end, - merge_flag, &vma->vm_flags); + merge_flag, &vm_flags); if (ret) { ret = H_STATE; break; } + reset_vm_flags(vma, vm_flags); start = vma->vm_end; } while (end > vma->vm_end); diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 3811d6c86d09..e47387f8be6d 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -2587,14 +2587,17 @@ int gmap_mark_unmergeable(void) { struct mm_struct *mm = current->mm; struct vm_area_struct *vma; + unsigned long vm_flags; int ret; VMA_ITERATOR(vmi, mm, 0); for_each_vma(vmi, vma) { + vm_flags = vma->vm_flags; ret = ksm_madvise(vma, vma->vm_start, vma->vm_end, - MADV_UNMERGEABLE, &vma->vm_flags); + MADV_UNMERGEABLE, &vm_flags); if (ret) return ret; + reset_vm_flags(vma, vm_flags); } mm->def_flags &= ~VM_MERGEABLE; return 0; diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 5cb401aa2b9d..5376246a3052 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -352,6 +352,8 @@ struct attribute_group khugepaged_attr_group = { int hugepage_madvise(struct vm_area_struct *vma, unsigned long *vm_flags, int advice) { + /* vma->vm_flags can be changed only using modifier functions */ + BUG_ON(vm_flags == &vma->vm_flags); switch (advice) { case MADV_HUGEPAGE: #ifdef CONFIG_S390 diff --git a/mm/ksm.c b/mm/ksm.c index dd02780c387f..d05c41b289db 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -2471,6 +2471,8 @@ int ksm_madvise(struct vm_area_struct *vma, unsigned long start, struct mm_struct *mm = vma->vm_mm; int err; + /* vma->vm_flags can be changed only using modifier functions */ + BUG_ON(vm_flags == &vma->vm_flags); switch (advice) { case MADV_MERGEABLE: /* From patchwork Mon Jan 9 20:53:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 12BA7C5479D for ; Mon, 9 Jan 2023 20:55:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=fcqL05HMHHqgwhaJvEffp4wyoiMUgUdpHYB3Lhq2Asw=; b=Q3P89efjmNI/Jh/qlbsuIdVKWY 6aC9KboIsQOb5cSJtSpEJE0mAahCYBmrjVsSGMAUvy8nyDpd17pGk/8k0dA9BA1hJKoKzX7L4ATiY b11D7oVazyn/DJ28JAU3X6ueCIMwt6fFFkiQX4JbO7oQAIPQZ9qGUtUSchni5uTl2LN8paK+Fp+ev nylgpHVn06RSpeKlrjpKPImifqXRrqfnoxxoAACSVtCIBqjWMPz6+OLHaEaUGfPeppAX9mcPCyiDd 7B0t8GVKsx/bJwbvyj4NWYcQJBAqdsmM7dpJ5K9AhhsiT0BCGuIVgXVgNQyEufS92FN+3a5AKKIi3 vLOgHJkw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAL-0040JU-2Y; Mon, 09 Jan 2023 20:54:45 +0000 Received: from mail-pg1-x54a.google.com ([2607:f8b0:4864:20::54a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzA2-00404Q-Lf for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:31 +0000 Received: by mail-pg1-x54a.google.com with SMTP id j21-20020a63fc15000000b00476d6932baeso4187290pgi.23 for ; Mon, 09 Jan 2023 12:54:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hDqG76VNjmKFCkYaVQCHk1jK9PV7CZBX+MdyRpSUw7Y=; b=oTTbEHXXf074UTT/LXfKIzTlYAOVSZBWjz/z2pi8uw2sgTsKOn0pf4M+FUmb3fhKEp t8utYb+BYUrtOmcMbBRfjQmj3HjZCnvt7nX8XXsh4x467hUw4mc17N9vBA4v3IH0gXYu Tf0NgJ8TRnq+dnC575YF2XUlugz8igLSTMm592zbLFSlzWIJOjCEMLUlmrt7z2QrvV2T gekww8A2UJVHgfp1zwqjj/KBiOxEMFrYzziSkpJrGHAD1djB4N2zU8kmwkE7PnEOrNUu hxvDlopUQbh5t04B3B6cW3kLOVcLSOY+TAkcMQBqHoc5+w1BV2sS39QIbB6FPKvgrwld s2sA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hDqG76VNjmKFCkYaVQCHk1jK9PV7CZBX+MdyRpSUw7Y=; b=D3HRHgeiIEFOB8ADtCT0BoEPDReormwIsQVfjBArstwJ96WJw8J8xyUp9Veb6Cb6RJ RpF3DzudfQekxa6bh5vhb04eyfqFpyo6fLMT8Q8uCZRMOe0hUVpCu5N9FBxV9pDrGmJB YzoPw2yebRhTf5b1n+SyBZS9pdEEelQb1qNdwS9hdBuRL31xhmsyH0NBf3dkc9cZzhkw 0F937T5lzWvhOv7cN9TDIxKrA8D3tIux+mP3aaQAZmdXBIhN7ozxps9wSkZChk9Dk7Fz nrG3eKtleyIY0B4gjI/VU1VggQBJlxQugR2eH72W7vg80qrQtIJn8/kQskZ+9EAMaFjl uSEw== X-Gm-Message-State: AFqh2koc6AKuPMz4cfRpZGAnvg5X4NQC3SAVuGTIeU/bIit4gTT+OsZ0 xGZzUGa/DAGAbr7WLVv7bwRWs3joiaM= X-Google-Smtp-Source: AMrXdXvx+5mjUKCd2siyFSRbOdBiHgQPNh6ZnBtsj8lXWONeXJTph7ewao1odYnxoExvO1xzjoIbjT4zuzo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a17:90a:638b:b0:221:52e3:1f56 with SMTP id f11-20020a17090a638b00b0022152e31f56mr5166916pjj.225.1673297664681; Mon, 09 Jan 2023 12:54:24 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:12 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-18-surenb@google.com> Subject: [PATCH 17/41] mm/mmap: move VMA locking before anon_vma_lock_write call From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_125426_755977_1A0341F0 X-CRM114-Status: GOOD ( 11.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Move VMA flag modification (which now implies VMA locking) before anon_vma_lock_write to match the locking order of page fault handler. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index fa994ae903d9..53d885e70a54 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2953,13 +2953,13 @@ static int do_brk_flags(struct ma_state *mas, struct vm_area_struct *vma, if (mas_preallocate(mas, vma, GFP_KERNEL)) goto unacct_fail; + set_vm_flags(vma, VM_SOFTDIRTY); vma_adjust_trans_huge(vma, vma->vm_start, addr + len, 0); if (vma->anon_vma) { anon_vma_lock_write(vma->anon_vma); anon_vma_interval_tree_pre_update_vma(vma); } vma->vm_end = addr + len; - set_vm_flags(vma, VM_SOFTDIRTY); mas_store_prealloc(mas, vma); if (vma->anon_vma) { From patchwork Mon Jan 9 20:53:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B0298C54EBD for ; Mon, 9 Jan 2023 20:56:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=NFdrshVc/yfMRhK+nHOh+JU5a+SvG7qI5kCw+K8hwJY=; b=f/Uza3PQkmBd7rmS3R+sawftQ5 vod6MKdWFzj7/XJuI6sYs7F1fil41Rj0ot0GYpd3VMZP2DHdi8QOwUbsQp1ZJ7004xj9Ew6OE6WUl zD6gZj9WBpmLhIC7hxO10PIDyRDqxwLkePaJuEb8ss8c7C9smdW5EEjzPpEzcjMYYLvEFNJl/1LNK il0gyCPnOVADUxUITOVCqZPeqvhg+lNW5mZlJSOU5RRb1j/qqZqMZTZnY1F1HKC7Y3u4O3IpJ3+dz buJ9VvoTLQ0/LB5VGeOQAdtNeEwmfj/GYJfbwyX4tZ0f/zieHFwrO3T/i2FuqmWilBulE4Ilwq55a NedhLfXQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzBC-0040qj-Ni; Mon, 09 Jan 2023 20:55:38 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzA5-00406h-AK for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:35 +0000 Received: by mail-yb1-xb49.google.com with SMTP id i17-20020a25bc11000000b007b59a5b74aaso10157253ybh.7 for ; Mon, 09 Jan 2023 12:54:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mttq8CcSxh6pk/PDvvqo/9XfYX/awSV+YKJHUec4DWQ=; b=EbLgVpiNWauCxYsPhZzluNh8i7Jc9od95XU7q4baD/bZuTFbACugKjaBqwn3ZlwjPK DyA2dJROokooe1X9hWF9cqxx7R8dzaLC4TQ2VgG1K6ZzCrV6Cn+bNgWtTm8qKV2EPlQI C3EgQy0lx8995KoeqPGehRxKjsY9DJj6pO/ydpCP34y4VCDsA5lFDA9xaQObsm54nagL gf/rqX49ePSI64VKn0w7yUB28hwRSupVRZyvn8G3inMqKMvhkMvaFS5iHqspfhqeeQGW sJQNxcx4h1OW/3tP5B8wZxgrbXFTXvC0zscvDZLigjE3sBlwiKuCEIrG4zHiBAmIFXhN sp1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mttq8CcSxh6pk/PDvvqo/9XfYX/awSV+YKJHUec4DWQ=; b=OXExeG9vxAvlMyIQ/0A+zOrv1jX0fW38ozDkpXcCd5Lx6jxmjIBuNfPtI1AvOH6c+c 4FY3Z6QiHSbsNPAa70ckIdAmnGPTpRt4f72YooVheiMT+YaGn/BRPaoRUn0HoLpuK7Tx PkYnBoR96pQLxUTwZrq465ykHDtMnwlkfuFUHNrgMEl5M0M7dwJN/wxaQL8f7jj6LfCY 9KCP99eqccIa/Bwh+f1JaIUVE1rGuIvi+LoZX+2+4mkDMxQ0zfV2jKUh2U5rSZl/KCnH eu9jqcmkMznVHln2H89Vw7x2l6MN8MQ8lD8nFm9PgNAR7tYgeQ1ymo2EbR9aOg68GboN T5qw== X-Gm-Message-State: AFqh2krMjz26sBo5UIecdrfJ19AI38uHOXwb5Lo9G/wgxmEs5c3KJ00G yREMQrA/LBb5E1J0m0GjJXY3R9Axv9E= X-Google-Smtp-Source: AMrXdXsB8fbMiTP1DYP5J5XEU88bTBTK0YYISNp2DJzOoqciXF/+cmYxWMxqmdOzbjRBqH/+xJjk8vQehVo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:850e:0:b0:6f8:42d8:2507 with SMTP id w14-20020a25850e000000b006f842d82507mr8097864ybk.110.1673297667509; Mon, 09 Jan 2023 12:54:27 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:13 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-19-surenb@google.com> Subject: [PATCH 18/41] mm/khugepaged: write-lock VMA while collapsing a huge page From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_125429_395112_6F2381C0 X-CRM114-Status: GOOD ( 11.77 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Protect VMA from concurrent page fault handler while collapsing a huge page. Page fault handler needs a stable PMD to use PTL and relies on per-VMA lock to prevent concurrent PMD changes. pmdp_collapse_flush(), set_huge_pmd() and collapse_and_free_pmd() can modify a PMD, which will not be detected by a page fault handler without proper locking. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 5376246a3052..d8d0647f0c2c 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1032,6 +1032,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, if (result != SCAN_SUCCEED) goto out_up_write; + vma_write_lock(vma); anon_vma_lock_write(vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, @@ -1503,6 +1504,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } + /* Lock the vma before taking i_mmap and page table locks */ + vma_write_lock(vma); + /* * We need to lock the mapping so that from here on, only GUP-fast and * hardware page walks can access the parts of the page tables that @@ -1690,6 +1694,7 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, result = SCAN_PTE_UFFD_WP; goto unlock_next; } + vma_write_lock(vma); collapse_and_free_pmd(mm, vma, addr, pmd); if (!cc->is_khugepaged && is_target) result = set_huge_pmd(vma, addr, pmd, hpage); From patchwork Mon Jan 9 20:53:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094373 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 67758C5479D for ; Mon, 9 Jan 2023 21:32:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=cZJShWYTUD0HHrH8SDb+NhiCs4Iwne2XPCqO21mShpY=; b=TjbUU+12C7bXL09Vafkt7i935N ymyZwg/3B7fPtUz+MV6v4d00X/Dlb/EcUr0/sku8seQ+Nrt3joF+mdq62UqNRHzSpBjOylhqARGCZ 22d20URNnQjuaP94UkZ07GFjiKVQmw+VQqGBHl5KKK7GWwX5QvFU0rYdpNLRU8T416xwsEiTk8fcQ cbVBeX7srNNVf7iF1gNFO2ci6PVr0cnSqeBNqSq+c+DuJBQBIfpYwQBoVZvbVoOWjbZIS1SvJyxRl 2udJJM80g9VJmZkJL/0he7wW8zK6HKTysetZV124pCexE84Fb0fokmtEXvguLqWWCcLCpFf4G0oIr K5JuTqNw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzj4-004EQ0-Qr; Mon, 09 Jan 2023 21:30:39 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzU7-0048n5-7t for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:15:11 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=ves4Zx76RBv25M18zzcj7gVloTdTuNMdyKmYC5DDvz0=; b=Vf+N8mMxGaZeBqYijEyAqpFBRe Uxg/RVWTue/ZcppV73Q3zlGzEhAwlzmhLmEw87JJUNU4SDOsrdDM8rXlTCpSgtbXqPhlAaNDJdV0O c1afh7MeQI47H/GluqlAlj3qvvfpZkfr2GZkuKNxLXTxkXexoju0pqABO97pY43M4wD6DjuE/q2vO tMSxASqFqL73sM5YB4zXXN1jt48kxyK+iG/CrBuepLWoEhKQ2BuQDXeg7w7nK8YPBpKVySEzUpjyy yCLDpgLTcwz4gP9oihxOE2+6gLEPuOFOMpPl82ZJtQKmr2mhkhOnReIa0gDSeKqRyVSN0FbMnwR0W g8uZmwbg==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAI-002cL3-Gw for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:45 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4bdeb1bbeafso103873127b3.4 for ; Mon, 09 Jan 2023 12:54:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ves4Zx76RBv25M18zzcj7gVloTdTuNMdyKmYC5DDvz0=; b=MKIFYUVg3C+KcUmk/DiJLvwQZvDLVNMlr8DQ0MCXFdkv6hN2eBfUWDzxIZ7BfVHp8I 9ZQKQbnc3cjtIsfRVDcXo71Ba38rTMbcUdq5jPPQ/O/qUYTP0dyF6taFqjbQZ9FRkBHu 4AhRyAaM/Xs1rNZHH4s644beTjAgsiLfyufSEIgttXRExgdKqL3ufxpDIm4mlvORyhtg 8KPqhXKz4czDblD/OO3yCfwh5TQTP8fNUj9/04RnIZEyuuEXUuTlBqAldxaXV0tj/mD/ bYzPmov2AWSWqL0UrgwZuK2hqucg36ElYTON5SUI9h/73LoxJPTGa7ERIj/8Aw4bHycM nzHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ves4Zx76RBv25M18zzcj7gVloTdTuNMdyKmYC5DDvz0=; b=FDGPCp3F14K6uaOClmuNypp8mO3Q+k0Ng3AfOvRH+xjPWwryGK/csx40LfrtIbxUmj mtBJHIRn6o7wAiRRu/TG4FzElixoFeyL6i9oKgoqKNNq4WyPTcB0HtEdjeXmYOywArtd 23HHTpf32dlT1u7qpVSNHChORzyb8sWY583VFUaguUDhu8ynSse+CMguPXDYgU1aA3zI T81W+NuWNM4V81uaAGNqhZDcr3QwqfpAJo+nLoImBgJC0sxWwhN/eTf+kLUAv2Hkj0XX A5jbjJAWvz6NncoFyhgLYeTgOdMRwolwxl4qBxEeGegy5Nynvn09wxjhT1h4FQAfyaqZ j0IA== X-Gm-Message-State: AFqh2kqHtT7lPZ8g/d4sY+gssZqi+AW4zOaKdIRfriIOmnee5z9WfSiT qFxoFyjHLIrRm9NC9nzWckSCJbu87FM= X-Google-Smtp-Source: AMrXdXvMc8uw91kFZQ5TZdRrbkHWe3EKuKErf5QuBh5uD1fMXXAng4XpjPLPHBE1E+VnZgCLzgnV8hcsb8Y= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a81:a013:0:b0:4a9:884a:20c4 with SMTP id x19-20020a81a013000000b004a9884a20c4mr4780346ywg.139.1673297669408; Mon, 09 Jan 2023 12:54:29 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:14 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-20-surenb@google.com> Subject: [PATCH 19/41] mm/mmap: write-lock VMAs before merging, splitting or expanding them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205442_571917_448B8D11 X-CRM114-Status: GOOD ( 13.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Decisions about whether VMAs can be merged, split or expanded must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Write-lock all VMAs which might be affected by a merge or split operation before making decision how such operations should be performed. Not sure if expansion really needs this, just being paranoid. Otherwise mmap_region and vm_brk_flags might not locking. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 53d885e70a54..f6ca4a87f9e2 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -254,8 +254,11 @@ SYSCALL_DEFINE1(brk, unsigned long, brk) */ mas_set(&mas, oldbrk); next = mas_find(&mas, newbrk - 1 + PAGE_SIZE + stack_guard_gap); - if (next && newbrk + PAGE_SIZE > vm_start_gap(next)) - goto out; + if (next) { + vma_write_lock(next); + if (newbrk + PAGE_SIZE > vm_start_gap(next)) + goto out; + } brkvma = mas_prev(&mas, mm->start_brk); /* Ok, looks good - let it rip. */ @@ -1017,10 +1020,17 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + if (prev) + vma_write_lock(prev); next = find_vma(mm, prev ? prev->vm_end : 0); mid = next; - if (next && next->vm_end == end) /* cases 6, 7, 8 */ + if (next) + vma_write_lock(next); + if (next && next->vm_end == end) { /* cases 6, 7, 8 */ next = find_vma(mm, next->vm_end); + if (next) + vma_write_lock(next); + } /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); @@ -2198,6 +2208,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma, int err; validate_mm_mt(mm); + vma_write_lock(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split(vma, addr); if (err) @@ -2564,6 +2575,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, /* Attempt to expand an old mapping */ /* Check next */ + if (next) + vma_write_lock(next); if (next && next->vm_start == end && !vma_policy(next) && can_vma_merge_before(next, vm_flags, NULL, file, pgoff+pglen, NULL_VM_UFFD_CTX, NULL)) { @@ -2573,6 +2586,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, } /* Check prev */ + if (prev) + vma_write_lock(prev); if (prev && prev->vm_end == addr && !vma_policy(prev) && (vma ? can_vma_merge_after(prev, vm_flags, vma->anon_vma, file, pgoff, vma->vm_userfaultfd_ctx, NULL) : @@ -2942,6 +2957,8 @@ static int do_brk_flags(struct ma_state *mas, struct vm_area_struct *vma, if (security_vm_enough_memory_mm(mm, len >> PAGE_SHIFT)) return -ENOMEM; + if (vma) + vma_write_lock(vma); /* * Expand the existing vma if possible; Note that singular lists do not * occur after forking, so the expand will only happen on new VMAs. From patchwork Mon Jan 9 20:53:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B178AC5479D for ; Mon, 9 Jan 2023 21:02:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=8cunu/65HKu7dsqKbysoBjtTzkTJUx0Wg1Lew2znNpE=; b=fN/s56swaGOeWrL2YpDum0Q8O/ fmrpuIZPS1/RTQ4hBaPCl62gBN03SjfmZ1CCjd+fDWIvO4XeRKuV1zJrQufAEAsUCDHLTT5Z05e1s ELMNBrRsU5dNuTY553Xf4i8ocH284P0vvh3cD44AjtcQDJdwz27SrEsQUNV33otm2vlAhIiI/epNf yYTxHkLIK2t6GNlsXqQMWf4CIKPUvPAQsiqy1OTocvPI2iL2Q+HU4C5y54YEJsmlPYxbJmSC+v1gR EhOhnbL5qAc61raS9NhqKtmX8ZZjvZhozP+djVvjyv9IgreKg7gwry8KwL6mlqlw91Zybrksu8eNA sU5lDvMQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzGH-0043VE-73; Mon, 09 Jan 2023 21:00:54 +0000 Received: from mail-oo1-xc49.google.com ([2607:f8b0:4864:20::c49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzG7-0043Qc-KZ for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 21:00:45 +0000 Received: by mail-oo1-xc49.google.com with SMTP id b6-20020a4a3406000000b004a5ffc77240so3456142ooa.17 for ; Mon, 09 Jan 2023 13:00:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FDedOk4BDhI0jXVeiXMnt29MYEOAHvgEHolCeEtDVh8=; b=m9qNPQfvAaSk/cZYvrILeQolkFuo8oX0rPOdz+ZeBZe0viaaMWSs2xM9wojEerSzR9 Ehc0liLdN4Rf9t9ITHbgyUo/TpjLKodzNOt3cYyWGhisrg0tQj8O7DOpS0TtK3tPQLQG qUB/3EWr0qIxJ9v9dQ2Qpf6kyj6P77wOGaZ+XxUB3NlRJ4nvQo0f4D36W3MPMO4ktFMc ysLVV5w6Mp+QnxSVtHOQDDwbxZ7kpIUBpnDdTFd6J6q0MuwMTyvve19XzRy6+EFEsY7p 5CEU/qiYVWTESVV3KMWaGXvUZnEBFFh/Tdv+49ubumcalCQjcVJZu4o8Eo8hhtY4lcfd RRGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FDedOk4BDhI0jXVeiXMnt29MYEOAHvgEHolCeEtDVh8=; b=w7jDRXF4E9PwLVjLo9TB31K6YDkNxUQ4knGJSIVPlaNUySV9rE3hk7xlQH8GjXa8N/ uFqD67YmrQDlSzIHbrzPpR2uuZ4fuEqexnWZfeLMXrwAbinpD6CjEfbnLz1ZqsQfz9LJ b+BhXrff1D2UglVimMrr9XtctLXu4gsxfh/6ayN9o8J5yR+lblpMY6wlWIUhpEV9WufM wvNT4oblB71gQXQvyoiDO8eFRZDbnAltGwpYf4gPPRjfKxnbu+fMKwX2Nha2zy6wRJOC bzCWmPzxO7CLFL383KAQKc/q63wkF67KbKkyZ+4B0+dZGsavCAo8k413+YC1yGPSx/tF gygg== X-Gm-Message-State: AFqh2ko4FrRZPRW012CMcUrVo0sVQCgf5H9tMzgFTQ9PIZ/zX/m9MZrP y1k69gAA3oiV1GnD9BYPP2/PhPRsJqc= X-Google-Smtp-Source: AMrXdXvGyOxiOv+MuSWvovzrd0so3mEpN1Edvcx+Z2HBXoeLvz+P6Hw4SXaxw2akuo9mk6+zHwkliVPrxQU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:1489:0:b0:6fb:ef1e:bb1a with SMTP id 131-20020a251489000000b006fbef1ebb1amr6925224ybu.168.1673297671808; Mon, 09 Jan 2023 12:54:31 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:15 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-21-surenb@google.com> Subject: [PATCH 20/41] mm/mmap: write-lock VMAs in vma_adjust From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_130043_725147_CC143F10 X-CRM114-Status: GOOD ( 16.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org vma_adjust modifies a VMA and possibly its neighbors. Write-lock them before making the modifications. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index f6ca4a87f9e2..1e2154137631 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -614,6 +614,12 @@ inline int vma_expand(struct ma_state *mas, struct vm_area_struct *vma, * The following helper function should be used when such adjustments * are necessary. The "insert" vma (if any) is to be inserted * before we drop the necessary locks. + * 'expand' vma is always locked before it's passed to __vma_adjust() + * from vma_merge() because vma should not change from the moment + * can_vma_merge_{before|after} decision is made. + * 'insert' vma is used only by __split_vma() and it's always a brand + * new vma which is not yet added into mm's vma tree, therefore no need + * to lock it. */ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, unsigned long end, pgoff_t pgoff, struct vm_area_struct *insert, @@ -633,6 +639,10 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, MA_STATE(mas, &mm->mm_mt, 0, 0); struct vm_area_struct *exporter = NULL, *importer = NULL; + vma_write_lock(vma); + if (next) + vma_write_lock(next); + if (next && !insert) { if (end >= next->vm_end) { /* @@ -676,8 +686,11 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, * If next doesn't have anon_vma, import from vma after * next, if the vma overlaps with it. */ - if (remove_next == 2 && !next->anon_vma) + if (remove_next == 2 && !next->anon_vma) { exporter = next_next; + if (exporter) + vma_write_lock(exporter); + } } else if (end > next->vm_start) { /* @@ -850,6 +863,8 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, if (remove_next == 2) { remove_next = 1; next = next_next; + if (next) + vma_write_lock(next); goto again; } } From patchwork Mon Jan 9 20:53:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094326 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D4FB8C5479D for ; Mon, 9 Jan 2023 21:05:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=bR/5nxSwdsQclGmXwIfizirBvJU4wTdm9t5ra80CsxM=; b=JzSCW6X+oIpKJgDSPg/xro2DNJ xAKF4K2Q1s4QCWcWhOSy1A9TtBgmSTGDN8VOtWeT+3d8MN5LU2vXPQG2d4a+c4WulmZ+3Nlzmeh6e e7voMOgaVdHh4DqeM693OSio+Ce78h7KfQrU0d+VX3tJmBvefu/aMPVfam38TYRq+ZQcoyKT06iTR 81loluf/y398ME+iJaaqzLYthVQ7b9vn5boac3EjhbLNu3LeME5Q+ZKy9xoRUGWWprKM3hKYgjzYR Q8iUpF7gRKcsYYOW3GfTLnafloskt0wZHEbHvUdNtWA1dJ9KZbOPXBWiSyLYSvtO3bhB1tBXzFi/7 Yzlngg1g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzJp-004523-Cw; Mon, 09 Jan 2023 21:04:35 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzHV-004422-2G for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 21:02:14 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4c8e781bc0aso53673587b3.22 for ; Mon, 09 Jan 2023 13:02:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1sz3d8Q6zHAextbGhc5zNdx6/hdXqibZ5+zjxQEPtcY=; b=dyW75mchIm8T3L3V0Aa5O5essWjp4aFupYS1LFKIISIPYObh/dim0X80nsutKOS2m/ 7vd6AcHVv3F7nL+fFOG4vgOpiulnyZtSLux8AIxMn/Q09ghgqU9hpb0yvL9ry+wKM1Wm UvSXVlZK7VJVWgc1kVtRM87LVA0ol6j3197yqrcZsN3y1TKaMHMTXNPpOvk38N+aLEKf U9xTtNcWgL4MRS7DZ4GXXUuetaMZa2ezppHTDMHc58pzPdKSGBN9qWFGFMwk1DYAuR6m mxxVO9djRkO0qH7ZMi64I+oMQN0AQu1sYjF82YaLSmfJI9fTdyDigl7Wn8SgruW4bLXZ WU7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1sz3d8Q6zHAextbGhc5zNdx6/hdXqibZ5+zjxQEPtcY=; b=OjP52jhW2q5+Pqq2EixVC+SeFbtJ6ruH9CXdlmvKGanbq2As2BFhV9ajFfXORtzSkI iStTufhN6BvdOEPgWyU73f2Rnh05MYB5MoJNTj79przMsk0dQ9eho0lg4pi1FGTpITSM OgrYpbJyyIBGXDihpmuXJEdLe/NvGGdhmwBqllzeBkp3kRvL2+jPt62DqssZdC1bqFJ3 KJQAu0WEIuJvUkzCFoGqM9rAW4EeIMNRsaKV14t64L1vb7kptYxtJwoMwL/IhVIDLtjE C2N5I2h9fC87SGIW4e//lRpzWCcdK0b2jkzryPHCVy1RwbtBz3ulkknlhmN0ykl9mENq PD4A== X-Gm-Message-State: AFqh2kqQKvAIZdI9Snh2uVGfTi1ICdaBR2Sn2NZjDBN+YG/ytX45ea7+ FzGVd66oxgnFS8jRN5C4an23goNdKUc= X-Google-Smtp-Source: AMrXdXvYJ0MKAPgXdwWnp9E6tLdKAmG2zH31TOsESiFv0D3eqJovHEz4lb9T2t60zhYoFlt31PHaEzcWVdQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:830b:0:b0:731:a583:5571 with SMTP id s11-20020a25830b000000b00731a5835571mr8144189ybk.320.1673297674437; Mon, 09 Jan 2023 12:54:34 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:16 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-22-surenb@google.com> Subject: [PATCH 21/41] mm/mmap: write-lock VMAs affected by VMA expansion From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_130213_228004_82A62E9A X-CRM114-Status: UNSURE ( 9.49 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org vma_expand changes VMA boundaries and might result in freeing an adjacent VMA. Write-lock affected VMAs to prevent concurrent page faults. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 1e2154137631..ff02cb51e7e7 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -544,6 +544,7 @@ inline int vma_expand(struct ma_state *mas, struct vm_area_struct *vma, if (mas_preallocate(mas, vma, GFP_KERNEL)) goto nomem; + vma_write_lock(vma); vma_adjust_trans_huge(vma, start, end, 0); if (file) { @@ -590,6 +591,7 @@ inline int vma_expand(struct ma_state *mas, struct vm_area_struct *vma, } if (remove_next) { + vma_write_lock(next); if (file) { uprobe_munmap(next, next->vm_start, next->vm_end); fput(file); From patchwork Mon Jan 9 20:53:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0CE6AC54EBD for ; Mon, 9 Jan 2023 21:01:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=eZNrPxMoHzUoyyJwQKEzjci0w+6gig6SU47DgezJOvw=; b=bO9KlHJX64Ugm3RwzUK9/pjBWY JYJBX/LnhFkekVCfKtpMEeNelVvjQc8yAvwCtVXm+S7VaFZOBI3GMd09lWozsf0oiGe5mKFab4I1g jTm3u0HuyOyQb9qlzY6HzQUUxEAy1+4XHas3yCP3IbYCny12EWdW2KoQvAeezPiV7GHNsJqnFkorP gXx8FPj2I4hotYRXRAGGHZdG4+Al5vnfIQeyeo4iaCOdjOZKTd9ypTpagbEdajH8RrvdIsqSCM9Fi IZRbcc3uvPfnWBKzRZAfHOK9NAJl5OQ1nc11mvyj9l3H3WCRmKB4i43GF9PN8wysZNvBLcCA4f8Qs r95MfYqg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzFQ-004361-W6; Mon, 09 Jan 2023 21:00:01 +0000 Received: from mail-oi1-x24a.google.com ([2607:f8b0:4864:20::24a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzFJ-004337-I7 for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:59:55 +0000 Received: by mail-oi1-x24a.google.com with SMTP id k2-20020a056808068200b00360fc885001so3040713oig.1 for ; Mon, 09 Jan 2023 12:59:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1FpUUiRmr7dBuhWWRB46HjSP7TgY2quKdzRuNf/hqHM=; b=AsrsBvZP9aQHF3L5Nl/u1fB+D2bm7e0YjzSCvgbBaPzVvDqUAEV/bKN3f/FWWFnJld y54pIfCVv+Sf28j1GT6GnDmjTV3QxG0dtIi870ieojG97R1YrLaN1+Hnnh1RqNC6IKhV FDsy8k4m0XBTPaqC3JVLB2amGI721PZ/e3uTJFIVXTdjA/KJiKwGIaizGxyvcp5J8QaW 5xEAO0mZOLLdDxM+ya/sdbkzi1vVwSyWReOOtmKkNqv4WGZ0uHxmN44Y8NrueULgHAw2 BBJ3cZtaXuduG2TgX3HtPnChX2RwWmECLeYtrImIURhb/vK80eEmATn6FHpKHV2WHJD9 N1QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1FpUUiRmr7dBuhWWRB46HjSP7TgY2quKdzRuNf/hqHM=; b=CS5xKzl7O/jMnms1nmSOlnosnN5Ti4OTpo6xUn8D8hhkWNOwyJyt8BOjy4QfCCK6Cw Z3gT46/8faWKLyuHjooQkN5Zh7i2VHBDDkShXmcWBQvR90aqPrllVWZgc0ouzNsf1aZo gx/ZUnN6FcO6dK+UOVqWzbLJDa7fVVEz9VA253jH7iSHpvRcHUd/OlD9hQihADhT7AYK kjeTPpORvFP1/pmgK/fQ4yFVMlktu3vvepn30qUkWUuDK01tw+S5p8xSEDb/XauHfdg7 wp3EDOHviyGWfnrPty7SVNy1cf3LwIn9LBYBFgk+hCZYFspr7rZ0QpeNUdBIewzeUFbZ cj2g== X-Gm-Message-State: AFqh2krQGvGDuNZTikKD4ztZtRZI8efjI8n4GE6VwffDGSgXWFlJWeIE tzWsGtT50TPn9RfhHo9KtC5qQMBGAws= X-Google-Smtp-Source: AMrXdXtAgZu+pAoAaYuQQAsM5E55paYeoDQZrstt2b6LAIgKd1Hk/SubADrA5GRLvEWllXjlmSi56GzDLGU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:4646:0:b0:799:3955:201f with SMTP id t67-20020a254646000000b007993955201fmr3367922yba.94.1673297676537; Mon, 09 Jan 2023 12:54:36 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:17 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-23-surenb@google.com> Subject: [PATCH 22/41] mm/mremap: write-lock VMA while remapping it to a new address range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_125954_063286_AE9E066C X-CRM114-Status: GOOD ( 10.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Write-lock VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index ff02cb51e7e7..da1908730828 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3261,6 +3261,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); + vma_write_lock(new_vma); if (vma_link(mm, new_vma)) goto out_vma_link; *need_rmap_locks = false; diff --git a/mm/mremap.c b/mm/mremap.c index 2ccdd1561f5b..d24a79bcb1a1 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -622,6 +622,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, return -ENOMEM; } + vma_write_lock(vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, &need_rmap_locks); From patchwork Mon Jan 9 20:53:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094328 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B2629C5479D for ; Mon, 9 Jan 2023 21:08:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1Af8qHi29hKmG6+S1JSt3+3c291Mkv+Q+UO2A5+a7kI=; b=PThZ8D42oBOYAbvYOGyOYk/NDp 3hkr0sZBUOjnD+pL6Sx+qmvWdSbrzq8oCaoTIeXunaF7kbo8AwRD74vyb5BEUFLzywr0bGjvZ2btO QRJy0RueXEb/+V5SoficUPuJ9n5a9hUPGTU4Be1T9+q61LtzkkKprEtAmA+Mi3RQrUdhBhl7Qu9fv p9Qrnsg2D9CDH2Arz8Lk6SithMSsVRZ3hXHlsA31h/S20Ju3DVJnaOGBEi9BkQYuW+rDVGXz2Ou4X wkIgvYzvDOzQdPVDxdq6OpI82gVYSjv996TAkhZXCpUb3xUb0WMDrM1BKa/lbejq4++cWuqgmijwv c0VqSVhw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzM8-0045rM-Rr; Mon, 09 Jan 2023 21:06:58 +0000 Received: from mail-qk1-x74a.google.com ([2607:f8b0:4864:20::74a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzII-0044Mr-A3 for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 21:02:59 +0000 Received: by mail-qk1-x74a.google.com with SMTP id v7-20020a05620a0f0700b006faffce43b2so7269478qkl.9 for ; Mon, 09 Jan 2023 13:02:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=olOpibg5eXy7KG3cFGJtGZQRi2wnSYuvllMlsZALFjE=; b=pKSCW4f0d77MorxMhTYeMa3Bkvztzi2l/11p+8AD9JlAcd7O/4nLaK3tMdO/JZYFZO oqL+3k7pxb64mumhhSbnD+8oqAK9T+YZbtnVKYdyWwz1xTl0lxuCTGJlmjyD4W6tkdoJ ScE8xkE+709mZSvnfMafhqWZz0JUfC8PX8Gqgq4hVWAiFqsbly88nJnbf2OtvsEjRJ+V Bh1jSeLHcoArd7dhSosnJ19E8FECzKPk1IhOzNYJUOL7f9tKHCqrIEP0OXg1wGsfV/sX JopWIX7MNjYmVxKWpBhB8gnFt1KMXJN0MSlcK65fQjg6FKg5IorU4J4ePrdeGZ792swl N3JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=olOpibg5eXy7KG3cFGJtGZQRi2wnSYuvllMlsZALFjE=; b=ycdYMWniEheQx796MWtH+YLX5kwnYpJ0ENW6DHxfto7HoT7Kz3bngn+W+Ehcs2PrMM fGb1XpvDf+/B8SGCOvdX2HvfQmreVPyhRq67TYp6GiIHfLoqpIw7C3oDFRwHhsEYPwdW +MzJsVuBpsSPGA7Gs8akzgXzaIGnzvAQTby6LpFFzJ2w9QkzSqw1fAr2t4oOokoIkgWL C8yzt46x5nBO1A28LDExVeOG5DPigF3JY3dGPWunrOW7GbHGnderohfh8Mn4/rXI7xzR kG2wCUoWhaq7/xhpATHkph68ZNK+n4T23kmsbtbc4Xo7RSha3hyq0NcxliUKj0vSi0cV RcWw== X-Gm-Message-State: AFqh2koFBuyC82LITH/5t14+cHglMc6HLaVJGTEFxyTHjJ4aJ438pd3J RACCRpUY1AjaNvRkBxiy3UooHWa5uTY= X-Google-Smtp-Source: AMrXdXsFsv+WY6uB2QAxQKVey5zkUH2AXIcpYb3YNAv+Gej95uP4vEcIzk/UCgPyK193sduP1xpoK0p/ZF8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a62:1501:0:b0:589:8362:c7ce with SMTP id 1-20020a621501000000b005898362c7cemr251559pfv.21.1673297679447; Mon, 09 Jan 2023 12:54:39 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:18 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-24-surenb@google.com> Subject: [PATCH 23/41] mm: write-lock VMAs before removing them from VMA tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_130258_371327_AFB7F1C5 X-CRM114-Status: GOOD ( 10.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Write-locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 ++ mm/nommu.c | 5 +++++ 2 files changed, 7 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index da1908730828..be289e0b693b 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -448,6 +448,7 @@ void vma_mas_store(struct vm_area_struct *vma, struct ma_state *mas) */ void vma_mas_remove(struct vm_area_struct *vma, struct ma_state *mas) { + vma_write_lock(vma); trace_vma_mas_szero(mas->tree, vma->vm_start, vma->vm_end - 1); mas->index = vma->vm_start; mas->last = vma->vm_end - 1; @@ -2300,6 +2301,7 @@ int split_vma(struct mm_struct *mm, struct vm_area_struct *vma, static inline int munmap_sidetree(struct vm_area_struct *vma, struct ma_state *mas_detach) { + vma_write_lock(vma); mas_set_range(mas_detach, vma->vm_start, vma->vm_end - 1); if (mas_store_gfp(mas_detach, vma, GFP_KERNEL)) return -ENOMEM; diff --git a/mm/nommu.c b/mm/nommu.c index b3154357ced5..7ae91337ef14 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -552,6 +552,7 @@ void vma_mas_store(struct vm_area_struct *vma, struct ma_state *mas) void vma_mas_remove(struct vm_area_struct *vma, struct ma_state *mas) { + vma_write_lock(vma); mas->index = vma->vm_start; mas->last = vma->vm_end - 1; mas_store_prealloc(mas, NULL); @@ -1551,6 +1552,10 @@ void exit_mmap(struct mm_struct *mm) mmap_write_lock(mm); for_each_vma(vmi, vma) { cleanup_vma_from_mm(vma); + /* + * No need to lock VMA because this is the only mm user and no + * page fault handled can race with it. + */ delete_vma(mm, vma); cond_resched(); } From patchwork Mon Jan 9 20:53:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094342 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 15D4CC54EBD for ; Mon, 9 Jan 2023 21:17:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=FnXK6AKlwiCRQivRNR/iWZqjesQvVssoUbERnL3sp+0=; b=X2tX9AsmmVbe+eXSYBVkVU+bae TNcKdCAdFR5MVPOajTPBXgWGZtHsIJlXmabSizZhIZp0vKnPyDzpoQYqmcrdxFgZ3iChPYwFQkUKx TKFR6WFKzL4tMPaf57Iqm5JTOabB6EbdqHJNcNwbrM4Vw7z3Rn6Jvd5QknFKNVn2Db169AXMSXA7G zuRzwn9vowhBFZcSX/V7TZq12cRo1/yuLvhrfwGRzHqhrneIMPHYPezLjag464d1xMKlEant0PO1Q AGWyHECfL+ROe4OFGUOCbV+AZUhiwQi4pTIvpbRbhzx4J2ztll/1uXwqDKlx3JCGc0xNaiQGEaXGr O1tM9EhQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzVA-0049JG-Ee; Mon, 09 Jan 2023 21:16:16 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzSy-0048Sy-AX for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:14:00 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=AwPKUNzvgMd9Kf7LcbfBV8kCxW3mIJvnnhSUrm6UVIM=; b=WCBfUyzgSGsnN8IuhJqDVOgaQC sqG6w+9vM2sirAGfttFqNec4WVD/n6bhOgQaNREPWyGrnOCr47Nv/3WcYo3+71pWj9LtCVly0RCTq oG5enWm033DoC4gj0KQo2aTF7DWaxSayCoyv8I3kxjThtLPrTjF4Iho81zRDZbRqGRvai1BiySaZD i0PI0XPPLcaQPadT+aSh8yF/ZyKhs2Ia8ThqpoJyv20Kd3Omw1oHcsuMxA2QAVfIFdswwrlCKvWr8 hnmR6EA1xljUsdvdqzYVMEhuBYkw0ZpmWV8anIFkvI8eNF5ueC9B6N613o/UOpxr2WU/1jV+M+YLM LiUsieXg==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pEzAC-002sqJ-1Y for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:38 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4c2d68b6969so94643927b3.7 for ; Mon, 09 Jan 2023 12:54:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AwPKUNzvgMd9Kf7LcbfBV8kCxW3mIJvnnhSUrm6UVIM=; b=PUe2nQQDxcDIZZYDvSnvU7x/WgxDknYnlEY7L4Z8C3iDtsfKJ6ipNpTRC5N0+o397w Zlr5LNQCm8jIObT2qwQm+iFFbmfCf3so91ezlzHPCZn3+26NE3G0VG8VYTX74hVikOaP hXnC1hgxqMkKtkdKCm4ja7+1OhhfuKZ8QD3Yt7TKznRnTkaA7k7w1KBOtddu/hOGuQp+ p5yPJ6DKvl3ugNtnIXuytmcOf3bXiekUrutBQkFLl092Px87ewbl/fVBINgSpD18bwdH tuw54xfQPY3gshaLVXYSV9JTTyWxRw4/COwHM+KjA25VHZQUSDq3VtesXUsv3nuWcWdU OueA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AwPKUNzvgMd9Kf7LcbfBV8kCxW3mIJvnnhSUrm6UVIM=; b=L4YYoy/e377PgOrQUWNmSVLd0hBv4ry3TD8d7V5uBDrRRqXXzmg2CJhxDU8kjkOpdB On4a1ajGLuGA+c2z4dLmRowc3HmKcJ5t1iIg4Y7eSe5jszCsASJ6JUGWEB808n9od3W2 +RTFPHFu0dxSc2mjkVZItOKL/NI9BLzAtzVBLiUcxJb86AJOtCUlwgkGAk9jwbELS73n K0IJUq5w69KBcetfHainNcHoiYUt5nZ+lt1gH1Vp/gvakQngiGW/Q13FJv2ogvJHkktc sl70S9Or4Q/x/BfANUXbpgEg4CCHaxcDGv2w9/LEF1EYsASG+5f4SSIsdeUcv+TUO9tJ FOmA== X-Gm-Message-State: AFqh2kohWrtUgZTVLOP0FUjGcz26r4L0rtF6uTSP02f1FNyDJTxiEhVw HViblJnf7fVSH30gnAgY3u9Vk2QdoCA= X-Google-Smtp-Source: AMrXdXsf9qNWBP/q4HbIX0JjhZtra/GO9q6jv0FDi4j+zY41mh+XeX3qFvEfYkTJ9YCl7ccm1I7VEz8cc3E= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a81:4702:0:b0:4cd:f764:1911 with SMTP id u2-20020a814702000000b004cdf7641911mr673145ywa.403.1673297681920; Mon, 09 Jan 2023 12:54:41 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:19 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-25-surenb@google.com> Subject: [PATCH 24/41] mm: conditionally write-lock VMA in free_pgtables From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205436_822589_92FF4CA4 X-CRM114-Status: GOOD ( 14.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Normally free_pgtables needs to lock affected VMAs except for the case when VMAs were isolated under VMA write-lock. munmap() does just that, isolating while holding appropriate locks and then downgrading mmap_lock and dropping per-VMA locks before freeing page tables. Add a parameter to free_pgtables and unmap_region for such scenario. Signed-off-by: Suren Baghdasaryan --- mm/internal.h | 2 +- mm/memory.c | 6 +++++- mm/mmap.c | 18 ++++++++++++------ 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index bcf75a8b032d..5ea4ff1a70e7 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -87,7 +87,7 @@ void folio_activate(struct folio *folio); void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *start_vma, unsigned long floor, - unsigned long ceiling); + unsigned long ceiling, bool lock_vma); void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); struct zap_details; diff --git a/mm/memory.c b/mm/memory.c index 2fabf89b2be9..9ece18548db1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -348,7 +348,7 @@ void free_pgd_range(struct mmu_gather *tlb, void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *vma, unsigned long floor, - unsigned long ceiling) + unsigned long ceiling, bool lock_vma) { MA_STATE(mas, mt, vma->vm_end, vma->vm_end); @@ -366,6 +366,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + if (lock_vma) + vma_write_lock(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); @@ -380,6 +382,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, && !is_vm_hugetlb_page(next)) { vma = next; next = mas_find(&mas, ceiling - 1); + if (lock_vma) + vma_write_lock(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); } diff --git a/mm/mmap.c b/mm/mmap.c index be289e0b693b..0d767ce043af 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -78,7 +78,7 @@ core_param(ignore_rlimit_data, ignore_rlimit_data, bool, 0644); static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, struct vm_area_struct *vma, struct vm_area_struct *prev, struct vm_area_struct *next, unsigned long start, - unsigned long end); + unsigned long end, bool lock_vma); static pgprot_t vm_pgprot_modify(pgprot_t oldprot, unsigned long vm_flags) { @@ -2202,7 +2202,7 @@ static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, struct vm_area_struct *vma, struct vm_area_struct *prev, struct vm_area_struct *next, - unsigned long start, unsigned long end) + unsigned long start, unsigned long end, bool lock_vma) { struct mmu_gather tlb; @@ -2211,7 +2211,8 @@ static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, update_hiwater_rss(mm); unmap_vmas(&tlb, mt, vma, start, end); free_pgtables(&tlb, mt, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, - next ? next->vm_start : USER_PGTABLES_CEILING); + next ? next->vm_start : USER_PGTABLES_CEILING, + lock_vma); tlb_finish_mmu(&tlb); } @@ -2468,7 +2469,11 @@ do_mas_align_munmap(struct ma_state *mas, struct vm_area_struct *vma, mmap_write_downgrade(mm); } - unmap_region(mm, &mt_detach, vma, prev, next, start, end); + /* + * We can free page tables without locking the vmas because they were + * isolated before we downgraded mmap_lock and dropped per-vma locks. + */ + unmap_region(mm, &mt_detach, vma, prev, next, start, end, !downgrade); /* Statistics and freeing VMAs */ mas_set(&mas_detach, start); remove_mt(mm, &mas_detach); @@ -2785,7 +2790,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vma->vm_file = NULL; /* Undo any partial mapping done by a device driver. */ - unmap_region(mm, mas.tree, vma, prev, next, vma->vm_start, vma->vm_end); + unmap_region(mm, mas.tree, vma, prev, next, vma->vm_start, vma->vm_end, + true); if (file && (vm_flags & VM_SHARED)) mapping_unmap_writable(file->f_mapping); free_vma: @@ -3130,7 +3136,7 @@ void exit_mmap(struct mm_struct *mm) mmap_write_lock(mm); mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, - USER_PGTABLES_CEILING); + USER_PGTABLES_CEILING, true); tlb_finish_mmu(&tlb); /* From patchwork Mon Jan 9 20:53:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A6950C54EBD for ; Mon, 9 Jan 2023 21:30:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=9oRIOKJTpAFg5Y3ApXNlswgbod3T03+QSIdMdmGTGBM=; b=drbP0fS4fLgQbU1uw00CS3Ce85 j15YzQlfdCUVKsTETzDMNyM5rCPeLOCb/ggh4D3ONl7QXttrDB9jkGiVjGa+AlRtTPFy+dMNlaJm4 aUW7uzWEYdG+zGO4liA9zdpWAGNs+9006P7bXez5UOn5hCXckG+A4tMdx5YCd5GWaWOlhTkF9KAno sFu3GaqMM08Jokb8BJZzvlxRBgMF77aMvvP9f9V9VTdgHIkH45BmfnX+1TGhifVBpw/M/Ee2qjted vAssaTrqY1o4gMAKRvm/0s13bfzvlPkB9eFfL3JR188LdaInySyq3SzF0mGYhBXLRYlDzz4vrWNTb gsBZTsYQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzhv-004Dv9-Ck; Mon, 09 Jan 2023 21:29:28 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzU6-0048n5-BM for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:15:10 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=89ITsq4nUiN8d4MMzUNaP/VKvoRvGlIDiLzssAyWMSw=; b=Ll+dujxsps85Sm94ICJ1YPNRZn j4izYUM6qKfcWX+DfP3iDfg3QeSbOTP3pCTSJV5UROKzgKh7H8XHILd7PF8PGoX8wQPsE8rKWMgy3 /mhu2z0fULHEMA48fShnse0P8ISSlEIWGvHHhxelGkuKir4RnxBn79oCLdC8hYcZ2cPAObTqGreBp tAxTCTgiEfo6rDtPCEMjla4ZDcEy2xSDViOj/dkyqMCy+fRKd0RdvTzMwjj2CijsR9idbtII+fBLJ Dg4OSIZERSQuef9dd9vC/FnQdlz58Pw4PCy/1meF9JytR0tD9ZwJ7Y7fk1oBtjWvEgMWjdQBJj2Vb nyMNoSXg==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAX-002cMq-GU for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:59 +0000 Received: by mail-yb1-xb49.google.com with SMTP id i10-20020a25f20a000000b006ea4f43c0ddso10381778ybe.21 for ; Mon, 09 Jan 2023 12:54:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=89ITsq4nUiN8d4MMzUNaP/VKvoRvGlIDiLzssAyWMSw=; b=nfh/0wSN7du3H+jbnftiNBYsH0L/ZzvAjrQDNKD6/Ux50j22Vuiz0S8HvVp3TvO1vo K/T56ndCuYOiwpC071dddEQYgomwd4Y1mEzWyHgdRsI4Zm5Szg4pGbhhjtZTIjjKwBZ2 DuLjlMuvIIQ0lWiW9Nfxk/1vqmTgUUR+Vpz2r/N+XxIUJS4L6MBvWuo91YhKUizAdsFD /Z7UYgJwWP6aH5EHzLl9AB9L/N1SliUIF0AbaZQSGUEgi4Q4Dc2tmxMGPV7cLiqwxdNW 0MiPjg6rJc2LN6ARMi2zMWyKxfOe4Y81VTcKlWoejCss5A49BpopZcqw0LB7HbZRjJwk DoGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=89ITsq4nUiN8d4MMzUNaP/VKvoRvGlIDiLzssAyWMSw=; b=xT0UFtRv08Wt7paNVj/NOtEbn80/NHpRn7l6L0Tbu2nLQm8vNMtcwhEtAHqnJrgDnp se9qJKF6Ve0dpDxNvr0XqZtazZSkHIa6SIYpfqcdXFqIbz3rxiMrQ8gQGsbfduG6Y/xL JjZ11CnDa0ibwT4ozg7JVJLOXCGCVB/Imh3ZLTG6Ypzrqo7dLnpVOIEt1t3iaq8vathH IMlbvCOSJJ/Kfl6qLQNqd8qoIyyvw1rNV0XXU2r+wQ20Sf0ZSxk2dkCsVyHJAATJjrKl xKVTVKQhBcUoH+cmBJzqlgAmrcZjpGMxNUTSF3ozd5P8aw72i7LQRy+GoJyjPYP1ngmd twTA== X-Gm-Message-State: AFqh2koTTRs1t5x+ZY7FbCy4RSTc03fVtA2ClV5bL+afcnOWZY3eo8uR 6pqHBe2ERP+B4YwAEGAOXQ3Cx+qAeFY= X-Google-Smtp-Source: AMrXdXvWplK/QqycNgSSdeBDfganGdyEUrMbsmwVLuA4csVljoZeQRUCGyXX2t9p/6nnKr16T46cKc1FR2o= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a0d:eacb:0:b0:485:126c:3834 with SMTP id t194-20020a0deacb000000b00485126c3834mr6100451ywe.360.1673297684211; Mon, 09 Jan 2023 12:54:44 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:20 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-26-surenb@google.com> Subject: [PATCH 25/41] mm/mmap: write-lock adjacent VMAs if they can grow into unmapped area From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205457_584533_9E90DF12 X-CRM114-Status: UNSURE ( 9.09 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org While unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases write-lock adjacent VMAs to prevent this growth. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 0d767ce043af..30c7d1c5206e 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2461,11 +2461,13 @@ do_mas_align_munmap(struct ma_state *mas, struct vm_area_struct *vma, * down_read(mmap_lock) and collide with the VMA we are about to unmap. */ if (downgrade) { - if (next && (next->vm_flags & VM_GROWSDOWN)) + if (next && (next->vm_flags & VM_GROWSDOWN)) { + vma_write_lock(next); downgrade = false; - else if (prev && (prev->vm_flags & VM_GROWSUP)) + } else if (prev && (prev->vm_flags & VM_GROWSUP)) { + vma_write_lock(prev); downgrade = false; - else + } else mmap_write_downgrade(mm); } From patchwork Mon Jan 9 20:53:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094364 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0F0B7C5479D for ; Mon, 9 Jan 2023 21:29:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=b2+qkaZORZZH54dd9M2fz54VY3O+rMSTYSzZM3wWlrs=; b=xDOvmVTvii7FtWkIUdu08hvag5 Qvie50B4gfbDiBwDn6NwzoiH+zrAGak4s/6kx70k006UWwJVQtHM3zxwpP2GLr/7oMoXQqC8JTAsx g/1H6Jkb4pI8arDiFEdY7og1RaXHoaYgZGiW3acLdThbd3h+P2Oq9R5Pc9rvhteaxcQlKtAONsfl+ tq6W1o2rFEvbU1k85dM2khUbCpszn+OG0+lCPPQgaU4Yh8t4ipRelukSW6QobSsuRo0z0IWv5aXBV lQMqTsmd9MBFwPkRYk0CUqk101m8gdQqkqxUhUpsjFWj+pTE6UtyyOBFHWByuLUI1dBTB8M/aWo3v 17zhNqNA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzgV-004DQY-2y; Mon, 09 Jan 2023 21:28:02 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzU5-0048n5-CP for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:15:09 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=L7MiPLggwHMt078/i5i4Bbk0h7mxL/B1TX4Loq0bJkA=; b=IH3TdoGLpZvI7N1Y7Ru1av5f2v OCbiyxym5+J9YvBPKD9o+RTHclunhrYViMHIsIbKgxt/ZAXCb3Y5iKNAbWmh/aJwSUofORM4hjpuy mCVXlQeakWk7+/uaeI0zo5cIMIBDpNsmj/GS1NV4vEdEg9n3s6pruIMoRu4b7UFjzn3tO05Sdunqo BDjiDlhx7WB0jlNwqs/evi0otaomS75vf+V3TRdQaj8/Q+DxQeRf91ka+M47M4nnOBcJ8Sv/dTgxn 4oW7HLr/J0Ap89Al1OSA2+XX0Fwaa3DqWQL6jOdGdEZyckxZOqBOC6xRCIjMNM3ngZGceC+C6DZLj ADKAEcfw==; Received: from mail-pl1-x64a.google.com ([2607:f8b0:4864:20::64a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAb-002cWc-RQ for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:55:03 +0000 Received: by mail-pl1-x64a.google.com with SMTP id t12-20020a170902b20c00b00192e3d10c5bso6997734plr.4 for ; Mon, 09 Jan 2023 12:54:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=L7MiPLggwHMt078/i5i4Bbk0h7mxL/B1TX4Loq0bJkA=; b=k/y/f651ouIVm5VYXSBuN3WGA3r0siHAwbforN7cl7XldyaXSobuMoKnfRbHsUp2Zw mT4Fy24gz9zLv4V1qPdlGMDZmt4uQYDL7SMguN64hvgl+OPwnYaemI3hYYgdYfqYy2ZE RZpsDEbxomOp03K+XTMpAlOj5/FuHalix+oD7gvliXvyOvoSifjWwtoSfq337yDgZBWx NsM/wn5sH1zgbh2ZxdX6VDsXj4dH/jKa39D017jsbV+IQ14YXnxgSRRt8KTb8ppwgMfm uVAI77HN37Vy1v+4HMEL/4fAI+3c+Ki/ub6d+F+N4Xj5NK0ZJDRKlJXLfRFrsnczSTJ7 ObUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=L7MiPLggwHMt078/i5i4Bbk0h7mxL/B1TX4Loq0bJkA=; b=lXfp2Ge2yw9G9Ar9WtgrSh6d/HKpGXFPZMK68/UQ+gIB0NJaqCqYTWLMKPTex/spgC mBBd32EX5Blm9myAhGGLTWNo+9kiohlnz9YksmbLQ/WqPz9a/UmnURFwnzuO78BVHBSg C75FZiv2Ga86migVu1ZehfIMh/iaWPaZfrrNcdCsQ7CoWddcTMeeV/Pg67xRKFBdRe3n Eg/e9u1NEYDFFE7Ky2BgYY/lxdc5pjZoqWvODNedQ0WEoeBwvRED1EykqjJtaX7/F421 WoyW3UZadXY4p/qp/4x6EVbhY1tRi+Jeo4MOf8eLxvdN6LjaUysiocb9VgCZwEkQ/dRX hzlg== X-Gm-Message-State: AFqh2kpIWW0TwIoWBJpBF2jiC5wZSzvhmFgpbamb+FYV1NPTbAmva6uq Dhky1eooJfrv6HgEAbCZ4TqgBEH7occ= X-Google-Smtp-Source: AMrXdXscxn+TZfW1mPWgpj2/8Haqh4tY58OgT8gYBU0unalxDUYH6+O0E4BOyMrkarY33KKQg8fQ3JxSbTo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a17:902:8e88:b0:190:becc:7e76 with SMTP id bg8-20020a1709028e8800b00190becc7e76mr6134633plb.1.1673297686693; Mon, 09 Jan 2023 12:54:46 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:21 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-27-surenb@google.com> Subject: [PATCH 26/41] kernel/fork: assert no VMA readers during its destruction From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205501_961591_C922F91B X-CRM114-Status: GOOD ( 10.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 ++++++++ kernel/fork.c | 2 ++ 2 files changed, 10 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 594e835bad9c..c464fc8a514c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -680,6 +680,13 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +static inline void vma_assert_no_reader(struct vm_area_struct *vma) +{ + VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock) && + vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), + vma); +} + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} @@ -688,6 +695,7 @@ static inline bool vma_read_trylock(struct vm_area_struct *vma) { return false; } static inline void vma_read_unlock(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} +static inline void vma_assert_no_reader(struct vm_area_struct *vma) {} #endif /* CONFIG_PER_VMA_LOCK */ diff --git a/kernel/fork.c b/kernel/fork.c index 1591dd8a0745..6d9f14e55ecf 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -485,6 +485,8 @@ static void __vm_area_free(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); + /* The vma should either have no lock holders or be write-locked. */ + vma_assert_no_reader(vma); kmem_cache_free(vm_area_cachep, vma); } #endif From patchwork Mon Jan 9 20:53:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5EE98C54EBD for ; Mon, 9 Jan 2023 21:18:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=5pc/qtoHmP3cwHAmxuQnYVbdYtg4s7vgLupkkslU+kM=; b=I7L2qKNpKGEAT6tE5nU2zMtnpK h8iSSWfvbq8pUk4yOt2MP1CN9uQ9ZkMGcD5y1kEm7Zu190LdliryPw57WJhd285CPPjlYS41dYeVB 3uV3Uo+h1CDBrMZjTqJA8r0yGN6Fpfo85uMToF24ip8n3i1d6xgqA1pys0fkp7ROBoFATE0ism67S CquUeNH0xoh9oa8qhHa1L3aH2aYMQ23nuy35rrxCK7Aas9/yPTXbF+aM7Rv2Gu82tseBy9e+/zfC6 j8is5XswlqTq3DzPdbcQnPHYmG8efyFfkngCAwDeHQ0DwLtGKWn9cVmgSOkQdb9PN+8r9pWCl0nOP Syq9AR5Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzVl-0049VQ-3S; Mon, 09 Jan 2023 21:16:54 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzT3-0048Ug-Hh for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:14:05 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=0pkQpuFsTWto6vflu1WV/u0ieoG+9lWN639GtltEg4c=; b=VymzhFc3clrWXmLmkaIpS4K2Vz ViXikN0rRMBFcMmV6diTWEiRpkMggRVEyRtqmksEO3XbZOJZcfAxrbS6BjU3jk89C05dgqwJqlkUR Niwn2M3bE94MW5tEEI5VSCU4IKNB6eRovsb2mCAEn2gHZuqi1G0pT3y14PtYzxXnKu25CrhB8/5O2 3bbZmnwj8GcFrbzjRrbH25AegEjVeeBfgZvzgEuhSSUbwcy3lWik2/jFUDdQYpbQTr4E+l+3CqQ2l C/mIgBgTYKWvhMMYDcQa6Ol50zb4quvtqrEm1+T3SqBFUbS5Ry6VBXpQ6IY9G78ne8Qlj6oZWfewJ YJIpxkdA==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pEzAK-002t2G-0o for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:46 +0000 Received: by mail-yb1-xb49.google.com with SMTP id k18-20020a25e812000000b0077cc9ab9dd9so10218872ybd.8 for ; Mon, 09 Jan 2023 12:54:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0pkQpuFsTWto6vflu1WV/u0ieoG+9lWN639GtltEg4c=; b=aWg1ukrhEhZM2G7oua95ritICYN4v22wUCcY0gEG4tVylqBmv829owF41k3CxkyAaw SuqbgaxpvQ3jo7YPBWRSVqYiX7AqAmWKt0XHBKUG5SuIsHysEKVzXAcfr0KHgTcUUWfl b8z+Kg1PDae7rSd6mmxN8sCBWlq1F+xCCZRkBLC5iLo/l2X3T6+e58rjDwLcesFby2PU TDwKsOLIZU7rmZo8L2TvumA953Nmin0x2oWW50UYSkBp0m/8PkZaL+q137UQ5Jg9l0HG VtNsJ9z3uJiIVmCVloCQiu4j8+XoGxPmrLqPQDbiOLf/gZJpQPJ4VggKimmoURCteWUt n6qA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0pkQpuFsTWto6vflu1WV/u0ieoG+9lWN639GtltEg4c=; b=7ZLIUnw/Vpfmzb998+bP+H1WHFctl8c5ulWk6ctK5ECLMTj2JPi9W4yfqdUVwBDEUP sxkuNKzX+1EjJnZo0KdCDT9zdkfEF4Q/3IaxsZ3Tw2nHhmK1grfEJAUQuH0YlNzN+HmL lkLXXGkgH1XIGJjkt7L0LfHmuw2iASxpoF0tP0X72VZcjMU0Gwcj8O8xSFonlRpAKcxL AUtS3YQRqJlZFtMoKkj511SqYsgr9dOcCfGaPjOaHensMHd6klHX0QZ7oQ7YHkmDAymJ Dbh4OkeTzBFpv7eGIBWKURZw5WgrdrELOHE2egHum4KUX0B3AuHdeX/WVBIch+8tB4K9 Gxpg== X-Gm-Message-State: AFqh2koc4bA5Ii3ROM8Bpmz0afcYQi5TDPCOGiiV8rTZGKJEDOdTD8oS KwNd1niUH2XOTYR8AOCzSzWPqgYkhQc= X-Google-Smtp-Source: AMrXdXvMFgW5AuRJO7w2AGyv4a3g+CMQq4rAqpoqS9oQdpJUn0mmZFftW7BbnOBV6/f8EJVvUFaCfl4rQG0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a81:e93:0:b0:3d6:233b:7292 with SMTP id 141-20020a810e93000000b003d6233b7292mr1998513ywo.476.1673297689205; Mon, 09 Jan 2023 12:54:49 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:22 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-28-surenb@google.com> Subject: [PATCH 27/41] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205444_637100_CC116AB5 X-CRM114-Status: GOOD ( 10.46 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Page fault handlers might need to fire MMU notifications while a new notifier is being registered. Modify mm_take_all_locks to write-lock all VMAs and prevent this race with fault handlers that would hold VMA locks. VMAs are locked before i_mmap_rwsem and anon_vma to keep the same locking order as in page fault handlers. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 30c7d1c5206e..a256deca0bc0 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3566,6 +3566,7 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping) * of mm/rmap.c: * - all hugetlbfs_i_mmap_rwsem_key locks (aka mapping->i_mmap_rwsem for * hugetlb mapping); + * - all vmas marked locked * - all i_mmap_rwsem locks; * - all anon_vma->rwseml * @@ -3591,6 +3592,7 @@ int mm_take_all_locks(struct mm_struct *mm) mas_for_each(&mas, vma, ULONG_MAX) { if (signal_pending(current)) goto out_unlock; + vma_write_lock(vma); if (vma->vm_file && vma->vm_file->f_mapping && is_vm_hugetlb_page(vma)) vm_lock_mapping(mm, vma->vm_file->f_mapping); @@ -3677,6 +3679,7 @@ void mm_drop_all_locks(struct mm_struct *mm) if (vma->vm_file && vma->vm_file->f_mapping) vm_unlock_mapping(vma->vm_file->f_mapping); } + vma_write_unlock_mm(mm); mutex_unlock(&mm_all_locks_mutex); } From patchwork Mon Jan 9 20:53:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094387 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7FEB8C63797 for ; Mon, 9 Jan 2023 21:37:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=OHDk8QLcD++oO6dXVHl3aPLkKw7MtfDOGq+fELM3eoY=; b=c92xeehuoajH3mLD0diNVKXN2i IHzrlxR7FdnpuJKikEjEDpUrSAC+Nuzvdovxuxn7NqGZ/As9tieVxuzheQxG1MFoE0ohMJ07J9IoM 9tqtlHX3HhipTGgZAFrUm2ORPGQduwkKLf7vVgHIdtCiJYgGOK05khemxRH+lmmZ5JnFt2pTG70Wb dVrxfekSHy6YzNgAqgvqYYQO2JIqF3lAj0NKsgfObRrYFhxi5kaoAV/AgfNrTpd96iOyH57xN2jlJ LV3fNyRVvCP6G0DMXNho4eXoQ7GtEJbeybpgbo4NhfvMZ7ey3Snaz5Gbehcsci8dDOvaW0sqpMMvK 4/YL2svA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzo4-004GC5-JS; Mon, 09 Jan 2023 21:35:49 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzUC-0048yD-Ca for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:15:18 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=UA5+XYk5SNGwY0wBqjbKeTB68Smdf1b2WfFGnAueiFI=; b=n5HoUTTXueCc4c/bBIaaApHn6h rF6oKX+somVboxm0TwndPd+Qd9FuYDcRIRbkCfqQ5n+FPQ4vbuItRCtbPLn9V/jouB/yUyuRJ12rE Qib5H7lLHR2fSPLKlo6LiyvbXP0/kZu4Ora5XY1fVY4Qimlj3pCZx+cXGzsIbej1/jlrNXsPZbpmM de2j1dASG3e2qSmT3hK0nfG6xQySE/OTiYrkP6Fq3T0Q8nD1OBgJ8UrQ4Dyk+LM5gdmXH3flAsqSx SRyyufQ73/rK/ffPaso7jOvqxVgMVdI1cuoPaD4G0iguWdWUkyp7/SViVaG7ObVHI7pdoZgH537x/ +QffGMlQ==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pEzAL-002sqL-19 for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:54:47 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id l194-20020a2525cb000000b007b411fbdc13so10391389ybl.23 for ; Mon, 09 Jan 2023 12:54:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UA5+XYk5SNGwY0wBqjbKeTB68Smdf1b2WfFGnAueiFI=; b=T55Sv4TWznVzBLEvpmky/jAR0QJoeEeCX90o+hjvZVdyHXxi4X+dLVqm78fwsxY8qM 7mstQqJ2q4Sn5cjstBMo6qt6HReHOpVXqnm5LMKSQ635M0MCSPIfIgxAkOTHc8oB12I/ ai6kXEsVt00A/xCygYMywSMRtpTqMTCmAJNGp6/HS0isXJekWM6FOqzBH2+trv87L4rS u+h7CTRZzB07Ow9ioaR2euNV+QdGF1fgPh0hG8fHh3bLFfxrE2kcC3qaflJTUkk0dQTh 8vb62xpka2Ra5fG5NjEzQJYLTJMKCpwH3JBwVijZjrLZOBQ13qNu1ZZEA2iHn/6uZCd9 oE+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UA5+XYk5SNGwY0wBqjbKeTB68Smdf1b2WfFGnAueiFI=; b=phAZsZtsN5W9KufMFQIUGBrntMoE4JuaCNYmvhImRzneVKXquAy/un15qV8KZGzQWa dT/PjI0QRspDULr+ycYih4kXS3laTaMIh2P6RskoUPL0AfrJXOZsdMLf5rX/LczeXt8C XZOFzJYjHAH2nD6jI02U5HxLuT4rl0ArANo7XKapQB/EMIySapGb+RYxKYKGQz/6veBz o8vwyEfUISX9cMwnQZs4xmymMWLpqcS88wMuTAMvCo55dXKx1kuSafsb2mdH5ohH98Gk cyRHRyWj1wvLxWCinxq3jmOmR0H7y5zfOSYHKKEYowwHM2s5omB9OIoCzRMBo1OrAls2 vcXA== X-Gm-Message-State: AFqh2kpkvJLODPH89Eh2OOFmX71dG3W+0VlP4atZvAX1olF+O0CP/4Ws IUY0syx/AfEtBCp16AnH6LaNbNujzSk= X-Google-Smtp-Source: AMrXdXsg8SKnvt1lG56Fk9uUJaowgWL0vbqBlxMLAmQhblw+uvVMcBKNRkD6OuTHfHWzX2ebIh14gNZU+Cs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a05:6902:920:b0:781:eeed:9abf with SMTP id bu32-20020a056902092000b00781eeed9abfmr7003326ybb.538.1673297691543; Mon, 09 Jan 2023 12:54:51 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:23 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-29-surenb@google.com> Subject: [PATCH 28/41] mm: introduce lock_vma_under_rcu to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205445_573819_34828C08 X-CRM114-Status: GOOD ( 16.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce lock_vma_under_rcu function to lookup and lock a VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. For now only anonymous VMAs can be searched this way. In other cases the function returns NULL. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index c464fc8a514c..d0fddf6a1de9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -687,6 +687,9 @@ static inline void vma_assert_no_reader(struct vm_area_struct *vma) vma); } +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 9ece18548db1..a658e26d965d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5242,6 +5242,57 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +/* + * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be + * stable and not isolated. If the VMA is not found or is being modified the + * function returns NULL. + */ +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + MA_STATE(mas, &mm->mm_mt, address, address); + struct vm_area_struct *vma, *validate; + + rcu_read_lock(); + vma = mas_walk(&mas); +retry: + if (!vma) + goto inval; + + /* Only anonymous vmas are supported for now */ + if (!vma_is_anonymous(vma)) + goto inval; + + if (!vma_read_trylock(vma)) + goto inval; + + /* Check since vm_start/vm_end might change before we lock the VMA */ + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { + vma_read_unlock(vma); + goto inval; + } + + /* Check if the VMA got isolated after we found it */ + mas.index = address; + validate = mas_walk(&mas); + if (validate != vma) { + vma_read_unlock(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + /* The area was replaced with another one. */ + vma = validate; + goto retry; + } + + rcu_read_unlock(); + return vma; +inval: + rcu_read_unlock(); + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. From patchwork Mon Jan 9 20:53:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F2995C5479D for ; Mon, 9 Jan 2023 21:28:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=AjteDnP3EsyiyHq24nRikWBw0C3FOxIAdjz/8s3vSl0=; b=CkE87cEWt3KfxPstULZ4gCzndA sBHHNvsoA/TN3G+uhJBFpn73ehgmfor4onKKvTpRe6uQhKymYuB9nRYDhzf9zkH4vMniJKpbzRZ+4 ZA8GvuBuQbcBy0A1WQlGh/3KrKdgVKCnKuFw5BRzz1lJ0dTYnQt7lUeFV76x78GoKKeAzijBMvgA+ N7Ve2kWxdCUurd8rUFst71FysimH9eHKTjzPWgwCjaojPL5lV9FozWLh7KFw/FXuwyg3F780HlegP 25vH6sR5HsnNKpm4E2ubSF0hUxZbAzOZLKSARYdi85GL8fReSVqFXk3uj+y/OBraBrvMvaurag5no CSWBZdxw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzfX-004D66-Cp; Mon, 09 Jan 2023 21:27:00 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzTy-0048n5-9Z for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:15:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=Bx36nWIWIaMqsyKC4DFh9Dy6zgkZnlvdwV69ikS09tQ=; b=utpV8Eel2heSYHtx24gFs0e3Zq ybQ8x/3h2vQ6sx4FCrkJUo5n2+Ld7jMMBmf5UVhnaOM9XfoWPzTHbSumaCekws3+LqhkmNOfh82vv o3JGxnFOJymbibdODDpWaKKFHYppL9ROBpnSCGa7r0SjJWnmf2GIJoqWg1T4YmLHTFum5YDM4XF6I Zlg3n7RVOZIguOBiVPNreL1pKXx7ukFbBzByTFkwpwgyP0/P5+owd1Cn3q+h3EiR6SCjWXsVljiIG oxqjp9q6yprQCD3l3J2s5druljL7Ay066UUTZMR11fuIwAeFUk29HDbe8Daof8yyA+t8in1A2PQIF zGjS8oyw==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAi-002cb4-F3 for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:55:11 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 11-20020a25050b000000b007b7968a6423so10317763ybf.13 for ; Mon, 09 Jan 2023 12:54:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Bx36nWIWIaMqsyKC4DFh9Dy6zgkZnlvdwV69ikS09tQ=; b=GjzlXD654bxHD8qHg+BCMZfuuyb19mHGKgBfyrdYtuHPZI3RXY2mylK8MrrZ6vi/KV XpBGPAViKdrJ111khQK0LarqUyqdysojbfZLW54fo2JQeFn5ElZgOwEGNBdof4Exx32h oce1agjXuGHjstOPH9icNCzJQUnGWQYtYgBGU792WEzYIKYlpSXiCOhWF2Ne+MX/DqXf xrifjQ0boHWSv5tXEGAlsRbW4dbyfRz7jLqAVJ0lSkujXDjZ2O7m0HOdnuYL7/iH8Jk5 Up4mHJat+wejXPO4fCLDWlYHvfZ/+JBFjf5EhWbTUQtFaZy2JybPs/XeOQ27EyoIyzsK wpvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Bx36nWIWIaMqsyKC4DFh9Dy6zgkZnlvdwV69ikS09tQ=; b=mFXRC9oafdM/ztB4f9T8NLY1xGl9GHV8xs7F5h/d5Oo9RQVzL6XfA9upC047XFjc7v GvvKG6cd5YQcLMQuonHSJGxqlzrCNR7PaOkHsb+xDCGf96LR/B7eP1VGS22OWtnjI7Qk m940AM0fRXQleH8f+WsGrbzWKH+q4k2gGAvjtxR3QeStzyUT419cqrGnt3uqiW8fJSyt 6G6+isN9SmWTaD6I1hNaEFB5kdVxZFQz+esLxenPavNFg1EKJWXE95Mr/ds+qmQ+tuOY yF+9c9d4qy9kQ3c0FqWx1/Xm1rgbDC7a/GLqyLF5qaeMbnO7CazmU8CEo+VS0N70Leqb YvcQ== X-Gm-Message-State: AFqh2kpax7O6V4AncAGsBNhyaXfzcyCN6/YR6Qh3eFIckYt875UYVj+K BzaatXeCV6NKECAMd7VI6CECLBcaMPA= X-Google-Smtp-Source: AMrXdXsUnoeDnhtdWoii8DMH8rCFUM9XgPs0a9yaOqp3Qv/VXJUnVpuxBD4Is/HyplWdMowbocjku3a0aLg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:3c83:0:b0:700:604:2e1b with SMTP id j125-20020a253c83000000b0070006042e1bmr7014317yba.246.1673297694067; Mon, 09 Jan 2023 12:54:54 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:24 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-30-surenb@google.com> Subject: [PATCH 29/41] mm: fall back to mmap_lock if vma->anon_vma is not yet set From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205508_505994_5F8FE956 X-CRM114-Status: UNSURE ( 9.40 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When vma->anon_vma is not set, pagefault handler will set it by either reusing anon_vma of an adjacent VMA if VMAs are compatible or by allocating a new one. find_mergeable_anon_vma() walks VMA tree to find a compatible adjacent VMA and that requires not only the faulting VMA to be stable but also the tree structure and other VMAs inside that tree. Therefore locking just the faulting VMA is not enough for this search. Fall back to taking mmap_lock when vma->anon_vma is not set. This situation happens only on the first page fault and should not affect overall performance. Signed-off-by: Suren Baghdasaryan --- mm/memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index a658e26d965d..2560524ad7f4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5264,6 +5264,10 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_is_anonymous(vma)) goto inval; + /* find_mergeable_anon_vma uses adjacent vmas which are not locked */ + if (!vma->anon_vma) + goto inval; + if (!vma_read_trylock(vma)) goto inval; From patchwork Mon Jan 9 20:53:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094362 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 251F7C5479D for ; Mon, 9 Jan 2023 21:27:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=YIy8hAXE4sD/cSUrH3zs84jeSChw8j8tDxfSGONNaNU=; b=Ge4vINtT6K29jhcvvwPFYhUmWw 88wxmM19LQvqH4lVYO8qahTMn1anKFwC4Fif4ubfSi/kILYrWZkuCC7u8ad6oADU3kganpGY7UjK0 knuPJEGBn2s7N2qdoNIsl//n4sMY8MCqbe7GBwR9Mqdk0GeWHuvqg+xdvTG18SrpfwiYIjRXwAX5G /HQrFQdfekP8cHcAyTSFk1OlWyVoQap7j1Wnn8u/xk2zwToBRUzEVsUS+XTDUUW9Hi+CfGr4fA18S vNLc9iUDUiUbkvVFvylykQWCwIqDG9IcunBcwKtksAL7ZGEGwL2W4L3UKqob9R2Szig2ixIGYWkMi ZyReHxuQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzee-004ClY-NI; Mon, 09 Jan 2023 21:26:04 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzTx-0048n5-1i for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:15:01 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=B6J4oGWYAsMb5e7sz4HwYEXURXCKE081WlUNIaNvLis=; b=Wi9tMYIYMNg+ZMArsITF1IjwEH 36c+fx2NhyEiFn8sN8GFuTZWMayobvqFgJsvNks7YPQv/9UK6GA4R+691NfoU+QajIrF9GOSf8+wm Vz1aknQXhTlxw1YKt8LdZm+RhtUYBuP1OmK3ZwOjmeo4mDl6+rhjR3mnO0tPWXnGV5umIPcMMsj+8 KyVWlYNXCdC6YrUU78Mr4EN/RWL4el0Girjw0PGGxKsntk8mpbDdVGt91FydElE9dZErjJHWVWDqM kcd6dOT2AUU9ytgBtNpVq0FDq8llGvakZ2Q/+PMKD7EmPsDi3eGgmfvY4nyRWYCfdViA3Bwp/qyJv v+8lzmIA==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAm-002cbq-5K for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:55:14 +0000 Received: by mail-yb1-xb49.google.com with SMTP id h66-20020a252145000000b0071a7340eea9so10193184ybh.6 for ; Mon, 09 Jan 2023 12:54:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=B6J4oGWYAsMb5e7sz4HwYEXURXCKE081WlUNIaNvLis=; b=RmMHlCmm38xVKeEsK8n8bfn/aY3rbZqU5QjWJwlg7PYpTvR8g1QwX79gxnClyI+zIW JCQh5e4SjFQ2Q8nxF5bjIKp19Ks1VtVZP+kR4i8xND+6Fj+6CTJ8vlU6n1mfUhVhmmz7 +sE7byDQX+/dNdDepWb8tj8CSiQt6RCyFFAm480IRiAV2JLM0c7a5e0c36lCq09bYHwP wg+XvRL1iHdJUlNZBka6/ieT4oXpnPfG2/UuJktrbb0wvhMWj6jl5RcTqQ78qy0Rsk8c 4BQSRlXhCecUdC5ZETwJhxsGpmEcobjcEL9EE3pAQY+ImA2TDfMr31CaGMQDdx5w7rZe dkaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=B6J4oGWYAsMb5e7sz4HwYEXURXCKE081WlUNIaNvLis=; b=Jk+BeR5rmMv0UWpPlziYt5UcqjSRYbpow7aTwYmVegxFMPdWtFptUpsx68IkuoXRws T0sgYKd5XMzH0Myj5KRXetkxLIwlft5I/PyoftqxKdg4Lj/u6WbLj4RreWUtGU7oVzbQ TY7ZevxY+Ow0eCaJIeODn7x8+OCF8as6IPDo+hdeqXAMiP6t4J3CCgD7IR5zFEf56Gan /pNR0TS7tLuvBQbKsHSmL3fFD3CMJNGqYTMweJVRu5UA6fAIyAWUzwtH3/UjcvS9oQXN Dz0UNETOPeS/hJQBBVoH13sI7GMuuo198kqPQ6uYcja2XYapKm9byRFTprcwObq4XGNL q8Fw== X-Gm-Message-State: AFqh2kq9uqO1CZHJI6+60utIBwVKThOc90Bq9C5t5YhpzEB1VM4CMhHV LKlQ00/8/crg76EZ0WbINLHgmeVUSy8= X-Google-Smtp-Source: AMrXdXvwV2DJ0LbEXE5ZgdwVnwlID+Hh7Y6icoi6WDsC3E+JAvvNZYvjiQv0jw9VN21R/xnRr9wwO4IvIuA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:e7c7:0:b0:7bf:cf52:ede5 with SMTP id e190-20020a25e7c7000000b007bfcf52ede5mr196156ybh.626.1673297696306; Mon, 09 Jan 2023 12:54:56 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:25 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-31-surenb@google.com> Subject: [PATCH 30/41] mm: add FAULT_FLAG_VMA_LOCK flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205512_219314_E82AFC97 X-CRM114-Status: GOOD ( 10.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d0fddf6a1de9..2e3be1d45371 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -467,7 +467,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_VMA_LOCK, "VMA_LOCK" } /* * vm_fault is filled by the pagefault handler and passed to the vma's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0d27edd3e63a..fce9113d979c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1103,6 +1103,7 @@ enum fault_flag { FAULT_FLAG_INTERRUPTIBLE = 1 << 9, FAULT_FLAG_UNSHARE = 1 << 10, FAULT_FLAG_ORIG_PTE_VALID = 1 << 11, + FAULT_FLAG_VMA_LOCK = 1 << 12, }; typedef unsigned int __bitwise zap_flags_t; From patchwork Mon Jan 9 20:53:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A4220C5479D for ; Mon, 9 Jan 2023 21:26:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=rjuUD6ICvDqyI1wxhqBWIw0uHpDf3LVoCWa9dBdclA4=; b=wazugCZRcHbxlCA6osgA8anh/e 3V9ACjpIYk8esWfuPUUdNNqth08xkpTrPKgFXazHkKKpwnMxeMKR5ROsQoPyTf0sw84bqXhfrwRws MFMN11SPhdHOUkI/leP5DKu163mE+NPoah4JZ78CYKJbKK0+zA0r5kLsgj88gJitdV6vdjUzWD05Z tpNVYUUZz0vUdjPlpbcfGChBS509JiuSk8/Ml3ACbrQgGqQu+oRCokC8XYNe4+TchgOfRhVrbHLo8 xwS2ilu0CW+fk6L/K7ImtHC7cJzMd+pMoTbAEqNynHwXXY/qvjUdGECna69VHEfvBRu0jgEK6czHI F8QPPwfg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzdv-004CSS-1j; Mon, 09 Jan 2023 21:25:20 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzTw-0048n5-Hx for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:15:00 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=wzUeswsih4PBYsAGoTi9IuotyhbGAzkWIDmV0FovK6Q=; b=tb2YcdNH2m2TR8m6/HxYcN6XvS vX2dGH8wUEPL1cTwGsSulidgPMmSOPctBXsuiUV0+R/WHJnM9IDYg9F7+Y/ApWI5uvAf+ZHj0mOZ7 o9mRRr6bCmLfieIZTZmAZQYozqFa/vQLKv/2Rv5LU4k7RhYBJCTR13ZrxKzaPi+4DfRdwhT0ltzum xcHXENQHfd5ir9xgyOYh5jK92euxizxDFsHB3JMbUwxvNwvnJ8k14ai/YW5XCyBbHzT+cv9xDBbql 1rThdTzYOLoSP4a6BjiRlFsFhpPCeiqgPv6N3nVXyFy9oCuex2Awcu1fjs44PB12z5JfLhNRZqcLz 3zrXGS7A==; Received: from mail-pl1-x64a.google.com ([2607:f8b0:4864:20::64a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAo-002ccT-3k for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:55:16 +0000 Received: by mail-pl1-x64a.google.com with SMTP id l15-20020a170903244f00b001927c3a0055so6951008pls.6 for ; Mon, 09 Jan 2023 12:55:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wzUeswsih4PBYsAGoTi9IuotyhbGAzkWIDmV0FovK6Q=; b=i8mCfCut8WK6+zjWPXPrBAn0OVK3TZZqUir2pgpkQNWhenu43X3/akUsts/8WrGL5U Ag1lqNnPrK3Lht8c6bYeEHv2p/KYkV84L0N5RKup3Lqh7FNgEiwz2qb9iAaK8aobnie5 SWRp3Ej3gvy6Gx0qa+UPT0Wm9SDIcbH9qbkEiBwV9a/y0P6JGxAF3x2bjosqkuWs4x2Q 7wdZhW31F8FFO8VMgBTx/6RIzfF9caT2DmPvdENgd9fKarg0jWwQoD/+83rrhQbpzEQp l4T3tmRoZNr6BoK1+IwDWlh9AP60sfKufgosjp9Mre5rIBM3jgdbCDpmS0FgJglfxzEq wvVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wzUeswsih4PBYsAGoTi9IuotyhbGAzkWIDmV0FovK6Q=; b=DwC40jmDt88KK0ABgdnReMPiNp3j/EenP9nRquO81/NHA2R5TD8Y2EZPpb8+z2sYyB pHJRi8wSgZH+NhlZrWoJ7SWSPD3Lb74Cr0D4FvmgLwMgKedXaqiBx4zc64fG0eQmhFqw nVkvR/cauS11HKIJ9632JNF21NKj+1/LnE4HmqcHYVuBZYrwCV4UOheA8P82SOoE4xlk mOWDGSwcC7RvgFVxoktdEST9OWaXqwM5muOMIQIHAamePJTNJHQ+rHWmokc42/OuyrBo FPaaVbDEXKa1B6vs0bHr9V3/miZ6t5rBovvhQZLFVeQ6QuwY5flDb+c/0cDELzeYXePf JAWQ== X-Gm-Message-State: AFqh2kq4r8fZugT1UgPT41BMeJCdPKkYR6xDf9x6Sf/ubo8qOY1gLwRA gvFwhOkShM8Mf9w9essfuCJTQdm6B5Y= X-Google-Smtp-Source: AMrXdXtXcDh9ACK+yDPMFgwqnCidD6d48HBoOS8XbTQx42HB6Mm5GZSyonzBvXPdXaN+wlg8/pxMUTj5geU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a17:902:7d8a:b0:189:754b:9d9c with SMTP id a10-20020a1709027d8a00b00189754b9d9cmr4530740plm.120.1673297699405; Mon, 09 Jan 2023 12:54:59 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:26 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-32-surenb@google.com> Subject: [PATCH 31/41] mm: prevent do_swap_page from handling page faults under VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205514_164214_B18EF204 X-CRM114-Status: UNSURE ( 7.99 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 2560524ad7f4..20806bc8b4eb 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3707,6 +3707,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!pte_unmap_same(vmf)) goto out; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + ret = VM_FAULT_RETRY; + goto out; + } + entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (is_migration_entry(entry)) { From patchwork Mon Jan 9 20:53:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4BFD4C5479D for ; Mon, 9 Jan 2023 21:25:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=W/p4Y2WMLJnwDHPgPFceehqqNlzOMaGfpCaP/pE11dg=; b=jGraXo/XOVJvazsz8HI3lzifN6 sdgUqHUQB6hbMqgoQNGd/GStyckjsi2eQXZhOX6dJhJzMZfblkvtEhyH9tN6osTyFjSAdvzG5YN0k 8laZ2MwQUJ03A9yqORtuVLiFemXP93w2xtconQ2Kz7FG6pVFS2Ia1PmFizgPjbDvdrO7mZGD8FaKO A1JEnqi2MAHAG+iC0ruTzT57yXwaj20fnSbqeNBsU1FUIK0mlogk0Gr9YD7MzrSOSmmC5w9mTgi7O 9JiBw7Jt1Dl8HucW2/59Wc/YLgP3IcA1GihtsnbZijUrFI7HFCjIVriyXH7znI6DmcCfR0vgSytfW voD+dnTA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzca-004C2X-U4; Mon, 09 Jan 2023 21:23:57 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzTw-0048n5-3r for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:15:00 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=kEbjJObwF3X8F3sFkf3mioj7FiTBVSLxL3AOZVg1QSY=; b=gFXsL7mNWPKXbt1F3rpphy0QBB C9KfFSODTMsdnQjppwdNGX+QzTvEF81z7HFxOqT59U1Kq3roCUlnljGqVWSbUXwO+Af1MSygKhi2P inebiiTVahHvD6LtLWCj6ZEazgkPHED3aA7I4xDDEnRQX1rbBYJah7YCPtDy44G7Z0Ea9QAEor3uF FW4IguR34Bsdyg5MC1v4MDWt5B1GensXiv7RFu/tglCYNDUZDx55g5BBDc9uLVHrfePJpPoL/Vzb2 lABc9/RtkaVhGFk7dj3R1Faxmn98765y10tGIUDJbjP4mtweTmykxd7jKhVaLWHYRcRIc1Erbhaex Wn4TqZlw==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAo-002cMq-NV for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:55:17 +0000 Received: by mail-yb1-xb49.google.com with SMTP id i10-20020a25f20a000000b006ea4f43c0ddso10382648ybe.21 for ; Mon, 09 Jan 2023 12:55:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kEbjJObwF3X8F3sFkf3mioj7FiTBVSLxL3AOZVg1QSY=; b=b707Zl9+U9/mZ9c+XkHaueCkegKOCNYI/oVAEusRFU63I3srOxecFlDkxyKx//4mW5 JwP5obEgccip2H4R9+fiuFL4CW2k5fc/GSQhSK1HsVxo3ZKjTR0Lux66eUg0E3PQlBTw wXh6GkCABKuopnTH7q3BfoKfyBQ/HE6fsvI1JacGKFSqz7wW0ul/c07mqYyM4k4N+tIJ sPYdgU87tyvftSSm83lgw37DswrvUBJMsShqXbiuQjXDGVNwSgoByZHvozZ1Xmfis7g3 cB964lLozfT6WEMDCdFsl0vR2EiUHB+3pHP9fkmg+z92l6SffusdjR7B04cQIU05PEZE 4Ang== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kEbjJObwF3X8F3sFkf3mioj7FiTBVSLxL3AOZVg1QSY=; b=KUMEcO5fgO1uUYfDy3zO/vrP6oDbOcffSue1edZVpEh3AF7pqhmNpSYRtTxVG5auZx 06ap6I7PUDGKAnxMPOl41c0pOViR9w+x/+cPdWVdGs5xQ3kgSW3okZLT+KYtNIgxwjbk nfUIOgj4aDHK/CxHNPP/Hh+DkOuuINJ9qDo0uFzlUzpAHArd/iBA5mJLoSr2r/uTDX3f elR3lp9gcXFZLtkNT9FSEtpZ1/hBi+SqGXOp8GvdIjNxI2+B3xvNvrwxy0faWpopQRiu WghJMrq7XZ4qGWMOAXvlB/sWpx4Et/y0b8fOCfAFjE2kNtwxbGYEyO/ZCt3FKeGcquh9 Y6Tg== X-Gm-Message-State: AFqh2kqRUjJfN8meXazeeGwBDE8JLrqbTcOzX81ojavd8vNf3aBd+xyT KAbIgu+VsWyZGcdF150L9p44CrcLKzM= X-Google-Smtp-Source: AMrXdXuMlfQ+hsH5bYnRbFVRQKfHDnoMxNv1bo1mLunTWqhQUT5+3qxDxTtfm+RaU7eYfnfY1D8yBkPOjiE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:9304:0:b0:7bf:ffcb:79d7 with SMTP id f4-20020a259304000000b007bfffcb79d7mr163411ybo.446.1673297701437; Mon, 09 Jan 2023 12:55:01 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:27 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-33-surenb@google.com> Subject: [PATCH 32/41] mm: prevent userfaults to be handled under per-vma lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205514_771890_AF636BFC X-CRM114-Status: UNSURE ( 7.65 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Due to the possibility of handle_userfault dropping mmap_lock, avoid fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Suggested-by: Peter Xu --- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 20806bc8b4eb..12508f4d845a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5273,6 +5273,13 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma->anon_vma) goto inval; + /* + * Due to the possibility of userfault handler dropping mmap_lock, avoid + * it for now and fall back to page fault handling under mmap_lock. + */ + if (userfaultfd_armed(vma)) + goto inval; + if (!vma_read_trylock(vma)) goto inval; From patchwork Mon Jan 9 20:53:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094358 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 968E7C5479D for ; Mon, 9 Jan 2023 21:24:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=fJFfzeC4Qbcffj4BvM5Dh/x5h89M9wYKieW/Kjb+h0A=; b=iy4y4ScTdBmml7hiaUDG/waw4D ZiGuofo5P19ZXHmxqSbNgNksuTP8HZSRVpdM+Nd8VBqndOyz9rW7Huev1J98Le2YX4pmSoyToIYyo qeUYfbqOrRhrxhkouaogb+U0KDGiLSPeyWVZvsqbWfLNpO01rLC6H5X0N53z9xEAGxpjhiuasUg+e g+PvL4dQYEHF7dn3Zob8DgLEbHbISZzqN593D2hvo11ADe3HnXcPmbTwUHmoc/0zKMoqMctKOz4rM VhaHPxMeRLrcuhjaXQzjCiMdWgvLpCU9l+D3KzimQBGQLWLs8kOkq5obx86ug9chTOcwWVWOKqbed zpoPYKHg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzbn-004Bll-I9; Mon, 09 Jan 2023 21:23:08 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzTv-0048n5-NW for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:14:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=mbt4mHoq0eWZO149X3WH8AIvdwQHHBNP+0MBWdKikso=; b=rICjo/W+aPxX+TUWpXY21SxO60 0z5kSpslM6wOyR+rfcl3N/j9e4oNFmOvS9uBP+KbWJ7mo0HKgCRv8k0OgkntjYRO5Gs4FRdTPd29m V7hDbOfalcdpHcMGLaX4Mm183QL4WFuYDLa1YQYto2KE8xvBuJQ61q1TYzdY2GToLxxmeqO11Xjpo h20Cs9M7vNFygjQqCJCjqv0C7tQpwAPdGjfcAP5YMUPTWTGwOjlQ+wYYQ0Q2HfvbHOPgZd9SJsRYa BHWhcklAKivQvQw0Yn6ojoIZCkvb8EX5NtdDrO0LZeSjc9uEBroph2M30ykXCpzCvN3E3AkJVGPXA k4A8koTA==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAs-002cdp-8z for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:55:20 +0000 Received: by mail-yb1-xb49.google.com with SMTP id g9-20020a25bdc9000000b0073727a20239so10202961ybk.4 for ; Mon, 09 Jan 2023 12:55:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mbt4mHoq0eWZO149X3WH8AIvdwQHHBNP+0MBWdKikso=; b=I5umo4kG0hMiq4gs5OujcHd7Sn1GGJbiybdYQYTj0HAR3QOkD0OA35qcaeXUMRzGAJ dIG3EBC52wC5ibcKgTHA/19I2NomM7CaXPc78eVykHWlIF2Vt/KiNizKg3yGh0eNsGHP jtagKS1D/qcsRmP7Fqc869E/kUJC8aHONABQKFab5CUW1r0+y/kJjLtE/M8OXtv+HWMx 0jlTEySwEhV749+tPTyOU0MfX8AR34H6BYDnkbNQmdajk1IIrv1WChV6C7Fwj8u25gID LgF2w8/r6WitNZ0Roa2WVZN3Fbrz2yHOP70XPp57APsjpBjV3GkkefNK2oUkx1yrmoLm XvuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mbt4mHoq0eWZO149X3WH8AIvdwQHHBNP+0MBWdKikso=; b=jAQxRIviMsYAEy6/Jchww4M4o6WO2nsqo6ur+5wT5v2LHkNLoBt32P3IIVvdte/OoK cydkwRC5jnx0xEC1tINuaad0E1fxANIPydWsT109tvq530RNyoiVkKalGdCdxk4nwOTI H1EqhWB0lLgpCV4u8b6t0NfQF8SMDS9AwWDK5zAT5zGY3iNwUNScpw+IuvVL6VqvJZ3L rkWBG97i0xuQbj5DbjRDxzlFxiSvn8IfgUt/xOp/c9hwyLjaOoFLLwnzFj53GovMFV9e Ly4k57J09LySbKa+Il8uDWMEYtghcnvSFZFOgkQ68k2O74BBFepTEM1KNU7ZuV+erhu+ 20Xg== X-Gm-Message-State: AFqh2kpRVHrnVe4qANB6vanMkQo33RRTUpPH2KxkPUkSj3hNQYOLNDuw MbqM0aZk/NVFv67/w5+mEKZkj0bIAf8= X-Google-Smtp-Source: AMrXdXtYuIZxYJtCLtoqxn2YJzAdgg4PkVPb1cvNxFA+XZbpVDno2DZhvITBpFn/vgBWgY0+QZ+ehdsSMWQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:9d82:0:b0:774:df47:b5f with SMTP id v2-20020a259d82000000b00774df470b5fmr6154650ybp.405.1673297703709; Mon, 09 Jan 2023 12:55:03 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:28 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-34-surenb@google.com> Subject: [PATCH 33/41] mm: introduce per-VMA lock statistics From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205518_330982_42F7843F X-CRM114-Status: UNSURE ( 8.69 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++++++ include/linux/vmstat.h | 6 ++++++ mm/Kconfig.debug | 8 ++++++++ mm/vmstat.c | 6 ++++++ 4 files changed, 26 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 7f5d1caf5890..8abfa1240040 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -149,6 +149,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, DIRECT_MAP_LEVEL3_SPLIT, +#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + VMA_LOCK_SUCCESS, + VMA_LOCK_ABORT, + VMA_LOCK_RETRY, + VMA_LOCK_MISS, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 19cf5b6892ce..fed855bae6d8 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -125,6 +125,12 @@ static inline void vm_events_fold_cpu(int cpu) #define count_vm_tlb_events(x, y) do { (void)(y); } while (0) #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS +#define count_vm_vma_lock_event(x) count_vm_event(x) +#else +#define count_vm_vma_lock_event(x) do {} while (0) +#endif + #define __count_zid_vm_events(item, zid, delta) \ __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index fca699ad1fb0..32a93b064590 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -207,3 +207,11 @@ config PTDUMP_DEBUGFS kernel. If in doubt, say N. + + +config PER_VMA_LOCK_STATS + bool "Statistics for per-vma locks" + depends on PER_VMA_LOCK + default y + help + Statistics for per-vma locks. diff --git a/mm/vmstat.c b/mm/vmstat.c index 1ea6a5ce1c41..4f1089a1860e 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1399,6 +1399,12 @@ const char * const vmstat_text[] = { "direct_map_level2_splits", "direct_map_level3_splits", #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + "vma_lock_success", + "vma_lock_abort", + "vma_lock_retry", + "vma_lock_miss", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Mon Jan 9 20:53:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3C95CC5479D for ; Mon, 9 Jan 2023 21:23:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Al4gm+LSkTeeVkMB2t36Iqw6Yhda1Ya/nF53jXMsObU=; b=gDqLkWVHBi0dnRsM8BrPM42Mqg 3gAICSV2cCfppc+UDWkqpvqYbG0uiUa9ANnjGFWrowLWCKQMMub0Undh8xKkHYsJSC6/8jfVETPoy j37jy8HthKfBbPptBHF6FgsAEEmTPSHjwNC7DzFPNLcqXoGayVxFFD1nJQV12yf+VC6X1ekhGuESH QE7AlzP54NbNYVrSZiq8Ct0BSfWOAgKl6X8IO1dfoxH1oZZo2x9aoUGT0O4PqMafierJbTka1o8PS ZsZsOYYWv7MEQssiy8i0RccQYokpGs6bsF7+9CTIjqmxmdHwMBaDP3GjW59VFoRJDNt8NjBNctF6t 5ltT8oiw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzas-004BRo-TQ; Mon, 09 Jan 2023 21:22:11 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzTu-0048n5-V1 for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:14:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=wQaK3IYB3deWxpAe9eNtEaAaK5GLZai42ZGvF8iLG5s=; b=rmZQ36G+by2Rx2UJslT5fZhh4b 7Ur+xFPtjxnz2cH/wpEOj9JYynlDTf/qE1cNqlHaJbfY6aAxhD/YU03UzyoRzZkHVjgpoESpyIVfu vG+FeIbbZcvK56qrPYKkBqosOv1UMTue8VvwntYqrPvP1bOC8+E3kFC3YoQyXMNT5pCbZTM/S47xe 2IUGP6HETqBr/bopFE4j24J6xdYWW4zmcpBjXH5strhtIjlZkgtDHZhlXRdmZi9AXa7pDDLylTlGK sb/zMLQQIosHk9SyL9y3FXOYos5i+SKhTDPomUjrJSR71IoSMyJt/11oW4ndabxfqduOlboJ/rKX8 tHAEds6Q==; Received: from mail-pf1-x44a.google.com ([2607:f8b0:4864:20::44a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAw-002ceY-5M for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:55:25 +0000 Received: by mail-pf1-x44a.google.com with SMTP id v23-20020aa78097000000b005748c087db1so3985724pff.2 for ; Mon, 09 Jan 2023 12:55:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wQaK3IYB3deWxpAe9eNtEaAaK5GLZai42ZGvF8iLG5s=; b=rw20v86zZuVQYP+Ot38AI7CSNxJo3kS0fh4+Xzkg0ua9O1bePCbmIGXyRAOceAwONa J0TZdcxQSasqUL/oBe6wf7jLdD0R+p+Bu2MPqV1YykB1eqTKi0/qjL5rdUZIcmvdelEF RBoPMFMogBGK0PrycWwKvVRHJ+Ch4XmPWscy4TJxhfZ7irUJVjF6OWFk+4uiaZ5g0a9h xVEF8daEuEJDm1wmFyKFMZlgoKpmxoJhKWB0BV89iFmA1f0B2X0D/WawWhEeTcmbbuya u6sy1rh3eP2M76YM3K6bmEc6ANBDahY+/H9HaeO8ptCkobUJsiIXuerRnexRjBXA4ux5 pQew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wQaK3IYB3deWxpAe9eNtEaAaK5GLZai42ZGvF8iLG5s=; b=UcdTYwnQg5UevJBFUrm4a9nMTmFZDK+ArxbHQbzuDDoKe3ueutMo/MCGSUzGg8yFxr 0Ql/46i37U7PD8VhzpgiiVzLjDP42rkDJqgaHKut8PtA9zo2FEpflgb29A4GKbIqe4ET y5t59ou3QGmzRihXJEwoncvFoV0qMM8KvpHzEY0wme2ptdjBXKYWFcXYjhkS1Ay6IwU0 da0obpO4NuS+WEtKls9BBbeyigzegP+zBp9O17V6GhGJkRyTSskelobKRjl4k6PZWMPg q8SyFu72mZsemwXHp7h9tjdFi6zf2pdJ4If8ZaUa0acI0Ch7XMLbqrY0J5BWxcZfDdWi qrFQ== X-Gm-Message-State: AFqh2kpg4OAfyHPYGL647jvrLlw0pv4QcHHvs/tjVTfjY2cTfulnnj9d CZvecutL/1SS/CHU04f1sJea+mU5CTI= X-Google-Smtp-Source: AMrXdXslCGRX6fAYSmsr9gs68MBt+wxNt6fnptg7xXYf9+oinmXIkRDOo+t8kmB5UcEQZNFBtsxG/pDk5Sg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a17:902:7583:b0:189:bee:65ee with SMTP id j3-20020a170902758300b001890bee65eemr3889654pll.107.1673297706216; Mon, 09 Jan 2023 12:55:06 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:29 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-35-surenb@google.com> Subject: [PATCH 34/41] x86/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205522_211922_C298FE84 X-CRM114-Status: GOOD ( 12.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig | 1 + arch/x86/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 3604074a878b..3647f7bdb110 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -27,6 +27,7 @@ config X86_64 # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 7b0d4ab894c8..983266e7c49b 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -19,6 +19,7 @@ #include /* faulthandler_disabled() */ #include /* efi_crash_gracefully_on_page_fault()*/ #include +#include /* find_and_lock_vma() */ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1354,6 +1355,38 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_error(error_code, vma))) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR, + ARCH_DEFAULT_PKEY); + return; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1454,6 +1487,9 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (likely(!(fault & VM_FAULT_ERROR))) return; From patchwork Mon Jan 9 20:53:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A6C44C5479D for ; Mon, 9 Jan 2023 20:57:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=J+MTkBXUxZhrcQZji8s9CA7vykfgbBMD4GSGMVLi6wA=; b=I2O9kpYtCOdX8CN9MdhXcAuZsN v5RaAbTiPspmV3lvjgwp8GMIJ5D349l11P0TnBSUGDwVgJprDFyC82Oydo5Nf3k5Y13iixHxte3sV BnKDcTivofS46T0VUsriniA+GrLnhkCdtacdVtL3xwK+jb1zhyNuqE+FaPH0Ohu1yU/etivNtZtrk QL9nWj9OhzSBlLPRVv0n+FRZ0+1yuhDJPE8p6sB8qUKmXUVafXoVXBMFnvB4f7DSrFM3LC8UNh7Lx iK6lnTnX3/peC+fPr2Do6UAF49i+zXDM6DOiVuw3k2l12qM6OwGQiYFWa3QhmLy8wraad/ZRIYqlt lNYjJcOQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzBW-00412R-TX; Mon, 09 Jan 2023 20:56:00 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAk-0040b4-C4 for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:55:12 +0000 Received: by mail-yb1-xb49.google.com with SMTP id w9-20020a05690210c900b007b20e8d0c99so10271682ybu.0 for ; Mon, 09 Jan 2023 12:55:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=m2UwVHuB7txMBXnAywM8cXmBB/zrDJgCqgF9heuZ7RI=; b=AELPgr/6hvWzT+1V7lD4mX315E9t1xeapwUzcEqCdKIYPiCzKk8dc+T92lAP4lgkah 7yRjN7psrT8Aldeuq4taS71dz7N2REjLgY5gl/EmOw6Bjz3k1TLxoTZM4lDuQnOU9Jya +UewXZCmP8ol69UqHnariEjrKTpgLVgEy+/Bgrxiodf6VRTx+vPl+chOix68VXTazEey Yz24CBUMjgzvQiO/GtdPs+W2QXpN/PA/WPYudNeoUrMmtRKn07dYT94LClSxQvVwRpgt N9P9H+WlrUeFi7Q8PmpAS+IgzxksFHRinwxPnTAAPwB27jZBUc2AgIvIZ8+W58HDzxG+ RgCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=m2UwVHuB7txMBXnAywM8cXmBB/zrDJgCqgF9heuZ7RI=; b=hbxLHQSgg/wssSia/bAEPFNphlBoYy9qt+abS5pLvEY3NwxriW4AHMfyo84usQXhOJ XXqPLxpyZjAYgmkTKsw3AtbJ1Ok0a8sV9ekWWKMpjlBs3292+qUJal87pmX/EA6mWI8q KWy3Lfts5XiUbFzw4wzntTD1FKVGyjVMRKpViDKW9VtDbccpJ7U2Wr0xPwvpXkY8gtTO ihKFwPxiGGz2dK2jc2GYpw/4JRvBrZBEfZys3ANgql6Ejra9/ppaii8WNIk9/mlq/qQe qbMntZq0zPgFb1l0pPsEfST3Z42RcG6BG9llXAQ7iB0bvksMJ85WDcVD0+uFpuSkJEGo 7C8g== X-Gm-Message-State: AFqh2kq4v98YiXM9h3F7Jxgr0+aFZUce2qSWZsuw85iKktfqby1j4zT+ ch6BWOFWxxPTrCIExTpGip554Kug6iY= X-Google-Smtp-Source: AMrXdXu86J9QPZ8qYzzFeKpvTLXFdkNuTVtCgOtoNpSPm2yn8vVjTEAXybleIDx2Sx/3r3Ulgd8R54EZk2M= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:b78c:0:b0:769:74cd:9c63 with SMTP id n12-20020a25b78c000000b0076974cd9c63mr6699949ybh.257.1673297708715; Mon, 09 Jan 2023 12:55:08 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:30 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-36-surenb@google.com> Subject: [PATCH 35/41] arm64/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_125510_448699_9E461A23 X-CRM114-Status: GOOD ( 14.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig | 1 + arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 03934808b2ed..829fa6d14a36 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -95,6 +95,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 596f46dabe4e..833fa8bab291 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -535,6 +535,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); +#ifdef CONFIG_PER_VMA_LOCK + struct vm_area_struct *vma; +#endif if (kprobe_page_fault(regs, esr)) return 0; @@ -585,6 +588,36 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); +#ifdef CONFIG_PER_VMA_LOCK + if (!(mm_flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, addr); + if (!vma) + goto lock_mmap; + + if (!(vma->vm_flags & vm_flags)) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -628,6 +661,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */ From patchwork Mon Jan 9 20:53:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 09371C54EBD for ; Mon, 9 Jan 2023 20:58:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=lmuaLUu6KAKo2Xhq8k+Sh7ghdzrYJUmLY2wZB+bzuvs=; b=yGrwEQ2S5BTanYkujmb9zyUUaI 9M5LgqyMc1NSAOQSn4b81lq4uJub5MZ6ArLq9qutA+TVJR9o4st+FmKXqGtAjM835FF528p5Ut2jw 3wqSzpeD1+UBoW/kky9uH6XrLhF+zMsj3Cd5LZbt15k75Tl7YgAMX4hEbwMS3liIaDJfGI5WD1t2O dIvgHktN9tz+ozgSv+t3d4Z2Y7f0n6mmkHPlwKJTaNUjb+o6s/oBAUmIxTiuiG1vM5uV5MqlouZXy gOmPKgqSCmobDG1lmMXQfXuhJfDGKpcOoGUN4d5Ic16aCIOBaZWZgJ89y/YYdA9j6PLdKE5vBZdPY EsFCT98g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzCE-0041N2-Hc; Mon, 09 Jan 2023 20:56:43 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAm-0040d0-LU for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:55:14 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id a4-20020a5b0004000000b006fdc6aaec4fso10375604ybp.20 for ; Mon, 09 Jan 2023 12:55:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=n9fXm3izjyyPtsvowAse/sqFmR2VD35eu3/dkFbABUU=; b=Lu1lZFQRLQ14pZBfGy4NklmBcU6A/6INZGZkbzsgSlo/xf+q4f3Srx8+Mwys+6coe+ Ffmm0USNrJ8AFhCvinJsbBeU/mUqellUur3KPDhj6fquokQjmdfa6asRXc95A2ZwxK0r 65ObusIEfZRdiZ2acFQ8p3bJxbaCwGG5NTSDswdl0j+wpKeIAqm5hRE9lT+/32AEgmrz B2Z5fNXwd3NAztwL7Tcvgx4jFF9uDwxQg3WbD1oe+/QJTBY53gcrTvhoGSAa89iq0kDV 4crl41ZGwNSTBBEwkTNkn6j9GWODsWMSurEgXbqgc2WbNqPAfDPrxauLcSQy2BQ/r3X3 zHyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=n9fXm3izjyyPtsvowAse/sqFmR2VD35eu3/dkFbABUU=; b=eTQqONBxcIA45DyTQN14lloyd6qcH+llYEzrlXLGHGCDbWbVTfTDxzyiHpAjegRUaJ BSKiB1DBnPWSJn2Uw84NKFzlBashZKBqWiWGEJcVxZ6ZzV1KRCaiiboWiLUro0188yYi Vl3oRzx+WxxF0p9DnzW1iOhgQipy1rEkfFQzwQoDDjpyMMBoK5JCYAVZCOEudXGMROo0 FeTb6VhwCk56OKb1MboPIC7a26mMttB49HUwX3AYMSthvCr7tBNOmK6yRhtpbuauALrI SQA6K/5QHlXPGAfRTozW2tDYYvMqCvJkGbhOhkHpip/N6Mzy5tJPcE9mQU2odsA5D3xM 80qg== X-Gm-Message-State: AFqh2ko0tTfM7rVQMfjJed9eabTjozSWgFTtGFrXiEMqm5HiFRzJdvYP 2rfhpzxkTLfijfPBnOANj3zeI+vel/c= X-Google-Smtp-Source: AMrXdXvNqK3Ohvvx3O6oqyebZlpPuhsn/AIt7Mf5iuM9SXcX9ARLOAj6UwU5PecAUl7FhPG0GswhG64wODs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:730f:0:b0:6fa:3287:8df6 with SMTP id o15-20020a25730f000000b006fa32878df6mr6834426ybc.424.1673297710804; Mon, 09 Jan 2023 12:55:10 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:31 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-37-surenb@google.com> Subject: [PATCH 36/41] powerc/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_125512_757595_414F48BB X-CRM114-Status: GOOD ( 14.51 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/powerpc/mm/fault.c | 41 ++++++++++++++++++++++++++ arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + 3 files changed, 43 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 2bef19cc1b98..f92f8956d5f2 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -469,6 +469,44 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + int rc = bad_access_pkey(regs, address, vma); + + vma_read_unlock(vma); + return rc; + } + + if (unlikely(access_error(is_write, is_exec, vma))) { + int rc = bad_access(regs, address); + + vma_read_unlock(vma); + return rc; + } + + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -545,6 +583,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, mmap_read_unlock(current->mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig index ae248a161b43..70a46acc70d6 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -16,6 +16,7 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP + select ARCH_SUPPORTS_PER_VMA_LOCK default y config OPAL_PRD diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig index a3b4d99567cb..e036a04ff1ca 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -21,6 +21,7 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB + select ARCH_SUPPORTS_PER_VMA_LOCK default y config PARAVIRT From patchwork Mon Jan 9 20:53:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094320 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 53B5CC6379F for ; Mon, 9 Jan 2023 20:58:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=jHXjp/ZAfhcCPt0/dyizlZIwOval5K5MdEMkmZAUg1Q=; b=pCETpWYeD+8O/sEllFpB1eOrx+ 42+nvIHIQyCj0gTfq9Pq5rzA4z0RP1Al5iAh4RCMPp/NECaeZ2THL9VdMmy4uls+jlR4lBHtWPsC3 Z3otld7oaQqpxFnrBXQ3lzAtsEtdsfjmIfAHLN5mVIHZP/5IABVaJOpho3GMeSAtvCZBh2EKiIquv cEZd5Cf078vE8T5toPaZic5FUEh8sFePFEcS7gYdFMngR2Wv0QVHR57BZwdQlIQ2AlkZYDha6AF+r qWEPwPIoh9LRk1NvPR7rZxvsow+m6yDwtUcMYdWParRxe2V54l2GsEWASGpx3a+rfRoNyiyjT0+9f BZ2lZjgA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzCd-0041cW-3F; Mon, 09 Jan 2023 20:57:08 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAp-0040fT-6N for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:55:17 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id y66-20020a25c845000000b00733b5049b6fso10234817ybf.3 for ; Mon, 09 Jan 2023 12:55:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JqE4Jv/Ie7jJyt+ylXoJ6xWWNYJ7mAmWvaygaHhGgmA=; b=C0RC5qgBD/V26736LQT1/agfFQE3oawoBsVodaZr+HfHKR+SKXSkxzHnqRRZUQyjCF /VKZZZ+vJ6kaEdIfo26R2t3ZnpRNnactrnv5DhopT9zYmpMMrFjwmeMN0NuvQGOG9Ma5 xgohCqYaIMlqK/c6f2BeWoXmU7iZ4/9gccjpi+n1Vpya+2P9xxgUSOybhe5uijKT6IMh I/M50QEDZcxaCjlrD2td9ExwPf2mPPj9P3UPSnAgQR/cCy9DPbczOjnWTPsq0XpCcdbi g/lQrqmdlC6kTELYjuH3lVXIdNQlERXCKK1DKiV3+4JsEhkwrKU78EAKu+F90rDI8Z8t c6mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JqE4Jv/Ie7jJyt+ylXoJ6xWWNYJ7mAmWvaygaHhGgmA=; b=7g9DU5MhS7iadTUIIxasAGoQqU1GnTrV3B7YNHPgYstMZO74W54+wTDmYJTXYKbrNL uNNgTT1/1qx+U87rh9OED0N5YKHjBYm0QU0HXssqemGOLaIQj4OtRO3NPYPH9C4oCrvb rFvB09Oo/EEcwPVKgX1ok0EIV44pIYQboSpSIddVkuJ2G1gZxwYZR+/2dQYbCOCnXn2X sGTn2kXximaKuEE6mwG6CuN8yMynI5cnwCgdTt7Ulc264cjxvZu60VrOeqbSCQNHQlQR mcPjTa8dY6ebYAOb5dioZDY4LyeFJ6tqb/ufDFA4iu0RabBw4JVyBCUwTX1mcT0c8C63 10Iw== X-Gm-Message-State: AFqh2kqkEEJwB5f2tsAy6Rm7B1y3mFzpQPJtoyHRbBVDZCg4yZQXZ5T3 Bw/MTArnYAPTqDcQMDCIIN0Oo0gjGmg= X-Google-Smtp-Source: AMrXdXsVSd5Wz07hI5hxUAzF2wzs/BFzwaLObvOXI24uMhgcUHGkQCCTKMo/jOVrGruw6pJEAc/B3Pke/WM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a0d:dbce:0:b0:4ad:7104:1f63 with SMTP id d197-20020a0ddbce000000b004ad71041f63mr3427748ywe.49.1673297713421; Mon, 09 Jan 2023 12:55:13 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:32 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-38-surenb@google.com> Subject: [PATCH 37/41] mm: introduce mod_vm_flags_nolock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_125515_277264_3250F0F1 X-CRM114-Status: UNSURE ( 9.30 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In cases when VMA flags are modified after VMA was isolated and mmap_lock was downgraded, flags modifications do not require per-VMA locking and an attempt to lock the VMA would result in an assertion because mmap write lock is not held. Introduce mod_vm_flags_nolock to be used in such situation. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2e3be1d45371..7d436a5027cc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -743,6 +743,14 @@ void clear_vm_flags(struct vm_area_struct *vma, unsigned long flags) vma->vm_flags &= ~flags; } +static inline +void mod_vm_flags_nolock(struct vm_area_struct *vma, + unsigned long set, unsigned long clear) +{ + vma->vm_flags |= set; + vma->vm_flags &= ~clear; +} + static inline void mod_vm_flags(struct vm_area_struct *vma, unsigned long set, unsigned long clear) From patchwork Mon Jan 9 20:53:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094331 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 60BB0C5479D for ; Mon, 9 Jan 2023 21:14:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=AT+WN4Pt1wurggj9Wbjp3INt10SySxrwvm2MRrHNyd0=; b=Sw8ViV7b5W5EYdlp6vgVNhL/Vg dRUU3gC9NsU9hp+NldNWlgjvPye/M46oT1B2C0c92gqacw9TvwTMfByzm7t9p1OOYK+Qk7HK7L3O7 3QqsK12BKK9ZAhdEg/sRm7aMZmlsVmQy8zC3Xdvj2soGpa7R2Lm/Jib+BZjxEiXiKgSMRYmheLltv QZlVirRF+J7opD35oYY0W1Q8szagGAWnEr/sakUzxUwMcfJUqfN9R2Di2rKQLH0xz11RESmovWsLZ 6xVvaWO7GvIxqg16x3QQXCOqFn11Mqjb//rZW6mtSu/lo9j8vsy+GDT+rqCW1SvH3Yeexd+DuTd8O xo+FIeew==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzSd-0048ML-QW; Mon, 09 Jan 2023 21:13:40 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzSY-0048LB-QG for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:13:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=iUx/OB3iQQFsS6Pcn0sWsw86mSb+m9wyFp0pF/RbXSk=; b=BaCugGSij/hlKL3seD9euVoG5K AwpUGYgRKAA2VNMPInS5QNvq4VjTs+xoS7YqQg1iCydTYVQ1JilMHxWJvdl0lodlvl4Mqf0X8npHb djr0SSEHggGcVvPgFYxYiInn8R2uAqJFzTsQRds7CoCSraEVZVBGj+4XAB/DLZymsdqjH0anfdHCz 3fYjqNkBDMz4HbMC35ByG/ewbvZJ2wPxE/kN/sd3KgUoAPnVxmHaNn15Aghvi0PpvvFdek9oDsrTg yzRbDknnRcAZ+ERQSATo2uzCZ5p3IOoo+38o6BpmGGN+Z17XeVzH39nXtyIguuIJYAjx9XZtL6ICA Xh7ESOcA==; Received: from mail-pg1-x54a.google.com ([2607:f8b0:4864:20::54a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pEzAr-002t8F-1e for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:55:24 +0000 Received: by mail-pg1-x54a.google.com with SMTP id g18-20020a63f412000000b004aef17e314cso2495447pgi.21 for ; Mon, 09 Jan 2023 12:55:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iUx/OB3iQQFsS6Pcn0sWsw86mSb+m9wyFp0pF/RbXSk=; b=dIxUWUkrPawF5nCoP+HUjvA+1vmI2x8utDxJKsWUDKbZyEnrEFBNIhzaM8Fi8ju0R6 eFijpcCpNnzwFYn4qQcfR/+HsKpxnsThGu7XxjIktqkOlh8pbTgjPjgxlF3er9jtpiFK 3GSfGJF3spx9qr3lx0Xbz1tHEn1iO2FRbitd8mVM168Ji21A6WJbwvjHXFAHlbmszk/z PHIXCswYcUFMYiFzIipRXQxj+Lq9k2DTCyxkIwApZosMOj5/aAMGnoES19O/qzXO1kO2 XId6gaep+ySgAUv8E1eSLybQz1XP2tMRZPlcvFIYudobcMhpLFz5bkGWnIsnoWm/3FYR M9Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iUx/OB3iQQFsS6Pcn0sWsw86mSb+m9wyFp0pF/RbXSk=; b=s8RCpqSvFvvfpDGEb5VVKr1rjl7zFH3DoxEkt8u0mggZANZ6sqR+tpOCVntw2PDBaV RR4OZ2gpXy6Uihba+uaL7a/Xy2eiCCeu+ZFj7FZg0Ue5kHwTDSLiUoS74ylSnMwo1US0 nAWjuevAi3N+pZa5QoUgOjowGZnP13IsR4IK16GFEFrr347+/QgPvfGOzTYfAF3KimZs XvdC371ESHXuXmuyZJZ62HLjFoxddtnF2BmA0Om+O9HiZfVGTF0zAoej3KA/cSdCBFMb GjDc0jUXol4qeTLVc1XaMlBhJyIKlF9H/b/kfK/VvlEGc4IvlBF4ArveJocRhGF7dnEn ZW+A== X-Gm-Message-State: AFqh2kpYCwifSNXpJCo7FmBXaajCL/VRxQmhzLpOvOm02vV8ureA4ugM YfWYS0d0nnp1uIjUCAJhwzhXgI2fWZw= X-Google-Smtp-Source: AMrXdXtKEO5NZ0LV26PgvCCbCCrX8iN+m+pnC68agl9KzwVT0TYKaTzsINmDv+j62IKmNfwju5kMUEnv/YA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a17:90b:370a:b0:226:f53b:d370 with SMTP id mg10-20020a17090b370a00b00226f53bd370mr875190pjb.75.1673297716576; Mon, 09 Jan 2023 12:55:16 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:33 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-39-surenb@google.com> Subject: [PATCH 38/41] mm: avoid assertion in untrack_pfn From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205522_261284_F1CAABF8 X-CRM114-Status: GOOD ( 16.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org untrack_pfn can be called after VMA was isolated and mmap_lock downgraded. An attempt to lock affected VMA would cause an assertion, therefore use mod_vm_flags_nolock in such situations. Signed-off-by: Suren Baghdasaryan --- arch/x86/mm/pat/memtype.c | 10 +++++++--- include/linux/mm.h | 2 +- include/linux/pgtable.h | 5 +++-- mm/memory.c | 15 ++++++++------- mm/memremap.c | 4 ++-- mm/mmap.c | 4 ++-- 6 files changed, 23 insertions(+), 17 deletions(-) diff --git a/arch/x86/mm/pat/memtype.c b/arch/x86/mm/pat/memtype.c index 9e490a372896..f71c8381430b 100644 --- a/arch/x86/mm/pat/memtype.c +++ b/arch/x86/mm/pat/memtype.c @@ -1045,7 +1045,7 @@ void track_pfn_insert(struct vm_area_struct *vma, pgprot_t *prot, pfn_t pfn) * can be for the entire vma (in which case pfn, size are zero). */ void untrack_pfn(struct vm_area_struct *vma, unsigned long pfn, - unsigned long size) + unsigned long size, bool lock_vma) { resource_size_t paddr; unsigned long prot; @@ -1064,8 +1064,12 @@ void untrack_pfn(struct vm_area_struct *vma, unsigned long pfn, size = vma->vm_end - vma->vm_start; } free_pfn_range(paddr, size); - if (vma) - clear_vm_flags(vma, VM_PAT); + if (vma) { + if (lock_vma) + clear_vm_flags(vma, VM_PAT); + else + mod_vm_flags_nolock(vma, 0, VM_PAT); + } } /* diff --git a/include/linux/mm.h b/include/linux/mm.h index 7d436a5027cc..3158f33e268c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2135,7 +2135,7 @@ void zap_page_range_single(struct vm_area_struct *vma, unsigned long address, unsigned long size, struct zap_details *details); void unmap_vmas(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *start_vma, unsigned long start, - unsigned long end); + unsigned long end, bool lock_vma); struct mmu_notifier_range; diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 1159b25b0542..eaa831bd675d 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1214,7 +1214,8 @@ static inline int track_pfn_copy(struct vm_area_struct *vma) * can be for the entire vma (in which case pfn, size are zero). */ static inline void untrack_pfn(struct vm_area_struct *vma, - unsigned long pfn, unsigned long size) + unsigned long pfn, unsigned long size, + bool lock_vma) { } @@ -1232,7 +1233,7 @@ extern void track_pfn_insert(struct vm_area_struct *vma, pgprot_t *prot, pfn_t pfn); extern int track_pfn_copy(struct vm_area_struct *vma); extern void untrack_pfn(struct vm_area_struct *vma, unsigned long pfn, - unsigned long size); + unsigned long size, bool lock_vma); extern void untrack_pfn_moved(struct vm_area_struct *vma); #endif diff --git a/mm/memory.c b/mm/memory.c index 12508f4d845a..5c7d5eaa60d8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1610,7 +1610,7 @@ void unmap_page_range(struct mmu_gather *tlb, static void unmap_single_vma(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, - struct zap_details *details) + struct zap_details *details, bool lock_vma) { unsigned long start = max(vma->vm_start, start_addr); unsigned long end; @@ -1625,7 +1625,7 @@ static void unmap_single_vma(struct mmu_gather *tlb, uprobe_munmap(vma, start, end); if (unlikely(vma->vm_flags & VM_PFNMAP)) - untrack_pfn(vma, 0, 0); + untrack_pfn(vma, 0, 0, lock_vma); if (start != end) { if (unlikely(is_vm_hugetlb_page(vma))) { @@ -1672,7 +1672,7 @@ static void unmap_single_vma(struct mmu_gather *tlb, */ void unmap_vmas(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *vma, unsigned long start_addr, - unsigned long end_addr) + unsigned long end_addr, bool lock_vma) { struct mmu_notifier_range range; struct zap_details details = { @@ -1686,7 +1686,8 @@ void unmap_vmas(struct mmu_gather *tlb, struct maple_tree *mt, start_addr, end_addr); mmu_notifier_invalidate_range_start(&range); do { - unmap_single_vma(tlb, vma, start_addr, end_addr, &details); + unmap_single_vma(tlb, vma, start_addr, end_addr, &details, + lock_vma); } while ((vma = mas_find(&mas, end_addr - 1)) != NULL); mmu_notifier_invalidate_range_end(&range); } @@ -1715,7 +1716,7 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start, update_hiwater_rss(vma->vm_mm); mmu_notifier_invalidate_range_start(&range); do { - unmap_single_vma(&tlb, vma, start, range.end, NULL); + unmap_single_vma(&tlb, vma, start, range.end, NULL, false); } while ((vma = mas_find(&mas, end - 1)) != NULL); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); @@ -1750,7 +1751,7 @@ void zap_page_range_single(struct vm_area_struct *vma, unsigned long address, * unmap 'address-end' not 'range.start-range.end' as range * could have been expanded for hugetlb pmd sharing. */ - unmap_single_vma(&tlb, vma, address, end, details); + unmap_single_vma(&tlb, vma, address, end, details, false); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } @@ -2519,7 +2520,7 @@ int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr, err = remap_pfn_range_notrack(vma, addr, pfn, size, prot); if (err) - untrack_pfn(vma, pfn, PAGE_ALIGN(size)); + untrack_pfn(vma, pfn, PAGE_ALIGN(size), true); return err; } EXPORT_SYMBOL(remap_pfn_range); diff --git a/mm/memremap.c b/mm/memremap.c index 08cbf54fe037..2f88f43d4a01 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -129,7 +129,7 @@ static void pageunmap_range(struct dev_pagemap *pgmap, int range_id) } mem_hotplug_done(); - untrack_pfn(NULL, PHYS_PFN(range->start), range_len(range)); + untrack_pfn(NULL, PHYS_PFN(range->start), range_len(range), true); pgmap_array_delete(range); } @@ -276,7 +276,7 @@ static int pagemap_range(struct dev_pagemap *pgmap, struct mhp_params *params, if (!is_private) kasan_remove_zero_shadow(__va(range->start), range_len(range)); err_kasan: - untrack_pfn(NULL, PHYS_PFN(range->start), range_len(range)); + untrack_pfn(NULL, PHYS_PFN(range->start), range_len(range), true); err_pfn_remap: pgmap_array_delete(range); return error; diff --git a/mm/mmap.c b/mm/mmap.c index a256deca0bc0..332af383f7cd 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2209,7 +2209,7 @@ static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, lru_add_drain(); tlb_gather_mmu(&tlb, mm); update_hiwater_rss(mm); - unmap_vmas(&tlb, mt, vma, start, end); + unmap_vmas(&tlb, mt, vma, start, end, lock_vma); free_pgtables(&tlb, mt, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, next ? next->vm_start : USER_PGTABLES_CEILING, lock_vma); @@ -3127,7 +3127,7 @@ void exit_mmap(struct mm_struct *mm) tlb_gather_mmu_fullmm(&tlb, mm); /* update_hiwater_rss(mm) here? but nobody should be looking */ /* Use ULONG_MAX here to ensure all VMAs in the mm are unmapped */ - unmap_vmas(&tlb, &mm->mm_mt, vma, 0, ULONG_MAX); + unmap_vmas(&tlb, &mm->mm_mt, vma, 0, ULONG_MAX, false); mmap_read_unlock(mm); /* From patchwork Mon Jan 9 20:53:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094356 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6604DC54EBD for ; Mon, 9 Jan 2023 21:22:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=A1F0yPZ7cwaqojEVZOq59aUxFvoHfp9cjlhispRrYDI=; b=4PSg37uQBBvnj/+HWfdSu2OGFP BzFykUs0eiDT0kw6BiaiOuCHQP8NBD3w93kGVOA5pXBd4okufnkBMxcS1PJDf4JMXhRCs0U2YaCBL neLSfv5GpzoKPOyOEYFlQlxUu+kaVNuUGw5fKiL33CPK0maJSMTt5cHn01K4wzHtZO56nEPtZ/Nk6 jbAJvtA7kMeVvAZMOIgPHHNKYyq1e61obhKvpprIzPYA21PPNMvGyjmWTTTYeaH9Wk+2gZzEd2NvO TYYwO6rh8kmkxW6nyeLldYm31cSdBvKhgHituxo1D+cS58r07UTN8iBLo8T7girW/65nxaF+7B6PB f4wRR9hQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEza0-004B98-4c; Mon, 09 Jan 2023 21:21:17 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzTu-0048n5-2Y for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:14:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=q4uicoifY7gC/4yqte16SYBfutCg5tFbw97tUXlS1Bs=; b=Mp9QbtFLkAdta6f0gP3Ofhh6jk n4W451XyCz/muMoPLOQOIq1AIz2Y6Cc3iZqbzBATTiE9+Ekc8ZEeIhmkf6WWrPSmLdruGyDbI/NvD WM2SG2o1k6wU6y6+KRDH+23B7jl0t/SAlpnWbuCCAqcMpW/812e55JVThfsJ30Gs0ZIa0BEeMtF8U vi5/0WB7n4zsUMlQ8zRQxmKwSN3ekvgkR7frSpOmwbncLJG4KLt11L3/5MZZZmFJ1VHZkf6wfYkgx /Sf9a0BTeuyXWo4PpYdKFMaDGjirSBXX1BdPqOBxAVqjx866Xwh2jFuibwVl/d9/cBSniu8EZFrZw Kn8LjcEg==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzBA-002cgE-8t for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:55:38 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-4c57d19de6fso76670857b3.20 for ; Mon, 09 Jan 2023 12:55:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=q4uicoifY7gC/4yqte16SYBfutCg5tFbw97tUXlS1Bs=; b=R+wIE/Pt4KhRz8VFgR0zMOS2JkvFmXy26q+OSQ9LjCyq0QISjHNPW3JBEjDsyfPMZn JHIhnIofZ7qGF96jAqweLgH0bhrxbWlUBYN16b0h3IUNriVtZxEflnZ/KJaCsGgJuROJ 7NtSec9kWHRW8dxlpNmOxFIq510sDiRPIwm32r6PlWpXwb1OQmeULeUN7CB+D4fk8aTq 1FAeEWt3FnKxXn+FEK6NZ2a1Nn+94uGmt6MDM7k61uLqgDX1MFlIF8o4Zu3p/TF9jRMz yr+WOD/8oTTja8WHghFFUJvljuYoTFD7DGiXYFX6wrr/dkh/UC1lDtuKJ2cCE65save6 YCEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=q4uicoifY7gC/4yqte16SYBfutCg5tFbw97tUXlS1Bs=; b=VAfbOkuGpV4hIe0eACyKv1IISNjbmKrASLeNnRO+3ntNYgzAZ1yqlimLACTOG4T+th oXrW4QY/dgTkKLBmub4kKmZWV/6eOTrkQKcC7gOkTzQnhCgnKEMZ48AGAkXrdsZuUetA +ZzYMNjF7x6VFAJ3yDH0yHfgrANWAGpuH+Kc69UaRhdGcaw50ymsJRSA0dn2qKFp0gY0 kLDy3bN7TjsnKko3AiRKxYAZBVlWTurCUbTk/k7ujJ5SqFA6Cr2eeSiHwBu8sl30wWOt O9C3zUxKAR8fYXJp4ZCGoRDfBr/fRejJVlBuqJY2E0dbzke8sKstX0UDjY67n9q8g6Zh 6/aw== X-Gm-Message-State: AFqh2kqH6YJU8+qc2iWwZMe06Tepw6UEwkTucaceic0ZMBeWdo1c5E/Q biHR3jdmyqJJNc3rDYKsDCuEiVO2+s0= X-Google-Smtp-Source: AMrXdXufHU+YEXa/HPbI4vLLyfNw163JJwFMEsJciRnPr1X88dc4ccEYYlMB1fPlEQ5O4fkaQPa8neGtLug= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:9246:0:b0:6f2:2d54:dcdf with SMTP id e6-20020a259246000000b006f22d54dcdfmr6080399ybo.144.1673297718997; Mon, 09 Jan 2023 12:55:18 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:34 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-40-surenb@google.com> Subject: [PATCH 39/41] kernel/fork: throttle call_rcu() calls in vm_area_free From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205536_369589_901A51CA X-CRM114-Status: GOOD ( 16.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. To minimize that impact, place VMAs into a list and free them in groups using one call_rcu() call per group. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 1 + include/linux/mm_types.h | 19 +++++++++-- kernel/fork.c | 68 +++++++++++++++++++++++++++++++++++----- mm/init-mm.c | 3 ++ mm/mmap.c | 1 + 5 files changed, 82 insertions(+), 10 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3158f33e268c..50c7a6dd9c7a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -250,6 +250,7 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +void drain_free_vmas(struct mm_struct *mm); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index fce9113d979c..c0e6c8e4700b 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -592,8 +592,18 @@ struct vm_area_struct { /* Information about our backing store: */ unsigned long vm_pgoff; /* Offset (within vm_file) in PAGE_SIZE units */ - struct file * vm_file; /* File we map to (can be NULL). */ - void * vm_private_data; /* was vm_pte (shared mem) */ + union { + struct { + /* File we map to (can be NULL). */ + struct file *vm_file; + + /* was vm_pte (shared mem) */ + void *vm_private_data; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct list_head vm_free_list; +#endif + }; #ifdef CONFIG_ANON_VMA_NAME /* @@ -693,6 +703,11 @@ struct mm_struct { */ #ifdef CONFIG_PER_VMA_LOCK int mm_lock_seq; + struct { + struct list_head head; + spinlock_t lock; + int size; + } vma_free_list; #endif diff --git a/kernel/fork.c b/kernel/fork.c index 6d9f14e55ecf..97f2b751f88d 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -481,26 +481,75 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) } #ifdef CONFIG_PER_VMA_LOCK -static void __vm_area_free(struct rcu_head *head) +static inline void __vm_area_free(struct vm_area_struct *vma) { - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); /* The vma should either have no lock holders or be write-locked. */ vma_assert_no_reader(vma); kmem_cache_free(vm_area_cachep, vma); } -#endif + +static void vma_free_rcu_callback(struct rcu_head *head) +{ + struct vm_area_struct *first_vma; + struct vm_area_struct *vma, *vma2; + + first_vma = container_of(head, struct vm_area_struct, vm_rcu); + list_for_each_entry_safe(vma, vma2, &first_vma->vm_free_list, vm_free_list) + __vm_area_free(vma); + __vm_area_free(first_vma); +} + +void drain_free_vmas(struct mm_struct *mm) +{ + struct vm_area_struct *first_vma; + LIST_HEAD(to_destroy); + + spin_lock(&mm->vma_free_list.lock); + list_splice_init(&mm->vma_free_list.head, &to_destroy); + mm->vma_free_list.size = 0; + spin_unlock(&mm->vma_free_list.lock); + + if (list_empty(&to_destroy)) + return; + + first_vma = list_first_entry(&to_destroy, struct vm_area_struct, vm_free_list); + /* Remove the head which is allocated on the stack */ + list_del(&to_destroy); + + call_rcu(&first_vma->vm_rcu, vma_free_rcu_callback); +} + +#define VM_AREA_FREE_LIST_MAX 32 + +void vm_area_free(struct vm_area_struct *vma) +{ + struct mm_struct *mm = vma->vm_mm; + bool drain; + + free_anon_vma_name(vma); + + spin_lock(&mm->vma_free_list.lock); + list_add(&vma->vm_free_list, &mm->vma_free_list.head); + mm->vma_free_list.size++; + drain = mm->vma_free_list.size > VM_AREA_FREE_LIST_MAX; + spin_unlock(&mm->vma_free_list.lock); + + if (drain) + drain_free_vmas(mm); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +void drain_free_vmas(struct mm_struct *mm) {} void vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); -#ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, __vm_area_free); -#else kmem_cache_free(vm_area_cachep, vma); -#endif } +#endif /* CONFIG_PER_VMA_LOCK */ + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { @@ -1150,6 +1199,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, INIT_LIST_HEAD(&mm->mmlist); #ifdef CONFIG_PER_VMA_LOCK WRITE_ONCE(mm->mm_lock_seq, 0); + INIT_LIST_HEAD(&mm->vma_free_list.head); + spin_lock_init(&mm->vma_free_list.lock); + mm->vma_free_list.size = 0; #endif mm_pgtables_bytes_init(mm); mm->map_count = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index 33269314e060..b53d23c2d7a3 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -39,6 +39,9 @@ struct mm_struct init_mm = { .mmlist = LIST_HEAD_INIT(init_mm.mmlist), #ifdef CONFIG_PER_VMA_LOCK .mm_lock_seq = 0, + .vma_free_list.head = LIST_HEAD_INIT(init_mm.vma_free_list.head), + .vma_free_list.lock = __SPIN_LOCK_UNLOCKED(init_mm.vma_free_list.lock), + .vma_free_list.size = 0, #endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, diff --git a/mm/mmap.c b/mm/mmap.c index 332af383f7cd..a0d5d3af1d95 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3159,6 +3159,7 @@ void exit_mmap(struct mm_struct *mm) trace_exit_mmap(mm); __mt_destroy(&mm->mm_mt); mmap_write_unlock(mm); + drain_free_vmas(mm); vm_unacct_memory(nr_accounted); } From patchwork Mon Jan 9 20:53:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094346 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D63BBC54EBD for ; Mon, 9 Jan 2023 21:20:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=n3g4GI7u1JwvJ8lePY5nNUau88rf9OnEZd1UcbySD/8=; b=Ihag+d6tpGA7YaLf/3m+9HLmlc 4kf67zozTvJ1pS9o7Wzu708kyz61HUA1xvKH7aogZxBT3LTLpyUA4eu92toTcFxEGlVmbbWcQxKDj KxidZE0yv6KcCaOLLJBF9PLL0h7nlAJa7v8EqA/manOgXXbrykpEmOqopDq2WuVSmiaakNdoSl0vR exfUAV2MM14/oASoeOk9Kl89WplZXnzbwj6KxlgIPICynoU/YFEtcO0yILDSDnpCkLwliS0nnuith s94Rzn3CT/lgi+4NNuo6YjsV5xCiWil2vqJqTgjLL8M/eYBOD76C6fxGzR+hAcRWqf3CFK7+zEVcy pP5hykrQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzXy-004ALW-75; Mon, 09 Jan 2023 21:19:11 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzTt-0048n5-Ld for linux-arm-kernel@bombadil.infradead.org; Mon, 09 Jan 2023 21:14:57 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=KsiQb/dE9yymRpEr0UBz2Qq4KLrRHehGb+mEU6hVzX0=; b=UEthq5gkwuyyNwpnyjxF07q+Vn SrOoBadQF/D+y2zzG2c8a11l6zGXy03huMBwrdeLy1NZBEp0wo0WaZhs5bk8Za3tdQB+BveWLGIq3 aR4xKVSj9sP2O/EWF7LGm6O63uVoWSnqIB9C2irI8IvMB/IGdABjLzDgistWyqMCcR0XES1MCqltJ Mcf6CFz1MEMxfFzPsA3qbL37JK7nmLFsYOIl0iMl+dnbeTC5cqDw5f3pAo8izRWC8l1qhHEyDheug a2vZ3Q2h0B4yvwvUTs4RODIR/GfyXvPVBooklsHv/3BTJKPGg0+hiZPMdLxON9omMZWP3J5M7tk0k xFGSaivg==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzBA-002cgV-Me for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:55:40 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-349423f04dbso105167757b3.13 for ; Mon, 09 Jan 2023 12:55:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KsiQb/dE9yymRpEr0UBz2Qq4KLrRHehGb+mEU6hVzX0=; b=sWtobSCBvoHV5TPkE/zYtWszgUW3jVRDiVjTt3sAcjFBaXlqjfdYiQ6AHG86rnNt0b mg+e+VoU+a89JO+rslq34FfyzRkAoGg7DuVDlMH1H1ePQ32MuB2+77t0gAtkJ+y+wX6p B/589/jsqLZApvm6BoQPTAcfoFYkEfWsjxaKMUYIa10AIrZvQul2s59ukdBVjUu/Ibrp egNG28U4wT5FZXec8pRJXGar6bKM6QyHpkRg0GkDbw6/Hl7olgQ7YiTkujC6hmd+0b7u 2fP91n1+O4wNRIiFMGmQ1g/TPPehatod7VMoh0dY0z9ogD+TX+MKDQ5sfW6QhS4fa9cZ O/Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KsiQb/dE9yymRpEr0UBz2Qq4KLrRHehGb+mEU6hVzX0=; b=fZPF0ADp9sWatS1UbeT9MQdyIPPT7USen6YLmHNNKPkZKe0mhABSaUGG9sLgru1GdM /XC2G8JOB3BQ2lT2rvMU+lKVKtNG+Oh2Sonbrmmb/0PhJOdWUcYJ4+UrmTJe6zBoXP3O 8Vd2co/qYvv3Zn4P6+6uBtFeFDeLrzOf70MrZY/jYhQAlUFaRfXQQPfUnwMj1MHgyYeg g8UdUQbzJS5MlyR5yvvxVc+RWix1Xs7cgSKUGlAOn3vKLiQeKq7hrJrvb708Z/TiUDPL m7qvwBW65CXVXDWSSbeplv+lOR2GZdzuU/ig103xj56iASHZpsUfodyBygFSigEJ1aJ6 nr7A== X-Gm-Message-State: AFqh2koDakp6p8mhGNUfYz1Jy1M6lWYl/XvUcYhPx4wF3H320D1QDghc kWsyVTQC+EocQPtPlBVkj3RSkpjL+64= X-Google-Smtp-Source: AMrXdXvB4GUb8flK/fogRdz9aeHP/yrxt0+V7dhaMMs1drQdYkqQuuAWfxuEu+0du5vQqeydWK6w8e3+WKU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:abc6:0:b0:755:29c5:63e with SMTP id v64-20020a25abc6000000b0075529c5063emr6840356ybi.142.1673297721512; Mon, 09 Jan 2023 12:55:21 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:35 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-41-surenb@google.com> Subject: [PATCH 40/41] mm: separate vma->lock from vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_205536_767189_16E5FD36 X-CRM114-Status: GOOD ( 20.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org vma->lock being part of the vm_area_struct causes performance regression during page faults because during contention its count and owner fields are constantly updated and having other parts of vm_area_struct used during page fault handling next to them causes constant cache line bouncing. Fix that by moving the lock outside of the vm_area_struct. All attempts to keep vma->lock inside vm_area_struct in a separate cache line still produce performance regression especially on NUMA machines. Smallest regression was achieved when lock is placed in the fourth cache line but that bloats vm_area_struct to 256 bytes. Considering performance and memory impact, separate lock looks like the best option. It increases memory footprint of each VMA but that will be addressed in the next patch. Note that after this change vma_init() does not allocate or initialize vma->lock anymore. A number of drivers allocate a pseudo VMA on the stack but they never use the VMA's lock, therefore it does not need to be allocated. The future drivers which might need the VMA lock should use vm_area_alloc()/vm_area_free() to allocate it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 25 ++++++------ include/linux/mm_types.h | 6 ++- kernel/fork.c | 82 ++++++++++++++++++++++++++++------------ 3 files changed, 74 insertions(+), 39 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 50c7a6dd9c7a..d40bf8a5e19e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -615,11 +615,6 @@ struct vm_operations_struct { }; #ifdef CONFIG_PER_VMA_LOCK -static inline void vma_init_lock(struct vm_area_struct *vma) -{ - init_rwsem(&vma->lock); - vma->vm_lock_seq = -1; -} static inline void vma_write_lock(struct vm_area_struct *vma) { @@ -635,9 +630,9 @@ static inline void vma_write_lock(struct vm_area_struct *vma) if (vma->vm_lock_seq == mm_lock_seq) return; - down_write(&vma->lock); + down_write(&vma->vm_lock->lock); vma->vm_lock_seq = mm_lock_seq; - up_write(&vma->lock); + up_write(&vma->vm_lock->lock); } /* @@ -651,17 +646,17 @@ static inline bool vma_read_trylock(struct vm_area_struct *vma) if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) return false; - if (unlikely(down_read_trylock(&vma->lock) == 0)) + if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) return false; /* * Overflow might produce false locked result. * False unlocked result is impossible because we modify and check - * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * vma->vm_lock_seq under vma->vm_lock protection and mm->mm_lock_seq * modification invalidates all existing locks. */ if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); return false; } return true; @@ -669,7 +664,7 @@ static inline bool vma_read_trylock(struct vm_area_struct *vma) static inline void vma_read_unlock(struct vm_area_struct *vma) { - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -684,7 +679,7 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) static inline void vma_assert_no_reader(struct vm_area_struct *vma) { - VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock) && + VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock) && vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } @@ -694,7 +689,6 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ -static inline void vma_init_lock(struct vm_area_struct *vma) {} static inline void vma_write_lock(struct vm_area_struct *vma) {} static inline bool vma_read_trylock(struct vm_area_struct *vma) { return false; } @@ -704,6 +698,10 @@ static inline void vma_assert_no_reader(struct vm_area_struct *vma) {} #endif /* CONFIG_PER_VMA_LOCK */ +/* + * WARNING: vma_init does not initialize vma->vm_lock. + * Use vm_area_alloc()/vm_area_free() if vma needs locking. + */ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -712,7 +710,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_init_lock(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index c0e6c8e4700b..faa61b400f9b 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -526,6 +526,10 @@ struct anon_vma_name { char name[]; }; +struct vma_lock { + struct rw_semaphore lock; +}; + /* * This struct describes a virtual memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory @@ -563,7 +567,7 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; - struct rw_semaphore lock; + struct vma_lock *vm_lock; #endif /* diff --git a/kernel/fork.c b/kernel/fork.c index 97f2b751f88d..95db6a521cf1 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -451,40 +451,28 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; -struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) -{ - struct vm_area_struct *vma; +#ifdef CONFIG_PER_VMA_LOCK - vma = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); - if (vma) - vma_init(vma, mm); - return vma; -} +/* SLAB cache for vm_area_struct.lock */ +static struct kmem_cache *vma_lock_cachep; -struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) +static bool vma_init_lock(struct vm_area_struct *vma) { - struct vm_area_struct *new = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); + vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); + if (!vma->vm_lock) + return false; - if (new) { - ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); - ASSERT_EXCLUSIVE_WRITER(orig->vm_file); - /* - * orig->shared.rb may be modified concurrently, but the clone - * will be reinitialized. - */ - *new = data_race(*orig); - INIT_LIST_HEAD(&new->anon_vma_chain); - vma_init_lock(new); - dup_anon_vma_name(orig, new); - } - return new; + init_rwsem(&vma->vm_lock->lock); + vma->vm_lock_seq = -1; + + return true; } -#ifdef CONFIG_PER_VMA_LOCK static inline void __vm_area_free(struct vm_area_struct *vma) { /* The vma should either have no lock holders or be write-locked. */ vma_assert_no_reader(vma); + kmem_cache_free(vma_lock_cachep, vma->vm_lock); kmem_cache_free(vm_area_cachep, vma); } @@ -540,6 +528,7 @@ void vm_area_free(struct vm_area_struct *vma) #else /* CONFIG_PER_VMA_LOCK */ +static bool vma_init_lock(struct vm_area_struct *vma) { return true; } void drain_free_vmas(struct mm_struct *mm) {} void vm_area_free(struct vm_area_struct *vma) @@ -550,6 +539,48 @@ void vm_area_free(struct vm_area_struct *vma) #endif /* CONFIG_PER_VMA_LOCK */ +struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) +{ + struct vm_area_struct *vma; + + vma = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); + if (!vma) + return NULL; + + vma_init(vma, mm); + if (!vma_init_lock(vma)) { + kmem_cache_free(vm_area_cachep, vma); + return NULL; + } + + return vma; +} + +struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) +{ + struct vm_area_struct *new; + + new = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); + if (!new) + return NULL; + + ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); + ASSERT_EXCLUSIVE_WRITER(orig->vm_file); + /* + * orig->shared.rb may be modified concurrently, but the clone + * will be reinitialized. + */ + *new = data_race(*orig); + if (!vma_init_lock(new)) { + kmem_cache_free(vm_area_cachep, new); + return NULL; + } + INIT_LIST_HEAD(&new->anon_vma_chain); + dup_anon_vma_name(orig, new); + + return new; +} + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { @@ -3138,6 +3169,9 @@ void __init proc_caches_init(void) NULL); vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT); +#ifdef CONFIG_PER_VMA_LOCK + vma_lock_cachep = KMEM_CACHE(vma_lock, SLAB_PANIC|SLAB_ACCOUNT); +#endif mmap_init(); nsproxy_cache_init(); } From patchwork Mon Jan 9 20:53:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DAA76C54EBD for ; Mon, 9 Jan 2023 21:06:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=6zx8HM3s35qouGTtXkMB9iKcaUMZnVif/P5aVSX6IfU=; b=gb2b5GjV0BF9dmBo0PGolt/Xdi /pVYZqYvoK11oGY4m/Pw6zEjcSBueOVOe/kcYdLtdoaLOWSNVRzvpQ4z7M3WJQgO/EAZ5KNRFhWal bTkl8dUsVaU/Zk9NnxZ1S68TLbymoW70iNHJuo4u4knsT6oAf8aK6aAhIbbxXjeitJzLdwiib6F1K 2tClgb5ARnN2EzXxtLbUwjcB2z4leGipN3ACfJEb53ywGzbb76yetB7y+m7mzLNqftSXhNFGxtybR 4n72tFKJttxDrRHnZpCT7MJ/TNWE3rEYFmARetlS7/YjzcXwgaArH62XRKm5z7bvK7bQ3S2vhp7NU nWvw79cA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzKs-0045Nc-Uw; Mon, 09 Jan 2023 21:05:39 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzHl-004489-Eb for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 21:02:27 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-46839d9ca5dso105162727b3.16 for ; Mon, 09 Jan 2023 13:02:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6eLzL6RHeFdZbRB5oFC0iJ795TBzqFT5ovI/MwJrKVk=; b=J47RXHly65L6DdSqhS16NmYDveuNz+mvTNG4sGvbQhUoWkeX6DBvsEbauZWeZvA/FL 6iJWwlnZTOJwBfCXcZD78WFVJCG7QdSQyabstEWgAbqGHA3DDp9ZOq1VOAfH1PKWr803 hD+hbwJSt9HgzHQXXidFA8sLeLstkKuaZuBnozg86r7c8tgk1iKjkQ1PqFU36BM8+uUF XrYI9BSRx6s/UM8T/tSpiSH9gONtyaFYLkfVKHbhv2aDCwIEnthDk+4faf5N9RzawpoY Qe27f6FOB5GsQ/NQQedhE4ZJmkp3i9Bbttmr9VE8Pj62jtmxe5G6ZqBn6CUeB13gsJFT GliQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6eLzL6RHeFdZbRB5oFC0iJ795TBzqFT5ovI/MwJrKVk=; b=B82waYDYH0UedQG8/bQ21STBwt6mUOfE9I+T3yXOXdOiec1n3uYUxhbVgwNKsOVERI d2EYfsdCXmE++BBIr6vCBbaHkDCVzKRr+mxMcUXKXSKe7rTwBDUOFbBa8UBmkuA5V8D4 1LrNCZnXCPWCmeA7Zw8BKNTZXG7ZOl05qItH2uN5oioSxT+CL2WMg5a2KMe2daHzTzTo RMq4BLW2JWCASk6lYwFl6O7cAJzYOjNw86/gxg7Tn/h8kFZ4lNtyGLdQCE1OefaPi3ko /480bzCoLqMju67MJQyt/GiQ9rSr5pkjhcFMgeZ4c2gXiKKb+K0N8xzyTE6EpkYh890s 3lYA== X-Gm-Message-State: AFqh2krUIm9WvkLcpa31fufQ5qFhxF4IivLVw45PT9M2LQPTHxXbTIKp 1/g3oMjOjtMLfnLqMMlZ3auRRiwtryA= X-Google-Smtp-Source: AMrXdXvrHv3ZAwS3fQ/OCRP/gup9EskfAqpVKQwrbofhFp/7IXGRuSpH+yzdGI9HlCLagGnSix+y0aiJr6A= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a5b:305:0:b0:707:473f:b763 with SMTP id j5-20020a5b0305000000b00707473fb763mr7071095ybp.158.1673297724036; Mon, 09 Jan 2023 12:55:24 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:36 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-42-surenb@google.com> Subject: [PATCH 41/41] mm: replace rw_semaphore with atomic_t in vma_lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_130225_512540_F978D0B8 X-CRM114-Status: GOOD ( 22.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org rw_semaphore is a sizable structure of 40 bytes and consumes considerable space for each vm_area_struct. However vma_lock has two important specifics which can be used to replace rw_semaphore with a simpler structure: 1. Readers never wait. They try to take the vma_lock and fall back to mmap_lock if that fails. 2. Only one writer at a time will ever try to write-lock a vma_lock because writers first take mmap_lock in write mode. Because of these requirements, full rw_semaphore functionality is not needed and we can replace rw_semaphore with an atomic variable. When a reader takes read lock, it increments the atomic unless the value is negative. If that fails read-locking is aborted and mmap_lock is used instead. When writer takes write lock, it resets atomic value to -1 if the current value is 0 (no readers). Since all writers take mmap_lock in write mode first, there can be only one writer at a time. If there are readers, writer will place itself into a wait queue using new mm_struct.vma_writer_wait waitqueue head. The last reader to release the vma_lock will signal the writer to wake up. vm_lock_seq is also moved into vma_lock and along with atomic_t they are nicely packed and consume 8 bytes, bringing the overhead from vma_lock from 44 to 16 bytes: slabinfo before the changes: ... : ... vm_area_struct ... 152 53 2 : ... slabinfo with vma_lock: ... : ... rw_semaphore ... 8 512 1 : ... vm_area_struct ... 160 51 2 : ... Assuming 40000 vm_area_structs, memory consumption would be: baseline: 6040kB vma_lock (vm_area_structs+vma_lock): 6280kB+316kB=6596kB Total increase: 556kB atomic_t might overflow if there are many competing readers, therefore vma_read_trylock() implements an overflow check and if that occurs it restors the previous value and exits with a failure to lock. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 37 +++++++++++++++++++++++++------------ include/linux/mm_types.h | 10 ++++++++-- kernel/fork.c | 6 +++--- mm/init-mm.c | 2 ++ 4 files changed, 38 insertions(+), 17 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d40bf8a5e19e..294dd44b2198 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -627,12 +627,16 @@ static inline void vma_write_lock(struct vm_area_struct *vma) * mm->mm_lock_seq can't be concurrently modified. */ mm_lock_seq = READ_ONCE(vma->vm_mm->mm_lock_seq); - if (vma->vm_lock_seq == mm_lock_seq) + if (vma->vm_lock->lock_seq == mm_lock_seq) return; - down_write(&vma->vm_lock->lock); - vma->vm_lock_seq = mm_lock_seq; - up_write(&vma->vm_lock->lock); + if (atomic_cmpxchg(&vma->vm_lock->count, 0, -1)) + wait_event(vma->vm_mm->vma_writer_wait, + atomic_cmpxchg(&vma->vm_lock->count, 0, -1) == 0); + vma->vm_lock->lock_seq = mm_lock_seq; + /* Write barrier to ensure lock_seq change is visible before count */ + smp_wmb(); + atomic_set(&vma->vm_lock->count, 0); } /* @@ -643,20 +647,28 @@ static inline void vma_write_lock(struct vm_area_struct *vma) static inline bool vma_read_trylock(struct vm_area_struct *vma) { /* Check before locking. A race might cause false locked result. */ - if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) + if (vma->vm_lock->lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) return false; - if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) + if (unlikely(!atomic_inc_unless_negative(&vma->vm_lock->count))) return false; + /* If atomic_t overflows, restore and fail to lock. */ + if (unlikely(atomic_read(&vma->vm_lock->count) < 0)) { + if (atomic_dec_and_test(&vma->vm_lock->count)) + wake_up(&vma->vm_mm->vma_writer_wait); + return false; + } + /* * Overflow might produce false locked result. * False unlocked result is impossible because we modify and check * vma->vm_lock_seq under vma->vm_lock protection and mm->mm_lock_seq * modification invalidates all existing locks. */ - if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { - up_read(&vma->vm_lock->lock); + if (unlikely(vma->vm_lock->lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { + if (atomic_dec_and_test(&vma->vm_lock->count)) + wake_up(&vma->vm_mm->vma_writer_wait); return false; } return true; @@ -664,7 +676,8 @@ static inline bool vma_read_trylock(struct vm_area_struct *vma) static inline void vma_read_unlock(struct vm_area_struct *vma) { - up_read(&vma->vm_lock->lock); + if (atomic_dec_and_test(&vma->vm_lock->count)) + wake_up(&vma->vm_mm->vma_writer_wait); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -674,13 +687,13 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) * current task is holding mmap_write_lock, both vma->vm_lock_seq and * mm->mm_lock_seq can't be concurrently modified. */ - VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); + VM_BUG_ON_VMA(vma->vm_lock->lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } static inline void vma_assert_no_reader(struct vm_area_struct *vma) { - VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock) && - vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), + VM_BUG_ON_VMA(atomic_read(&vma->vm_lock->count) > 0 && + vma->vm_lock->lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index faa61b400f9b..a6050c38ca2e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -527,7 +527,13 @@ struct anon_vma_name { }; struct vma_lock { - struct rw_semaphore lock; + /* + * count > 0 ==> read-locked with 'count' number of readers + * count < 0 ==> write-locked + * count = 0 ==> unlocked + */ + atomic_t count; + int lock_seq; }; /* @@ -566,7 +572,6 @@ struct vm_area_struct { unsigned long vm_flags; #ifdef CONFIG_PER_VMA_LOCK - int vm_lock_seq; struct vma_lock *vm_lock; #endif @@ -706,6 +711,7 @@ struct mm_struct { * by mmlist_lock */ #ifdef CONFIG_PER_VMA_LOCK + struct wait_queue_head vma_writer_wait; int mm_lock_seq; struct { struct list_head head; diff --git a/kernel/fork.c b/kernel/fork.c index 95db6a521cf1..b221ad182d98 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -461,9 +461,8 @@ static bool vma_init_lock(struct vm_area_struct *vma) vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); if (!vma->vm_lock) return false; - - init_rwsem(&vma->vm_lock->lock); - vma->vm_lock_seq = -1; + atomic_set(&vma->vm_lock->count, 0); + vma->vm_lock->lock_seq = -1; return true; } @@ -1229,6 +1228,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); #ifdef CONFIG_PER_VMA_LOCK + init_waitqueue_head(&mm->vma_writer_wait); WRITE_ONCE(mm->mm_lock_seq, 0); INIT_LIST_HEAD(&mm->vma_free_list.head); spin_lock_init(&mm->vma_free_list.lock); diff --git a/mm/init-mm.c b/mm/init-mm.c index b53d23c2d7a3..0088e31e5f7e 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -38,6 +38,8 @@ struct mm_struct init_mm = { .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), #ifdef CONFIG_PER_VMA_LOCK + .vma_writer_wait = + __WAIT_QUEUE_HEAD_INITIALIZER(init_mm.vma_writer_wait), .mm_lock_seq = 0, .vma_free_list.head = LIST_HEAD_INIT(init_mm.vma_free_list.head), .vma_free_list.lock = __SPIN_LOCK_UNLOCKED(init_mm.vma_free_list.lock),